@polycode-projects/the-mechanical-code-talker 2.10.2 → 2.10.5

package/README.md

@@ -120,13 +120,18 @@ tmct> /exit
 
 **[Try it live in your browser →](https://polycode-projects.gitlab.io/the-mechanical-code-talker/)**
 runs the actual query engine client-side. No server, no install. The landing
-page answers codebase questions live, and six more pages each ground their
-own domain: a full chat seeded with 19,228 facts (the same nine bands as
-`npm run init:xl`, capped to a 22.8 MB download), the **memory ledger**
-(every fact as a readable sentence; drill by clicking the terms inside), a
-Towers-of-Hanoi plan replayed move by move, the spider-and-fly and
-text-adventure games, and a sprite gallery whose chat dock answers from
-1,033 generated sprite facts.
+page answers codebase questions live, and eight more pages each ground their
+own domain: a full chat seeded with 32,646 facts (the same nine bands as
+`npm run init:xl`, capped to a 38.7 MB download, 2.1 MB on the wire), the
+**memory ledger** (every fact as a readable sentence; drill by clicking the
+terms inside), the **code explorer** (the same ledger UI refocused on a code
+graph, with a hint rail of suggested next questions), an **ingest page** that
+turns pasted or dropped text into grounded facts and downloads them as
+canonical JSONL, a Towers-of-Hanoi plan replayed move by move, the
+spider-and-fly and text-adventure games, and a sprite gallery whose chat dock
+answers from 1,033 generated sprite facts. The chat page and the ledger take
+the same paste-or-drop text in place; every page that holds a fact store
+exports it as JSONL.
 The site hosts its own copy of wink-nlp, ships its assets precompressed,
 and a service worker precaches the big ones, so a second visit works
 offline. `tmct chat --render spider-fly|adventure|sprites [--output <path>]`
@@ -166,6 +171,11 @@ over the same graph. A hint rail suggests the next question from what the graph
 actually holds — "what does X import", "which functions call Y", "list
 functions" — so every suggestion resolves to a real answer.
 
+The identical page also runs as a plain hosted page, over the demo code
+graph — **[try it live →](https://polycode-projects.gitlab.io/the-mechanical-code-talker/code.html)**
+— for a look with nothing to install. The desktop build below is for
+exploring your own repo or graph, which the hosted page cannot reach.
+
 Electron is a dev-only dependency and never ships in the npm package. Because
 `.npmrc` sets `ignore-scripts=true`, installing it does not fetch the runtime
 binary; fetch it once, then build and launch:
@@ -469,6 +479,11 @@ cited read-out, CC BY-SA, pinned to the revision it read (provenance
 `reference:wikipedia-live:<Title>@<revid>`). A failed lookup (no matching
 title, a timeout, a rate limit) leaves the honest miss byte-identical.
 
+A live article carries its own source-type prior (`referenceLive`, 0.5) below
+the curated revision-pinned pack (`reference`, 0.6). A fact fetched live never
+outranks the shipped article on the same term, and a fact you teach outranks
+both.
+
 ## How it remembers
 
 tmct's memory has two layers, both fed by every parsed request and response and
@@ -479,6 +494,11 @@ by cleaned session logs:
 - **text blocks under a PageRank-style index**, pulled into context on
   relevance rather than loaded wholesale.
 
+Every session also writes its own human-readable transcript,
+`.tmct/session-<id>.md` — a glow-friendly Markdown file with one heading per
+turn, the question as a blockquote, the reply in a fenced block. The
+browser chat page's "export .md" button writes the same shape.
+
 With no graph at all, tmct starts empty and remembers what you tell it. The
 `.tmct/` graph is created from the conversation. On a first run it seeds the
 committed vocabulary so it knows what it's talking about from turn one: an
@@ -531,6 +551,17 @@ rather than the code graph. A taught class answers both shapes too: after
 "dog is a kind of animal", "how many animals are there" counts its members and
 "list all animals" reads them back, each cited to where it came from.
 
+When you ask about a term, the read-back shows each "is a kind of" object with
+its own superclass chain: "what is rover" answers "rover is a kind of dog →
+canine → mammal → animal". If one label carries two unrelated senses — you
+taught "rover is a dog" and a corpus row says "rover is a scout" — the answer
+groups by concept ("rover, the dog:" / "rover, the scout:") instead of listing
+two unrelated lines as if they were one thing. The split is deterministic over
+the stored hierarchy: two senses part when a stored disjointness separates
+their ancestors, when their chains never meet, or when they meet only at the
+very top. When the evidence is thin the answer stays a flat list — grouping is
+presentation, and it never retracts or reranks a fact.
+
 Teaching doesn't have to be typed, either. `tmct extract` runs a plain text
 file through the same recognizer the chat's teach lane uses. Sentences the
 recognizer grounds become fact rows; everything else is skipped and counted,
@@ -551,6 +582,20 @@ Pass `--repo <path>` instead to write the recognized facts straight into
 that repo's memory, or `--out <file.jsonl>` to save the rows. Each one
 carries an `extracted:<file>` provenance tag at its own trust tier.
 
+`--optimistic` adds a second, lower-trust tier over the sentences the strict
+recognizer skips: a copula or a known relation verb flanked by two nouns
+becomes a candidate triple, stored under its own `optimistic-extract:<file>`
+provenance (prior 0.35, below every curated pack) with no operator tag riding
+alongside — so a fuzzy guess can never corroborate a curated fact. It is an
+attempt, not full NLU: a sentence with no clean pair yields nothing.
+`--canonical` prints each grounded fact as a triple, noting how each endpoint
+already links into the store.
+
+The same pipeline is one library seam, `ingestText(text, options)` (exported
+as `@polycode-projects/the-mechanical-code-talker/ingest`), and one cold tool,
+`tmct_ingest` — so a browser page, a script, or a tool-calling agent can ground
+text without the CLI.
+
 ### Provenance and trust
 
 Every fact and text block records **where it came from and when**. Sources are
@@ -571,8 +616,15 @@ provenance** rather than silently picking a winner.
 batch: a forward-chaining materialisation over the memory's OWL 2 RL rule
 kernels (the classical syllogism is one of them, and the verb keeps Aristotle's
 broader sense; see the bibliography) that writes new **entailed** facts. They
-are low-trust and retractable, never outranking a stated fact, and this never
-runs on the chat's hot path.
+are low-trust and retractable, never outranking a stated fact.
+
+The full-store batch is a maintenance job, off the chat's hot path. One bounded
+sibling does run inline: when a learn-on-miss load pulls new facts in (a child
+pack, a reference or live-Wikipedia article), a small focus-scoped pass around
+the loaded term connects those new facts to what's already remembered, so a load
+becomes durable knowledge rather than an island. It shares the same kernels, the
+same `entailed:*` provenance and the same low, retractable trust; only its scope
+and budget shrink.
 
 ## Install & use
 
@@ -711,6 +763,9 @@ above — the same teach recognizer, reading a file instead of your typing:
        [--repo <abs>]          write the facts into that repo's own tmct memory; without
                                it nothing on disk is mutated and the facts print as JSONL
        [--out <file.jsonl>]    write that JSONL to a file instead of stdout
+       [--optimistic]          also run a lower-trust fuzzy tier over the sentences the
+                               strict recognizer skips; candidates rank below every curated pack
+       [--canonical]           print each grounded fact as a triple linked into the store
 ```
 
 `tmct extend --validate` checks a third-party extension pack's declared resources
@@ -1009,7 +1064,7 @@ loads a repo's graph into a `{ dispatch, resolve, graph }` context,
 
 ## The tool surface
 
-Everything above runs on the same 24 tools. Each one is read-only, answers one question in a single call, and returns bounded output. None of them calls a model. A tool that cannot ground an answer says so — the same honest miss you get everywhere else in tmct.
+Everything above runs on the same 25 tools. Each one is read-only, answers one question in a single call, and returns bounded output. None of them calls a model. A tool that cannot ground an answer says so — the same honest miss you get everywhere else in tmct.
 
 Three of them are **hot**: their schemas stay resident, so an agent driving tmct sees them every turn and reaches for one call instead of a Read/Grep loop.
 
@@ -1082,6 +1137,7 @@ The remaining tools are **cold**: still served, but not billed to an agent every
 | `tmct_cochanges` | Modules that historically change in the same commit as a symbol's module (git co-change). | `symbol` (required) |
 | `tmct_context_more` | The bundle sections a lean tmct_context omitted (siblings / tests / cochange / class members / re-exports). | `symbol` (required) |
 | `tmct_export` | Every stored memory fact as JSONL (subject/predicate/object/provenance) — the shape `tmct extract` emits, for backup or audit. | none |
+| `tmct_ingest` | Ground plain text into memory facts with the same deterministic recognizer the chat teach lane uses — the strict tier, plus an optional lower-trust fuzzy tier — and report the canonical triples. | `text` (required), `optimistic` |
 
 Add `repo_path` to any of them to point at a repository other than the working directory. `tmct init` also writes this catalog, with a worked invocation per tool, to `.tmct/TOOLS.md` inside the repo it indexed.
 

package/bin/tmct.mjs

@@ -55,7 +55,7 @@ Usage:
 ${renderUsage()}
 
 On a terminal, chat opens the full-screen TUI; piped input gets the plain shell.
-In chat: /help lists slash-commands; /exit leaves. Session log → <repo>/.tmct/session-<id>.log.
+In chat: /help lists slash-commands; /exit leaves. Session log → <repo>/.tmct/session-<id>.md.
 
 Shared graph-path precedence (chat/serve/cli; see src/services/cli-args.mjs): --graph flag(s) >
 TMCT_GRAPH_FILE env > tmct.toml graph_file/graph_files > --repo-derived
@@ -266,7 +266,10 @@ async function runCliMode() {
     }
     const config = await configFor(args.repo_path);
     try {
-      const text = await dispatchTool(sub, args, { config });
+      // The recognizer seam a tool like tmct_ingest needs: injected here (bin
+      // sits above the service layer) rather than imported by the tool layer.
+      const { ingestText } = await import("../src/services/extract-facts.mjs");
+      const text = await dispatchTool(sub, args, { config, ingest: ingestText });
       process.stdout.write(text + "\n");
     } catch (e) {
       process.stderr.write(`tmct: ${e?.message || e}\n`);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@polycode-projects/the-mechanical-code-talker",
-  "version": "2.10.2",
+  "version": "2.10.5",
   "private": false,
   "type": "module",
   "description": "The Mechanical Code Talker (tmct) — a tolerant, offline, $0 chat surface that guides you toward precision queries about a software repository. ELIZA/PARRY-style but domain-obsessed with code. No model calls; no codebase index of its own.",
@@ -45,6 +45,7 @@
     "./plan": "./src/domain/router/drive.mjs",
     "./generateCompletion": "./src/services/completions.mjs",
     "./createCompletionsGraphAdapter": "./src/services/completions.mjs",
+    "./ingest": "./src/services/extract-facts.mjs",
     "./repository-interface": "./src/adapters/repository-interface.mjs",
     "./conformance": "./src/tools/conformance.mjs"
   },
@@ -139,6 +140,7 @@
     "build:spider-fly-bundle": "node scripts/build-spider-fly-bundle.mjs",
     "build:plan-bundle": "node scripts/build-plan-bundle.mjs",
     "build:ledger-bundle": "node scripts/build-ledger-bundle.mjs",
+    "build:ingest-bundle": "node scripts/build-ingest-bundle.mjs",
     "build:code-explorer-bundle": "node scripts/build-code-explorer-bundle.mjs",
     "build:electron": "node scripts/build-electron-app.mjs",
     "electron": "electron electron/main.mjs",

package/src/adapters/memory/core.mjs

@@ -94,7 +94,7 @@ const MEMORY_VOCABULARY = [
   { prop: DERIVED_FROM_PROP, predicate: "derivedFrom", note: "umbrella: a Fact derived from a Source (or another Fact). ext ref prov:wasDerivedFrom (UNVERIFIED-pending-web-check)" },
   { prop: STATED_BY_PROP, predicate: "statedBy", note: "subPropertyOf derivedFrom: a Source directly asserts this Fact (one edge per independent source — replaces the factProvenance union)" },
   { prop: CANONICALISED_FROM_PROP, predicate: "canonicalisedFrom", note: "subPropertyOf derivedFrom: a canonical Fact cleaned from a raw Block/Source, never replacing it" },
-  { prop: "mgx:sourceType", note: "a Source's kind: operator | teach | provider | corpus | corpusWeak | reference | extracted | web | entailed (the trust-prior key)" },
+  { prop: "mgx:sourceType", note: "a Source's kind: operator | teach | provider | corpus | corpusWeak | reference | referenceLive | extracted | optimisticExtract | web | entailed (the trust-prior key)" },
   { prop: "mgx:sourceUrl", note: "a web Source's URL" },
   { prop: "mgx:sourceRule", note: "an entailed Source's rule id" },
   { prop: "mgx:sourceReliability", note: "actor-level (session-scoped) trust nudge in [0.5,1.5], neutral 1.0 when absent — materialised by recomputeSourceReliability from a session's asserted-vs-contradicted track record (memory/trust.mjs's sessionReliabilityFrom); folds into computeTrust's per-source prior" },
@@ -772,8 +772,13 @@ function sourceIdFor(desc) {
     // One Source per pack article (the @revid stays in the article segment),
     // so two facts from the same article corroborate nothing extra.
     case "reference": return { id: `src:reference:${desc.pack}:${desc.article}`, type: "reference" };
+    // The live-Wikipedia pack: same per-article Source id, but a lower trust
+    // type so a live lookup ranks below the curated revision-pinned pack.
+    case "referenceLive": return { id: `src:reference:${desc.pack}:${desc.article}`, type: "referenceLive" };
     // One Source per source-file basename, not per extraction run.
     case "extracted": return { id: `src:extracted:${desc.name}`, type: "extracted" };
+    // The fuzzy tier's candidates: one low-trust Source per source label.
+    case "optimisticExtract": return { id: `src:optimistic-extract:${desc.name}`, type: "optimisticExtract" };
     case "web": return { id: `src:learned:web:${fnv1aHex(String(desc.url || ""))}`, type: "web", url: String(desc.url || "") };
     case "entailed": return { id: `src:entailed:${desc.rule}`, type: "entailed", rule: String(desc.rule || "") };
     default: return null;
@@ -794,8 +799,10 @@ const PROV_CLASS_BY_SOURCE_TYPE = Object.freeze({
   corpus: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
   corpusWeak: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
   reference: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
+  referenceLive: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
   web: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
   extracted: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
+  optimisticExtract: { subClass: "tmct:DocumentSource", prov: "prov:Entity" },
   entailed: { subClass: "tmct:ActivitySource", prov: "prov:Activity" },
 });
 

package/src/domain/cli-verbs.mjs

@@ -92,6 +92,8 @@ export const CLI_VERBS = [
       { flag: "[--file <text-file>]", prose: ["teach recognizer and keep the facts it grounds; every", "other sentence is skipped and counted, never paraphrased"] },
       { flag: "[--repo <abs>]", prose: ["write the facts into that repo's own tmct memory; without", "it nothing on disk is mutated and the facts print as JSONL"] },
       { flag: "[--out <file.jsonl>]", prose: ["write that JSONL to a file instead of stdout"] },
+      { flag: "[--optimistic]", prose: ["also run a lower-trust fuzzy tier over the sentences the", "strict recognizer skips; candidates rank below every curated pack"] },
+      { flag: "[--canonical]", prose: ["print each grounded fact as a triple linked into the store"] },
     ],
   },
   {

package/src/domain/domain.mjs

@@ -5,9 +5,18 @@
 // arrives as data from the memory store's fact/Rule rows. Plugs into
 // planning.mjs's findActionPath as its applyActions.
 
+import { provenanceTagToSource } from "./memory/trust.mjs";
+
 const MEMBER_EDGE_PREDICATES = new Set(["rdfs:subClassOf", "rdf:type"]);
 const SNAPSHOT_RE = /^(.+)@step(\d+)$/;
 
+/** The class-member list a grounding step ranges over. The default "all" is
+ *  every typed member; "taught" restricts to instances a person taught, the
+ *  scope the plan lane uses so a merged corpus never widens a quantified goal
+ *  or the movable set. Falls back to nothing for an unknown scope. */
+const membersInScope = (domain, cls, scope) =>
+  (scope === "taught" ? domain.taughtClassMembers : domain.classMembers)?.[cls] || [];
+
 /** Trim a taught term defensively: some teach frames keep a sentence's
  *  trailing punctuation in the captured object. */
 const normTerm = (value) => String(value ?? "").trim().replace(/[.!?]+$/, "");
@@ -110,18 +119,34 @@ export function compileDomain(factRows, ruleRows) {
 
   // Class membership from typing edges. A member is a subject with a typing
   // edge into the class and no typing edge pointing at itself (a leaf).
-  const edges = (factRows || []).map(normRow).filter((r) => MEMBER_EDGE_PREDICATES.has(r.predicate));
-  const hasIncoming = new Set(edges.map((r) => r.object));
-  const classMembers = {};
-  for (const edge of edges) {
-    if (hasIncoming.has(edge.subject)) continue;
-    (classMembers[edge.object] ??= []).push(edge.subject);
-  }
-  for (const members of Object.values(classMembers)) {
-    members.sort();
-    // de-dup while keeping order
-    for (let i = members.length - 1; i > 0; i -= 1) if (members[i] === members[i - 1]) members.splice(i, 1);
-  }
+  const membersFromEdges = (typingEdges) => {
+    const hasIncoming = new Set(typingEdges.map((r) => r.object));
+    const members = {};
+    for (const edge of typingEdges) {
+      if (hasIncoming.has(edge.subject)) continue;
+      (members[edge.object] ??= []).push(edge.subject);
+    }
+    for (const list of Object.values(members)) {
+      list.sort();
+      // de-dup while keeping order
+      for (let i = list.length - 1; i > 0; i -= 1) if (list[i] === list[i - 1]) list.splice(i, 1);
+    }
+    return members;
+  };
+  const typingRows = (factRows || []).filter((r) => MEMBER_EDGE_PREDICATES.has(normTerm(r.predicate)));
+  const edges = typingRows.map(normRow);
+  const classMembers = membersFromEdges(edges);
+  // The same membership, restricted to instances a person taught (an operator
+  // assert or a teach-lane frame). A planner that quantifies over a class —
+  // "every X" — ranges over the taught individuals only: a merged corpus can
+  // type dozens of unrelated things into that class, and sweeping those into
+  // the goal makes it unsatisfiable and the search fruitless. classMembers
+  // keeps every member for the readers that want them all.
+  const taughtEdges = edges.filter((_, i) => {
+    const kind = provenanceTagToSource(typingRows[i].provenance)?.kind;
+    return kind === "operator" || kind === "teach";
+  });
+  const taughtClassMembers = membersFromEdges(taughtEdges);
 
   // A class-bound word (an effect role or constraint term that is neither
   // "subject" nor "target") is substituted by its class's sole member at
@@ -158,7 +183,7 @@ export function compileDomain(factRows, ruleRows) {
     .filter((r) => !dynamicPredicates.has(r.predicate) && !MEMBER_EDGE_PREDICATES.has(r.predicate))
     .sort(rowSort);
 
-  return { actions, classMembers, dynamicPredicates, ordering };
+  return { actions, classMembers, taughtClassMembers, dynamicPredicates, ordering };
 }
 
 const domainIndividuals = (domain) => {
@@ -312,13 +337,16 @@ function constraintViolated(action, nextState, domain) {
 }
 
 /** Every legal grounded action from `state`, with its successor.
- *  Deterministic: actions, signatures, and members are walked sorted. */
-export function movesFromRules(state, domain, { budget = 5000 } = {}) {
+ *  Deterministic: actions, signatures, and members are walked sorted. `scope`
+ *  ("all" | "taught") picks which class members ground the moves — the plan
+ *  lane passes "taught" so corpus members of a taught class never enter the
+ *  movable set. */
+export function movesFromRules(state, domain, { budget = 5000, scope = "all" } = {}) {
   let groundings = 0;
   for (const action of domain.actions) {
     for (const sig of action.signatures) {
-      groundings += (domain.classMembers[sig.subjectClass] || []).length *
-        (domain.classMembers[sig.targetClass] || []).length;
+      groundings += membersInScope(domain, sig.subjectClass, scope).length *
+        membersInScope(domain, sig.targetClass, scope).length;
     }
   }
   if (groundings > budget) throw new PlanBudgetError(groundings, budget);
@@ -327,8 +355,8 @@ export function movesFromRules(state, domain, { budget = 5000 } = {}) {
   for (const action of domain.actions) {
     const [verb, particle] = action.name.split(/\s+/);
     for (const sig of action.signatures) {
-      for (const subject of domain.classMembers[sig.subjectClass] || []) {
-        for (const target of domain.classMembers[sig.targetClass] || []) {
+      for (const subject of membersInScope(domain, sig.subjectClass, scope)) {
+        for (const target of membersInScope(domain, sig.targetClass, scope)) {
           if (subject === target) continue;
           let ok = true;
           for (const precond of action.preconds) {
@@ -360,7 +388,7 @@ export function movesFromRules(state, domain, { budget = 5000 } = {}) {
  *  state predicate. Satisfaction is a transitive walk along the goal
  *  predicate: a stacked member reaches the goal object through its support
  *  chain, which a direct row lookup cannot see. */
-export function compileGoal(goalSpecs, domain) {
+export function compileGoal(goalSpecs, domain, { scope = "all" } = {}) {
   const specs = (goalSpecs || []).map((g) => ({
     universal: Boolean(g.universal),
     term: normTerm(g.term),
@@ -369,7 +397,7 @@ export function compileGoal(goalSpecs, domain) {
   }));
   const checks = [];
   for (const spec of specs) {
-    const members = spec.universal ? domain.classMembers[spec.term] || [] : [spec.term];
+    const members = spec.universal ? membersInScope(domain, spec.term, scope) : [spec.term];
     if (spec.universal && members.length === 0) {
       throw new Error(`the goal names "${spec.term}" as a class, but it has no known members`);
     }

package/src/domain/memory/trust.mjs

@@ -46,10 +46,25 @@ function parseChatTagRest(rest) {
  *   reference:<pack>:<article>[@revid] -> { kind:"reference", pack, article }
  *     (split on the first two colons only; the article keeps any @revid and
  *      any spaces — "reference:simplewiki:Polar bear@912" stays one article)
+ *     EXCEPT the live-Wikipedia pack: reference:wikipedia-live:<article> parses
+ *      as kind "referenceLive", which scores below the curated revision-pinned
+ *      pack, so a live lookup never outranks the shipped article on the same term.
  *   extracted:<file-basename>  -> { kind:"extracted", name:<file-basename> }
+ *   optimistic-extract:<file-basename> -> { kind:"optimisticExtract", name }
+ *     (the fuzzy tier of `tmct extract --optimistic`: a candidate the strict
+ *      recognizer skipped, stored under its OWN low prior so it never
+ *      corroborates a curated pack — no operator/teach tag rides alongside)
  *   entailed:<rule>            -> { kind:"entailed",  rule:<rule> }
  * chat:/session: refs map to the operator; an unknown tag -> null (no Source).
  */
+// The one reference pack whose content is fetched live at query time rather
+// than shipped revision-pinned. Its facts score at the referenceLive prior
+// (below curated `reference`), so a live lookup never outranks the shipped pack.
+// Must equal reference-pack.mjs's LIVE_PACK_NAME; kept as a local literal so
+// trust.mjs stays import-free of the lexicon-loading pack module.
+const LIVE_REFERENCE_PACK = "wikipedia-live";
+const referenceKindFor = (pack) => (pack === LIVE_REFERENCE_PACK ? "referenceLive" : "reference");
+
 export function provenanceTagToSource(tag) {
   const t = String(tag || "").trim();
   if (!t) return null;
@@ -59,7 +74,8 @@ export function provenanceTagToSource(tag) {
     const rest = t.slice("reference:".length);
     const colon = rest.indexOf(":");
     if (colon < 0) return { kind: "reference", pack: rest || "unknown", article: "" };
-    return { kind: "reference", pack: rest.slice(0, colon) || "unknown", article: rest.slice(colon + 1) };
+    const pack = rest.slice(0, colon) || "unknown";
+    return { kind: referenceKindFor(pack), pack, article: rest.slice(colon + 1) };
   }
   const head = t.split(/\s+/)[0]; // drop trailing " /r/IsA" etc.
   if (head.startsWith("corpus-weak:")) return { kind: "corpusWeak", name: head.slice("corpus-weak:".length) || "unknown" };
@@ -78,6 +94,7 @@ export function provenanceTagToSource(tag) {
   }
   if (head.startsWith("web:")) return { kind: "web", url: head.slice("web:".length) };
   if (head.startsWith("url:")) return { kind: "web", url: head.slice("url:".length) };
+  if (head.startsWith("optimistic-extract:")) return { kind: "optimisticExtract", name: head.slice("optimistic-extract:".length) || "unknown" };
   if (head.startsWith("extracted:")) return { kind: "extracted", name: head.slice("extracted:".length) || "unknown" };
   if (head.startsWith("entailed:")) return { kind: "entailed", rule: head.slice("entailed:".length) };
   if (head.startsWith("chat:") || head.startsWith("session:") || head.startsWith("operator")) return { kind: "operator" };
@@ -93,8 +110,10 @@ export const SOURCE_PRIOR = Object.freeze({
   corpus: 0.7,
   reference: 0.6,
   corpusWeak: 0.55,
-  web: 0.4,
+  referenceLive: 0.5,
   extracted: 0.45,
+  web: 0.4,
+  optimisticExtract: 0.35,
   entailed: 0.3,
 });
 

package/src/domain/sense-split.mjs

@@ -0,0 +1,203 @@
+// sense-split.mjs — word-sense discrimination over a taught/stored class
+// hierarchy. When one label carries two is-a facts ("rover is a kind of dog",
+// "rover is a kind of scout"), their superclass ancestries decide whether the
+// two end classes name the same concept or two different ones. The verdict is
+// an implicit owl:differentFrom between the senses; the mechanism is pure and
+// deterministic over the stored subClassOf closure and any stored
+// owl:disjointWith pairs — never a model.
+//
+// The distinctness test, in order:
+//   1. one class subsumes the other  -> same lineage, not distinct.
+//   2. any ancestor pair is disjoint -> distinct (the cax-dw ⊑-lift, read-only).
+//   3. ancestries never meet below ⊤ -> distinct (wholly non-intersecting).
+//   4. the least common subsumer is a root, or scores below threshold on a
+//      Wu-Palmer depth ratio (or a supplied Resnik information-content map)
+//                                    -> distinct.
+//   5. otherwise                     -> not distinct.
+
+import { SUBCLASS_PREDICATE } from "./syllogise.mjs";
+
+/** child term -> Set of its direct superclasses, from `[child, parent]` edges.
+ *  Self-loops and blank endpoints are dropped. */
+export function subClassParents(subClassEdges) {
+  const parents = new Map();
+  for (const [child, parent] of subClassEdges || []) {
+    if (!child || !parent || child === parent) continue;
+    if (!parents.has(child)) parents.set(child, new Set());
+    parents.get(child).add(parent);
+  }
+  return parents;
+}
+
+/** The single rendering chain above `term` — `[term, parent, grandparent, …]`
+ *  — following one deterministic (lowest-sorted) parent per hop, capped at
+ *  `cap` nodes, cycle-safe. A branch point still yields one readable line;
+ *  the full-ancestor-set comparisons below never rely on this single path. */
+export function ancestryChain(term, parents, { cap = 6 } = {}) {
+  const chain = [term];
+  const seen = new Set([term]);
+  let node = term;
+  while (chain.length < cap) {
+    const ups = [...(parents.get(node) || [])].filter((p) => !seen.has(p)).sort();
+    if (!ups.length) break;
+    node = ups[0];
+    seen.add(node);
+    chain.push(node);
+  }
+  return chain;
+}
+
+/** Every ancestor of `term`, including `term` itself, as a Set. Cycle-safe,
+ *  bounded by the reachable set. */
+export function ancestorSet(term, parents, { cap = 64 } = {}) {
+  const seen = new Set([term]);
+  const queue = [term];
+  while (queue.length && seen.size < cap) {
+    const node = queue.shift();
+    for (const p of parents.get(node) || []) {
+      if (!seen.has(p)) { seen.add(p); queue.push(p); }
+    }
+  }
+  return seen;
+}
+
+/** Depth from a root, measured as the LONGEST upward chain length: a root
+ *  (no parents) is depth 1, its children depth 2, and so on. Deterministic and
+ *  cycle-safe; used for the Wu-Palmer ratio and to pick the deepest common
+ *  subsumer as the least common subsumer. */
+function depthFrom(term, parents, memo, stack) {
+  if (memo.has(term)) return memo.get(term);
+  if (stack.has(term)) return 1; // cycle guard — treat the re-entry as a root
+  stack.add(term);
+  const ups = parents.get(term);
+  let depth = 1;
+  if (ups && ups.size) {
+    let deepest = 0;
+    for (const p of ups) deepest = Math.max(deepest, depthFrom(p, parents, memo, stack));
+    depth = deepest + 1;
+  }
+  stack.delete(term);
+  memo.set(term, depth);
+  return depth;
+}
+
+/** A symmetric lookup "is A disjoint with B" over stored owl:disjointWith
+ *  pairs. Both directions of every pair are indexed. */
+function disjointIndex(disjointEdges) {
+  const of = new Map();
+  for (const [a, b] of disjointEdges || []) {
+    if (!a || !b) continue;
+    if (!of.has(a)) of.set(a, new Set());
+    of.get(a).add(b);
+    if (!of.has(b)) of.set(b, new Set());
+    of.get(b).add(a);
+  }
+  return of;
+}
+
+/** Decide whether classes `a` and `b` name distinct concepts, given the parent
+ *  index, the disjointness index, a depth memo, and the scoring options.
+ *  Returns `{ distinct, reason, lcs }`. */
+function pairVerdict(a, b, parents, disjointOf, depthMemo, { threshold, rootDepth, icByTerm }) {
+  const ancA = ancestorSet(a, parents);
+  const ancB = ancestorSet(b, parents);
+  if (ancA.has(b) || ancB.has(a)) return { distinct: false, reason: "subsumes", lcs: ancA.has(b) ? b : a };
+
+  if (disjointOf.size) {
+    for (const x of ancA) {
+      const partners = disjointOf.get(x);
+      if (!partners) continue;
+      for (const y of ancB) if (partners.has(y)) return { distinct: true, reason: "disjoint", lcs: null };
+    }
+  }
+
+  const common = [...ancA].filter((x) => ancB.has(x));
+  if (!common.length) {
+    // Non-intersection only means "distinct" when both sides carry real
+    // ancestry evidence. A bare class with no recorded superclass tells us
+    // nothing about its sense — a sparse taxonomy that simply never linked
+    // two storage classes must not read as two concepts. Inconclusive keeps
+    // the flat list.
+    if (ancA.size < 2 || ancB.size < 2) return { distinct: false, reason: "inconclusive", lcs: null };
+    return { distinct: true, reason: "non-intersecting", lcs: null };
+  }
+
+  const depth = (t) => depthFrom(t, parents, depthMemo, new Set());
+  let lcs = common[0];
+  let lcsDepth = depth(lcs);
+  for (const c of common) {
+    const d = depth(c);
+    if (d > lcsDepth || (d === lcsDepth && c < lcs)) { lcs = c; lcsDepth = d; }
+  }
+
+  if (lcsDepth <= rootDepth) return { distinct: true, reason: "root-subsumer", lcs };
+
+  if (icByTerm) {
+    const ic = icByTerm.get(lcs);
+    if (typeof ic === "number" && ic < threshold) return { distinct: true, reason: "low-ic", lcs };
+    return { distinct: false, reason: "shared-lineage", lcs };
+  }
+
+  const wuPalmer = (2 * lcsDepth) / (depth(a) + depth(b));
+  if (wuPalmer < threshold) return { distinct: true, reason: "shallow-subsumer", lcs };
+  return { distinct: false, reason: "shared-lineage", lcs };
+}
+
+/** Cluster same-predicate end classes into senses. Two classes join the same
+ *  sense when they are NOT distinct (union-find over the pairwise verdicts);
+ *  the answer splits when more than one cluster survives.
+ *
+ *  Options:
+ *   - subClassEdges / parents: the taxonomy (`parents` wins if both given).
+ *   - disjointEdges: stored owl:disjointWith pairs.
+ *   - threshold: Wu-Palmer ratio (or Resnik IC) below which a shared subsumer
+ *     still reads as two senses. Default 0.5.
+ *   - rootDepth: a subsumer at or above this shallow depth is treated as ⊤.
+ *     Default 1 (only the very root).
+ *   - icByTerm: an optional term->information-content map; when supplied the
+ *     LCS is scored by Resnik IC instead of the Wu-Palmer depth ratio.
+ *
+ *  Returns `{ split, clusters, pairs }`. Each cluster is
+ *  `{ objects, label }` where `label` is the cluster's most specific class.
+ *  `pairs` records every pairwise verdict for inspection/tests. */
+export function clusterSenses(objects, {
+  subClassEdges = [], parents: parentsIn = null, disjointEdges = [],
+  threshold = 0.5, rootDepth = 1, icByTerm = null,
+} = {}) {
+  const unique = [...new Set((objects || []).filter(Boolean))];
+  const parents = parentsIn || subClassParents(subClassEdges);
+  const disjointOf = disjointIndex(disjointEdges);
+  const depthMemo = new Map();
+
+  const parent = new Map(unique.map((o) => [o, o]));
+  const find = (x) => { while (parent.get(x) !== x) { parent.set(x, parent.get(parent.get(x))); x = parent.get(x); } return x; };
+  const union = (a, b) => { const ra = find(a), rb = find(b); if (ra !== rb) parent.set(ra < rb ? rb : ra, ra < rb ? ra : rb); };
+
+  const pairs = [];
+  for (let i = 0; i < unique.length; i += 1) {
+    for (let j = i + 1; j < unique.length; j += 1) {
+      const v = pairVerdict(unique[i], unique[j], parents, disjointOf, depthMemo, { threshold, rootDepth, icByTerm });
+      pairs.push({ a: unique[i], b: unique[j], ...v });
+      if (!v.distinct) union(unique[i], unique[j]);
+    }
+  }
+
+  const byRoot = new Map();
+  for (const o of unique) {
+    const r = find(o);
+    if (!byRoot.has(r)) byRoot.set(r, []);
+    byRoot.get(r).push(o);
+  }
+  const depth = (t) => depthFrom(t, parents, depthMemo, new Set());
+  const clusters = [...byRoot.values()].map((members) => {
+    const sorted = [...members].sort();
+    let label = sorted[0];
+    let best = depth(label);
+    for (const m of sorted) { const d = depth(m); if (d > best) { best = d; label = m; } }
+    return { objects: sorted, label };
+  }).sort((x, y) => x.label.localeCompare(y.label));
+
+  return { split: clusters.length > 1, clusters, pairs };
+}
+
+export { SUBCLASS_PREDICATE };