@polycode-projects/the-mechanical-code-talker 1.3.2 → 1.4.1

package/README.md

@@ -4,9 +4,77 @@
 
 A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage:
 pattern-driven, best-efforts, and obsessed with software the way PARRY was
-obsessed with the mafia. No model calls anywhere. Interpretation is mechanical
-(deterministic language libraries, template sets, committed corpuses). Memory is
-a graph on disk. Every answer is either grounded or an honest miss.
+obsessed with the mafia. No model calls anywhere.
+
+tmct turns natural language directly into a graph database. The graph starts
+seeded with an **ontology** (a curated software vocabulary with real
+definitions), a **lexicon** (everyday words mapped onto that vocabulary), and
+a **corpus** (a filtered ConceptNet slice of general-world terms). Teach it a
+fact in plain English and it mints a node. Ask it a question and it answers
+from what it was seeded with, what you taught it, and what it can derive by
+rule from both. Every answer is either grounded or an honest miss.
+
+## Teach it, then ask it to reason
+
+This is real, runnable output. No cherry-picking, no model anywhere in the
+loop. Copy it into a file and run it:
+
+```js
+import { runChat } from "@polycode-projects/the-mechanical-code-talker";
+import { Readable, PassThrough } from "node:stream";
+import { mkdtemp, mkdir, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+// A graph producer feeds tmct a code graph like this one (see "The repository
+// interface" below). This is a 4-node slice: two modules, a base class, and a
+// class that inherits it.
+const graph = {
+  individuals: [
+    { id: "mod:src/handlers/base.mjs", label: "src/handlers/base.mjs", class: "Module" },
+    { id: "mod:src/handlers/tasks.mjs", label: "src/handlers/tasks.mjs", class: "Module" },
+    { id: "fn:src/handlers/base.mjs#Controller", label: "Controller", class: "Class" },
+    { id: "fn:src/handlers/tasks.mjs#TaskController", label: "TaskController", class: "Class" },
+  ],
+  objectProperties: [{
+    predicate: "inherits", prop: "seon:hasSuperType", count: 1,
+    examples: [{ subject: "fn:src/handlers/tasks.mjs#TaskController", object: "fn:src/handlers/base.mjs#Controller",
+                 subjectLabel: "TaskController", objectLabel: "Controller" }],
+  }],
+};
+
+const repoPath = await mkdtemp(join(tmpdir(), "tmct-demo-"));
+await mkdir(join(repoPath, ".tmct"), { recursive: true });
+await writeFile(join(repoPath, ".tmct", "graph.json"), JSON.stringify(graph));
+
+// tmct has one API surface for both teaching and asking: a chat turn, in
+// English. Each call below is a short session over the same repo, so what
+// gets taught in the first call is still remembered in the second.
+async function tell(line) {
+  const out = new PassThrough();
+  let transcript = "";
+  out.on("data", (chunk) => { transcript += chunk; });
+  await runChat({ repoPath, input: Readable.from([line + "\n", "/exit\n"]), output: out });
+  return transcript.split("\n").find((l) => l.startsWith("tmct> "));
+}
+
+await tell("a controller is a kind of handler");         // Learn
+console.log(await tell("is TaskController a handler"));  // Infer from learnings
+```
+
+Output, captured from an actual run:
+
+```
+tmct> yes — the code graph says TaskController inherits Controller, and you
+told me: controller is a kind of handler (source: ace:chat:<session-id>@<timestamp>)
+```
+
+Nothing here was told that "handler" and "Controller" relate. tmct combined a
+fact already in the graph (`TaskController inherits Controller`) with a fact
+you just taught it in English (`controller is a kind of handler`) and wrote
+the connecting sentence itself, citing both sources. The `source: ace:chat:…`
+part is a real provenance receipt. Every fact tmct stores records where it
+came from and when (more on that below).
 
 ```
 $ tmct
@@ -21,6 +89,67 @@ tmct> /exit
 is a real, interactive chat demo running client-side. Your browser runs the
 actual query engine against a small example codebase, no server, no install.
 
+## Compared to other JS libraries
+
+Nothing found in the JS ecosystem does this exact combination: teach in
+English, reason over a seeded ontology, answer in English, with no model call
+anywhere. Three kinds of library each cover one piece of it well.
+
+**[compromise](https://github.com/spencermountain/compromise)** (`npm i
+compromise`, 12k+ GitHub stars, describes itself as "modest natural language
+processing") is the general-purpose NLP toolkit of the three. Its own README
+example:
+
+```js
+import nlp from 'compromise'
+let doc = nlp('she sells seashells by the seashore.')
+doc.verbs().toPastTense()
+doc.text()
+// 'she sold seashells by the seashore.'
+```
+
+It tags part-of-speech, conjugates verbs, and parses fractions and money far
+more broadly than tmct attempts to. It has no graph, no ontology, and no
+memory between calls. Each call is stateless text in, text out.
+
+**[N3.js](https://github.com/rdfjs/N3.js)** (`npm i n3`) is a mature,
+spec-compliant RDF/OWL toolkit: parsing, writing, and in-memory storage of
+triples. Its own README example:
+
+```js
+const parser = new N3.Parser();
+parser.parse(tomAndJerry, (error, quad, prefixes) => {
+  if (quad) console.log(quad);
+  else console.log("That's all, folks!", prefixes);
+});
+```
+
+(`tomAndJerry` is Turtle text the caller writes by hand: `c:Tom a c:Cat.
+c:Jerry a c:Mouse; c:smarterThan c:Tom.`) N3.js is the right choice if you
+already have RDF and need to parse or serialize it fast. It has no
+natural-language front end (you write the triples yourself) and no built-in
+reasoning beyond an optional, limited basic-graph-pattern reasoner.
+
+**[elizabot](https://github.com/tkafka/node-elizabot)** (`npm i elizabot`) is
+the direct ELIZA lineage in JS, the same territory tmct's chat surface sits
+in. Its own README example:
+
+```js
+var eliza = new ElizaBot();
+var initial = eliza.getInitial();
+var reply = eliza.transform(inputstring);
+```
+
+It is deterministic and needs no model, same as tmct. But it has no graph and
+no persistent memory: a fact from one line never carries into the next, and
+its "reasoning" is pattern substitution, not a stored, queryable fact.
+
+Put together: broad NLP without a graph (compromise), a graph without a
+natural-language front end (N3.js), or a conversational front end without a
+graph (elizabot). Combining ontology-seeded graph memory, English teaching,
+and rule-based inference into one no-model pipeline is what looks distinctive
+about tmct as of this writing.
+
 ## How it interprets you
 
 Every message runs through **multiple concurrent interpretation strategies**:
@@ -199,7 +328,8 @@ or a bare user gets a working install in one command.
 ### Try it on an example graph
 
 tmct *consumes* a code graph at `<repo>/.tmct/graph.json`; it does not build
-one. Two ready-made example graphs ship in `examples/` so you can see it answer
+one. Two ready-made example graphs live in `examples/` in this repo (not in the
+published npm package — clone the repo to use them) so you can see it answer
 real questions with no setup:
 
 ```bash
@@ -237,9 +367,13 @@ full tours.
 import { runChat, ask, resolveObject, fetchEntities } from "@polycode-projects/the-mechanical-code-talker";
 ```
 
-The `exports` map and the chat primitives (`ask`, `resolveObject`,
-`relationKind`, `impactClosure`, `dispatchTool`, `fetchEntities`) are the
-extension surface.
+`runChat` is the full teach-and-ask surface (see "Teach it, then ask it to
+reason" above — it works over injectable streams, so a script can drive a
+session the same way the tests do). `ask`/`resolveObject` are the lower-level,
+read-only query primitives over an already-loaded graph, for a caller that
+wants to query without a chat session. The `exports` map and the chat
+primitives (`ask`, `resolveObject`, `relationKind`, `impactClosure`,
+`dispatchTool`, `fetchEntities`) are the extension surface.
 
 ## The repository interface
 

package/ROADMAP.md

@@ -407,6 +407,28 @@ consolidates three near-duplicate "research frontier" essays (the frame problem,
 disambiguation/ontology scale, Winograd-hard coreference) that had independently grown across the
 now-archived docs.
 
+**v1.4.0 — the first PLAN_AGENTS.md uplift batch (2026-07-11)**: built as four parallel,
+worktree-isolated background tracks, merged sequentially, watched throughout by a background
+strategy-advisor agent (full tick-by-tick record in `STRATEGY_ADVISOR.log`). 1543/1543 tests green
+at the final merge. Shipped: the Repository Interface wrapper fixes from the seonix audit (ranked
+`search()`, a real graph-only `context()` — `INTERFACE_VERSION` 1.0.0→1.1.0 — depth-capped
+`impact()`, source-backed `snippet()`, `edges()`/`search()` pagination, telemetry wiring), a
+path-traversal security fix found and closed along the way (`src/source-slice.mjs`), hub-dampened
+memory-fact ranking (on by default — the build found the original "modest degree, modest penalty"
+assumption was mathematically wrong and proved the real bound instead), memory-tree versioning
+(`snapshotMemory()`, manual trigger only), full session-scoped actor-level trust (shipped
+unconditionally, no config flag — operator decision, single consumer), the extension-pack seam
+(`src/extensions.mjs`, `[extensions]`/`[bias]` in `tmct.toml`, `tmct extend --validate`, and a
+deliberate bug fix — `tmct init` now seeds SEON as well as ConceptNet), bias-weighted fact ranking
+(`src/memory/bias.mjs`, verified by control-flow tracing to never drop a fact, only reorder it), and
+`tmct init --with-persona <name>`. One real merge conflict (Track A's `tel` param and Track D's
+`biasByBundle` param both threading through the same `chat.mjs` function signatures — resolved by
+keeping both). One bug the strategy advisor caught that the original brief missed: the
+path-traversal guard failed closed *incorrectly* under a relative `TMCT_GRAPH_FILE`, rejecting
+legitimate reads, not just traversal attempts — fixed at the source and defensively in the guard.
+One scope decision made mid-build: multi-language AST extraction stays in seonix permanently, not
+tmct's job — full detail in `PLAN_AGENTS.md` §13.
+
 ## The umbrella product definition (item 1)
 
 **A tolerant, ELIZA/PARRY-style chat, obsessed with software.** A best-efforts

package/bin/tmct.mjs

@@ -49,6 +49,11 @@ Usage:
                                offline, $0; init is tier-1-only unless asked
        [--detect]              suggest a tier-2 corpus from the repo's manifests
                                (pyproject.toml → python, pom.xml → java); never seeds unasked
+       [--with-persona <name>] write an explicit [extensions]/[bias] preset into tmct.toml
+                               ("code" — today's implicit default, made explicit)
+  tmct extend --validate <dir>  validate a third-party extension pack's declared
+                               resources (corpus/lexicon/templates) before activating
+                               it in any repo's tmct.toml; exits non-zero on failure
   tmct syllogise [--repo <abs>] speculative inference (offline maintenance job): forward-
        [--depth <n>] [--budget <n>]  chain the memory's rdfs:subClassOf closure, materialising
                                bounded, low-trust, retractable entailed facts (never on the chat path)
@@ -316,41 +321,96 @@ async function main() {
 
   if (mode === "init") {
     // `tmct init` — the Repository-Interface onboarding surface: scaffold .tmct/,
-    // write tmct.toml, seed the tier-1 corpus (offline, opt-out via TMCT_NO_SEED),
-    // and record provenance. Idempotent; --force rewrites config + re-records.
+    // write tmct.toml, seed the corpus (offline, opt-out via TMCT_NO_SEED), and
+    // record provenance. Idempotent; --force rewrites config + re-records.
     //
-    // TIERING POLICY: init is OFFLINE, $0 and TIER-1-ONLY by default. A tier-2
-    // domain/language corpus (corpus/tier2/: aws, python, java) is added ONLY when
+    // TIERING POLICY: init is OFFLINE, $0 and TIER-1-ONLY by default (seon +
+    // conceptnet — src/extensions.mjs's BUILTIN_EXTENSIONS). A tier-2 domain/
+    // language corpus (corpus/tier2/: aws, python, java) is added ONLY when
     // explicitly asked via `--corpus <id>`. The `--detect` auto-detect is a
     // documented STUB: it inspects the repo's manifests (pyproject.toml → python,
     // pom.xml → java) and SUGGESTS the matching corpus, but never seeds it unasked.
     const rest = process.argv.slice(3);
-    const { initRepo } = await import("../src/init.mjs");
-    const res = await initRepo(process.cwd(), { force: rest.includes("--force") });
-    process.stdout.write(res.message + "\n");
+    const { initRepo, defaultConfig, renderTomlConfig, CONFIG_FILE, PERSONA_PRESETS } = await import("../src/init.mjs");
+    const { loadTomlConfig } = await import("../src/toml-config.mjs");
 
     const ci = rest.indexOf("--corpus");
     const corpusId = ci !== -1 ? rest[ci + 1] : undefined;
+    let manifest = null;
+    let manifestEntry = null;
     if (corpusId) {
-      // Seed a tier-2 corpus by id from corpus/tier2/ (same slice shape as tier-1;
-      // provenance-tagged corpus:tier2:<id>). Idempotent (content-hashed fact ids).
+      // Validate the id against the tier-2 manifest (same source of truth the
+      // old ad hoc path used) BEFORE touching anything on disk.
       const { readFile } = await import("node:fs/promises");
-      const { join, dirname } = await import("node:path");
-      const { seedMemory, TIER2_MANIFEST_FILE } = await import("../src/corpus/conceptnet.mjs");
-      let manifest;
+      const { TIER2_MANIFEST_FILE } = await import("../src/corpus/conceptnet.mjs");
       try { manifest = JSON.parse(await readFile(TIER2_MANIFEST_FILE, "utf8")); }
       catch (e) { process.stderr.write(`tmct init: cannot read the tier-2 manifest — ${e?.message || e}\n`); process.exit(1); }
-      const entry = (manifest.corpuses || []).find((c) => c.id === corpusId);
-      if (!entry) {
+      manifestEntry = (manifest.corpuses || []).find((c) => c.id === corpusId);
+      if (!manifestEntry) {
         const ids = (manifest.corpuses || []).map((c) => c.id).join(", ");
         process.stderr.write(`tmct init: unknown --corpus "${corpusId}". Available tier-2 corpuses: ${ids}.\n`);
         process.exit(2);
       }
-      const slicePath = join(dirname(TIER2_MANIFEST_FILE), entry.file);
-      const seeded = await seedMemory(process.cwd(), { slicePath, provenancePrefix: `corpus:tier2:${entry.id}` });
+    }
+
+    // `--with-persona <name>` (Part 7): resolve + validate BEFORE touching
+    // disk, mirroring `--corpus`'s own unknown-id error handling — a bad
+    // persona name never scaffolds anything.
+    const pi = rest.indexOf("--with-persona");
+    const personaName = pi !== -1 ? rest[pi + 1] : undefined;
+    let personaPreset = null;
+    if (personaName) {
+      if (!Object.prototype.hasOwnProperty.call(PERSONA_PRESETS, personaName)) {
+        const names = Object.keys(PERSONA_PRESETS).join(", ");
+        process.stderr.write(`tmct init: unknown --with-persona "${personaName}". Available personas: ${names}.\n`);
+        process.exit(2);
+      }
+      personaPreset = PERSONA_PRESETS[personaName];
+    }
+
+    const res = await initRepo(process.cwd(), { force: rest.includes("--force"), persona: personaPreset });
+    process.stdout.write(res.message + "\n");
+
+    if (manifestEntry) {
+      // `--corpus <id>` now means "activate extensions.tier2-<id> and PERSIST
+      // that into tmct.toml" — so a second `tmct init` (or the next chat
+      // bootstrap) remembers the choice, unlike the old ad hoc path, which had
+      // to be repeated every time. This changes the tier-2 provenance tag from
+      // the old colon-separated "corpus:tier2:<id>" to the hyphenated
+      // "corpus:tier2-<id>" (matching the TOML-legal extension name) — a
+      // deliberate, low-risk rename; nothing in chat.mjs's runtime logic keys
+      // on the old colon-separated string (verified via grep).
+      const extName = `tier2-${manifestEntry.id}`;
+      const raw = await loadTomlConfig(process.cwd()); // just-written by initRepo above
+      const cfg = { ...defaultConfig() };
+      if (raw?.graph_file !== undefined) cfg.graphFile = String(raw.graph_file);
+      if (raw?.corpus?.tier !== undefined) cfg.corpus = { tier: raw.corpus.tier };
+      if (raw?.seed) {
+        cfg.seed = { ...cfg.seed };
+        if (raw.seed.enabled !== undefined) cfg.seed.enabled = Boolean(raw.seed.enabled);
+        if (raw.seed.limit !== undefined) cfg.seed.limit = Number(raw.seed.limit);
+      }
+      cfg.extensions = { ...(raw?.extensions || {}), [extName]: { ...(raw?.extensions?.[extName] || {}), active: true } };
+      if (raw?.bias !== undefined) cfg.bias = raw.bias;
+      const { writeFile } = await import("node:fs/promises");
+      const { join } = await import("node:path");
+      await writeFile(join(process.cwd(), CONFIG_FILE), renderTomlConfig(cfg));
+
+      // Seed it now too — through the SAME unified corpus loader every other
+      // bundle goes through (src/extensions.mjs), not a bespoke seedMemory call.
+      const { resolveExtensions, seedActiveCorpusEntries } = await import("../src/extensions.mjs");
+      const { entries } = await resolveExtensions(process.cwd());
+      const entry = entries.get(extName);
+      const { perBundle } = await seedActiveCorpusEntries(process.cwd(), new Map([[extName, entry]]));
+      const seeded = perBundle[extName];
+      if (seeded.error) {
+        process.stderr.write(`tmct init: could not seed tier-2 corpus "${manifestEntry.id}" — ${seeded.error}\n`);
+        process.exit(1);
+      }
       process.stdout.write(
-        `seeded tier-2 corpus "${entry.id}" (${entry.kind}) — ${seeded.appended} fact(s) added`
-        + `${seeded.skipped ? `, ${seeded.skipped} already present` : ""}. Source: corpus/tier2/${entry.file} (${entry.license}).\n`,
+        `seeded tier-2 corpus "${manifestEntry.id}" (${manifestEntry.kind}) — ${seeded.appended} fact(s) added`
+        + `${seeded.skipped ? `, ${seeded.skipped} already present` : ""}. Source: corpus/tier2/${manifestEntry.file} (${manifestEntry.license}). `
+        + `Activated in tmct.toml — future \`tmct init\`/chat sessions seed it automatically.\n`,
       );
       return;
     }
@@ -377,6 +437,56 @@ async function main() {
     return;
   }
 
+  if (mode === "extend") {
+    // `tmct extend --validate <dir>` — validate a THIRD-PARTY extension pack
+    // (the shape a package like seonix/marginalia ships) BEFORE it's activated
+    // in any repo's tmct.toml. Reuses existing throw-loudly primitives
+    // (loadSlice/loadMap/toFacts, loadLexicon, loadTemplates) via
+    // src/extensions.mjs's validateExtensionPack — never invents new
+    // shape-checking logic. `<dir>` must carry its own tmct.toml declaring one
+    // or more `[extensions.<name>]` host entries (the SAME [extensions] table
+    // shape a repo's own tmct.toml uses) naming the resource(s) to validate;
+    // the shipped builtins (seon/conceptnet/tier2-*) are never re-validated
+    // here — this command is about a PACK's OWN declared resources.
+    const rest = process.argv.slice(3);
+    const vi = rest.indexOf("--validate");
+    const dirArg = vi !== -1 ? rest[vi + 1] : undefined;
+    if (!dirArg) {
+      process.stderr.write("tmct extend: --validate <dir> requires a directory\n");
+      process.exit(2);
+    }
+    const { resolve: resolvePath } = await import("node:path");
+    const target = resolvePath(process.cwd(), dirArg);
+    const { resolveExtensions, BUILTIN_EXTENSIONS, validateExtensionPack } = await import("../src/extensions.mjs");
+    let entries;
+    try {
+      ({ entries } = await resolveExtensions(target));
+    } catch (e) {
+      process.stderr.write(`tmct extend --validate: ${e?.message || e}\n`);
+      process.exit(1);
+    }
+    const hostEntries = [...entries].filter(([name]) => !(name in BUILTIN_EXTENSIONS));
+    if (!hostEntries.length) {
+      process.stderr.write(`tmct extend --validate: no host-declared [extensions.*] entries found in ${target}/tmct.toml\n`);
+      process.exit(1);
+    }
+    let allOk = true;
+    for (const [name, entry] of hostEntries) {
+      process.stdout.write(`${name} (${entry.kind}):\n`);
+      const { ok, results } = await validateExtensionPack(target, entry);
+      if (!ok) allOk = false;
+      for (const r of results) {
+        const status = r.ok ? "PASS" : "FAIL";
+        const detail = r.ok
+          ? (r.counts ? ` (${Object.entries(r.counts).map(([k, v]) => `${k}=${v}`).join(", ")})` : "")
+          : ` — ${r.error}`;
+        process.stdout.write(`  [${status}] ${r.kind}: ${r.path}${detail}\n`);
+      }
+    }
+    process.stdout.write(allOk ? "tmct extend --validate: all resources passed.\n" : "tmct extend --validate: one or more resources FAILED.\n");
+    process.exit(allOk ? 0 : 1);
+  }
+
   if (mode === "syllogise") {
     // `tmct syllogise` — the explicit speculative-inference batch (never on the chat
     // hot path): forward-chain the memory's rdfs:subClassOf closure into bounded,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@polycode-projects/the-mechanical-code-talker",
-  "version": "1.3.2",
+  "version": "1.4.1",
   "private": false,
   "type": "module",
   "description": "The Mechanical Code Talker (tmct) — a tolerant, offline, $0 chat surface that guides you toward precision queries about a software repository. ELIZA/PARRY-style but domain-obsessed with code. No model calls; no codebase index of its own.",