@polycode-projects/the-mechanical-code-talker 1.3.1 → 1.4.0

package/README.md

@@ -4,9 +4,77 @@
 
 A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage:
 pattern-driven, best-efforts, and obsessed with software the way PARRY was
-obsessed with the mafia. No model calls anywhere. Interpretation is mechanical
-(deterministic language libraries, template sets, committed corpuses). Memory is
-a graph on disk. Every answer is either grounded or an honest miss.
+obsessed with the mafia. No model calls anywhere.
+
+tmct turns natural language directly into a graph database. The graph starts
+seeded with an **ontology** (a curated software vocabulary with real
+definitions), a **lexicon** (everyday words mapped onto that vocabulary), and
+a **corpus** (a filtered ConceptNet slice of general-world terms). Teach it a
+fact in plain English and it mints a node. Ask it a question and it answers
+from what it was seeded with, what you taught it, and what it can derive by
+rule from both. Every answer is either grounded or an honest miss.
+
+## Teach it, then ask it to reason
+
+This is real, runnable output. No cherry-picking, no model anywhere in the
+loop. Copy it into a file and run it:
+
+```js
+import { runChat } from "@polycode-projects/the-mechanical-code-talker";
+import { Readable, PassThrough } from "node:stream";
+import { mkdtemp, mkdir, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+// A graph producer feeds tmct a code graph like this one (see "The repository
+// interface" below). This is a 4-node slice: two modules, a base class, and a
+// class that inherits it.
+const graph = {
+  individuals: [
+    { id: "mod:src/handlers/base.mjs", label: "src/handlers/base.mjs", class: "Module" },
+    { id: "mod:src/handlers/tasks.mjs", label: "src/handlers/tasks.mjs", class: "Module" },
+    { id: "fn:src/handlers/base.mjs#Controller", label: "Controller", class: "Class" },
+    { id: "fn:src/handlers/tasks.mjs#TaskController", label: "TaskController", class: "Class" },
+  ],
+  objectProperties: [{
+    predicate: "inherits", prop: "seon:hasSuperType", count: 1,
+    examples: [{ subject: "fn:src/handlers/tasks.mjs#TaskController", object: "fn:src/handlers/base.mjs#Controller",
+                 subjectLabel: "TaskController", objectLabel: "Controller" }],
+  }],
+};
+
+const repoPath = await mkdtemp(join(tmpdir(), "tmct-demo-"));
+await mkdir(join(repoPath, ".tmct"), { recursive: true });
+await writeFile(join(repoPath, ".tmct", "graph.json"), JSON.stringify(graph));
+
+// tmct has one API surface for both teaching and asking: a chat turn, in
+// English. Each call below is a short session over the same repo, so what
+// gets taught in the first call is still remembered in the second.
+async function tell(line) {
+  const out = new PassThrough();
+  let transcript = "";
+  out.on("data", (chunk) => { transcript += chunk; });
+  await runChat({ repoPath, input: Readable.from([line + "\n", "/exit\n"]), output: out });
+  return transcript.split("\n").find((l) => l.startsWith("tmct> "));
+}
+
+await tell("a controller is a kind of handler");         // Learn
+console.log(await tell("is TaskController a handler"));  // Infer from learnings
+```
+
+Output, captured from an actual run:
+
+```
+tmct> yes — the code graph says TaskController inherits Controller, and you
+told me: controller is a kind of handler (source: ace:chat:<session-id>@<timestamp>)
+```
+
+Nothing here was told that "handler" and "Controller" relate. tmct combined a
+fact already in the graph (`TaskController inherits Controller`) with a fact
+you just taught it in English (`controller is a kind of handler`) and wrote
+the connecting sentence itself, citing both sources. The `source: ace:chat:…`
+part is a real provenance receipt. Every fact tmct stores records where it
+came from and when (more on that below).
 
 ```
 $ tmct
@@ -21,6 +89,67 @@ tmct> /exit
 is a real, interactive chat demo running client-side. Your browser runs the
 actual query engine against a small example codebase, no server, no install.
 
+## Compared to other JS libraries
+
+Nothing found in the JS ecosystem does this exact combination: teach in
+English, reason over a seeded ontology, answer in English, with no model call
+anywhere. Three kinds of library each cover one piece of it well.
+
+**[compromise](https://github.com/spencermountain/compromise)** (`npm i
+compromise`, 12k+ GitHub stars, describes itself as "modest natural language
+processing") is the general-purpose NLP toolkit of the three. Its own README
+example:
+
+```js
+import nlp from 'compromise'
+let doc = nlp('she sells seashells by the seashore.')
+doc.verbs().toPastTense()
+doc.text()
+// 'she sold seashells by the seashore.'
+```
+
+It tags part-of-speech, conjugates verbs, and parses fractions and money far
+more broadly than tmct attempts to. It has no graph, no ontology, and no
+memory between calls. Each call is stateless text in, text out.
+
+**[N3.js](https://github.com/rdfjs/N3.js)** (`npm i n3`) is a mature,
+spec-compliant RDF/OWL toolkit: parsing, writing, and in-memory storage of
+triples. Its own README example:
+
+```js
+const parser = new N3.Parser();
+parser.parse(tomAndJerry, (error, quad, prefixes) => {
+  if (quad) console.log(quad);
+  else console.log("That's all, folks!", prefixes);
+});
+```
+
+(`tomAndJerry` is Turtle text the caller writes by hand: `c:Tom a c:Cat.
+c:Jerry a c:Mouse; c:smarterThan c:Tom.`) N3.js is the right choice if you
+already have RDF and need to parse or serialize it fast. It has no
+natural-language front end (you write the triples yourself) and no built-in
+reasoning beyond an optional, limited basic-graph-pattern reasoner.
+
+**[elizabot](https://github.com/tkafka/node-elizabot)** (`npm i elizabot`) is
+the direct ELIZA lineage in JS, the same territory tmct's chat surface sits
+in. Its own README example:
+
+```js
+var eliza = new ElizaBot();
+var initial = eliza.getInitial();
+var reply = eliza.transform(inputstring);
+```
+
+It is deterministic and needs no model, same as tmct. But it has no graph and
+no persistent memory: a fact from one line never carries into the next, and
+its "reasoning" is pattern substitution, not a stored, queryable fact.
+
+Put together: broad NLP without a graph (compromise), a graph without a
+natural-language front end (N3.js), or a conversational front end without a
+graph (elizabot). Combining ontology-seeded graph memory, English teaching,
+and rule-based inference into one no-model pipeline is what looks distinctive
+about tmct as of this writing.
+
 ## How it interprets you
 
 Every message runs through **multiple concurrent interpretation strategies**:
@@ -170,7 +299,7 @@ on the chat's hot path.
   entailment (`tmct syllogise`) is mechanical OWL rule materialization applied
   offline, rule-by-rule and retractable, not an LLM. There is **no LLM anywhere
   in the product**. (An LLM-as-judge exists only in the offline eval harness
-  that tunes tmct, see `SKILL_TUNING_CYCLE.md`, never in the product path.)
+  that tunes tmct, see `SKILL_BENCHMARK_CHAT.md`, never in the product path.)
 - **It never guesses silently.** When it cannot resolve your question it says
   so and nudges you toward a query it *can* answer.
 
@@ -199,7 +328,8 @@ or a bare user gets a working install in one command.
 ### Try it on an example graph
 
 tmct *consumes* a code graph at `<repo>/.tmct/graph.json`; it does not build
-one. Two ready-made example graphs ship in `examples/` so you can see it answer
+one. Two ready-made example graphs live in `examples/` in this repo (not in the
+published npm package — clone the repo to use them) so you can see it answer
 real questions with no setup:
 
 ```bash
@@ -237,9 +367,13 @@ full tours.
 import { runChat, ask, resolveObject, fetchEntities } from "@polycode-projects/the-mechanical-code-talker";
 ```
 
-The `exports` map and the chat primitives (`ask`, `resolveObject`,
-`relationKind`, `impactClosure`, `dispatchTool`, `fetchEntities`) are the
-extension surface.
+`runChat` is the full teach-and-ask surface (see "Teach it, then ask it to
+reason" above — it works over injectable streams, so a script can drive a
+session the same way the tests do). `ask`/`resolveObject` are the lower-level,
+read-only query primitives over an already-loaded graph, for a caller that
+wants to query without a chat session. The `exports` map and the chat
+primitives (`ask`, `resolveObject`, `relationKind`, `impactClosure`,
+`dispatchTool`, `fetchEntities`) are the extension surface.
 
 ## The repository interface
 

package/ROADMAP.md

@@ -382,6 +382,31 @@ resolver/guardrail/planner (96% plan completion, 0% hallucination, closed-world
 levers; the `../bedrock-meter` $0 rung; the playtest; Stage-2/Stage-5 research notes. Full detail:
 `CHATBENCH_0.8.0.md`, `AGENTBENCH_0.8.0.md`.
 
+**Doc restructuring — `PLAN_AGENTS.md` (2026-07-10)**: `PLAN_TMCT_ECOSYSTEM_INTEGRATION.md` was
+rewritten and renamed to `PLAN_AGENTS.md`, absorbing six sibling docs (`PLAN_AGI_ARCHITECTURE.md`,
+`PLAN_CAPABILITY_ROUTER.md`, `PLAN_TAUGHT_RELATIONS.md`, `PLAN_OSS_ACE_PARSER.md`,
+`PLAN_ontology-hierarchies.md`, `PLAN_ADVANCED_GRAMMAR.md` — all now in `archive/`) and sequencing
+their durable content into Phase 0 (foundations) through Phase 4 (tmct as a pluggable LLM rung for
+Claude Code/Bedrock/Copilot), plus a tiered research horizon (R1–R3). Two fresh comparative audits
+of `../marginalia` and `../seonix` fed a new §2, "tmct uplift" — mechanisms those sibling repos
+already have that tmct lacks or does more crudely (memory-tree versioning, actor-level trust, a
+declarative SHACL-style ingest gate, real multi-language AST extraction, Chronograph-style temporal
+diffing, and several Repository-Interface wrapper gaps that just need to be pointed at logic already
+sitting in tmct's own `codegraph.mjs`). A new sibling doc, `PLAN_COMPLETIONS.md`, specs a second,
+competing "tmct produces an artifact" capability alongside `PLAN_CODE.md`'s program synthesis:
+mechanical, extractive text generation (broad search → group → infer between groups → summarize →
+prune → grammar/voice pass), never LLM-style free generation. Separately, the benchmark/skill doc
+landscape was unified: CHATBENCH stops splitting report+transcripts into two files going forward;
+`SKILL_CHAT_PLAYTEST.md` and `SKILL_PLAYTEST_SPRINT.md` merged into `SKILL_BENCHMARK_PLAYTEST.md`
+(with a new `PLAYTESTBENCH_<version>.md` report convention); `SKILL_TUNING_CYCLE.md` and
+`SKILL_INFERENCE_TESTING.md` renamed to `SKILL_BENCHMARK_CHAT.md`/`SKILL_BENCHMARK_INFERENCE.md`;
+a new `SKILL_BENCHMARK_AGENT.md` formalizes the previously-ad-hoc AGENTBENCH cycle;
+`SKILL_STRATEGY_ADVISOR.md`/`SKILL_PLAIN_PROSE.md` renamed to `SKILL_AGENT_STRATEGY_ADVISOR.md`/
+`SKILL_AGENT_PLAIN_PROSE.md`. A new shared reference doc, `docs/references/research-horizon.md`,
+consolidates three near-duplicate "research frontier" essays (the frame problem, word-sense
+disambiguation/ontology scale, Winograd-hard coreference) that had independently grown across the
+now-archived docs.
+
 ## The umbrella product definition (item 1)
 
 **A tolerant, ELIZA/PARRY-style chat, obsessed with software.** A best-efforts
@@ -1094,6 +1119,16 @@ Features we have deliberately shaped seams for but will not build until the phas
 earned them. **Not everything below is deferred for the same reason** — the design horizon,
 stated explicitly (2026-07-08 research pass):
 
+**`PLAN_AGENTS.md` (2026-07-10) is the governing plan for tmct's next major arc** — mounting tmct
+hard into `../marginalia` and `../seonix` as their shared NL↔graph engine and tool-loop/completions
+API, plus a pluggable LLM rung for Claude Code, Amazon Bedrock, and GitHub Copilot. It sequences
+Phase 0 (foundations — an extension-pack/corpus-lexicon seam, RI wrapper fixes, several other
+small known-how items) through Phase 4 (the LLM-rung protocol shims), a "tmct uplift" section
+grounded in fresh comparative audits of both sibling repos, and a tiered research horizon (R1–R3).
+It supersedes the six phase/track pointers below that reference now-archived docs — treat this
+paragraph as the up-to-date entry point, and the items below as historical record of how those six
+docs' scope was reached before consolidation.
+
 ### Future direction: a genuine planning/agentic loop (flagged 2026-07-09, research pass done, not implemented)
 
 The operator's own framing, explicitly out of scope for the routing-level `GOAL_BY_COMMAND`/

package/bin/tmct.mjs

@@ -49,6 +49,11 @@ Usage:
                                offline, $0; init is tier-1-only unless asked
        [--detect]              suggest a tier-2 corpus from the repo's manifests
                                (pyproject.toml → python, pom.xml → java); never seeds unasked
+       [--with-persona <name>] write an explicit [extensions]/[bias] preset into tmct.toml
+                               ("code" — today's implicit default, made explicit)
+  tmct extend --validate <dir>  validate a third-party extension pack's declared
+                               resources (corpus/lexicon/templates) before activating
+                               it in any repo's tmct.toml; exits non-zero on failure
   tmct syllogise [--repo <abs>] speculative inference (offline maintenance job): forward-
        [--depth <n>] [--budget <n>]  chain the memory's rdfs:subClassOf closure, materialising
                                bounded, low-trust, retractable entailed facts (never on the chat path)
@@ -316,41 +321,96 @@ async function main() {
 
   if (mode === "init") {
     // `tmct init` — the Repository-Interface onboarding surface: scaffold .tmct/,
-    // write tmct.toml, seed the tier-1 corpus (offline, opt-out via TMCT_NO_SEED),
-    // and record provenance. Idempotent; --force rewrites config + re-records.
+    // write tmct.toml, seed the corpus (offline, opt-out via TMCT_NO_SEED), and
+    // record provenance. Idempotent; --force rewrites config + re-records.
     //
-    // TIERING POLICY: init is OFFLINE, $0 and TIER-1-ONLY by default. A tier-2
-    // domain/language corpus (corpus/tier2/: aws, python, java) is added ONLY when
+    // TIERING POLICY: init is OFFLINE, $0 and TIER-1-ONLY by default (seon +
+    // conceptnet — src/extensions.mjs's BUILTIN_EXTENSIONS). A tier-2 domain/
+    // language corpus (corpus/tier2/: aws, python, java) is added ONLY when
     // explicitly asked via `--corpus <id>`. The `--detect` auto-detect is a
     // documented STUB: it inspects the repo's manifests (pyproject.toml → python,
     // pom.xml → java) and SUGGESTS the matching corpus, but never seeds it unasked.
     const rest = process.argv.slice(3);
-    const { initRepo } = await import("../src/init.mjs");
-    const res = await initRepo(process.cwd(), { force: rest.includes("--force") });
-    process.stdout.write(res.message + "\n");
+    const { initRepo, defaultConfig, renderTomlConfig, CONFIG_FILE, PERSONA_PRESETS } = await import("../src/init.mjs");
+    const { loadTomlConfig } = await import("../src/toml-config.mjs");
 
     const ci = rest.indexOf("--corpus");
     const corpusId = ci !== -1 ? rest[ci + 1] : undefined;
+    let manifest = null;
+    let manifestEntry = null;
     if (corpusId) {
-      // Seed a tier-2 corpus by id from corpus/tier2/ (same slice shape as tier-1;
-      // provenance-tagged corpus:tier2:<id>). Idempotent (content-hashed fact ids).
+      // Validate the id against the tier-2 manifest (same source of truth the
+      // old ad hoc path used) BEFORE touching anything on disk.
       const { readFile } = await import("node:fs/promises");
-      const { join, dirname } = await import("node:path");
-      const { seedMemory, TIER2_MANIFEST_FILE } = await import("../src/corpus/conceptnet.mjs");
-      let manifest;
+      const { TIER2_MANIFEST_FILE } = await import("../src/corpus/conceptnet.mjs");
       try { manifest = JSON.parse(await readFile(TIER2_MANIFEST_FILE, "utf8")); }
       catch (e) { process.stderr.write(`tmct init: cannot read the tier-2 manifest — ${e?.message || e}\n`); process.exit(1); }
-      const entry = (manifest.corpuses || []).find((c) => c.id === corpusId);
-      if (!entry) {
+      manifestEntry = (manifest.corpuses || []).find((c) => c.id === corpusId);
+      if (!manifestEntry) {
         const ids = (manifest.corpuses || []).map((c) => c.id).join(", ");
         process.stderr.write(`tmct init: unknown --corpus "${corpusId}". Available tier-2 corpuses: ${ids}.\n`);
         process.exit(2);
       }
-      const slicePath = join(dirname(TIER2_MANIFEST_FILE), entry.file);
-      const seeded = await seedMemory(process.cwd(), { slicePath, provenancePrefix: `corpus:tier2:${entry.id}` });
+    }
+
+    // `--with-persona <name>` (Part 7): resolve + validate BEFORE touching
+    // disk, mirroring `--corpus`'s own unknown-id error handling — a bad
+    // persona name never scaffolds anything.
+    const pi = rest.indexOf("--with-persona");
+    const personaName = pi !== -1 ? rest[pi + 1] : undefined;
+    let personaPreset = null;
+    if (personaName) {
+      if (!Object.prototype.hasOwnProperty.call(PERSONA_PRESETS, personaName)) {
+        const names = Object.keys(PERSONA_PRESETS).join(", ");
+        process.stderr.write(`tmct init: unknown --with-persona "${personaName}". Available personas: ${names}.\n`);
+        process.exit(2);
+      }
+      personaPreset = PERSONA_PRESETS[personaName];
+    }
+
+    const res = await initRepo(process.cwd(), { force: rest.includes("--force"), persona: personaPreset });
+    process.stdout.write(res.message + "\n");
+
+    if (manifestEntry) {
+      // `--corpus <id>` now means "activate extensions.tier2-<id> and PERSIST
+      // that into tmct.toml" — so a second `tmct init` (or the next chat
+      // bootstrap) remembers the choice, unlike the old ad hoc path, which had
+      // to be repeated every time. This changes the tier-2 provenance tag from
+      // the old colon-separated "corpus:tier2:<id>" to the hyphenated
+      // "corpus:tier2-<id>" (matching the TOML-legal extension name) — a
+      // deliberate, low-risk rename; nothing in chat.mjs's runtime logic keys
+      // on the old colon-separated string (verified via grep).
+      const extName = `tier2-${manifestEntry.id}`;
+      const raw = await loadTomlConfig(process.cwd()); // just-written by initRepo above
+      const cfg = { ...defaultConfig() };
+      if (raw?.graph_file !== undefined) cfg.graphFile = String(raw.graph_file);
+      if (raw?.corpus?.tier !== undefined) cfg.corpus = { tier: raw.corpus.tier };
+      if (raw?.seed) {
+        cfg.seed = { ...cfg.seed };
+        if (raw.seed.enabled !== undefined) cfg.seed.enabled = Boolean(raw.seed.enabled);
+        if (raw.seed.limit !== undefined) cfg.seed.limit = Number(raw.seed.limit);
+      }
+      cfg.extensions = { ...(raw?.extensions || {}), [extName]: { ...(raw?.extensions?.[extName] || {}), active: true } };
+      if (raw?.bias !== undefined) cfg.bias = raw.bias;
+      const { writeFile } = await import("node:fs/promises");
+      const { join } = await import("node:path");
+      await writeFile(join(process.cwd(), CONFIG_FILE), renderTomlConfig(cfg));
+
+      // Seed it now too — through the SAME unified corpus loader every other
+      // bundle goes through (src/extensions.mjs), not a bespoke seedMemory call.
+      const { resolveExtensions, seedActiveCorpusEntries } = await import("../src/extensions.mjs");
+      const { entries } = await resolveExtensions(process.cwd());
+      const entry = entries.get(extName);
+      const { perBundle } = await seedActiveCorpusEntries(process.cwd(), new Map([[extName, entry]]));
+      const seeded = perBundle[extName];
+      if (seeded.error) {
+        process.stderr.write(`tmct init: could not seed tier-2 corpus "${manifestEntry.id}" — ${seeded.error}\n`);
+        process.exit(1);
+      }
       process.stdout.write(
-        `seeded tier-2 corpus "${entry.id}" (${entry.kind}) — ${seeded.appended} fact(s) added`
-        + `${seeded.skipped ? `, ${seeded.skipped} already present` : ""}. Source: corpus/tier2/${entry.file} (${entry.license}).\n`,
+        `seeded tier-2 corpus "${manifestEntry.id}" (${manifestEntry.kind}) — ${seeded.appended} fact(s) added`
+        + `${seeded.skipped ? `, ${seeded.skipped} already present` : ""}. Source: corpus/tier2/${manifestEntry.file} (${manifestEntry.license}). `
+        + `Activated in tmct.toml — future \`tmct init\`/chat sessions seed it automatically.\n`,
       );
       return;
     }
@@ -377,6 +437,56 @@ async function main() {
     return;
   }
 
+  if (mode === "extend") {
+    // `tmct extend --validate <dir>` — validate a THIRD-PARTY extension pack
+    // (the shape a package like seonix/marginalia ships) BEFORE it's activated
+    // in any repo's tmct.toml. Reuses existing throw-loudly primitives
+    // (loadSlice/loadMap/toFacts, loadLexicon, loadTemplates) via
+    // src/extensions.mjs's validateExtensionPack — never invents new
+    // shape-checking logic. `<dir>` must carry its own tmct.toml declaring one
+    // or more `[extensions.<name>]` host entries (the SAME [extensions] table
+    // shape a repo's own tmct.toml uses) naming the resource(s) to validate;
+    // the shipped builtins (seon/conceptnet/tier2-*) are never re-validated
+    // here — this command is about a PACK's OWN declared resources.
+    const rest = process.argv.slice(3);
+    const vi = rest.indexOf("--validate");
+    const dirArg = vi !== -1 ? rest[vi + 1] : undefined;
+    if (!dirArg) {
+      process.stderr.write("tmct extend: --validate <dir> requires a directory\n");
+      process.exit(2);
+    }
+    const { resolve: resolvePath } = await import("node:path");
+    const target = resolvePath(process.cwd(), dirArg);
+    const { resolveExtensions, BUILTIN_EXTENSIONS, validateExtensionPack } = await import("../src/extensions.mjs");
+    let entries;
+    try {
+      ({ entries } = await resolveExtensions(target));
+    } catch (e) {
+      process.stderr.write(`tmct extend --validate: ${e?.message || e}\n`);
+      process.exit(1);
+    }
+    const hostEntries = [...entries].filter(([name]) => !(name in BUILTIN_EXTENSIONS));
+    if (!hostEntries.length) {
+      process.stderr.write(`tmct extend --validate: no host-declared [extensions.*] entries found in ${target}/tmct.toml\n`);
+      process.exit(1);
+    }
+    let allOk = true;
+    for (const [name, entry] of hostEntries) {
+      process.stdout.write(`${name} (${entry.kind}):\n`);
+      const { ok, results } = await validateExtensionPack(target, entry);
+      if (!ok) allOk = false;
+      for (const r of results) {
+        const status = r.ok ? "PASS" : "FAIL";
+        const detail = r.ok
+          ? (r.counts ? ` (${Object.entries(r.counts).map(([k, v]) => `${k}=${v}`).join(", ")})` : "")
+          : ` — ${r.error}`;
+        process.stdout.write(`  [${status}] ${r.kind}: ${r.path}${detail}\n`);
+      }
+    }
+    process.stdout.write(allOk ? "tmct extend --validate: all resources passed.\n" : "tmct extend --validate: one or more resources FAILED.\n");
+    process.exit(allOk ? 0 : 1);
+  }
+
   if (mode === "syllogise") {
     // `tmct syllogise` — the explicit speculative-inference batch (never on the chat
     // hot path): forward-chain the memory's rdfs:subClassOf closure into bounded,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@polycode-projects/the-mechanical-code-talker",
-  "version": "1.3.1",
+  "version": "1.4.0",
   "private": false,
   "type": "module",
   "description": "The Mechanical Code Talker (tmct) — a tolerant, offline, $0 chat surface that guides you toward precision queries about a software repository. ELIZA/PARRY-style but domain-obsessed with code. No model calls; no codebase index of its own.",