@polycode-projects/the-mechanical-code-talker 2.0.2 → 2.2.0

package/README.md

@@ -103,12 +103,15 @@ The test suite replays every runnable example in this README against the
 live product, this transcript included. If the chat behavior ever drifts
 from the output above, the suite fails and says so.
 
-```session
-$ tmct
-tmct> what talks to the payment module?
-…
-tmct> /callers checkout
-…
+Point it at a codebase's graph and the same engine answers structural questions.
+`examples/mini-webapp` ships in this repo, so this runs as written:
+
+```session cwd=repo
+$ node bin/tmct.mjs chat --repo examples/mini-webapp --ephemeral
+tmct> what does app.mjs talk to?
+src/server/router.mjs and src/handlers/tasks.mjs and src/handlers/users.mjs and src/lib/logger.mjs.
+tmct> what talks to store.mjs?
+src/handlers/tasks.mjs and src/handlers/users.mjs.
 tmct> /exit
 ```
 
@@ -177,7 +180,7 @@ resolves to a real graph traversal or declines honestly:
   (*because/although/while*), conditionals, and false-premise flags ("why
   does X still import Y" when it no longer does).
 
-The full catalog with measured coverage lives in `CAPABILITIES_1.7.3.md` and
+The full catalog with measured coverage lives in `CAPABILITIES_2.0.3.md` and
 the `BENCHMARK_*.md` reports.
 
 **Response finishing.** Before an answer prints, it is segmented into typed

package/ROADMAP.md

@@ -100,7 +100,30 @@ blending web-sourced facts with graph/operator facts.
 
 ## Design docs
 
-Every substantial design lives in its own `PLAN_*.md` at the repo root (active) or `archive/`
-(shipped and closed) — this file points to them, it doesn't repeat their content. `SKILL_*.md` docs
-specify the repeatable measurement/build cycles (benchmarks, the background strategy advisor,
-plain-prose writing). `HANDOVER.md` is the single current-open-items list.
+Every substantial design lives in its own `PLAN_*.md` at the repo root; `archive/` holds the shipped
+and closed ones. This file points to them, it doesn't repeat their content. Each plan states its own
+status in its opening lines — read it there, because a status quoted here would rot.
+
+| Plan | What it's for |
+| --- | --- |
+| [PLAN_ADVENTURE.md](PLAN_ADVENTURE.md) | a text adventure as an architectural stretch: imperative command grammar, NPC turn scheduler, room-look digest |
+| [PLAN_AGENTS.md](PLAN_AGENTS.md) | the governing plan for the multi-repo arc (marginalia, seonix, a pluggable LLM rung), with its own phase sequencing |
+| [PLAN_CHILD_CORPUS.md](PLAN_CHILD_CORPUS.md) | a wider default seed corpus, chosen by age of acquisition |
+| [PLAN_CLASS_QUERY.md](PLAN_CLASS_QUERY.md) | "list/count all X of class Y", reconciled against what already shipped |
+| [PLAN_CODE.md](PLAN_CODE.md) | program synthesis over tmct's closed DSLs, plus JS/HTML/CSS fragments and goal-directed program repair |
+| [PLAN_CONSISTENCY_CHECK.md](PLAN_CONSISTENCY_CHECK.md) | tmct as a consistency service for an LLM tool loop |
+| [PLAN_EMBEDDINGS.md](PLAN_EMBEDDINGS.md) | the semantic-similarity axis, and the way back to it |
+| [PLAN_GRAPH_SCAN.md](PLAN_GRAPH_SCAN.md) | seed and query cost at `init:xl`/`init:xxl` corpus scale |
+| [PLAN_GUESS_NUMBER.md](PLAN_GUESS_NUMBER.md) | closed-loop planning over hidden state, via belief-interval bisection |
+| [PLAN_MUD.md](PLAN_MUD.md) | persistent, shared tmct worlds over a `server:` memory backend |
+| [PLAN_NLU_BENCHMARKS.md](PLAN_NLU_BENCHMARKS.md) | scoring tmct on the CLINC150 and HWU64 intent sets |
+| [PLAN_OPEN_ITEMS.md](PLAN_OPEN_ITEMS.md) | the build order closing the backlog `HANDOVER.md` carries |
+| [PLAN_PARAPHRASE_VERIFICATION.md](PLAN_PARAPHRASE_VERIFICATION.md) | checking a paraphrase against the graph before it prints |
+| [PLAN_PURGE.md](PLAN_PURGE.md) | promote the load-bearing code, delete the dead weight |
+| [PLAN_REPO_INDEX.md](PLAN_REPO_INDEX.md) | tmct grows its own code parsers, ported from seonix |
+| [PLAN_SYLLOGIST.md](PLAN_SYLLOGIST.md) | the reasoning engine's incrementality and retraction horizon |
+| [PLAN_SYLLOGIST_EL_DL.md](PLAN_SYLLOGIST_EL_DL.md) | beyond OWL 2 RL: an EL classifier, then a DL tableau prover |
+
+`SKILL_*.md` docs specify the repeatable measurement and build cycles (the benchmarks, the capability
+audit, the background strategy advisor, plain-prose writing). `HANDOVER.md` is the single
+current-open-items list.

package/corpus/generated/ace-surface-variants.jsonl

@@ -1,3 +1,4 @@
+{"kind":"rescue","sentence":"Code search is a retrieval task","rescued":"Code hunt is a retrieval task","from":"search","to":"hunt","pos":"noun","synsetId":"00947217-n","sourceFile":"PLAN_EMBEDDINGS.md","provenance":"wordnet:00947217-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the one of trumps is a sure winner","from":"ace","to":"one","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the single of trumps is a sure winner","from":"ace","to":"single","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the unity of trumps is a sure winner","from":"ace","to":"unity","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}

package/corpus/generated/manifest.json

@@ -2,8 +2,8 @@
   "version": 1,
   "generated": "by scripts/generate-template-variants.mjs",
   "file": "ace-surface-variants.jsonl",
-  "rows": 17,
-  "bytes": 4774,
-  "sha256": "5364415f73f7a366371f1fdcc072b5059f81871ef48b106088f05b1441def569",
+  "rows": 18,
+  "bytes": 5012,
+  "sha256": "28fae5a3353f5dd1df365b4407e0ac2a5c66b98b232d8bf454fe3573f2c16760",
   "license": "CC-BY-4.0 (WordNet-derived synonym substitutions of Open English WordNet / SemCor example sentences and this repo's own MPL-2.0 docs prose — see corpus/generated/README.md)"
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@polycode-projects/the-mechanical-code-talker",
-  "version": "2.0.2",
+  "version": "2.2.0",
   "private": false,
   "type": "module",
   "description": "The Mechanical Code Talker (tmct) — a tolerant, offline, $0 chat surface that guides you toward precision queries about a software repository. ELIZA/PARRY-style but domain-obsessed with code. No model calls; no codebase index of its own.",

package/src/adapters/toml-config.mjs

@@ -139,7 +139,6 @@ export async function normalizeConfig(raw, { configDir } = {}) {
   if (t.impl_of_interface !== undefined) tune.implOfInterface = t.impl_of_interface;
   if (t.beam_search !== undefined) tune.beamSearch = t.beam_search;
   if (t.beam_width !== undefined) tune.beamWidth = t.beam_width;
-  if (t.embed_rank !== undefined) tune.embedRank = t.embed_rank;
   if (t.prose_layers !== undefined) tune.proseLayers = t.prose_layers;
   const exp = t.expansion || {};
   const expansion = {};

package/src/domain/ask-vocab.mjs

@@ -52,6 +52,9 @@ export const RELATIONS = {
     comment: "query-side union: imports (Module->Module) + calls (Module->Module) + callsSymbol (fn->fn).",
     verbs: [
       "uses", "use", "makes use of", "make use of",
+      // casual: "what does app.mjs talk to" is the union question a newcomer
+      // actually asks — imports and calls together, which is what "uses" is.
+      "talks to", "talk to",
       // gerund (g-drop normalization)
       "using",
     ],

package/src/domain/ask.mjs

@@ -2779,7 +2779,13 @@ export function traverse(graph, parsed, { contextId = null, prev = null, pinnedO
     }
     const edges = fwdKinds.flatMap((k) => edgesOfKind(graph, k)).filter((e) => e.subject === objMatch.id);
     const targets = edges.map((e) => graph.byId.get(e.object)).filter(Boolean);
-    const deduped = subjIsFineSymbol ? uniqueById(targets) : targets;
+    // Dedupe unconditionally: a forward answer enumerates distinct targets, so
+    // reaching one twice is never information. It matters for a query-side
+    // union (KIND_UNIONS' "uses" scans imports+calls+callsSymbol), where a
+    // module that both imports AND calls another was listed twice — the
+    // reverse traversal already collapses these, so the two directions
+    // disagreed about the same pair.
+    const deduped = uniqueById(targets);
     // Keep only matches of the asked class, so a forward answer never leaks
     // a wrong-class match once an entityType was actually asked for.
     let matches = deduped;

package/src/domain/codegraph.mjs

@@ -1,5 +1,4 @@
 import { lookupByProseTokens, proseLayerHits, splitIdentifierWords } from "./prose.mjs";
-import { cosine } from "./vector.mjs";
 import { CREATED_AT_PROP, UPDATED_AT_PROP } from "./memory/trust.mjs";
 
 // Pure (no-network, no-fs) query logic over the typed `entities` payload that the
@@ -511,15 +510,6 @@ const LIT_COMP_CAP = 4;
 const LIT_FRAC = 1.0;
 const LIT_CAP_FRAC = 0.9;
 
-// opt-in via embedRank + an injected embedder: static-embedding re-rank over
-// path/symbol/doc text read from the graph (never source), cached per-process in EMB_CACHE
-const EMB_FRAC = 0.2;
-const EMB_CAP_FRAC = 0.35;
-const EMB_TEXT_SYMBOL_CAP = 64;
-const EMB_TEXT_DOC_CAP = 12;
-const EMB_CACHE = new WeakMap(); // graph -> { embedder, texts, vecs: Map<moduleId, Float32Array> }
-let embedWarned = false;
-
 // opt-in via beamSearch: multi-ply adaptive expansion of the proximity nudge above.
 // Beam width is a margin relative to each ply's best score (not a fixed count), so a
 // weak-then-strong candidate isn't prematurely discarded. Successors are generated
@@ -544,30 +534,6 @@ const SPIRAL_HOP_DECAY = 0.6;
 const SPIRAL_PROX_FRAC = 0.2;
 const SPIRAL_PROX_CAP_FRAC = 0.35;
 
-/** embedRank: per-module embeddable text from path components + defined symbol
- *  names + doc first-lines, cached alongside the vectors in EMB_CACHE. */
-function moduleEmbedTexts(graph) {
-  const texts = new Map(); // moduleId -> text
-  const defIdx = definesIndex(graph);
-  const docs = new Map();  // moduleId -> [doc first-lines]
-  for (const ind of graph.individuals) {
-    const doc = (ind.attributes || []).find((a) => a.key === "doc")?.value;
-    if (!doc) continue;
-    const modId = (ind.class || "") === "Module" ? ind.id : moduleIdOf(graph, ind);
-    if (!modId) continue;
-    let arr = docs.get(modId);
-    if (!arr) docs.set(modId, (arr = []));
-    if (arr.length < EMB_TEXT_DOC_CAP) arr.push(String(doc).split("\n")[0]);
-  }
-  for (const ind of graph.individuals) {
-    if ((ind.class || "") !== "Module") continue;
-    const parts = String(ind.label).split(/[^a-zA-Z0-9_]+/).filter(Boolean);
-    const syms = (defIdx.get(ind.id) || []).slice(0, EMB_TEXT_SYMBOL_CAP);
-    texts.set(ind.id, [...parts, ...syms, ...(docs.get(ind.id) || [])].join(" "));
-  }
-  return texts;
-}
-
 /** Split a lowercased path label into boundary components: django/utils/text.py →
  *  {django,utils,text,py}. Component equality (not substring) stops "text" matching "ci<text>". */
 function pathComponents(labelLc) {
@@ -773,7 +739,7 @@ export function spiralExpand(graph, scored = [], {
  *  IDF-weights each query token, scores path/symbol/exact-symbol matches, and
  *  re-ranks with a bounded import-proximity bonus. Pure; deterministic. */
 function scoreModules(graph, tokens, opts = {}) {
-  const { demoteNonProd = false, callAdjacency = false, implOfInterface = false, beamSearch = false, spiral = false, proseBoost = false, proseLayers = false, literalMention = false, embedRank = false, rawQuery = "" } = opts;
+  const { demoteNonProd = false, callAdjacency = false, implOfInterface = false, beamSearch = false, spiral = false, proseBoost = false, proseLayers = false, literalMention = false, rawQuery = "" } = opts;
   const beamWidth = Number.isFinite(opts.beamWidth) && opts.beamWidth > 0 ? opts.beamWidth : 8;
   const defIdx = definesIndex(graph);
   // Precompute each module's path components + defined-symbol exact/component sets, once.
@@ -967,36 +933,6 @@ function scoreModules(graph, tokens, opts = {}) {
       s.score += Math.min(signal * PROSE_LAYER_FRAC, s.score * PROSE_LAYER_CAP_FRAC);
     }
   }
-  // embedRank: the embedder is injected so this module stays fs-free; absent -> a one-time
-  // stderr note, never a failure
-  if (embedRank) {
-    if (!opts.embedder) {
-      if (!embedWarned) {
-        embedWarned = true;
-        process.stderr.write("tmct: embedRank requested but no embedder available (weights not fetched? see `npm run refs:embeddings`) — flag is a no-op\n");
-      }
-    } else if (scored.length) {
-      const embedder = opts.embedder;
-      let cache = EMB_CACHE.get(graph);
-      if (!cache || cache.embedder !== embedder) {
-        cache = { embedder, texts: moduleEmbedTexts(graph), vecs: new Map() };
-        EMB_CACHE.set(graph, cache);
-      }
-      const qv = embedder.embed(rawQuery || tokens.join(" "));
-      let maxBase = 0;
-      for (const s of scored) maxBase = Math.max(maxBase, s.score);
-      for (const s of scored) {
-        let v = cache.vecs.get(s.ind.id);
-        if (!v) {
-          v = embedder.embed(cache.texts.get(s.ind.id) || String(s.ind.label));
-          cache.vecs.set(s.ind.id, v);
-        }
-        const sim = Math.max(0, cosine(qv, v)); // negative similarity never penalises
-        if (!sim) continue;
-        s.score += Math.min(sim * maxBase * EMB_FRAC, s.score * EMB_CAP_FRAC);
-      }
-    }
-  }
   // beamSearch (opt-in): multi-ply generalization of the single-hop families above.
   if (beamSearch && scored.length > 1) beamExpand(graph, scored, beamWidth);
   // SPIRAL (opt-in): bounded-radius ego walk that may introduce lexically-invisible modules — runs
@@ -1024,9 +960,9 @@ export function searchModulesRanked(graph, query, opts = {}) {
   const tokens = raw.toLowerCase().split(/[^a-z0-9_]+/).filter(Boolean);
   if (!tokens.length) return [];
   // literalMention needs the query BEFORE tokenization (the tokenizer destroys the dotted refs
-  // it matches on) and embedRank embeds the raw phrasing; threaded only when a flag that
-  // consumes it is on, so the OFF path is provably unchanged.
-  const effOpts = (opts.literalMention || opts.embedRank) ? { ...opts, rawQuery: raw } : opts;
+  // it matches on); threaded only when a flag that consumes it is on, so the OFF path is
+  // provably unchanged.
+  const effOpts = opts.literalMention ? { ...opts, rawQuery: raw } : opts;
   return scoreModules(graph, tokens, effOpts).map((s) => ({ path: String(s.ind.label), score: s.score }));
 }
 

package/src/domain/markdown-links.mjs

@@ -0,0 +1,55 @@
+// markdown-links.mjs — pull the relative link targets out of a markdown string.
+// Pure: a string in, `[{ target, line }]` out, no filesystem and no imports, so
+// the CI jobs that run without `npm ci` can reach it.
+//
+// Code is not prose. A doc that writes `[text](target)` inside backticks is
+// showing you what a link looks like, not making one, and a checker that cannot
+// tell the difference reports the example as a broken link to a file named
+// "target". So the spans are blanked before the link patterns run — blanked
+// rather than cut, because every offset behind them still has to name the right
+// line number.
+
+const blank = (line) => line.replace(/[^\n]/g, " ");
+const FENCE = /^[ \t]{0,3}(`{3,}|~{3,})/;
+
+/** Replace every fenced block and inline code span with spaces, keeping the
+ *  string's length and its newlines so later offsets still map to their line.
+ *  Fences are walked line by line: a lazy multiline regex stops at the end of
+ *  the opening fence's own line and blanks only the markers. */
+export function blankCodeSpans(markdown) {
+  let fence = null;
+  const lines = markdown.split("\n").map((line) => {
+    const marker = line.match(FENCE)?.[1];
+    if (fence) {
+      if (marker && marker[0] === fence[0] && marker.length >= fence.length) fence = null;
+      return blank(line);
+    }
+    if (marker) fence = marker;
+    return marker ? blank(line) : line;
+  });
+  return lines.join("\n").replace(/(`+)[\s\S]*?\1/g, blank);
+}
+
+// Inline links/images: [text](target "title") — target ends at the first
+// whitespace or closing paren. Reference definitions: [label]: target.
+const INLINE_LINK = /!?\[[^\]]*\]\(([^)\s]+)(?:\s+"[^"]*")?\)/g;
+const REFERENCE_DEF = /^\s{0,3}\[[^\]]+\]:\s+(\S+)/gm;
+
+/** Every repo-relative link target in `markdown`, with the 1-based line it sits
+ *  on. External URLs, bare #anchors and absolute paths are out of scope. */
+export function relativeTargets(markdown) {
+  const prose = blankCodeSpans(markdown);
+  const targets = [];
+  for (const regex of [INLINE_LINK, REFERENCE_DEF]) {
+    for (const match of prose.matchAll(regex)) {
+      let target = match[1];
+      if (/^(https?|mailto|ftp):/i.test(target)) continue;
+      if (target.startsWith("#") || target.startsWith("/") || target.startsWith("<")) continue;
+      target = decodeURIComponent(target.split("#")[0].split("?")[0]);
+      if (!target) continue;
+      const line = prose.slice(0, match.index).split("\n").length;
+      targets.push({ target, line });
+    }
+  }
+  return targets;
+}