@polycode-projects/the-mechanical-code-talker 0.6.0 → 0.7.1

package/README.md

@@ -49,6 +49,31 @@ under a guard that proves the protected spans came through byte-for-byte. Today
 that pass fixes the a/an article defect; broader voice and agreement rules are
 implemented but parked until they earn their place on the benchmark.
 
+## How it guides you
+
+When you touch a **concept** without asking a precise question — "what is a
+class", "what about imports", "what calls are there" — tmct answers in three
+bands instead of dead-ending:
+
+1. the **definition** (a plain-English one-liner: *"A class is a template that
+   defines the structure and behaviour of objects."* / *"To import is to bring
+   another module's definitions into the current one."*);
+2. **real instances from your graph** — *"In this codebase, for example: Record,
+   Task and User (10 classes)"*, or actual edges *"src/core/store.mjs imports
+   src/core/model.mjs (18 import edges)"*;
+3. **guided follow-ups** — two or three concrete next questions, each one
+   *pre-checked against your graph* so every suggestion is guaranteed to resolve:
+   *"Want to go deeper? Try: which classes inherit from Record / what does Task
+   contain / where is User defined"*.
+
+It fires for both **noun** concepts (class, module, function, method) and
+**relation** concepts (imports, calls, contains, inherits, tests), and only when
+tmct genuinely knows the concept *and* has instances of it — otherwise the honest
+miss stands. The effect is a conversation that drills down from a vague opener to
+a useful answer without ever hitting a wall. Natural phrasings are routed to the
+capability you meant: *"what functions are in Task"* → its members, *"what
+defined saveStore"* → where it's defined.
+
 ## How it remembers
 
 tmct's memory has two layers, both fed by every parsed request and response and
@@ -60,9 +85,12 @@ by cleaned session logs:
   relevance rather than loaded wholesale.
 
 With no graph at all, tmct starts empty and remembers what you tell it — the
-`.tmct/` graph is created from the conversation. Committed corpuses seed the
-vocabulary; a filtered **ConceptNet slice** (CC-BY-SA 4.0) is planned — see
-`ROADMAP.md` Phase 2.
+`.tmct/` graph is created from the conversation. On a first run it seeds the
+committed vocabulary so it knows what it's talking about from turn one: a curated
+**SEON** software ontology plus the whole filtered **ConceptNet slice**
+(CC-BY-SA 4.0) — every term carries an English definition, so "what is a cache?"
+answers offline, from disk, on turn one. `--ephemeral` (used by the shipped
+`npm run example:*` demos) reads a graph but writes nothing back.
 
 ### Provenance and trust
 

package/ROADMAP.md

@@ -550,6 +550,51 @@ Repository Interface library-surface work (Phase 8) settling the extraction boun
 `PLAN_OSS_ACE_PARSER.md`. Sibling publish-candidates (the bounded-Damerau fuzzy matcher, the
 PageRank+IDF block ranker) may follow the same path if there is demand.
 
+## Phase 10 — Conversational competence & onboarding (0.6.0 → 0.7.0)
+
+*(Operator-directed, from live new-user testing.)* Once a graph is loaded the engine is strong;
+the weak surfaces were the FIRST RUN and the VAGUE question. This phase makes the miss graceful,
+the empty state honest, and the vague touch a guided answer — realising item 1's "tolerant, guides
+you toward precision" promise on the conversational surface.
+
+- **Onboarding UX (shipped 0.6.0):** the grammar wall moved behind `/help` (a short, tailored miss
+  instead); intent lanes for memory/teach ("remember that X"), meta/self ("what is this codebase",
+  "what do you know"), routed only when a graph query would miss; empty/degenerate-graph
+  orientation that distinguishes CODE STRUCTURE (needs a `.tmct/graph.json` via a producer or
+  `--repo`; tmct reads graphs, it does not index code) from VOCABULARY (`tmct init`/bootstrap seeds
+  concepts); `TMCT_GRAPH_FILE` honoured by chat; slash-optional commands (`stats`≡`/stats`);
+  `/memory` explore hooks; up/down-arrow prompt history in the TUI.
+- **Knowledge (shipped 0.6.0 → 0.7.0):** the curated `corpus/seon` ontology — a software-sense
+  definition for EVERY lexicon term, language-neutral (Java/C#/Python `class` → one concept); the
+  ConceptNet slice quality-filtered (word-sense noise cut) and regrown to ~40k facts; tier-2
+  specialised corpuses (aws/python/java) with `tmct init --corpus`; batched `appendFacts` (one
+  write, 419s→2.5s) enabling **seed-all** so a fresh repo knows the whole curated vocabulary.
+- **The concept force (shipped 0.7.0):** a vague touch on a concept X, where tmct knows X and has instances,
+  answers in three bands — **the definition** (from `corpus/seon`), **the examples** (real code-graph
+  + memory instances of X), and **a soft guided follow-up** ("Want to go deeper?" + 2–3 questions
+  built from the real instances × the query shapes valid for that kind, each pre-validated to
+  resolve). Applies to NOUN concepts (`what is a class` → define + Base/Widget/Button + drill-ins)
+  AND **RELATION concepts** (`what about imports` / `what calls are there` → the verb definition +
+  example edges + guided queries), fixing the vague-query dead-ends. Fact rendering is
+  de-anthropomorphised (no first-person "i learned:" over-claim — corpus facts read as data +
+  provenance; `you told me` stays for operator-asserted facts); listings cap at 32 with a "say
+  'more'" pagination that holds the remainder in session state.
+- **Dead-end routing + read-only demos (shipped 0.7.0):** natural drill-down phrasings are routed
+  onto the canonical shapes they mean — `what functions are in X` → members-of-class, `what defined
+  X` → where-is-X-defined, a no-context `what about X` → the concept/relation force (the discourse
+  continuation still wins when there IS a prior answer). `tmct chat --ephemeral` (and the
+  `npm run example:*` demos) reads a graph but writes nothing back, so a checked-in example is never
+  dirtied by a demo run.
+- **The dialogue-flow loop (`SKILL_CHAT_PLAYTEST.md`, 0.7.0):** a fast, qualitative tuning loop that
+  complements the LLM-judge benchmark — Claude plays a curious user, hunts *dead-ends* (walls,
+  "unknown qualifier", phrasing-misses, invited follow-ups the engine can't take), fixes them by
+  ROUTING to existing capabilities, replays the same conversations until they flow, freezes them as
+  regression transcripts, then ratchets the complexity tier. The drill-down transcript above is its
+  first frozen fixture (`test/chatflow-drilldown.test.mjs`).
+- **Measured** by the version-matched benchmark (`CHATBENCH_<version>` per `SKILL_TUNING_CYCLE.md`),
+  with new graded cells for the miss / empty-graph / concept-touch surfaces so these become
+  regression-protected levers, not one-off polish.
+
 ## Phase LATER — recognized, deferred, not now
 
 Features we have deliberately shaped seams for but will not build until the phases above have

package/bin/tmct.mjs

@@ -34,6 +34,7 @@ software repository. No model calls; no codebase index of its own.
 Usage:
   tmct                         interactive chat (the headline surface)
   tmct chat [--repo <abs>]     chat over a specific repo's graph
+       [--ephemeral]           read the graph but write nothing back (demo/read-only)
        [--plain]               force the plain readline shell (the default when
                                stdin/stdout is not a terminal)
   tmct memory [--repo <abs>]   what tmct remembers: facts, utterances, sessions,
@@ -267,16 +268,21 @@ async function main() {
     const rest = process.argv.slice(3);
     const i = rest.indexOf("--repo");
     const repoPath = i !== -1 ? rest[i + 1] : undefined;
+    // `--ephemeral` (used by the shipped `npm run example:*` scripts): read the
+    // target graph but write nothing back — no session folded into the committed
+    // code graph, no .tmct/memory dropped under it. A demo you can run repeatedly
+    // on a checked-in example without ever dirtying it.
+    const ephemeral = rest.includes("--ephemeral");
     // The shell gate: a real terminal gets the full-screen Ink TUI; `--plain` or a
     // non-TTY stream (pipes, scripts, the test suite) gets the readline shell. Both
     // drive the same createSession sink — only the drawing differs.
     const plain = rest.includes("--plain") || !process.stdin.isTTY || !process.stdout.isTTY;
     if (plain) {
       const { runChat } = await import("../src/chat.mjs");
-      await runChat({ repoPath });
+      await runChat({ repoPath, ephemeral });
     } else {
       const { runTui } = await import("../src/tui/app.mjs");
-      await runTui({ repoPath });
+      await runTui({ repoPath, ephemeral });
     }
     return;
   }

package/corpus/conceptnet/README.md

@@ -15,7 +15,9 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
 - **Source:** ConceptNet 5.7.0 assertions dump,
   `https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-assertions-5.7.0.csv.gz`
   (published 2019-07-03).
-- **Retrieved + filtered:** 2026-07-04.
+- **Retrieved + filtered:** 2026-07-04; **regrown to the ~40k tier-1 target on
+  2026-07-05** (same dump, same licence) by widening the tech-seed domain and
+  raising the size budget — see "Growing the slice" below.
 - **Why the dump, not the API:** the public API (`api.conceptnet.io`) was
   hard-down (HTTP 502 from its nginx front-end on every request across ~15
   attempts over 10+ minutes on 2026-07-04), so the slice was stream-filtered
@@ -33,12 +35,16 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
    policy: `/r/EtymologicallyRelatedTo`, `/r/EtymologicallyDerivedFrom`,
    `/r/ExternalURL` (etymology noise and link-outs — no consumer in tmct).
 3. **Tech-domain seed terms**: at least one endpoint's bare term is in the
-   90-term software/tech seed list exported as `SEED_TERMS` from
+   tech seed list — the ~90-term base `SEED_TERMS` exported from
    `fetch-slice.mjs` (software, computer, program, code, module, function,
    database, server, network, bug, test, file, memory, algorithm, keyboard,
-   programmer, repository, commit, …).
+   programmer, repository, commit, …) **plus** the ~230-term `EXTRA_SEEDS`
+   growth list in `filter-dump.mjs` (programming languages, frameworks, data
+   structures, cloud/infra, protocols, tools, ML — python, java, docker,
+   kubernetes, git, neural_network, tcp, kernel, hashtable, …), added
+   2026-07-05 to reach the ~40k tier-1 target while staying in the tech domain.
 4. **Dedupe** by `(start, rel, end)`, keeping the higher weight.
-5. **Size budget** (committed slice ≤ 1.5 MB; target ~1.4 MB), **two-tier**:
+5. **Size budget** (committed slice ≤ 5 MB; target ~4.3 MB), **two-tier**:
    assertions whose relation maps to an ACE-OWL pattern (`ace != "none"` in
    `conceptnet-map.toml`) are kept first, weight-descending; `ace = "none"`
    relations (`RelatedTo`, `Synonym`, …) fill the remaining budget — they
@@ -46,7 +52,7 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
    facts.
 6. Deterministic output order: `(rel, start, end)`.
 
-## Quality-filter pass (2026-07-05)
+## Quality-filter pass (regrown slice, 2026-07-05)
 
 `filter-dump.mjs` keeps the DATA honest (tech-seed match, canonical relations,
 budget) but not the SEMANTICS: ConceptNet's crowd-sourced "Verbosity"/Open-Mind
@@ -66,66 +72,66 @@ row):
   evidence-based set (`elegance, evil, gloom, unreality, universalism, dumb,
   free, junk`) — never a class
 
-**Result: 14,258 → 13,880 rows (378 cut), 1,399,979 → 1,348,361 bytes.** Cuts by
-reason: sentence-fragment 328, single-char 27, numeric 11, opinion-object 8,
-definitional-phrase 4. Of the 4,170 seedable (mapped, `ace≠none`) facts, 286
-noise facts were removed, leaving **3,884 clean seedable facts**. No relation
-disappeared entirely (the drift guard stays satisfied). Re-run any time with:
+**Result (2026-07-05 regrow): 45,633 → 44,947 rows (686 cut), 4,308,850 →
+4,220,629 bytes.** Cuts by reason: sentence-fragment 613, single-char 48,
+numeric 11, opinion-object 8, definitional-phrase 6. Of the seedable (mapped,
+`ace≠none`) facts, the clean slice carries **6,255** — up from 3,884 in the
+1.35 MB slice. No relation disappeared entirely (the drift guard stays
+satisfied). Re-run any time with:
 
 ```bash
 node corpus/conceptnet/quality-filter.mjs --in-place corpus/conceptnet/slice.jsonl
 ```
 
-## Growing the slice toward ~40k facts — the budget blocker
+## Growing the slice toward the ~40k tier-1 target (done 2026-07-05)
 
-Growing tier-1 to the operator's ~40k-fact target is **feasible data-wise but
-blocked by the committed budget**: the ConceptNet dump S3 endpoint is reachable,
-but `test/corpus-conceptnet.test.mjs` asserts `size <= 1_500_000`, and the clean
-slice is already 1.35 MB (13,880 rows). 40k facts is ≈ 4 MB — 2.6× over the cap.
-Reaching it needs a **Wave-2 policy change** (raise `MAX_BYTES` in
-`filter-dump.mjs` AND the budget assertion in the test), which is out of scope
-for a data-only pass. When the budget is raised, regrow with a bigger seed list
-+ budget, then re-run the quality filter:
+The operator's ~40k-fact tier-1 target is **shipped**: the slice was regrown
+from the same ConceptNet dump by (a) widening the tech domain with the
+`EXTRA_SEEDS` list in `filter-dump.mjs` (~230 tech terms — languages,
+frameworks, data structures, cloud/infra, protocols, tools, ML) and (b) raising
+`MAX_BYTES` in `filter-dump.mjs` from 1.4 MB to 4.5 MB with the matching test
+budget assertion (`test/corpus-conceptnet.test.mjs`) raised from 1.5 MB to
+5 MB. The widened seed set matched **45,633 unique en→en assertions** (all under
+budget, so no tier-trimming was needed this pass), and the quality filter
+trimmed them to the committed **44,947 clean facts**. Regenerate with:
 
 ```bash
-# 1. widen the domain and the budget (edit SEED_TERMS in fetch-slice.mjs, MAX_BYTES in filter-dump.mjs)
+# 1. stream-filter the dump (widened seeds + raised budget already live in filter-dump.mjs)
 curl -s https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-assertions-5.7.0.csv.gz \
   | gunzip -c | node corpus/conceptnet/filter-dump.mjs > corpus/conceptnet/slice.jsonl
 # 2. re-apply the semantic quality filter
 node corpus/conceptnet/quality-filter.mjs --in-place corpus/conceptnet/slice.jsonl
-# 3. raise the test's budget assertion to match
 ```
 
-## Row counts (pre-filter baseline, 2026-07-04)
+## Row counts (regrown slice, 2026-07-05)
 
-**14,258 assertions, 1,399,979 bytes** — the raw filter-dump output, BEFORE the
-2026-07-05 quality-filter pass above trimmed it to 13,880 rows / 1,348,361 bytes
-(34,074,917 dump lines scanned;
-28,802 unique en→en seed assertions matched = 4,170 mappable + 24,632
-`ace="none"`; ALL 4,170 mappable kept, 10,088 none-rows fill the budget).
-29 of the 31 non-filtered canonical relations are present:
+**44,947 clean assertions, 4,220,629 bytes** — the quality-filtered committed
+slice (raw filter-dump output was 45,633 rows / 4,308,850 bytes; 34,074,917 dump
+lines scanned; 45,633 unique en→en seed assertions matched = 6,670 mappable +
+38,963 `ace="none"`; ALL kept under the 4.5 MB budget, then 686 noise rows cut).
+30 of the 31 non-filtered canonical relations are present; **6,255 seedable
+(`ace≠none`) facts**:
 
 | Relation | Rows | | Relation | Rows |
 |---|---|---|---|---|
-| `/r/RelatedTo` | 4911 | | `/r/HasA` | 39 |
-| `/r/HasContext` | 2634 | | `/r/HasProperty` | 27 |
-| `/r/IsA` | 2594 | | `/r/Antonym` | 25 |
-| `/r/DerivedFrom` | 1906 | | `/r/MotivatedByGoal` | 25 |
-| `/r/AtLocation` | 459 | | `/r/SimilarTo` | 20 |
-| `/r/Synonym` | 368 | | `/r/DistinctFrom` | 19 |
-| `/r/UsedFor` | 312 | | `/r/DefinedAs` | 16 |
-| `/r/FormOf` | 224 | | `/r/MadeOf` | 14 |
-| `/r/CapableOf` | 164 | | `/r/CausesDesire` | 12 |
-| `/r/MannerOf` | 141 | | `/r/HasLastSubevent` | 12 |
-| `/r/PartOf` | 115 | | `/r/Causes` | 11 |
-| `/r/HasPrerequisite` | 103 | | `/r/CreatedBy` | 10 |
-| `/r/HasSubevent` | 42 | | `/r/Desires` | 7 |
-| `/r/ReceivesAction` | 40 | | `/r/HasFirstSubevent` | 7 |
-| | | | `/r/LocatedNear` | 1 |
-
-Absent from the slice (nothing matched the seed terms): `/r/ObstructedBy`,
-`/r/SymbolOf` — both still have mapping rows, so a regenerated slice that
-surfaces them stays covered.
+| `/r/RelatedTo` | 29016 | | `/r/HasSubevent` | 42 |
+| `/r/HasContext` | 4376 | | `/r/DistinctFrom` | 32 |
+| `/r/IsA` | 4173 | | `/r/HasProperty` | 30 |
+| `/r/DerivedFrom` | 3388 | | `/r/ReceivesAction` | 27 |
+| `/r/Synonym` | 1126 | | `/r/MotivatedByGoal` | 26 |
+| `/r/AtLocation` | 642 | | `/r/MadeOf` | 20 |
+| `/r/FormOf` | 528 | | `/r/Causes` | 18 |
+| `/r/UsedFor` | 378 | | `/r/CreatedBy` | 15 |
+| `/r/CapableOf` | 231 | | `/r/CausesDesire` | 13 |
+| `/r/MannerOf` | 230 | | `/r/Desires` | 9 |
+| `/r/PartOf` | 196 | | `/r/HasLastSubevent` | 8 |
+| `/r/Antonym` | 155 | | `/r/HasFirstSubevent` | 7 |
+| `/r/HasPrerequisite` | 108 | | `/r/LocatedNear` | 3 |
+| `/r/SimilarTo` | 102 | | `/r/DefinedAs` | 2 |
+| `/r/HasA` | 45 | | `/r/SymbolOf` | 1 |
+
+Absent from the slice (nothing matched the seed terms): `/r/ObstructedBy` — it
+still has a mapping row, so a regenerated slice that surfaces it stays covered.
 
 ## How to regenerate / extend
 
@@ -139,11 +145,12 @@ curl -s https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-asse
   | node corpus/conceptnet/filter-dump.mjs > corpus/conceptnet/slice.jsonl
 ```
 
-To extend the domain, add seed terms to `SEED_TERMS` in `fetch-slice.mjs` and
-re-run. `npm test` guards the contract: every relation present in the slice
-must have a row in `src/corpus/conceptnet-map.toml` (drift guard), en→en shape
-and the ≤ 1.5 MB budget are asserted, and the seeding path is exercised
-end-to-end.
+To extend the domain, add seed terms to `EXTRA_SEEDS` in `filter-dump.mjs` (the
+dump route that produces the committed slice) or `SEED_TERMS` in
+`fetch-slice.mjs` (the API route) and re-run. `npm test` guards the contract:
+every relation present in the slice must have a row in
+`src/corpus/conceptnet-map.toml` (drift guard), en→en shape and the ≤ 5 MB
+budget are asserted, and the seeding path is exercised end-to-end.
 
 ## Consumers
 

package/corpus/conceptnet/filter-dump.mjs

@@ -30,8 +30,65 @@ import { createInterface } from "node:readline";
 import { SEED_TERMS, CANONICAL_RELS, FILTERED_RELS, bareEnTerm } from "./fetch-slice.mjs";
 import { loadMap } from "../../src/corpus/conceptnet.mjs";
 
-const MAX_BYTES = 1_400_000; // committed-slice budget (hard cap 1.5 MB)
-const SEEDS = new Set(SEED_TERMS);
+const MAX_BYTES = 4_500_000; // committed-slice budget (hard cap 5 MB), grown for the ~40k tier-1 target
+
+// Tier-1 growth seeds (2026-07-05): the original SEED_TERMS in fetch-slice.mjs
+// stay the canonical ~90-term base; this list WIDENS the tech domain toward the
+// ~40k-fact target without leaving the software/tech world — languages,
+// frameworks, data structures, cloud/infra, protocols, tools, ML. Kept here (an
+// owned maintainer file) rather than in fetch-slice.mjs so the API route's seed
+// contract is untouched. Union with SEED_TERMS below.
+const EXTRA_SEEDS = [
+  // programming languages & ecosystems
+  "python", "java", "javascript", "typescript", "ruby", "perl", "php", "golang",
+  "rust", "kotlin", "swift", "scala", "haskell", "lisp", "clojure", "erlang",
+  "fortran", "cobol", "pascal", "assembly", "sql", "html", "css", "json", "xml",
+  "yaml", "markdown", "bash", "powershell",
+  // paradigms & concepts
+  "programming", "coding", "recursion", "iteration", "inheritance",
+  "polymorphism", "abstraction", "encapsulation", "concurrency", "parallelism",
+  "multithreading", "asynchronous", "synchronization", "serialization",
+  "optimization", "refactoring", "debugging", "compilation", "runtime",
+  "object_oriented", "functional_programming", "computer_science",
+  "software_engineering", "computer_programming",
+  // data structures & algorithms
+  "hash", "hashtable", "tree", "graph", "list", "tuple", "dictionary", "heap",
+  "matrix", "vector", "node", "recursion", "sorting", "searching", "linked_list",
+  "binary_tree", "hash_table", "data_type", "datatype",
+  // frameworks, tools & platforms
+  "git", "github", "docker", "kubernetes", "jenkins", "react", "angular",
+  "nodejs", "django", "flask", "spring", "rails", "tensorflow", "pytorch",
+  "hadoop", "kafka", "redis", "mongodb", "mysql", "postgresql", "sqlite",
+  "elasticsearch", "apache", "nginx", "wordpress", "eclipse", "vim", "emacs",
+  // hardware & systems
+  "kernel", "driver", "register", "transistor", "semiconductor", "motherboard",
+  "microprocessor", "microcontroller", "gpu", "ram", "rom", "ssd", "firmware",
+  "bootloader", "filesystem", "partition", "peripheral", "microchip",
+  "circuit_board", "integrated_circuit", "hard_drive", "graphics_card",
+  // networking & protocols
+  "packet", "bandwidth", "latency", "router", "firewall", "gateway", "proxy",
+  "dns", "tcp", "ftp", "smtp", "ssh", "ssl", "vpn", "ethernet", "wifi",
+  "bluetooth", "socket", "port", "packet_switching", "ip_address",
+  // web & the net
+  "url", "cookie", "session", "webpage", "hyperlink", "frontend", "backend",
+  "webserver", "webservice", "hosting", "domain", "html5", "ajax",
+  // cloud, infra & devops
+  "cloud_computing", "container", "virtualization", "serverless",
+  "microservice", "deployment", "devops", "pipeline", "infrastructure",
+  "datacenter", "cluster", "scalability", "availability", "redundancy",
+  // security
+  "authentication", "authorization", "cryptography", "hashing", "firewall",
+  "malware", "virus", "vulnerability", "cybersecurity", "cipher",
+  // data & ml
+  "dataset", "database", "datastore", "query", "index", "schema", "transaction",
+  "neural_network", "deep_learning", "classification", "regression",
+  "clustering", "training", "inference", "algorithm", "computation",
+  "big_data", "data_mining", "data_science", "analytics",
+  // concurrency & os primitives
+  "semaphore", "mutex", "deadlock", "scheduler", "interrupt", "syscall",
+  "daemon", "multitasking", "buffer", "pipe",
+];
+const SEEDS = new Set([...SEED_TERMS, ...EXTRA_SEEDS]);
 const termOf = (uri) => uri.slice("/c/en/".length);
 
 const byKey = new Map(); // "start rel end" -> row