@polycode-projects/the-mechanical-code-talker 0.5.0 → 0.7.0

package/README.md

@@ -2,12 +2,11 @@
 
 `@polycode-projects/the-mechanical-code-talker`
 
-A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage —
+A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage:
 pattern-driven, best-efforts, and obsessed with software the way PARRY was
-obsessed with the mafia. No model calls anywhere in the product: interpretation
-is mechanical (deterministic language libraries, template sets, committed
-corpuses), memory is a graph on disk, and every answer is either grounded or an
-honest miss.
+obsessed with the mafia. No model calls anywhere. Interpretation is mechanical
+(deterministic language libraries, template sets, committed corpuses). Memory is
+a graph on disk. Every answer is either grounded or an honest miss.
 
 ```
 $ tmct
@@ -50,6 +49,31 @@ under a guard that proves the protected spans came through byte-for-byte. Today
 that pass fixes the a/an article defect; broader voice and agreement rules are
 implemented but parked until they earn their place on the benchmark.
 
+## How it guides you
+
+When you touch a **concept** without asking a precise question — "what is a
+class", "what about imports", "what calls are there" — tmct answers in three
+bands instead of dead-ending:
+
+1. the **definition** (a plain-English one-liner: *"A class is a template that
+   defines the structure and behaviour of objects."* / *"To import is to bring
+   another module's definitions into the current one."*);
+2. **real instances from your graph** — *"In this codebase, for example: Record,
+   Task and User (10 classes)"*, or actual edges *"src/core/store.mjs imports
+   src/core/model.mjs (18 import edges)"*;
+3. **guided follow-ups** — two or three concrete next questions, each one
+   *pre-checked against your graph* so every suggestion is guaranteed to resolve:
+   *"Want to go deeper? Try: which classes inherit from Record / what does Task
+   contain / where is User defined"*.
+
+It fires for both **noun** concepts (class, module, function, method) and
+**relation** concepts (imports, calls, contains, inherits, tests), and only when
+tmct genuinely knows the concept *and* has instances of it — otherwise the honest
+miss stands. The effect is a conversation that drills down from a vague opener to
+a useful answer without ever hitting a wall. Natural phrasings are routed to the
+capability you meant: *"what functions are in Task"* → its members, *"what
+defined saveStore"* → where it's defined.
+
 ## How it remembers
 
 tmct's memory has two layers, both fed by every parsed request and response and
@@ -61,9 +85,12 @@ by cleaned session logs:
   relevance rather than loaded wholesale.
 
 With no graph at all, tmct starts empty and remembers what you tell it — the
-`.tmct/` graph is created from the conversation. Committed corpuses seed the
-vocabulary; a filtered **ConceptNet slice** (CC-BY-SA 4.0) is planned — see
-`ROADMAP.md` Phase 2.
+`.tmct/` graph is created from the conversation. On a first run it seeds the
+committed vocabulary so it knows what it's talking about from turn one: a curated
+**SEON** software ontology plus the whole filtered **ConceptNet slice**
+(CC-BY-SA 4.0) — every term carries an English definition, so "what is a cache?"
+answers offline, from disk, on turn one. `--ephemeral` (used by the shipped
+`npm run example:*` demos) reads a graph but writes nothing back.
 
 ### Provenance and trust
 
@@ -125,6 +152,41 @@ or a bare user gets a working install in one command.
 > Install-size note: tmct depends on wink-nlp's deterministic English language
 > model (~3.8 MB installed). That model is a lookup table, not an LLM.
 
+### Try it on an example graph
+
+tmct *consumes* a code graph at `<repo>/.tmct/graph.json` — it does not build
+one. Two ready-made example graphs ship in `examples/` so you can see it answer
+real questions with no setup:
+
+```bash
+npm run example:mini       # "Questboard" — a small task-tracker web app (12 modules)
+npm run example:polyglot   # one shared OWL vocabulary across Java / Python / C#
+npm run chat:repo -- ./any/path   # chat over any repo that has a .tmct/graph.json
+```
+
+Questions the **mini-webapp** graph answers:
+
+```
+what classes are there
+describe Task
+how many modules
+which modules import src/core/model.mjs
+what tests cover src/handlers/tasks.mjs
+```
+
+The **polyglot** graph shows the language-neutral idea — Java, Python and C#
+entities all typed to the same `seon:Class` / `seon:Method` / `seon:Module`
+concepts, so one query reasons across every language at once:
+
+```
+how many classes          # 9 — Java + Python + C# counted as one concept
+what classes are there     # Order (Java), Inventory (Python), PaymentService (C#), …
+which modules define PaymentService
+```
+
+See `examples/mini-webapp/README.md` and `examples/polyglot/README.md` for the
+full tours.
+
 ### As a library
 
 ```js
@@ -138,29 +200,31 @@ extension surface.
 ## The repository interface
 
 tmct is not an indexer, so it consumes a graph through a typed contract any
-producer can implement. That contract is now first-class: a **versioned
-(1.0.0), OWL-grounded, machine-readable service definition**
-(`docs/repository-interface.md` + a JSON schema) of every service, its
-arguments, result types, and error contract. A **miss is a value, not a throw** —
-the interface models "no answer" explicitly. tmct ships **reference providers**
-(a fixture graph and the empty/bootstrap graph) that implement every service,
-and a **runnable conformance suite**: tmct's own providers pass it in `npm test`,
-and any external graph producer (seonix first) runs the *same* suite against its
-native implementation to claim conformance. Conformance is the suite, not prose.
-The design deliberately inverts the original relationship — tmct was lifted out
-of seonix; now seonix reorients as a *user* that imports the tmct library and
-exposes its graph to tmct as a service, with the LLM agent staying outside tmct
-exactly as the no-LLM ethos requires.
+producer can implement. That contract is first-class: a **versioned (1.0.0),
+OWL-grounded, machine-readable service definition** (`docs/repository-interface.md`
+plus a JSON schema) of every service, its arguments, result types, and error
+contract. A **miss is a value, not a throw**: the interface models "no answer"
+explicitly. tmct ships **reference providers** (a fixture graph and the
+empty/bootstrap graph) that implement every service, and a **runnable conformance
+suite**. tmct's own providers pass it in `npm test`. Any external graph producer
+(seonix first) runs the same suite against its native implementation to claim
+conformance. Conformance is the suite, not prose. This inverts the original
+relationship: tmct was lifted out of seonix, and seonix now reorients as a *user*
+that imports the tmct library and exposes its graph to tmct as a service. The LLM
+agent stays outside tmct, as the no-LLM ethos requires.
 
 ## Security and supply chain
 
 tmct is $0 to run and meant to be trusted offline, so the supply chain is
-hardened accordingly: CI runs **SAST and secret detection**, a **nightly
-`npm audit` + OSV-Scanner** job watches dependencies, releases are published with
-**npm provenance** (`--provenance`), and there is a coordinated-disclosure
-`SECURITY.md` policy. The content-address hash is single-sourced in
-`src/hash.mjs` so the cross-version-stable fact-id contract has exactly one
-definition.
+hardened:
+
+- CI runs **SAST and secret detection**.
+- A nightly **`npm audit` + OSV-Scanner** job watches dependencies.
+- Releases are published with **npm provenance** (`--provenance`).
+- A coordinated-disclosure `SECURITY.md` policy covers reports.
+
+The content-address hash is single-sourced in `src/hash.mjs`, so the
+cross-version-stable fact-id contract has exactly one definition.
 
 ## Provenance
 

package/ROADMAP.md

@@ -550,6 +550,51 @@ Repository Interface library-surface work (Phase 8) settling the extraction boun
 `PLAN_OSS_ACE_PARSER.md`. Sibling publish-candidates (the bounded-Damerau fuzzy matcher, the
 PageRank+IDF block ranker) may follow the same path if there is demand.
 
+## Phase 10 — Conversational competence & onboarding (0.6.0 → 0.7.0)
+
+*(Operator-directed, from live new-user testing.)* Once a graph is loaded the engine is strong;
+the weak surfaces were the FIRST RUN and the VAGUE question. This phase makes the miss graceful,
+the empty state honest, and the vague touch a guided answer — realising item 1's "tolerant, guides
+you toward precision" promise on the conversational surface.
+
+- **Onboarding UX (shipped 0.6.0):** the grammar wall moved behind `/help` (a short, tailored miss
+  instead); intent lanes for memory/teach ("remember that X"), meta/self ("what is this codebase",
+  "what do you know"), routed only when a graph query would miss; empty/degenerate-graph
+  orientation that distinguishes CODE STRUCTURE (needs a `.tmct/graph.json` via a producer or
+  `--repo`; tmct reads graphs, it does not index code) from VOCABULARY (`tmct init`/bootstrap seeds
+  concepts); `TMCT_GRAPH_FILE` honoured by chat; slash-optional commands (`stats`≡`/stats`);
+  `/memory` explore hooks; up/down-arrow prompt history in the TUI.
+- **Knowledge (shipped 0.6.0 → 0.7.0):** the curated `corpus/seon` ontology — a software-sense
+  definition for EVERY lexicon term, language-neutral (Java/C#/Python `class` → one concept); the
+  ConceptNet slice quality-filtered (word-sense noise cut) and regrown to ~40k facts; tier-2
+  specialised corpuses (aws/python/java) with `tmct init --corpus`; batched `appendFacts` (one
+  write, 419s→2.5s) enabling **seed-all** so a fresh repo knows the whole curated vocabulary.
+- **The concept force (shipped 0.7.0):** a vague touch on a concept X, where tmct knows X and has instances,
+  answers in three bands — **the definition** (from `corpus/seon`), **the examples** (real code-graph
+  + memory instances of X), and **a soft guided follow-up** ("Want to go deeper?" + 2–3 questions
+  built from the real instances × the query shapes valid for that kind, each pre-validated to
+  resolve). Applies to NOUN concepts (`what is a class` → define + Base/Widget/Button + drill-ins)
+  AND **RELATION concepts** (`what about imports` / `what calls are there` → the verb definition +
+  example edges + guided queries), fixing the vague-query dead-ends. Fact rendering is
+  de-anthropomorphised (no first-person "i learned:" over-claim — corpus facts read as data +
+  provenance; `you told me` stays for operator-asserted facts); listings cap at 32 with a "say
+  'more'" pagination that holds the remainder in session state.
+- **Dead-end routing + read-only demos (shipped 0.7.0):** natural drill-down phrasings are routed
+  onto the canonical shapes they mean — `what functions are in X` → members-of-class, `what defined
+  X` → where-is-X-defined, a no-context `what about X` → the concept/relation force (the discourse
+  continuation still wins when there IS a prior answer). `tmct chat --ephemeral` (and the
+  `npm run example:*` demos) reads a graph but writes nothing back, so a checked-in example is never
+  dirtied by a demo run.
+- **The dialogue-flow loop (`SKILL_CHAT_PLAYTEST.md`, 0.7.0):** a fast, qualitative tuning loop that
+  complements the LLM-judge benchmark — Claude plays a curious user, hunts *dead-ends* (walls,
+  "unknown qualifier", phrasing-misses, invited follow-ups the engine can't take), fixes them by
+  ROUTING to existing capabilities, replays the same conversations until they flow, freezes them as
+  regression transcripts, then ratchets the complexity tier. The drill-down transcript above is its
+  first frozen fixture (`test/chatflow-drilldown.test.mjs`).
+- **Measured** by the version-matched benchmark (`CHATBENCH_<version>` per `SKILL_TUNING_CYCLE.md`),
+  with new graded cells for the miss / empty-graph / concept-touch surfaces so these become
+  regression-protected levers, not one-off polish.
+
 ## Phase LATER — recognized, deferred, not now
 
 Features we have deliberately shaped seams for but will not build until the phases above have

package/bin/tmct.mjs

@@ -34,12 +34,17 @@ software repository. No model calls; no codebase index of its own.
 Usage:
   tmct                         interactive chat (the headline surface)
   tmct chat [--repo <abs>]     chat over a specific repo's graph
+       [--ephemeral]           read the graph but write nothing back (demo/read-only)
        [--plain]               force the plain readline shell (the default when
                                stdin/stdout is not a terminal)
   tmct memory [--repo <abs>]   what tmct remembers: facts, utterances, sessions,
        [--verbose]             folded blocks (the /memory chat command, from the shell)
   tmct init [--force]          initialize the current directory for tmct: .tmct/,
                                tmct.toml, tier-1 corpus seed, provenance record
+       [--corpus <id>]         also seed a tier-2 corpus (aws|python|java) — opt-in,
+                               offline, $0; init is tier-1-only unless asked
+       [--detect]              suggest a tier-2 corpus from the repo's manifests
+                               (pyproject.toml → python, pom.xml → java); never seeds unasked
   tmct syllogise [--repo <abs>] speculative inference (offline maintenance job): forward-
        [--depth <n>] [--budget <n>]  chain the memory's rdfs:subClassOf closure, materialising
                                bounded, low-trust, retractable entailed facts (never on the chat path)
@@ -263,16 +268,21 @@ async function main() {
     const rest = process.argv.slice(3);
     const i = rest.indexOf("--repo");
     const repoPath = i !== -1 ? rest[i + 1] : undefined;
+    // `--ephemeral` (used by the shipped `npm run example:*` scripts): read the
+    // target graph but write nothing back — no session folded into the committed
+    // code graph, no .tmct/memory dropped under it. A demo you can run repeatedly
+    // on a checked-in example without ever dirtying it.
+    const ephemeral = rest.includes("--ephemeral");
     // The shell gate: a real terminal gets the full-screen Ink TUI; `--plain` or a
     // non-TTY stream (pipes, scripts, the test suite) gets the readline shell. Both
     // drive the same createSession sink — only the drawing differs.
     const plain = rest.includes("--plain") || !process.stdin.isTTY || !process.stdout.isTTY;
     if (plain) {
       const { runChat } = await import("../src/chat.mjs");
-      await runChat({ repoPath });
+      await runChat({ repoPath, ephemeral });
     } else {
       const { runTui } = await import("../src/tui/app.mjs");
-      await runTui({ repoPath });
+      await runTui({ repoPath, ephemeral });
     }
     return;
   }
@@ -295,10 +305,62 @@ async function main() {
     // `tmct init` — the Repository-Interface onboarding surface: scaffold .tmct/,
     // write tmct.toml, seed the tier-1 corpus (offline, opt-out via TMCT_NO_SEED),
     // and record provenance. Idempotent; --force rewrites config + re-records.
+    //
+    // TIERING POLICY: init is OFFLINE, $0 and TIER-1-ONLY by default. A tier-2
+    // domain/language corpus (corpus/tier2/: aws, python, java) is added ONLY when
+    // explicitly asked via `--corpus <id>`. The `--detect` auto-detect is a
+    // documented STUB: it inspects the repo's manifests (pyproject.toml → python,
+    // pom.xml → java) and SUGGESTS the matching corpus, but never seeds it unasked.
     const rest = process.argv.slice(3);
     const { initRepo } = await import("../src/init.mjs");
     const res = await initRepo(process.cwd(), { force: rest.includes("--force") });
     process.stdout.write(res.message + "\n");
+
+    const ci = rest.indexOf("--corpus");
+    const corpusId = ci !== -1 ? rest[ci + 1] : undefined;
+    if (corpusId) {
+      // Seed a tier-2 corpus by id from corpus/tier2/ (same slice shape as tier-1;
+      // provenance-tagged corpus:tier2:<id>). Idempotent (content-hashed fact ids).
+      const { readFile } = await import("node:fs/promises");
+      const { join, dirname } = await import("node:path");
+      const { seedMemory, TIER2_MANIFEST_FILE } = await import("../src/corpus/conceptnet.mjs");
+      let manifest;
+      try { manifest = JSON.parse(await readFile(TIER2_MANIFEST_FILE, "utf8")); }
+      catch (e) { process.stderr.write(`tmct init: cannot read the tier-2 manifest — ${e?.message || e}\n`); process.exit(1); }
+      const entry = (manifest.corpuses || []).find((c) => c.id === corpusId);
+      if (!entry) {
+        const ids = (manifest.corpuses || []).map((c) => c.id).join(", ");
+        process.stderr.write(`tmct init: unknown --corpus "${corpusId}". Available tier-2 corpuses: ${ids}.\n`);
+        process.exit(2);
+      }
+      const slicePath = join(dirname(TIER2_MANIFEST_FILE), entry.file);
+      const seeded = await seedMemory(process.cwd(), { slicePath, provenancePrefix: `corpus:tier2:${entry.id}` });
+      process.stdout.write(
+        `seeded tier-2 corpus "${entry.id}" (${entry.kind}) — ${seeded.appended} fact(s) added`
+        + `${seeded.skipped ? `, ${seeded.skipped} already present` : ""}. Source: corpus/tier2/${entry.file} (${entry.license}).\n`,
+      );
+      return;
+    }
+
+    if (rest.includes("--detect")) {
+      // AUTO-DETECT STUB (documented, non-seeding): map a build manifest to the
+      // tier-2 corpus that fits, and tell the operator how to add it. Kept a stub on
+      // purpose — the $0/offline default never expands the corpus without an ask.
+      const { access } = await import("node:fs/promises");
+      const { join } = await import("node:path");
+      const has = (f) => access(join(process.cwd(), f)).then(() => true, () => false);
+      const DETECT = [["pyproject.toml", "python"], ["pom.xml", "java"]];
+      const found = [];
+      for (const [file, id] of DETECT) if (await has(file)) found.push([file, id]);
+      if (!found.length) {
+        process.stdout.write("no tier-2 corpus auto-detected (looked for pyproject.toml → python, pom.xml → java).\n");
+      } else {
+        for (const [file, id] of found) {
+          process.stdout.write(`detected ${file} — run \`tmct init --corpus ${id}\` to add the ${id} tier-2 corpus (offline, $0).\n`);
+        }
+      }
+      return;
+    }
     return;
   }
 

package/corpus/README.md

@@ -5,14 +5,40 @@ The corpuses tmct ships so that an **empty** tmct still has a vocabulary
 in `src/corpus/`. Related committed data lives in `data/` (response templates
 + the SE phrase book — items 4+7).
 
+## The tiering policy (tier-1 / tier-2 / tier-3)
+
+tmct's knowledge arrives in three tiers, distinguished by **when** it lands and
+**whether it ships in the npm package**:
+
+| Tier | What | Ships in the package? | Lands when | Provenance |
+|---|---|---|---|---|
+| **1 — base** | the general English/tech ConceptNet slice + the response templates + the SE phrasebook — the vocabulary every tmct has out of the box | **yes**, committed here | `tmct init` seeds `.tmct/` from committed data (offline, $0) | `corpus:conceptnet /r/…` |
+| **2 — specialised** | LANGUAGE- or DOMAIN-specific fact sets (`aws`, `python`, `java`, …) so tmct can "expand into a concept for an applicable codebase" | **no** — selected per repo | `tmct init` fetches/generates the chosen tier-2 corpuses into `.tmct/` (Wave-2 wiring) | `corpus:tier2:<id> /r/…` |
+| **3 — learned** | facts tmct writes from the actual conversation / the actual codebase it is pointed at | never committed | at runtime, into `.tmct/memory/` | `chat:…`, `codegraph:…` |
+
+**Offline / $0 is the default at every tier.** Tier-1 is committed. Tier-2's
+sample corpuses are *curated* (generated locally by `tier2/generate.mjs`, no
+network); a tier-2 corpus too large to curate by hand may declare a `fetch`
+source (a URL + a `sha256`), but the network is **opt-in only** — nothing
+reaches out unless the operator asks for it. Tier-3 is whatever the user says.
+
+**Checksums / integrity.** Every tier-2 corpus carries a `sha256` and a `bytes`
+count in `tier2/manifest.json`; a `fetch`-sourced corpus is checksum-verified on
+download (`generate.mjs` `fetchCorpus()`), so a corrupt or tampered fetch fails
+loudly instead of seeding garbage.
+
 ## What's here
 
 | Path | What | Size | Licence |
 |---|---|---|---|
-| `conceptnet/slice.jsonl` | filtered English/tech-domain ConceptNet 5.7 slice (one assertion per line) | ~1.4 MB | **CC-BY-SA 4.0** (see `conceptnet/LICENSE-NOTICE`) |
+| `conceptnet/slice.jsonl` | tier-1: filtered English/tech-domain ConceptNet 5.7 slice, quality-filtered (one assertion per line) | ~1.35 MB | **CC-BY-SA 4.0** (see `conceptnet/LICENSE-NOTICE`) |
 | `conceptnet/fetch-slice.mjs` | regeneration tool — the ConceptNet **API** route (polite, ~1 req/s) | — | MPL-2.0 |
 | `conceptnet/filter-dump.mjs` | regeneration tool — the ConceptNet **dump** route (produced the committed slice; the API was down) | — | MPL-2.0 |
+| `conceptnet/quality-filter.mjs` | second-pass noise filter (drops sentence-fragment/numeric/opinion rows); produced the committed clean slice | — | MPL-2.0 |
 | `conceptnet/README.md` | provenance, retrieval date, seed terms, filter rules, row counts | — | — |
+| `tier2/manifest.json` | tier-2: index of specialised corpuses (id, kind, description, source, sha256, size) | — | MPL-2.0 |
+| `tier2/generate.mjs` | tier-2: the curated-corpus generator + manifest writer (+ opt-in network-fetch path) | — | MPL-2.0 |
+| `tier2/{aws,python,java}.jsonl` | tier-2 SAMPLE corpuses — same fact shape as the tier-1 slice, loadable via the same path | ~4 KB each | MPL-2.0 |
 
 And alongside (same phase, different directory because it is tmct-original
 data, not a derived corpus):
@@ -44,9 +70,55 @@ await seedMemory(repoDir, { limit: 500 }); // capped (fast bootstrap)
 - `ace = "none"` relations (RelatedTo, Synonym, FormOf, …) are deliberately
   NOT seeded — they are kept in the slice for future lexicon/fuzzy-match use.
 
+## Tier-2 specialised corpuses (`tier2/`)
+
+A tier-2 corpus is a small, curated, LANGUAGE- or DOMAIN-specific fact set in
+the **exact tier-1 fact shape** — one JSON object per line,
+`{"start":"/c/en/…","rel":"/r/…","end":"/c/en/…","weight":N,"surfaceText":"…"}`,
+with `rel` drawn only from the mapped relations in
+`src/corpus/conceptnet-map.toml`. Because the shape is identical, a tier-2 file
+loads and seeds through the very same `loadSlice()`/`toFacts()` path as the
+tier-1 slice — `tier2/generate.mjs --verify` proves it (each sample loads and
+all its facts seed cleanly, no `ace=none` dead rows).
+
+The idea: a Python repo pulls the `python` corpus so tmct knows "a dict is a
+kind of hash table"; an AWS project pulls `aws` so it knows "S3 is object
+storage, a bucket is part of S3". Language terms unify onto the **shared CS
+concept vocabulary** (`list → array`, `hashmap → hash table`) so specialised
+knowledge connects to the tier-1 graph instead of floating apart.
+
+**To add a corpus:** add an entry to `CORPUSES` in `tier2/generate.mjs` (a list
+of `[subject, relation, concept]` triples) and run `node tier2/generate.mjs
+--verify`. It rewrites `<id>.jsonl` and `manifest.json` (facts count, byte size,
+sha256) in one deterministic pass. Curated data is authored in that file so it
+stays reviewable; a corpus too big to hand-curate is a `fetch` manifest entry
+(URL + sha256, opt-in network — `fetchCorpus()` is the reference downloader).
+
+**How the coordinator should wire tier-2 into `tmct init` (Wave-2 — NOT done
+here):**
+
+1. Add a tier-2-aware seeder next to `seedMemory()` (e.g. `seedTier2(dir, id)`)
+   that reads `corpus/tier2/manifest.json`, resolves the requested corpus's
+   `file` (curated → already on disk; `fetch` → download + `sha256`-verify
+   first, only when network is explicitly enabled), then runs the SAME
+   `loadSlice → toFacts → appendFact` pipeline — **but stamps provenance
+   `corpus:tier2:<id> <rel>`** instead of the hard-coded `corpus:conceptnet …`
+   string in `toFacts()`. (Simplest: give `toFacts` an optional
+   `provenancePrefix` argument, default `"corpus:conceptnet"`.)
+2. Give `tmct init` a `--corpus <id>[,<id>…]` flag (and/or codebase
+   auto-detection: a `requirements.txt`/`pyproject.toml` → `python`, a
+   `pom.xml`/`build.gradle` → `java`, an AWS SDK dep / `serverless.yml` →
+   `aws`). Default stays tier-1-only, offline, $0.
+3. Idempotency is free — `seedMemory`'s content-hashed fact ids and pre-read
+   skip already handle re-seeds; tier-2 rides the same path.
+
+None of that touches `package.json` or the tier-1 budget; tier-2 files are not
+shipped, so they do not count against the ≤ 1.5 MB slice budget.
+
 ## How to regenerate / extend
 
-See `conceptnet/README.md` — one command per route (API vs dump), plus the
-seed-term list to extend. The test suite (`test/corpus-conceptnet.test.mjs`,
-`test/corpus-templates.test.mjs`) guards the contracts: slice/mapping drift,
-en→en shape, the ≤ 1.5 MB budget, template ids/slots, and end-to-end seeding.
+See `conceptnet/README.md` — one command per route (API vs dump), the
+quality-filter pass, plus the seed-term list to extend. The test suite
+(`test/corpus-conceptnet.test.mjs`, `test/corpus-templates.test.mjs`) guards the
+contracts: slice/mapping drift, en→en shape, the ≤ 1.5 MB budget, template
+ids/slots, and end-to-end seeding.

package/corpus/conceptnet/README.md

@@ -15,7 +15,9 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
 - **Source:** ConceptNet 5.7.0 assertions dump,
   `https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-assertions-5.7.0.csv.gz`
   (published 2019-07-03).
-- **Retrieved + filtered:** 2026-07-04.
+- **Retrieved + filtered:** 2026-07-04; **regrown to the ~40k tier-1 target on
+  2026-07-05** (same dump, same licence) by widening the tech-seed domain and
+  raising the size budget — see "Growing the slice" below.
 - **Why the dump, not the API:** the public API (`api.conceptnet.io`) was
   hard-down (HTTP 502 from its nginx front-end on every request across ~15
   attempts over 10+ minutes on 2026-07-04), so the slice was stream-filtered
@@ -33,12 +35,16 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
    policy: `/r/EtymologicallyRelatedTo`, `/r/EtymologicallyDerivedFrom`,
    `/r/ExternalURL` (etymology noise and link-outs — no consumer in tmct).
 3. **Tech-domain seed terms**: at least one endpoint's bare term is in the
-   90-term software/tech seed list exported as `SEED_TERMS` from
+   tech seed list — the ~90-term base `SEED_TERMS` exported from
    `fetch-slice.mjs` (software, computer, program, code, module, function,
    database, server, network, bug, test, file, memory, algorithm, keyboard,
-   programmer, repository, commit, …).
+   programmer, repository, commit, …) **plus** the ~230-term `EXTRA_SEEDS`
+   growth list in `filter-dump.mjs` (programming languages, frameworks, data
+   structures, cloud/infra, protocols, tools, ML — python, java, docker,
+   kubernetes, git, neural_network, tcp, kernel, hashtable, …), added
+   2026-07-05 to reach the ~40k tier-1 target while staying in the tech domain.
 4. **Dedupe** by `(start, rel, end)`, keeping the higher weight.
-5. **Size budget** (committed slice ≤ 1.5 MB; target ~1.4 MB), **two-tier**:
+5. **Size budget** (committed slice ≤ 5 MB; target ~4.3 MB), **two-tier**:
    assertions whose relation maps to an ACE-OWL pattern (`ace != "none"` in
    `conceptnet-map.toml`) are kept first, weight-descending; `ace = "none"`
    relations (`RelatedTo`, `Synonym`, …) fill the remaining budget — they
@@ -46,34 +52,86 @@ see `LICENSE-NOTICE` in this directory for the full attribution.
    facts.
 6. Deterministic output order: `(rel, start, end)`.
 
-## Row counts (committed slice, 2026-07-04)
+## Quality-filter pass (regrown slice, 2026-07-05)
+
+`filter-dump.mjs` keeps the DATA honest (tech-seed match, canonical relations,
+budget) but not the SEMANTICS: ConceptNet's crowd-sourced "Verbosity"/Open-Mind
+rows leave sentence-fragment "concepts" and opinion axioms that read as nonsense
+once seeded ("a computer is a kind of dumb", "a class is a kind of elegance",
+"mouse AtLocation taloned_grip_of_owl", "2 is a kind of software"). A second
+pass, `quality-filter.mjs`, removes those by term/relation shape (never per
+row):
+
+- **numeric endpoint** — bare term all digits (`2`, `1000`, `80386`)
+- **single-char endpoint** — bare term length ≤ 1 (`a`, `r`, `m`)
+- **sentence fragment** — ≥ 4 underscore-words on either endpoint
+  (`taloned_grip_of_owl`, `worlds_largest_interconnected_network_of_networks`)
+- **definitional phrase** — `/r/DefinedAs` with a ≥ 3-word object (real
+  `DefinedAs` is a synonym: `cpu → processor`)
+- **opinion object** — `/r/IsA` / `/r/DefinedAs` whose object is in a small
+  evidence-based set (`elegance, evil, gloom, unreality, universalism, dumb,
+  free, junk`) — never a class
+
+**Result (2026-07-05 regrow): 45,633 → 44,947 rows (686 cut), 4,308,850 →
+4,220,629 bytes.** Cuts by reason: sentence-fragment 613, single-char 48,
+numeric 11, opinion-object 8, definitional-phrase 6. Of the seedable (mapped,
+`ace≠none`) facts, the clean slice carries **6,255** — up from 3,884 in the
+1.35 MB slice. No relation disappeared entirely (the drift guard stays
+satisfied). Re-run any time with:
 
-**14,258 assertions, 1,399,979 bytes** (34,074,917 dump lines scanned;
-28,802 unique en→en seed assertions matched = 4,170 mappable + 24,632
-`ace="none"`; ALL 4,170 mappable kept, 10,088 none-rows fill the budget).
-29 of the 31 non-filtered canonical relations are present:
+```bash
+node corpus/conceptnet/quality-filter.mjs --in-place corpus/conceptnet/slice.jsonl
+```
+
+## Growing the slice toward the ~40k tier-1 target (done 2026-07-05)
+
+The operator's ~40k-fact tier-1 target is **shipped**: the slice was regrown
+from the same ConceptNet dump by (a) widening the tech domain with the
+`EXTRA_SEEDS` list in `filter-dump.mjs` (~230 tech terms — languages,
+frameworks, data structures, cloud/infra, protocols, tools, ML) and (b) raising
+`MAX_BYTES` in `filter-dump.mjs` from 1.4 MB to 4.5 MB with the matching test
+budget assertion (`test/corpus-conceptnet.test.mjs`) raised from 1.5 MB to
+5 MB. The widened seed set matched **45,633 unique en→en assertions** (all under
+budget, so no tier-trimming was needed this pass), and the quality filter
+trimmed them to the committed **44,947 clean facts**. Regenerate with:
+
+```bash
+# 1. stream-filter the dump (widened seeds + raised budget already live in filter-dump.mjs)
+curl -s https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-assertions-5.7.0.csv.gz \
+  | gunzip -c | node corpus/conceptnet/filter-dump.mjs > corpus/conceptnet/slice.jsonl
+# 2. re-apply the semantic quality filter
+node corpus/conceptnet/quality-filter.mjs --in-place corpus/conceptnet/slice.jsonl
+```
+
+## Row counts (regrown slice, 2026-07-05)
+
+**44,947 clean assertions, 4,220,629 bytes** — the quality-filtered committed
+slice (raw filter-dump output was 45,633 rows / 4,308,850 bytes; 34,074,917 dump
+lines scanned; 45,633 unique en→en seed assertions matched = 6,670 mappable +
+38,963 `ace="none"`; ALL kept under the 4.5 MB budget, then 686 noise rows cut).
+30 of the 31 non-filtered canonical relations are present; **6,255 seedable
+(`ace≠none`) facts**:
 
 | Relation | Rows | | Relation | Rows |
 |---|---|---|---|---|
-| `/r/RelatedTo` | 4911 | | `/r/HasA` | 39 |
-| `/r/HasContext` | 2634 | | `/r/HasProperty` | 27 |
-| `/r/IsA` | 2594 | | `/r/Antonym` | 25 |
-| `/r/DerivedFrom` | 1906 | | `/r/MotivatedByGoal` | 25 |
-| `/r/AtLocation` | 459 | | `/r/SimilarTo` | 20 |
-| `/r/Synonym` | 368 | | `/r/DistinctFrom` | 19 |
-| `/r/UsedFor` | 312 | | `/r/DefinedAs` | 16 |
-| `/r/FormOf` | 224 | | `/r/MadeOf` | 14 |
-| `/r/CapableOf` | 164 | | `/r/CausesDesire` | 12 |
-| `/r/MannerOf` | 141 | | `/r/HasLastSubevent` | 12 |
-| `/r/PartOf` | 115 | | `/r/Causes` | 11 |
-| `/r/HasPrerequisite` | 103 | | `/r/CreatedBy` | 10 |
-| `/r/HasSubevent` | 42 | | `/r/Desires` | 7 |
-| `/r/ReceivesAction` | 40 | | `/r/HasFirstSubevent` | 7 |
-| | | | `/r/LocatedNear` | 1 |
-
-Absent from the slice (nothing matched the seed terms): `/r/ObstructedBy`,
-`/r/SymbolOf` — both still have mapping rows, so a regenerated slice that
-surfaces them stays covered.
+| `/r/RelatedTo` | 29016 | | `/r/HasSubevent` | 42 |
+| `/r/HasContext` | 4376 | | `/r/DistinctFrom` | 32 |
+| `/r/IsA` | 4173 | | `/r/HasProperty` | 30 |
+| `/r/DerivedFrom` | 3388 | | `/r/ReceivesAction` | 27 |
+| `/r/Synonym` | 1126 | | `/r/MotivatedByGoal` | 26 |
+| `/r/AtLocation` | 642 | | `/r/MadeOf` | 20 |
+| `/r/FormOf` | 528 | | `/r/Causes` | 18 |
+| `/r/UsedFor` | 378 | | `/r/CreatedBy` | 15 |
+| `/r/CapableOf` | 231 | | `/r/CausesDesire` | 13 |
+| `/r/MannerOf` | 230 | | `/r/Desires` | 9 |
+| `/r/PartOf` | 196 | | `/r/HasLastSubevent` | 8 |
+| `/r/Antonym` | 155 | | `/r/HasFirstSubevent` | 7 |
+| `/r/HasPrerequisite` | 108 | | `/r/LocatedNear` | 3 |
+| `/r/SimilarTo` | 102 | | `/r/DefinedAs` | 2 |
+| `/r/HasA` | 45 | | `/r/SymbolOf` | 1 |
+
+Absent from the slice (nothing matched the seed terms): `/r/ObstructedBy` — it
+still has a mapping row, so a regenerated slice that surfaces it stays covered.
 
 ## How to regenerate / extend
 
@@ -87,11 +145,12 @@ curl -s https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-asse
   | node corpus/conceptnet/filter-dump.mjs > corpus/conceptnet/slice.jsonl
 ```
 
-To extend the domain, add seed terms to `SEED_TERMS` in `fetch-slice.mjs` and
-re-run. `npm test` guards the contract: every relation present in the slice
-must have a row in `src/corpus/conceptnet-map.toml` (drift guard), en→en shape
-and the ≤ 1.5 MB budget are asserted, and the seeding path is exercised
-end-to-end.
+To extend the domain, add seed terms to `EXTRA_SEEDS` in `filter-dump.mjs` (the
+dump route that produces the committed slice) or `SEED_TERMS` in
+`fetch-slice.mjs` (the API route) and re-run. `npm test` guards the contract:
+every relation present in the slice must have a row in
+`src/corpus/conceptnet-map.toml` (drift guard), en→en shape and the ≤ 5 MB
+budget are asserted, and the seeding path is exercised end-to-end.
 
 ## Consumers