@polycode-projects/the-mechanical-code-talker 2.3.0 → 2.5.0

package/README.md

@@ -2,9 +2,8 @@
 
 `@polycode-projects/the-mechanical-code-talker`
 
-A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage:
-pattern-driven, best-efforts, and focused on software as its subject matter.
-It makes no model calls.
+A pure-JS, **no-LLM**, offline, **$0** chatbot in the ELIZA/PARRY lineage.
+It is pattern-driven and focused on software, and it makes no model calls.
 
 tmct turns natural language directly into a graph database. On first run it
 seeds an everyday **human-world persona**: people, places, objects, nature,
@@ -100,8 +99,12 @@ parts are real provenance receipts. Every fact tmct stores records where it
 came from and when (more on that below).
 
 The test suite replays every runnable example in this README against the
-live product, this transcript included. If the chat behavior ever drifts
-from the output above, the suite fails and says so.
+live product, this transcript included. Every line shown must be a line the
+product prints, in the order shown, so if the chat behavior drifts from the
+output above, the suite fails and says so. Two blocks below are marked `skip=`
+and never run: one would touch the network, the other needs an LLM judge.
+`docs/public-examples.md` maps every example on every public surface to the
+test that holds it.
 
 Point it at a codebase's graph and the same engine answers structural questions.
 `examples/mini-webapp` ships in this repo, so this runs as written:
@@ -160,7 +163,7 @@ Text that doesn't fit the grammar still gets the tolerant strategies. Nothing
 is rejected for being loose, fuzzy, or misspelled.
 
 On top of that base, tmct reads the shapes people actually use, and each one
-resolves to a real graph traversal or declines honestly:
+resolves to a real graph traversal or declines cleanly:
 
 - everyday question forms: "what is Commit", "what's model.mjs for",
   "recent commits" as real dated history;
@@ -353,8 +356,13 @@ Goal (inferred): Plan a move sequence from the current state to the goal (7 move
 ```
 
 "next" executes one move at a time, writing each board state into memory as
-facts; the final step re-reads those facts and confirms the goal from them,
-never assuming success. The search is genuine and domain-general: the test
+facts stamped with the step that produced them ("disk-1@step1 rests on peg-c",
+sourced to the plan). The final step re-reads the store and confirms the goal
+from those written facts, never assuming success. The stamp is what makes each
+step a separate record; a question about the piece itself ("where does disk-1
+rest?", "is disk-1 clear?") reads the current board — the latest step's facts,
+not every step at once. The search is
+domain-general: the test
 suite teaches Towers of Hanoi purely as sentences for 1 to 8 disks and
 asserts the plan is exactly 2^n − 1 moves every time, and a second game
 (`crates.txt`, stacking crates with different rules and a two-goal
@@ -417,14 +425,15 @@ nudges you to ground one side first. Quantified teaching stores the
 quantifier ("some functions are risky" … "how many functions are risky" →
 "A few."), and "how many facts are there" counts the store back.
 
-Teaching doesn't have to be typed, either. `npm run extract:facts` (from a
-clone) runs a plain text file through the same recognizer the chat's teach
-lane uses. Sentences the recognizer grounds become fact rows; everything
-else is skipped and counted, never paraphrased:
+Teaching doesn't have to be typed, either. `tmct extract` runs a plain text
+file through the same recognizer the chat's teach lane uses. Sentences the
+recognizer grounds become fact rows; everything else is skipped and counted,
+never paraphrased. Add `--repo <abs>` to write them into that repo's own
+memory; without it nothing on disk is mutated and the facts print as JSONL:
 
 ```bash cwd=repo
 printf 'We deployed redis last week. a cache is a kind of store. Why was it slow?\n' > /tmp/notes.txt
-node scripts/extract-facts-from-text.mjs /tmp/notes.txt
+node bin/tmct.mjs extract /tmp/notes.txt
 ```
 
 ```output
@@ -537,13 +546,13 @@ provenance record. Most of its flags choose what gets seeded and where config is
        [--corpus <id|path>]    also seed a corpus — a tier-2 manifest id (aws|python|java|
                                general) or a jsonl file path — opt-in, offline, $0
        [--ontology <name|path>]  activate+seed an ontology bundle (a recognized name or a path)
-       [--lexicon <name|path>]   activate a lexicon bundle (recognized name or a path;
+       [--lexicon <name|path>]  activate a lexicon bundle (recognized name or a path;
                                merged read-time, never seeded — see mergedLexiconExtra)
        [--graph <path>]        set graph_file/graph_files in tmct.toml (repeatable)
        [--config <path>]       write to an alternate tmct.toml location
        [--detect]              suggest a tier-2 corpus from the repo's manifests
                                (pyproject.toml → python, pom.xml → java); never seeds unasked
-       [--with-persona <name>] write an explicit [extensions]/[bias] preset into tmct.toml
+       [--with-persona <name>]  write an explicit [extensions]/[bias] preset into tmct.toml
                                ("code" — today's implicit default, made explicit)
        [--persona-size <medium|large>]  grow the default "human" persona's fact count
                                beyond Small (the default): "medium" activates
@@ -564,7 +573,7 @@ already set up. Its `--graph` flag works differently from the others: it appends
   tmct import [--repo <abs>]   activate+seed into an ALREADY-initialized repo (any
        [--corpus <id|path>]    combination of these flags in one call). --graph is a
        [--ontology <name|path>]  DIFFERENT operation from the others: it APPENDS to
-       [--lexicon <name|path>]   tmct.toml's graph_files array (multi-graph growth),
+       [--lexicon <name|path>]  tmct.toml's graph_files array (multi-graph growth),
        [--graph <path>]        never an extensions-bundle activation.
        [--file <definition.txt>]  teach a plain-text definition file sentence by
                                sentence (# lines are comments); any declined
@@ -573,12 +582,24 @@ already set up. Its `--graph` flag works differently from the others: it appends
        [--config <path>]
 ```
 
+`tmct extract` is the document route into memory described under "Teach it"
+above — the same teach recognizer, reading a file instead of your typing:
+
+```output:help:extract
+  tmct extract <text-file>     read a plain text file's sentences through the chat's own
+       [--file <text-file>]    teach recognizer and keep the facts it grounds; every
+                               other sentence is skipped and counted, never paraphrased
+       [--repo <abs>]          write the facts into that repo's own tmct memory; without
+                               it nothing on disk is mutated and the facts print as JSONL
+       [--out <file.jsonl>]    write that JSONL to a file instead of stdout
+```
+
 `tmct extend --validate` checks a third-party extension pack's declared resources
 before you switch any repo's `tmct.toml` over to it:
 
 ```output:help:extend
   tmct extend --validate <dir>  validate a third-party extension pack's declared
-       [--config <path>]      resources (corpus/lexicon/templates) before activating
+       [--config <path>]       resources (corpus/lexicon/templates) before activating
                                it in any repo's tmct.toml; exits non-zero on failure
 ```
 
@@ -586,9 +607,9 @@ before you switch any repo's `tmct.toml` over to it:
 inference" above:
 
 ```output:help:syllogise
-  tmct syllogise [--repo <abs>] speculative inference (offline maintenance job): forward-
+  tmct syllogise [--repo <abs>]  speculative inference (offline maintenance job): forward-
        [--depth <n>] [--budget <n>]  chain the memory's rdfs:subClassOf closure, materialising
-       [--config <path>]      bounded, low-trust, retractable entailed facts (never on the chat path)
+       [--config <path>]       bounded, low-trust, retractable entailed facts (never on the chat path)
 ```
 
 `tmct viz` renders the memory graph as the ledger explorer — a single,
@@ -596,11 +617,11 @@ self-contained HTML file you can open in a browser:
 
 ```output:help:viz
   tmct viz [--repo <abs>]      write one self-contained HTML page: the memory graph as a
-       [--focus <term>]       readable ledger of fact-sentences around one focus term,
-       [--term <word>]        with segments, a two-hop minimap, and an in-page chat dock
-       [--limit <n>]          that answers from the embedded graph. Focuses on the newest
-       [--output <path>]      taught fact's subject by default (--focus <term> or
-       [--config <path>]      --term <word> override it); --output defaults to
+       [--focus <term>]        readable ledger of fact-sentences around one focus term,
+       [--term <word>]         with segments, a two-hop minimap, and an in-page chat dock
+       [--limit <n>]           that answers from the embedded graph. Focuses on the newest
+       [--output <path>]       taught fact's subject by default (--focus <term> or
+       [--config <path>]       --term <word> override it); --output defaults to
                                ledger.html in the cwd; --limit caps the embedded fact
                                rows; --term resolves via the same normalization chat uses.
 ```
@@ -927,7 +948,7 @@ node bin/tmct.mjs cli tmct_untested '{"repo_path":"examples/mini-webapp"}'
 ## The repository interface
 
 tmct is not an indexer, so it consumes a graph through a typed contract any
-producer can implement. That contract is first-class: a **versioned (1.0.0),
+producer can implement. That contract is first-class: a **versioned (1.1.0),
 OWL-grounded, machine-readable service definition** (`docs/repository-interface.md`
 plus a JSON schema) of every service, its arguments, result types, and error
 contract. The interface returns a miss as a normal value. It never throws to
@@ -942,6 +963,19 @@ service. The LLM agent stays outside tmct, as the no-LLM ethos requires.
 
 ## Measuring it
 
+What the 2.0.3 cycle measured, on 2026-07-16. Each figure links to its method
+and carries, in the same row, the caveat that changes what it means. The full
+tables, judge scores, and transcripts are in the linked write-ups.
+
+| What it does | Result (2.0.3) | Read the number with this | Method |
+|---|---|---|---|
+| Multi-hop entailment | 219/219 chat cases and 80/80 kernel cases, 0% fabrication, all six bands pass | 50 of the 219 greens (23%) are graded against a declared floor, not the classical answer. INF-C2's 20/20 grades that the engine answers contradictory memory without fabricating, never that it detects the clash. | `BENCHMARK_INFERENCE_2.0.3.md` |
+| Tool-call planning | 56/56 cases, 100% plan-completion, 100% result-completion, 0% hallucination, every rung A0→C2 | Goal driver. All 11 C2 cases pass, so the ladder now has more headroom than the case set exercises. | `BENCHMARK_AGENT_2.0.3.md` |
+| Groundedness | Every answer carries a source, and an empty graph reports itself empty (`bootstrap-empty` 2.000/2). Judge-scored groundedness 1.857/2 over 98 cases. | The 1.857 is judged (`claude-haiku-4-5-20251001`, `judge-prompt-v1`) at N=1 over 9 of 23 construction shapes, so read it as indicative. The judge runs in the offline eval harness, never in the product. | `BENCHMARK_CEFR_ENGLISH_2.0.3.md` |
+| Abstention (the honest miss) | 0% fabrication across 299 inference rows and 0% hallucination across 168 agent rows | Structural, not a tuned threshold. tmct abstains because nothing matched, so the rows test a property of a no-model design rather than a score. | `BENCHMARK_INFERENCE_2.0.3.md`, `BENCHMARK_AGENT_2.0.3.md` |
+| Determinism | Byte-identical on rerun, 0 verdict changes across 299 inference rows against the prior cycle, a 109-case replay in 877ms at $0 per turn | A property of the no-model pipeline. | `BENCHMARK_INFERENCE_2.0.3.md`, `CAPABILITIES_2.0.3.md` |
+| Dialogue robustness (role and polarity) | An adversarial persona sweep could not force a single role or polarity inversion in 55 probes — active/passive, forward/reverse, negation and the converse trap all compiled to the correct canonical shape | The same five-frame sweep (~200 probes) surfaced 25 dialogue dead-ends, eight of them confidently wrong, all from words dropped before the parser rather than from the reasoning. | `BENCHMARK_CONVERSATION_2.0.3.md` |
+
 Three offline benchmark rigs live in a clone (they are not in the npm
 package). Each replays a committed case set through the real product and
 writes graded rows you can diff between runs:
@@ -971,9 +1005,6 @@ one place an LLM is allowed, never the product:
 npm run chatbench:judge -- --product /tmp/chatbench-smoke/product.jsonl
 ```
 
-The headline numbers and their conditions live in the `BENCHMARK_*.md`
-write-ups.
-
 ## Security and supply chain
 
 tmct is $0 to run and meant to be trusted offline, so the supply chain is
@@ -995,6 +1026,74 @@ fallback, the code-extraction stack, and the MCP server were all removed. The
 naming, license, and memory model were reset to the vision above. See
 `ROADMAP.md` for the phase plan.
 
+## Standards and bibliography
+
+tmct's vocabulary is grounded in published standards where they exist, and says where they don't.
+Each alignment below is a triple in `ontology/tmct-core.ttl` and a test in
+`test/adapters/grammar-ontology.test.mjs`. `docs/references/` holds an entry per source: the
+edition, the retrieval date, the terms tmct uses, and what could not be verified.
+`PLAN_NORMATIVE.md` holds the reconciliation, one verdict per term.
+
+### The data model
+
+| source | edition | what tmct uses it for |
+|---|---|---|
+| [W3C OWL 2 Primer](https://www.w3.org/TR/owl2-primer/) · [Profiles](https://www.w3.org/TR/owl2-profiles/) | Recommendation, 2012-12-11 | The triple model. The grammar emits `rdfs:subClassOf`, `owl:Restriction`, `owl:someValuesFrom`, `owl:disjointWith` and cardinality axioms. The inference engine implements OWL 2 RL/RDF rules and uses their names: `scm-sco`, `cax-sco`, `cax-dw`, `cls-svf1`, `scm-svf1`. |
+| [RDF 1.1 Semantics](https://www.w3.org/TR/rdf11-mt/) | Recommendation, 2014-02-25 | Facts are reified statements. Appendix D.1 endorses reification for provenance, which is what tmct uses it for. [RDF 1.2](https://www.w3.org/TR/rdf12-concepts/) (Candidate Recommendation, 2026-04-07) reclassifies that vocabulary as legacy and points new systems at triple terms and `rdf:reifies`. tmct has not moved, and `docs/references/schemas/rdf-reification-and-rdf-star.md` says why. |
+| [W3C PROV-O](https://www.w3.org/TR/prov-o/) | Recommendation, 2013-04-30 | Provenance. A fact's source links sit under `prov:wasInfluencedBy`; a fact cleaned from a raw utterance is a `prov:wasDerivedFrom`; a session is a `prov:Activity` and the utterances in it are `prov:Entity`s it generated. |
+| [W3C SKOS](https://www.w3.org/TR/skos-reference/) | Recommendation, 2009-08-18 | Read, and mostly not used. `skos:related` needs `skos:Concept` at both ends, and tmct's corpus terms are bare strings. `docs/references/schemas/skos.md` records what a concept-identity pass would need. |
+| [SEON](http://se-on.org/) `code.owl` | 2012/02 | Code-graph vocabulary: `seon:hasSuperType`, `seon:containsCodeEntity`, `seon:declaresMethod`, `seon:invokesMethod` and 15 more. Where SEON has no term, tmct coins under its own `mgx:` prefix rather than borrowing SEON's. |
+
+### Language
+
+| source | edition | what tmct uses it for |
+|---|---|---|
+| [Attempto Controlled English](http://attempto.ifi.uzh.ch/site/docs/) | ACE 6.7, 2013 | The controlled-English fragment. tmct implements 9 of ACE's declarative sentence patterns. |
+| Kuhn, "A Survey and Classification of Controlled Natural Languages" | *Computational Linguistics* 40(1), 2014 | Where ACE sits among controlled languages. |
+| [ConceptNet](https://github.com/commonsense/conceptnet5/wiki/Relations) | slice pins 5.7.0 | The commonsense corpus. 25 relations are mirrored into `mgx:` and each cites its `/r/` origin. |
+| Damerau, *CACM* 7(3), 1964 · Levenshtein, *Soviet Physics Doklady* 10(8), 1966 | — | Fuzzy matching. `fuzzy.mjs` implements **Optimal String Alignment** — restricted Damerau-Levenshtein, which allows adjacent transposition but edits no substring twice. |
+
+### Reasoning and planning
+
+| source | edition | what tmct uses it for |
+|---|---|---|
+| Fikes & Nilsson, "STRIPS" | *Artificial Intelligence* 2(3–4), 1971 | The action model: operator, precondition, effect. |
+| McDermott et al., PDDL | Yale CVC TR-98-003, 1998 | The action-rule vocabulary. |
+| Doyle, "A Truth Maintenance System" | *Artificial Intelligence* 12(3), 1979 | Justification and premise. tmct records which rule entailed a fact; it does not yet record which facts fed the rule. |
+| Meszaros, *xUnit Test Patterns* | Addison-Wesley, 2007 | The test-double taxonomy — stub, spy, mock, fake, dummy — and the fixture patterns. `docs/references/testing-vocabulary.md` records where tmct's own tiers depart from the standard taxonomy, and that "blast radius" is an ops metaphor for what the literature calls Regression Test Selection. |
+| Aristotle, *Prior Analytics* I.1 (24b18–20) · Bobzien, "Ancient Logic", *SEP* | — | **Why the command is called `syllogise`.** The word is used in the older, broader sense of *sullogismos* — Aristotle's own definition is "discourse in which, certain things being stated, something other than what is stated follows of necessity", with no mention of three terms or two premises, and the Stoics used the same word for a system in which modus ponens is a *sullogismos*. Two of tmct's rules are the narrow thing exactly: `scm-sco` is **Barbara**, `cax-sco` is the **Socrates syllogism**. The others reach past term logic. The operation's own names are **forward chaining** and **materialisation**, and the code uses those. |
+
+### Storage
+
+| source | edition | what tmct uses it for |
+|---|---|---|
+| Jensen et al., "A Consensus Glossary of Temporal Database Concepts" | *SIGMOD Record* 23(1), 1994 | The time vocabulary. `mgx:utteranceTs` is valid time; `mgx:createdAt` is a transaction-time start. tmct is **not** bitemporal: `mgx:updatedAt` is an audit stamp, so tmct cannot answer what it believed last Tuesday. |
+| RFC 9923, "The FNV Non-Cryptographic Hash Algorithm" | Informational, 2026 | FNV hashes the narrow non-fact-id pools (paraphrase keys, per-URL source ids, corpus dedupe). Fact ids are content-addressed with a **64-bit truncation of SHA-256**, so a fact id is collision-resistant at tmct's corpus sizes; tmct is still **not** a Merkle tree and offers no tamper-evidence. |
+| Green, Karvounarakis, Tannen, "Provenance Semirings" | PODS 2007 | The distinction tmct's docs keep: it records source annotation and PROV-style attribution, not how-provenance. |
+
+### Measuring it
+
+| source | edition | what tmct uses it for |
+|---|---|---|
+| Council of Europe, CEFR — Companion volume | 2020, ISBN 978-92-871-8621-8 | The band labels A1–C2 the chat benchmark grades against. CEFR measures what a *person* can do communicatively; grading the difficulty of *prompts* by band is tmct's adaptation, not a CEFR-validated use. The band descriptions in `chatbench/GRADED.md` are tmct's own prose. |
+| Reiter, "On Closed World Data Bases" | *Logic and Data Bases*, Plenum, 1978, pp. 55–76 | Both halves of the honest miss. The planner's operator model is **closed-world**, which is what makes a plan checkable. The chat layer is **open-world**: it will not read "no matching rule" as "the answer is no". |
+| Chow, "On optimum recognition error and reject tradeoff" | *IEEE Trans. Information Theory* 16(1), 1970 | Prior art for the goal. The literature calls a refusal **abstention**, or selective prediction, and Chow's reject option is its root. Those methods threshold a confidence score; tmct has none, and abstains because nothing matched — which is why the row above names the mechanism. |
+| Ji et al., "Survey of Hallucination in Natural Language Generation" | *ACM Computing Surveys* 55(12), 2023 | Groundedness, and what tmct is avoiding by having no model to hallucinate with. |
+
+### Where no standard fits
+
+- **Trust.** PROV records who said a thing, not whether to believe them, and no W3C Recommendation
+  covers trust. `mgx:trustScore` and its inputs are tmct's own. Candidate literature: Artz & Gil,
+  "A survey of trust in computer science and the Semantic Web", *Journal of Web Semantics* 5(2),
+  2007.
+- **Negation.** tmct negates with its own `mgxneg:` prefix, which applies to any predicate.
+  `owl:disjointWith` would over-claim, since "john is not a man" denies one membership rather than
+  a class axiom, and OWL 2's `negativePropertyAssertion` needs a reified shape the flat JSON store
+  has no room for.
+- **Dialogue acts.** tmct has no intent vocabulary. ISO 24617-2 (SemAF) is the standard for one, and
+  `docs/references/schemas/iso-24617-2-dialogue-acts.md` maps tmct's behaviour onto it so that if
+  one is built it uses the standard's names.
+
 ## Licensing
 
 **MPL-2.0.** Free for commercial use. If you modify the covered files and

package/ROADMAP.md

@@ -112,14 +112,15 @@ status in its opening lines — read it there, because a status quoted here woul
 | [PLAN_CLASS_QUERY.md](PLAN_CLASS_QUERY.md) | "list/count all X of class Y", reconciled against what already shipped |
 | [PLAN_CODE.md](PLAN_CODE.md) | program synthesis over tmct's closed DSLs, plus JS/HTML/CSS fragments and goal-directed program repair |
 | [PLAN_CONSISTENCY_CHECK.md](PLAN_CONSISTENCY_CHECK.md) | tmct as a consistency service for an LLM tool loop |
+| [PLAN_DIALOGUE_ACTS.md](PLAN_DIALOGUE_ACTS.md) | naming tmct's turn types to ISO 24617-2 dialogue acts, deterministically |
 | [PLAN_EMBEDDINGS.md](PLAN_EMBEDDINGS.md) | the semantic-similarity axis, and the way back to it |
 | [PLAN_GRAPH_SCAN.md](PLAN_GRAPH_SCAN.md) | seed and query cost at `init:xl`/`init:xxl` corpus scale |
 | [PLAN_GUESS_NUMBER.md](PLAN_GUESS_NUMBER.md) | closed-loop planning over hidden state, via belief-interval bisection |
 | [PLAN_MUD.md](PLAN_MUD.md) | persistent, shared tmct worlds over a `server:` memory backend |
 | [PLAN_NLU_BENCHMARKS.md](PLAN_NLU_BENCHMARKS.md) | scoring tmct on the CLINC150 and HWU64 intent sets |
-| [PLAN_OPEN_ITEMS.md](PLAN_OPEN_ITEMS.md) | the build order closing the backlog `HANDOVER.md` carries |
+| [PLAN_OPEN_ITEMS.md](archive/PLAN_OPEN_ITEMS.md) | delivered — the 2.0.3-cycle build order, archived |
 | [PLAN_PARAPHRASE_VERIFICATION.md](PLAN_PARAPHRASE_VERIFICATION.md) | checking a paraphrase against the graph before it prints |
-| [PLAN_PURGE.md](PLAN_PURGE.md) | promote the load-bearing code, delete the dead weight |
+| [PLAN_PURGE.md](archive/PLAN_PURGE.md) | delivered — promoted the load-bearing code, deleted the dead weight, archived |
 | [PLAN_REPO_INDEX.md](PLAN_REPO_INDEX.md) | tmct grows its own code parsers, ported from seonix |
 | [PLAN_SYLLOGIST.md](PLAN_SYLLOGIST.md) | the reasoning engine's incrementality and retraction horizon |
 | [PLAN_SYLLOGIST_EL_DL.md](PLAN_SYLLOGIST_EL_DL.md) | beyond OWL 2 RL: an EL classifier, then a DL tableau prover |

package/bin/tmct.mjs

@@ -28,100 +28,19 @@
 // shape and the green test suite. See README.md for what tmct is and
 // deliberately is NOT, and ROADMAP.md for where it is going.
 
+// The verb list itself is data (src/domain/cli-verbs.mjs) and both the Usage
+// block below and the unknown-invocation line read it, so a new verb is one
+// entry rather than two edits that drift. It is pure and imports nothing, so a
+// static import here costs `tmct --help` nothing.
+import { renderUsage, unknownInvocationMessage } from "../src/domain/cli-verbs.mjs";
+
 const HELP = `tmct — The Mechanical Code Talker
 
 A tolerant, offline, $0 chat that guides you toward precision queries about a
 software repository. No model calls; no codebase index of its own.
 
 Usage:
-  tmct                         interactive chat (the headline surface)
-  tmct chat [--repo <abs>]     chat over a specific repo's graph
-       [--graph <path>]        explicit graph file (repeatable — multiple graphs merge;
-                               see src/adapters/graph-merge.mjs); wins over --repo/TMCT_GRAPH_FILE/tmct.toml
-       [--config <path>]       an alternate tmct.toml location (a file or a directory)
-       [--ephemeral]           read the graph but write nothing back (demo/read-only)
-       [--prompt "<text>"]     one-shot: run the prompt's sentences as turns and print
-                               the final answer (teach state first, trigger last)
-       [--render blocks]       with --prompt: when the final turn produced a plan,
-                               write it as a self-contained animated page
-       [--output <path>]       the rendered page's path (default plan.html)
-       [--narrate]             start with narrate mode on — a verbose, developer-facing
-                               trace of decision points/matched pattern/results/goal per
-                               turn, appended under a "--- narrate ---" marker (also
-                               TMCT_NARRATE=1; toggle mid-session with /narrate on|off)
-       [--plain]               force the plain readline shell (the default when
-                               stdin/stdout is not a terminal)
-       [--memory-backend <default|memory|sqlite>]  storage backend for taught facts this
-                               session (CLI flag > TMCT_MEMORY_BACKEND env > tmct.toml's
-                               [memory] backend > "default", the flat .tmct/ JSON file)
-  tmct memory [--repo <abs>]   what tmct remembers: facts, utterances, sessions,
-       [--config <path>]       folded blocks (the /memory chat command, from the shell)
-       [--verbose]
-  tmct init [--repo <abs>]     initialize a repo for tmct (default: cwd): .tmct/,
-       [--force]               tmct.toml, .tmct/TOOLS.md (the cold-tool catalog),
-                               tier-1 corpus seed, provenance record
-       [--corpus <id|path>]    also seed a corpus — a tier-2 manifest id (aws|python|java|
-                               general) or a jsonl file path — opt-in, offline, $0
-       [--ontology <name|path>]  activate+seed an ontology bundle (a recognized name or a path)
-       [--lexicon <name|path>]   activate a lexicon bundle (recognized name or a path;
-                               merged read-time, never seeded — see mergedLexiconExtra)
-       [--graph <path>]        set graph_file/graph_files in tmct.toml (repeatable)
-       [--config <path>]       write to an alternate tmct.toml location
-       [--detect]              suggest a tier-2 corpus from the repo's manifests
-                               (pyproject.toml → python, pom.xml → java); never seeds unasked
-       [--with-persona <name>] write an explicit [extensions]/[bias] preset into tmct.toml
-                               ("code" — today's implicit default, made explicit)
-       [--persona-size <medium|large>]  grow the default "human" persona's fact count
-                               beyond Small (the default): "medium" activates
-                               human-medium.jsonl (~1,608 facts total), "large" also
-                               activates human-large.jsonl (~13,600 facts total,
-                               with genuine multi-hop hypernym chains) — additive
-                               size tiers of the SAME bundle, not separate personas
-       [--memory-backend <default|memory|sqlite>]  write tmct.toml's [memory] backend
-                               (same flag name as \`tmct chat\`) — a later \`tmct chat\`
-                               in this repo picks it up with no flag needed
-  tmct import [--repo <abs>]   activate+seed into an ALREADY-initialized repo (any
-       [--corpus <id|path>]    combination of these flags in one call). --graph is a
-       [--ontology <name|path>]  DIFFERENT operation from the others: it APPENDS to
-       [--lexicon <name|path>]   tmct.toml's graph_files array (multi-graph growth),
-       [--graph <path>]        never an extensions-bundle activation.
-       [--file <definition.txt>]  teach a plain-text definition file sentence by
-                               sentence (# lines are comments); any declined
-                               sentence exits non-zero with the sentence named
-       [--memory-backend <default|memory|sqlite>]  same knob as \`tmct init\`
-       [--config <path>]
-  tmct extend --validate <dir>  validate a third-party extension pack's declared
-       [--config <path>]      resources (corpus/lexicon/templates) before activating
-                               it in any repo's tmct.toml; exits non-zero on failure
-  tmct syllogise [--repo <abs>] speculative inference (offline maintenance job): forward-
-       [--depth <n>] [--budget <n>]  chain the memory's rdfs:subClassOf closure, materialising
-       [--config <path>]      bounded, low-trust, retractable entailed facts (never on the chat path)
-  tmct viz [--repo <abs>]      write one self-contained HTML page: the memory graph as a
-       [--focus <term>]       readable ledger of fact-sentences around one focus term,
-       [--term <word>]        with segments, a two-hop minimap, and an in-page chat dock
-       [--limit <n>]          that answers from the embedded graph. Focuses on the newest
-       [--output <path>]      taught fact's subject by default (--focus <term> or
-       [--config <path>]      --term <word> override it); --output defaults to
-                               ledger.html in the cwd; --limit caps the embedded fact
-                               rows; --term resolves via the same normalization chat uses.
-  tmct serve [--repo <abs>]    run the Anthropic Messages API-compatible endpoint
-       [--host <h>] [--port <n>]  (POST /v1/messages) over the graph — a deterministic,
-       [--graph <path>]        no-LLM "model" a tool-loop client can call; $0 usage.
-       [--config <path>]       Defaults: host 127.0.0.1, port 8787. Ctrl+C to stop.
-  tmct plan "<request>"        the capability router: compose/execute read-only graph-
-       [--repo <abs>]          query tool calls for a compound or maintenance-goal
-       [--graph <path>]        request ("of the modules impacted by X, which are
-       [--config <path>]       untested", "what most needs a test") — a real STRIPS/
-       [--tools <a,b,...>]     PDDL planner (src/domain/router/*), never a guessed call.
-       [--json]                Prints the grounded step sequence + composed answer,
-                               or an honest "no plan found". --tools restricts the
-                               declared toolset; --json prints the full loop result.
-  tmct cli <tool> '{…}'        invoke a graph tool directly (carry-over, de-emphasized)
-       [--repo <abs>]          the repo to answer from; the payload's "repo_path" says
-       [--graph <path>]        the same thing. --graph names the graph file outright
-       [--config <path>]       (repeatable), --config an alternate tmct.toml
-  tmct cli digest '{…}'        architecture map + per-module context bundles
-  tmct --help                  show this help
+${renderUsage()}
 
 On a terminal, chat opens the full-screen TUI; piped input gets the plain shell.
 In chat: /help lists slash-commands; /exit leaves. Session log → <repo>/.tmct/session-<id>.log.
@@ -1117,6 +1036,15 @@ async function main() {
     return;
   }
 
+  if (mode === "extract") {
+    // `tmct extract` — run a text file's sentences through the chat's own teach
+    // recognizer and keep what it grounds. Lazily imported: it pulls the whole
+    // chat stack, which `tmct --help` and chat startup must not pay for.
+    const { main: extractFacts } = await import("../src/services/extract-facts.mjs");
+    await extractFacts(process.argv.slice(3));
+    return;
+  }
+
   if (mode === "viz") {
     // `tmct viz` — the ledger explorer: one self-contained HTML page rendering
     // the memory graph as readable fact-sentences around a focus term, with
@@ -1336,8 +1264,7 @@ async function main() {
 
   // An unknown mode gets the instructive usage line and exit 2. (A bare invocation
   // never lands here — the argv splice above rewrote it to `chat`.)
-  process.stderr.write(`tmct: unknown invocation "${process.argv.slice(2).join(" ")}". ` +
-    "Use `chat`, `memory`, `init`, `import`, `extend --validate`, `syllogise`, `serve`, `plan`, `cli digest …`, or `cli <tool> …`.\n");
+  process.stderr.write(unknownInvocationMessage(process.argv.slice(2).join(" ")));
   process.exit(2);
 }
 

package/corpus/LICENSES.json

@@ -101,11 +101,26 @@
       "notice": "corpus/tier2/manifest.json"
     },
     {
-      "path": "corpus/generated/ace-surface-variants.jsonl",
-      "upstream": "mechanically generated from Open English WordNet / SemCor example sentences and this repository's own docs prose",
-      "license": "CC-BY-4.0",
+      "path": "corpus/prose/sqlite/",
+      "upstream": "SQLite documentation (sqlite.org), plain-text extractions of 12 pages",
+      "license": "public-domain",
       "shareAlike": false,
-      "notice": "corpus/generated/manifest.json"
+      "notice": "corpus/prose/sqlite/LICENSE-NOTICE"
+    },
+    {
+      "path": "corpus/prose/wikipedia/",
+      "upstream": "Wikipedia — simple.wikipedia.org lead sections (56 everyday-concept articles) and en.wikipedia.org full articles (12 in tmct's technical domain), fetched via the MediaWiki extracts API",
+      "license": "CC-BY-SA-4.0",
+      "shareAlike": true,
+      "notice": "corpus/prose/wikipedia/LICENSE-NOTICE"
+    },
+    {
+      "path": "corpus/generated/ace-surface-variants.jsonl",
+      "upstream": "mechanically generated from Open English WordNet / SemCor example sentences (CC-BY-4.0) and corpus/prose/, which includes CC-BY-SA-4.0 Wikipedia text",
+      "license": "CC-BY-SA-4.0",
+      "shareAlike": true,
+      "notice": "corpus/generated/manifest.json",
+      "note": "CC-BY-SA-4.0, not CC-BY-4.0: the rescue rows quote and word-substitute Wikipedia sentences from corpus/prose/wikipedia/, so this published file is a modified derivative of CC-BY-SA-4.0 text and inherits its viral share-alike. It carried CC-BY-4.0 while the prose corpus was this repository's own docs; adding Wikipedia is what changed it. See corpus/prose/wikipedia/LICENSE-NOTICE."
     },
     {
       "path": "data/",

package/corpus/README.md

@@ -45,6 +45,9 @@ notice}), guarded by `test/estate/corpus-licences.test.mjs`.
 | `tier2/{aws,python,java,general}.jsonl` | tier-2 SAMPLE corpuses — same fact shape as the tier-1 slice, loadable via the same path (`general` is the one deliberately non-code-domain bundle) | ~4-6 KB each | MPL-2.0 |
 | `tier2/human.jsonl` | the DEFAULT active bundle: the everyday "human-world" persona, Small tier (664 facts) — hand-curated from Open English WordNet, bridged to Schema.org's top-level classes | ~80 KB | MPL-2.0 (hand-authored fact set; not a verbatim WordNet/Schema.org excerpt — see `tier2/generate.mjs`'s own header comment) |
 | `tier2/human-medium.jsonl` / `tier2/human-large.jsonl` | SIZE tiers of the SAME `human` bundle — each holds ONLY the facts that size adds beyond the previous one; both shipped INACTIVE by default, activated via `tmct init --persona-size medium\|large`. Built by `scripts/build-persona-tiers.mjs` from the same WordNet source, automatically curated (sense-ranked, blocklist/denylist-filtered) rather than hand-typed one at a time, given the scale (944 / ~12,000 facts) | ~111 KB / ~1.4 MB | MPL-2.0 (same "hand-authored in homage to the source's shape" basis as `human.jsonl`) |
+| `prose/sqlite/*.txt` | the frozen prose corpus, code half: plain-text extractions of 12 SQLite documentation pages | ~375 KB | **public domain** (see `prose/sqlite/LICENSE-NOTICE`) |
+| `prose/wikipedia/*.txt` | the frozen prose corpus, English half: 56 simple-English lead sections (everyday concepts) + 12 en.wikipedia full articles (NLP/OWL/logic) | ~356 KB | **CC-BY-SA 4.0** (see `prose/wikipedia/LICENSE-NOTICE`) |
+| `prose/manifest.json` | provenance for the above: source URL, byte count and sha256 per file, plus the fetch date | — | MPL-2.0 |
 | `tier2/human-examples.jsonl` / `tier2/human-examples-medium.jsonl` / `tier2/human-examples-large.jsonl` | the example-sentence corpus — real natural-language sentences mapped to the same curated vocabulary, NOT fact triples. Small/Medium tiers are 100% WordNet's own inline `example:` field (same CC-BY-4.0 basis as the fact bundles). Large tier ALSO includes a SemCor-filtered supplement (real Brown Corpus text, re-tagged to modern OEWN senses) for categories where WordNet's own inline coverage is thin (`human-nature` especially — `noun.animal.yaml`'s inline rate is under 1%) | ~11 KB / ~43 KB / ~257 KB | **The WordNet-inline entries**: CC-BY-4.0 (Princeton WordNet + Open English WordNet team), reproduced verbatim (these ARE the source's own example sentences, not a paraphrase). **The `source: "semcor:…"` entries**: CC-BY-4.0 per this SemCor fork's own `LICENSE.md` (a local uncommitted checkout of `globalwordnet/semcor`, real Brown Corpus text re-tagged to modern senses — the original 1960s Brown Corpus permissions aren't independently re-verified beyond that fork's own license statement; proceeding with attribution was an explicit operator decision) — each entry's own `source` field names its origin file (`semcor:<genre>/<file>.yaml`) for exactly this reason |
 
 And alongside (same phase, different directory because it is tmct-original
@@ -55,6 +58,51 @@ data, not a derived corpus):
 | `../data/templates/responses.jsonl` | ~56 response templates ({id, class, template, register}) | MPL-2.0 |
 | `../data/phrasebook/software-phrases.txt` | ~170 SE phrase patterns + 31 synonym families | MPL-2.0 |
 
+## The prose corpus (`prose/`) — external and frozen, on purpose
+
+`prose/` is plain English text. It is not seeded and holds no facts. It exists
+to be *measured against*: `scripts/template-coverage.mjs` asks how much of it
+tmct's ACE grammar parses, and the rescue pass in
+`scripts/generate-template-variants.mjs` mines it for near-misses.
+
+It used to be this repository's own root `*.md` docs, and both reasons it moved
+are worth stating, because both were real bugs:
+
+- **A doc edit could drift a shipped artifact.** The corpus was globbed from
+  every root `*.md`, so editing a README or a plan changed
+  `generated/ace-surface-variants.jsonl` — a file npm ships. A rescue row could
+  be, and once was, a sentence someone had written into a plan that morning.
+  Committed text fetched from a recorded URL cannot do that.
+- **The coverage metric was not comparable across versions.** A hit rate only
+  means something against a fixed corpus. When the corpus moves with every doc
+  edit, two versions' numbers are not measuring the same thing, and a change in
+  the number says nothing about a change in the grammar.
+
+So the corpus is external (nothing in this repo can edit it), frozen (a
+snapshot, checksummed per file in `prose/manifest.json`), and re-fetched only
+when someone deliberately runs `npm run gen:prose-corpus`. `test/estate/prose-corpus.test.mjs`
+checks the committed bytes against the manifest, so a hand-edit fails loudly.
+
+**Licensing picked the sources, not preference.** The rescue pass substitutes
+words and commits the result to a file npm publishes, so this corpus is
+republished as a *modified derivative*. Only sources that permit derivatives
+qualify. IETF RFCs and W3C specifications — including the OWL 2 Primer, the
+best domain match there is — permit redistribution but **not** modification, so
+they are unusable here and must not be added. SQLite's documentation is public
+domain; Wikipedia is CC BY-SA 4.0. Adding Wikipedia is what makes
+`generated/ace-surface-variants.jsonl` CC-BY-SA-4.0 rather than CC-BY-4.0:
+share-alike is viral, and it reaches the generated file.
+
+**Why simple-English lead sections for the everyday half.** Measured, on the
+metric the rescue pass actually consumes (sentences with exactly one undeclared
+word): simple.wikipedia.org lead sections yield ~0.33 rescue candidates per KB
+against ~0.12 for technical prose. Lead sections are dense definitional English
+("A penguin is a bird that cannot fly"), which is the shape the ACE grammar was
+built to parse; later article sections drift into history and citations. The
+titles cover the same everyday concepts the persona clumps model
+(`scripts/build-persona-tiers.mjs`'s `CLUMP_FILES`). The technical half stays
+because tmct's own domain is code.
+
 ## How seeding works
 
 `src/adapters/corpus/conceptnet.mjs` turns the slice into tmct memory facts:

package/corpus/generated/README.md

@@ -17,7 +17,7 @@ measurement: `node scripts/template-coverage.mjs` (baseline) and
 
 | `kind` | What | Fields |
 |---|---|---|
-| `rescue` | A real docs-corpus sentence that almost fit the ACE grammar (exactly one undeclared word), rescued by substituting a WordNet synonym of that word that's ALSO already declared in `src/domain/grammar/lexicon-core.json` | `sentence`, `rescued`, `from`, `to`, `pos`, `synsetId`, `sourceFile` |
+| `rescue` | A real sentence from the frozen prose corpus (`corpus/prose/`) that almost fit the ACE grammar (exactly one undeclared word), rescued by substituting a WordNet synonym of that word that's ALSO already declared in `src/domain/grammar/lexicon-core.json` | `sentence`, `rescued`, `from`, `to`, `pos`, `synsetId`, `sourceFile` |
 | `variant` | A real WordNet/SemCor example sentence (`corpus/tier2/human-examples*.jsonl`) that already hits `parseAce`, with one content word swapped for a same-synset sibling (both ends independently declared in tmct's own lexicon) | `seed`, `generated`, `from`, `to`, `pos`, `synsetId`, `sourceCorpus` |
 | `alt-phrasing` | A possessive-pattern (#7) hit rewritten in the ACE grammar's OTHER declared surface form for the same triple (`"X's Y is Z"` <-> `"the Y of X is Z"`, both routed through `buildPossessive` in `src/domain/grammar/ace.mjs`) | `seed`, `generated`, `pattern`, `form` |
 
@@ -31,12 +31,27 @@ product code — it is committed raw material, verified by
 `scripts/template-coverage.mjs`. Wiring it into live answer rendering is a
 separate, future phase.
 
-## Licence
+## Licence — CC-BY-SA-4.0
 
-Open English WordNet content (synset `members`/`example` fields) is
-CC-BY-4.0; SemCor-derived sentences (`corpus/tier2/human-examples-large.jsonl`)
-are CC-BY-4.0 per that fork's own `LICENSE.md` (same basis `corpus/README.md`
-already documents for that file). A `variant`/`rescue` row is a one-word
-derivative of that material — CC-BY-4.0. An `alt-phrasing` row is a pure
-grammar-level rephrasing of the same source sentence — CC-BY-4.0 also, no new
-creative content introduced either way.
+**`ace-surface-variants.jsonl` is CC-BY-SA-4.0**, and this file ships in the npm
+package. It is not under this repository's MPL-2.0.
+
+Two source licences meet in this file:
+
+- Open English WordNet content (synset `members`/`example` fields) is CC-BY-4.0;
+  SemCor-derived sentences (`corpus/tier2/human-examples-large.jsonl`) are
+  CC-BY-4.0 per that fork's own `LICENSE.md` (the same basis `corpus/README.md`
+  already documents for that file). `variant` and `alt-phrasing` rows derive
+  from this material.
+- `corpus/prose/` feeds the `rescue` rows, and it includes Wikipedia text under
+  **CC-BY-SA-4.0** (`corpus/prose/wikipedia/LICENSE-NOTICE`). A `rescue` row
+  quotes such a sentence and substitutes one word into it, so the row is a
+  published, modified derivative of CC-BY-SA-4.0 text.
+
+Share-alike is viral, so the combined file takes the stricter licence:
+**CC-BY-SA-4.0**. Redistributing it, modified or not, means doing so under
+CC-BY-SA 4.0 with the attribution in `corpus/prose/wikipedia/LICENSE-NOTICE`.
+
+This file was labelled CC-BY-4.0 while the rescue corpus was this repository's
+own MPL-2.0 docs. Repointing the corpus at Wikipedia is what changed it.
+`corpus/prose/sqlite/` is public domain and adds no condition of its own.

package/corpus/generated/ace-surface-variants.jsonl

@@ -1,4 +1,7 @@
-{"kind":"rescue","sentence":"Code search is a retrieval task","rescued":"Code hunt is a retrieval task","from":"search","to":"hunt","pos":"noun","synsetId":"00947217-n","sourceFile":"PLAN_EMBEDDINGS.md","provenance":"wordnet:00947217-n"}
+{"kind":"rescue","sentence":"The claim of the previous paragraph is extensively checked in the SQLite regression test suite using a special test harness that simulates the effects on a database file of operating system crashes and power failures","rescued":"The claim of the old paragraph is extensively checked in the SQLite regression test suite using a special test harness that simulates the effects on a database file of operating system crashes and power failures","from":"previous","to":"old","pos":"adjective","synsetId":"00128009-s","sourceFile":"sqlite/transactional.txt","provenance":"wordnet:00128009-s"}
+{"kind":"rescue","sentence":"The meaning of sentences is described as a small set of interpretation rules","rescued":"The significance of sentences is described as a small set of interpretation rules","from":"meaning","to":"significance","pos":"noun","synsetId":"06613680-n","sourceFile":"wikipedia/Attempto_Controlled_English.txt","provenance":"wordnet:06613680-n"}
+{"kind":"rescue","sentence":"The difficulty of planning is dependent on the simplifying assumptions employed","rescued":"The trouble of planning is dependent on the simplifying assumptions employed","from":"difficulty","to":"trouble","pos":"noun","synsetId":"00625102-n","sourceFile":"wikipedia/Automated_planning_and_scheduling.txt","provenance":"wordnet:00625102-n"}
+{"kind":"rescue","sentence":"The consumption of food is normally enjoyable to humans","rescued":"The intake of food is normally enjoyable to humans","from":"consumption","to":"intake","pos":"noun","synsetId":"00839759-n","sourceFile":"wikipedia/Food.txt","provenance":"wordnet:00839759-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the one of trumps is a sure winner","from":"ace","to":"one","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the single of trumps is a sure winner","from":"ace","to":"single","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the unity of trumps is a sure winner","from":"ace","to":"unity","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}

package/corpus/generated/manifest.json

@@ -2,8 +2,8 @@
   "version": 1,
   "generated": "by scripts/generate-template-variants.mjs",
   "file": "ace-surface-variants.jsonl",
-  "rows": 18,
-  "bytes": 5012,
-  "sha256": "28fae5a3353f5dd1df365b4407e0ac2a5c66b98b232d8bf454fe3573f2c16760",
-  "license": "CC-BY-4.0 (WordNet-derived synonym substitutions of Open English WordNet / SemCor example sentences and this repo's own MPL-2.0 docs prose — see corpus/generated/README.md)"
+  "rows": 21,
+  "bytes": 6417,
+  "sha256": "6e1daca10e43455608e5ee91ca41e20c5a1c5b0127f14ffd087c85580fe1f0ad",
+  "license": "CC-BY-SA-4.0 (WordNet-derived synonym substitutions of Open English WordNet / SemCor example sentences (CC-BY-4.0) and corpus/prose/, which includes CC-BY-SA-4.0 Wikipedia text. Share-alike is viral and the rescue rows quote and modify that text, so the combined file is CC-BY-SA-4.0 — the stricter of the two. See corpus/prose/wikipedia/LICENSE-NOTICE and corpus/generated/README.md)"
 }