@polycode-projects/the-mechanical-code-talker 1.9.2 → 1.10.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. package/README.md +441 -202
  2. package/bin/tmct.mjs +126 -1
  3. package/package.json +4 -2
  4. package/src/answer-variants.mjs +8 -36
  5. package/src/ask-browser-entry.mjs +5 -23
  6. package/src/ask-browser.bundle.js +1 -2
  7. package/src/ask-nlp.mjs +9 -23
  8. package/src/ask-vocab.mjs +139 -589
  9. package/src/ask.mjs +627 -1729
  10. package/src/chat.mjs +1684 -2872
  11. package/src/cli-args.mjs +14 -28
  12. package/src/codegraph.mjs +236 -644
  13. package/src/completions/complete.mjs +18 -62
  14. package/src/completions/graph-adapter.mjs +14 -60
  15. package/src/completions/group.mjs +12 -68
  16. package/src/completions/infer.mjs +38 -126
  17. package/src/completions/prune.mjs +17 -70
  18. package/src/completions/rank.mjs +16 -69
  19. package/src/completions/search.mjs +8 -31
  20. package/src/concept.mjs +32 -88
  21. package/src/conformance.mjs +11 -15
  22. package/src/corpus/conceptnet.mjs +31 -89
  23. package/src/corpus/templates.mjs +19 -45
  24. package/src/corpus/unknown-ingest.mjs +31 -92
  25. package/src/embed.mjs +10 -22
  26. package/src/extensions.mjs +50 -154
  27. package/src/finish.mjs +35 -91
  28. package/src/grammar/ace.mjs +16 -40
  29. package/src/grammar/assert.mjs +1 -1
  30. package/src/grammar/lexicon-core.json +1 -1
  31. package/src/grammar/lexicon.mjs +9 -27
  32. package/src/graph-merge.mjs +2 -3
  33. package/src/hash.mjs +6 -14
  34. package/src/index.mjs +6 -10
  35. package/src/init.mjs +38 -125
  36. package/src/interpret/fuzzy.mjs +10 -29
  37. package/src/interpret/merge.mjs +9 -27
  38. package/src/interpret/normalize.mjs +137 -585
  39. package/src/interpret/pipeline.mjs +23 -71
  40. package/src/interpret/strategies/ace.mjs +7 -31
  41. package/src/interpret/strategies/constructions.mjs +14 -41
  42. package/src/interpret/strategies/grammar.mjs +21 -60
  43. package/src/interpret/strategies/keywords.mjs +42 -131
  44. package/src/interpret/strategies/noise-strip.mjs +18 -89
  45. package/src/memory/bias.mjs +11 -54
  46. package/src/memory/blocks.mjs +18 -69
  47. package/src/memory/core.mjs +171 -591
  48. package/src/memory/fold.mjs +0 -0
  49. package/src/memory/inspect.mjs +7 -25
  50. package/src/memory/shacl.mjs +10 -39
  51. package/src/memory/trust.mjs +26 -127
  52. package/src/memory-ask-browser-entry.mjs +7 -30
  53. package/src/memory-ask-browser.bundle.js +1 -1
  54. package/src/paraphrase.mjs +20 -53
  55. package/src/planning.mjs +15 -157
  56. package/src/prose-nlp.mjs +4 -17
  57. package/src/prose.mjs +19 -67
  58. package/src/providers/bootstrap.mjs +1 -2
  59. package/src/providers/fixture.mjs +1 -2
  60. package/src/providers/graph-service.mjs +28 -59
  61. package/src/repository-interface.mjs +6 -8
  62. package/src/router/drive.mjs +183 -0
  63. package/src/router/goal-reasoner.mjs +66 -231
  64. package/src/router/guardrail.mjs +20 -58
  65. package/src/router/planner.mjs +15 -46
  66. package/src/router/registry.mjs +13 -43
  67. package/src/router/resolver.mjs +46 -131
  68. package/src/router/results.mjs +231 -0
  69. package/src/schema-docs.mjs +10 -27
  70. package/src/server-http.mjs +10 -19
  71. package/src/server.mjs +22 -28
  72. package/src/sessions.mjs +15 -30
  73. package/src/source-slice.mjs +5 -7
  74. package/src/source.mjs +10 -20
  75. package/src/syllogise.mjs +187 -575
  76. package/src/telemetry.mjs +3 -3
  77. package/src/toml-config.mjs +4 -4
  78. package/src/tui/app.mjs +9 -19
  79. package/src/viz.mjs +66 -123
  80. package/src/wink-model.mjs +10 -24
@@ -1,54 +1,21 @@
1
- // extensions.mjs — the extension-pack seam: one place a host repo (or a
2
- // third-party package such as seonix/marginalia) declares which corpus/
3
- // lexicon/templates bundles feed tmct, and how much each bundle's facts are
4
- // trusted relative to the others.
1
+ // extensions.mjs — the extension-pack seam: one place a host repo (or a third-party
2
+ // package) declares which corpus/lexicon/templates bundles feed tmct, and how much each
3
+ // bundle's facts are trusted relative to the others.
5
4
  //
6
5
  // resolveExtensions(repoRoot) → { entries: Map<name, ResolvedEntry>, biasByBundle }
7
6
  //
8
- // BUILTIN_EXTENSIONS' DEFAULT ACTIVE BUNDLE IS `human` (PLAN_SEED.md, the
9
- // persona flip): a fresh repo now seeds an everyday-world vocabulary (people,
10
- // places, objects, nature, time/events, body/food, mind hand-curated from
11
- // Open English WordNet + Schema.org) rather than the old implicit code-domain
12
- // default. `seon` and `conceptnet` are SHIPPED but now INACTIVE — both are
13
- // equally code/tech-domain-biased (conceptnet's committed slice was filtered
14
- // via a tech-domain seed-term match, PLAN_SEED.md §2), so BOTH flip together,
15
- // not just seon — a repo that wants the old behavior asks for it explicitly
16
- // (`tmct init --with-persona code`, or `[extensions.seon]`/`[extensions.
17
- // conceptnet]` `active = true`). Four more shipped-but-INACTIVE tier-2 bundles
18
- // (`tier2-aws` / `tier2-python` / `tier2-java` / `tier2-general`) round out the
19
- // catalog. Activating any of these is a config-only edit (`tmct init --corpus
20
- // aws`, or a `[extensions.tier2-aws] active = true` in tmct.toml) — zero code
21
- // change. `tier2-general`'s own 49-fact animal/weather set (PLAN_AGENTS.md
22
- // Phase 1) is superseded IN DEFAULT ROLE by `human`'s much larger
23
- // `human-nature` clump, but stays shipped/selectable on its own for a caller
24
- // that wants that narrow slice without the rest of the human persona.
7
+ // `human` is the default active bundle (everyday-world vocabulary); `seon`/`conceptnet`
8
+ // (code/tech-domain) and four tier-2 bundles ship inactive, activated via `tmct init
9
+ // --with-persona`/`--corpus <id>` or a `[extensions.<name>] active = true` override.
10
+ // `human-medium`/`human-large` are additive SIZE TIERS of `human`, not separate personas.
25
11
  //
26
- // `human-medium`/`human-large` (also shipped-but-INACTIVE) are SIZE TIERS of
27
- // the SAME `human` bundle, not separate personas (PLAN_SEED.md §3) each
28
- // holds only the facts that size adds beyond the previous one. Activated
29
- // together via `tmct init --persona-size medium|large` (bin/tmct.mjs), which
30
- // resolves them through this same registry.
12
+ // A `tmct.toml` `[extensions]` table-of-tables may override a recognized builtin, or
13
+ // declare a new host entry with its own `kind` (corpus | lexicon | templates | pack |
14
+ // ontology). A separate flat `[bias]` table (bundle-name number) feeds
15
+ // src/memory/bias.mjs's ranking.
31
16
  //
32
- // A `tmct.toml` may carry a top-level `[extensions]` table-of-tables
33
- // (`[extensions.tier2-aws]`, …): a RECOGNIZED name (one of the builtins above)
34
- // may override `active`/paths/etc; an UNRECOGNIZED name declares a brand new
35
- // host entry and MUST carry a `kind` (corpus | lexicon | templates | pack |
36
- // ontology) — a `pack` entry may combine any of corpus_path/lexicon_path/
37
- // templates_path/phrasebook_path under one `active` flag and one provenance
38
- // name, the shape a third-party vocabulary package hands tmct. `ontology` is a
39
- // DISTINCT, nameable kind for an ontology bundle (as opposed to a plain
40
- // `corpus` bundle) — its `ontology_path` key is just an alias populating the
41
- // SAME internal `corpusPath` field a corpus entry uses, so every downstream
42
- // seeder/loader needs zero branching by kind name.
43
- //
44
- // A SEPARATE top-level `[bias]` table (flat: bundle-name → number) feeds
45
- // src/memory/bias.mjs's ranking — never nested under `[extensions.*]`.
46
- //
47
- // Entries are returned in a FIXED, deterministic order — `seon` first, then
48
- // `conceptnet`, then every other entry sorted by name — mirroring the
49
- // seon-before-conceptnet idempotency-ordering precedent chat.mjs's
50
- // seedBootstrapMemory already establishes (seon's curated facts should win the
51
- // content-hash idempotency race over general ConceptNet noise).
17
+ // Entries are returned in a fixed order: `seon` first, then `conceptnet`, then the rest
18
+ // sorted by name (so seon's curated facts win idempotency races over ConceptNet noise).
52
19
 
53
20
  import { isAbsolute, join, resolve, dirname } from "node:path";
54
21
  import { readFile } from "node:fs/promises";
@@ -65,67 +32,45 @@ import {
65
32
  toFacts,
66
33
  } from "./corpus/conceptnet.mjs";
67
34
 
68
- // corpus/namenet/generate.mjs's output — same small-top-up shape as the
69
- // wordnet-xl/wordnet-full entries below, just a single bundle (not
70
- // worth PKG_ROOT-style plumbing through corpus/conceptnet.mjs for one
71
- // directory constant, so computed locally here instead).
35
+ // corpus/namenet/generate.mjs's output — a single small top-up bundle.
72
36
  const NAMENET_DIR = join(dirname(fileURLToPath(import.meta.url)), "..", "corpus", "namenet");
73
37
 
74
38
  export const EXTENSION_KINDS = Object.freeze(["corpus", "lexicon", "templates", "pack", "ontology"]);
75
39
 
76
- // The definitional-band-first predicate order chat.mjs's bootstrap has always
77
- // passed for the ConceptNet seed (SEED_PREFER) re-declared here (not
78
- // imported from chat.mjs) to keep this module off chat.mjs's heavy graph, the
79
- // same "re-declare, don't import" discipline init.mjs's own SEED_PREFER uses.
40
+ // The definitional-band-first predicate order for the ConceptNet seed (re-declared,
41
+ // not imported, to keep this module off chat.mjs's heavy graph).
80
42
  const CONCEPTNET_PREFER = ["rdfs:subClassOf", "rdf:type", "mgx:usedFor", "mgx:partOf", "mgx:capableOf"];
81
43
 
82
44
  /** The shipped defaults — a FRESH object per call, so a caller can never
83
45
  * accidentally mutate a module-level singleton. */
84
46
  function builtinExtensions() {
85
47
  return {
86
- // WAS active:true (the implicit code-domain default) — now opt-in.
87
- // PLAN_SEED.md §2: re-activate explicitly (`tmct init --with-persona
88
- // code`, or `[extensions.seon] active = true`) for the old behavior.
48
+ // Opt-in code-domain bundle.
89
49
  seon: {
90
50
  kind: "corpus",
91
51
  active: false,
92
52
  corpusPath: SEON_CONCEPTS_FILE,
93
53
  provenancePrefix: "corpus:seon",
94
54
  },
95
- // WAS active:true now opt-in too, not just seon (PLAN_SEED.md §2: the
96
- // committed slice is itself tech-domain-filtered, equally biased).
55
+ // Opt-in too: the committed slice is tech-domain-filtered, equally biased.
97
56
  conceptnet: {
98
57
  kind: "corpus",
99
58
  active: false,
100
59
  corpusPath: CONCEPTNET_SLICE_FILE,
101
60
  provenancePrefix: "corpus:conceptnet",
102
- // matches chat.mjs's seedBootstrapMemory exactly: uncapped, definitional
103
- // band first.
104
61
  limit: undefined,
105
62
  prefer: CONCEPTNET_PREFER,
106
63
  },
107
- // NEW — the default active bundle (PLAN_SEED.md). Everyday-world
108
- // vocabulary: people, places, objects, nature, time/events, body/food,
109
- // mind, plus the human-base/human-bridge scaffolding connecting WordNet's
110
- // and Schema.org's independently-built taxonomies (PLAN_SEED.md §3, §8).
64
+ // The default active bundle: everyday-world vocabulary plus the scaffolding
65
+ // connecting WordNet's and Schema.org's independently-built taxonomies.
111
66
  human: {
112
67
  kind: "corpus",
113
68
  active: true,
114
69
  corpusPath: join(TIER2_DIR, "human.jsonl"),
115
70
  provenancePrefix: "corpus:human",
116
71
  },
117
- // NEW — Medium/Large SIZE tiers of the SAME `human` bundle (PLAN_SEED.md
118
- // §3), not separate personas: each file holds ONLY the facts that size
119
- // adds beyond the previous one (Medium beyond Small, Large beyond
120
- // Medium), so activating them is purely ADDITIVE alongside `human`
121
- // (never a replacement for it). Both ship INACTIVE — Small stays the
122
- // unconditional default — and are activated together via `tmct init
123
- // --persona-size medium|large` (bin/tmct.mjs), which resolves them
124
- // through this SAME BUILTIN_EXTENSIONS lookup and the ordinary
125
- // `--corpus <id>` activation seam (activatePluggableInput). "large"
126
- // activates BOTH human-medium and human-large (Large's facts are
127
- // Medium's plus its own — both bundles must be active to reach the
128
- // full ~13,600-fact total).
72
+ // Medium/Large SIZE tiers of `human` (additive, not separate personas): each file
73
+ // holds only the facts that size adds beyond the previous one.
129
74
  "human-medium": {
130
75
  kind: "corpus",
131
76
  active: false,
@@ -162,17 +107,8 @@ function builtinExtensions() {
162
107
  corpusPath: join(TIER2_DIR, "general.jsonl"),
163
108
  provenancePrefix: "corpus:tier2-general",
164
109
  },
165
- // corpus/wordnet/generate.mjs's output: a mechanical ConceptNet-shape
166
- // conversion of Open English WordNet's structural relations, not
167
- // hand-curated like the tier-2 bundles above (NOT called "tier-3" —
168
- // corpus/README.md's tiering policy already uses that name for something
169
- // else, runtime-learned facts that are never committed; this bundle is
170
- // curated + committed, tier-2-shaped, just too large to hand-author).
171
- // Named directly "wordnet-xl"/"wordnet-full" so `tmct import --corpus
172
- // wordnet-xl` resolves straight through this BUILTIN_EXTENSIONS lookup,
173
- // the same seam every other recognized name already uses — no change
174
- // needed to bin/tmct.mjs's tier-2-manifest-id resolution path. Shipped
175
- // inactive, like every other opt-in bundle here.
110
+ // corpus/wordnet/generate.mjs's output: a mechanical ConceptNet-shape conversion of
111
+ // Open English WordNet, too large to hand-curate like the tier-2 bundles above.
176
112
  "wordnet-xl": {
177
113
  kind: "corpus",
178
114
  active: false,
@@ -185,12 +121,8 @@ function builtinExtensions() {
185
121
  corpusPath: join(WORDNET_DIR, "wordnet-full.jsonl"),
186
122
  provenancePrefix: "corpus:wordnet-full",
187
123
  },
188
- // corpus/namenet/generate.mjs's output: species/common-name and
189
- // Wikidata-label/WordNet-lemma synonym pairs, mechanically derived from
190
- // three human-reviewed Open English Namenet linking tables. A small,
191
- // explicitly OPTIONAL top-up bundle (not a primary corpus) — same
192
- // BUILTIN_EXTENSIONS seam as wordnet-xl/wordnet-full above, so `tmct
193
- // import --corpus namenet` resolves directly here. Shipped inactive.
124
+ // corpus/namenet/generate.mjs's output: species/common-name and Wikidata/WordNet
125
+ // synonym pairs. A small, optional top-up bundle, not a primary corpus.
194
126
  namenet: {
195
127
  kind: "corpus",
196
128
  active: false,
@@ -202,9 +134,8 @@ function builtinExtensions() {
202
134
 
203
135
  export const BUILTIN_EXTENSIONS = Object.freeze(builtinExtensions());
204
136
 
205
- /** Validate one RESOLVED extension entry — throws a clear, specific error
206
- * naming the offending key. Shared by resolveExtensions (every entry, always
207
- * on) and Part 4's validateExtensionPack (a candidate pack directory). */
137
+ /** Validate one RESOLVED extension entry — throws a clear, specific error naming the
138
+ * offending key. Shared by resolveExtensions and validateExtensionPack. */
208
139
  export function validateExtensionEntry(name, entry) {
209
140
  if (!entry || typeof entry !== "object") {
210
141
  throw new Error(`extension "${name}": entry must be an object`);
@@ -278,20 +209,14 @@ function mergeExtensionEntry(name, builtin, override, repoRoot) {
278
209
  }
279
210
 
280
211
  /**
281
- * Resolve every extension entry a repo carries — the shipped builtins plus
282
- * whatever `tmct.toml`'s `[extensions]`/`[bias]` tables add or override.
283
- * Returns `{ entries, biasByBundle }`:
284
- * - `entries`: Map<name, ResolvedEntry> in FIXED order (seon, conceptnet,
285
- * then the rest sorted by name). EVERY entry is present (active or not)
286
- * callers filter by `.active` themselves (Part 2's corpus loader loop).
287
- * - `biasByBundle`: { bundleName: number } from the flat top-level `[bias]`
288
- * table (default {} — every bundle then ranks at bias 1, see bias.mjs).
289
- * No `tmct.toml` (or one with no `[extensions]`/`[bias]` tables) resolves to
290
- * exactly today's implicit seon+conceptnet default, byte-identical.
212
+ * Resolve every extension entry a repo carries — the shipped builtins plus whatever
213
+ * `tmct.toml`'s `[extensions]`/`[bias]` tables add or override. Returns
214
+ * `{ entries, biasByBundle }`: `entries` is Map<name, ResolvedEntry> in fixed order (every
215
+ * entry present, active or not callers filter by `.active`); `biasByBundle` is
216
+ * { bundleName: number } from the flat `[bias]` table (default {}).
291
217
  *
292
- * `configFile` (optional): an explicit tmct.toml path override `tmct extend
293
- * --validate <dir> --config <path>` read INSTEAD of `<repoRoot>/tmct.toml`;
294
- * `repoRoot` still anchors every resource path (unchanged).
218
+ * `configFile` (optional): an explicit tmct.toml path read instead of
219
+ * `<repoRoot>/tmct.toml`; `repoRoot` still anchors every resource path.
295
220
  */
296
221
  export async function resolveExtensions(repoRoot, { configFile } = {}) {
297
222
  const raw = repoRoot ? await loadTomlConfig(repoRoot, configFile ? { file: configFile } : {}) : null;
@@ -324,25 +249,14 @@ export async function resolveExtensions(repoRoot, { configFile } = {}) {
324
249
 
325
250
  // ---- Part 2: the unified corpus loader loop ---------------------------------
326
251
 
327
- /** Seed every ACTIVE `corpus`/`ontology`-kind entry, plus any ACTIVE `pack`-kind
328
- * entry that declares a `corpusPath` (in the Map's own fixed order seon,
329
- * conceptnet, then the rest sorted by name) into `repo`'s memory, ONE
330
- * seedMemory() call per bundle. Shared by chat.mjs's first-run bootstrap,
331
- * `tmct init`'s seed step and `tmct init --corpus <id>` — so all three read
332
- * the SAME loop instead of three independent hardcoded call sites.
333
- *
334
- * BUGFIX (this batch): a `pack`-kind entry's `corpusPath` used to be silently
335
- * skipped here despite this module's own docblock claiming pack entries
336
- * combine corpus_path/lexicon_path/etc — a pack's corpus facts never made it
337
- * into memory. Fixed by seeding any active pack entry that declares a
338
- * corpusPath, alongside corpus/ontology entries.
252
+ /** Seed every ACTIVE `corpus`/`ontology`-kind entry, plus any ACTIVE `pack`-kind entry
253
+ * that declares a `corpusPath`, into `repo`'s memory, one seedMemory() call per bundle.
254
+ * Shared by chat.mjs's first-run bootstrap, `tmct init`'s seed step, and
255
+ * `tmct init --corpus <id>`.
339
256
  *
340
- * FAILURE-TOLERANT per bundle (init.mjs's own doctrine: a missing/broken
341
- * corpus degrades to "not seeded", never a crash): one bad third-party pack's
342
- * seedMemory throw is CAUGHT and recorded as `perBundle[name].error` logged
343
- * in the structured result rather than silently swallowed — while every
344
- * OTHER bundle still seeds normally. Returns
345
- * `{ appended, skipped, total, perBundle: { name: {appended,skipped,total,error?} } }`. */
257
+ * FAILURE-TOLERANT per bundle: one bad third-party pack's seedMemory throw is caught and
258
+ * recorded as `perBundle[name].error` while every other bundle still seeds normally.
259
+ * Returns `{ appended, skipped, total, perBundle: { name: {appended,skipped,total,error?} } }`. */
346
260
  export async function seedActiveCorpusEntries(repo, entries) {
347
261
  const { seedMemory } = await import("./corpus/conceptnet.mjs");
348
262
  const perBundle = {};
@@ -350,13 +264,6 @@ export async function seedActiveCorpusEntries(repo, entries) {
350
264
  let skipped = 0;
351
265
  let total = 0;
352
266
  for (const [name, entry] of entries instanceof Map ? entries : new Map()) {
353
- // PLAN_SEED.md §2 bug fix: a "pack"-kind entry with its own corpusPath
354
- // combines corpus/lexicon/templates under one active flag (this module's
355
- // own docblock says so) but was previously never actually seeded here —
356
- // only bare `kind: "corpus"` entries were. Broadened, not narrowed: every
357
- // existing `kind: "corpus"`/`kind: "ontology"` entry (seon/conceptnet/
358
- // human/tier2-*) behaves identically to before; only a pack entry that
359
- // DOES carry a corpusPath newly qualifies.
360
267
  if (!entry.active) continue;
361
268
  const seedable = entry.kind === "corpus" || entry.kind === "ontology" || (entry.kind === "pack" && entry.corpusPath);
362
269
  if (!seedable) continue;
@@ -373,8 +280,6 @@ export async function seedActiveCorpusEntries(repo, entries) {
373
280
  skipped += res.skipped;
374
281
  total += res.total;
375
282
  } catch (err) {
376
- // Logged (in the structured result), never silently swallowed — but this
377
- // ONE bundle's failure never aborts the others.
378
283
  perBundle[name] = { appended: 0, skipped: 0, total: 0, error: err && err.message ? err.message : String(err) };
379
284
  }
380
285
  }
@@ -384,16 +289,10 @@ export async function seedActiveCorpusEntries(repo, entries) {
384
289
  // ---- Part 3: lexicon-bundle merge -------------------------------------------
385
290
 
386
291
  /** Merge every ACTIVE `lexicon`/`pack` entry's declared lexicon file into one
387
- * `{nouns, verbs, adjectives, properNames}` object the exact shape
388
- * grammar/lexicon.mjs's `loadLexicon(extra)` already accepts. Bundles merge
389
- * in ASCENDING bias order (lowest first) so `loadLexicon`'s existing "extra
390
- * entries win on conflict" last-write-wins semantics resolve a same-lemma
391
- * collision by BIAS, deterministically, rather than by arbitrary load order —
392
- * a higher-bias bundle's entry always wins. Ties (equal/absent bias) keep the
393
- * entries' `entries` Map iteration order (itself the fixed seon/conceptnet/
394
- * sorted-rest order). Entries with no lexiconPath are skipped. Returns `null`
395
- * when nothing merges (so a caller can pass `undefined` through to
396
- * `loadLexicon` unchanged — the byte-identical no-extension default). */
292
+ * `{nouns, verbs, adjectives, properNames}` object (grammar/lexicon.mjs's `loadLexicon`
293
+ * shape). Bundles merge in ascending bias order so a same-lemma collision resolves by
294
+ * bias, deterministically — a higher-bias bundle's entry always wins. Returns `null`
295
+ * when nothing merges. */
397
296
  export async function mergedLexiconExtra(entries, biasByBundle = {}) {
398
297
  const candidates = [];
399
298
  for (const [name, entry] of entries instanceof Map ? entries : new Map()) {
@@ -425,13 +324,10 @@ export async function mergedLexiconExtra(entries, biasByBundle = {}) {
425
324
 
426
325
  // ---- Part 4: `tmct extend --validate <dir>` ---------------------------------
427
326
 
428
- /** Validate one CANDIDATE extension pack entry against a directory reuses
429
- * the existing throw-loudly primitives (loadSlice/loadMap/toFacts,
430
- * loadLexicon, loadTemplates) rather than inventing new shape-checking logic.
431
- * `candidate` is a resolved-shape entry (see mergeExtensionEntry) whose paths
432
- * are absolute or resolved against `dir`. Returns
433
- * `{ ok, results: [{kind, path, ok, error?, counts?}] }` — never throws;
434
- * every failure is CAUGHT and reported as one `results[]` row. */
327
+ /** Validate one CANDIDATE extension pack entry against a directory, reusing the existing
328
+ * throw-loudly primitives (loadSlice/loadMap/toFacts, loadLexicon, loadTemplates).
329
+ * `candidate` is a resolved-shape entry whose paths are absolute or resolved against
330
+ * `dir`. Returns `{ ok, results: [{kind, path, ok, error?, counts?}] }`; never throws. */
435
331
  export async function validateExtensionPack(dir, candidate) {
436
332
  const results = [];
437
333
  const abs = (p) => (p ? (isAbsolute(p) ? p : resolve(dir, p)) : p);
package/src/finish.mjs CHANGED
@@ -1,25 +1,11 @@
1
- // finish.mjs — Phase 7 response finishing: the segmentation IR seam.
2
- // (archive/PLAN_RESPONSE_FINISHING.md, "The segmentation IR (lever 1)".)
1
+ // finish.mjs — response finishing: the segmentation IR seam.
3
2
  //
4
- // The governing principle is fact-invariance BY CONSTRUCTION. An answer is a
5
- // list of typed spans, [{ type, text }, …], carried alongside the flat string
6
- // (never replacing it). Every type except `prose` is PROTECTED: entities,
7
- // paths, numbers, code, provenance and receipts are byte-copied through
8
- // finishing untouched, and only prose spans are ever handed to a (future)
9
- // grammar-rule engine. Segmentation makes "turn app/lib/a.mjs into an.mjs"
10
- // UNREPRESENTABLE — the protected spans are not in the rule engine's input.
11
- //
12
- // This module is the FOUNDATION step: pure structure, ZERO behaviour change.
13
- // It provides:
14
- // - the segment type vocabulary + the protected/prose split,
15
- // - maskSegments(answer, { graph }) — a conservative masker for the composed
16
- // (non-template) path (the templated path segments in corpus/templates.mjs),
17
- // - an INVARIANCE CHECKER (the protected-span multiset must survive any
18
- // prose-only transform), the property future grammar rules are gated on,
19
- // - a NO-OP finish(result, ctx) — the seam a later wave wires into chat.mjs.
20
- //
21
- // Byte-exact reconstruction is the whole contract here: flatten(segments) ===
22
- // answer for every producer, and finish() returns its input byte-for-byte.
3
+ // Fact-invariance by construction: an answer is a list of typed spans,
4
+ // [{ type, text }, …], carried alongside the flat string. Every type except
5
+ // `prose` is PROTECTED byte-copied through finishing untouched so only
6
+ // prose spans ever reach the grammar-rule engine; a rule can't touch a fact
7
+ // because a fact is never in its input. Byte-exact reconstruction is the
8
+ // whole contract: flatten(segments) === answer for every producer.
23
9
 
24
10
  import { readFileSync } from "node:fs";
25
11
  import { fileURLToPath } from "node:url";
@@ -31,7 +17,7 @@ import { flatten } from "./corpus/templates.mjs";
31
17
  export { flatten };
32
18
 
33
19
  const GRAMMAR_DIR = dirname(fileURLToPath(import.meta.url));
34
- /** The data-driven grammar-rule table (Phase 7, lever 2). */
20
+ /** The data-driven grammar-rule table. */
35
21
  export const GRAMMAR_RULES_FILE = join(GRAMMAR_DIR, "..", "data", "templates", "grammar-rules.toml");
36
22
 
37
23
  /** The segment type vocabulary. `prose` is the only unprotected type. */
@@ -49,15 +35,10 @@ export function isProtected(type) {
49
35
  return type !== "prose";
50
36
  }
51
37
 
52
- // --- Conservative masker for the composed path ------------------------------
53
- // The templated path gets segments almost for free (corpus/templates.mjs
54
- // renderSegments). The composed path (ask engine, plain/conversational turns)
55
- // hands finishing a hand-built flat string; maskSegments walks it and marks
56
- // PROTECTED anything matching one of the patterns below, leaving everything
57
- // else prose. Policy: CONSERVATIVE — when unsure, protect. An un-adopted render
58
- // site simply presents its whole answer as a single prose span (pass no graph
59
- // and match nothing → one prose segment), and the invariance checker still
60
- // guards it. flatten(maskSegments(answer, ctx)) === answer, always.
38
+ // --- Conservative masker for the composed (non-template) path ---------------
39
+ // Walks a hand-built flat answer string and marks PROTECTED anything matching
40
+ // one of the patterns below; everything else is prose. Conservative: when
41
+ // unsure, protect. flatten(maskSegments(answer, ctx)) === answer, always.
61
42
 
62
43
  // Parenthesized receipts: "(traversal: calls edges where object = fnAlpha)" and
63
44
  // the repair receipt 'read as "which modules import a.mjs"'.
@@ -195,16 +176,11 @@ export function assertInvariance(before, after) {
195
176
  return after;
196
177
  }
197
178
 
198
- // --- The grammar-rule engine (lever 2) --------------------------------------
199
- // applyGrammar transforms ONLY the prose spans of a segment list. It NEVER
200
- // regexes the flat answer string and NEVER touches a protected span the
201
- // invariance checker is treated as NECESSARY-BUT-NOT-SUFFICIENT (a token the
202
- // masker failed to protect would sit in a prose span, and a corrupting rule that
203
- // mangled it would pass the multiset check because prose is unchecked). So the
204
- // only defence is that a rule literally cannot receive a protected span: every
205
- // handler below filters `type === "prose"` and byte-copies the rest. Each rule's
206
- // NEUTRAL behaviour is byte-stable; the only byte changes are GENUINE fixes to
207
- // defects tmct itself generates. Rules are chosen to commute → idempotent.
179
+ // --- The grammar-rule engine -------------------------------------------------
180
+ // applyGrammar transforms ONLY the prose spans; every handler below filters
181
+ // `type === "prose"` and byte-copies the rest, since the invariance checker
182
+ // alone can't catch a rule that corrupts a span the masker mis-protected
183
+ // (prose is unchecked). Rules are chosen to commute idempotent.
208
184
 
209
185
  const escapeRe2 = (s) => String(s).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
210
186
 
@@ -247,8 +223,8 @@ function pluralityOf(seg) {
247
223
 
248
224
  // Rule 1 — article selection (a/an). Reads the following word: in-span when the
249
225
  // whole "a word" pair is inside one prose span; across the boundary when the
250
- // prose ends in "a"/"an" and the next span supplies the word (guard #3: it only
251
- // fires when it can read the real next token, else it leaves the article alone).
226
+ // prose ends in "a"/"an" and the next span supplies the word (only fires when
227
+ // it can read the real next token, else it leaves the article alone).
252
228
  function ruleArticle(segments, rule) {
253
229
  const out = segments.map((s) => ({ ...s }));
254
230
  for (let i = 0; i < out.length; i += 1) {
@@ -312,20 +288,10 @@ function ruleAgreement(segments, rule) {
312
288
  return out;
313
289
  }
314
290
 
315
- // Rule 3 — sentence capitalisation. Capitalises (a) the very first character
316
- // when the answer OPENS on a prose span (the ORIGINAL single-answer scope,
317
- // unchanged), (b) every INTERNAL sentence boundary inside a single prose span
318
- // a run of terminal punctuation + whitespace followed by a lowercase letter
319
- // — and (c) a sentence boundary that CROSSES a span boundary: a prose span
320
- // ends in terminal punctuation (+ optional trailing whitespace) and the next
321
- // real-content span is itself prose starting lowercase (any purely-whitespace
322
- // prose spans in between are skipped over). (b) and (c) are the
323
- // PLAN_COMPLETIONS.md Stage 6 generalisation — a genuinely multi-sentence
324
- // completion needs every internal boundary capitalised, not just the whole-
325
- // answer opener. A boundary that lands on a PROTECTED span (path/entity/…) is
326
- // left exactly as grounded — a protected span's casing is never
327
- // rule-transformed, the same guard (a) always applied to an answer that opens
328
- // on one.
291
+ // Rule 3 — sentence capitalisation: (a) the answer-initial letter, (b) every
292
+ // internal sentence boundary within one prose span, (c) a boundary that
293
+ // crosses a span boundary (skipping purely-whitespace spans in between). A
294
+ // boundary landing on a protected span is left alone never rule-transformed.
329
295
  function ruleCapitalise(segments) {
330
296
  if (!segments.length) return segments;
331
297
  const out = segments.map((s) => ({ ...s }));
@@ -380,12 +346,10 @@ function ruleList(segments, rule) {
380
346
 
381
347
  // Rule 5 — terminal punctuation. Collapses ANY run of 2+ sentence stops within
382
348
  // a prose span to a single stop ("done.." → "done."; "Sentence one.. Sentence
383
- // two!!" → "Sentence one. Sentence two!") generalised (PLAN_COMPLETIONS.md
384
- // Stage 6) from the original single-answer scope (only the LAST prose span,
385
- // only a run anchored at the very end of the answer) to every internal
386
- // sentence boundary a multi-sentence completion can carry. A legitimate
387
- // fragment/list answer that ends without a stop is unaffected — there is
388
- // nothing to collapse.
349
+ // two!!" → "Sentence one. Sentence two!"), across every internal sentence
350
+ // boundary a multi-sentence completion can carry, not just the answer's end.
351
+ // A legitimate fragment/list answer that ends without a stop is unaffected
352
+ // there is nothing to collapse.
389
353
  function ruleTerminal(segments, rule) {
390
354
  const stops = (rule.stops && rule.stops.length ? rule.stops : [".", "!", "?"]).map(escapeRe2).join("");
391
355
  const re = new RegExp(`([${stops}])(?:\\s*[${stops}])+(\\s*)`, "g");
@@ -420,7 +384,7 @@ export function grammarRules() {
420
384
 
421
385
  /** Apply the grammar rules to a segment list, in file order, transforming ONLY
422
386
  * prose spans. Asserts fact-invariance at the end (the protected multiset must
423
- * be identical) and returns the transformed segments. Idempotent by design:
387
+ * be identical) and returns the transformed segments. Idempotent:
424
388
  * applyGrammar(applyGrammar(x)) yields the same flattened text as applyGrammar(x). */
425
389
  export function applyGrammar(segments, rules = grammarRules()) {
426
390
  let cur = segments;
@@ -435,37 +399,17 @@ export function applyGrammar(segments, rules = grammarRules()) {
435
399
  }
436
400
 
437
401
  // --- The finish() seam ------------------------------------------------------
438
- // finish() is the LAST transform in a turn:
439
- // 1. take the result's `segments` (attached by a producer) or mask its
440
- // `answer` via maskSegments(result.answer, ctx),
441
- // 2. map ONLY the prose spans through the grammar-rule engine (applyGrammar),
442
- // 3. re-flatten, ASSERTING the invariance checker holds (guard #4: this runs
443
- // in production, not just in tests — a corruption throws, never ships),
444
- // 4. rewrite result.answer (and thus logLines); `via` is unchanged.
445
- //
446
- // NEUTRAL finishing is BYTE-STABLE: when no rule fires, the flattened answer is
447
- // byte-identical to the input and finish() returns its argument by REFERENCE, so
448
- // every byte-exact assertion (test/showcase.test.mjs) stays green untouched. A
449
- // genuine fix (e.g. "a artifact" → "an artifact") rebuilds the result with the
450
- // corrected answer. Idempotent by construction: finish(finish(x)) === finish(x).
451
- //
452
- // INTENDED chat.mjs SEAM (a sibling/later wave wires this, foreign file): in
453
- // runTurn, at the `withLast` seam, `result = finish(result, { graph })` so every
454
- // producer passes through once. Until then finish() is exercised by its unit +
455
- // golden tests; wiring it changes no fact, only fixes our own generated defects.
402
+ // The last transform in a turn: mask (or reuse existing) segments, run
403
+ // applyGrammar over the prose spans, re-flatten, and assert invariance before
404
+ // rewriting result.answer/logLines. Byte-stable when neutral: finish() returns
405
+ // its argument by REFERENCE when no rule fired. Idempotent by construction.
456
406
  //
457
- // ctx.rules (optional, additive): a caller-supplied rule table overriding the
458
- // cached grammarRules() for this call only e.g. completions/complete.mjs's
459
- // Stage 6 pass force-enables the sentence-capitalisation rule (PARKED in the
460
- // live chat table per grammar-rules.toml's own cycle-006 note) because a
461
- // genuinely multi-sentence extractive completion needs every internal sentence
462
- // boundary capitalised to read as one voice, without touching the chat
463
- // pipeline's default live/parked flags. Every EXISTING call site omits
464
- // ctx.rules and is therefore byte-identical to before this option existed.
407
+ // ctx.rules (optional): a caller-supplied rule table overriding the cached
408
+ // grammarRules() for this call only. Every existing call site omits it.
465
409
 
466
410
  /** Finish a turn result: grammar-correct its prose spans, preserving every fact.
467
411
  * Byte-stable when neutral (returns its argument unchanged); rebuilds only on a
468
- * genuine fix. Throws if finishing would move any protected span (guard #4).
412
+ * genuine fix. Throws if finishing would move any protected span.
469
413
  * @param {{rules?: object[]}} [ctx.rules] optional rule-table override (see above) */
470
414
  export function finish(result, ctx = {}) {
471
415
  if (!result || typeof result.answer !== "string") return result;
@@ -1,5 +1,5 @@
1
- // grammar/ace.mjs — tmct's deterministic ACE-OWL sub-fragment parser (ROADMAP
2
- // Phase 2, item 2). Implements the 8 controlled-English sentence patterns of
1
+ // grammar/ace.mjs — tmct's deterministic ACE-OWL sub-fragment parser.
2
+ // Implements the 8 controlled-English sentence patterns of
3
3
  // docs/references/schemas/ace-owl-fragment.md and nothing more: fitting the
4
4
  // grammar is a strong signal, missing it is a FEATURE — parseAce returns null
5
5
  // (or an empty-triples result carrying the unknown words as `residue`) and the
@@ -18,21 +18,13 @@
18
18
  // empty — feeds the pipeline's "if you mean X…" surround).
19
19
  // null the sentence does not fit the fragment at all.
20
20
  //
21
- // Term style: classes/individuals are `tmct:<lexeme>` CURIEs (lexicon lemma
22
- // for nouns, canonical spelling for proper names, the literal token for
23
- // code-shaped references like chat.mjs); predicates are the OWL/RDF(S)
24
- // vocabulary terms or the lexicon verb's tmct:<3sg> predicate
25
- // (lexicon.mjs's predicateOf). Restriction and intersection class
26
- // expressions get READABLE deterministic node names (tmct:some-imports-test,
27
- // tmct:module-that-imports-test) instead of blank nodes, so the same
28
- // sentence always re-emits the same triples and appendFact stays idempotent.
29
- // An intersection is flattened to repeated owl:intersectionOf triples (one
30
- // per member) — the flat-JSON stand-in for an RDF list, documented in
31
- // ontology/tmct-core.ttl. `lexicon.ns` is always "tmct:" here (lexicon.mjs's
32
- // DEFAULT_NS) — every term this module mints is namespaced off `lexicon.ns`
33
- // rather than a hardcoded literal purely so a caller can supply its own
34
- // already-namespaced lexicon (extensions.mjs's mergedLexiconExtra); tmct
35
- // itself only ever runs one namespace.
21
+ // Term style: classes/individuals are `tmct:<lexeme>` CURIEs; predicates are
22
+ // OWL/RDF(S) vocabulary terms or the lexicon verb's tmct:<3sg> predicate
23
+ // (lexicon.mjs's predicateOf). Restriction/intersection nodes get READABLE
24
+ // deterministic names (tmct:some-imports-test) instead of blank nodes, so the
25
+ // same sentence always re-emits the same triples and appendFact stays
26
+ // idempotent; an intersection flattens to repeated owl:intersectionOf triples
27
+ // (documented in ontology/tmct-core.ttl).
36
28
 
37
29
  import {
38
30
  loadLexicon, lookupNoun, lookupVerb, lookupAdjective, lookupProperName,
@@ -190,22 +182,11 @@ function parseRelation(lexicon, toks, lower) {
190
182
  return null;
191
183
  }
192
184
 
193
- // ---- ambiguity: breadth-first candidate parses, dead ends pruned, survivors
194
- // surfaced rather than guessed (the operator's own framing see
195
- // PLAN_DID_YOU_SEE_HER_DUCK.md's Origin section). parseRelation just above is
196
- // UNCHANGED — it is still the greedy, first-verb-position-wins fast path
197
- // every existing caller keeps using, so every single-reading sentence (the
198
- // overwhelming majority) is completely unaffected. parseRelationHits and
199
- // parseAceAmbiguous below are a separate, ADDITIVE scan that a caller opts
200
- // into only when it wants to know whether more than one reading survives. ----
185
+ // ---- ambiguity: an additive scan a caller opts into to see every surviving
186
+ // reading, never displacing parseRelation's own greedy first-match path. ----
201
187
 
202
- /** Pattern 3 — EVERY verb-position split, not just the first: for each token
203
- * index that lookupVerb recognizes, resolve both sides and keep it ONLY if
204
- * it is a complete, valid parse (a genuine hit — a missOrNull/null split is
205
- * a dead end, pruned here rather than surfaced as "ambiguity"). Duplicate
206
- * logic with parseRelation is deliberate: parseRelation must stay byte-for-
207
- * byte unchanged for every existing caller, so this is a standalone reader,
208
- * not a refactor of shared internals. */
188
+ /** Pattern 3 — every verb-position split (not just the first), keeping only
189
+ * complete, valid parses; a dead-end split is pruned, not surfaced. */
209
190
  function parseRelationHits(lexicon, toks, lower) {
210
191
  const hits = [];
211
192
  for (let i = 1; i < toks.length - 1; i += 1) {
@@ -233,14 +214,9 @@ function parseRelationHits(lexicon, toks, lower) {
233
214
  return hits;
234
215
  }
235
216
 
236
- /** Public ambiguity surface: parse `sentence` and, ONLY when more than one
237
- * independent, COMPLETE relation-pattern reading survives (parseRelationHits
238
- * above), return them all, each labeled by the token it read as the verb.
239
- * Returns null for the overwhelming majority of sentences: anything not
240
- * relation-shaped (mirrors parseAce's own dispatch gate exactly, so this
241
- * only ever fires on a sentence parseAce would ALSO route to parseRelation),
242
- * and any relation-shaped sentence with 0 or 1 surviving readings — the
243
- * ordinary parseAce path is authoritative and untouched either way. */
217
+ /** Public ambiguity surface: null unless 2+ independent, complete
218
+ * relation-pattern readings survive, in which case all are returned, each
219
+ * labeled by the token read as its verb. */
244
220
  export function parseAceAmbiguous(sentence, lexicon = loadLexicon()) {
245
221
  const toks = tokenize(sentence);
246
222
  if (toks.length < 4) return null; // 3 tokens: exactly one verb position is even possible
@@ -1,6 +1,6 @@
1
1
  // grammar/assert.mjs — the grammar→memory bridge: parseAce a sentence and land
2
2
  // every emitted triple in tmct's OWN memory graph via memory/core.mjs's
3
- // appendFact (ROADMAP Phase 2 item 2 meeting Phase 1 item 9).
3
+ // appendFact.
4
4
  //
5
5
  // appendFact normalizes each triple's subject/object through normFactTerm
6
6
  // (tmct:Legacy-module → "legacy-module"; the predicate keeps its vocabulary
@@ -1,5 +1,5 @@
1
1
  {
2
- "comment": "tmct's starter lexicon (ROADMAP Phase 2, item 2), now covering BOTH the original software domain and (PLAN_SEED.md, the default 'human-world' persona) everyday people/places/objects/nature/time/body/mind vocabulary. Every word the ACE-OWL sub-fragment parser (src/grammar/ace.mjs) is allowed to understand is DECLARED here — tmct never guesses a word's category. Nouns may declare a possessive property typing ('data' or 'object', pattern 7) or an irregular 'plural' (WordNet-sourced human nouns like 'man'/'child' need this — the morphology fold only handles regular -s/-ies/-es endings); adjectives MUST declare a type ('subclass' forms a class, 'data' asserts a boolean-ish datatype property, pattern 8); verbs may declare a preposition ('depend' + 'on' → dependsOn). The human-domain additions are flat and simple by design (breadth over depth — no gender/kinship taxonomy, no formal role hierarchy), matching corpus/tier2/generate.mjs's CORPUSES.human.lexicon sub-key, which --verify drift-checks against. Extend via loadLexicon(extra) with this same shape.",
2
+ "comment": "tmct's starter lexicon (ROADMAP Phase 2, item 2), now covering BOTH the original software domain and (PLAN_SEED.md, the default 'human-world' persona) everyday people/places/objects/nature/time/body/mind vocabulary. Every word the ACE-OWL sub-fragment parser (src/grammar/ace.mjs) is allowed to understand is DECLARED here — tmct never guesses a word's category. Nouns may declare a possessive property typing ('data' or 'object', pattern 7) or an irregular 'plural' (WordNet-sourced human nouns like 'man'/'child' need this — the morphology fold only handles regular -s/-ies/-es endings); adjectives MUST declare a type ('subclass' forms a class, 'data' asserts a boolean-ish datatype property, pattern 8); verbs may declare a preposition ('depend' + 'on' → dependsOn). The human-domain additions are flat and simple (breadth over depth — no gender/kinship taxonomy, no formal role hierarchy), matching corpus/tier2/generate.mjs's CORPUSES.human.lexicon sub-key, which --verify drift-checks against. Extend via loadLexicon(extra) with this same shape.",
3
3
  "nouns": {
4
4
  "module": {},
5
5
  "class": {},