@polycode-projects/the-mechanical-code-talker 2.3.0 → 2.3.1

package/corpus/LICENSES.json

@@ -101,11 +101,26 @@
       "notice": "corpus/tier2/manifest.json"
     },
     {
-      "path": "corpus/generated/ace-surface-variants.jsonl",
-      "upstream": "mechanically generated from Open English WordNet / SemCor example sentences and this repository's own docs prose",
-      "license": "CC-BY-4.0",
+      "path": "corpus/prose/sqlite/",
+      "upstream": "SQLite documentation (sqlite.org), plain-text extractions of 12 pages",
+      "license": "public-domain",
       "shareAlike": false,
-      "notice": "corpus/generated/manifest.json"
+      "notice": "corpus/prose/sqlite/LICENSE-NOTICE"
+    },
+    {
+      "path": "corpus/prose/wikipedia/",
+      "upstream": "Wikipedia — simple.wikipedia.org lead sections (56 everyday-concept articles) and en.wikipedia.org full articles (12 in tmct's technical domain), fetched via the MediaWiki extracts API",
+      "license": "CC-BY-SA-4.0",
+      "shareAlike": true,
+      "notice": "corpus/prose/wikipedia/LICENSE-NOTICE"
+    },
+    {
+      "path": "corpus/generated/ace-surface-variants.jsonl",
+      "upstream": "mechanically generated from Open English WordNet / SemCor example sentences (CC-BY-4.0) and corpus/prose/, which includes CC-BY-SA-4.0 Wikipedia text",
+      "license": "CC-BY-SA-4.0",
+      "shareAlike": true,
+      "notice": "corpus/generated/manifest.json",
+      "note": "CC-BY-SA-4.0, not CC-BY-4.0: the rescue rows quote and word-substitute Wikipedia sentences from corpus/prose/wikipedia/, so this published file is a modified derivative of CC-BY-SA-4.0 text and inherits its viral share-alike. It carried CC-BY-4.0 while the prose corpus was this repository's own docs; adding Wikipedia is what changed it. See corpus/prose/wikipedia/LICENSE-NOTICE."
     },
     {
       "path": "data/",

package/corpus/README.md

@@ -45,6 +45,9 @@ notice}), guarded by `test/estate/corpus-licences.test.mjs`.
 | `tier2/{aws,python,java,general}.jsonl` | tier-2 SAMPLE corpuses — same fact shape as the tier-1 slice, loadable via the same path (`general` is the one deliberately non-code-domain bundle) | ~4-6 KB each | MPL-2.0 |
 | `tier2/human.jsonl` | the DEFAULT active bundle: the everyday "human-world" persona, Small tier (664 facts) — hand-curated from Open English WordNet, bridged to Schema.org's top-level classes | ~80 KB | MPL-2.0 (hand-authored fact set; not a verbatim WordNet/Schema.org excerpt — see `tier2/generate.mjs`'s own header comment) |
 | `tier2/human-medium.jsonl` / `tier2/human-large.jsonl` | SIZE tiers of the SAME `human` bundle — each holds ONLY the facts that size adds beyond the previous one; both shipped INACTIVE by default, activated via `tmct init --persona-size medium\|large`. Built by `scripts/build-persona-tiers.mjs` from the same WordNet source, automatically curated (sense-ranked, blocklist/denylist-filtered) rather than hand-typed one at a time, given the scale (944 / ~12,000 facts) | ~111 KB / ~1.4 MB | MPL-2.0 (same "hand-authored in homage to the source's shape" basis as `human.jsonl`) |
+| `prose/sqlite/*.txt` | the frozen prose corpus, code half: plain-text extractions of 12 SQLite documentation pages | ~375 KB | **public domain** (see `prose/sqlite/LICENSE-NOTICE`) |
+| `prose/wikipedia/*.txt` | the frozen prose corpus, English half: 56 simple-English lead sections (everyday concepts) + 12 en.wikipedia full articles (NLP/OWL/logic) | ~356 KB | **CC-BY-SA 4.0** (see `prose/wikipedia/LICENSE-NOTICE`) |
+| `prose/manifest.json` | provenance for the above: source URL, byte count and sha256 per file, plus the fetch date | — | MPL-2.0 |
 | `tier2/human-examples.jsonl` / `tier2/human-examples-medium.jsonl` / `tier2/human-examples-large.jsonl` | the example-sentence corpus — real natural-language sentences mapped to the same curated vocabulary, NOT fact triples. Small/Medium tiers are 100% WordNet's own inline `example:` field (same CC-BY-4.0 basis as the fact bundles). Large tier ALSO includes a SemCor-filtered supplement (real Brown Corpus text, re-tagged to modern OEWN senses) for categories where WordNet's own inline coverage is thin (`human-nature` especially — `noun.animal.yaml`'s inline rate is under 1%) | ~11 KB / ~43 KB / ~257 KB | **The WordNet-inline entries**: CC-BY-4.0 (Princeton WordNet + Open English WordNet team), reproduced verbatim (these ARE the source's own example sentences, not a paraphrase). **The `source: "semcor:…"` entries**: CC-BY-4.0 per this SemCor fork's own `LICENSE.md` (a local uncommitted checkout of `globalwordnet/semcor`, real Brown Corpus text re-tagged to modern senses — the original 1960s Brown Corpus permissions aren't independently re-verified beyond that fork's own license statement; proceeding with attribution was an explicit operator decision) — each entry's own `source` field names its origin file (`semcor:<genre>/<file>.yaml`) for exactly this reason |
 
 And alongside (same phase, different directory because it is tmct-original
@@ -55,6 +58,51 @@ data, not a derived corpus):
 | `../data/templates/responses.jsonl` | ~56 response templates ({id, class, template, register}) | MPL-2.0 |
 | `../data/phrasebook/software-phrases.txt` | ~170 SE phrase patterns + 31 synonym families | MPL-2.0 |
 
+## The prose corpus (`prose/`) — external and frozen, on purpose
+
+`prose/` is plain English text. It is not seeded and holds no facts. It exists
+to be *measured against*: `scripts/template-coverage.mjs` asks how much of it
+tmct's ACE grammar parses, and the rescue pass in
+`scripts/generate-template-variants.mjs` mines it for near-misses.
+
+It used to be this repository's own root `*.md` docs, and both reasons it moved
+are worth stating, because both were real bugs:
+
+- **A doc edit could drift a shipped artifact.** The corpus was globbed from
+  every root `*.md`, so editing a README or a plan changed
+  `generated/ace-surface-variants.jsonl` — a file npm ships. A rescue row could
+  be, and once was, a sentence someone had written into a plan that morning.
+  Committed text fetched from a recorded URL cannot do that.
+- **The coverage metric was not comparable across versions.** A hit rate only
+  means something against a fixed corpus. When the corpus moves with every doc
+  edit, two versions' numbers are not measuring the same thing, and a change in
+  the number says nothing about a change in the grammar.
+
+So the corpus is external (nothing in this repo can edit it), frozen (a
+snapshot, checksummed per file in `prose/manifest.json`), and re-fetched only
+when someone deliberately runs `npm run gen:prose-corpus`. `test/estate/prose-corpus.test.mjs`
+checks the committed bytes against the manifest, so a hand-edit fails loudly.
+
+**Licensing picked the sources, not preference.** The rescue pass substitutes
+words and commits the result to a file npm publishes, so this corpus is
+republished as a *modified derivative*. Only sources that permit derivatives
+qualify. IETF RFCs and W3C specifications — including the OWL 2 Primer, the
+best domain match there is — permit redistribution but **not** modification, so
+they are unusable here and must not be added. SQLite's documentation is public
+domain; Wikipedia is CC BY-SA 4.0. Adding Wikipedia is what makes
+`generated/ace-surface-variants.jsonl` CC-BY-SA-4.0 rather than CC-BY-4.0:
+share-alike is viral, and it reaches the generated file.
+
+**Why simple-English lead sections for the everyday half.** Measured, on the
+metric the rescue pass actually consumes (sentences with exactly one undeclared
+word): simple.wikipedia.org lead sections yield ~0.33 rescue candidates per KB
+against ~0.12 for technical prose. Lead sections are dense definitional English
+("A penguin is a bird that cannot fly"), which is the shape the ACE grammar was
+built to parse; later article sections drift into history and citations. The
+titles cover the same everyday concepts the persona clumps model
+(`scripts/build-persona-tiers.mjs`'s `CLUMP_FILES`). The technical half stays
+because tmct's own domain is code.
+
 ## How seeding works
 
 `src/adapters/corpus/conceptnet.mjs` turns the slice into tmct memory facts:

package/corpus/generated/README.md

@@ -17,7 +17,7 @@ measurement: `node scripts/template-coverage.mjs` (baseline) and
 
 | `kind` | What | Fields |
 |---|---|---|
-| `rescue` | A real docs-corpus sentence that almost fit the ACE grammar (exactly one undeclared word), rescued by substituting a WordNet synonym of that word that's ALSO already declared in `src/domain/grammar/lexicon-core.json` | `sentence`, `rescued`, `from`, `to`, `pos`, `synsetId`, `sourceFile` |
+| `rescue` | A real sentence from the frozen prose corpus (`corpus/prose/`) that almost fit the ACE grammar (exactly one undeclared word), rescued by substituting a WordNet synonym of that word that's ALSO already declared in `src/domain/grammar/lexicon-core.json` | `sentence`, `rescued`, `from`, `to`, `pos`, `synsetId`, `sourceFile` |
 | `variant` | A real WordNet/SemCor example sentence (`corpus/tier2/human-examples*.jsonl`) that already hits `parseAce`, with one content word swapped for a same-synset sibling (both ends independently declared in tmct's own lexicon) | `seed`, `generated`, `from`, `to`, `pos`, `synsetId`, `sourceCorpus` |
 | `alt-phrasing` | A possessive-pattern (#7) hit rewritten in the ACE grammar's OTHER declared surface form for the same triple (`"X's Y is Z"` <-> `"the Y of X is Z"`, both routed through `buildPossessive` in `src/domain/grammar/ace.mjs`) | `seed`, `generated`, `pattern`, `form` |
 
@@ -31,12 +31,27 @@ product code — it is committed raw material, verified by
 `scripts/template-coverage.mjs`. Wiring it into live answer rendering is a
 separate, future phase.
 
-## Licence
+## Licence — CC-BY-SA-4.0
 
-Open English WordNet content (synset `members`/`example` fields) is
-CC-BY-4.0; SemCor-derived sentences (`corpus/tier2/human-examples-large.jsonl`)
-are CC-BY-4.0 per that fork's own `LICENSE.md` (same basis `corpus/README.md`
-already documents for that file). A `variant`/`rescue` row is a one-word
-derivative of that material — CC-BY-4.0. An `alt-phrasing` row is a pure
-grammar-level rephrasing of the same source sentence — CC-BY-4.0 also, no new
-creative content introduced either way.
+**`ace-surface-variants.jsonl` is CC-BY-SA-4.0**, and this file ships in the npm
+package. It is not under this repository's MPL-2.0.
+
+Two source licences meet in this file:
+
+- Open English WordNet content (synset `members`/`example` fields) is CC-BY-4.0;
+  SemCor-derived sentences (`corpus/tier2/human-examples-large.jsonl`) are
+  CC-BY-4.0 per that fork's own `LICENSE.md` (the same basis `corpus/README.md`
+  already documents for that file). `variant` and `alt-phrasing` rows derive
+  from this material.
+- `corpus/prose/` feeds the `rescue` rows, and it includes Wikipedia text under
+  **CC-BY-SA-4.0** (`corpus/prose/wikipedia/LICENSE-NOTICE`). A `rescue` row
+  quotes such a sentence and substitutes one word into it, so the row is a
+  published, modified derivative of CC-BY-SA-4.0 text.
+
+Share-alike is viral, so the combined file takes the stricter licence:
+**CC-BY-SA-4.0**. Redistributing it, modified or not, means doing so under
+CC-BY-SA 4.0 with the attribution in `corpus/prose/wikipedia/LICENSE-NOTICE`.
+
+This file was labelled CC-BY-4.0 while the rescue corpus was this repository's
+own MPL-2.0 docs. Repointing the corpus at Wikipedia is what changed it.
+`corpus/prose/sqlite/` is public domain and adds no condition of its own.

package/corpus/generated/ace-surface-variants.jsonl

@@ -1,4 +1,7 @@
-{"kind":"rescue","sentence":"Code search is a retrieval task","rescued":"Code hunt is a retrieval task","from":"search","to":"hunt","pos":"noun","synsetId":"00947217-n","sourceFile":"PLAN_EMBEDDINGS.md","provenance":"wordnet:00947217-n"}
+{"kind":"rescue","sentence":"The claim of the previous paragraph is extensively checked in the SQLite regression test suite using a special test harness that simulates the effects on a database file of operating system crashes and power failures","rescued":"The claim of the old paragraph is extensively checked in the SQLite regression test suite using a special test harness that simulates the effects on a database file of operating system crashes and power failures","from":"previous","to":"old","pos":"adjective","synsetId":"00128009-s","sourceFile":"sqlite/transactional.txt","provenance":"wordnet:00128009-s"}
+{"kind":"rescue","sentence":"The meaning of sentences is described as a small set of interpretation rules","rescued":"The significance of sentences is described as a small set of interpretation rules","from":"meaning","to":"significance","pos":"noun","synsetId":"06613680-n","sourceFile":"wikipedia/Attempto_Controlled_English.txt","provenance":"wordnet:06613680-n"}
+{"kind":"rescue","sentence":"The difficulty of planning is dependent on the simplifying assumptions employed","rescued":"The trouble of planning is dependent on the simplifying assumptions employed","from":"difficulty","to":"trouble","pos":"noun","synsetId":"00625102-n","sourceFile":"wikipedia/Automated_planning_and_scheduling.txt","provenance":"wordnet:00625102-n"}
+{"kind":"rescue","sentence":"The consumption of food is normally enjoyable to humans","rescued":"The intake of food is normally enjoyable to humans","from":"consumption","to":"intake","pos":"noun","synsetId":"00839759-n","sourceFile":"wikipedia/Food.txt","provenance":"wordnet:00839759-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the one of trumps is a sure winner","from":"ace","to":"one","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the single of trumps is a sure winner","from":"ace","to":"single","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}
 {"kind":"variant","seed":"the ace of trumps is a sure winner","generated":"the unity of trumps is a sure winner","from":"ace","to":"unity","pos":"noun","synsetId":"13764713-n","sourceCorpus":"corpus/tier2/human-examples-large.jsonl","provenance":"wordnet:13764713-n"}

package/corpus/generated/manifest.json

@@ -2,8 +2,8 @@
   "version": 1,
   "generated": "by scripts/generate-template-variants.mjs",
   "file": "ace-surface-variants.jsonl",
-  "rows": 18,
-  "bytes": 5012,
-  "sha256": "28fae5a3353f5dd1df365b4407e0ac2a5c66b98b232d8bf454fe3573f2c16760",
-  "license": "CC-BY-4.0 (WordNet-derived synonym substitutions of Open English WordNet / SemCor example sentences and this repo's own MPL-2.0 docs prose — see corpus/generated/README.md)"
+  "rows": 21,
+  "bytes": 6417,
+  "sha256": "6e1daca10e43455608e5ee91ca41e20c5a1c5b0127f14ffd087c85580fe1f0ad",
+  "license": "CC-BY-SA-4.0 (WordNet-derived synonym substitutions of Open English WordNet / SemCor example sentences (CC-BY-4.0) and corpus/prose/, which includes CC-BY-SA-4.0 Wikipedia text. Share-alike is viral and the rescue rows quote and modify that text, so the combined file is CC-BY-SA-4.0 — the stricter of the two. See corpus/prose/wikipedia/LICENSE-NOTICE and corpus/generated/README.md)"
 }