@polycode-projects/the-mechanical-code-talker 3.0.5 → 3.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -6
- package/corpus/child/README.md +2 -2
- package/corpus/child/index.json.gz +0 -0
- package/corpus/child/manifest.json +74 -74
- package/corpus/child/shards/child-00.jsonl.gz +0 -0
- package/corpus/child/shards/child-01.jsonl.gz +0 -0
- package/corpus/child/shards/child-02.jsonl.gz +0 -0
- package/corpus/child/shards/child-03.jsonl.gz +0 -0
- package/corpus/child/shards/child-04.jsonl.gz +0 -0
- package/corpus/child/shards/child-05.jsonl.gz +0 -0
- package/corpus/child/shards/child-06.jsonl.gz +0 -0
- package/corpus/child/shards/child-07.jsonl.gz +0 -0
- package/corpus/child/shards/child-08.jsonl.gz +0 -0
- package/corpus/child/shards/child-09.jsonl.gz +0 -0
- package/corpus/child/shards/child-0a.jsonl.gz +0 -0
- package/corpus/child/shards/child-0b.jsonl.gz +0 -0
- package/corpus/child/shards/child-0c.jsonl.gz +0 -0
- package/corpus/child/shards/child-0d.jsonl.gz +0 -0
- package/corpus/child/shards/child-0e.jsonl.gz +0 -0
- package/corpus/child/shards/child-0f.jsonl.gz +0 -0
- package/corpus/child/shards/child-10.jsonl.gz +0 -0
- package/corpus/child/shards/child-11.jsonl.gz +0 -0
- package/corpus/child/shards/child-12.jsonl.gz +0 -0
- package/corpus/child/shards/child-13.jsonl.gz +0 -0
- package/corpus/child/shards/child-14.jsonl.gz +0 -0
- package/corpus/child/shards/child-15.jsonl.gz +0 -0
- package/corpus/child/shards/child-16.jsonl.gz +0 -0
- package/corpus/child/shards/child-17.jsonl.gz +0 -0
- package/corpus/child/shards/child-18.jsonl.gz +0 -0
- package/corpus/child/shards/child-19.jsonl.gz +0 -0
- package/corpus/child/shards/child-1a.jsonl.gz +0 -0
- package/corpus/child/shards/child-1b.jsonl.gz +0 -0
- package/corpus/child/shards/child-1c.jsonl.gz +0 -0
- package/corpus/child/shards/child-1d.jsonl.gz +0 -0
- package/corpus/child/shards/child-1e.jsonl.gz +0 -0
- package/corpus/child/shards/child-1f.jsonl.gz +0 -0
- package/corpus/conceptnet/quality-filter.mjs +28 -4
- package/corpus/tier2/generate.mjs +4 -8
- package/corpus/tier2/human-large.jsonl +0 -2
- package/corpus/tier2/human-medium.jsonl +0 -2
- package/corpus/tier2/manifest.json +6 -6
- package/data/templates/constructions/digest-sentence-structures.toml +146 -1
- package/package.json +1 -1
- package/src/adapters/corpus/digest-bank.mjs +15 -3
- package/src/domain/digest/compose.mjs +142 -10
- package/src/domain/digest/config.json +5 -0
- package/src/domain/digest/structures.mjs +39 -14
- package/src/services/chat-page-viz.mjs +15 -4
- package/src/services/chat.mjs +118 -49
- package/src/services/extract-facts.mjs +46 -4
- package/src/services/research.mjs +80 -3
- package/src/surfaces/web/chat-browser-entry.mjs +23 -3
- package/src/surfaces/web/memory-ask-browser.bundle.js +108 -108
package/README.md
CHANGED
|
@@ -1185,12 +1185,12 @@ The full tables, judge scores, and transcripts are in the linked write-ups.
|
|
|
1185
1185
|
|
|
1186
1186
|
| What it does | Result (2.7.12) | Read the number with this | Method |
|
|
1187
1187
|
|---|---|---|---|
|
|
1188
|
-
| Multi-hop entailment | 379/379 chat cases and 100/100 kernel cases, 0% fabrication, all bands pass | The case set is unchanged from 2.6.0 (same templates, same counts). The one real move this cycle is INF-4's ceiling-graded count dropping 35→30: five cases that now pass as genuine capability instead of against the declared honest-miss floor. | `BENCHMARK_INFERENCE_2.7.12.md` |
|
|
1189
|
-
| Tool-call planning | 68/68 cases, 100% plan-completion, 100% result-completion, 0% hallucination, every rung A0→C2 | Goal driver. 2.6.0 gated at TOOL-7 (62/66, 94%). This cycle's router uplift (a guarded RECOVER step, a tied-candidate composer) cleared it: a real capability move, not a ruler change. | `BENCHMARK_AGENT_2.7.12.md` |
|
|
1190
|
-
| Groundedness | Every answer carries a source, and an empty graph reports itself empty. Judge-scored mean 1.809/2 over 138 cases, 5 hard fails, 136/138 tier-1. | Judged (`claude-haiku-4-5-20251001`, `judge-prompt-v2`) at N=1. The judge prompt moved v1→v2 since 2.6.0, so this is a measurement, not a clean lever comparison against the prior cycle. The judge runs in the offline eval harness, never in the product. | `BENCHMARK_CEFR_ENGLISH_2.7.12.md` |
|
|
1191
|
-
| Abstention (the honest miss) | 0% fabrication across 479 inference rows (379 chat + 100 kernel) and 0% hallucination across 272 agent rows | Structural, not a tuned threshold. tmct abstains because nothing matched, so the rows test a property of a no-model design rather than a score. | `BENCHMARK_INFERENCE_2.7.12.md`, `BENCHMARK_AGENT_2.7.12.md` |
|
|
1192
|
-
| Determinism | Byte-identical on rerun: a 379-case `--replay` clean across 2 runs, no LLM, no network, $0 per turn | A property of the no-model pipeline. | `BENCHMARK_INFERENCE_2.7.12.md` |
|
|
1193
|
-
| Dialogue robustness (persona sweep) | A 6-persona sweep (textbook logician, casual newcomer, new developer, adversarial sceptic, returning user, planning user) fixed 25 of the prior cycle's 29 routed findings (21 clean, 4 with a residual noted); 4 remain broken, 2 in a shape distinct from the original complaint | Free exploration across all six personas surfaced roughly 60 fresh findings beyond the ratchet check. The single highest-signal pattern: tmct's own suggested repair text was itself frequently broken when followed verbatim (since fixed, see `NEXT.md`). | `BENCHMARK_CONVERSATION_2.7.11.md` |
|
|
1188
|
+
| Multi-hop entailment | 379/379 chat cases and 100/100 kernel cases, 0% fabrication, all bands pass | The case set is unchanged from 2.6.0 (same templates, same counts). The one real move this cycle is INF-4's ceiling-graded count dropping 35→30: five cases that now pass as genuine capability instead of against the declared honest-miss floor. | `archive/BENCHMARK_INFERENCE_2.7.12.md` |
|
|
1189
|
+
| Tool-call planning | 68/68 cases, 100% plan-completion, 100% result-completion, 0% hallucination, every rung A0→C2 | Goal driver. 2.6.0 gated at TOOL-7 (62/66, 94%). This cycle's router uplift (a guarded RECOVER step, a tied-candidate composer) cleared it: a real capability move, not a ruler change. | `archive/BENCHMARK_AGENT_2.7.12.md` |
|
|
1190
|
+
| Groundedness | Every answer carries a source, and an empty graph reports itself empty. Judge-scored mean 1.809/2 over 138 cases, 5 hard fails, 136/138 tier-1. | Judged (`claude-haiku-4-5-20251001`, `judge-prompt-v2`) at N=1. The judge prompt moved v1→v2 since 2.6.0, so this is a measurement, not a clean lever comparison against the prior cycle. The judge runs in the offline eval harness, never in the product. | `archive/BENCHMARK_CEFR_ENGLISH_2.7.12.md` |
|
|
1191
|
+
| Abstention (the honest miss) | 0% fabrication across 479 inference rows (379 chat + 100 kernel) and 0% hallucination across 272 agent rows | Structural, not a tuned threshold. tmct abstains because nothing matched, so the rows test a property of a no-model design rather than a score. | `archive/BENCHMARK_INFERENCE_2.7.12.md`, `archive/BENCHMARK_AGENT_2.7.12.md` |
|
|
1192
|
+
| Determinism | Byte-identical on rerun: a 379-case `--replay` clean across 2 runs, no LLM, no network, $0 per turn | A property of the no-model pipeline. | `archive/BENCHMARK_INFERENCE_2.7.12.md` |
|
|
1193
|
+
| Dialogue robustness (persona sweep) | A 6-persona sweep (textbook logician, casual newcomer, new developer, adversarial sceptic, returning user, planning user) fixed 25 of the prior cycle's 29 routed findings (21 clean, 4 with a residual noted); 4 remain broken, 2 in a shape distinct from the original complaint | Free exploration across all six personas surfaced roughly 60 fresh findings beyond the ratchet check. The single highest-signal pattern: tmct's own suggested repair text was itself frequently broken when followed verbatim (since fixed, see `NEXT.md`). | `archive/BENCHMARK_CONVERSATION_2.7.11.md` |
|
|
1194
1194
|
|
|
1195
1195
|
Three offline benchmark rigs live in a clone (they are not in the npm
|
|
1196
1196
|
package). Each replays a committed case set through the real product and
|
package/corpus/child/README.md
CHANGED
|
@@ -20,7 +20,7 @@ Layout (mirrors corpus/reference/):
|
|
|
20
20
|
budgets, the acceptance metrics, and a sha256 for every emitted file.
|
|
21
21
|
- `LICENSE-NOTICE` — CC-BY-SA 4.0; read it before reusing these files.
|
|
22
22
|
|
|
23
|
-
|
|
23
|
+
46883 terms, 92638 keyed facts. Built 2026-07-24.
|
|
24
24
|
|
|
25
25
|
Predicates are already mapped into tmct's vocabulary
|
|
26
26
|
(conceptnet-map.toml): `rdfs:subClassOf`, `mgx:capableOf`,
|
|
@@ -63,7 +63,7 @@ acceptance test — the plan's hand-counted baseline drifted once). Numbers belo
|
|
|
63
63
|
are for this committed pack:
|
|
64
64
|
|
|
65
65
|
```
|
|
66
|
-
Measured over this pack:
|
|
66
|
+
Measured over this pack: 69140 facts
|
|
67
67
|
kinds of bird 1881 [abbott's booby, abyssinian woodpecker, acadian flycatcher, accipiter, acorn woodpecker, acridotheres, acrocephalus, actitis, adélie penguin, african black duck, african cuckoo hawk, african finfoot, african fish eagle, african marsh harrier, african penguin, african pygmy goose, african sacred ibis, african skimmer, aix, akiapola au]
|
|
68
68
|
capabilities on birds 71 [bird can attempt to fly, chicken can become food, bird can build nest, bird can call with chirps, bird can chirp, bird can eat grasshopper, bird can experience flight, bird can eye fish, bird can fall from sky, bird can fly, bird can fly high, bird can head south, bird can kill fish, bird can land on beach, bird can land on branch, bird can land on tree, bird can learn to fly, bird can listen to insect, bird can person's pet, bird can pet]
|
|
69
69
|
things that can fly 39 [animals, ants, bat, bats, bats and birds, bee, beetles, bird, butterfly, doves, dragon, dragonfly, flies, frisbee, geese, gnat, goose, helicoptors, hoatzins, insects]
|
|
Binary file
|
|
@@ -14,30 +14,30 @@
|
|
|
14
14
|
"version": "5.7.0",
|
|
15
15
|
"sha256": "accd65fe94038584295574ddc26e1500c1919c8c4532bf771811cafd0948af7e"
|
|
16
16
|
},
|
|
17
|
-
"built": "2026-07-
|
|
17
|
+
"built": "2026-07-24",
|
|
18
18
|
"pack": "conceptnet",
|
|
19
19
|
"shardCount": 32,
|
|
20
20
|
"counts": {
|
|
21
|
-
"terms":
|
|
22
|
-
"facts":
|
|
21
|
+
"terms": 46883,
|
|
22
|
+
"facts": 92638,
|
|
23
23
|
"shards": 32
|
|
24
24
|
},
|
|
25
25
|
"budgets": {
|
|
26
26
|
"facts": {
|
|
27
|
-
"used":
|
|
27
|
+
"used": 92638,
|
|
28
28
|
"max": 120000
|
|
29
29
|
},
|
|
30
30
|
"shardsGzBytes": {
|
|
31
|
-
"used":
|
|
31
|
+
"used": 1097302,
|
|
32
32
|
"max": 3145728
|
|
33
33
|
},
|
|
34
34
|
"indexGzBytes": {
|
|
35
|
-
"used":
|
|
35
|
+
"used": 463262,
|
|
36
36
|
"max": 716800
|
|
37
37
|
}
|
|
38
38
|
},
|
|
39
39
|
"acceptance": {
|
|
40
|
-
"totalFacts":
|
|
40
|
+
"totalFacts": 69140,
|
|
41
41
|
"kindsOfBird": {
|
|
42
42
|
"count": 1881,
|
|
43
43
|
"sample": [
|
|
@@ -136,8 +136,8 @@
|
|
|
136
136
|
"files": [
|
|
137
137
|
{
|
|
138
138
|
"file": "index.json.gz",
|
|
139
|
-
"bytes":
|
|
140
|
-
"sha256": "
|
|
139
|
+
"bytes": 463262,
|
|
140
|
+
"sha256": "cfe726d696e7a3bd37a33171943b4cbee79f6be38441f0ba8425b8e303c97076"
|
|
141
141
|
},
|
|
142
142
|
{
|
|
143
143
|
"file": "LICENSE-NOTICE",
|
|
@@ -147,167 +147,167 @@
|
|
|
147
147
|
{
|
|
148
148
|
"file": "README.md",
|
|
149
149
|
"bytes": 4728,
|
|
150
|
-
"sha256": "
|
|
150
|
+
"sha256": "b4a0916b8315b2276df7dc0c612136e241c8e24b3f4debbdbbce5ae4082a6de2"
|
|
151
151
|
},
|
|
152
152
|
{
|
|
153
153
|
"file": "shards/child-00.jsonl.gz",
|
|
154
|
-
"bytes":
|
|
155
|
-
"sha256": "
|
|
154
|
+
"bytes": 31885,
|
|
155
|
+
"sha256": "8fa0c70c391a7cd06262358fe32d52ac35d9e8ef40b976373f1b341cecf19758"
|
|
156
156
|
},
|
|
157
157
|
{
|
|
158
158
|
"file": "shards/child-01.jsonl.gz",
|
|
159
|
-
"bytes":
|
|
160
|
-
"sha256": "
|
|
159
|
+
"bytes": 33446,
|
|
160
|
+
"sha256": "7f33ec49bb0d1a33b1ed4e44d66609a72b88af16d48f7ccb354e2936af042081"
|
|
161
161
|
},
|
|
162
162
|
{
|
|
163
163
|
"file": "shards/child-02.jsonl.gz",
|
|
164
|
-
"bytes":
|
|
165
|
-
"sha256": "
|
|
164
|
+
"bytes": 39355,
|
|
165
|
+
"sha256": "56d731fad9aabc630c7d8794424513f99f4abbde84580f24a0aad6cbd6d57a73"
|
|
166
166
|
},
|
|
167
167
|
{
|
|
168
168
|
"file": "shards/child-03.jsonl.gz",
|
|
169
|
-
"bytes":
|
|
170
|
-
"sha256": "
|
|
169
|
+
"bytes": 31745,
|
|
170
|
+
"sha256": "b14ddc87a2fe7434701e8ada94673d9b20b2c0ccceb00e49168a33f198cadfce"
|
|
171
171
|
},
|
|
172
172
|
{
|
|
173
173
|
"file": "shards/child-04.jsonl.gz",
|
|
174
|
-
"bytes":
|
|
175
|
-
"sha256": "
|
|
174
|
+
"bytes": 33202,
|
|
175
|
+
"sha256": "7890d1a63601132a9fd1054a8b088065bd41d9e2a26ab908582b1858beae0c36"
|
|
176
176
|
},
|
|
177
177
|
{
|
|
178
178
|
"file": "shards/child-05.jsonl.gz",
|
|
179
|
-
"bytes":
|
|
180
|
-
"sha256": "
|
|
179
|
+
"bytes": 34872,
|
|
180
|
+
"sha256": "0bb756c6e78d598a31fe7c57385785c26f7f7ddbab16f3a5143f715cfff76fb1"
|
|
181
181
|
},
|
|
182
182
|
{
|
|
183
183
|
"file": "shards/child-06.jsonl.gz",
|
|
184
|
-
"bytes":
|
|
185
|
-
"sha256": "
|
|
184
|
+
"bytes": 36268,
|
|
185
|
+
"sha256": "bd150f3d13b0519cb97155657bd5edac4860b338d988db7c05c74e2d6b9dc440"
|
|
186
186
|
},
|
|
187
187
|
{
|
|
188
188
|
"file": "shards/child-07.jsonl.gz",
|
|
189
|
-
"bytes":
|
|
190
|
-
"sha256": "
|
|
189
|
+
"bytes": 33641,
|
|
190
|
+
"sha256": "797a5dd7a575c633ebae145b6811750b401f527e558b034d84761e9e8fa08df8"
|
|
191
191
|
},
|
|
192
192
|
{
|
|
193
193
|
"file": "shards/child-08.jsonl.gz",
|
|
194
|
-
"bytes":
|
|
195
|
-
"sha256": "
|
|
194
|
+
"bytes": 34071,
|
|
195
|
+
"sha256": "2c39d8f05e5246a26efe84a479484f992a49eb2125d82ad01ad94c49e2a9a573"
|
|
196
196
|
},
|
|
197
197
|
{
|
|
198
198
|
"file": "shards/child-09.jsonl.gz",
|
|
199
|
-
"bytes":
|
|
200
|
-
"sha256": "
|
|
199
|
+
"bytes": 35865,
|
|
200
|
+
"sha256": "9d8363f3c04cfe80643b2c1e1cd4222544c8e80c347ea0042789df7b752287f8"
|
|
201
201
|
},
|
|
202
202
|
{
|
|
203
203
|
"file": "shards/child-0a.jsonl.gz",
|
|
204
|
-
"bytes":
|
|
205
|
-
"sha256": "
|
|
204
|
+
"bytes": 34618,
|
|
205
|
+
"sha256": "17a5cd9946fc6a19f69e51277f9486c92c8d43e3953439580adf71e90736376d"
|
|
206
206
|
},
|
|
207
207
|
{
|
|
208
208
|
"file": "shards/child-0b.jsonl.gz",
|
|
209
|
-
"bytes":
|
|
210
|
-
"sha256": "
|
|
209
|
+
"bytes": 31978,
|
|
210
|
+
"sha256": "f447a23e7e3191f5247a0d2cbfc187ba707f347d4fca01c4fedd3ef6446e1996"
|
|
211
211
|
},
|
|
212
212
|
{
|
|
213
213
|
"file": "shards/child-0c.jsonl.gz",
|
|
214
|
-
"bytes":
|
|
215
|
-
"sha256": "
|
|
214
|
+
"bytes": 32540,
|
|
215
|
+
"sha256": "d333ca1ca7167fa22f8af37fd0a493e735ff5b993f76d8f8d975c1f9bf2ef188"
|
|
216
216
|
},
|
|
217
217
|
{
|
|
218
218
|
"file": "shards/child-0d.jsonl.gz",
|
|
219
|
-
"bytes":
|
|
220
|
-
"sha256": "
|
|
219
|
+
"bytes": 32826,
|
|
220
|
+
"sha256": "f52920d032e74322750dd6420e6dc8cea57a023a515b81b7f77ade63e25e952d"
|
|
221
221
|
},
|
|
222
222
|
{
|
|
223
223
|
"file": "shards/child-0e.jsonl.gz",
|
|
224
|
-
"bytes":
|
|
225
|
-
"sha256": "
|
|
224
|
+
"bytes": 37216,
|
|
225
|
+
"sha256": "55688d2f5233345816403611cd9e9e9990a5566b1df08c061529bfa8fcf89575"
|
|
226
226
|
},
|
|
227
227
|
{
|
|
228
228
|
"file": "shards/child-0f.jsonl.gz",
|
|
229
|
-
"bytes":
|
|
230
|
-
"sha256": "
|
|
229
|
+
"bytes": 32817,
|
|
230
|
+
"sha256": "f767faeb048dab56a018e1a547a8063b1bd907b256b8110f56b6da3b58c3a5a6"
|
|
231
231
|
},
|
|
232
232
|
{
|
|
233
233
|
"file": "shards/child-10.jsonl.gz",
|
|
234
|
-
"bytes":
|
|
235
|
-
"sha256": "
|
|
234
|
+
"bytes": 36194,
|
|
235
|
+
"sha256": "bd90001e185c2b84239d7362ad7765593c7f94fb422f60a35a86bb9330903f12"
|
|
236
236
|
},
|
|
237
237
|
{
|
|
238
238
|
"file": "shards/child-11.jsonl.gz",
|
|
239
|
-
"bytes":
|
|
240
|
-
"sha256": "
|
|
239
|
+
"bytes": 36842,
|
|
240
|
+
"sha256": "f5a603e3290b29a463af799258838ec692933db492a0d372dd2dab32bbdc3868"
|
|
241
241
|
},
|
|
242
242
|
{
|
|
243
243
|
"file": "shards/child-12.jsonl.gz",
|
|
244
|
-
"bytes":
|
|
245
|
-
"sha256": "
|
|
244
|
+
"bytes": 34716,
|
|
245
|
+
"sha256": "daba6bd6859ded6d016418a01fe89699803b87d5f1629cd2ebca5e7fc97cb3f4"
|
|
246
246
|
},
|
|
247
247
|
{
|
|
248
248
|
"file": "shards/child-13.jsonl.gz",
|
|
249
|
-
"bytes":
|
|
250
|
-
"sha256": "
|
|
249
|
+
"bytes": 35643,
|
|
250
|
+
"sha256": "914501ada0aa3cdd959603ec095597bed731b649e5c26477f390b3db4f8db93f"
|
|
251
251
|
},
|
|
252
252
|
{
|
|
253
253
|
"file": "shards/child-14.jsonl.gz",
|
|
254
|
-
"bytes":
|
|
255
|
-
"sha256": "
|
|
254
|
+
"bytes": 34171,
|
|
255
|
+
"sha256": "343285630b15e1b5a221dcc23770d2430c366ef320429f6ff38ef64e9c1b338b"
|
|
256
256
|
},
|
|
257
257
|
{
|
|
258
258
|
"file": "shards/child-15.jsonl.gz",
|
|
259
|
-
"bytes":
|
|
260
|
-
"sha256": "
|
|
259
|
+
"bytes": 34878,
|
|
260
|
+
"sha256": "8541b35ca94f56f49c58abcf99dbeaff5a576aaba11802fb9851239daee07b2f"
|
|
261
261
|
},
|
|
262
262
|
{
|
|
263
263
|
"file": "shards/child-16.jsonl.gz",
|
|
264
|
-
"bytes":
|
|
265
|
-
"sha256": "
|
|
264
|
+
"bytes": 36348,
|
|
265
|
+
"sha256": "54139274ab7d073126f9bec37cc02c67fbaf41b62aa507c487096d98168966a8"
|
|
266
266
|
},
|
|
267
267
|
{
|
|
268
268
|
"file": "shards/child-17.jsonl.gz",
|
|
269
|
-
"bytes":
|
|
270
|
-
"sha256": "
|
|
269
|
+
"bytes": 34902,
|
|
270
|
+
"sha256": "2874b45a1dc8dc829f960f9fafa05b15d7b8b5801cccdf9e490242c830466e93"
|
|
271
271
|
},
|
|
272
272
|
{
|
|
273
273
|
"file": "shards/child-18.jsonl.gz",
|
|
274
|
-
"bytes":
|
|
275
|
-
"sha256": "
|
|
274
|
+
"bytes": 33492,
|
|
275
|
+
"sha256": "e248f9137d314573917469b958bfbb1d3e6393feae119488976d9ce32d9707bf"
|
|
276
276
|
},
|
|
277
277
|
{
|
|
278
278
|
"file": "shards/child-19.jsonl.gz",
|
|
279
|
-
"bytes":
|
|
280
|
-
"sha256": "
|
|
279
|
+
"bytes": 35739,
|
|
280
|
+
"sha256": "f84e5b6584d5e46bb5566a39ba941c71e293ed9f8cc83f0eb627eff6dfddd32b"
|
|
281
281
|
},
|
|
282
282
|
{
|
|
283
283
|
"file": "shards/child-1a.jsonl.gz",
|
|
284
|
-
"bytes":
|
|
285
|
-
"sha256": "
|
|
284
|
+
"bytes": 32215,
|
|
285
|
+
"sha256": "f59bc66d6b5dfe8c50e95636098c15afa2430c4bc1973a2a71855b41e87c191e"
|
|
286
286
|
},
|
|
287
287
|
{
|
|
288
288
|
"file": "shards/child-1b.jsonl.gz",
|
|
289
|
-
"bytes":
|
|
290
|
-
"sha256": "
|
|
289
|
+
"bytes": 32942,
|
|
290
|
+
"sha256": "0ab7e4649c448ef556b54384575ffdd112877444969f7634b4f6c5195b08ccce"
|
|
291
291
|
},
|
|
292
292
|
{
|
|
293
293
|
"file": "shards/child-1c.jsonl.gz",
|
|
294
|
-
"bytes":
|
|
295
|
-
"sha256": "
|
|
294
|
+
"bytes": 34145,
|
|
295
|
+
"sha256": "336e70ce7e06d55fbf97b306a65f817e424b5c0beada5921013d1b8fc80f43a4"
|
|
296
296
|
},
|
|
297
297
|
{
|
|
298
298
|
"file": "shards/child-1d.jsonl.gz",
|
|
299
|
-
"bytes":
|
|
300
|
-
"sha256": "
|
|
299
|
+
"bytes": 31999,
|
|
300
|
+
"sha256": "880b79cff62cf3a3a1b8cc9f1a6c45c5a3e9dd5b88893740a59bd1574a6ba0a9"
|
|
301
301
|
},
|
|
302
302
|
{
|
|
303
303
|
"file": "shards/child-1e.jsonl.gz",
|
|
304
|
-
"bytes":
|
|
305
|
-
"sha256": "
|
|
304
|
+
"bytes": 32844,
|
|
305
|
+
"sha256": "ddefecf98f3d44a41e0899db9794be105a509ffae02e56d2b83dc5c47819fba0"
|
|
306
306
|
},
|
|
307
307
|
{
|
|
308
308
|
"file": "shards/child-1f.jsonl.gz",
|
|
309
|
-
"bytes":
|
|
310
|
-
"sha256": "
|
|
309
|
+
"bytes": 33887,
|
|
310
|
+
"sha256": "2b5abd6c51d20eedbba38c65e0679aea9842ba9f28ae4fb7b76cfabac1643301"
|
|
311
311
|
}
|
|
312
312
|
]
|
|
313
313
|
}
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -23,13 +23,19 @@
|
|
|
23
23
|
// 3. sentence fragment — bare term is >= 4 underscore-words on EITHER
|
|
24
24
|
// endpoint ("taloned_grip_of_owl",
|
|
25
25
|
// "worlds_largest_interconnected_network_of_networks")
|
|
26
|
-
// 4.
|
|
26
|
+
// 4. trailing-preposition fragment — bare term is a 2-3 underscore-word
|
|
27
|
+
// endpoint whose LAST word is a preposition
|
|
28
|
+
// ("pot_in") — a truncated crowd-sourced
|
|
29
|
+
// prepositional-phrase fragment, not a concept
|
|
30
|
+
// 5. known misspelling — bare endpoint is in a small, hand-curated
|
|
31
|
+
// KNOWN_MISSPELLINGS set ("vessle")
|
|
32
|
+
// 6. definitional phrase — /r/DefinedAs whose object is >= 3 words
|
|
27
33
|
// (real DefinedAs is a synonym: cpu->processor)
|
|
28
|
-
//
|
|
34
|
+
// 7. opinion object — /r/IsA or /r/DefinedAs whose object is one of a
|
|
29
35
|
// small, evidence-based OPINION set (adjectives /
|
|
30
36
|
// value words that never name a class)
|
|
31
37
|
//
|
|
32
|
-
// Rules 1-
|
|
38
|
+
// Rules 1-5 apply to every relation (they only ever remove junk); 6-7 are the
|
|
33
39
|
// definitional band the sims flagged. Stats land on stderr; JSONL on stdout.
|
|
34
40
|
|
|
35
41
|
import { readFile, writeFile } from "node:fs/promises";
|
|
@@ -46,6 +52,19 @@ export const OPINION_OBJECTS = new Set([
|
|
|
46
52
|
"elegance", "evil", "gloom", "unreality", "universalism", "dumb", "free", "junk",
|
|
47
53
|
]);
|
|
48
54
|
|
|
55
|
+
// A short, closed preposition set. A 2-3 underscore-word endpoint ending in
|
|
56
|
+
// one of these ("pot_in") is a truncated crowd-sourced prepositional-phrase
|
|
57
|
+
// fragment, not a standalone concept.
|
|
58
|
+
const TRAILING_PREPOSITIONS = new Set(["in", "on", "at", "of", "for", "with", "to", "from", "by"]);
|
|
59
|
+
|
|
60
|
+
// Evidence-based: crowd-sourced misspellings seen in the committed slice.
|
|
61
|
+
// Kept explicit (not a spellcheck dependency — wink-nlp carries no spellcheck
|
|
62
|
+
// signal, so a general heuristic here would risk cutting a legitimate but
|
|
63
|
+
// unfamiliar word) so it never cuts anything but a confirmed typo.
|
|
64
|
+
export const KNOWN_MISSPELLINGS = new Set([
|
|
65
|
+
"vessle",
|
|
66
|
+
]);
|
|
67
|
+
|
|
49
68
|
/** Why this row is noise, or null to keep it. Pure function of the row shape. */
|
|
50
69
|
export function cutReason(row) {
|
|
51
70
|
const s = bareTerm(row.start);
|
|
@@ -53,7 +72,12 @@ export function cutReason(row) {
|
|
|
53
72
|
for (const t of [s, e]) {
|
|
54
73
|
if (/^\d+$/.test(t)) return "numeric-endpoint";
|
|
55
74
|
if (t.length <= 1) return "single-char-endpoint";
|
|
56
|
-
|
|
75
|
+
const parts = t.split("_").filter(Boolean);
|
|
76
|
+
if (parts.length >= 4) return "sentence-fragment";
|
|
77
|
+
if (parts.length >= 2 && parts.length <= 3 && TRAILING_PREPOSITIONS.has(parts[parts.length - 1])) {
|
|
78
|
+
return "trailing-preposition-fragment";
|
|
79
|
+
}
|
|
80
|
+
if (KNOWN_MISSPELLINGS.has(t)) return "known-misspelling";
|
|
57
81
|
}
|
|
58
82
|
if (row.rel === "/r/DefinedAs" && words(e) >= 3) return "definitional-phrase";
|
|
59
83
|
if ((row.rel === "/r/IsA" || row.rel === "/r/DefinedAs") && OPINION_OBJECTS.has(e)) return "opinion-object";
|
|
@@ -719,7 +719,7 @@ export const CORPUSES = {
|
|
|
719
719
|
nouns: [
|
|
720
720
|
// human-core (+186 words: people, family, common roles)
|
|
721
721
|
"draw", "set", "heavy", "stand", "free", "pack", "dead", "pick",
|
|
722
|
-
"double", "
|
|
722
|
+
"double", "cast", "dry", "bank", "stamp", "tender", "fair",
|
|
723
723
|
"active", "regular", "blue", "hack", "band", "brush", "guard", "master",
|
|
724
724
|
"transfer", "firm", "side", "bull", "mate", "rank", "board", "brace",
|
|
725
725
|
"grade", "better", "course", "natural", "butt", "camp", "crop", "date",
|
|
@@ -733,7 +733,7 @@ export const CORPUSES = {
|
|
|
733
733
|
"steady", "stream", "combine", "company", "convert", "general", "moderate", "alternate",
|
|
734
734
|
"ace", "crab", "flop", "fund", "pair", "rake", "rock", "sick",
|
|
735
735
|
"blade", "cheat", "sport", "caller", "column", "couple", "divine", "pledge",
|
|
736
|
-
"
|
|
736
|
+
"sponge", "system", "wanton", "sleeper", "business", "associate", "instrument",
|
|
737
737
|
"substitute", "gun", "sum", "alto", "best", "buff", "dear", "mass",
|
|
738
738
|
"cream", "creep", "giant", "labor", "pilot", "sneak", "stack", "staff",
|
|
739
739
|
"world", "gutter", "jumper", "labour", "lather", "picket", "reader", "savage",
|
|
@@ -852,7 +852,6 @@ export const CORPUSES = {
|
|
|
852
852
|
["dead", "/r/HasA", "dead person"],
|
|
853
853
|
["pick", "/r/IsA", "elite"],
|
|
854
854
|
["double", "/r/IsA", "actor"],
|
|
855
|
-
["straight", "/r/IsA", "person"],
|
|
856
855
|
["cast", "/r/IsA", "gathering"],
|
|
857
856
|
["dry", "/r/IsA", "reformer"],
|
|
858
857
|
["bank", "/r/IsA", "financial institution"],
|
|
@@ -968,7 +967,6 @@ export const CORPUSES = {
|
|
|
968
967
|
["couple", "/r/IsA", "pair"],
|
|
969
968
|
["divine", "/r/IsA", "reverend"],
|
|
970
969
|
["pledge", "/r/IsA", "member"],
|
|
971
|
-
["simple", "/r/IsA", "person"],
|
|
972
970
|
["sponge", "/r/IsA", "learner"],
|
|
973
971
|
["system", "/r/IsA", "group"],
|
|
974
972
|
["system", "/r/HasA", "hierarchy"],
|
|
@@ -1845,7 +1843,7 @@ export const CORPUSES = {
|
|
|
1845
1843
|
"ideal", "noble", "octet", "quack", "saint", "scrag", "shark", "shoal",
|
|
1846
1844
|
"silly", "skunk", "super", "witch", "zombi", "agency", "anchor", "backup",
|
|
1847
1845
|
"batter", "buster", "canary", "circus", "dealer", "domain", "doomed", "genius",
|
|
1848
|
-
"gossip", "herald", "hoodoo", "hopper", "
|
|
1846
|
+
"gossip", "herald", "hoodoo", "hopper", "jockey", "legion", "member",
|
|
1849
1847
|
"nature", "orphan", "parade", "peanut", "pillar", "pirate", "player", "porter",
|
|
1850
1848
|
"pusher", "putter", "senior", "septet", "sestet", "sextet", "surety", "tapper",
|
|
1851
1849
|
"tiller", "zombie", "adjunct", "butcher", "captive", "cascade", "chapter", "charity",
|
|
@@ -1884,7 +1882,7 @@ export const CORPUSES = {
|
|
|
1884
1882
|
"canvasser", "cockscomb", "collector", "comforter", "commander", "companion", "conductor", "crossover",
|
|
1885
1883
|
"eccentric", "executive", "fledgling", "franchise", "freelance", "incumbent", "mercenary", "moderator",
|
|
1886
1884
|
"neighbour", "newspaper", "patriarch", "patrician", "portfolio", "quartette", "quintette", "repertory",
|
|
1887
|
-
"secretary", "segregate", "spearhead", "
|
|
1885
|
+
"secretary", "segregate", "spearhead", "summation", "transient", "trumpeter", "voluntary",
|
|
1888
1886
|
"absolutism", "anthropoid", "apparition", "assemblage", "collective", "contractor", "delinquent", "episcopate",
|
|
1889
1887
|
"exaltation", "exposition", "illiterate", "judicature", "leadership", "lieutenant", "minimalist", "percussion",
|
|
1890
1888
|
"profligate", "provincial", "rendezvous", "brotherhood", "determinant", "distributor", "interpreter", "stipendiary",
|
|
@@ -3267,7 +3265,6 @@ export const CORPUSES = {
|
|
|
3267
3265
|
["hoodoo", "/r/IsA", "priest"],
|
|
3268
3266
|
["priest", "/r/IsA", "spiritual leader"],
|
|
3269
3267
|
["hopper", "/r/IsA", "jumper"],
|
|
3270
|
-
["immune", "/r/IsA", "person"],
|
|
3271
3268
|
["jockey", "/r/IsA", "equestrian"],
|
|
3272
3269
|
["legion", "/r/IsA", "military unit"],
|
|
3273
3270
|
["legion", "/r/HasA", "legionnaire"],
|
|
@@ -3755,7 +3752,6 @@ export const CORPUSES = {
|
|
|
3755
3752
|
["friend", "/r/IsA", "actor"],
|
|
3756
3753
|
["segregate", "/r/IsA", "actor"],
|
|
3757
3754
|
["spearhead", "/r/IsA", "leader"],
|
|
3758
|
-
["stigmatic", "/r/IsA", "person"],
|
|
3759
3755
|
["summation", "/r/IsA", "collection"],
|
|
3760
3756
|
["transient", "/r/IsA", "traveler"],
|
|
3761
3757
|
["trumpeter", "/r/IsA", "messenger"],
|
|
@@ -438,7 +438,6 @@
|
|
|
438
438
|
{"start":"/c/en/hoodoo","rel":"/r/IsA","end":"/c/en/priest","weight":1,"surfaceText":"[[hoodoo]] IsA [[priest]]"}
|
|
439
439
|
{"start":"/c/en/priest","rel":"/r/IsA","end":"/c/en/spiritual leader","weight":1,"surfaceText":"[[priest]] IsA [[spiritual leader]]"}
|
|
440
440
|
{"start":"/c/en/hopper","rel":"/r/IsA","end":"/c/en/jumper","weight":1,"surfaceText":"[[hopper]] IsA [[jumper]]"}
|
|
441
|
-
{"start":"/c/en/immune","rel":"/r/IsA","end":"/c/en/person","weight":1,"surfaceText":"[[immune]] IsA [[person]]"}
|
|
442
441
|
{"start":"/c/en/jockey","rel":"/r/IsA","end":"/c/en/equestrian","weight":1,"surfaceText":"[[jockey]] IsA [[equestrian]]"}
|
|
443
442
|
{"start":"/c/en/legion","rel":"/r/IsA","end":"/c/en/military unit","weight":1,"surfaceText":"[[legion]] IsA [[military unit]]"}
|
|
444
443
|
{"start":"/c/en/legion","rel":"/r/HasA","end":"/c/en/legionnaire","weight":1,"surfaceText":"[[legion]] HasA [[legionnaire]]"}
|
|
@@ -926,7 +925,6 @@
|
|
|
926
925
|
{"start":"/c/en/friend","rel":"/r/IsA","end":"/c/en/actor","weight":1,"surfaceText":"[[friend]] IsA [[actor]]"}
|
|
927
926
|
{"start":"/c/en/segregate","rel":"/r/IsA","end":"/c/en/actor","weight":1,"surfaceText":"[[segregate]] IsA [[actor]]"}
|
|
928
927
|
{"start":"/c/en/spearhead","rel":"/r/IsA","end":"/c/en/leader","weight":1,"surfaceText":"[[spearhead]] IsA [[leader]]"}
|
|
929
|
-
{"start":"/c/en/stigmatic","rel":"/r/IsA","end":"/c/en/person","weight":1,"surfaceText":"[[stigmatic]] IsA [[person]]"}
|
|
930
928
|
{"start":"/c/en/summation","rel":"/r/IsA","end":"/c/en/collection","weight":1,"surfaceText":"[[summation]] IsA [[collection]]"}
|
|
931
929
|
{"start":"/c/en/transient","rel":"/r/IsA","end":"/c/en/traveler","weight":1,"surfaceText":"[[transient]] IsA [[traveler]]"}
|
|
932
930
|
{"start":"/c/en/trumpeter","rel":"/r/IsA","end":"/c/en/messenger","weight":1,"surfaceText":"[[trumpeter]] IsA [[messenger]]"}
|
|
@@ -9,7 +9,6 @@
|
|
|
9
9
|
{"start":"/c/en/dead","rel":"/r/HasA","end":"/c/en/dead person","weight":1,"surfaceText":"[[dead]] HasA [[dead person]]"}
|
|
10
10
|
{"start":"/c/en/pick","rel":"/r/IsA","end":"/c/en/elite","weight":1,"surfaceText":"[[pick]] IsA [[elite]]"}
|
|
11
11
|
{"start":"/c/en/double","rel":"/r/IsA","end":"/c/en/actor","weight":1,"surfaceText":"[[double]] IsA [[actor]]"}
|
|
12
|
-
{"start":"/c/en/straight","rel":"/r/IsA","end":"/c/en/person","weight":1,"surfaceText":"[[straight]] IsA [[person]]"}
|
|
13
12
|
{"start":"/c/en/cast","rel":"/r/IsA","end":"/c/en/gathering","weight":1,"surfaceText":"[[cast]] IsA [[gathering]]"}
|
|
14
13
|
{"start":"/c/en/dry","rel":"/r/IsA","end":"/c/en/reformer","weight":1,"surfaceText":"[[dry]] IsA [[reformer]]"}
|
|
15
14
|
{"start":"/c/en/bank","rel":"/r/IsA","end":"/c/en/financial institution","weight":1,"surfaceText":"[[bank]] IsA [[financial institution]]"}
|
|
@@ -125,7 +124,6 @@
|
|
|
125
124
|
{"start":"/c/en/couple","rel":"/r/IsA","end":"/c/en/pair","weight":1,"surfaceText":"[[couple]] IsA [[pair]]"}
|
|
126
125
|
{"start":"/c/en/divine","rel":"/r/IsA","end":"/c/en/reverend","weight":1,"surfaceText":"[[divine]] IsA [[reverend]]"}
|
|
127
126
|
{"start":"/c/en/pledge","rel":"/r/IsA","end":"/c/en/member","weight":1,"surfaceText":"[[pledge]] IsA [[member]]"}
|
|
128
|
-
{"start":"/c/en/simple","rel":"/r/IsA","end":"/c/en/person","weight":1,"surfaceText":"[[simple]] IsA [[person]]"}
|
|
129
127
|
{"start":"/c/en/sponge","rel":"/r/IsA","end":"/c/en/learner","weight":1,"surfaceText":"[[sponge]] IsA [[learner]]"}
|
|
130
128
|
{"start":"/c/en/system","rel":"/r/IsA","end":"/c/en/group","weight":1,"surfaceText":"[[system]] IsA [[group]]"}
|
|
131
129
|
{"start":"/c/en/system","rel":"/r/HasA","end":"/c/en/hierarchy","weight":1,"surfaceText":"[[system]] HasA [[hierarchy]]"}
|
|
@@ -81,9 +81,9 @@
|
|
|
81
81
|
"tool": "corpus/tier2/generate.mjs"
|
|
82
82
|
},
|
|
83
83
|
"file": "human-medium.jsonl",
|
|
84
|
-
"facts":
|
|
85
|
-
"bytes":
|
|
86
|
-
"sha256": "
|
|
84
|
+
"facts": 942,
|
|
85
|
+
"bytes": 110564,
|
|
86
|
+
"sha256": "152df1c9026ace0ecf899e89b23dab77617e97fe0d7fafd50866e26a792f0454",
|
|
87
87
|
"license": "MPL-2.0"
|
|
88
88
|
},
|
|
89
89
|
{
|
|
@@ -95,9 +95,9 @@
|
|
|
95
95
|
"tool": "corpus/tier2/generate.mjs"
|
|
96
96
|
},
|
|
97
97
|
"file": "human-large.jsonl",
|
|
98
|
-
"facts":
|
|
99
|
-
"bytes":
|
|
100
|
-
"sha256": "
|
|
98
|
+
"facts": 11999,
|
|
99
|
+
"bytes": 1464616,
|
|
100
|
+
"sha256": "fbbe8135d4915f2bc041bf577e7b0bd1a8fd1c1bab804c8215b1d05e39db27f9",
|
|
101
101
|
"license": "MPL-2.0"
|
|
102
102
|
}
|
|
103
103
|
],
|