@hviana/sema 0.5.7 → 0.5.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/HOW_IT_WORKS.md +74 -0
- package/dist/src/mind/articulation.js +15 -2
- package/dist/src/mind/learning.js +166 -3
- package/dist/src/mind/traverse.d.ts +18 -0
- package/dist/src/mind/traverse.js +24 -0
- package/jsr.json +1 -1
- package/package.json +1 -1
- package/src/mind/articulation.ts +16 -2
- package/src/mind/learning.ts +159 -4
- package/src/mind/traverse.ts +29 -0
- package/test/76-type-level-company.test.mjs +342 -0
package/HOW_IT_WORKS.md
CHANGED
|
@@ -512,6 +512,73 @@ similarity into _distributional_ similarity, silently shifting the halo null
|
|
|
512
512
|
model that the concept threshold's derivation (unrelated halos ⇒ cosine 0 ±
|
|
513
513
|
1/√D) depends on.
|
|
514
514
|
|
|
515
|
+
#### Company must be keyed on types, not tokens
|
|
516
|
+
|
|
517
|
+
A signature derived from the _whole_ partner's node id records a **token**:
|
|
518
|
+
"occurred next to node #4711992". But the distributional hypothesis is a claim
|
|
519
|
+
about **types**: "occurred next to a city name". On a content-addressed store of
|
|
520
|
+
natural language the difference is fatal, because a whole deposit essentially
|
|
521
|
+
never repeats: on a trained 15.7M-node store, 331,110 trained pairs produced
|
|
522
|
+
325,615 distinct contexts — 98.3% of them unique, so barely one span in sixty
|
|
523
|
+
ever recurs. Two genuine synonyms in different sentences then share no partner
|
|
524
|
+
id at all, and their halos come out quasi-orthogonal _by construction_: the best
|
|
525
|
+
distributional sibling of "Eiffel Tower" measured 0.146 against a concept
|
|
526
|
+
threshold of 0.516, with its own attested translations absent entirely. The
|
|
527
|
+
whole concept-hop, articulation and analogy layer was inert at corpus scale —
|
|
528
|
+
not for want of data, but because the key was too fine.
|
|
529
|
+
|
|
530
|
+
Sema therefore pours a **company profile**: the partner's own signature
|
|
531
|
+
superposed with the signatures of the partner's _constituents_. Two episodes
|
|
532
|
+
then share halo mass exactly when their partners are **made of** something in
|
|
533
|
+
common, which is what "keeps similar company" was always meant to mean.
|
|
534
|
+
|
|
535
|
+
Which constituents count is decided by four rules, each one load-bearing:
|
|
536
|
+
|
|
537
|
+
- **Every depth, not just the partner's children.** Cuts are content-defined
|
|
538
|
+
over a rolling window, so a chunk boundary depends on the bytes _around_ a
|
|
539
|
+
unit. "The Eiffel Tower is in Paris" folds to `The Eiffel` ·
|
|
540
|
+
`Tower is in
|
|
541
|
+
Paris`, and "Tour Eiffel dia any Paris" to `Tour Eiffel` ·
|
|
542
|
+
`dia any Paris` — the shared unit `Paris` is a child of neither. Reading only
|
|
543
|
+
the children merely moves the token problem from whole-partner identity down
|
|
544
|
+
to top-level-chunk identity, which for full sentences is nearly as rare. The
|
|
545
|
+
descent finds `Paris` and `ffel`; the top level finds nothing.
|
|
546
|
+
- **Never gate on recurrence-so-far.** The tempting rule — descend while a
|
|
547
|
+
constituent is corpus-unique, stop at the first unit attested twice — is
|
|
548
|
+
order-dependent: when the _first_ of a pair is deposited its shared unit has
|
|
549
|
+
been seen once, so the descent runs past it, and only the second partner ever
|
|
550
|
+
profiles it. Whether two synonyms become siblings must not depend on which was
|
|
551
|
+
taught first. The hub test below is the one reading that does consult corpus
|
|
552
|
+
state, and it can only ever remove a term — never decide which units are
|
|
553
|
+
found.
|
|
554
|
+
- **Nothing narrower than the fold's own window `W`.** Sub-window shards are
|
|
555
|
+
fragments of a unit, not units; admitting them lets mid-frequency byte
|
|
556
|
+
coincidences leak company across unrelated domains.
|
|
557
|
+
- **Minimal units only, and nothing that half-dominates the partner.** A
|
|
558
|
+
constituent that still contains a constituent of its own is a composite, and
|
|
559
|
+
superposing both counts the same content twice. This matters most for _nested_
|
|
560
|
+
partners — an accumulated conversation, where turn _k_'s context is a prefix
|
|
561
|
+
of turn _k+1_'s — which share their large chunks structurally rather than
|
|
562
|
+
distributionally.
|
|
563
|
+
|
|
564
|
+
Hubs (more than √N structural parents — Section 8.8) are excluded as
|
|
565
|
+
scaffolding, since a term shared by every deposit would put a common component
|
|
566
|
+
in every profile and collapse the null model the concept threshold depends on.
|
|
567
|
+
Byte atoms are excluded for the same reason: an atom's fan-in is the alphabet's.
|
|
568
|
+
|
|
569
|
+
The result stays **normalized**, so one episode still pours one unit of mass and
|
|
570
|
+
every mass-based reading is unchanged; and every term is still a seeded function
|
|
571
|
+
of a **node identity**, never a gist, so the hygiene rule above is untouched.
|
|
572
|
+
|
|
573
|
+
Similarity is **graded and size-relative**: two partners meet in proportion to
|
|
574
|
+
how much of what they are made of they share, so a shared unit inside a long
|
|
575
|
+
partner says less than the same unit inside a short one. This is the honest
|
|
576
|
+
reading of the evidence, not a defect — but it means the layer speaks clearly at
|
|
577
|
+
sentence scale and softly for very long partners. One further honest limit: the
|
|
578
|
+
hub bound is √N, so on a _small_ store almost nothing reads as scaffolding and
|
|
579
|
+
frame words do enter profiles. That is the correct floor for a corpus that
|
|
580
|
+
cannot yet say what discriminates, and it resolves itself as N grows.
|
|
581
|
+
|
|
515
582
|
Two nodes whose halos are similar have occurred in similar circumstances — they
|
|
516
583
|
are **distributional siblings**: synonyms, paraphrases, items of the same
|
|
517
584
|
category, two names for one thing. Note the complementarity:
|
|
@@ -532,6 +599,13 @@ embedding model:
|
|
|
532
599
|
rule).
|
|
533
600
|
- **Articulation.** An answer is re-voiced in the asker's own vocabulary by
|
|
534
601
|
substituting answer forms with query forms that share a halo (Section 24).
|
|
602
|
+
Halo similarity alone is not sufficient authority for this: the strongest
|
|
603
|
+
company any two forms can keep is standing next to each other, so a question
|
|
604
|
+
and the answer it is answered by are _maximally_ similar distributionally. Two
|
|
605
|
+
exact-side vetoes keep that from being read as synonymy — a stored
|
|
606
|
+
continuation edge between the two forms, and a candidate form spanning the
|
|
607
|
+
_whole_ answer (substituting that is replacement, not re-voicing). Without
|
|
608
|
+
them, "it hangs in madrid" is faithfully re-voiced as "where is it kept now".
|
|
535
609
|
- **Analogy strength.** Whether two entities are genuinely analogous — the gate
|
|
536
610
|
on counterfactual comparison (Section 18) — is measured by halo similarity,
|
|
537
611
|
directly or through shared siblings (a second-order distributional test).
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
// using concept (halo) resonance to match the voices.
|
|
5
5
|
import { spliceAll } from "./types.js";
|
|
6
6
|
import { recognise } from "./recognition.js";
|
|
7
|
-
import { contains } from "./traverse.js";
|
|
7
|
+
import { answers, contains } from "./traverse.js";
|
|
8
8
|
import { bestHaloMate } from "./match.js";
|
|
9
9
|
import { coverSequence } from "../derive/src/index.js";
|
|
10
10
|
import { rItem, rNode, traceDerivation } from "./trace.js";
|
|
@@ -67,9 +67,22 @@ export async function articulate(ctx, answer, query) {
|
|
|
67
67
|
if (!found)
|
|
68
68
|
continue;
|
|
69
69
|
const voice = found.item;
|
|
70
|
-
if (voice.node === s.payload || contains(ctx, voice.node, s.payload)
|
|
70
|
+
if (voice.node === s.payload || contains(ctx, voice.node, s.payload) ||
|
|
71
|
+
answers(ctx, voice.node, s.payload)) {
|
|
71
72
|
continue;
|
|
72
73
|
}
|
|
74
|
+
// A form spanning the WHOLE answer is not a concept inside the answer to
|
|
75
|
+
// revoice — substituting it discards the answer and emits the asker's own
|
|
76
|
+
// words back, which is what a conversational store makes tempting: an
|
|
77
|
+
// answer and the question it answers keep maximal company, so the whole
|
|
78
|
+
// answer resonates with the whole query above any concept threshold
|
|
79
|
+
// (measured on the CONV fixture at 0.809 against 0.516). Articulation
|
|
80
|
+
// splices the asker's wording INTO an answer where the same concept
|
|
81
|
+
// appears; when the "concept" is the entire answer there is nothing left
|
|
82
|
+
// of it, and "where is it kept now" comes back in place of "it hangs in
|
|
83
|
+
// madrid". §5's contract is re-voicing, never replacement.
|
|
84
|
+
if (s.start === 0 && s.end === answer.length)
|
|
85
|
+
continue;
|
|
73
86
|
substitutions.set(s.payload, voice.bytes);
|
|
74
87
|
}
|
|
75
88
|
if (substitutions.size === 0) {
|
|
@@ -2,10 +2,13 @@
|
|
|
2
2
|
//
|
|
3
3
|
// Learning is DEPOSITION: perceive a stream into a tree and intern every
|
|
4
4
|
// node. A fact is an EDGE between node ids; recall traverses edges.
|
|
5
|
+
import { addInto, normalize, zeros } from "../vec.js";
|
|
5
6
|
import { bindSeat, companySignature, isChunk } from "../sema.js";
|
|
6
7
|
import { changedNodes } from "./types.js";
|
|
7
8
|
import { inputBytes, perceiveDeposit, resolve, } from "./primitives.js";
|
|
8
9
|
import { canonicalWindows, leafIdPrefix } from "./canonical.js";
|
|
10
|
+
import { hubBound } from "./traverse.js";
|
|
11
|
+
import { dominates } from "../geometry.js";
|
|
9
12
|
import { fold as foldVecs } from "../sema.js";
|
|
10
13
|
/** Intern a perceived tree into node ids, bottom-up, sharing equal subtrees.
|
|
11
14
|
* Returns the root node id and a map from tree nodes to their ids.
|
|
@@ -188,6 +191,163 @@ async function propagateSuffixes(ctx, src, dst) {
|
|
|
188
191
|
await ctx.store.link(id, dst);
|
|
189
192
|
}
|
|
190
193
|
}
|
|
194
|
+
/** How many constituents one profile may VISIT. A partner's constituent tree
|
|
195
|
+
* is O(len/W) nodes, so an uncapped descent would make a pour cost grow with
|
|
196
|
+
* the partner's LENGTH — and a partner is a whole deposit, which may be a
|
|
197
|
+
* paragraph. The budget is what keeps a pour O(1) in the input, the property
|
|
198
|
+
* that lets {@link companyProfile} claim no new cost class. It binds only on
|
|
199
|
+
* long partners whose constituents are all corpus-unique; the descent's own
|
|
200
|
+
* stop rule (below) reaches recurring units far sooner on a trained store. */
|
|
201
|
+
const PROFILE_VISITS = 64;
|
|
202
|
+
/** The COMPANY PROFILE of a partner: its own identity signature superposed
|
|
203
|
+
* with the signatures of its RECURRING content-defined constituents.
|
|
204
|
+
*
|
|
205
|
+
* WHY THE WHOLE-PARTNER SIGNATURE ALONE IS NOT ENOUGH. The distributional
|
|
206
|
+
* hypothesis is a claim about TYPES ("occurs near a city name"), but a
|
|
207
|
+
* signature keyed on the whole partner's node id records a TOKEN ("occurred
|
|
208
|
+
* near node #4711992"). Two nodes are then distributional siblings only when
|
|
209
|
+
* their partners are the very same node — and a content-addressed store of
|
|
210
|
+
* natural language almost never repeats a whole deposit (measured on the
|
|
211
|
+
* trained store: whole-span dedup 0.98×, i.e. effectively none). So the
|
|
212
|
+
* halos of genuine synonyms came out quasi-orthogonal BY CONSTRUCTION: the
|
|
213
|
+
* best distributional sibling of "Eiffel Tower" scored 0.146 against a
|
|
214
|
+
* concept threshold of 0.516, with its own attested translations absent
|
|
215
|
+
* entirely, and the whole concept-hop / articulation / analogy layer was
|
|
216
|
+
* inert at corpus scale. (Re-verified under the store's OWN training seed:
|
|
217
|
+
* company signatures key on NODE ID, not the alphabet, so this reading is
|
|
218
|
+
* seed-independent and the figures are identical either way. Worth stating
|
|
219
|
+
* because a Mind built with a seed other than the store's makes every GIST
|
|
220
|
+
* comparison meaningless while leaving halo comparisons untouched.)
|
|
221
|
+
*
|
|
222
|
+
* WHY THE DESCENT MUST NOT STOP AT DEPTH 1. Reading only `rec.kids` does
|
|
223
|
+
* NOT deliver this. Cuts are content-defined over a rolling window, so a
|
|
224
|
+
* chunk boundary depends on the bytes AROUND a unit: "The Eiffel Tower is in
|
|
225
|
+
* Paris" folds to "The Eiffel " + "Tower is in Paris", and "Tour Eiffel dia
|
|
226
|
+
* any Paris" to "Tour Eiffel " + "dia any Paris". The shared unit "Paris"
|
|
227
|
+
* is a node in NEITHER — depth-1 profiles of that pair intersect in the
|
|
228
|
+
* EMPTY SET, and their halos measured 0.0319 against 0.0416 for an unrelated
|
|
229
|
+
* control: no signal at all. A depth-1 read merely moves the token problem
|
|
230
|
+
* from whole-partner identity down to top-level-chunk identity, which for
|
|
231
|
+
* full sentences is nearly as rare. Descending, the same pair shares
|
|
232
|
+
* " Paris" and "ffel " while the control still shares nothing — the units
|
|
233
|
+
* the distributional hypothesis is actually about.
|
|
234
|
+
*
|
|
235
|
+
* EVERY DEPTH CONTRIBUTES, AND THE RULE MUST NOT DEPEND ON ARRIVAL ORDER.
|
|
236
|
+
* The tempting stop rule — descend only while a constituent is corpus-unique,
|
|
237
|
+
* stop at the first unit attested in ≥ 2 forms — is wrong, and measurably so.
|
|
238
|
+
* Recurrence is a property of the corpus SO FAR: when the first of a pair is
|
|
239
|
+
* deposited its shared unit has fan-in 1, so the descent runs past it, and
|
|
240
|
+
* only the second partner ever profiles it. The pair then never meets
|
|
241
|
+
* (measured on the fixture above: 0.0165 against a 0.0375 control — still
|
|
242
|
+
* nothing). Whether two synonyms become distributional siblings cannot be
|
|
243
|
+
* allowed to depend on which was trained first. So the walk descends through
|
|
244
|
+
* EVERY constituent within its budget and superposes each one that is not a
|
|
245
|
+
* hub, at whatever depth it sits. A partner's own unique chunks contribute
|
|
246
|
+
* terms unique to that partner, which dilute but never mislead; the shared
|
|
247
|
+
* units contribute the signal.
|
|
248
|
+
*
|
|
249
|
+
* HUBS ARE THE ONE EXCLUSION, read LIMITed as `parentsFirst(n, bound+1)` —
|
|
250
|
+
* the store's own exact hub-or-not probe (a result longer than the bound
|
|
251
|
+
* means MORE than the bound), never a fan-in-sized read. A constituent with
|
|
252
|
+
* more than √N structural parents is scaffolding by §8.8's bound: " is ",
|
|
253
|
+
* "the ". Superposing it would put a term shared by every deposit into every
|
|
254
|
+
* profile, ALL halos would correlate, and the concept threshold's null model
|
|
255
|
+
* (unrelated halos at 0 ± 1/√D) that §4.1's hygiene note protects would
|
|
256
|
+
* collapse. It is still DESCENDED into — a hub chunk can contain a rare
|
|
257
|
+
* unit — but contributes nothing itself.
|
|
258
|
+
*
|
|
259
|
+
* Byte atoms are skipped in BOTH representations (a negative id and a stored
|
|
260
|
+
* kid-less node): an atom's fan-in is the alphabet's, so it can only ever
|
|
261
|
+
* read as a hub, and a short partner folding FLAT would otherwise put a
|
|
262
|
+
* handful of alphabet signatures into every profile — which is what silenced
|
|
263
|
+
* CAST's analogy gate in the first version of this function (measured:
|
|
264
|
+
* analogy strength 0.3636 -> 0.2004, "no halo-tier company evidence",
|
|
265
|
+
* test/29 C1).
|
|
266
|
+
*
|
|
267
|
+
* A FUNCTION OF THE NODE AND THE CORPUS STATE — stated precisely, because
|
|
268
|
+
* the weaker claim is the true one. The constituents are read from the
|
|
269
|
+
* STORE, never from the depositing tree's id map: that map holds only the
|
|
270
|
+
* nodes THIS deposit newly interned, so a partner met a second time yielded a
|
|
271
|
+
* profile missing exactly those constituents, the exact-partner case fell
|
|
272
|
+
* from cosine 1 to 1/√(1+k), and the geometry stopped meaning anything.
|
|
273
|
+
* Reading the store fixes that. It does NOT make the profile permanent: the
|
|
274
|
+
* hub test reads fan-in against √N and both grow with training, so a partner
|
|
275
|
+
* poured early and again late can profile differently. That residue is
|
|
276
|
+
* confined to the hub EXCLUSION — which terms are dropped as scaffolding —
|
|
277
|
+
* and never to which units are found, because the descent itself is now
|
|
278
|
+
* order-independent. The drift is one-directional and benign: a term can
|
|
279
|
+
* only ever go from contributing to being excluded as scaffolding. Replay of
|
|
280
|
+
* a fixed training order is bit-identical, so §2.1 holds. What must not be
|
|
281
|
+
* claimed is that a node's profile is fixed for all time; it is fixed given
|
|
282
|
+
* the corpus that has been seen.
|
|
283
|
+
*
|
|
284
|
+
* THE NULL MODEL IS OTHERWISE UNTOUCHED (§4.1). Every term is still a seeded
|
|
285
|
+
* function of a NODE IDENTITY, never a gist, so no byte-similarity between
|
|
286
|
+
* partners can leak content similarity into distributional similarity. The
|
|
287
|
+
* result is normalized, so ONE episode still pours ONE unit of mass:
|
|
288
|
+
* {@link Store.haloMass} keeps counting episodes and every mass-based
|
|
289
|
+
* reading is unchanged. Two partners sharing j of k discriminating
|
|
290
|
+
* constituents meet at j/(1+k) — graded evidence, above the 1/√D noise floor
|
|
291
|
+
* and below conceptThreshold until the overlap is most of the content, which
|
|
292
|
+
* is the semantics "same company" should have.
|
|
293
|
+
*
|
|
294
|
+
* Bounded: at most {@link PROFILE_VISITS} constituents are classified, each
|
|
295
|
+
* by ONE LIMITed structural-parent read, so a pour costs O(1) reads in the
|
|
296
|
+
* partner's size and performs no scan. */
|
|
297
|
+
function companyProfile(ctx, id) {
|
|
298
|
+
const acc = zeros(ctx.space.D);
|
|
299
|
+
addInto(acc, companySignature(ctx.space, id));
|
|
300
|
+
const bound = hubBound(ctx);
|
|
301
|
+
const W = ctx.space.maxGroup;
|
|
302
|
+
const whole = Math.max(1, ctx.store.contentLen(id));
|
|
303
|
+
const frontier = [];
|
|
304
|
+
const seen = new Set([id]);
|
|
305
|
+
const descend = (n) => {
|
|
306
|
+
const kids = ctx.store.get(n)?.kids;
|
|
307
|
+
if (!kids)
|
|
308
|
+
return;
|
|
309
|
+
for (const kid of kids)
|
|
310
|
+
if (!seen.has(kid))
|
|
311
|
+
frontier.push(kid);
|
|
312
|
+
};
|
|
313
|
+
descend(id);
|
|
314
|
+
for (let visits = 0; visits < PROFILE_VISITS && frontier.length > 0;) {
|
|
315
|
+
const n = frontier.shift();
|
|
316
|
+
if (seen.has(n))
|
|
317
|
+
continue;
|
|
318
|
+
seen.add(n);
|
|
319
|
+
visits++;
|
|
320
|
+
// Atoms in both representations — negative id, or a stored kid-less node.
|
|
321
|
+
if (n < 0 || ctx.store.get(n)?.kids == null)
|
|
322
|
+
continue;
|
|
323
|
+
descend(n);
|
|
324
|
+
const len = ctx.store.contentLen(n, whole);
|
|
325
|
+
if (len < W || dominates(len, whole))
|
|
326
|
+
continue;
|
|
327
|
+
// MINIMAL units only: a constituent that still has a constituent of its
|
|
328
|
+
// own at or above W is a composite, and superposing it as well as its
|
|
329
|
+
// parts would count the same content twice. Nested partners — an
|
|
330
|
+
// accumulated conversation, where turn k's context is a prefix of turn
|
|
331
|
+
// k+1's — share their large chunks structurally rather than
|
|
332
|
+
// distributionally, so those composites are exactly the terms that make
|
|
333
|
+
// adjacent turns read as synonyms (measured: consecutive turns at 0.809
|
|
334
|
+
// and 0.740 against a 0.516 concept threshold). The smallest units at or
|
|
335
|
+
// above the fold's own window are the word-sized types company should be
|
|
336
|
+
// keyed at.
|
|
337
|
+
const kids = ctx.store.get(n).kids;
|
|
338
|
+
let composite = false;
|
|
339
|
+
for (const kid of kids) {
|
|
340
|
+
if (kid >= 0 && ctx.store.contentLen(kid, W) >= W)
|
|
341
|
+
composite = true;
|
|
342
|
+
}
|
|
343
|
+
if (composite)
|
|
344
|
+
continue;
|
|
345
|
+
if (ctx.store.parentsFirst(n, bound + 1).length > bound)
|
|
346
|
+
continue;
|
|
347
|
+
addInto(acc, companySignature(ctx.space, n));
|
|
348
|
+
}
|
|
349
|
+
return normalize(acc);
|
|
350
|
+
}
|
|
191
351
|
/** Ingest a pair (context, continuation) — learn an edge and pour halos.
|
|
192
352
|
* Returns the deposited root ids (context, continuation) — a pure
|
|
193
353
|
* read-out; callers that ignore it behave exactly as before. */
|
|
@@ -203,12 +363,15 @@ export async function ingestPair(ctx, ctxInput, cont) {
|
|
|
203
363
|
await ctx.store.link(ctxId, contId);
|
|
204
364
|
await propagateSuffixes(ctx, ctxId, contId);
|
|
205
365
|
// Halos pour company SIGNATURES (identity), not gists (content) — see
|
|
206
|
-
// companySignature in sema.ts
|
|
207
|
-
|
|
366
|
+
// companySignature in sema.ts — as a TYPE-level profile: the partner's own
|
|
367
|
+
// signature superposed with its discriminating constituents' (see
|
|
368
|
+
// companyProfile), so company is shared by what partners are MADE OF and
|
|
369
|
+
// not only by partner identity.
|
|
370
|
+
const contSeat = bindSeat(ctx.space, companyProfile(ctx, contId), 1);
|
|
208
371
|
for (const part of c.changed) {
|
|
209
372
|
const partId = c.ids.get(part);
|
|
210
373
|
await ctx.store.pourHalo(partId, contSeat);
|
|
211
|
-
await ctx.store.pourHalo(contId, bindSeat(ctx.space,
|
|
374
|
+
await ctx.store.pourHalo(contId, bindSeat(ctx.space, companyProfile(ctx, partId), 0));
|
|
212
375
|
}
|
|
213
376
|
return { ctxId, contId };
|
|
214
377
|
}
|
|
@@ -80,6 +80,24 @@ export declare function hubCap<T>(ctx: MindContext, ids: readonly T[]): readonly
|
|
|
80
80
|
* descent. Used by articulation to keep a voice from revoicing a fragment
|
|
81
81
|
* OF that voice. */
|
|
82
82
|
export declare function contains(ctx: MindContext, ancestor: number, descendant: number): boolean;
|
|
83
|
+
/** Whether a continuation edge joins the two forms, in either direction —
|
|
84
|
+
* the EXACT half's veto on calling them synonyms.
|
|
85
|
+
*
|
|
86
|
+
* Halos measure company, and the strongest company any two forms can keep is
|
|
87
|
+
* standing next to each other: a question and its answer co-occur in every
|
|
88
|
+
* episode that taught the pair, so their halos SHOULD be similar, and on a
|
|
89
|
+
* conversational store they are (measured on the CONV fixture: consecutive
|
|
90
|
+
* turns at 0.809 against a 0.516 concept threshold). A gate reading halo
|
|
91
|
+
* cosine alone therefore reads adjacency as synonymy and revoices an answer
|
|
92
|
+
* in the words of the question it answers — "it hangs in madrid" spliced back
|
|
93
|
+
* into "where is it kept now". The distributional layer cannot tell the two
|
|
94
|
+
* relations apart, because to it they are the same observation; the exact
|
|
95
|
+
* half can, for free, because it stored the edge. §4.1's division of labour
|
|
96
|
+
* exactly: approximate proposes, exact decides.
|
|
97
|
+
*
|
|
98
|
+
* Read LIMITed in both directions at the hub bound — a common continuation's
|
|
99
|
+
* fan-in is corpus-sized, and no single decision may scale with it. */
|
|
100
|
+
export declare function answers(ctx: MindContext, a: number, b: number): boolean;
|
|
83
101
|
/** The best-scoring item by cosine against `query`, among items scoring at
|
|
84
102
|
* or above `threshold` — the shared arg-max every Pattern-A "which of these
|
|
85
103
|
* resonates best" decision reduces to. `strict` picks the tie-break a
|
|
@@ -490,6 +490,30 @@ export function contains(ctx, ancestor, descendant) {
|
|
|
490
490
|
}
|
|
491
491
|
return false;
|
|
492
492
|
}
|
|
493
|
+
/** Whether a continuation edge joins the two forms, in either direction —
|
|
494
|
+
* the EXACT half's veto on calling them synonyms.
|
|
495
|
+
*
|
|
496
|
+
* Halos measure company, and the strongest company any two forms can keep is
|
|
497
|
+
* standing next to each other: a question and its answer co-occur in every
|
|
498
|
+
* episode that taught the pair, so their halos SHOULD be similar, and on a
|
|
499
|
+
* conversational store they are (measured on the CONV fixture: consecutive
|
|
500
|
+
* turns at 0.809 against a 0.516 concept threshold). A gate reading halo
|
|
501
|
+
* cosine alone therefore reads adjacency as synonymy and revoices an answer
|
|
502
|
+
* in the words of the question it answers — "it hangs in madrid" spliced back
|
|
503
|
+
* into "where is it kept now". The distributional layer cannot tell the two
|
|
504
|
+
* relations apart, because to it they are the same observation; the exact
|
|
505
|
+
* half can, for free, because it stored the edge. §4.1's division of labour
|
|
506
|
+
* exactly: approximate proposes, exact decides.
|
|
507
|
+
*
|
|
508
|
+
* Read LIMITed in both directions at the hub bound — a common continuation's
|
|
509
|
+
* fan-in is corpus-sized, and no single decision may scale with it. */
|
|
510
|
+
export function answers(ctx, a, b) {
|
|
511
|
+
const bound = hubBound(ctx);
|
|
512
|
+
if (ctx.store.hasNext(a) && ctx.store.nextFirst(a, bound).includes(b)) {
|
|
513
|
+
return true;
|
|
514
|
+
}
|
|
515
|
+
return ctx.store.hasNext(b) && ctx.store.nextFirst(b, bound).includes(a);
|
|
516
|
+
}
|
|
493
517
|
// ── Edge disambiguation (Section 6) ──────────────────────────────────────
|
|
494
518
|
/** The best-scoring item by cosine against `query`, among items scoring at
|
|
495
519
|
* or above `threshold` — the shared arg-max every Pattern-A "which of these
|
package/jsr.json
CHANGED
package/package.json
CHANGED
package/src/mind/articulation.ts
CHANGED
|
@@ -7,7 +7,7 @@ import { Vec } from "../vec.js";
|
|
|
7
7
|
import type { MindContext } from "./types.js";
|
|
8
8
|
import { spliceAll } from "./types.js";
|
|
9
9
|
import { recognise } from "./recognition.js";
|
|
10
|
-
import { contains } from "./traverse.js";
|
|
10
|
+
import { answers, contains } from "./traverse.js";
|
|
11
11
|
import { bestHaloMate } from "./match.js";
|
|
12
12
|
import type { Site } from "./graph-search.js";
|
|
13
13
|
import type { CandidateSpan } from "../derive/src/index.js";
|
|
@@ -75,9 +75,23 @@ export async function articulate(
|
|
|
75
75
|
const found = bestHaloMate(ctx, h, voices, (v) => v.halo);
|
|
76
76
|
if (!found) continue;
|
|
77
77
|
const voice = found.item;
|
|
78
|
-
if (
|
|
78
|
+
if (
|
|
79
|
+
voice.node === s.payload || contains(ctx, voice.node, s.payload) ||
|
|
80
|
+
answers(ctx, voice.node, s.payload)
|
|
81
|
+
) {
|
|
79
82
|
continue;
|
|
80
83
|
}
|
|
84
|
+
// A form spanning the WHOLE answer is not a concept inside the answer to
|
|
85
|
+
// revoice — substituting it discards the answer and emits the asker's own
|
|
86
|
+
// words back, which is what a conversational store makes tempting: an
|
|
87
|
+
// answer and the question it answers keep maximal company, so the whole
|
|
88
|
+
// answer resonates with the whole query above any concept threshold
|
|
89
|
+
// (measured on the CONV fixture at 0.809 against 0.516). Articulation
|
|
90
|
+
// splices the asker's wording INTO an answer where the same concept
|
|
91
|
+
// appears; when the "concept" is the entire answer there is nothing left
|
|
92
|
+
// of it, and "where is it kept now" comes back in place of "it hangs in
|
|
93
|
+
// madrid". §5's contract is re-voicing, never replacement.
|
|
94
|
+
if (s.start === 0 && s.end === answer.length) continue;
|
|
81
95
|
substitutions.set(s.payload, voice.bytes);
|
|
82
96
|
}
|
|
83
97
|
if (substitutions.size === 0) {
|
package/src/mind/learning.ts
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// Learning is DEPOSITION: perceive a stream into a tree and intern every
|
|
4
4
|
// node. A fact is an EDGE between node ids; recall traverses edges.
|
|
5
5
|
|
|
6
|
-
import { Vec } from "../vec.js";
|
|
6
|
+
import { addInto, normalize, Vec, zeros } from "../vec.js";
|
|
7
7
|
import { bindSeat, companySignature, isChunk, Sema } from "../sema.js";
|
|
8
8
|
import type { Input, MindContext } from "./types.js";
|
|
9
9
|
import { changedNodes } from "./types.js";
|
|
@@ -15,6 +15,8 @@ import {
|
|
|
15
15
|
resolve,
|
|
16
16
|
} from "./primitives.js";
|
|
17
17
|
import { canonicalWindows, leafIdPrefix } from "./canonical.js";
|
|
18
|
+
import { hubBound } from "./traverse.js";
|
|
19
|
+
import { dominates } from "../geometry.js";
|
|
18
20
|
import { fold as foldVecs } from "../sema.js";
|
|
19
21
|
|
|
20
22
|
/** Intern a perceived tree into node ids, bottom-up, sharing equal subtrees.
|
|
@@ -233,6 +235,156 @@ export interface DepositReport {
|
|
|
233
235
|
continuationId?: number;
|
|
234
236
|
}
|
|
235
237
|
|
|
238
|
+
/** How many constituents one profile may VISIT. A partner's constituent tree
|
|
239
|
+
* is O(len/W) nodes, so an uncapped descent would make a pour cost grow with
|
|
240
|
+
* the partner's LENGTH — and a partner is a whole deposit, which may be a
|
|
241
|
+
* paragraph. The budget is what keeps a pour O(1) in the input, the property
|
|
242
|
+
* that lets {@link companyProfile} claim no new cost class. It binds only on
|
|
243
|
+
* long partners whose constituents are all corpus-unique; the descent's own
|
|
244
|
+
* stop rule (below) reaches recurring units far sooner on a trained store. */
|
|
245
|
+
const PROFILE_VISITS = 64;
|
|
246
|
+
|
|
247
|
+
/** The COMPANY PROFILE of a partner: its own identity signature superposed
|
|
248
|
+
* with the signatures of its RECURRING content-defined constituents.
|
|
249
|
+
*
|
|
250
|
+
* WHY THE WHOLE-PARTNER SIGNATURE ALONE IS NOT ENOUGH. The distributional
|
|
251
|
+
* hypothesis is a claim about TYPES ("occurs near a city name"), but a
|
|
252
|
+
* signature keyed on the whole partner's node id records a TOKEN ("occurred
|
|
253
|
+
* near node #4711992"). Two nodes are then distributional siblings only when
|
|
254
|
+
* their partners are the very same node — and a content-addressed store of
|
|
255
|
+
* natural language almost never repeats a whole deposit (measured on the
|
|
256
|
+
* trained store: whole-span dedup 0.98×, i.e. effectively none). So the
|
|
257
|
+
* halos of genuine synonyms came out quasi-orthogonal BY CONSTRUCTION: the
|
|
258
|
+
* best distributional sibling of "Eiffel Tower" scored 0.146 against a
|
|
259
|
+
* concept threshold of 0.516, with its own attested translations absent
|
|
260
|
+
* entirely, and the whole concept-hop / articulation / analogy layer was
|
|
261
|
+
* inert at corpus scale. (Re-verified under the store's OWN training seed:
|
|
262
|
+
* company signatures key on NODE ID, not the alphabet, so this reading is
|
|
263
|
+
* seed-independent and the figures are identical either way. Worth stating
|
|
264
|
+
* because a Mind built with a seed other than the store's makes every GIST
|
|
265
|
+
* comparison meaningless while leaving halo comparisons untouched.)
|
|
266
|
+
*
|
|
267
|
+
* WHY THE DESCENT MUST NOT STOP AT DEPTH 1. Reading only `rec.kids` does
|
|
268
|
+
* NOT deliver this. Cuts are content-defined over a rolling window, so a
|
|
269
|
+
* chunk boundary depends on the bytes AROUND a unit: "The Eiffel Tower is in
|
|
270
|
+
* Paris" folds to "The Eiffel " + "Tower is in Paris", and "Tour Eiffel dia
|
|
271
|
+
* any Paris" to "Tour Eiffel " + "dia any Paris". The shared unit "Paris"
|
|
272
|
+
* is a node in NEITHER — depth-1 profiles of that pair intersect in the
|
|
273
|
+
* EMPTY SET, and their halos measured 0.0319 against 0.0416 for an unrelated
|
|
274
|
+
* control: no signal at all. A depth-1 read merely moves the token problem
|
|
275
|
+
* from whole-partner identity down to top-level-chunk identity, which for
|
|
276
|
+
* full sentences is nearly as rare. Descending, the same pair shares
|
|
277
|
+
* " Paris" and "ffel " while the control still shares nothing — the units
|
|
278
|
+
* the distributional hypothesis is actually about.
|
|
279
|
+
*
|
|
280
|
+
* EVERY DEPTH CONTRIBUTES, AND THE RULE MUST NOT DEPEND ON ARRIVAL ORDER.
|
|
281
|
+
* The tempting stop rule — descend only while a constituent is corpus-unique,
|
|
282
|
+
* stop at the first unit attested in ≥ 2 forms — is wrong, and measurably so.
|
|
283
|
+
* Recurrence is a property of the corpus SO FAR: when the first of a pair is
|
|
284
|
+
* deposited its shared unit has fan-in 1, so the descent runs past it, and
|
|
285
|
+
* only the second partner ever profiles it. The pair then never meets
|
|
286
|
+
* (measured on the fixture above: 0.0165 against a 0.0375 control — still
|
|
287
|
+
* nothing). Whether two synonyms become distributional siblings cannot be
|
|
288
|
+
* allowed to depend on which was trained first. So the walk descends through
|
|
289
|
+
* EVERY constituent within its budget and superposes each one that is not a
|
|
290
|
+
* hub, at whatever depth it sits. A partner's own unique chunks contribute
|
|
291
|
+
* terms unique to that partner, which dilute but never mislead; the shared
|
|
292
|
+
* units contribute the signal.
|
|
293
|
+
*
|
|
294
|
+
* HUBS ARE THE ONE EXCLUSION, read LIMITed as `parentsFirst(n, bound+1)` —
|
|
295
|
+
* the store's own exact hub-or-not probe (a result longer than the bound
|
|
296
|
+
* means MORE than the bound), never a fan-in-sized read. A constituent with
|
|
297
|
+
* more than √N structural parents is scaffolding by §8.8's bound: " is ",
|
|
298
|
+
* "the ". Superposing it would put a term shared by every deposit into every
|
|
299
|
+
* profile, ALL halos would correlate, and the concept threshold's null model
|
|
300
|
+
* (unrelated halos at 0 ± 1/√D) that §4.1's hygiene note protects would
|
|
301
|
+
* collapse. It is still DESCENDED into — a hub chunk can contain a rare
|
|
302
|
+
* unit — but contributes nothing itself.
|
|
303
|
+
*
|
|
304
|
+
* Byte atoms are skipped in BOTH representations (a negative id and a stored
|
|
305
|
+
* kid-less node): an atom's fan-in is the alphabet's, so it can only ever
|
|
306
|
+
* read as a hub, and a short partner folding FLAT would otherwise put a
|
|
307
|
+
* handful of alphabet signatures into every profile — which is what silenced
|
|
308
|
+
* CAST's analogy gate in the first version of this function (measured:
|
|
309
|
+
* analogy strength 0.3636 -> 0.2004, "no halo-tier company evidence",
|
|
310
|
+
* test/29 C1).
|
|
311
|
+
*
|
|
312
|
+
* A FUNCTION OF THE NODE AND THE CORPUS STATE — stated precisely, because
|
|
313
|
+
* the weaker claim is the true one. The constituents are read from the
|
|
314
|
+
* STORE, never from the depositing tree's id map: that map holds only the
|
|
315
|
+
* nodes THIS deposit newly interned, so a partner met a second time yielded a
|
|
316
|
+
* profile missing exactly those constituents, the exact-partner case fell
|
|
317
|
+
* from cosine 1 to 1/√(1+k), and the geometry stopped meaning anything.
|
|
318
|
+
* Reading the store fixes that. It does NOT make the profile permanent: the
|
|
319
|
+
* hub test reads fan-in against √N and both grow with training, so a partner
|
|
320
|
+
* poured early and again late can profile differently. That residue is
|
|
321
|
+
* confined to the hub EXCLUSION — which terms are dropped as scaffolding —
|
|
322
|
+
* and never to which units are found, because the descent itself is now
|
|
323
|
+
* order-independent. The drift is one-directional and benign: a term can
|
|
324
|
+
* only ever go from contributing to being excluded as scaffolding. Replay of
|
|
325
|
+
* a fixed training order is bit-identical, so §2.1 holds. What must not be
|
|
326
|
+
* claimed is that a node's profile is fixed for all time; it is fixed given
|
|
327
|
+
* the corpus that has been seen.
|
|
328
|
+
*
|
|
329
|
+
* THE NULL MODEL IS OTHERWISE UNTOUCHED (§4.1). Every term is still a seeded
|
|
330
|
+
* function of a NODE IDENTITY, never a gist, so no byte-similarity between
|
|
331
|
+
* partners can leak content similarity into distributional similarity. The
|
|
332
|
+
* result is normalized, so ONE episode still pours ONE unit of mass:
|
|
333
|
+
* {@link Store.haloMass} keeps counting episodes and every mass-based
|
|
334
|
+
* reading is unchanged. Two partners sharing j of k discriminating
|
|
335
|
+
* constituents meet at j/(1+k) — graded evidence, above the 1/√D noise floor
|
|
336
|
+
* and below conceptThreshold until the overlap is most of the content, which
|
|
337
|
+
* is the semantics "same company" should have.
|
|
338
|
+
*
|
|
339
|
+
* Bounded: at most {@link PROFILE_VISITS} constituents are classified, each
|
|
340
|
+
* by ONE LIMITed structural-parent read, so a pour costs O(1) reads in the
|
|
341
|
+
* partner's size and performs no scan. */
|
|
342
|
+
function companyProfile(ctx: MindContext, id: number): Vec {
|
|
343
|
+
const acc = zeros(ctx.space.D);
|
|
344
|
+
addInto(acc, companySignature(ctx.space, id));
|
|
345
|
+
const bound = hubBound(ctx);
|
|
346
|
+
const W = ctx.space.maxGroup;
|
|
347
|
+
const whole = Math.max(1, ctx.store.contentLen(id));
|
|
348
|
+
const frontier: number[] = [];
|
|
349
|
+
const seen = new Set<number>([id]);
|
|
350
|
+
const descend = (n: number) => {
|
|
351
|
+
const kids = ctx.store.get(n)?.kids;
|
|
352
|
+
if (!kids) return;
|
|
353
|
+
for (const kid of kids) if (!seen.has(kid)) frontier.push(kid);
|
|
354
|
+
};
|
|
355
|
+
descend(id);
|
|
356
|
+
for (let visits = 0; visits < PROFILE_VISITS && frontier.length > 0;) {
|
|
357
|
+
const n = frontier.shift()!;
|
|
358
|
+
if (seen.has(n)) continue;
|
|
359
|
+
seen.add(n);
|
|
360
|
+
visits++;
|
|
361
|
+
// Atoms in both representations — negative id, or a stored kid-less node.
|
|
362
|
+
if (n < 0 || ctx.store.get(n)?.kids == null) continue;
|
|
363
|
+
descend(n);
|
|
364
|
+
const len = ctx.store.contentLen(n, whole);
|
|
365
|
+
if (len < W || dominates(len, whole)) continue;
|
|
366
|
+
// MINIMAL units only: a constituent that still has a constituent of its
|
|
367
|
+
// own at or above W is a composite, and superposing it as well as its
|
|
368
|
+
// parts would count the same content twice. Nested partners — an
|
|
369
|
+
// accumulated conversation, where turn k's context is a prefix of turn
|
|
370
|
+
// k+1's — share their large chunks structurally rather than
|
|
371
|
+
// distributionally, so those composites are exactly the terms that make
|
|
372
|
+
// adjacent turns read as synonyms (measured: consecutive turns at 0.809
|
|
373
|
+
// and 0.740 against a 0.516 concept threshold). The smallest units at or
|
|
374
|
+
// above the fold's own window are the word-sized types company should be
|
|
375
|
+
// keyed at.
|
|
376
|
+
const kids = ctx.store.get(n)!.kids!;
|
|
377
|
+
let composite = false;
|
|
378
|
+
for (const kid of kids) {
|
|
379
|
+
if (kid >= 0 && ctx.store.contentLen(kid, W) >= W) composite = true;
|
|
380
|
+
}
|
|
381
|
+
if (composite) continue;
|
|
382
|
+
if (ctx.store.parentsFirst(n, bound + 1).length > bound) continue;
|
|
383
|
+
addInto(acc, companySignature(ctx.space, n));
|
|
384
|
+
}
|
|
385
|
+
return normalize(acc);
|
|
386
|
+
}
|
|
387
|
+
|
|
236
388
|
/** Ingest a pair (context, continuation) — learn an edge and pour halos.
|
|
237
389
|
* Returns the deposited root ids (context, continuation) — a pure
|
|
238
390
|
* read-out; callers that ignore it behave exactly as before. */
|
|
@@ -255,14 +407,17 @@ export async function ingestPair(
|
|
|
255
407
|
await propagateSuffixes(ctx, ctxId, contId);
|
|
256
408
|
|
|
257
409
|
// Halos pour company SIGNATURES (identity), not gists (content) — see
|
|
258
|
-
// companySignature in sema.ts
|
|
259
|
-
|
|
410
|
+
// companySignature in sema.ts — as a TYPE-level profile: the partner's own
|
|
411
|
+
// signature superposed with its discriminating constituents' (see
|
|
412
|
+
// companyProfile), so company is shared by what partners are MADE OF and
|
|
413
|
+
// not only by partner identity.
|
|
414
|
+
const contSeat = bindSeat(ctx.space, companyProfile(ctx, contId), 1);
|
|
260
415
|
for (const part of c.changed) {
|
|
261
416
|
const partId = c.ids.get(part)!;
|
|
262
417
|
await ctx.store.pourHalo(partId, contSeat);
|
|
263
418
|
await ctx.store.pourHalo(
|
|
264
419
|
contId,
|
|
265
|
-
bindSeat(ctx.space,
|
|
420
|
+
bindSeat(ctx.space, companyProfile(ctx, partId), 0),
|
|
266
421
|
);
|
|
267
422
|
}
|
|
268
423
|
return { ctxId, contId };
|
package/src/mind/traverse.ts
CHANGED
|
@@ -558,6 +558,35 @@ export function contains(
|
|
|
558
558
|
return false;
|
|
559
559
|
}
|
|
560
560
|
|
|
561
|
+
/** Whether a continuation edge joins the two forms, in either direction —
|
|
562
|
+
* the EXACT half's veto on calling them synonyms.
|
|
563
|
+
*
|
|
564
|
+
* Halos measure company, and the strongest company any two forms can keep is
|
|
565
|
+
* standing next to each other: a question and its answer co-occur in every
|
|
566
|
+
* episode that taught the pair, so their halos SHOULD be similar, and on a
|
|
567
|
+
* conversational store they are (measured on the CONV fixture: consecutive
|
|
568
|
+
* turns at 0.809 against a 0.516 concept threshold). A gate reading halo
|
|
569
|
+
* cosine alone therefore reads adjacency as synonymy and revoices an answer
|
|
570
|
+
* in the words of the question it answers — "it hangs in madrid" spliced back
|
|
571
|
+
* into "where is it kept now". The distributional layer cannot tell the two
|
|
572
|
+
* relations apart, because to it they are the same observation; the exact
|
|
573
|
+
* half can, for free, because it stored the edge. §4.1's division of labour
|
|
574
|
+
* exactly: approximate proposes, exact decides.
|
|
575
|
+
*
|
|
576
|
+
* Read LIMITed in both directions at the hub bound — a common continuation's
|
|
577
|
+
* fan-in is corpus-sized, and no single decision may scale with it. */
|
|
578
|
+
export function answers(
|
|
579
|
+
ctx: MindContext,
|
|
580
|
+
a: number,
|
|
581
|
+
b: number,
|
|
582
|
+
): boolean {
|
|
583
|
+
const bound = hubBound(ctx);
|
|
584
|
+
if (ctx.store.hasNext(a) && ctx.store.nextFirst(a, bound).includes(b)) {
|
|
585
|
+
return true;
|
|
586
|
+
}
|
|
587
|
+
return ctx.store.hasNext(b) && ctx.store.nextFirst(b, bound).includes(a);
|
|
588
|
+
}
|
|
589
|
+
|
|
561
590
|
// ── Edge disambiguation (Section 6) ──────────────────────────────────────
|
|
562
591
|
|
|
563
592
|
/** The best-scoring item by cosine against `query`, among items scoring at
|
|
@@ -0,0 +1,342 @@
|
|
|
1
|
+
// 76 — TYPE-LEVEL COMPANY (the halo pour's constituent profile).
|
|
2
|
+
//
|
|
3
|
+
// What is under test is ONE claim: two forms become distributional siblings
|
|
4
|
+
// when their partners are MADE OF a shared content unit, even though the
|
|
5
|
+
// partners share no node at the top of the fold. That is the difference
|
|
6
|
+
// between a halo keyed on a TOKEN ("occurred next to node #4711992") and one
|
|
7
|
+
// keyed on a TYPE ("occurred next to something containing 'Paris'").
|
|
8
|
+
//
|
|
9
|
+
// WHY THESE TESTS CANNOT PASS BY ACCIDENT. Every positive assertion is paired
|
|
10
|
+
// with a NEGATIVE CONTROL drawn from the same fixture, trained in the same
|
|
11
|
+
// store, of the same shape and comparable length — so a change that merely
|
|
12
|
+
// made all halos correlate (the null model collapsing) fails the control
|
|
13
|
+
// instead of passing the positive. Every bar is DERIVED from D, never tuned:
|
|
14
|
+
// `significanceBar` (3/√D) for "this is not chance" and `conceptThreshold` for
|
|
15
|
+
// "these are the same concept". And T1 computes, in-test, the fact that makes
|
|
16
|
+
// the whole file a test of the constituent descent rather than of halos in
|
|
17
|
+
// general: the two partners' depth-1 constituent sets are DISJOINT, so a
|
|
18
|
+
// profile reading only `rec.kids` has literally nothing in common to find.
|
|
19
|
+
//
|
|
20
|
+
// WHAT EACH TEST IS, STATED HONESTLY. T1 and T2 are the CAPABILITY tests:
|
|
21
|
+
// run against the previous depth-1 profile they fail, on the capability
|
|
22
|
+
// assertion itself and not on a precondition — measured -0.0086 against the
|
|
23
|
+
// 0.0938 bar, while the fixture's own preconditions still passed, so the
|
|
24
|
+
// failure is the missing capability and nothing else. T3, T4 and T5 pass
|
|
25
|
+
// under BOTH implementations by construction: they are not evidence for the
|
|
26
|
+
// capability, they are the invariants it must not buy itself with, and each
|
|
27
|
+
// one pins a regression this work actually hit — the null model collapsing
|
|
28
|
+
// when the descent superposed scaffolding, mass tracking constituents instead
|
|
29
|
+
// of episodes, and the profile being read from the deposit's id map instead
|
|
30
|
+
// of the store. Claiming all five as proof of the capability would be false;
|
|
31
|
+
// dropping the three would leave the two unfalsifiable.
|
|
32
|
+
//
|
|
33
|
+
// WHAT IS DELIBERATELY NOT TESTED HERE, AND WHY. A sixth test asserting that
|
|
34
|
+
// company GRADES with shared content (more shared units => more company) was
|
|
35
|
+
// written and removed: it is false as stated. Cosine normalizes by profile
|
|
36
|
+
// size, so a pair sharing 3 constituents out of a larger profile scores BELOW
|
|
37
|
+
// a pair sharing 1 out of a smaller one — measured at 0.085 against 0.111,
|
|
38
|
+
// consistently across all four seeds. The design's own `shared / (1 + k)`
|
|
39
|
+
// reading is size-RELATIVE, and asserting the absolute form encodes a law the
|
|
40
|
+
// system does not obey. Testing the relative form from outside would require
|
|
41
|
+
// re-deriving the profile's term-selection rules inside the test, which makes
|
|
42
|
+
// the test a mirror of the implementation and worthless as a check on it.
|
|
43
|
+
//
|
|
44
|
+
// Also untested, and a real limitation rather than an oversight: on a store
|
|
45
|
+
// this small the hub bound √N is large enough that frame scaffolding is not
|
|
46
|
+
// excluded, so short partners sharing only a frame do keep some company. That
|
|
47
|
+
// is the documented honest floor — a corpus that cannot yet say what
|
|
48
|
+
// discriminates — but it means these fixtures must share genuine CONTENT
|
|
49
|
+
// units, which T1 now asserts rather than assumes.
|
|
50
|
+
//
|
|
51
|
+
// Each test pins a DIFFERENT rule. Deleting any one of them lets a specific,
|
|
52
|
+
// named regression back in; none of them subsumes another.
|
|
53
|
+
|
|
54
|
+
import { test } from "node:test";
|
|
55
|
+
import assert from "node:assert/strict";
|
|
56
|
+
import {
|
|
57
|
+
conceptThreshold,
|
|
58
|
+
cosine,
|
|
59
|
+
Mind,
|
|
60
|
+
significanceBar,
|
|
61
|
+
} from "../dist/src/index.js";
|
|
62
|
+
|
|
63
|
+
const D = 1024;
|
|
64
|
+
const BAR = significanceBar(D); // 3/√D — above chance
|
|
65
|
+
const enc = new TextEncoder();
|
|
66
|
+
|
|
67
|
+
// Company signatures key on NODE ID, not on the alphabet, so a seeded
|
|
68
|
+
// keyring cannot be what makes these comparisons come out — but that is an
|
|
69
|
+
// argument, and the capability tests below check it instead, across seeds.
|
|
70
|
+
const SEEDS = [7, 1, 42, 99];
|
|
71
|
+
const newMind = (seed = 7) => new Mind({ seed, D });
|
|
72
|
+
const idOf = (m, s) => m.resolve(enc.encode(s));
|
|
73
|
+
const haloOf = (m, s) => {
|
|
74
|
+
const id = idOf(m, s);
|
|
75
|
+
return id === null || id === undefined ? null : m.store.halo(id);
|
|
76
|
+
};
|
|
77
|
+
|
|
78
|
+
/** The halo read must work at all before any comparison means anything. A
|
|
79
|
+
* missing halo silently makes every cosine below unreachable, and a broken
|
|
80
|
+
* one makes them meaningless — this is the control whose absence has voided
|
|
81
|
+
* whole investigations on this codebase. */
|
|
82
|
+
const assertHaloControl = (m, cues) => {
|
|
83
|
+
for (const c of cues) {
|
|
84
|
+
const h = haloOf(m, c);
|
|
85
|
+
assert.ok(h, `CONTROL: no halo poured for ${JSON.stringify(c)}`);
|
|
86
|
+
assert.ok(
|
|
87
|
+
Math.abs(cosine(h, h) - 1) < 1e-9,
|
|
88
|
+
`CONTROL: halo of ${JSON.stringify(c)} is not self-identical`,
|
|
89
|
+
);
|
|
90
|
+
}
|
|
91
|
+
};
|
|
92
|
+
|
|
93
|
+
/** The continuation a cue was trained with, as a node id. */
|
|
94
|
+
const partnerOf = (m, cue) => m.store.next(idOf(m, cue))[0];
|
|
95
|
+
|
|
96
|
+
/** Depth-1 constituents — what a profile reading only `rec.kids` would see. */
|
|
97
|
+
const depth1 = (m, node) => new Set(m.store.get(node)?.kids ?? []);
|
|
98
|
+
|
|
99
|
+
/** Every constituent reachable below `node`, to a depth the fold cannot
|
|
100
|
+
* exceed for these fixtures — what the descent can see. */
|
|
101
|
+
const deepConstituents = (m, node, depth = 6, out = new Set()) => {
|
|
102
|
+
if (depth === 0) return out;
|
|
103
|
+
for (const k of m.store.get(node)?.kids ?? []) {
|
|
104
|
+
if (k < 0) continue;
|
|
105
|
+
out.add(k);
|
|
106
|
+
deepConstituents(m, k, depth - 1, out);
|
|
107
|
+
}
|
|
108
|
+
return out;
|
|
109
|
+
};
|
|
110
|
+
|
|
111
|
+
const intersect = (a, b) => [...a].filter((x) => b.has(x));
|
|
112
|
+
|
|
113
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
114
|
+
// T1 — THE CAPABILITY, with its own impossibility proof for the old rule.
|
|
115
|
+
//
|
|
116
|
+
// Two sentences in different languages that both mention Paris. Content-
|
|
117
|
+
// defined cuts put the shared unit in DIFFERENT top-level chunks:
|
|
118
|
+
// "The Eiffel Tower is in Paris" -> "The Eiffel " · "Tower is in Paris"
|
|
119
|
+
// "Tour Eiffel dia any Paris" -> "Tour Eiffel " · "dia any Paris"
|
|
120
|
+
// so their depth-1 constituents are disjoint — asserted below, not assumed.
|
|
121
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
122
|
+
test("T1: partners sharing a unit BELOW the top of the fold keep company", async () => {
|
|
123
|
+
for (const seed of SEEDS) {
|
|
124
|
+
const m = newMind(seed);
|
|
125
|
+
await m.ingest([
|
|
126
|
+
["cue_en", "The Eiffel Tower is in Paris"],
|
|
127
|
+
["cue_mg", "Tour Eiffel dia any Paris"],
|
|
128
|
+
["cue_zz", "Bananas are grown in humid climates"],
|
|
129
|
+
]);
|
|
130
|
+
assertHaloControl(m, ["cue_en", "cue_mg", "cue_zz"]);
|
|
131
|
+
|
|
132
|
+
const pEn = partnerOf(m, "cue_en");
|
|
133
|
+
const pMg = partnerOf(m, "cue_mg");
|
|
134
|
+
|
|
135
|
+
// THE IMPOSSIBILITY PROOF. A profile built from `rec.kids` alone sees these
|
|
136
|
+
// sets and nothing else; they do not intersect, so no depth-1 rule — however
|
|
137
|
+
// weighted, however filtered — can make these two partners share a term.
|
|
138
|
+
// This test therefore measures the DESCENT, not halos in general.
|
|
139
|
+
assert.equal(
|
|
140
|
+
intersect(depth1(m, pEn), depth1(m, pMg)).length,
|
|
141
|
+
0,
|
|
142
|
+
"fixture no longer exercises the descent: the two partners now share a " +
|
|
143
|
+
"depth-1 constituent, so a depth-1 profile could pass T1 as well",
|
|
144
|
+
);
|
|
145
|
+
// And the units the descent is supposed to find must actually be there —
|
|
146
|
+
// AND be eligible to become profile terms. A fixture whose only shared
|
|
147
|
+
// constituents are sub-window shards or frame scaffolding measures frame
|
|
148
|
+
// similarity while reading like a content test; one was written during this
|
|
149
|
+
// work and passed for exactly that wrong reason. The shared unit must be
|
|
150
|
+
// at least the fold's own window wide.
|
|
151
|
+
const W = m.space.maxGroup;
|
|
152
|
+
const shared = intersect(
|
|
153
|
+
deepConstituents(m, pEn),
|
|
154
|
+
deepConstituents(m, pMg),
|
|
155
|
+
);
|
|
156
|
+
assert.ok(
|
|
157
|
+
shared.some((n) => m.store.contentLen(n, W) >= W),
|
|
158
|
+
`fixture is broken: the partners share no constituent of at least W=${W} ` +
|
|
159
|
+
`bytes, so nothing they share can enter a profile`,
|
|
160
|
+
);
|
|
161
|
+
|
|
162
|
+
const related = cosine(haloOf(m, "cue_en"), haloOf(m, "cue_mg"));
|
|
163
|
+
const control = cosine(haloOf(m, "cue_en"), haloOf(m, "cue_zz"));
|
|
164
|
+
|
|
165
|
+
assert.ok(
|
|
166
|
+
related >= BAR,
|
|
167
|
+
`seed ${seed}: partners sharing a content unit must keep measurable ` +
|
|
168
|
+
`company: got ${related.toFixed(4)}, need >= ${
|
|
169
|
+
BAR.toFixed(4)
|
|
170
|
+
} (3/sqrt(D))`,
|
|
171
|
+
);
|
|
172
|
+
// The control is what makes the line above falsifiable: without it, a
|
|
173
|
+
// regression that made EVERY halo correlate would pass.
|
|
174
|
+
assert.ok(
|
|
175
|
+
control < BAR,
|
|
176
|
+
`seed ${seed}: partners sharing nothing must stay at chance: got ` +
|
|
177
|
+
`${control.toFixed(4)}, need < ${
|
|
178
|
+
BAR.toFixed(4)
|
|
179
|
+
} — null model collapsed`,
|
|
180
|
+
);
|
|
181
|
+
await m.store.close();
|
|
182
|
+
}
|
|
183
|
+
});
|
|
184
|
+
|
|
185
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
186
|
+
// T2 — ORDER INDEPENDENCE.
|
|
187
|
+
//
|
|
188
|
+
// The tempting stop rule ("descend while a constituent is corpus-unique, stop
|
|
189
|
+
// at the first one attested twice") passes T1 in exactly one training order
|
|
190
|
+
// and fails in the other: when the FIRST partner is poured its shared unit has
|
|
191
|
+
// fan-in 1, so the descent runs past it and only the second partner ever
|
|
192
|
+
// profiles it. Whether two forms become siblings must not depend on which was
|
|
193
|
+
// taught first.
|
|
194
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
195
|
+
test("T2: company does not depend on which partner was taught first", async () => {
|
|
196
|
+
const measure = async (first, second, seed) => {
|
|
197
|
+
const m = newMind(seed);
|
|
198
|
+
await m.ingest([
|
|
199
|
+
[first[0], first[1]],
|
|
200
|
+
[second[0], second[1]],
|
|
201
|
+
["cue_zz", "Bananas are grown in humid climates"],
|
|
202
|
+
]);
|
|
203
|
+
assertHaloControl(m, ["cue_en", "cue_mg", "cue_zz"]);
|
|
204
|
+
return {
|
|
205
|
+
related: cosine(haloOf(m, "cue_en"), haloOf(m, "cue_mg")),
|
|
206
|
+
control: cosine(haloOf(m, "cue_en"), haloOf(m, "cue_zz")),
|
|
207
|
+
};
|
|
208
|
+
};
|
|
209
|
+
const EN = ["cue_en", "The Eiffel Tower is in Paris"];
|
|
210
|
+
const MG = ["cue_mg", "Tour Eiffel dia any Paris"];
|
|
211
|
+
|
|
212
|
+
for (const seed of SEEDS) {
|
|
213
|
+
const forward = await measure(EN, MG, seed);
|
|
214
|
+
const reverse = await measure(MG, EN, seed);
|
|
215
|
+
|
|
216
|
+
for (const [name, r] of [["forward", forward], ["reverse", reverse]]) {
|
|
217
|
+
assert.ok(
|
|
218
|
+
r.related >= BAR,
|
|
219
|
+
`seed ${seed} ${name} order: shared-unit company must survive ` +
|
|
220
|
+
`training order — got ${r.related.toFixed(4)}, need >= ` +
|
|
221
|
+
`${BAR.toFixed(4)}`,
|
|
222
|
+
);
|
|
223
|
+
assert.ok(
|
|
224
|
+
r.control < BAR,
|
|
225
|
+
`seed ${seed} ${name} order: control must stay at chance, got ` +
|
|
226
|
+
`${r.control.toFixed(4)}`,
|
|
227
|
+
);
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
});
|
|
231
|
+
|
|
232
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
233
|
+
// T3 — THE NULL MODEL SURVIVES.
|
|
234
|
+
//
|
|
235
|
+
// The failure mode opposite to T1's: a descent that superposed everything it
|
|
236
|
+
// walked past would put terms shared by every deposit into every profile, and
|
|
237
|
+
// ALL halos would correlate. That regression passes T1 handsomely. Here a
|
|
238
|
+
// population of mutually unrelated partners must stay mutually at chance —
|
|
239
|
+
// and, because it is the same store, T1's positive is re-checked against this
|
|
240
|
+
// population's own noise level rather than against a bar alone.
|
|
241
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
242
|
+
test("T3: unrelated partners stay mutually at chance", async () => {
|
|
243
|
+
const m = newMind();
|
|
244
|
+
const FACTS = [
|
|
245
|
+
["c1", "Volcanoes erupt when magma reaches the surface"],
|
|
246
|
+
["c2", "The violin has four strings tuned in fifths"],
|
|
247
|
+
["c3", "Penguins are flightless birds of the southern seas"],
|
|
248
|
+
["c4", "Concrete gains strength for weeks after it is poured"],
|
|
249
|
+
["c5", "The abacus was used for arithmetic in many cultures"],
|
|
250
|
+
["c6", "Lightning heats the air it passes through"],
|
|
251
|
+
];
|
|
252
|
+
await m.ingest(FACTS);
|
|
253
|
+
assertHaloControl(m, FACTS.map((f) => f[0]));
|
|
254
|
+
|
|
255
|
+
let worst = -1, worstPair = "";
|
|
256
|
+
for (let i = 0; i < FACTS.length; i++) {
|
|
257
|
+
for (let j = i + 1; j < FACTS.length; j++) {
|
|
258
|
+
const c = cosine(haloOf(m, FACTS[i][0]), haloOf(m, FACTS[j][0]));
|
|
259
|
+
if (c > worst) {
|
|
260
|
+
worst = c;
|
|
261
|
+
worstPair = `${FACTS[i][0]}~${FACTS[j][0]}`;
|
|
262
|
+
}
|
|
263
|
+
}
|
|
264
|
+
}
|
|
265
|
+
assert.ok(
|
|
266
|
+
worst < BAR,
|
|
267
|
+
`unrelated partners must not keep company: worst pair ${worstPair} at ` +
|
|
268
|
+
`${worst.toFixed(4)}, need < ${BAR.toFixed(4)}. A profile that ` +
|
|
269
|
+
`superposes scaffolding makes every halo correlate.`,
|
|
270
|
+
);
|
|
271
|
+
});
|
|
272
|
+
|
|
273
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
274
|
+
// T4 — ONE EPISODE POURS ONE UNIT OF MASS.
|
|
275
|
+
//
|
|
276
|
+
// The profile is normalized precisely so that enriching it cannot inflate the
|
|
277
|
+
// evidence it represents: `haloMass` counts EPISODES, and every mass-based
|
|
278
|
+
// reading in the system (recall's corroboration counts, the disambiguation
|
|
279
|
+
// tiers) depends on that staying true. A profile that forgot to normalize
|
|
280
|
+
// would pass T1 and T3 and silently re-weight the whole distributional layer.
|
|
281
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
282
|
+
test("T4: enriching the profile does not inflate halo mass", async () => {
|
|
283
|
+
const m = newMind();
|
|
284
|
+
await m.ingest([
|
|
285
|
+
// A partner with MANY constituents, and one with very few — if mass
|
|
286
|
+
// tracked constituent count instead of episodes, these would differ.
|
|
287
|
+
["rich", "The quick brown fox jumps over the lazy dog beside the river"],
|
|
288
|
+
["lean", "Ice melts"],
|
|
289
|
+
]);
|
|
290
|
+
assertHaloControl(m, ["rich", "lean"]);
|
|
291
|
+
|
|
292
|
+
const massRich = m.store.haloMass(idOf(m, "rich"));
|
|
293
|
+
const massLean = m.store.haloMass(idOf(m, "lean"));
|
|
294
|
+
assert.equal(
|
|
295
|
+
massRich,
|
|
296
|
+
massLean,
|
|
297
|
+
`halo mass must count episodes, not constituents: a 59-byte partner ` +
|
|
298
|
+
`poured ${massRich} against a 9-byte partner's ${massLean}`,
|
|
299
|
+
);
|
|
300
|
+
assert.equal(massRich, 1, `one episode must pour exactly one unit of mass`);
|
|
301
|
+
});
|
|
302
|
+
|
|
303
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
304
|
+
// T5 — THE PROFILE IS A FUNCTION OF THE STORE, NOT OF THE DEPOSIT.
|
|
305
|
+
//
|
|
306
|
+
// The constituents must be read from the STORE. Read instead from the
|
|
307
|
+
// depositing tree's id map — which holds only the nodes THIS deposit newly
|
|
308
|
+
// interned — and a partner met a SECOND time profiles differently from the
|
|
309
|
+
// first, because its subtrees are already stored and therefore absent from the
|
|
310
|
+
// map. The exact-partner case then falls from cosine 1 to 1/sqrt(1+k) and the
|
|
311
|
+
// geometry stops meaning anything. Two cues sharing the SAME partner are the
|
|
312
|
+
// direct probe: their halos must be identical, whatever else changed between
|
|
313
|
+
// the two deposits.
|
|
314
|
+
// ═══════════════════════════════════════════════════════════════════════════
|
|
315
|
+
test("T5: the same partner profiles identically on every episode", async () => {
|
|
316
|
+
const m = newMind();
|
|
317
|
+
const PARTNER = "Paris is the capital city of France";
|
|
318
|
+
await m.ingest([
|
|
319
|
+
["first", PARTNER],
|
|
320
|
+
// An unrelated deposit in between, so the second pour happens against a
|
|
321
|
+
// store that has grown and a tree whose subtrees are all already interned.
|
|
322
|
+
["filler", "Sandstone forms from compressed grains"],
|
|
323
|
+
["second", PARTNER],
|
|
324
|
+
]);
|
|
325
|
+
assertHaloControl(m, ["first", "second"]);
|
|
326
|
+
|
|
327
|
+
const same = cosine(haloOf(m, "first"), haloOf(m, "second"));
|
|
328
|
+
assert.ok(
|
|
329
|
+
same > 1 - 1e-6,
|
|
330
|
+
`two cues sharing one partner must have identical halos: got ` +
|
|
331
|
+
`${same.toFixed(6)}. The profile is being read from the deposit's id ` +
|
|
332
|
+
`map rather than from the store.`,
|
|
333
|
+
);
|
|
334
|
+
// Falsifiability: identical halos must not be an artefact of ALL halos in
|
|
335
|
+
// this store being identical.
|
|
336
|
+
const different = cosine(haloOf(m, "first"), haloOf(m, "filler"));
|
|
337
|
+
assert.ok(
|
|
338
|
+
different < conceptThreshold(D),
|
|
339
|
+
`control: a different partner must not yield the same halo, got ` +
|
|
340
|
+
`${different.toFixed(4)}`,
|
|
341
|
+
);
|
|
342
|
+
});
|