@hviana/sema 0.5.3 → 0.5.4

package/AGENTS.md

@@ -179,12 +179,12 @@ matchers (`locate`, `alignRuns`, `alignGraded`, `alignAround`/`frameSlots`,
 `bestHaloMate`, `haloSiblings`, `analogyStrength` and its structural tier
 `sharedFrameStrength`, `spanHalo`, `spanSynonymStrength`), projections
 (`follow`, `reverseContext`, `project`, `conceptHop`), the span-shape family
-(`skillExemplar`, `isSpanShaped`, `containsSpan`), and the two STRUCTURAL gates
-that are byte predicates rather than derived thresholds (`isSpanShaped`,
-`carriesFillers`). `mind/traverse.ts` owns the graph readings (`edgeAncestors`,
-`reachOf`, `chooseNext`/`chooseAmong`, `guidedFirst`, `leadsSomewhere`,
-`allWindowsAreScaffolding`, `formsOpenedBy`) and the corpus scale (`corpusN`,
-`hubBound`, `hubCap`, `atomReach`).
+(`skillExemplar`, `isSpanShaped`, `containsSpan`), and the STRUCTURAL gates that
+are byte predicates rather than derived thresholds (`isSpanShaped`,
+`carriesFillers`, `voicesDisplacedFiller`). `mind/traverse.ts` owns the graph
+readings (`edgeAncestors`, `reachOf`, `chooseNext`/`chooseAmong`, `guidedFirst`,
+`leadsSomewhere`, `allWindowsAreScaffolding`, `formsOpenedBy`) and the corpus
+scale (`corpusN`, `hubBound`, `hubCap`, `atomReach`).
 
 _Follow it:_ before writing a new generalising mechanism, express it as a
 (matcher, direction, gate) triple. If those already exist, the mechanism is a

package/HOW_IT_WORKS.md

@@ -4314,6 +4314,22 @@ The tier accounts for exactly the query span whose evidence carried the winning
 point of attention (§17.6's `start`–`end`), not the whole query — a consensus
 vote for "ice" among scaffolding does not explain the word "steel". One CONCEPT.
 
+One further refusal is specific to grounding _through an anchor_. This tier
+voices the anchor's continuation, so when the query is the same structure as the
+anchor with one position filled differently, that continuation speaks the
+**anchor's** occupant of the position — fluent, specific, and about the wrong
+thing (`How do I compile main.c?` answered `Run gcc hello.c`). A displaced slot
+alone does not license the refusal, because it cannot separate a different
+_thing_ from a different _word_: `symbol` ← `formula` has exactly the shape
+`main` ← `hello` does, and refusing it silences a correct answer phrased in the
+corpus's own words. The discriminator is whether the projection still shares
+perceivable content with the query outside that position — one river window is
+enough. `The chemical formula for water is H2O.` shares `for water`;
+`Run gcc hello.c` shares only `.c`, two bytes, below the scale at which overlap
+stops being chance. The gate refuses only; putting the asker's own occupant in
+its place is a stronger claim needing the corpus's carriage evidence, which is
+§20.5's business.
+
 ### 21.4 Tier 3 — the nearest grounded hit, at the query-relative fraction
 
 Walk the hits nearest-first and ground the first whose grounding explains enough

package/jsr.json

@@ -1,6 +1,6 @@
 {
   "$schema": "https://jsr.io/schema/config-file.v1.json",
   "name": "@hviana/sema",
-  "version": "0.5.3",
+  "version": "0.5.4",
   "exports": "./src/index.ts"
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hviana/sema",
-  "version": "0.5.3",
+  "version": "0.5.4",
   "description": "Sema: a non-parametric, instance-based reasoning system.",
   "type": "module",
   "main": "dist/src/index.js",

package/src/mind/match.ts

@@ -612,6 +612,87 @@ export function frameSlots(
   return { id, slots, matched: spans, covered };
 }
 
+/** THE DISPLACED-FILLER GATE: does `projection` speak the ANCHOR's occupant of
+ *  a position the query fills differently?
+ *
+ *  A mechanism grounding through an anchor voices that anchor's continuation.
+ *  When the query is the same structure as the anchor with one position filled
+ *  differently — a different filename, a different word — the anchor's
+ *  continuation is ABOUT THE ANCHOR'S occupant, and voicing it answers a
+ *  question the asker did not ask.  It is worse than silence, because it is
+ *  fluent and specific and wrong:
+ *
+ *      trained  `How do I compile hello.c?` -> `Run gcc hello.c`
+ *      asked    `How do I compile main.c?`
+ *      voiced   `Run gcc hello.c`            <- the corpus's file, not the asker's
+ *
+ *  The same shape on the trained 15.7M-node store: `How do you say 'flurbish'
+ *  in French?` answers "the way to say hello is \"Bonjour\"".
+ *
+ *  THIS IS NOT THE RESTATED-FRAGMENT GUARD.  That one asks whether the
+ *  projection is a piece of the QUERY; this asks whether it is a piece of the
+ *  ANCHOR that the query displaced.  Neither implies the other, and the
+ *  observed failures pass the restatement guard cleanly.
+ *
+ *  Three conditions, all byte-exact and all necessary:
+ *
+ *  1. the query and the anchor must be ONE STRUCTURE — what they share has to
+ *     dominate the query, or the query is not a variant of the anchor at all
+ *     and the anchor's occupant of anything is beside the point;
+ *  2. both sides of the position must reach one river window — below it byte
+ *     overlap is chance, not evidence (the floor identityBar and the bridge's
+ *     attestedQ both draw);
+ *  3. the projection must voice the anchor's filler and NOT the query's
+ *     referent.  Voicing both is a projection that carried the asker's own
+ *     occupant through, which is exactly what a licensed reference does and
+ *     must stay allowed;
+ *  4. and the projection must share NO perceivable content with the query
+ *     outside that position — no run of one river window.
+ *
+ *  GATE 4 IS WHAT SEPARATES A DIFFERENT THING FROM A DIFFERENT WORD, and
+ *  without it this refuses correct answers.  A displaced slot alone cannot
+ *  tell them apart: `symbol` <- `formula` and `main` <- `hello` are the same
+ *  shape to the matcher — one substitution slot, frame dominating.  Measured
+ *  on the trained store, gates 1-3 alone silenced
+ *
+ *      Q `What is the chemical symbol for water?`
+ *      A `The chemical formula for water is H2O.`
+ *
+ *  which is right, and merely phrased in the corpus's own words.  The answer
+ *  shares `the chemical ` and ` for water` with the question, so it is plainly
+ *  about what was asked.  `Run gcc hello.c` against `How do I compile main.c?`
+ *  shares nothing but `.c` — two bytes, below the window where overlap stops
+ *  being chance — so it is not about what was asked at all.  No new constant:
+ *  W is the same floor identityBar, attestedQ and the site test already draw. */
+export function voicesDisplacedFiller(
+  ctx: MindContext,
+  query: Uint8Array,
+  anchor: Uint8Array,
+  projection: Uint8Array,
+): boolean {
+  const W = ctx.space.maxGroup;
+  const inst = frameSlots(ctx, query, anchor, 0);
+  if (inst === null) return false;
+  if (!dominates(inst.covered, query.length)) return false;
+  for (const slot of inst.slots) {
+    if (slot.kind !== "substitution") continue;
+    if (slot.qe - slot.qs < W || slot.filler.length < W) continue;
+    if (indexOf(projection, slot.filler, 0) < 0) continue;
+    const referent = query.subarray(slot.qs, slot.qe);
+    if (indexOf(projection, referent, 0) >= 0) continue;
+    // Gate 4: does the projection still speak about the query's FRAME?  A run
+    // of one window anywhere outside the displaced position is enough — the
+    // answer is then about the thing that was asked, in the corpus's own
+    // wording.  Sharing nothing means it is about something else.
+    const shared = alignRuns(ctx, query, projection).some((r) =>
+      r.qe - r.qs >= W && (r.qe <= slot.qs || r.qs >= slot.qe)
+    );
+    if (shared) continue;
+    return true;
+  }
+  return false;
+}
+
 /** Whether every member is byte-distinct from the others. */
 export function distinct(items: readonly Uint8Array[]): boolean {
   for (let i = 0; i < items.length; i++) {

package/src/mind/mechanisms/recall.ts

@@ -17,7 +17,12 @@ import type { MindContext } from "../types.js";
 import { gistOf, read, resolve } from "../primitives.js";
 import { bytesEqual, indexOf } from "../../bytes.js";
 import { allWindowsAreScaffolding, corpusN, hubBound } from "../traverse.js";
-import { follow, project, reverseContext } from "../match.js";
+import {
+  follow,
+  project,
+  reverseContext,
+  voicesDisplacedFiller,
+} from "../match.js";
 import { CONCEPT, STEP } from "../graph-search.js";
 import { unexplainedLabel } from "../rationale.js";
 import type { PipelineMechanism, Precomputed } from "../pipeline-mechanism.js";
@@ -349,7 +354,28 @@ export async function recallByResonance(
         (dominates(forest[0].breadth, 1) && forest[0].peak > Math.LN2))
     ) {
       const g = await project(ctx, forest[0].anchor, queryGist);
-      // The anchor cleared the consensus floor, but the floor prices the
+      // THE ANCHOR'S OCCUPANT IS NOT THE ASKER'S.  This tier grounds an anchor
+      // the climb elected, and voices that anchor's continuation.  When the
+      // query is the same structure with one position filled differently, the
+      // continuation speaks the ANCHOR's occupant of that position — a fluent,
+      // specific, wrong answer (see voicesDisplacedFiller).  Refusing here is
+      // silence, not a redirection: putting the asker's referent in its place
+      // is a stronger claim that needs the corpus's own carriage evidence, and
+      // that is `reference`'s job.
+      const anchorBytes = read(
+        ctx,
+        forest[0].anchor,
+        query.length * ctx.space.maxGroup + 1,
+      );
+      if (g !== null && voicesDisplacedFiller(ctx, query, anchorBytes, g)) {
+        ctx.trace?.step(
+          "displacedFiller",
+          [rItem(query, "query"), rNode(ctx, forest[0].anchor, "anchor")],
+          [rItem(g, "withheld")],
+          "refused — the anchor's continuation speaks the anchor's own " +
+            "occupant of a position this query fills differently",
+        );
+      } // The anchor cleared the consensus floor, but the floor prices the
       // ANCHOR's evidence, not the projection's: a junk attractor can clear
       // it and project a PIECE OF THE QUERY back at it (the observed
       // "buenos días in English" → "English" fragment).  A projection that
@@ -357,7 +383,7 @@ export async function recallByResonance(
       // — never an answer (the same principle as `restates` above, extended
       // to fragments).  Genuine anchor groundings — longer than the query,
       // or disjoint from it — pass untouched.
-      if (g && !(g.length < query.length && indexOf(query, g, 0) >= 0)) {
+      else if (g && !(g.length < query.length && indexOf(query, g, 0) >= 0)) {
         return ground(
           g,
           "scaffolding-dominated query — ground the consensus-climb anchor",

package/test/76-reference-binding.test.mjs

@@ -403,6 +403,78 @@ test("a referent may not be lifted out of the engine's OWN reply", async () => {
   await m.store.close();
 });
 
+// ── THE DISPLACED-FILLER GATE ──────────────────────────────────────────────
+//
+// Reference needs two instances before it will voice a binding: one instance
+// agrees with nothing, so nothing distinguishes "Run gcc hello.c" (a carriage
+// that follows the filename) from "hello.c needs -lm" (a fact about that one
+// file).  Abstaining there is right.
+//
+// What is NOT right is what used to happen next.  Recall's
+// scaffolding-dominated tier grounds the anchor the climb elected and voices
+// that anchor's continuation — which speaks the ANCHOR's occupant of the very
+// position the query fills differently.  Measured on a one-example corpus,
+// `How do I compile main.c?` answered `Run gcc hello.c`; on the trained
+// 15.7M-node store, `How do you say 'flurbish' in French?` answers "the way to
+// say hello is \"Bonjour\"".  Fluent, specific, and about the wrong thing.
+//
+// The gate REFUSES ONLY.  It never puts the asker's referent in the corpus's
+// place — that is a stronger claim needing the corpus's own carriage evidence,
+// which is what the licence above supplies.  Where the corpus cannot license a
+// binding, silence is the honest degradation.
+
+test("a projection may not voice the ANCHOR's occupant of a displaced slot", async () => {
+  const m = await mindWith([
+    ["How do I compile hello.c?", "Run gcc hello.c"],
+    ["What is the capital of France?", "Paris"],
+  ]);
+  const out = clean((await m.respond("How do I compile main.c?")).bytes).trim();
+  assert.ok(
+    !out.includes("hello"),
+    `voiced the corpus's filename for the asker's file: ${JSON.stringify(out)}`,
+  );
+  assert.equal(out, "", "the honest degradation here is silence");
+  await m.store.close();
+});
+
+test("the gate is not the restated-fragment guard", async () => {
+  // The refused projection is not a piece of the QUERY — it passes the
+  // restatement guard cleanly — it is a piece of the ANCHOR the query
+  // displaced.  Neither guard implies the other.
+  const m = await mindWith([
+    ["How do I compile hello.c?", "Run gcc hello.c"],
+    ["What is the capital of France?", "Paris"],
+  ]);
+  const q = "How do I compile main.c?";
+  assert.ok(
+    !q.includes("Run gcc hello.c"),
+    "fixture broken: the projection must NOT be a query fragment",
+  );
+  assert.equal(clean((await m.respond(q)).bytes).trim(), "");
+  await m.store.close();
+});
+
+test("a projection that carries the ASKER's own occupant is untouched", async () => {
+  // Voicing the anchor's filler AND the query's referent is what a licensed
+  // reference does; the gate must not touch it.
+  const m = await mindWith(CARRIED);
+  const r = await m.respond("How do I compile main.c?");
+  assert.equal(clean(r.bytes).trim(), "Run gcc main.c");
+  assert.equal(r.provenance, "reference");
+  await m.store.close();
+});
+
+test("a sub-quantum difference does not trip the gate", async () => {
+  // Below one river window byte overlap is chance, not a displaced slot.
+  const m = await mindWith([
+    ["What is the capital of France?", "Paris"],
+    ["Who wrote Hamlet?", "Shakespeare"],
+  ]);
+  const out = clean((await m.respond("What is the capital of Franc?")).bytes);
+  assert.match(out, /Paris/, `a near-identical query lost its answer: ${out}`);
+  await m.store.close();
+});
+
 // ── NO HIJACK ──────────────────────────────────────────────────────────────
 
 test("a trained query is answered by its own edge, not by reference", async () => {

package/dist/example/demo.d.ts

@@ -1 +0,0 @@
-export {};

package/dist/example/demo.js

@@ -1,39 +0,0 @@
-// demo.ts — one short session that drives the WHOLE pipeline from one memory.
-//
-// We give Sema a handful of plain notes, then ask things that no single note
-// answers. The headline query is the third one: from three worked examples Sema
-// learns the shape of "X was painted by Y", lifts the painter out of a sentence
-// it has NEVER seen, and then — in the same pass — reasons forward to a separate
-// fact about that painter. The reply contains no word from the question. That is
-// retrieval, generalization, and reasoning composing as a single act, with every
-// step traceable back to the notes behind it.
-import { Mind } from "../src/index.js";
-import { SQliteStore } from "../src/store-sqlite.js";
-async function main() {
-    const mind = new Mind({ store: new SQliteStore({ path: ":memory:" }) });
-    const ask = async (q) => (await mind.respondText(q)).trim();
-    // ── Jot down what we know. Each line is just (context → what follows). ──
-    await mind.ingest([
-        // One relation, shown three times — a pattern taught purely by example:
-        ["The Mona Lisa was painted by Leonardo da Vinci.", "Leonardo da Vinci"],
-        ["The Starry Night was painted by Vincent van Gogh.", "Vincent van Gogh"],
-        [
-            "The Night Watch was painted by Rembrandt van Rijn.",
-            "Rembrandt van Rijn",
-        ],
-        // One stray fact, keyed on a name none of the examples mention:
-        ["Pablo Picasso", "Pablo Picasso co-founded the Cubist movement"],
-    ]);
-    // 1) GENERALIZE — apply the learned pattern to an unseen sentence and read out
-    //    the painter. "Pablo Picasso" was never given as an answer; Sema locates it
-    //    by analogy to the three examples.
-    console.log(await ask("The Weeping Woman was painted by Pablo Picasso."));
-    // → "Pablo Picasso co-founded the Cubist movement"
-    //   …and, having found the painter, it KEEPS GOING: the name bridges into the
-    //   one fact it holds about him. The answer appears in no word of the question.
-    // 2) COMPUTE — exact arithmetic, grounded right where the notes go silent.
-    console.log(await ask("a museum charges 12*4 for a family ticket"));
-    // → "48"
-    await mind.store.close();
-}
-main();

package/dist/example/train_base.d.ts

@@ -1,93 +0,0 @@
-export interface Episode {
-    context: string;
-    continuation: string;
-}
-export type TrainingItem = string | Episode;
-/** Dedup + trim a concept's items: drop empty/degenerate pairs and exact
- *  repeats so a concept never deposits the same form twice. */
-export declare function refineItems(items: TrainingItem[]): TrainingItem[];
-/** One normalized SmolSent row. */
-export interface SmolSentRow {
-    src: string;
-    trg: string;
-    sl: string;
-    tl: string;
-}
-/** Normalize a raw datasets-server row into a SmolSentRow, or null when it lacks
- *  both sides or a side is implausibly large (a dump, not a sentence). */
-export declare function toSmolSentRow(row: unknown): SmolSentRow | null;
-/** Translate ONE SmolSent pair into SEMA facts: the two sentences are one
- *  meaning in two languages, so bind them BOTH ways. refineItems drops the
- *  degenerate case where src === trg. */
-export declare function smolSentRowToItems(row: SmolSentRow): TrainingItem[];
-/** One normalized Aya row. */
-export interface AyaRow {
-    inputs: string;
-    targets: string;
-    language: string;
-}
-/** Normalize a raw datasets-server row object into an AyaRow, or null when it
- *  lacks a usable prompt/answer or a field is implausibly large (a dump, not a
- *  cognitive example). Trims surrounding whitespace; keeps inner text verbatim
- *  (human prose, possibly multi-paragraph). */
-export declare function toAyaRow(row: unknown): AyaRow | null;
-/** Translate ONE Aya row into SEMA training items. A row is a single human
- *  (question → answer) exchange — exactly one FACT, the (inputs → targets) edge.
- *  No standalone-answer experience and no one-exchange "cumulative" walk: a lone
- *  Q→A is not multi-turn, and both would only replicate the same edge. */
-export declare function ayaRowToItems(row: AyaRow): TrainingItem[];
-/** A single oasst2 message node (the fields we use; the tree nests via replies). */
-interface OasstNode {
-    role?: string;
-    text?: string;
-    rank?: number | null;
-    deleted?: boolean;
-    replies?: OasstNode[];
-}
-/** One conversational turn extracted from a tree. */
-export interface OasstTurn {
-    role: string;
-    text: string;
-}
-/** Collapse a conversation tree to ONE linear path: at each node, descend into
- *  its best-ranked, non-deleted reply (rank 0 preferred; unranked sorts last).
- *  Returns the ordered turns (already strictly alternating in this corpus). */
-export declare function bestOasstPath(root: OasstNode): OasstTurn[];
-/** Translate ONE multi-turn oasst2 conversation into SEMA training items.
- *
- *  This is the ONE stage where cumulative continuous context is truly necessary:
- *  the data is a real multi-turn dialogue, and what must be learned is how each
- *  turn follows from the WHOLE conversation so far — not from the previous turn
- *  alone. The conversation is emitted ONLY as the accumulated walk; standalone
- *  turn experiences and local adjacent-pair facts are NOT emitted (they are
- *  subsumed by it and would merely replicate the content).
- *
- *  The walk is the pattern proven in test/13-conversation.test.mjs
- *  ("teachConversation"): each turn is the continuation of all prior turns,
- *  with BARE turn text — NO "User:/Assistant:" labels.  The SHAPE is identical
- *  (cumulative context → next turn); the join string is not, and does not need
- *  to be — that file joins with nothing and this corpus joins with "\n" (see
- *  `accumulate`).  Saying "byte-for-byte", as this comment used to, invites the
- *  reading that the two must agree on a separator.  They must not agree,
- *  because there is nothing to agree about: turn boundaries are offsets, and
- *  the join string is just corpus text. Roles already
- *  alternate by position in an oasst2 best-path (the root is a prompter), so a
- *  label adds nothing the position does not, while a clean continuation matches
- *  the test's recall (predictNext queries bare prior turns) and lets a turn share
- *  its gist with the same text elsewhere (e.g. an Aya question stored bare).
- *
- *  Returns [] for a conversation below the multi-turn threshold, so callers can
- *  simply skip empties. */
-export declare function oasstConversationToItems(turns: OasstTurn[]): TrainingItem[];
-/** One normalized General-Knowledge row. */
-export interface GenKnowRow {
-    question: string;
-    answer: string;
-}
-/** Normalize a raw datasets-server row into a GenKnowRow, or null when it lacks
- *  a usable question/answer or a side is implausibly large (corruption). */
-export declare function toGenKnowRow(row: unknown): GenKnowRow | null;
-/** Translate ONE General-Knowledge row into SEMA items: exactly one
- *  (question → answer) FACT. refineItems drops a degenerate question === answer. */
-export declare function genKnowRowToItems(row: GenKnowRow): TrainingItem[];
-export {};