@ai-matrx/content-ir 0.16.2 → 0.17.0

package/CHANGELOG.md

@@ -1,11 +1,66 @@
 # Changelog
 
+## 0.17.0 — 2026-09-26
+
+**`/source` streaming: what "settled" promises, pinned — and the one
+documented exception** (verify-RC-B3 § Re-verification (0.16.0): G3, G6, H1,
+H2). The verifier's mutations M1–M13 each turn
+`__tests__/source-attack.test.ts` red ON ITS OWN — including the three
+holdbacks the 0.16.0 suite could not see (M9 unclosed backtick, M10 open `{`
+line, M12 pending `$$`), the `$$` open-tail rule (M11) and the held footnote
+(M13). Unmutated: green in CI and in 10 consecutive long runs under 24 CPU
+spinners.
+
+- **G3** — the suite's bounds are counts, never the clock, so no test may be
+  failed by the clock either: an explicit suite-wide timeout (15 min CI, 1 h
+  long). 0.16.0's long mode died on vitest's default 5 s timeout while every
+  work assertion held.
+- **G6** — each streaming holdback is pinned directly and by a flicker
+  message, and a new property checks, over random documents with and without
+  a final newline, that every island a stream shows as settled is the one-shot
+  island.
+- **H1** — a footnote whose continuation is undecided (it ends the text, or
+  the next line is still arriving — even one that currently looks like a
+  block) is held open (`complete: false`), never shown settled.
+- **H2 (chair ruling, 2026-09-26)** — a late orphan `</thinking>` /
+  `</think>` / `</reasoning>` closer is the ONE documented exception: it folds
+  the text before it into a thinking region retroactively. Consumers re-read
+  the finished text one-shot when a stream completes (README § streaming
+  mode); matrx-frontend's chat renderer does the same for its accumulator.
+- The property found and fixed further settle-then-change classes, each now
+  held back in streaming mode: a lone allowed HTML tag at any block start on
+  the open line; an undecided single-`$` span (everything from its opener);
+  a code span the final paragraph may still close (`pendingCodeSpanStart`,
+  counting a single trailing newline as still-open) — held only for the `$$`
+  readings it can change; `$$` pairs that a completing backtick / tilde run
+  would re-pair. The paragraph's end is always computed exactly as one-shot
+  computes it; only what is reported stops at the provisional point.
+
+**New (`/source` `findCodeRanges`, `CodeRange`): THE ONE code-range rule**
+(RC-B10 one-canonical). Fenced blocks and inline code spans by CommonMark —
+a fence opens only at line start (≤3 spaces) and closes by the one
+fence-nesting rule; a code span pairs equal backtick runs within one
+paragraph. Moved out of `math-pairs` (with `pendingCodeSpanStart`) so every
+package asks one function: `@ai-matrx/kit` delimiter-guard and
+`@ai-matrx/chat` speakable text now import it. Shared vectors:
+`__tests__/fixtures/code-range-vectors.json` (read by content-ir and kit).
+Guard: `__tests__/one-code-range-rule.test.ts` fails on any NEW fence/code-span
+detection in `apps/shared` (shrink-only census of the existing walkers).
+
+**Consumer action:** streaming consumers re-tokenize one-shot on completion
+(the H2 contract). Stored text is unaffected.
+
 ## 0.16.2
 
 - **Fixed (`/source` `pairDisplayMath`, `pendingDisplayMathOpener`):** a
   `$$...$$` / `$$…$$` placeholder in prose was paired as display math. New
   `isMathPlaceholder(inner)`; such a pair is consumed whole as literal text.
   Same rule as `@ai-matrx/kit` `delimiter-guard`.
+- **Fixed (`/source` `pairDisplayMath`): an inline triple backtick in prose
+  hid every later `$$` formula.** Code is now found the CommonMark way — a fence
+  opens only at line start (≤3 spaces), closes by the one fence-nesting rule
+  (`classifyInnerFenceLine`), and a code span pairs equal backtick runs inside
+  one paragraph. `tokenizeSource` math islands inherit it.
 - Consumer action: none — refresh to latest.
 
 ## 0.16.1

package/README.md

@@ -43,6 +43,30 @@ import {
 
 The package has no browser, React runtime, application store, database client, routing, or renderer dependency. `ajv` is its only runtime dependency. Host-specific registry fetching, component compilation, rendering, and error capture remain adapters around this kernel.
 
+## `/source` streaming mode: what "settled" promises
+
+`tokenizeSource(text)` reads stored text; `tokenizeSource(prefix, { streaming: true })`
+reads a message still arriving. In streaming mode anything the next bytes could
+still change is held back as plain prose (an open last line, an unclosed code
+span, a `$$` that can still pair, a footnote whose continuation is undecided, …),
+so **an island reported `complete` is the island the finished text has** — with
+ONE documented exception:
+
+- **A late orphan reasoning closer.** Models do emit closer-only reasoning (the
+  `<thinking>` opener lost upstream). When a `</thinking>` / `</think>` /
+  `</reasoning>` arrives with no opener, the one-shot reading folds everything
+  before it into a thinking region — retroactively. No streaming reader can
+  know that in advance short of holding ALL prose provisional until the stream
+  ends, which costs more than it saves (chair ruling, 2026-09-26; common-docs
+  `projects/rich-content-unification/REGISTER.md`, RC-B3 rulings).
+
+**Consumer contract:** when a stream completes, re-read the finished text in
+one-shot mode (`tokenizeSource(text)`, no `streaming`) and show THAT — the
+final screen always equals the one-shot result, which is also what a reload
+shows. Stored text never goes through streaming mode, so none of this can
+change a stored byte. (matrx-frontend's chat renderer does the same for its
+own live accumulator: `components/mardown-display/chat-markdown/settle-stream-blocks.ts`.)
+
 ## Release gate
 
 ```bash

package/dist/index.cjs

@@ -5288,32 +5288,8 @@ function singleDollarMathEnd(text, open, limit = text.length) {
   return isSingleDollarMath(content, text[close + 1]) ? close + 1 : -1;
 }
 
-// source/math-pairs.ts
-var MAX_MATH_SPAN = 600;
-var STRUCTURAL_MARKDOWN = /\]\(|https?:\/\/|\*\*|(?:^|\n)[ \t]{0,3}#{1,6}[ \t]|(?:^|\n)[ \t]*[-*+][ \t]+|(?:^|\n)[ \t]*\d+[.)][ \t]/;
-var LATEX_COMMAND = /\\[a-zA-Z]/;
-var PROSE_WORD = /[A-Za-z]{3,}/g;
-var PROSE_WORD_LIMIT = 6;
-function isMathPlaceholder(inner) {
-  return /^[\s.…·⋯_-]*$/.test(inner) && /[.…·⋯]/.test(inner);
-}
-function opensFlowBlock(text, open) {
-  for (let k = open - 1; k >= 0 && text[k] !== "\n"; k--) if (text[k] !== " " && text[k] !== "	") return false;
-  for (let k = open + 2; k < text.length && text[k] !== "\n"; k++) if (text[k] !== " " && text[k] !== "	" && text[k] !== "\r") return false;
-  return true;
-}
-function looksLikeDisplayMath(inner, flow = false) {
-  const s = inner.trim();
-  if (!s) return false;
-  if (STRUCTURAL_MARKDOWN.test(s)) return false;
-  if (s.length > MAX_MATH_SPAN) return false;
-  if (!flow && /\n[ \t]*\n/.test(inner)) return false;
-  if (LATEX_COMMAND.test(s)) {
-    return (s.match(PROSE_WORD) ?? []).length < PROSE_WORD_LIMIT * 3;
-  }
-  return (s.match(PROSE_WORD) ?? []).length < PROSE_WORD_LIMIT;
-}
-function codeRanges(text) {
+// source/code-ranges.ts
+function findCodeRanges(text) {
   const ranges = [];
   const n = text.length;
   let fence = null;
@@ -5345,7 +5321,7 @@ function codeRanges(text) {
       }
       const [s0] = runs[r];
       const [s1, l1] = runs[j];
-      ranges.push([s0, s1 + l1]);
+      ranges.push({ start: s0, end: s1 + l1, kind: "span" });
       r = j + 1;
     }
   };
@@ -5371,7 +5347,7 @@ function codeRanges(text) {
         closes = indent <= 3 && t >= fence.len && trimFenceLine(trimmed.slice(t)) === "";
       }
       if (closes) {
-        ranges.push([fence.start, lineEnd]);
+        ranges.push({ start: fence.start, end: lineEnd, kind: "fence" });
         fence = null;
       }
     } else {
@@ -5391,9 +5367,106 @@ function codeRanges(text) {
     if (nl === -1) break;
     pos = nl + 1;
   }
-  if (fence) ranges.push([fence.start, n]);
+  if (fence) ranges.push({ start: fence.start, end: n, kind: "fence" });
   else if (para !== -1) spans(para, n);
-  return ranges.sort((a, b) => a[0] - b[0]);
+  return ranges.sort((a, b) => a.start - b.start);
+}
+function pendingCodeSpanStart(text) {
+  const n = text.length;
+  if (n === 0 || /\n[ \t]*\n[ \t]*$/.test(text)) return -1;
+  let fence = null;
+  let para = -1;
+  let pos = 0;
+  while (pos <= n) {
+    const nl = text.indexOf("\n", pos);
+    const lineEnd = nl === -1 ? n : nl;
+    const line = text.slice(pos, lineEnd);
+    let indent = 0;
+    while (indent < line.length && line[indent] === " ") indent++;
+    const body = line.slice(indent);
+    if (fence) {
+      const trimmed = trimFenceLine(line);
+      let closes = false;
+      if (fence.char === "`") {
+        const kind = classifyInnerFenceLine(trimmed, fence.len, fence.nests, fence.depth);
+        if (kind === "open-nested") fence.depth++;
+        else if (kind === "close-nested") fence.depth--;
+        else if (kind === "close-outer") closes = indent <= 3;
+      } else {
+        let t = 0;
+        while (t < trimmed.length && trimmed[t] === "~") t++;
+        closes = indent <= 3 && t >= fence.len && trimFenceLine(trimmed.slice(t)) === "";
+      }
+      if (closes) fence = null;
+    } else {
+      const opener = indent <= 3 ? /^(`{3,}|~{3,})(.*)$/.exec(body) : null;
+      const info = opener ? opener[2] : "";
+      if (opener && !(opener[1][0] === "`" && info.includes("`"))) {
+        para = -1;
+        const char = opener[1][0];
+        const parsed = char === "`" ? parseFenceOpener(line) : null;
+        fence = { char, len: opener[1].length, nests: fenceNestsInnerFences(parsed?.lang), depth: 0 };
+      } else if (body.trim() === "") {
+        if (!(nl === -1 && line === "")) para = -1;
+      } else if (para === -1) para = pos;
+    }
+    if (nl === -1) break;
+    pos = nl + 1;
+  }
+  if (fence || para === -1) return -1;
+  const runs = [];
+  for (let k = para; k < n; ) {
+    if (text.charCodeAt(k) !== 96) {
+      k++;
+      continue;
+    }
+    let len = 1;
+    while (k + len < n && text.charCodeAt(k + len) === 96) len++;
+    runs.push([k, len]);
+    k += len;
+  }
+  const growing = (r) => {
+    const [start, len] = runs[r];
+    return start + len === n;
+  };
+  for (let r = 0; r < runs.length; ) {
+    if (growing(r)) return runs[r][0];
+    const len = runs[r][1];
+    let j = r + 1;
+    while (j < runs.length && (runs[j][1] !== len || growing(j))) j++;
+    if (j >= runs.length) return runs[r][0];
+    r = j + 1;
+  }
+  return -1;
+}
+
+// source/math-pairs.ts
+var MAX_MATH_SPAN = 600;
+var STRUCTURAL_MARKDOWN = /\]\(|https?:\/\/|\*\*|(?:^|\n)[ \t]{0,3}#{1,6}[ \t]|(?:^|\n)[ \t]*[-*+][ \t]+|(?:^|\n)[ \t]*\d+[.)][ \t]/;
+var LATEX_COMMAND = /\\[a-zA-Z]/;
+var PROSE_WORD = /[A-Za-z]{3,}/g;
+var PROSE_WORD_LIMIT = 6;
+function isMathPlaceholder(inner) {
+  return /^[\s.…·⋯_-]*$/.test(inner) && /[.…·⋯]/.test(inner);
+}
+function opensFlowBlock(text, open) {
+  for (let k = open - 1; k >= 0 && text[k] !== "\n"; k--) if (text[k] !== " " && text[k] !== "	") return false;
+  for (let k = open + 2; k < text.length && text[k] !== "\n"; k++) if (text[k] !== " " && text[k] !== "	" && text[k] !== "\r") return false;
+  return true;
+}
+function looksLikeDisplayMath(inner, flow = false) {
+  const s = inner.trim();
+  if (!s) return false;
+  if (STRUCTURAL_MARKDOWN.test(s)) return false;
+  if (s.length > MAX_MATH_SPAN) return false;
+  if (!flow && /\n[ \t]*\n/.test(inner)) return false;
+  if (LATEX_COMMAND.test(s)) {
+    return (s.match(PROSE_WORD) ?? []).length < PROSE_WORD_LIMIT * 3;
+  }
+  return (s.match(PROSE_WORD) ?? []).length < PROSE_WORD_LIMIT;
+}
+function codeRanges(text) {
+  return findCodeRanges(text).map((r) => [r.start, r.end]);
 }
 function pendingDisplayMathOpener(text) {
   if (!text.includes("$$")) return -1;
@@ -5805,6 +5878,14 @@ var Tokenizer = class {
     this.provisionalFrom = streaming ? pendingDisplayMathOpener(text) : -1;
     if (this.provisionalFrom === -1) this.provisionalFrom = Number.POSITIVE_INFINITY;
     if (streaming && text.includes("$$")) {
+      const span = pendingCodeSpanStart(text);
+      if (span !== -1) {
+        const firstToken = text.indexOf("$$", span);
+        if (firstToken !== -1) this.provisionalFrom = Math.min(this.provisionalFrom, firstToken);
+        for (const [open, close] of this.mathPairs) {
+          if (close >= span) this.provisionalFrom = Math.min(this.provisionalFrom, open);
+        }
+      }
       for (const pad of ["`", "``", "```", "~~~"]) {
         const alt = pairDisplayMath(text + pad);
         for (const [open, close] of this.mathPairs) {
@@ -6160,8 +6241,8 @@ var Tokenizer = class {
         last = k;
         continue;
       }
-      if (FOOTNOTE_DEF.test(line.trimStart()) || this.detectBlock(this.lineStarts[k], k, false)) break;
       if (this.isOpenTail(k)) return this.heldFootnote(p, open[1]);
+      if (FOOTNOTE_DEF.test(line.trimStart()) || this.detectBlock(this.lineStarts[k], k, false)) break;
       last = k;
     }
     const end = this.contentEnds[last];
@@ -7211,6 +7292,7 @@ exports.executesAtOutputRoot = executesAtOutputRoot;
 exports.fenceDiscriminator = fenceDiscriminator;
 exports.fenceNestsInnerFences = fenceNestsInnerFences;
 exports.fieldsToDbPayload = fieldsToDbPayload;
+exports.findCodeRanges = findCodeRanges;
 exports.fingerprintText = fingerprintText;
 exports.formatBlockLabel = formatBlockLabel;
 exports.getParseSession = getParseSession;