@ai-matrx/content-ir 0.18.5 → 0.18.6

package/CHANGELOG.md

@@ -1,18 +1,72 @@
 # Changelog
 
-## 0.18.5
+## 0.18.6
 
 Automatic changed-only republish (docs/metadata drift since the last tag — see
-`git diff npm/content-ir-core/v0.18.4..npm/content-ir-core/v0.18.5 -- apps/shared/content-ir-core`).
+`git diff npm/content-ir-core/v0.18.5..npm/content-ir-core/v0.18.6 -- apps/shared/content-ir-core`).
 No source changes intended and no consumer action required.
 
+## 0.19.0
+
+**Carries: RC-B3 fix round (verify-RC-B3 § 0.18.3) and RC-B3r R3 — builder:
+the RC-B3 standard lane.** (aidream 1174edb640 — RC-B10, the print lane —
+shipped in 0.18.3, and 094594d3a1 — the escaped-bracket rule, "Unreleased"
+below — shipped in 0.18.4; both under generic "no source changes" notes.)
+
+**`/source` streaming: settled means settled** (F3/F4). Found with the
+verifier's generator (realistic model fragments, every prefix, with and
+without a final newline), now permanent in `__tests__/source-attack.test.ts`
+(`LLM-fragment documents…`; 12,000 documents in `SOURCE_ATTACK=long`) plus 27
+pinned reproducers. `tokenizeSource(prefix, { streaming: true })` now also
+holds back, as prose or as an open island:
+- lines a box-drawing tree still forming at the end may claim as its root
+  label (the renderer's walk-back), the open last line included;
+- a paragraph cut short by a block that may still dissolve (a `$$` pair
+  closing on the open line, an unbalanced JSON head, any opener still
+  arriving on the open line) — and a footnote before one;
+- everything inline on the open last line, and a block island ending there;
+- a nested-fence closer found only by the strict-CommonMark retry, a tilde
+  fence whose inner block may still grow past it, braces that balance on
+  the open last line;
+- a code span that a blank-so-far or `>`-only open line may still close
+  (`pendingCodeSpanStart`, `source/code-ranges.ts`).
+The two lone-HTML-tag holdbacks (M-N6/N7) collapsed into the open-tail block
+rule. Each holdback turns the attack suite red on its own when removed
+(planted mutations on a scratch copy). The H2 exception is unchanged.
+
+**`/source` one-shot: a mid-line reasoning mention that never closes is prose**
+(RC-B3r R3). `grep -c '<thinking>' log.txt > count.txt` no longer opens an
+`xml_region` to the end of the text (the renderer shows prose). While a stream
+is still arriving it stays an open region (its closer may yet come).
+
+**`isSingleDollarMath` refuses shell and template variable chains** (F5):
+`$USER$HOSTNAME`, `A=$A$B`, `df$col$sub`, `$name.$ext`, `PS1='$USER@$HOST'`,
+`${a}${b}` stay text; `$n$th`, `$x_i$s` stay math. Measured read-only with
+matrx-frontend `scripts/measure-single-dollar-rule.ts` over 85,325 assistant
+messages and 4,317 notes: 0 spans gained, 3 lost (2 shell-variable-chain, 1
+template-literal), each an intended false positive; notes unchanged. New
+export `singleDollarAfter(text, close)` — the `after` argument the rule now
+reads (the following name, `\n` marking the end of text). Shared vectors grew
+(`__tests__/fixtures/single-dollar-vectors.json`, including the trailing-space
+guard case `echo $HOME $PATH`).
+- Consumer action: callers of `isSingleDollarMath` pass
+  `singleDollarAfter(text, close)` instead of `text[close + 1]` (a one-character
+  `after` still works; it just cannot see a variable chain).
+
+## 0.18.5
+
+Automatic republish. Despite the generated note, this version DID carry source
+changes: the first half of the 0.19.0 work above (aidream edb3ec222d streaming
+holds, 3a1dae805c + 9a6c8db0b2 variable-chain guard, 60d3f61803
+`singleDollarAfter`). 0.19.0 supersedes it.
+
 ## 0.18.4
 
 Automatic changed-only republish (docs/metadata drift since the last tag — see
 `git diff npm/content-ir-core/v0.18.3..npm/content-ir-core/v0.18.4 -- apps/shared/content-ir-core`).
 No source changes intended and no consumer action required.
 
-## Unreleased
+## Shipped in 0.18.4 (listed here before the release)
 
 **`/source`: THE escaped-bracket rule** (verify-RC-B4 R3-2). `\[ … \]` is a
 display-math island only when its content is real TeX — a command, `^ { } =
@@ -897,3 +951,4 @@ unchanged. Two things are worth doing at your next working session:
 - Promoted the existing verified Content IR twin into an independently installable public package.
 - Added ESM and declaration builds, package linting, packed-artifact analysis, and an isolated consumer canary.
 - Preserved all parser, session, normalization, schema conversion, and structural-validation behavior with the existing shared fixtures and fingerprint vectors.
+

package/README.md

@@ -48,7 +48,9 @@ The package has no browser, React runtime, application store, database client, r
 `tokenizeSource(text)` reads stored text; `tokenizeSource(prefix, { streaming: true })`
 reads a message still arriving. In streaming mode anything the next bytes could
 still change is held back as plain prose (an open last line, an unclosed code
-span, a `$$` that can still pair, a footnote whose continuation is undecided, …),
+span, a `$$` that can still pair, a footnote whose continuation is undecided,
+lines a box-drawing tree still forming below may claim as its root label, a
+paragraph cut short by a block that may still dissolve, …),
 so **an island reported `complete` is the island the finished text has** — with
 ONE documented exception:
 

package/dist/index.cjs

@@ -6214,10 +6214,13 @@ var Tokenizer = class {
   /**
    * Streaming: an open block that may still turn back into prose — a `$$`
    * pair whose closer is on the open last line (prose after it makes it
-   * inline), or a JSON head whose braces have not balanced (it may never parse).
+   * inline), a JSON head whose braces have not balanced (it may never parse),
+   * or any opener that is itself still arriving on the open last line.
    */
   mayDissolve(block) {
-    return this.streaming && !block.complete && (block.type === "math_block" || block.type === "json");
+    if (!this.streaming || block.complete) return false;
+    if (block.type === "math_block" || block.type === "json") return true;
+    return this.isOpenTail(this.lineOf(block.start));
   }
   /** Streaming: paragraph ends where an undecided, dissolvable block cut them short. */
   undecidedBreaks = /* @__PURE__ */ new Set();
@@ -6238,6 +6241,12 @@ var Tokenizer = class {
     }
     return lo;
   }
+  /** How many whitespace characters open the text at `from` (up to `to`). */
+  leadingSpace(from, to) {
+    let k = from;
+    while (k < to && /\s/.test(this.text[k])) k++;
+    return k - from;
+  }
   isBlank(from, to) {
     return !NOT_WHITESPACE.test(this.text.slice(from, to));
   }
@@ -6262,6 +6271,10 @@ var Tokenizer = class {
         break;
       }
       const island = this.detectBlock(pos, li, true);
+      if (island && pos + this.leadingSpace(pos, this.contentEnds[li]) >= this.provisionalFrom) {
+        this.push({ kind: "prose", start: pos, end: this.n, island: null, inlines: [] });
+        break;
+      }
       if (island) {
         this.push({ kind: "island", start: island.start, end: island.end, island, inlines: [] });
         pos = island.end;
@@ -6322,36 +6335,55 @@ var Tokenizer = class {
    * region…) is never a tree line nor a root label, so it stops the walk.
    */
   treeRootHoldFrom() {
-    const claimable = (k2) => {
+    const claimFrom = (k2) => {
+      const lineStart = this.lineStarts[k2];
+      const lineEnd = this.contentEnds[k2];
+      let pos = lineStart;
       for (; ; ) {
-        const block = this.blockAt(this.lineStarts[k2]);
-        if (!block || block.kind !== "island" || block.island?.fromParagraph === true) return true;
-        const opened = this.lineOf(block.start);
-        if (this.lineStarts[opened] === block.start || opened === k2) return false;
-        k2 = opened;
+        const block = this.blockAt(pos);
+        if (!block || block.kind !== "island" || !block.island) return pos;
+        if (block.island.fromParagraph === true || this.mayDissolve(block.island)) return pos;
+        if (block.start < lineStart) {
+          const opened = this.lineOf(block.start);
+          if (this.lineStarts[opened] !== block.start && claimFrom(opened) !== -1) return pos;
+          if (block.end >= lineEnd) return -1;
+          pos = block.end;
+          continue;
+        }
+        if (block.end >= lineEnd) return -1;
+        pos = block.end;
       }
     };
     let k = this.lineCount - 1;
     let start = this.lineCount;
+    let startPos = this.n;
     if (k >= 0 && this.isOpenTail(k)) {
-      if (!claimable(k)) return this.n;
+      const at = claimFrom(k);
+      if (at === -1) return this.n;
       start = k;
+      startPos = at;
       k--;
     }
-    for (; k >= 0 && claimable(k); k--) {
+    for (; k >= 0; k--) {
       this.tick(1);
+      const at = claimFrom(k);
+      if (at === -1) break;
       if (this.lineIsBlank(k)) continue;
       if (!TREE_GLYPHS.test(this.lineText(k))) break;
       start = k;
+      startPos = at;
     }
-    let root = start;
-    for (let j = start - 1; j >= 0 && claimable(j); j--) {
+    let rootPos = startPos;
+    for (let j = start - 1; j >= 0; j--) {
+      const at = claimFrom(j);
+      if (at === -1) break;
       const trimmed = this.lineText(j).trim();
       this.tick(trimmed.length + 1);
       if (!trimmed || isSplitterTreeLine(trimmed) || /^#{1,6}\s/.test(trimmed)) break;
-      root = j;
+      rootPos = at;
+      if (at > this.lineStarts[j]) break;
     }
-    return root < this.lineCount ? this.lineStarts[root] : this.n;
+    return rootPos;
   }
   /** The block (so far) containing offset `pos`, or null. */
   blockAt(pos) {
@@ -7121,6 +7153,10 @@ var Tokenizer = class {
             const closer = `</${name}>`;
             const close = this.next(closer, tagEnd);
             const blockType = isAttr ? "xml_attr" : "xml_region";
+            if (close === -1 && isThinking && !this.streaming) {
+              i = tagEnd;
+              continue;
+            }
             if (close === -1) return stop(this.island(blockType, i, this.n, false, { tag: name }));
             const end = close + closer.length;
             if (end > pe) return stop(this.spill(blockType, i, end, pe, { tag: name }));