@ai-matrx/content-ir 0.16.2 → 0.18.0

package/CHANGELOG.md

@@ -1,11 +1,105 @@
 # Changelog
 
+## 0.18.0 — 2026-09-25
+
+**`/source` `isSingleDollarMath`: `$FC$` is math.** (RC-B9 verifier, RC-B3.)
+`where $FC$ is fixed costs.` drew literal dollars on screen and in every
+export, because the rule wanted an operator, a lone variable or a TeX
+command. It now follows Pandoc's `tex_math_dollars` delimiters (what
+remark-math approximates, and how ChatGPT and Claude render): the opener is
+followed by a non-space; the closer is preceded by a non-space and not
+followed by a digit. `$5 and $10`, `$5-$10`, `$5/$10` stay currency. On top,
+content that reads as something else is refused: an unescaped `%`; a closing
+bracket or `/` first, an opening bracket last; a straight quote at either
+end; `:` `;` `,` `/` last; a number glued to a following letter
+(`$10$N9qo…`); a placeholder (`$...$`); and prose (two plain words, a
+contraction, an abbreviation) unless a TeX command or operator is present.
+Measured before choosing (matrx-frontend
+`scripts/measure-single-dollar-rule.ts`, 85,102 assistant messages and 4,298
+notes, read-only): 128 spans become math, each one a formula on review
+(`$FC$`, `$VC$`, `$(0, 0)$`, `$2a$`, `$3x + 6 = 15$`, `$XY$`, `$p(doom)$`);
+22 spans the old rule drew as math stop (jQuery `$("#id")`, template literals
+`${id}:${…}`, a printf `%1$s`, broken TeX); 0 currency spans become math.
+Pure Pandoc would have added 36 more false positives (shell `$HOME/…:$PATH`,
+`$\ge$20% …**$`, `[$X] … [$Y]`, bcrypt costs). Shared vectors:
+`__tests__/fixtures/single-dollar-vectors.json` (red on 0.17.0). No other
+implementation of this rule exists (the Python server has none).
+
+**New: `looksLikeOpenInlineMath(partial)`** — whether an UNCLOSED `$…`
+already reads as a formula while a stream is still arriving. It keeps the
+old signal rule (a TeX command or operator, a lone variable, a single-letter
+shape; a leading digit needs a TeX command), because without a closer there
+are no delimiter facts yet: `$35 per stop` keeps showing mid-stream.
+
+Also carries the committed RC-B10 code-range work in `source/code-ranges.ts`
+and `source/tokenize.ts`. The print lane owns its entry.
+
+**Consumer action:** a streaming healer that asks `isSingleDollarMath` about
+an unclosed `$…` tail must ask `looksLikeOpenInlineMath` instead. With the
+wider rule, a price like `$35 per stop` would otherwise be held back until
+the line ends. matrx-frontend `stream-heal.ts` is the one known caller.
+
+## 0.17.0 — 2026-09-26
+
+**`/source` streaming: what "settled" promises, pinned — and the one
+documented exception** (verify-RC-B3 § Re-verification (0.16.0): G3, G6, H1,
+H2). The verifier's mutations M1–M13 each turn
+`__tests__/source-attack.test.ts` red ON ITS OWN — including the three
+holdbacks the 0.16.0 suite could not see (M9 unclosed backtick, M10 open `{`
+line, M12 pending `$$`), the `$$` open-tail rule (M11) and the held footnote
+(M13). Unmutated: green in CI and in 10 consecutive long runs under 24 CPU
+spinners.
+
+- **G3** — the suite's bounds are counts, never the clock, so no test may be
+  failed by the clock either: an explicit suite-wide timeout (15 min CI, 1 h
+  long). 0.16.0's long mode died on vitest's default 5 s timeout while every
+  work assertion held.
+- **G6** — each streaming holdback is pinned directly and by a flicker
+  message, and a new property checks, over random documents with and without
+  a final newline, that every island a stream shows as settled is the one-shot
+  island.
+- **H1** — a footnote whose continuation is undecided (it ends the text, or
+  the next line is still arriving — even one that currently looks like a
+  block) is held open (`complete: false`), never shown settled.
+- **H2 (chair ruling, 2026-09-26)** — a late orphan `</thinking>` /
+  `</think>` / `</reasoning>` closer is the ONE documented exception: it folds
+  the text before it into a thinking region retroactively. Consumers re-read
+  the finished text one-shot when a stream completes (README § streaming
+  mode); matrx-frontend's chat renderer does the same for its accumulator.
+- The property found and fixed further settle-then-change classes, each now
+  held back in streaming mode: a lone allowed HTML tag at any block start on
+  the open line; an undecided single-`$` span (everything from its opener);
+  a code span the final paragraph may still close (`pendingCodeSpanStart`,
+  counting a single trailing newline as still-open) — held only for the `$$`
+  readings it can change; `$$` pairs that a completing backtick / tilde run
+  would re-pair. The paragraph's end is always computed exactly as one-shot
+  computes it; only what is reported stops at the provisional point.
+
+**New (`/source` `findCodeRanges`, `CodeRange`): THE ONE code-range rule**
+(RC-B10 one-canonical). Fenced blocks and inline code spans by CommonMark —
+a fence opens only at line start (≤3 spaces) and closes by the one
+fence-nesting rule; a code span pairs equal backtick runs within one
+paragraph. Moved out of `math-pairs` (with `pendingCodeSpanStart`) so every
+package asks one function: `@ai-matrx/kit` delimiter-guard and
+`@ai-matrx/chat` speakable text now import it. Shared vectors:
+`__tests__/fixtures/code-range-vectors.json` (read by content-ir and kit).
+Guard: `__tests__/one-code-range-rule.test.ts` fails on any NEW fence/code-span
+detection in `apps/shared` (shrink-only census of the existing walkers).
+
+**Consumer action:** streaming consumers re-tokenize one-shot on completion
+(the H2 contract). Stored text is unaffected.
+
 ## 0.16.2
 
 - **Fixed (`/source` `pairDisplayMath`, `pendingDisplayMathOpener`):** a
   `$$...$$` / `$$…$$` placeholder in prose was paired as display math. New
   `isMathPlaceholder(inner)`; such a pair is consumed whole as literal text.
   Same rule as `@ai-matrx/kit` `delimiter-guard`.
+- **Fixed (`/source` `pairDisplayMath`): an inline triple backtick in prose
+  hid every later `$$` formula.** Code is now found the CommonMark way — a fence
+  opens only at line start (≤3 spaces), closes by the one fence-nesting rule
+  (`classifyInnerFenceLine`), and a code span pairs equal backtick runs inside
+  one paragraph. `tokenizeSource` math islands inherit it.
 - Consumer action: none — refresh to latest.
 
 ## 0.16.1
@@ -14,7 +108,7 @@ Automatic changed-only republish (docs/metadata drift since the last tag — see
 `git diff npm/content-ir-core/v0.16.0..npm/content-ir-core/v0.16.1 -- apps/shared/content-ir-core`).
 No source changes intended and no consumer action required.
 
-## Unreleased
+## 0.16.2 (also shipped)
 
 **`/source` — a callout or `:::` directive nested in a list item stays in the
 list's prose block** (verify-RC-B8 round 2, SAFETY). Guard:

package/README.md

@@ -43,6 +43,30 @@ import {
 
 The package has no browser, React runtime, application store, database client, routing, or renderer dependency. `ajv` is its only runtime dependency. Host-specific registry fetching, component compilation, rendering, and error capture remain adapters around this kernel.
 
+## `/source` streaming mode: what "settled" promises
+
+`tokenizeSource(text)` reads stored text; `tokenizeSource(prefix, { streaming: true })`
+reads a message still arriving. In streaming mode anything the next bytes could
+still change is held back as plain prose (an open last line, an unclosed code
+span, a `$$` that can still pair, a footnote whose continuation is undecided, …),
+so **an island reported `complete` is the island the finished text has** — with
+ONE documented exception:
+
+- **A late orphan reasoning closer.** Models do emit closer-only reasoning (the
+  `<thinking>` opener lost upstream). When a `</thinking>` / `</think>` /
+  `</reasoning>` arrives with no opener, the one-shot reading folds everything
+  before it into a thinking region — retroactively. No streaming reader can
+  know that in advance short of holding ALL prose provisional until the stream
+  ends, which costs more than it saves (chair ruling, 2026-09-26; common-docs
+  `projects/rich-content-unification/REGISTER.md`, RC-B3 rulings).
+
+**Consumer contract:** when a stream completes, re-read the finished text in
+one-shot mode (`tokenizeSource(text)`, no `streaming`) and show THAT — the
+final screen always equals the one-shot result, which is also what a reload
+shows. Stored text never goes through streaming mode, so none of this can
+change a stored byte. (matrx-frontend's chat renderer does the same for its
+own live accumulator: `components/mardown-display/chat-markdown/settle-stream-blocks.ts`.)
+
 ## Release gate
 
 ```bash