@ai-react-markdown/engine 2.4.2 → 2.4.5

package/README.md

@@ -1,5 +1,18 @@
 # @ai-react-markdown/engine
 
+[![npm version](https://img.shields.io/npm/v/@ai-react-markdown/engine?logo=npm&color=cb3837)](https://www.npmjs.com/package/@ai-react-markdown/engine)
+[![npm downloads](https://img.shields.io/npm/dm/@ai-react-markdown/engine?color=blue)](https://www.npmjs.com/package/@ai-react-markdown/engine)
+[![minzipped size](https://img.shields.io/bundlephobia/minzip/@ai-react-markdown/engine?label=minzip)](https://bundlephobia.com/package/@ai-react-markdown/engine)
+[![types](https://img.shields.io/npm/types/@ai-react-markdown/engine?logo=typescript&logoColor=white&color=3178c6)](https://www.typescriptlang.org/)
+
+[![Node ≥20](https://img.shields.io/badge/Node-%E2%89%A520-339933?logo=nodedotjs&logoColor=white)](https://nodejs.org/)
+[![ESM + CJS](https://img.shields.io/badge/module-ESM%20%2B%20CJS-f7df1e?logo=javascript&logoColor=black)](#installation)
+[![license](https://img.shields.io/npm/l/@ai-react-markdown/engine?color=green)](https://github.com/AIEPhoenix/ai-react-markdown/blob/main/LICENSE)
+
+[![CI](https://img.shields.io/github/actions/workflow/status/AIEPhoenix/ai-react-markdown/ci.yml?branch=main&label=CI&logo=githubactions&logoColor=white)](https://github.com/AIEPhoenix/ai-react-markdown/actions/workflows/ci.yml)
+[![Release](https://img.shields.io/github/actions/workflow/status/AIEPhoenix/ai-react-markdown/release.yml?label=release&logo=githubactions&logoColor=white)](https://github.com/AIEPhoenix/ai-react-markdown/actions/workflows/release.yml)
+[![part of ai-react-markdown](https://img.shields.io/badge/monorepo-ai--react--markdown-8a2be2?logo=github)](https://github.com/AIEPhoenix/ai-react-markdown)
+
 Framework-agnostic Markdown engine for [ai-react-markdown](https://github.com/AIEPhoenix/ai-react-markdown) — incremental parsing, LaTeX preprocessing, definition/footnote machinery, and the unified plugin pipeline. Takes Markdown text in, produces a [hast](https://github.com/syntax-tree/hast) tree plus incremental-parse state out; rendering that tree is the job of a framework adapter such as [`@ai-react-markdown/core`](https://www.npmjs.com/package/@ai-react-markdown/core) (React).
 
 > **Status: internal supplier.** This package exists to serve
@@ -10,12 +23,94 @@ Framework-agnostic Markdown engine for [ai-react-markdown](https://github.com/AI
 > instead; this package is interesting to you only if you are building a
 > framework adapter of your own.
 
+## What's inside
+
+Everything is exported from the package root (`import { … } from '@ai-react-markdown/engine'`); the barrel is grouped by layer:
+
+| Layer                    | Modules                                                                                                                                        | Highlights                                                                                                                                                                                                                                                                                     |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Preprocessors            | `preprocessors/latex`, `preprocessors/remend`, `preprocessAIMDContent`                                                                         | `preprocessLaTeX(text)` (currency `$`, `\[…\]` / `\(…\)` normalization, code-fence and inline-code protection), `createIncrementalLatexPreprocessor()` for append-only streams, `remend` for unterminated-markup mending                                                                       |
+| Incremental parsing      | `incrementalParse/*`                                                                                                                           | `advanceIncrementalParse(state, content, options)` — the prefix-freeze engine: a line scanner decides a verified-safe freeze boundary, only the tail re-parses, and the two trees are spliced; every frame is deep-equal to a full parse (enforced by the arbiter suites) or falls back to one |
+| Pipeline assembly        | `markdown/*`, `pluginChain`, `plugins/catalog`, `customMdastHandlers`, `remarkInjectPhantomDefs`, `rehypeRebaseHashLinks`, `rehypeFooterAdorn` | `buildCoreRemarkPlugins` / `buildCoreRehypePlugins` / `buildCoreRemarkRehypeOptions` — the exact chains the React renderer uses; the sealed engine-plugin catalog (`highlight`, `definitionList`, `removeComments`, `smartypants`, `pangu`, `defaultEnginePlugins`)                            |
+| Cross-chunk coordination | `documentRegistry`, `collectDefLabels`, `extractContributions`, `extractDefBodiesFromHast`, `crossChunkUrlSanitize`                            | `createRegistry()` — the per-document store that numbers footnotes and resolves link definitions across chunks; `sanitizeCrossChunkUrl()` mirrors the standalone two-gate URL policy                                                                                                           |
+| Sanitization             | `sanitizeSchema`, `extendSanitizeSchema`, `markdown/urlTransform`                                                                              | The library default `rehype-sanitize` schema (read-only singleton — clone with `extendSanitizeSchema`), `defaultUrlTransform`                                                                                                                                                                  |
+| Streaming                | `smoothStream/controller`                                                                                                                      | `createSmoothStreamController()` — the framework-agnostic typewriter pacing state machine behind `<AIMarkdownSmoothStream>`, with `SMOOTH_STREAM_PACING_PRESETS`                                                                                                                               |
+| Leaves                   | `hastPredicates`, `normalizeId`, `shortenDocumentId`, `devStageTimings`, `fixtures/scenarios`                                                  | Small pure helpers and the shared test corpus                                                                                                                                                                                                                                                  |
+
+## Install
+
+```bash
+npm install @ai-react-markdown/engine
+```
+
+Dual ESM/CJS build with types for both. No React dependency. The only peer is `katex` (`^0.16 || ^0.17`, **optional** — needed only if you render math). It ships transitively via `rehype-katex`, so hoisted installers resolve it automatically; strict-isolation installers (yarn PnP, `pnpm --node-linker=isolated`) must install it explicitly in your app.
+
+## Example: the LaTeX preprocessor on its own
+
+```ts
+import { preprocessLaTeX } from '@ai-react-markdown/engine';
+
+preprocessLaTeX('Price is $100, and \\(x^2\\) is inline math.');
+// → 'Price is \\$100, and $$x^2$$ is inline math.'
+// (currency `$` escaped; `\\(…\\)` normalized to the `$$…$$` form remark-math's inline rule accepts)
+```
+
+The same function runs inside `@ai-react-markdown/core` before every parse; the incremental variant (`createIncrementalLatexPreprocessor`) reuses work across append-only frames.
+
+## Example: driving the incremental parser
+
+```ts
+import {
+  advanceIncrementalParse,
+  buildCoreRemarkPlugins,
+  buildCoreRehypePlugins,
+  buildCoreRemarkRehypeOptions,
+  defaultEnginePlugins,
+  sanitizeSchema,
+} from '@ai-react-markdown/engine';
+
+const options = {
+  remarkPlugins: buildCoreRemarkPlugins(defaultEnginePlugins),
+  rehypePlugins: buildCoreRehypePlugins(sanitizeSchema, ''),
+  remarkRehypeOptions: buildCoreRemarkRehypeOptions(false),
+  depsKey: [],
+  defListEnabled: false,
+};
+
+let state = null;
+for (const frame of ['# Hello', '# Hello\n\nworld', '# Hello\n\nworld and more']) {
+  const result = advanceIncrementalParse(state, frame, options);
+  state = result.nextState;
+  // result.hast — the full-document hast for this frame
+  // result.usedIncremental / result.boundary — whether the frame spliced, and where
+}
+```
+
+`AdvanceOptions` is documented in `incrementalParse/advanceIncrementalParse.ts`; the React renderer's `MarkdownContent` is the reference consumer.
+
+## Verification
+
+The incremental engine ships with a five-layer equivalence stack (fixture pins, fuzz arbiter, direction battery, exhaustive census, arbiter-sensitivity meta-suite) plus a release-gate soak (`scripts/run-soak.sh`); the full record lives in `src/experiments/prefixFreeze/README.md`. Every reachable divergence found so far is pinned as a deterministic test.
+
 ## Runtime support
 
 Pure computation over strings and syntax trees: no DOM access, no
 Node-only APIs, and no unguarded environment reads. Runs in browsers,
 Node, workers, and embedded JS runtimes (e.g. Hermes/JavaScriptCore).
 
+## Versioning
+
+Lockstep with `@ai-react-markdown/core`, which pins this package **exactly** — the export surface follows what core consumes and may change in any release before 3.0.0 (see the status note above). Release notes: [release highlights](https://github.com/AIEPhoenix/ai-react-markdown/blob/main/docs/release-highlights.md).
+
+## Package family
+
+| Package                                                                                                              | Role                                                                                                        | Version policy                                                      |
+| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
+| [`@ai-react-markdown/core`](https://www.npmjs.com/package/@ai-react-markdown/core)                                   | The React renderer — `<AIMarkdown>`, `<AIMarkdownSmoothStream>`, `<AIMarkdownDocuments>`, hooks, providers  | Release train                                                       |
+| [`@ai-react-markdown/mantine`](https://www.npmjs.com/package/@ai-react-markdown/mantine)                             | Mantine UI bindings — themed typography, code-highlight tabs, Mermaid, color-scheme wiring                  | Release train (lockstep with core)                                  |
+| [`@ai-react-markdown/engine`](https://www.npmjs.com/package/@ai-react-markdown/engine)                               | Framework-agnostic engine — incremental parsing, LaTeX preprocessing, plugin pipeline, cross-chunk registry | Release train (lockstep, pinned exactly by core; internal supplier) |
+| [`@ai-react-markdown/remark-mark-highlight`](https://www.npmjs.com/package/@ai-react-markdown/remark-mark-highlight) | remark plugin for `==mark==` highlight syntax                                                               | Independent semver                                                  |
+
 ## License
 
 MIT

package/dist/index.cjs

@@ -269,6 +269,11 @@ var defaultUrlTransform = (value) => {
 var import_micromark_util_html_tag_name = require("micromark-util-html-tag-name");
 var import_micromark_util_normalize_identifier = require("micromark-util-normalize-identifier");
 var TYPE6_NAMES = new Set(import_micromark_util_html_tag_name.htmlBlockNames);
+var TABLE_PART_NAMES = /* @__PURE__ */ new Set(["td", "th", "tr", "tbody", "thead", "tfoot", "caption", "col", "colgroup"]);
+var TYPE1_NAMES = /* @__PURE__ */ new Set(["script", "pre", "style", "textarea"]);
+var TYPE6_START_RE = /^<\/?([A-Za-z][A-Za-z0-9-]*)(?:[ \t\r]|\/?>|$)/;
+var TYPE1_START_RE = /^<(script|pre|style|textarea)(?:[ \t\r]|>|$)/i;
+var TYPE7_LINE_RE = /^<(\/?)([A-Za-z][A-Za-z0-9-]*)(?:[ \t\r][^>]*|\/)?>[ \t\r]*$/;
 var VOID_TAGS = /* @__PURE__ */ new Set([
   "area",
   "base",
@@ -407,7 +412,11 @@ function freshCheckpoint(defListEnabled, mathFlow, referenceTaint) {
     htmlFlowSinceBlank: false,
     htmlSeamPending: false,
     phasePoisonedAt: Infinity,
-    pendingTruncatedTags: []
+    pendingTruncatedTags: [],
+    pendingTruncatedCloses: [],
+    tagAcrossLines: false,
+    tagAcrossLinesIndent: 0,
+    htmlFlowReal: false
   };
 }
 function isPlausibleLinkDefRest(rest) {
@@ -704,6 +713,8 @@ function processConfirmedLine(cp, ln, text) {
       for (const tag of cp.pendingTruncatedTags) applyTag(tag, true);
       cp.pendingTruncatedTags = [];
     }
+    cp.pendingTruncatedCloses = [];
+    cp.tagAcrossLines = false;
     cp.blankRun += 1;
     cp.lastBlankStart = ln.start;
     cp.candidates.push({
@@ -717,6 +728,7 @@ function processConfirmedLine(cp, ln, text) {
     cp.paragraphHasUnpairedRun = false;
     cp.openBracket = null;
     cp.htmlFlowSinceBlank = false;
+    cp.htmlFlowReal = false;
     cp.prevLineBlank = true;
     cp.prevLineWasText = false;
     cp.prevLineWasValidDef = false;
@@ -732,6 +744,16 @@ function processConfirmedLine(cp, ln, text) {
   if (tagStart) {
     cp.htmlFlowSinceBlank = true;
     if (!TYPE6_NAMES.has(tagStart[1].toLowerCase())) cp.hazardVerdict = true;
+    if (!cp.htmlFlowReal) {
+      const t = mdTrimStart(ln.text);
+      const t6 = TYPE6_START_RE.exec(t);
+      const t7 = TYPE7_LINE_RE.exec(t);
+      if (t6 !== null && TYPE6_NAMES.has(t6[1].toLowerCase()) || TYPE1_START_RE.test(t) || // Type 7 cannot interrupt a paragraph, and excludes the raw-text
+      // names (those are type 1 as start tags, paragraph as end tags).
+      t7 !== null && !cp.prevLineWasText && !TYPE1_NAMES.has(t7[2].toLowerCase())) {
+        cp.htmlFlowReal = true;
+      }
+    }
   }
   const inRawText = cp.htmlFlowSinceBlank || rawOpenAtLineStart;
   const rawFlowStart = ln.indent <= 3 && /^<(?:!--|\?|![A-Za-z]|!\[CDATA\[)/.test(mdTrimStart(ln.text));
@@ -872,7 +894,21 @@ ${cont(scanText)}` };
   for (const [from, to] of rawSpans) {
     tagText = tagText.slice(0, from) + " ".repeat(to - from) + tagText.slice(to);
   }
-  {
+  let skipTagScan = false;
+  if (cp.tagAcrossLines) {
+    if (ln.indent < cp.tagAcrossLinesIndent) poisonRawDivergence();
+    const gt = ln.text.indexOf(">");
+    if (gt === -1) {
+      skipTagScan = true;
+    } else {
+      if (/["']/.test(ln.text.slice(0, gt))) poisonRawDivergence();
+      for (const tag of cp.pendingTruncatedCloses) applyTag(tag, true);
+      cp.pendingTruncatedCloses = [];
+      cp.tagAcrossLines = false;
+      tagText = " ".repeat(gt + 1) + tagText.slice(gt + 1);
+    }
+  }
+  if (!skipTagScan) {
     TAG_OR_COMMENT_RE.lastIndex = 0;
     let m;
     let lastCommentOpenerIdx = -1;
@@ -902,6 +938,7 @@ ${cont(scanText)}` };
       if (cp.commentOpen) continue;
       const closing = m[1] === "/";
       const tag = m[2].toLowerCase();
+      if (TABLE_PART_NAMES.has(tag)) cp.phasePoisonedAt = Math.min(cp.phasePoisonedAt, ln.start + m.index);
       const selfClosing = m[3] !== void 0 && /\/\s*$/.test(m[3]);
       if (VOID_TAGS.has(tag) || selfClosing) continue;
       applyTag(tag, closing);
@@ -922,6 +959,7 @@ ${cont(scanText)}` };
         if (startMasked || wholeVisible || inRaw(mr.index) || cp.commentOpen) continue;
         const closing = mr[1] === "/";
         const tag = mr[2].toLowerCase();
+        if (TABLE_PART_NAMES.has(tag)) cp.phasePoisonedAt = Math.min(cp.phasePoisonedAt, ln.start + mr.index);
         const selfClosing = mr[3] !== void 0 && /\/\s*$/.test(mr[3]);
         if (VOID_TAGS.has(tag) || selfClosing) continue;
         applyTag(tag, closing);
@@ -937,7 +975,14 @@ ${cont(scanText)}` };
         if (m2) {
           const closing = m2[1] === "/";
           const tag = m2[2].toLowerCase();
-          if (!VOID_TAGS.has(tag)) {
+          if (TABLE_PART_NAMES.has(tag)) cp.phasePoisonedAt = Math.min(cp.phasePoisonedAt, ln.start + lastLt);
+          if (cp.htmlFlowReal) {
+            cp.tagAcrossLines = true;
+            cp.tagAcrossLinesIndent = ln.indent;
+          }
+          if (closing) {
+            if (!VOID_TAGS.has(tag) && cp.htmlFlowReal) cp.pendingTruncatedCloses.push(tag);
+          } else if (!VOID_TAGS.has(tag)) {
             applyTag(tag, closing);
             const rawLastLt = ln.text.lastIndexOf("<");
             const rawTruncated = rawLastLt !== -1 && !ln.text.includes(">", rawLastLt);
@@ -1149,6 +1194,8 @@ function rebaseDualWalk(node, segments, maxEnd, offsetDelta, lineDelta) {
     for (const child of children) rebaseDualWalk(child, segments, maxEnd, offsetDelta, lineDelta);
   }
 }
+var TABLE_PART_TAG_RE = /<(?:td|th|tr|tbody|thead|tfoot|caption|col|colgroup)\b/i;
+var STRAY_SYNTHESIZED_END_TAG_RE = /<\/(?:br|p)\b/i;
 function spliceTrees(input) {
   const { prevMdast, prevHast, tailMdast, tailHast, content, boundary, injectionPrefix, injectedSegments } = input;
   const injectedLen = injectionPrefix.length;
@@ -1205,6 +1252,12 @@ function spliceTrees(input) {
     return !(start !== void 0 && start < injectedLen);
   });
   const tailWrapVisible = tailMdastChildren.some((child) => !isWrapInvisible(child));
+  if (prefixMdast.some((c) => c.type === "html" && TABLE_PART_TAG_RE.test(c.value))) return null;
+  for (const child of tailMdastChildren) {
+    if (isWrapInvisible(child)) continue;
+    if (child.type !== "html") break;
+    if (STRAY_SYNTHESIZED_END_TAG_RE.test(child.value) || TABLE_PART_TAG_RE.test(child.value)) return null;
+  }
   const aligned = alignPrefixCut(prefixMdast, cutRegion, tailWrapVisible);
   if (aligned === null) return null;
   const hastChildren = aligned.children;
@@ -1330,6 +1383,9 @@ function alignPrefixCut(prefixMdast, cutRegion, tailWrapVisible) {
     return null;
   }
   const lastIsLiteral = last !== void 0 && last.type === "text" && last.value.trim() !== "";
+  if (lastIsLiteral && pairIdx >= 0 && visibles[pairIdx].type !== "html") {
+    return null;
+  }
   const litOwnerEnd = pairIdx >= 0 ? visibles[pairIdx].position?.end?.offset : void 0;
   const litEnd = lastIsLiteral ? last.position?.end?.offset : void 0;
   if (lastIsLiteral && last.position !== void 0 && (litEnd === void 0 || litOwnerEnd === void 0)) {
@@ -1418,8 +1474,9 @@ function stripInjectedHast(tailMdast, tailHast, injectedLen, tailWrapVisible) {
 }
 function tailLeadingTextIsHoist(tailMdastChildren, tailHastChildren) {
   const firstText = tailHastChildren[0];
-  if (!firstText) return false;
   const firstVisible = tailMdastChildren.find((c) => !isWrapInvisible(c));
+  if (firstVisible?.type === "html" && STRAY_SYNTHESIZED_END_TAG_RE.test(firstVisible.value)) return null;
+  if (!firstText) return false;
   if (!isSeparatorText(firstText)) {
     if (firstText.type === "text" && firstText.position === void 0 && firstVisible?.type === "html") {
       if (/^\s*<\/[A-Za-z][A-Za-z0-9-]*\s*>/.test(firstVisible.value)) return true;
@@ -1468,6 +1525,9 @@ function countTrailingNewlines(value) {
 function countNewlines(text, end = text.length) {
   let count = 0;
   for (let i = text.indexOf("\n"); i !== -1 && i < end; i = text.indexOf("\n", i + 1)) count += 1;
+  for (let i = text.indexOf("\r"); i !== -1 && i < end; i = text.indexOf("\r", i + 1)) {
+    if (text.charCodeAt(i + 1) !== 10) count += 1;
+  }
   return count;
 }
 
@@ -1587,7 +1647,7 @@ function normalizeId(s) {
   return (0, import_micromark_util_normalize_identifier3.normalizeIdentifier)(s);
 }
 function normalizeForMatch(s) {
-  return (0, import_micromark_util_normalize_identifier3.normalizeIdentifier)(s.replace(/\\(.)/g, "$1"));
+  return (0, import_micromark_util_normalize_identifier3.normalizeIdentifier)(s.replace(/\\([!-/:-@[-`{-~])/g, "$1"));
 }
 
 // src/components/collectDefLabels.ts
@@ -1860,17 +1920,8 @@ function phantomSuffixCloser(content) {
 }
 
 // src/components/extractContributions.ts
-function fakeAnchorElement(url) {
-  return { type: "element", tagName: "a", properties: { href: url }, children: [] };
-}
-function sanitizeDefUrl(url, urlTransform) {
-  if (!urlTransform) return url;
-  const result = urlTransform(url, "href", fakeAnchorElement(url));
-  return result == null ? "" : String(result);
-}
 function* extractContributions(mdast, options = {}) {
   const phantomFn = options.phantomFootnoteLabels;
-  const urlTransform = options.urlTransform;
   const out = [];
   (0, import_unist_util_visit3.visit)(mdast, (n) => {
     if (n.type === "footnoteReference") {
@@ -1898,7 +1949,7 @@ function* extractContributions(mdast, options = {}) {
       out.push({
         kind: "linkDef",
         label: normalizeId(d.identifier),
-        url: sanitizeDefUrl(d.url, urlTransform),
+        url: d.url,
         title: d.title
       });
     }
@@ -3612,8 +3663,7 @@ function isProtocolAllowed(url, allowed) {
 }
 function sanitizeCrossChunkUrl(rawUrl, key, tagName, urlTransform, schema) {
   rawUrl = (0, import_micromark_util_sanitize_uri2.normalizeUri)(rawUrl);
-  const callerProtocols = schema.protocols;
-  const allowed = callerProtocols === void 0 || callerProtocols === null ? sanitizeSchema.protocols?.[key] : callerProtocols[key];
+  const allowed = Object.hasOwn(schema, "protocols") ? schema.protocols?.[key] : sanitizeSchema.protocols?.[key];
   if (allowed && allowed.length > 0 && !isProtocolAllowed(rawUrl, allowed)) return null;
   const transformed = urlTransform(rawUrl, key, fakeElement(tagName, key, rawUrl));
   if (transformed == null) return null;
@@ -4011,23 +4061,28 @@ function lineHasBacktick(content, pos) {
   }
   return false;
 }
-function isAtLineStart(content, pos) {
+function lineIndentBefore(content, pos) {
   let i = pos - 1;
-  let spaces = 0;
-  while (i >= 0 && content[i] === " ") {
-    spaces++;
-    if (spaces > 3) return false;
+  let indent = 0;
+  while (i >= 0 && (content[i] === " " || content[i] === "	")) {
+    indent += content[i] === "	" ? 4 : 1;
     i--;
   }
-  return i < 0 || content[i] === "\n" || content[i] === "\r";
+  return i < 0 || content[i] === "\n" || content[i] === "\r" ? indent : -1;
 }
 function findClosingBacktickRun(content, start, n) {
   let i = start;
   while (i < content.length) {
-    if (content[i] === "`") {
+    const ch = content[i];
+    if (ch === "`") {
       const runLen = getRepeatedMarkerLength(content, i, "`");
       if (runLen === n) return i;
       i += runLen;
+    } else if (ch === "\n") {
+      let j = i + 1;
+      while (j < content.length && (content[j] === " " || content[j] === "	" || content[j] === "\r")) j += 1;
+      if (j >= content.length || content[j] === "\n") return -1;
+      i += 1;
     } else {
       i += 1;
     }
@@ -4040,6 +4095,7 @@ function splitByProtectedRegions(content) {
   let multilineStart = -1;
   let multilineFenceMarker = null;
   let multilineFenceLength = 0;
+  let multilineFenceIndent = 0;
   function pushProtected(start, end) {
     if (start > lastIndex) {
       segments.push({ text: content.substring(lastIndex, start), isCode: false });
@@ -4053,7 +4109,8 @@ function splitByProtectedRegions(content) {
     if (multilineStart !== -1) {
       if (char === multilineFenceMarker) {
         const runLen = getRepeatedMarkerLength(content, i, multilineFenceMarker);
-        if (runLen >= multilineFenceLength && isAtLineStart(content, i) && restOfLineIsBlank(content, i + runLen)) {
+        const closerIndent = lineIndentBefore(content, i);
+        if (runLen >= multilineFenceLength && closerIndent !== -1 && closerIndent <= multilineFenceIndent + 3 && restOfLineIsBlank(content, i + runLen)) {
           pushProtected(multilineStart, i + runLen);
           multilineStart = -1;
           multilineFenceMarker = null;
@@ -4069,10 +4126,12 @@ function splitByProtectedRegions(content) {
     }
     if (char === "`" || char === "~") {
       const runLen = getRepeatedMarkerLength(content, i, char);
-      if (runLen >= 3 && isAtLineStart(content, i) && !(char === "`" && lineHasBacktick(content, i + runLen))) {
+      const openerIndent = lineIndentBefore(content, i);
+      if (runLen >= 3 && openerIndent !== -1 && !(char === "`" && lineHasBacktick(content, i + runLen))) {
         multilineStart = i;
         multilineFenceMarker = char;
         multilineFenceLength = runLen;
+        multilineFenceIndent = openerIndent;
         i += runLen;
         continue;
       }
@@ -4145,16 +4204,20 @@ function escapeCurrencyDollarSigns(text) {
       currentLineProcessed += segment;
     }
     let needEscape = true;
-    let restBeforeNextMatchOrEnd = "";
-    if (i < currencyMatches.length - 1) {
-      const nextMatch = currencyMatches[i + 1];
-      if (nextMatch.index - match.index > 1) {
-        restBeforeNextMatchOrEnd = text.substring(match.index + 1, nextMatch.index);
+    const restStart = match.index + 1;
+    const restEnd = i < currencyMatches.length - 1 ? currencyMatches[i + 1].index : text.length;
+    let firstLineBeforeNextMatch = "";
+    if (restEnd - restStart > 0) {
+      let eol = restEnd;
+      for (let k = restStart; k < restEnd; k++) {
+        const c = text.charCodeAt(k);
+        if (c === 10 || c === 13) {
+          eol = k;
+          break;
+        }
       }
-    } else {
-      restBeforeNextMatchOrEnd = text.substring(match.index + 1);
+      firstLineBeforeNextMatch = text.substring(restStart, eol);
     }
-    const firstLineBeforeNextMatch = restBeforeNextMatchOrEnd.split(/\r\n|\r|\n/g)[0];
     if (Array.from(firstLineBeforeNextMatch.matchAll(NO_ESCAPED_DOLLAR_REGEX)).length % 2 !== 0) {
       const wholeLineBeforeNextMatchWithoutCurrentDollar = currentLineProcessed + firstLineBeforeNextMatch;
       if (Array.from(wholeLineBeforeNextMatchWithoutCurrentDollar.matchAll(NO_ESCAPED_DOLLAR_REGEX)).length % 2 !== 0) {
@@ -4320,39 +4383,49 @@ function hasUnclosedTextCommand(text) {
   return false;
 }
 var RESIDUAL_OPEN_BRACKET_RE = /(?<!!)\\\[/;
-function processSliceInstrumented(slice) {
+var LEADING_DOUBLE_DOLLAR_RE = /^\s*\$\$/;
+function processSliceInstrumented(slice, probe = true) {
   const segments = splitByProtectedRegions(slice);
-  let out = "";
+  const parts = [];
   let quiescent = true;
   let truncatedAtSeamStart = false;
   for (let index = 0; index < segments.length; index++) {
     const segment = segments[index];
     if (segment.isCode) {
-      out += segment.text;
+      parts.push(segment.text);
       continue;
     }
     let text = segment.text;
     text = escapeMhchemCommands(text);
     text = escapeCurrencyDollarSigns(text);
     text = convertLatexDelimiters(text);
-    if (RESIDUAL_OPEN_BRACKET_RE.test(text)) quiescent = false;
+    if (probe && RESIDUAL_OPEN_BRACKET_RE.test(text)) quiescent = false;
     text = escapeLatexPipes(text);
-    if (findUnclosedDelimiterStart(text, "both") !== -1) quiescent = false;
+    if (probe && findUnclosedDelimiterStart(text, "both") !== -1) quiescent = false;
     text = escapeLatexPipesInUnclosed(text);
-    if (hasUnclosedTextCommand(text)) quiescent = false;
+    if (probe && hasUnclosedTextCommand(text)) quiescent = false;
     text = escapeTextUnderscores(text);
     text = convertSingleToDoubleDollar(text);
-    const unclosedDouble = findUnclosedDelimiterStart(text, "double-only");
-    if (unclosedDouble !== -1) {
-      quiescent = false;
-      if (index === 0 && text.slice(0, unclosedDouble).trim() === "") {
-        truncatedAtSeamStart = true;
+    if (probe || index === 0 && LEADING_DOUBLE_DOLLAR_RE.test(text)) {
+      const unclosedDouble = findUnclosedDelimiterStart(text, "double-only");
+      if (unclosedDouble !== -1) {
+        quiescent = false;
+        if (index === 0 && text.slice(0, unclosedDouble).trim() === "") {
+          truncatedAtSeamStart = true;
+        }
       }
     }
     text = truncateUnclosedLatexBlock(text);
-    out += text;
+    parts.push(text);
   }
-  return { out, quiescent, truncatedAtSeamStart };
+  return { out: parts.join(""), quiescent, truncatedAtSeamStart };
+}
+function isBlankRawLine(text, from, to) {
+  for (let i = from; i < to; i++) {
+    const c = text.charCodeAt(i);
+    if (c !== 32 && c !== 9 && c !== 13) return false;
+  }
+  return true;
 }
 function findRawSafeCut(active) {
   const segments = splitByProtectedRegions(active);
@@ -4367,10 +4440,12 @@ function findRawSafeCut(active) {
       continue;
     }
     const text = segment.text;
+    let atLineStart = offset === 0 || active.charCodeAt(offset - 1) === 10;
     let lineStart = 0;
     while (lineStart <= text.length) {
       const nl = text.indexOf("\n", lineStart);
       const lineEnd = nl === -1 ? text.length : nl;
+      if (atLineStart && nl !== -1 && isBlankRawLine(text, lineStart, lineEnd)) backtickHazard = false;
       for (let i = lineStart; i < lineEnd; i++) {
         const ch = text[i];
         if (ch === "`") backtickHazard = true;
@@ -4383,6 +4458,7 @@ function findRawSafeCut(active) {
       if (nl === -1) break;
       if (!backtickHazard && !latentLt) lastCut = offset + nl + 1;
       lineStart = nl + 1;
+      atLineStart = true;
     }
     offset += text.length;
   }
@@ -4391,11 +4467,14 @@ function findRawSafeCut(active) {
 var DEFAULT_FREEZE_ATTEMPT_THRESHOLD = 512;
 function createIncrementalLatexPreprocessor(options) {
   const freezeThreshold = options?.freezeThreshold ?? DEFAULT_FREEZE_ATTEMPT_THRESHOLD;
+  const onAttempt = options?.onAttempt;
+  const backoff = options?.backoff ?? true;
   let prevSource = "";
   let prevOutput = "";
   let frozenSrcEnd = 0;
   let frozenOut = "";
   let triggered = false;
+  let nextAttemptLen = 0;
   return function incrementalPreprocessLaTeX(source) {
     if (source === prevSource) return prevOutput;
     const isAppend = source.length > prevSource.length && source.startsWith(prevSource);
@@ -4403,6 +4482,7 @@ function createIncrementalLatexPreprocessor(options) {
       frozenSrcEnd = 0;
       frozenOut = "";
       triggered = false;
+      nextAttemptLen = 0;
     }
     if (!triggered) {
       const checkFrom = isAppend ? Math.max(0, prevSource.length - 1) : 0;
@@ -4414,18 +4494,26 @@ function createIncrementalLatexPreprocessor(options) {
       triggered = true;
     }
     let active = source.slice(frozenSrcEnd);
-    if (active.length > freezeThreshold) {
+    if (active.length > freezeThreshold && active.length >= nextAttemptLen) {
+      const activeLength = active.length;
+      let advanced = false;
+      let frozenBytes = 0;
+      const freeze = (cut2, slice) => {
+        frozenOut += slice.out;
+        frozenSrcEnd += cut2;
+        active = source.slice(frozenSrcEnd);
+        advanced = true;
+        frozenBytes = cut2;
+      };
       const cut = findRawSafeCut(active);
       if (cut > 0) {
         const candidate = processSliceInstrumented(active.slice(0, cut));
-        if (candidate.quiescent) {
-          frozenOut += candidate.out;
-          frozenSrcEnd += cut;
-          active = source.slice(frozenSrcEnd);
-        }
+        if (candidate.quiescent) freeze(cut, candidate);
       }
+      nextAttemptLen = advanced || !backoff ? 0 : active.length * 2;
+      onAttempt?.({ activeLength, frozenBytes });
     }
-    const tail = processSliceInstrumented(active);
+    const tail = processSliceInstrumented(active, false);
     const head = tail.truncatedAtSeamStart ? frozenOut.replace(/\s+$/, "") : frozenOut;
     const out = head + tail.out;
     prevSource = source;