brookmd 0.26.1 → 0.27.0

package/CHANGELOG.md

@@ -4,6 +4,49 @@ Notable changes to brookmd (formerly `flux-md`). Format based on
 [Keep a Changelog](https://keepachangelog.com/); this project aims to follow
 [Semantic Versioning](https://semver.org/).
 
+## 0.27.0 — 2026-07-31
+
+### Added
+
+- **Streaming syntax highlighting — open code blocks now highlight live, on by
+  default.** Since 0.19 the built-in highlighter deliberately waited for a
+  fence to close; a streaming block showed plain text. Open blocks now render
+  highlighted as they grow, and the mechanism preserves every guarantee the
+  deferred design existed to protect:
+
+  - **Byte-identical at settle.** All committed markup comes from seeded runs
+    of the same resumable tokenizer that `highlight()` uses — the frozen
+    prefix is only ever extended at *checkpoints* (positions provably outside
+    any future token: after a newline in whitespace, ≥ 3 bytes behind the
+    stream head, with no unterminated string/comment/template live; HTML
+    checkpoints after `>` instead). When the block closes, one seeded run
+    finishes the tail, so the final bytes equal a one-shot `highlight()` —
+    pinned by a ~44,000-case fuzz (all languages, chunk sizes down to 1 byte,
+    plus speculative-revision streams) asserting settle-identity and that
+    every intermediate frozen prefix is a byte-prefix of the final output.
+  - **Bounded work, independent of block size.** Between checkpoints, an
+    unterminated string/comment advances via an O(1)-state scanner over only
+    the new bytes; the tail re-scan is capped at 8 KB (past the cap the tail
+    renders plain until the next checkpoint — correctness unaffected). Total
+    tokenization work measures ≈ 5.7× the streamed bytes end-to-end against
+    the real parser, flat from 288 B to 23 KB blocks (naive re-highlighting
+    measures 812× more at 23 KB); a work-bound test enforces < 6×. The
+    50 000-char plain-escape guard still applies and discards streaming state.
+  - **The tail is speculative.** The few tokens nearest the stream head may
+    change color as bytes arrive (an unterminated `"` reads as plain until its
+    closer lands) — the same speculative-tail behavior brookmd's links and
+    emphasis already have. Frozen output never changes.
+
+  Opt out with `streamingHighlight={false}` on `BrookMarkdown` (React) or
+  `streamingHighlight: false` in `mountBrookMarkdown` options (DOM, and via it
+  the Web Component and Vue/Svelte/Solid adapters) — that restores 0.26.1's
+  plain-until-close exactly. Not a `ParserConfig` field: highlighting is a
+  renderer concern; the parser, wire, and WASM are untouched. SSR is
+  unchanged (closed blocks, synchronous), and `components.CodeBlock` /
+  `pre` / `code` overrides still bypass the built-in highlighter entirely.
+  Closing a block that streamed in now also highlights near-instantly: the
+  close-time pass reuses the accumulated prefix instead of starting over.
+
 ## 0.26.1 — 2026-07-30
 
 Two fixes found by benchmarking 0.26.0 against real chat traffic. Requires

package/README.md

@@ -4,7 +4,7 @@ Zero-dep streaming markdown for the browser. Rust→WASM core, one Web Worker pe
 
 Drop in a streaming-aware renderer — **React, Vue, Svelte, Solid, a framework-agnostic `<brook-markdown>` Web Component, or the vanilla DOM mount** — wire each LLM stream to a `BrookClient`, and the markdown renders incrementally off the main thread, block by block, with stable identities so unchanged blocks never re-reconcile.
 
-Parsing runs entirely **off the main thread** — each stream gets its own pooled Web Worker, so many concurrent LLM responses render without contending for the UI thread. On each token the parser re-parses only the **active tail**, not the whole document; patches cross the worker boundary as **verified splices** (not full re-sends, so emitted bytes stay O(n) even for one giant growing block); and heavy renderers (syntax highlighting, math, mermaid) are **deferred until a block closes**. The result is low retained memory and a main thread that stays responsive while streaming. See [the live demo](https://md.hsingh.app/).
+Parsing runs entirely **off the main thread** — each stream gets its own pooled Web Worker, so many concurrent LLM responses render without contending for the UI thread. On each token the parser re-parses only the **active tail**, not the whole document; patches cross the worker boundary as **verified splices** (not full re-sends, so emitted bytes stay O(n) even for one giant growing block); and heavy renderers (math, mermaid) are **deferred until a block closes** — code fences highlight as they stream, re-tokenizing only the last line rather than the whole block per chunk. The result is low retained memory and a main thread that stays responsive while streaming. See [the live demo](https://md.hsingh.app/).
 
 > **Beyond the browser:** the same Rust core also powers experimental React
 > Native, Swift (iOS/macOS), Kotlin/Android, Flutter, and C-ABI bindings —
@@ -248,7 +248,9 @@ controlled-string helpers wrap; in vanilla you call it directly.
 
 `mountBrookMarkdown(client, container, options?)` returns `{ destroy(), refresh() }`.
 Options: `components`, `sanitize`, `virtualize`, `stickToBottom`, `highlightCode`
-(default true), `batch` (default true — one DOM write per `requestAnimationFrame`),
+(default true), `streamingHighlight` (default true — highlight a code fence while
+it is still streaming; see [Streaming syntax highlighting](#streaming-syntax-highlighting)),
+`batch` (default true — one DOM write per `requestAnimationFrame`),
 `morphOpenBlocks` (default false — morph a growing generic open block's subtree in
 place instead of rebuilding it via `innerHTML`, so only the changed parts repaint
 and focus/selection in the streaming tail survive; the rendered result is
@@ -1167,6 +1169,34 @@ import { highlight } from "brookmd/highlight";
 const html = highlight("const x = 1;", "ts");
 ```
 
+### Streaming syntax highlighting
+
+A code fence is highlighted **while it streams**, not only once it closes. On by
+default; turn it off with `streamingHighlight={false}` (React) or
+`{ streamingHighlight: false }` (the DOM mount options / Vue / Svelte / Solid),
+which restores the plain-until-close behaviour.
+
+It stays O(n) over the whole block rather than re-highlighting the fence on every
+chunk. An open block keeps a **frozen prefix** — markup for everything behind a
+checkpoint, which later bytes provably cannot rewrite — and re-tokenizes only the
+**tail** after it, about one source line's worth per patch. When the fence closes,
+the tokenizer resumes from that checkpoint instead of starting over, so a block
+that streamed in highlights near-instantly.
+
+Two things worth knowing:
+
+- **The tail is speculative.** A prefix of source does not tokenize like the same
+  prefix of a longer source: `const s = "hello` is a stray quote plus an
+  identifier until its closing quote lands, and `123.456e` is `123` plus loose
+  fragments until a digit arrives. So the last line's colours can shift as bytes
+  come in. Nothing behind the checkpoint ever changes.
+- **The settled markup is byte-identical** to `highlight(text, lang)` either way.
+  Turning this on or off changes when colour appears, never what it is.
+
+Blocks past the highlighter's 50 000-character guard, unknown languages, and any
+fence taken over by a `components.CodeBlock` / `pre` / `code` override are
+unaffected — they behave exactly as before.
+
 ## Coverage
 
 **CommonMark 0.31: 100% (652/652 spec examples), byte-exact** — every section,
@@ -1218,8 +1248,6 @@ By design, not yet, or only partially:
   (`<span>`/`<div class="math …">` with `gfmMath` on) and a `Mermaid` slot, but
   stays zero-dep: bring your own KaTeX / mermaid pass (or a `components.MathBlock`
   / `components.Mermaid` override) for the actual SVG/MathML output.
-- **Syntax highlighting on open code blocks** — deferred until close. This is a
-  deliberate perf choice.
 
 ## Performance
 

package/dist/dom.d.ts

@@ -64,6 +64,17 @@ export interface MountOptions {
     /** Use the built-in code highlighter. Default true; suppressed when a
      *  `components.CodeBlock` override is supplied. */
     highlightCode?: boolean;
+    /**
+     * Highlight a code fence **while it is still streaming**, instead of showing
+     * plain escaped text until it closes. On by default (parity with the React
+     * renderer's `streamingHighlight` prop).
+     *
+     * An open block keeps a frozen prefix and re-tokenizes only its tail on each
+     * patch, so this stays linear in the block's size. The settled markup is
+     * byte-identical either way — only the tail's colours are provisional, and
+     * they may shift as bytes arrive. Set `false` for the pre-0.27 behaviour.
+     */
+    streamingHighlight?: boolean;
     /** Coalesce patches into one DOM write per animation frame. Default true. */
     batch?: boolean;
     /**

package/dist/dom.js

@@ -1,4 +1,5 @@
 import { highlightDeferred } from "./hi-defer.js";
+import { createInc, incHighlight, incSeed } from "./hi-inc.js";
 import { morph } from "./morph.js";
 import { blockProps, extractLang } from "./block-props.js";
 import { decorateSegments } from "./decorate.js";
@@ -28,6 +29,7 @@ function mountBrookMarkdown(client, container, options = {}) {
   const hasInlineTransforms = !!decorators || !!urlTransform;
   const hasPerf = typeof performance !== "undefined";
   const highlightCode = options.highlightCode !== false && !components?.CodeBlock;
+  const streamingHighlight = options.streamingHighlight !== false;
   const batch = options.batch !== false && typeof requestAnimationFrame === "function";
   const morphOpenBlocks = options.morphOpenBlocks === true;
   const root = document.createElement("div");
@@ -115,6 +117,7 @@ function mountBrookMarkdown(client, container, options = {}) {
         continue;
       }
       existing.table = void 0;
+      if (existing.inc && b.kind.type !== "CodeBlock") existing.inc = void 0;
       if (existing.highlight) {
         existing.highlight.cancel();
         existing.highlight = void 0;
@@ -133,6 +136,7 @@ function mountBrookMarkdown(client, container, options = {}) {
       for (const [id, mb] of mounted) {
         if (!seen.has(id)) {
           if (mb.highlight) mb.highlight.cancel();
+          mb.inc = void 0;
           mb.node.remove();
           mounted.delete(id);
         }
@@ -349,8 +353,19 @@ function mountBrookMarkdown(client, container, options = {}) {
   function renderCodeBlock(b, mb) {
     const lang = extractLang(b.html) || "text";
     const text = b.open ? "" : codeText(b);
-    const run = text ? highlightDeferred(text, lang) : null;
-    const highlighted = run ? run.html : null;
+    let openMarkup = null;
+    if (b.open && streamingHighlight) {
+      let inc = mb.inc;
+      if (inc === void 0 || inc.lang !== lang.toLowerCase()) {
+        inc = createInc(lang) ?? void 0;
+        mb.inc = inc;
+      }
+      if (inc !== void 0) openMarkup = incHighlight(inc, codeText(b));
+    }
+    const seed = !b.open && mb.inc !== void 0 ? incSeed(mb.inc, text, lang) : void 0;
+    const run = text ? highlightDeferred(text, lang, seed) : null;
+    if (!b.open) mb.inc = void 0;
+    const highlighted = openMarkup ?? (run ? run.html : null);
     const block = document.createElement("div");
     block.className = "brook-code-block" + (b.open ? " brook-streaming" : "");
     const header = document.createElement("div");
@@ -499,6 +514,7 @@ function mountBrookMarkdown(client, container, options = {}) {
           mb.highlight.cancel();
           mb.highlight = void 0;
         }
+        mb.inc = void 0;
       }
       root.remove();
     },

package/dist/hi-defer.d.ts

@@ -1,3 +1,4 @@
+import { type HighlightState } from "./hi.js";
 /**
  * Test-only: shrink the per-slice budget so a suite can force the deferred path
  * deterministically instead of betting on how fast the machine is. Call with no
@@ -27,9 +28,9 @@ export interface DeferredHighlight {
  * renderer can call it from a render pass and only reach for
  * {@link highlightDeferred} when this comes back empty.
  */
-export declare function highlightWithin(code: string, lang: string): string | null;
+export declare function highlightWithin(code: string, lang: string, seed?: HighlightState): string | null;
 /**
  * Highlight `code` without blocking: the first slice runs here, synchronously,
  * and the rest (if any) continues on later tasks. See {@link DeferredHighlight}.
  */
-export declare function highlightDeferred(code: string, lang: string): DeferredHighlight;
+export declare function highlightDeferred(code: string, lang: string, seed?: HighlightState): DeferredHighlight;

package/dist/hi-defer.js

@@ -8,12 +8,15 @@ function __setSliceMs(ms) {
 }
 const noop = () => {
 };
-function highlightWithin(code, lang) {
-  const state = { pos: 0, out: "" };
+function seeded(seed) {
+  return seed === void 0 ? { pos: 0, out: "" } : { pos: seed.pos, out: seed.out };
+}
+function highlightWithin(code, lang, seed) {
+  const state = seeded(seed);
   return runSlice(code, lang, state) ? state.out : null;
 }
-function highlightDeferred(code, lang) {
-  const state = { pos: 0, out: "" };
+function highlightDeferred(code, lang, seed) {
+  const state = seeded(seed);
   if (runSlice(code, lang, state)) {
     return { html: state.out, rest: null, cancel: noop };
   }

package/dist/hi-inc.d.ts

@@ -0,0 +1,102 @@
+import { type HighlightState } from "./hi.js";
+/**
+ * How an unbounded opener recognizes its terminator, with the O(1) state each
+ * one carries between patches:
+ *
+ * - `char` — the first occurrence of `ch` closes (`[^ch]*ch` forms: Go raw
+ *   strings, HTML/CSS attribute strings, Rust `#[…]`, Bash `'…'` and `${…}`).
+ * - `starSlash` — `prevStar`, for `/* … *\/`.
+ * - `dashGt` — `dashRun`, for `<!-- … -->`.
+ * - `esc` — `esc`, for backslash-escaped strings and template literals. A
+ *   backslash immediately before a newline is {@link DEAD}, not an escape: the
+ *   pattern's `\\.` does not match a newline, so the form can never close.
+ * - `triple` — `runLen`, for Python `"""` / `'''`.
+ * - `dbl` — `pendingQuote`, for SQL's doubled-quote escaping (`'a''b'`).
+ */
+type PredKind = {
+    k: "char";
+    ch: string;
+} | {
+    k: "starSlash";
+} | {
+    k: "dashGt";
+} | {
+    k: "esc";
+    ch: string;
+} | {
+    k: "triple";
+    ch: string;
+} | {
+    k: "dbl";
+    ch: string;
+};
+/** @internal */
+export interface Opener {
+    /** Sticky regex for the OPENING delimiter only — bounded, O(1) to test. */
+    re: RegExp;
+    /** The pattern class the tokenizer emits when this form IS terminated. */
+    cls: string;
+    /** The literal terminator. */
+    end: string;
+    pred: PredKind;
+}
+/** @internal The carried state of one opener's continuation predicate. */
+export interface OpenerScanState {
+    n: number;
+    f: boolean;
+}
+/** Per-block incremental state. Create with {@link createInc}, feed {@link incHighlight}. */
+export interface IncState {
+    /** The language key this state's tables were chosen for. */
+    readonly lang: string;
+    /** Settled through here: `[0, c)` of the source will never re-tokenize. */
+    c: number;
+    /** Markup for `[0, c)`. Always a byte-prefix of the block's final markup. */
+    frozenHtml: string;
+    /** The checkpoint BEFORE `c`, and the `frozenHtml` length that went with it —
+     *  the one step of rewind a tail revision needs (see {@link adopt}). */
+    c0: number;
+    frozenLen0: number;
+    /** The unbounded opener live at the tail, or `null`. */
+    opener: Opener | null;
+    /** `opener`'s carried predicate state. */
+    scan: OpenerScanState;
+    /** `opener` can never close — stop re-scanning and render the tail plain. */
+    sealed: boolean;
+    /** The source last fed in, for the append/revision guard. */
+    text: string;
+    /** Escaped source for `[c, plainUpto)` — the plain tail, extended in place. */
+    plain: string;
+    plainFrom: number;
+    plainUpto: number;
+    /** The markup last handed out, so a repeated feed of the same text is free. */
+    html: string | null;
+}
+/**
+ * State for a block in `lang`, or `null` when the language has no table (the
+ * plain-escape fallback has no token boundaries to checkpoint on).
+ */
+export declare function createInc(lang: string): IncState | null;
+/**
+ * Feed the block's CURRENT full source and get the markup for all of it, or
+ * `null` when the incremental path has bowed out (past {@link CLIFF}) and the
+ * caller should render the plain escaped body exactly as it does today.
+ *
+ * Calls must be append-only. Anything else — a `reset()`, a speculative tail
+ * revision, one block's id being reused for different content — is detected
+ * (`text.startsWith(prev)`, the same guard the DOM renderer's prefix-append fast
+ * path uses) and simply restarts the state from scratch.
+ */
+export declare function incHighlight(st: IncState, text: string): string | null;
+/**
+ * The `{pos, out}` a close-time `highlightWithin`/`highlightDeferred` run should
+ * resume from, or `undefined` when nothing was frozen or the state does not
+ * belong to `text`/`lang` (a revised block, a different language, a block that
+ * crossed the cliff). Then the close-time run is the unseeded one it always was.
+ */
+export declare function incSeed(st: IncState, text: string, lang: string): HighlightState | undefined;
+/** @internal Test-only: total source bytes re-tokenized since the last reset. */
+export declare function __getIncScanned(): number;
+/** @internal Test-only. */
+export declare function __resetIncScanned(): void;
+export {};

package/dist/hi-inc.js

@@ -0,0 +1,317 @@
+import { escapeHtml, stepHighlight } from "./hi.js";
+const GAP = 3;
+const CAP = 8192;
+const CLIFF = 5e4;
+const OPEN = 0;
+const CLOSED = 1;
+const DEAD = 2;
+const BLOCK_COMMENT = { re: /\/\*/y, cls: "com", end: "*/", pred: { k: "starSlash" } };
+const JS_OPENERS = [
+  BLOCK_COMMENT,
+  { re: /`/y, cls: "str", end: "`", pred: { k: "esc", ch: "`" } }
+];
+const RUST_OPENERS = [
+  BLOCK_COMMENT,
+  { re: /b?"/y, cls: "str", end: '"', pred: { k: "esc", ch: '"' } },
+  { re: /#!?\[/y, cls: "attr", end: "]", pred: { k: "char", ch: "]" } }
+];
+const PY_OPENERS = [
+  { re: /[fFrRbB]{0,2}"""/y, cls: "str", end: '"""', pred: { k: "triple", ch: '"' } },
+  { re: /[fFrRbB]{0,2}'''/y, cls: "str", end: "'''", pred: { k: "triple", ch: "'" } }
+];
+const GO_OPENERS = [
+  BLOCK_COMMENT,
+  { re: /`/y, cls: "str", end: "`", pred: { k: "char", ch: "`" } }
+];
+const BASH_OPENERS = [
+  { re: /"/y, cls: "str", end: '"', pred: { k: "esc", ch: '"' } },
+  { re: /'/y, cls: "str", end: "'", pred: { k: "char", ch: "'" } },
+  { re: /\$\{/y, cls: "var", end: "}", pred: { k: "char", ch: "}" } }
+];
+const SQL_OPENERS = [
+  BLOCK_COMMENT,
+  { re: /'/y, cls: "str", end: "'", pred: { k: "dbl", ch: "'" } },
+  { re: /"/y, cls: "str", end: '"', pred: { k: "dbl", ch: '"' } }
+];
+const HTML_OPENERS = [
+  { re: /<!--/y, cls: "com", end: "-->", pred: { k: "dashGt" } },
+  { re: /"/y, cls: "str", end: '"', pred: { k: "char", ch: '"' } },
+  { re: /'/y, cls: "str", end: "'", pred: { k: "char", ch: "'" } }
+];
+const CSS_OPENERS = [
+  BLOCK_COMMENT,
+  { re: /"/y, cls: "str", end: '"', pred: { k: "char", ch: '"' } },
+  { re: /'/y, cls: "str", end: "'", pred: { k: "char", ch: "'" } }
+];
+const OPENERS = {
+  js: JS_OPENERS,
+  javascript: JS_OPENERS,
+  ts: JS_OPENERS,
+  tsx: JS_OPENERS,
+  jsx: JS_OPENERS,
+  typescript: JS_OPENERS,
+  rust: RUST_OPENERS,
+  rs: RUST_OPENERS,
+  py: PY_OPENERS,
+  python: PY_OPENERS,
+  go: GO_OPENERS,
+  bash: BASH_OPENERS,
+  sh: BASH_OPENERS,
+  shell: BASH_OPENERS,
+  json: [],
+  sql: SQL_OPENERS,
+  html: HTML_OPENERS,
+  xml: HTML_OPENERS,
+  css: CSS_OPENERS
+};
+const GT_CHECKPOINT = /* @__PURE__ */ new Set(["html", "xml"]);
+function createInc(lang) {
+  const key = lang.toLowerCase();
+  if (!Object.prototype.hasOwnProperty.call(OPENERS, key)) return null;
+  return {
+    lang: key,
+    c: 0,
+    frozenHtml: "",
+    c0: 0,
+    frozenLen0: 0,
+    opener: null,
+    scan: { n: 0, f: false },
+    sealed: false,
+    text: "",
+    plain: "",
+    plainFrom: 0,
+    plainUpto: 0,
+    html: null
+  };
+}
+function divergence(a, b) {
+  const n = a.length < b.length ? a.length : b.length;
+  let i = 0;
+  while (i < n && a.charCodeAt(i) === b.charCodeAt(i)) i++;
+  return i;
+}
+function adopt(st, d) {
+  if (st.c > 0 && d >= st.c + GAP) {
+    dropTail(st);
+    return true;
+  }
+  if (st.c0 > 0 && d >= st.c0 + GAP) {
+    st.frozenHtml = st.frozenHtml.slice(0, st.frozenLen0);
+    st.c = st.c0;
+    st.c0 = 0;
+    st.frozenLen0 = 0;
+    dropTail(st);
+    return true;
+  }
+  return false;
+}
+function dropTail(st) {
+  st.opener = null;
+  st.scan = { n: 0, f: false };
+  st.sealed = false;
+  st.plain = "";
+  st.plainFrom = st.c;
+  st.plainUpto = st.c;
+  st.html = null;
+}
+function reset(st) {
+  st.c = 0;
+  st.frozenHtml = "";
+  st.c0 = 0;
+  st.frozenLen0 = 0;
+  st.opener = null;
+  st.scan = { n: 0, f: false };
+  st.sealed = false;
+  st.text = "";
+  st.plain = "";
+  st.plainFrom = 0;
+  st.plainUpto = 0;
+  st.html = null;
+}
+function incHighlight(st, text) {
+  if (st.text === text) return st.html;
+  const d = divergence(st.text, text);
+  const appended = d === st.text.length && text.length > st.text.length;
+  let from = st.text.length;
+  if (!appended) {
+    if (!adopt(st, d)) reset(st);
+    from = 0;
+  }
+  st.text = text;
+  if (text.length > CLIFF) {
+    reset(st);
+    st.text = text;
+    return null;
+  }
+  const live = st.opener;
+  if (live !== null) {
+    const r = st.sealed ? OPEN : feed(live.pred, st.scan, text, from, text.length);
+    if (r === DEAD) st.sealed = true;
+    if (r !== CLOSED) {
+      st.html = st.frozenHtml + plainTail(st, text);
+      return st.html;
+    }
+    st.opener = null;
+  }
+  if (text.length - st.c > CAP) {
+    st.html = st.frozenHtml + plainTail(st, text);
+    return st.html;
+  }
+  st.html = rescan(st, text);
+  return st.html;
+}
+function incSeed(st, text, lang) {
+  if (st.c === 0 || st.lang !== lang.toLowerCase()) return void 0;
+  if (text.length > CLIFF || text.length < st.c) return void 0;
+  if (!text.startsWith(st.text.slice(0, st.c))) return void 0;
+  return { pos: st.c, out: st.frozenHtml };
+}
+let scanned = 0;
+function __getIncScanned() {
+  return scanned;
+}
+function __resetIncScanned() {
+  scanned = 0;
+}
+function plainTail(st, text) {
+  if (st.plainFrom !== st.c || st.plainUpto > text.length) {
+    st.plainFrom = st.c;
+    st.plain = "";
+    st.plainUpto = st.c;
+  }
+  if (st.plainUpto < text.length) {
+    st.plain += escapeHtml(text.slice(st.plainUpto));
+    st.plainUpto = text.length;
+  }
+  return st.plain;
+}
+function rescan(st, text) {
+  const openers = OPENERS[st.lang];
+  const gt = GT_CHECKPOINT.has(st.lang);
+  const limit = text.length - GAP;
+  let cp = -1;
+  let cpOut = 0;
+  let liveAt = -1;
+  let liveOp = null;
+  let liveLen = 0;
+  const sink = (cls, start, end, outLen) => {
+    if (liveAt >= 0) return;
+    for (let i = 0; i < openers.length; i++) {
+      const op = openers[i];
+      op.re.lastIndex = start;
+      const m = op.re.exec(text);
+      if (m === null || m.index !== start) continue;
+      const terminated = cls === op.cls && end - start >= m[0].length + op.end.length && text.startsWith(op.end, end - op.end.length);
+      if (terminated) continue;
+      liveAt = start;
+      liveOp = op;
+      liveLen = m[0].length;
+      return;
+    }
+    if (gt) {
+      if (cls === "pun" && end - start === 1 && text.charCodeAt(start) === 62 && end <= limit) {
+        cp = end;
+        cpOut = outLen;
+      }
+      return;
+    }
+    if (cls !== "ws") return;
+    const hi = end < limit ? end : limit;
+    if (hi <= start) return;
+    const nl = text.lastIndexOf("\n", hi - 1);
+    if (nl < start) return;
+    cp = nl + 1;
+    cpOut = outLen - (end - cp);
+  };
+  const state = { pos: st.c, out: "" };
+  scanned += text.length - st.c;
+  while (!stepHighlight(text, st.lang, state, text.length, sink)) {
+  }
+  const full = st.frozenHtml + state.out;
+  if (cp > st.c) {
+    st.c0 = st.c;
+    st.frozenLen0 = st.frozenHtml.length;
+    st.frozenHtml += state.out.slice(0, cpOut);
+    st.c = cp;
+  }
+  const found = liveOp;
+  if (found === null) {
+    st.opener = null;
+  } else {
+    st.scan = { n: 0, f: false };
+    const r = feed(found.pred, st.scan, text, liveAt + liveLen, text.length);
+    st.opener = r === CLOSED ? null : found;
+    st.sealed = r === DEAD;
+  }
+  return full;
+}
+function feed(p, st, s, from, to) {
+  switch (p.k) {
+    case "char": {
+      const i = s.indexOf(p.ch, from);
+      return i >= 0 && i < to ? CLOSED : OPEN;
+    }
+    case "starSlash": {
+      for (let i = from; i < to; i++) {
+        const ch = s[i];
+        if (st.f && ch === "/") return CLOSED;
+        st.f = ch === "*";
+      }
+      return OPEN;
+    }
+    case "dashGt": {
+      for (let i = from; i < to; i++) {
+        const ch = s[i];
+        if (ch === ">" && st.n >= 2) return CLOSED;
+        st.n = ch === "-" ? st.n + 1 : 0;
+      }
+      return OPEN;
+    }
+    case "esc": {
+      for (let i = from; i < to; i++) {
+        const ch = s[i];
+        if (st.f) {
+          st.f = false;
+          if (ch === "\n") return DEAD;
+          continue;
+        }
+        if (ch === "\\") {
+          st.f = true;
+          continue;
+        }
+        if (ch === p.ch) return CLOSED;
+      }
+      return OPEN;
+    }
+    case "triple": {
+      for (let i = from; i < to; i++) {
+        if (s[i] === p.ch) {
+          st.n++;
+          if (st.n === 3) return CLOSED;
+        } else {
+          st.n = 0;
+        }
+      }
+      return OPEN;
+    }
+    case "dbl": {
+      for (let i = from; i < to; i++) {
+        const ch = s[i];
+        if (st.f) {
+          st.f = false;
+          if (ch !== p.ch) return CLOSED;
+          continue;
+        }
+        if (ch === p.ch) st.f = true;
+      }
+      return OPEN;
+    }
+  }
+}
+export {
+  __getIncScanned,
+  __resetIncScanned,
+  createInc,
+  incHighlight,
+  incSeed
+};

package/dist/hi.d.ts

@@ -8,6 +8,7 @@
  * highlight an open (streaming) block, which avoids re-highlighting the same
  * code on every chunk — the main perf win for streaming code.
  */
+export declare function escapeHtml(s: string): string;
 /**
  * The resumable tokenizer's cursor: `pos` is the next source index to consume,
  * `out` the markup emitted so far. Start a run at `{ pos: 0, out: "" }`.
@@ -16,6 +17,21 @@ export interface HighlightState {
     pos: number;
     out: string;
 }
+/**
+ * Called once per token the tokenizer emits, AFTER its markup is appended:
+ * `(cls, start, end, outLen)` where `cls` is the PATTERN class (`ws`, `str`,
+ * `com`, `pun`, `ident`… — not the `kw`/`fn`/`ty` refinement), `[start, end)` is
+ * the source span, and `outLen` is `state.out.length` once the token has been
+ * written. The catch-all one-character fallback reports `cls === ""`.
+ *
+ * Passing no sink is the default and costs one `undefined` test per token; the
+ * escape-fallback path (unknown language / over the size guard) emits no tokens
+ * and so reports nothing.
+ *
+ * @internal The incremental streaming path (hi-inc.ts) is the only consumer —
+ * it needs token boundaries to pick a checkpoint that survives an append.
+ */
+export type TokenSink = (cls: string, start: number, end: number, outLen: number) => void;
 /**
  * One resumable slice of {@link highlight}. Consumes WHOLE tokens from
  * `state.pos` until at least `chars` source characters have been taken (or the
@@ -32,6 +48,6 @@ export interface HighlightState {
  *
  * @internal Not part of the semver surface — use {@link highlight}.
  */
-export declare function stepHighlight(code: string, lang: string, state: HighlightState, chars: number): boolean;
+export declare function stepHighlight(code: string, lang: string, state: HighlightState, chars: number, sink?: TokenSink): boolean;
 export declare function highlight(code: string, lang: string): string;
 export declare function supportedLangs(): string[];

package/dist/hi.js

@@ -169,7 +169,7 @@ function escapeHtml(s) {
   }
   return out + s.slice(last);
 }
-function stepHighlight(code, lang, state, chars) {
+function stepHighlight(code, lang, state, chars, sink) {
   const conf = code.length > 5e4 ? void 0 : LANGS[lang.toLowerCase()];
   const stop = state.pos + (chars > 0 ? chars : 1);
   if (!conf) {
@@ -201,6 +201,7 @@ function stepHighlight(code, lang, state, chars) {
           finalCls = "ty";
         } else {
           out += escapeHtml(text);
+          if (sink) sink(cls, pos, after, out.length);
           pos = after;
           matched = true;
           break;
@@ -211,12 +212,14 @@ function stepHighlight(code, lang, state, chars) {
       } else {
         out += `<span class="t-${finalCls}">${escapeHtml(text)}</span>`;
       }
+      if (sink) sink(cls, pos, after, out.length);
       pos = after;
       matched = true;
       break;
     }
     if (!matched) {
       out += escapeHtml(code[pos]);
+      if (sink) sink("", pos, pos + 1, out.length);
       pos += 1;
     }
   }
@@ -234,6 +237,7 @@ function supportedLangs() {
   return Object.keys(LANGS);
 }
 export {
+  escapeHtml,
   highlight,
   stepHighlight,
   supportedLangs

package/dist/react.d.ts

@@ -128,6 +128,22 @@ interface BrookMarkdownProps {
      * applies only to the streaming tail.
      */
     childMemo?: boolean;
+    /**
+     * Highlight a code fence **while it is still streaming**, instead of showing
+     * plain escaped text until it closes. On by default.
+     *
+     * An open block keeps a frozen prefix and re-tokenizes only its tail on each
+     * patch, so this stays linear in the block's size (it does not re-highlight
+     * the whole fence per chunk). The settled markup is byte-identical either way
+     * — only the tail's colours are provisional, and they may shift as bytes
+     * arrive (`"hello` is a stray quote plus an identifier until its closing quote
+     * lands). Set `false` for the pre-0.27 behaviour: plain body until close.
+     *
+     * No effect on SSR (the server renders closed blocks only), and none at all
+     * when `components.CodeBlock` / `components.pre` / `components.code` take over
+     * the block — an override bypasses the built-in highlighter entirely.
+     */
+    streamingHighlight?: boolean;
     /** Appended to the root's `className` (the `brook-md` class is always present). */
     className?: string;
     /** Set on the root element. */
@@ -264,6 +280,7 @@ interface BlockViewProps {
     virtualize?: boolean;
     sanitize?: (html: string) => string;
     childMemo?: boolean;
+    streamingHighlight?: boolean;
     onRenderMetrics?: RenderMetricsHook;
     decorators?: Decorator[];
     urlTransform?: UrlTransform;

package/dist/react.js

@@ -52,6 +52,7 @@ function BrookMarkdownFromClient({
   stickToBottom,
   sanitize,
   childMemo,
+  streamingHighlight,
   className,
   id,
   role,
@@ -104,6 +105,7 @@ function BrookMarkdownFromClient({
                 virtualize,
                 sanitize,
                 childMemo,
+                streamingHighlight,
                 onRenderMetrics: onMetrics,
                 decorators,
                 urlTransform,
@@ -486,6 +488,7 @@ function renderBlockContent({
   components,
   sanitize,
   childMemo,
+  streamingHighlight,
   decorators,
   urlTransform
 }) {
@@ -515,7 +518,8 @@ function renderBlockContent({
           {
             html: block.html,
             open: block.open,
-            code: typeof source === "string" ? source : void 0
+            code: typeof source === "string" ? source : void 0,
+            streamingHighlight
           }
         );
       }
@@ -580,7 +584,7 @@ function renderBlockContent({
 }
 function blocksEqual(prev, next) {
   if (prev.block == null || next.block == null) return prev.block === next.block;
-  return prev.block.id === next.block.id && prev.block.html === next.block.html && prev.block.open === next.block.open && prev.block.speculative === next.block.speculative && prev.components === next.components && prev.virtualize === next.virtualize && prev.sanitize === next.sanitize && prev.childMemo === next.childMemo && prev.onRenderMetrics === next.onRenderMetrics && // Identity compare: an unstable decorators/urlTransform (fresh each render)
+  return prev.block.id === next.block.id && prev.block.html === next.block.html && prev.block.open === next.block.open && prev.block.speculative === next.block.speculative && prev.components === next.components && prev.virtualize === next.virtualize && prev.sanitize === next.sanitize && prev.childMemo === next.childMemo && prev.streamingHighlight === next.streamingHighlight && prev.onRenderMetrics === next.onRenderMetrics && // Identity compare: an unstable decorators/urlTransform (fresh each render)
   // busts the memo so every committed block re-decorates — the O(n²) footgun
   // the dev warning calls out. A hoisted/memoized value keeps the memo holding.
   prev.decorators === next.decorators && prev.urlTransform === next.urlTransform && // Same identity rule as onRenderMetrics: an inline `onBlockError={() => …}`

package/dist/renderers/CodeBlock.d.ts

@@ -8,7 +8,9 @@ interface Props {
      * as the highlight itself. Absent (blockData off) the HTML is decoded here.
      */
     code?: string;
+    /** Highlight the block while it is still open. Default true. */
+    streamingHighlight?: boolean;
 }
-declare function CodeBlockImpl({ html, open, code }: Props): import("react/jsx-runtime").JSX.Element;
+declare function CodeBlockImpl({ html, open, code, streamingHighlight }: Props): import("react/jsx-runtime").JSX.Element;
 export declare const CodeBlock: import("react").MemoExoticComponent<typeof CodeBlockImpl>;
 export {};

package/dist/renderers/CodeBlock.js

@@ -2,19 +2,43 @@ import { Fragment, jsx, jsxs } from "react/jsx-runtime";
 import { memo, useCallback, useEffect, useMemo, useRef, useState } from "react";
 import { highlight } from "../hi.js";
 import { highlightDeferred, highlightWithin } from "../hi-defer.js";
+import { createInc, incHighlight, incSeed } from "../hi-inc.js";
 import { extractLang } from "../block-props.js";
 function decodeText(html) {
   const m = html.match(/<pre><code[^>]*>([\s\S]*?)<\/code><\/pre>/);
   if (!m) return "";
   return m[1].replace(/&lt;/g, "<").replace(/&gt;/g, ">").replace(/&quot;/g, '"').replace(/&#39;/g, "'").replace(/&amp;/g, "&");
 }
-function CodeBlockImpl({ html, open, code }) {
+function CodeBlockImpl({ html, open, code, streamingHighlight }) {
   const lang = extractLang(html) || "text";
   const text = useMemo(() => open ? "" : code ?? decodeText(html), [html, open, code]);
+  const streaming = open && streamingHighlight !== false;
+  const openText = useMemo(
+    () => streaming ? code ?? decodeText(html) : "",
+    [streaming, code, html]
+  );
+  const incRef = useRef(null);
+  const [inc, setInc] = useState(null);
   const sync = useMemo(() => {
     if (!text) return null;
-    return typeof window === "undefined" ? highlight(text, lang) : highlightWithin(text, lang);
+    if (typeof window === "undefined") return highlight(text, lang);
+    const st = incRef.current;
+    return highlightWithin(text, lang, st ? incSeed(st, text, lang) : void 0);
   }, [text, lang]);
+  useEffect(() => {
+    if (!streaming || typeof window === "undefined") {
+      incRef.current = null;
+      setInc((prev) => prev === null ? prev : null);
+      return;
+    }
+    let st = incRef.current;
+    if (st === null || st.lang !== lang.toLowerCase()) {
+      st = createInc(lang);
+      incRef.current = st;
+    }
+    const markup = st === null ? null : incHighlight(st, openText);
+    setInc(markup === null ? null : { lang, html: markup });
+  }, [streaming, openText, lang]);
   const [slow, setSlow] = useState(null);
   useEffect(() => {
     if (!text || sync !== null) {
@@ -38,7 +62,10 @@ function CodeBlockImpl({ html, open, code }) {
       run.cancel();
     };
   }, [text, lang, sync]);
-  const highlighted = sync ?? (slow !== null && slow.text === text && slow.lang === lang ? slow.html : null);
+  const highlighted = sync ?? (slow !== null && slow.text === text && slow.lang === lang ? slow.html : null) ?? // The streaming tail. Not gated on `openText` identity: the markup lags the
+  // props by one commit, and showing last patch's spans beats flashing the
+  // whole block back to plain every tick. A language change does invalidate it.
+  (streaming && inc !== null && inc.lang === lang ? inc.html : null);
   const [copied, setCopied] = useState(false);
   const timerRef = useRef(null);
   useEffect(() => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "brookmd",
-  "version": "0.26.1",
+  "version": "0.27.0",
   "description": "Zero-dep streaming markdown for the browser. Rust→WASM core, Web Worker per stream, incremental parse with speculative closure.",
   "type": "module",
   "sideEffects": ["./dist/worker.js", "./dist/styles.css"],