mathpix-markdown-it 2.0.38 → 2.0.40

package/pr-specs/2026-04-global-state-cleanup-and-perf.md

@@ -0,0 +1,212 @@
+# PR: Global state cleanup and performance improvements
+
+Status: Implemented
+Owner: @OlgaRedozubova
+
+---
+
+## Context
+
+Audit of the codebase revealed multiple module-level mutable state variables that accumulate data across documents in long-lived processes. Several hot-path data structures used `Array` + `findIndex()` for O(n) lookups that should be O(1) Maps, and `getInlineCodeListFromString` result was scanned with `.find()` per character — O(n × m).
+
+Two additional memory-retention bugs were uncovered during this work:
+
+- `mdPluginTOC` stored the parse `state` in a module-level `gstate` variable so the TOC render rule could reach the top-level token list. That reference was never cleared and kept the ENTIRE token tree pinned across unrelated parses until the next `md.use()` call. On tabular-heavy documents this alone retained hundreds of megabytes.
+
+- `coreInline` rebound `state.env = Object.assign({}, ...)` inside the inline loop. That desynced state.env from the env reference the caller of `md.render(src, env)` still held, so parse-time mutations (including our TOC / cache bookkeeping) became invisible to render rules that received the original env.
+
+---
+
+## Goal
+
+Fix memory leaks from module-level state, improve lookup performance for tabular-related data structures, add per-parse cleanup guarantees for reused md instances, and preserve the caller's env reference contract between parse and render.
+
+---
+
+## Non-Goals
+
+- Highlight dedup optimization (order-dependent behavior is pre-existing, changing risks regression without dedicated test coverage).
+- Migrating `mathTable` / `subTabular` / `extractedCodeBlocks` to `state.env` (they now work correctly with the two-hook scheme — `reset_tabular_state` at the start of parse plus `cleanup_tabular_state` at the end).
+- Browser-only concerns (click handlers, setInterval, context menu listeners).
+
+---
+
+## Current Behavior (before)
+
+- `Clear*` functions run once at `md.use()` time. If the md instance is reused for multiple documents, module-level state accumulates.
+- `mdPluginTOC` pins every parsed token tree on module-level `gstate` indefinitely.
+- `coreInline` replaces `state.env` with a fresh object per inline token, breaking the shared-env contract.
+- `diagboxTable` has no cleanup function at all — unbounded growth.
+- `subTabular`, `extractedCodeBlocks` use `Array` + `findIndex()` — O(n) per lookup.
+- `findEndMarker()` calls `.find()` on inline code list for every character position — O(n × m).
+- `labelsList` uses `Array` for all lookups — O(n) per `find`/`findIndex`.
+- `makeTagRegexes` creates 6 new `RegExp` objects per HTML block.
+- `SetItemizeLevelTokens` clones the entire `md.options` object on every call.
+
+---
+
+## Desired Behavior (after)
+
+- Every `md.parse()` begins with a `reset_tabular_state` core-ruler hook that clears module-level tabular state. A second `cleanup_tabular_state` hook runs at the end of the core pipeline and drops parse-only caches (`subTabular`, `mathTable`, `extractedCodeBlocks`, `diagboxTable`, and the column-style intern cache) so they're not retained through render.
+- `mdPluginTOC` stores the token list on `state.env[TOC_ENV_KEY]`, so the reference is released together with env when the parse ends. The TOC render rule reads the list from env instead of the removed `gstate`.
+- `coreInline` mutates `state.env` in place for the inline pass and derives a private `inlineEnv` for the nested `inline.parse()` call. The caller's env binding stays intact for downstream render rules.
+- All lookup data structures use `Map` for O(1) access.
+- Inline code position check is O(1) via `Set<number>`.
+- Tag-regex objects are cached and reused; `g`-flag regexes are matched with `.match()` to avoid `lastIndex` corruption across calls.
+- `SetItemizeLevelTokens` saves/restores only `outMath` with `try/finally`.
+
+---
+
+## Constraints / Invariants
+
+- `reset_tabular_state` and `cleanup_tabular_state` both respect `renderElement.startLine` — partial renders skip cleanup so the enclosing re-parse still sees the cached state.
+- `labelsByKey` / `labelsByUuid` survive `cleanup_tabular_state` — labels are read by render rules for `\ref{}` / `\eqref{}` resolution.
+- Highlight rendering files (`render-rule-highlights.ts`, `common.ts`) are NOT modified — reverted to master to avoid behavioural regression.
+- `labelsList` export kept for deep-import backward compatibility (deprecated, exposed as a `Proxy` that returns a version-cached snapshot of `labelsByKey.values()` — snapshot is rebuilt only when `addIntoLabelsList` / `clearLabelsList` bumps the version. Supports `.length`, iteration, and Array methods. Mutations (`.push`, index assignment) target the throwaway target array and are effectively ignored).
+
+### Per-parse cross-plugin state reset
+
+Sub-plugins (TOC, theorem, labels, footnotes, list-env, text counters) each maintain module-level state that was previously cleared only at `md.use` time or inside the `initMathpixMarkdown.parse` / `renderer.render` wrappers. Direct callers of `markdownIt().use(mathpixMarkdownPlugin)` that reuse one md instance across documents saw drift:
+
+- TOC slug registry (`slugsTocItems`) accumulated → second parse of the same document produced `#introduction-2`, `#introduction-3`, …
+- `theoremEnvironments` / `environmentsCounter` / `counterProof` kept stale theorem numbers across documents.
+- `labelsByKey` / `labelsByUuid` still held prior document's `\label{}` entries, so `\ref{}` could resolve to the wrong block.
+- `mmd_footnotes_list` carried old footnote items.
+- `itemizeLevelTokens` held parsed token trees from prior `\renewcommand{\labelitemi}` — visible as stale marker tokens in the next document.
+- `resetTextCounter` / size counter / `MathJax.Reset()` (equation numbering) were also wrapper-only.
+- `ParseErrorList` was the worst offender: `ClearParseErrorList()` was defined but never called anywhere — tabular parse errors grew monotonically.
+
+A new core-ruler hook `reset_mmd_global_state` (registered `before('normalize')` from `mathpixMarkdownPlugin`) calls all the reset functions at the start of every `md.parse()`. It respects `renderElement.startLine` so partial re-renders don't tear down the enclosing parse's cross-reference state. The same implementation is exported as `resetMmdGlobalState()` from the package root so one-shot converters (e.g. DOCX export) can release module-level state immediately after render without waiting for the next parse.
+
+### Additional parse-only retention fixes
+
+- `cleanup_math_cache` core-ruler hook (pushed, end of pipeline) clears `state.env.__mathpix`. Previously the per-parse math dedup cache was only initialized, never released — MathJax html/svg strings for every unique expression stayed on env until the caller dropped it.
+- `mdPluginTOC.grab_state` stashes `state.tokens` on `state.env[TOC_ENV_KEY]` only when the document actually used `[[toc]]`, detected by a one-pass scan of inline-token children for `toc_body`. Documents without `[[toc]]` no longer retain the whole token tree on env.
+
+### `markdownToHtmlPipelineSegments` balance fix
+
+The segments renderer tracked a single `pendingCloseTag` + `pendingLevel` pair, so a nested same-type same-level `_open` terminated the segment mid-block. Concrete case: md-theorem wraps an inner `paragraph_open` at level 0 inside the outer `paragraph_open` class `theorem_block` — the first `paragraph_close` closed the segment prematurely, leaving `<div><div>...</div>` in one segment and `</div></div>...` in the next. Added a `pendingDepth` counter so nested opens of the same type at the same level increment depth; the segment closes only when depth drops back to zero. Regression test in `tests/_html-segments.js` covers 38 scenarios across all block rules from `mmdRules.ts`.
+
+---
+
+## Done When
+
+- [x] Per-parse `reset_tabular_state` hook clears all tabular module-level state at the start of parse
+- [x] Post-parse `cleanup_tabular_state` hook drops parse-only caches (`subTabular`, `mathTable`, `extractedCodeBlocks`, `diagboxTable`, column-style intern cache) at the end of the core pipeline
+- [x] `mdPluginTOC` stores the token list on `state.env[TOC_ENV_KEY]`; module-level `gstate` removed
+- [x] `coreInline` mutates `state.env` in place instead of rebinding it
+- [x] `diagboxTable` has `ClearDiagboxTable()` + `diagboxById` reverse Map
+- [x] `subTabular` converted from Array to Map
+- [x] `extractedCodeBlocks` converted from Array to Map
+- [x] `labelsByKey` + `labelsByUuid` Map indexes for O(1) lookups
+- [x] `buildInlineCodePositionSet()` → `Set<number>` for O(1) position check
+- [x] `tagRegexCache` memoization + `.test()` → `.match()` fix for g-regex `lastIndex`
+- [x] `utf8Encode` uses `parts[]` + `join()` instead of `+=`
+- [x] `SetItemizeLevelTokens` saves/restores only `outMath` with `try/finally`
+- [x] All 3,286 tests pass
+
+---
+
+## Architecture
+
+### Two-hook cleanup scheme
+
+```
+core.ruler.before('normalize', 'reset_tabular_state', resetHook)
+  // clears state before parsing starts (defensive — also catches the case
+  // where the previous parse threw)
+core.ruler.push(                 'cleanup_tabular_state', cleanupHook)
+  // drops parse-only caches after the last core rule — they're never read
+  // during render
+```
+
+### TOC token-list handoff
+
+Old (leaked entire token tree):
+
+```ts
+let gstate;                                      // module scope
+md.core.ruler.push('grab_state', (state) => { gstate = state; });
+// ...
+const dataToc = getTocList(0, gstate.tokens, ...);
+```
+
+New (per-parse scope):
+
+```ts
+const TOC_ENV_KEY = '__mathpix_toc_tokens';
+md.core.ruler.push('grab_state', (state) => {
+  if (!state.env) state.env = {};
+  state.env[TOC_ENV_KEY] = state.tokens;
+});
+// ...
+const allTokens = (env && env[TOC_ENV_KEY]) || tokens || [];
+const dataToc = getTocList(0, allTokens, ...);
+```
+
+### coreInline env preservation
+
+Old (broke caller env):
+
+```ts
+state.env = Object.assign({}, {...state.env}, {
+  currentTag, ...envToInline,
+});
+state.md.inline.parse(token.content, state.md, state.env, token.children);
+```
+
+New (preserves caller env binding):
+
+```ts
+const inlineEnv = Object.assign({}, state.env, {currentTag}, envToInline);
+state.env.currentTag = currentTag;
+if (envToInline && typeof envToInline === 'object') {
+  Object.assign(state.env, envToInline);
+}
+state.md.inline.parse(token.content, state.md, inlineEnv, token.children);
+```
+
+The same pattern is applied in the deeper recursive walker `walkInlineInTokens` (used by `footnote_latex` / `tabular` deep-walk): it now also builds a private `inlineEnv` per token and mutates `state.env` in place, rather than rebinding it.
+
+---
+
+## Memory impact
+
+Benchmark document: 16 MB MMD with 13,713 tabular blocks, ~479K `<td>` cells, and ~49K inline math expressions.
+
+### Full SVG+HTML render
+
+| Stage                       | Before  | After   | Δ            |
+|-----------------------------|--------:|--------:|-------------:|
+| Peak heap (html held)       | 2597 MB |  778 MB | −1819 (−70%) |
+| Heap after releasing html   | 1887 MB |   68 MB | −1819 (−96%) |
+
+The bulk of the reduction comes from the TOC `gstate` / `coreInline` env fix: without it the token tree stayed pinned across parses and dominated the retained heap. The `cleanup_tabular_state` hook removes the remaining ~45 MB of parse-only caches that used to survive into the render phase.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/markdown/mdPluginTableTabular.ts` | `clearTabularState()` + two core-ruler hooks: `reset_tabular_state` (before normalize) and `cleanup_tabular_state` (push, end of pipeline) |
+| `src/markdown/mdPluginTOC.ts` | `TOC_ENV_KEY` on `state.env` replaces module-level `gstate`; render rule reads from env |
+| `src/markdown/md-inline-rule/core-inline.ts` | In-place `state.env` mutation, derived `inlineEnv` for nested `inline.parse` |
+| `src/markdown/md-block-rule/begin-tabular/sub-cell.ts` | `ClearDiagboxTable()`, `diagboxById` reverse Map, `buildInlineCodePositionSet` in `extractNextBraceContent` |
+| `src/markdown/md-block-rule/begin-tabular/sub-tabular.ts` | Array → Map, all `findIndex` → `.get()` |
+| `src/markdown/md-block-rule/begin-tabular/sub-code.ts` | Array → Map |
+| `src/markdown/md-block-rule/begin-tabular/sub-math.ts` | Single `tail` slice in `getMathTableContent` |
+| `src/markdown/common.ts` | `buildInlineCodePositionSet()`, `findEndMarker` uses `Set.has()` |
+| `src/markdown/common/labels.ts` | `labelsByKey` + `labelsByUuid` Map indexes |
+| `src/markdown/md-block-rule/mmd-html-block.ts` | `tagRegexCache` memoization, `.test()` → `.match()` for g-regex |
+| `src/markdown/md-svg-to-base64/base64.ts` | `parts[]` + `join()` in utf8Encode |
+| `src/markdown/md-latex-lists-env/re-level.ts` | Save/restore only `outMath` with `try/finally` + `beginCacheBypass`/`endCacheBypass` |
+| `src/markdown/mdPluginSeparateForBlock.ts` | Removed — dead code, never imported anywhere |
+
+---
+
+## Testing
+
+- All 3,286 tests pass
+- Per-parse cleanup verified: 100 sequential parses on the same md instance show no memory growth
+- Highlight rendering files reverted to master — zero risk of behavioural regression

package/pr-specs/2026-04-optimize-tabular-parsing.md

@@ -0,0 +1,211 @@
+# PR: Optimize tabular parsing performance
+
+Status: Implemented
+Owner: @OlgaRedozubova
+
+---
+
+## Context
+
+Parsing and rendering large documents with many `\begin{tabular}` blocks was slow and memory-hungry. The benchmark document — a 16 MB MMD with 13,713 tabular blocks, ~479K `<td>` cells, and ~49K inline math expressions — showed a peak heap of 2.6 GB for the full SVG/HTML render path.
+
+Profiling revealed three classes of waste:
+
+1. **Algorithmic**: `getSubMath()` rebuilt the placeholder string on every math expression found (O(N×M)); several lookup tables used `Array + findIndex()`; MathJax was invoked for every math token, including 67% duplicates.
+2. **Per-token allocation**: CSS border strings, `attrs` arrays, and close-token objects were allocated fresh for every `<td>`/`<tr>`/`<table>` even though the underlying content was identical across thousands of tokens.
+3. **Speculative output**: `renderInlineTokenBlock` always built all five outputs (HTML, TSV, CSV, tableMd, tableSmoothed) regardless of which one the caller actually consumed; `token.mathEquation` was stored on every math token even when the caller only walked the token tree.
+
+---
+
+## Goal
+
+Reduce parse time and memory usage for documents with many tabular environments and repeated math expressions. Avoid building per-token data that the caller will not read.
+
+---
+
+## Non-Goals
+
+- Persistent cross-parse MathJax cache (decided against — complexity outweighs benefit for the supported use cases; cache is per-parse via `state.env`).
+- Migrating all module-level state to `state.env` (tracked in the sibling `global-state-cleanup-and-perf.md` spec).
+- Changing the markdown-it Token shape (kept for downstream compatibility).
+
+---
+
+## Done When
+
+- [x] `getSubMath` is iterative single-pass
+- [x] `mathTable` is `Map<string, string>`
+- [x] Per-parse math cache in `state.env.__mathpix` deduplicates identical math
+- [x] Cache bypass for `SetItemizeLevelTokens` forDocx mutation
+- [x] Accessibility `mjx-mml-*` IDs remain unique on cache hits
+- [x] `outMath.skipMathToHtml` option — skips SVG serialization and `token.mathEquation` storage for token-only callers
+- [x] Border-style strings pre-interned; composed cell style deduplicated per parse
+- [x] Shared attrs arrays for `td_open` / `tr_open` / `table_open` / `tbody_open` with clone-on-write
+- [x] Frozen singleton close-tokens for `td_close` / `tr_close` / `table_close`
+- [x] `StatePushTabulars` skips `content` / `children = []` assignments on marker tokens
+- [x] `res.concat` → in-place `res.push` inside the tabular construction loop
+- [x] `renderInlineTokenBlock` and `renderNonTableTokenIntoCell` build each output only when the caller requested it via `options.outMath.include_*`
+- [x] HTML-visual attrs (style, `_empty` class, `table_tabular` class, `border-*` tr style) skipped when the caller sets `forMD` or `forLatex`
+- [x] `token.mathData.svg` retained only when highlights are active
+- [x] Empty `labels` object is returned as `null` from MathJax `OuterData` (no empty `{}` per math token)
+- [x] All 3,286 tests pass
+- [x] New unit tests cover: `getSubMath` edge cases, math-cache dedup/bypass/isolation, accessibility ID uniqueness, cell-attrs sharing, round-trip placeholder restoration
+
+---
+
+## Architecture
+
+### Iterative getSubMath (sub-math.ts)
+
+Single-pass scan with a local `new RegExp(RE_MATH_OPEN.source, 'g')`:
+- `RE_MATH_OPEN` stored as literal without the `/g` flag (immutable template)
+- Each call creates a local copy — reentrant-safe
+- The `startPos: number = 0` optional parameter is preserved for signature compatibility with the earlier recursive implementation; it is seeked to via `re.lastIndex = startPos` before the scan loop
+- `getEndMarker()` uses capture groups for eqref/ref detection (no substring matching)
+- `shouldSkipDollar()` validates `$`/`$$` edge cases
+- `mathTablePush()` accepts both `(id, content)` and `({id, content})` for backward compatibility
+
+### colsToFixWidth: Array → Set (parse-tabular.ts)
+
+The per-table set of column indices that need fixed width was tracked as a `number[]` with `.includes()` + `.push()` for dedup — O(N²) in cell count for wide tables. Converted to a local `Set<number>` with `.add()` (O(1) dedup). Converted to a plain array once at `tableOpen.meta.colsToFixWidth` assignment so downstream consumers (`shouldRewriteColSpec`) keep their existing array input shape. Removes four identical `if-includes-push` fragments along the way.
+
+### Dead split/join round-trips removed (common.ts)
+
+`getColumnLines` and `getColumnAlign` ended with `.split('').join('')` — an identity operation that allocates a per-call character array for nothing. The neighbouring `.split('').join(' ')` (which inserts spaces between characters) is not a no-op and is preserved.
+
+### Per-parse math cache (convert-math-to-html.ts)
+
+```
+state.env.__mathpix = {
+  inlineCache:  Map<math, CachedResult>,   // inline_math tokens
+  displayCache: Map<math, CachedResult>,   // display_math tokens
+  cacheBypass:  number,                    // >0 disables cache
+}
+```
+
+Initialized by the `init_math_cache` core-ruler hook. On cache hit with accessibility, the original `mjx-assistive-mml` id is replaced with a fresh one from `MathJax.nextAssistiveId()`.
+
+Cache hits mark the returned `TypesetResult` with `_labelsRegistered: true` so `convertMathToHtml` skips the label-registration loop — the `state.md.inline.parse` for every `\label{}` tag and the `addIntoLabelsList` side-effect already ran on the first miss and are idempotent for the same key+content. `idLabels` is still computed from `Object.keys(token.labels)` so downstream ref/eqref resolution is unaffected.
+
+### `outMath.skipMathToHtml` (convert-math-to-html.ts)
+
+When set, `applyTypesetResultToToken` does not copy the serialized math HTML to `token.mathEquation`, and `typesetMathForToken` path 4 forces `outMath.include_svg = false` for the MathJax call so the SVG string is never built. Other MathJax outputs (mathml_word, asciimath, metrics) still populate when the caller enabled them.
+
+### Pre-interned border styles + per-parse style intern (tabular-td.ts)
+
+`verticalCellLine` / `horizontalCellLine` switch on the line token and return one of 16 module-level border strings (solid/double/dashed/none × 4 sides) — template-literal allocation per call is gone. The composed cell style (`composeCellStyle`) is interned in a per-parse `columnStyleCache` so repeated cells share a single string instance.
+
+### Shared attrs for structural tabular tokens
+
+`getSharedCellAttrs(style, isEmpty, skipVisual)`, `getSharedTableOpenAttrs(extraClass?, skipVisual)`, `getSharedTbodyOpenAttrs(numCol)`, and `getSharedTrOpenAttrs(skipVisual)` return read-only attrs arrays cached per-parse by key. The shared arrays carry the non-enumerable `Symbol.for('mathpix.tabular.attrsShared')` marker so mutation sites (`tokenAttrSet` in the tabular renderer, `addAttributesToParentTokenByType` in utils) detach a private clone before writing.
+
+### Frozen close-token singletons
+
+`td_close`, `tr_close`, `table_close`, and `tbody_close` (non-forLatex only) have no variable data. A single `Object.freeze`d instance per kind is exported and pushed everywhere these markers appear. Under `forLatex`, `tbody_close` carries a per-table `latex` payload and is allocated per-instance.
+
+`StatePushTabulars` no longer assigns `content` / `children = []` onto marker tokens — those fields are never read on open/close markers and assignment would throw on the frozen close singletons. The multi-column branch of `parse-tabular.ts` also pushes `SHARED_TD_CLOSE` directly instead of allocating a fresh `{token:'td_close', ...}` object per cell.
+
+`addStyle` / `addHLineIntoStyle` (`tabular-td.ts`) check the input attrs for the `attrsSharedMarker` symbol and clone before mutating so that callers which pass in a shared-attrs array do not corrupt the cached object.
+
+### Output gating in renderInlineTokenBlock / renderNonTableTokenIntoCell
+
+The tabular renderer computes flags via a shared `computeOutputGates(options)` helper and only populates the outputs the caller asked for:
+
+```ts
+const { needHtml, needTsv, needCsv, needMd, needSmoothed } = computeOutputGates(options);
+// needHtml: !forMD && include_table_html !== false
+// needTsv/needCsv: include_tsv / include_csv
+// needMd: forMD || include_table_markdown
+// needSmoothed: forPptx
+```
+
+`renderInlineTokenBlock` and `renderNonTableTokenIntoCell` both use the same helper so their gating cannot drift. Every `result += ...`, `arr*.push(...)`, `cellMd += ...`, and `formatTsvCell` / `formatCsvCell` call is gated on the corresponding flag.
+
+Leaf-token handling still invokes `slf.renderInline([token], options, env)` even when `needHtml` is false — the `latex_list_item_open` render rule sets `token.meta.itemizeLevel` as a side effect, which `handleListTokensForCellMarkdown` reads to emit list markers.
+
+### HTML-visual attrs skipped for forMD / forLatex
+
+`td_open.style`, `td_open.class='_empty'`, `tr_open.style`, `table_open.class='tabular'`, and the `table_tabular` class + text-align style on the wrapping `paragraph_open` are HTML/CSS-only. When the caller sets `forMD` or `forLatex`, `AddTd` / `AddTdSubTable` / `getMultiColumnMultiRow` / `StatePushParagraphOpen` skip these. Colspan / rowspan on multicol/multirow cells and `paragraph_open.data-align` (for forLatex) are preserved.
+
+### mathData.svg only retained under highlights
+
+`applyTypesetResultToToken` shallow-clones `mathData` without the `svg` field unless `options.highlights?.length` is set. The SVG is still available on `token.mathEquation` (the HTML render rule path) — only the duplicate copy on `mathData` is dropped. The highlight render path re-populates `mathData.svg` in `convertMathToHtmlWithHighlight`. Invariant: the `highlights` length is read at parse time; mutating `options.highlights` between parse and render does not retroactively re-populate `mathData.svg`.
+
+### Hot-path outMath spread memoization (skipMathToHtml path)
+
+`typesetMathForToken` under `skipMathToHtml: true` forces `include_svg: false` for the MathJax call. The spread `{ ...options.outMath, include_svg: false }` would run per math token (~49K times on the benchmark). A `WeakMap<outMath, clone>` cache memoizes the spread — first token pays the allocation, the rest reuse the cached clone. GC-friendly: when the parse's `options.outMath` is released, the cache entry goes with it.
+
+### OuterData empty-labels guard (mathjax/index.ts)
+
+When the current equation has no labels, `res.labels` is set to `null` instead of `{...emptyObject}` so each math token does not carry an empty `{}` allocation.
+
+### res.concat → res.push in tabular construction
+
+Tabular token arrays are assembled in a hot loop. `res = res.concat(data.res)` allocated a new array on every iteration; replaced with `for (const t of data.res) res.push(t)` to mutate in place.
+
+---
+
+## Public API changes
+
+| Option | Type | Default | Effect |
+|--------|------|--------:|--------|
+| `outMath.skipMathToHtml` | boolean | `false` | When `true`, skips SVG serialization and `token.mathEquation` storage; other MathJax outputs still respect their own `include_*` flags. Intended for callers that walk tokens directly and never read the serialized math HTML. |
+
+No other options introduced. Existing flags — `options.forMD`, `options.forLatex`, `options.forDocx`, `options.forPptx`, `outMath.include_tsv` / `include_csv` / `include_table_html` / `include_table_markdown`, `options.highlights` — drive the output-gating decisions described above.
+
+---
+
+## Memory impact
+
+Benchmark document: 16 MB MMD with 13,713 tabular blocks, ~479K `<td>` cells, and ~49K inline math expressions.
+
+### Full SVG/HTML render path
+
+| Stage                   | Before  | After  | Δ            |
+|-------------------------|--------:|-------:|-------------:|
+| Peak heap (html held)   | 2597 MB | 778 MB | −1819 (−70%) |
+| Heap after drop html    | 1887 MB |  68 MB | −1819 (−96%) |
+| Parse time              |  17.9 s | 14.6 s |        −18%  |
+| HTML output size        |  355 MB | 355 MB |             0 |
+
+### Token-only path (`forMD: true`, `outMath.skipMathToHtml: true`)
+
+| Stage                   | Before  | After  | Δ            |
+|-------------------------|--------:|-------:|-------------:|
+| Peak heap               | 2597 MB | 443 MB | −2154 (−83%) |
+| Heap after drop output  | 1887 MB |  81 MB | −1806 (−96%) |
+| Parse time              |  17.9 s | 20.5 s |               |
+| Serialized output size  |  355 MB | 165 MB |        −190  |
+
+Parse time on the token-only path includes more bookkeeping but skips SVG serialization; wall-clock depends heavily on which MathJax outputs the caller enables.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/markdown/md-block-rule/begin-tabular/sub-math.ts` | Iterative `getSubMath`; `mathTable` Array→Map; `mathTablePush` overload; `getEndMarker` with capture groups |
+| `src/markdown/md-block-rule/begin-tabular/tabular-td.ts` | Pre-interned border strings; `columnStyleCache` per-parse intern; `getSharedCellAttrs` / `getSharedTableOpenAttrs` / `getSharedTrOpenAttrs` / `getSharedTbodyOpenAttrs` with clone-on-write marker; frozen `SHARED_TD_CLOSE` / `SHARED_TR_CLOSE` / `SHARED_TABLE_CLOSE`; `skipVisual` pathway in `AddTd` / `AddTdSubTable` |
+| `src/markdown/md-block-rule/begin-tabular/parse-tabular.ts` | Derives `skipVisual = forMD \|\| forLatex`; gates style-only calls (`addHLineIntoStyle`, shared attrs helpers); `res.concat` → `res.push` in the construction loop |
+| `src/markdown/md-block-rule/begin-tabular/multi-column-row.ts` | `getMultiColumnMultiRow` honors `skipVisual`; colspan/rowspan always preserved, style/width skipped under the flag |
+| `src/markdown/md-block-rule/begin-tabular/index.ts` | `StatePushTabulars` skips `content` / `children = []` on open/close markers; `StatePushParagraphOpen` gates class/style on `forMD \|\| forLatex`, preserves `data-align` for forLatex |
+| `src/markdown/md-inline-rule/tabular.ts` | Subtable `table_open` uses `getSharedTableOpenAttrs('subtable')` instead of mutating shared attrs |
+| `src/markdown/md-renderer-rules/render-tabular.ts` | Clone-on-write `tokenAttrSet`; per-output `needHtml` / `needTsv` / `needCsv` / `needMd` / `needSmoothed` gates throughout `renderInlineTokenBlock` and `renderNonTableTokenIntoCell`; leaf-token `renderInline` retained for list-marker side effect |
+| `src/markdown/common/convert-math-to-html.ts` | Per-parse cache in `state.env.__mathpix`; `initMathCache`; `beginCacheBypass` / `endCacheBypass`; accessibility ID replacement; `outMath.skipMathToHtml` path that disables SVG serialization and `mathEquation` storage; `mathData.svg` gated on `options.highlights` |
+| `src/markdown/mdPluginRaw.ts` | Registers `init_math_cache` core-ruler hook |
+| `src/markdown/md-latex-lists-env/re-level.ts` | `beginCacheBypass` / `endCacheBypass` around forDocx mutation; `try/finally` for `outMath` restoration |
+| `src/markdown/utils.ts` | Clone-on-write in `addAttributesToParentTokenByType` (shared attrs marker) |
+| `src/mathjax/index.ts` | `OuterData` returns `null` for empty `labels` instead of an empty `{}` clone |
+| `tests/_sub-math.js` | Unit tests for `getSubMath` edge cases |
+| `tests/_typeset-cache.js` | Tests for dedup, env isolation, mutation protection, accessibility IDs, bypass counter, forDocx integration, mathData svg gating |
+| `tests/_accessibility.js` | Tests for unique IDs with cache, aria-labelledby consistency |
+
+---
+
+## Testing
+
+- All 3,286 tests pass
+- `outMath.skipMathToHtml` is opt-in; existing callers see identical output
+- Shared attrs + close-token singletons verified correct under highlight/diagbox/forDocx paths (tests exercise cloning)
+- MD list markers in tabular cells verified — leaf-token `renderInline` is still invoked so the `latex_list_item_open` render rule can set `token.meta.itemizeLevel`