@coldsmirk/inkstone-monaco 0.29.0 → 0.31.0

package/README.md

@@ -6,20 +6,21 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 
 ## What you get
 
-- **`ensureMonacoHost(options?)`** — the process-wide host: resolves the local `monaco-editor` module (never the CDN), routes **all** built-in language workers (base editor, TypeScript/JavaScript, JSON, CSS/scss/less, HTML/handlebars/razor) as bundler-emitted chunks so everything works offline, and applies `locale: "zh-cn"` via Monaco's official NLS catalog when asked — plus, on every editor created under that host, zh-CN descriptions for the six command-palette actions Monaco describes in English literals that no catalog can reach (Fold, Unfold, Find with Arguments, Refactor..., Source Action..., Paste As...), each held to the installed `monaco-editor` by an upgrade-tripwire test. Idempotent — every caller gets the same module; a load failure is not cached, so a fresh call retries.
+- **`ensureMonacoHost(options?)`** — the process-wide host: resolves the local `monaco-editor` module (never the CDN) and the parts of it the Shiki tokenization stands on, routes **all** built-in language workers (base editor, TypeScript/JavaScript, JSON, CSS/scss/less, HTML/handlebars/razor) and inkstone's own tokenization worker (the `textmate` label) as bundler-emitted chunks so everything works offline, and applies `locale: "zh-cn"` via Monaco's official NLS catalog when asked — plus, on every editor created under that host, zh-CN descriptions for the six command-palette actions Monaco describes in English literals that no catalog can reach (Fold, Unfold, Find with Arguments, Refactor..., Source Action..., Paste As...), each held to the installed `monaco-editor` by an upgrade-tripwire test. Idempotent — every caller gets the same module; a load failure is not cached, so a fresh call retries.
 - **`getWorker` extension point** — plug third-party language workers (e.g. monaco-yaml) into the routing; return `undefined` to fall back to the built-in routes.
 - **`monacoLanguages`** — the sorted catalog of every built-in language id the installed Monaco registers (with the `MonacoBuiltInLanguage` union type behind it), held to the installed `monaco-editor` by an upgrade-tripwire test. These are *ids*, not display names (`proto`, `sol`, `aes`); an unknown id silently falls back to `plaintext`.
 - **`disabledKeybindings` / `installDisabledKeybindings`** — swallow the find/replace chords and the command palette's F1 for editors that opt in. Monaco's standalone keybinding service is page-global and its rules are permanent, so the install registers each chord once per monaco module's service (duplicate copies of this package share the ledger; independent monaco bundles register their own) and scopes it with a per-capability context key created on the opted-in editor; the chord table is held to the installed `monaco-editor` by an upgrade-tripwire test.
 - **`installJsxTagClosing`** — VS Code's auto-closing JSX tags: a typed `>` that completes an opening tag inserts the matching closing tag and keeps the caret between the pair. Parser-backed: the bundled TS worker subclasses monaco's stock worker to expose `getJsxClosingTagAtPosition` — a language-service call monaco never proxied (VS Code ships the feature in its TypeScript extension, outside the editor core) — so generics, comparisons, arrow functions, and already-closed elements never trigger. Only a *typed* `>` triggers: the gates require one bare single-character insertion into a **writable** TypeScript/JavaScript model whose path ends in `.tsx` / `.jsx`, and the reply lands only while the caret still sits right after that `>` — so `replaceInEditor` replacements, `StreamingEditorController` appends (the stream holds the editor read-only), and other programmatic writes never earn a closing tag, and the insert arrives as its own undo step (one undo removes just the closing tag). A `getWorker` override that routes the `typescript` label to a worker without the subclass is detected on its first lookup and the install disarms silently. Multi-cursor typing and the diff editor's panes are deliberately out of scope. `<MonacoEditor>` installs it automatically.
 - **`dropMonacoWorkerFallbacks()`** (from **`@coldsmirk/inkstone-monaco/vite`**) — a Vite plugin that removes monaco's built-in worker fallbacks at build time. Monaco's language workerManagers statically reference their workers as `new Worker(new URL(...))` fallbacks that never run under `ensureMonacoHost` (the environment check prefers `getWorker`), yet bundlers emit a chunk per reference — including a ~7MB TypeScript worker duplicate that is never fetched. The plugin rewrites the fallbacks into located throws so the duplicates are never emitted; if a monaco bump reshapes the pattern it warns and leaves the file untouched (the dead chunk returns, nothing breaks).
-- **Embedded-editor fixes** — works around [microsoft/monaco-editor#5177](https://github.com/microsoft/monaco-editor/issues/5177) (find-widget button tooltips landing on top of the buttons they describe) with a `width: max-content` hover-layer style tagged `data-inkstone="monaco-hover-geometry"`; deleted the moment upstream fixes it. The tooltip layer positions against the editor's container — give that container `position: relative`. The command palette's row tooltips have the same trouble inside the palette: 0.56 positions them against the palette's list, which sets no position of its own, so they landed above the row they describe (by the height of the filter box); a `position: relative` style on the list, tagged `data-inkstone="monaco-quick-input-geometry"`, puts them on their row until upstream does.
+- **Embedded-editor fixes** — works around [microsoft/monaco-editor#5177](https://github.com/microsoft/monaco-editor/issues/5177) (find-widget button tooltips landing on top of the buttons they describe) with a `width: max-content` hover-layer style tagged `data-inkstone="monaco-hover-geometry"`; deleted the moment upstream fixes it. And 0.57 draws an editor's context menu under `fixedOverflowWidgets` (inkstone's default) in a shadow root hosted in the node the editor was created in, which no theme color reaches — a transparent menu, its items over the code ([microsoft/monaco-editor#5511](https://github.com/microsoft/monaco-editor/issues/5511)); the host makes that node a `.monaco-component`, the class Monaco writes its theme's colors on, so the menu inherits them, and `<MonacoEditor>` keeps the class in the `className` it renders.
 - **`StreamingEditorController`** — stream generated text into an editor without routing every token through app state: `begin()` / `append(chunk)` / `end()` / `discard()`, `attach(editor)` / `detach(editor)` with buffered replay for late-mounting editors (and for a model swapped onto the attached editor mid-stream, including after the final chunk), and `isStreaming()` for the onChange guard. The stream locks every editor it touches read-only and `end()` / `discard()` restore each one's own setting — by handle, so an editor detached mid-stream is restored too; `discard()` puts every displaced document back into the model it came from, and `end()` does the same for every model except the one keeping the streamed text. Full-buffer writes and restorations preserve each model's UTF-8 BOM. Editors and models the app disposed mid-stream are skipped, never touched — teardown can race the stream without wedging it.
 - **`replaceInEditor(editor, old, new, all)`** — exact find-and-replace on an editor handle via `executeEdits`, preserving undo/cursor/folds; returns whether the edit landed (`false` = no editor / empty search / no match / a read-only editor refused it — e.g. one locked by a stream in progress), so a tool-call handler knows to fall back to a state write.
-- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module, once per distinct module: defines the highlighter's themes as Monaco themes, registers each loaded grammar's canonical name as a Monaco language where Monaco has none and installs a Shiki tokens provider for it, and patches `setTheme` so the Shiki theme ids route through the highlighter while non-Shiki ids (vs/vs-dark, caller-defined) keep Monaco's original path. Called again on a bridged module, it installs the grammars loaded since. A failed registration is restored and retried on the next call.
-- **Which regex engine** — the bridge tokenizes whole files through the shared highlighter, and `@coldsmirk/inkstone-core`'s default JavaScript engine translates each grammar's patterns at the first tokenization, on the main thread (a TypeScript file's first sixty lines: about 170 ms, colour landing a beat after the text on every first open of a language). An app that opens files in Monaco should configure `onigurumaEngine` from `@coldsmirk/inkstone-core` (`configureHighlighter({ engine: onigurumaEngine })`, before the first `getHighlighter()`): its WASM is inlined into your bundle and compiles the same lines in about 20 ms.
-- **`registerShikiLanguage(monaco, highlighter, language)`** — the lazy path: an app that loads grammars as files are opened calls this after each `highlighter.loadLanguage(...)` and before a model names the language. Registers the language where Monaco lacks it (`toml`, `shellscript`), installs its tokens provider once under the grammar's canonical name (an alias like `ts` collapses), and leaves the theme the editor shows alone — the one thing re-running `@shikijs/monaco`'s `shikiToMonaco` cannot promise, which is why the bridge is inkstone's own. Throws for a grammar the highlighter has not loaded.
-- **`installDesignTokens()`** — defines the VS Code design tokens monaco-editor 0.56's chrome reads and the standalone editor never does ([microsoft/monaco-editor#5408](https://github.com/microsoft/monaco-editor/issues/5408)): `--vscode-cornerRadius-{small,medium,large,xLarge}` and `--vscode-shadow-{md,lg,xl}`, on `.monaco-editor` (the context menu's shadow root and a host's `overflowWidgetsDomNode` inherit them). Without them the context menu, completion, hover, and find widgets ship square and shadowless. Each resolves through the host's `--inkstone-radius-*` / `--inkstone-shadow-*` token first (see `@coldsmirk/inkstone-core`), then a fallback scale in the spirit of 0.55's look. It also keeps an input box's focus ring single: the standalone editor outlines both the `.monaco-inputbox` container (its `synthetic-focus` ring, which follows the box's radius) and the inner `<input>` (square, 1px inside), and the inner ring's corners poke out as soon as the box has corners — the container's ring is the one ring. `ensureMonacoHost` installs it; idempotent.
-- **`DEFAULT_MONACO_OPTIONS`** — the inkstone construction-option baseline (no minimap, 14px/1.6 type on the shared `--inkstone-font-family-code` token, embedded-friendly find widget, tuned completion, thin scrollbars). Spread it to extend: `{ ...DEFAULT_MONACO_OPTIONS, wordWrap: "on" }`.
+- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module (one from `ensureMonacoHost`), once per distinct module: defines the highlighter's themes as Monaco themes, registers each loaded grammar's canonical name as a Monaco language where Monaco has none and its Shiki tokenization in Monaco's registry — over Monaco's own, JSON's included (its mode's tokenizer is turned off through its configuration, its other features left on) — and patches `setTheme` so the Shiki theme ids route through the highlighter while non-Shiki ids (vs/vs-dark, caller-defined) keep Monaco's original path. Called again on a bridged module, it registers the grammars loaded since. A failed registration is restored and retried on the next call.
+- **Tokenization off the page's thread** — VS Code's design, through the same door in Monaco: each model an editor shows is tokenized whole in a worker of its own, built from what the shared highlighter was built from (`highlighterSources`), the tokens and the state each line ends in sent back as they are made and moved past the edits the worker has not yet seen. The page's thread tokenizes only what Monaco still asks of it before the worker reaches it — a model's first lines in view, the lines an edit touches — and of those only a line it can afford: one of 1,000 characters or more, or one the grammar does not finish in 50 ms, is drawn in the default color until the worker's tokens arrive, the line after it read from the state before it (a minified line costs the grammar about 10 ms per 1,000 characters, 370 over 10,000, and one in view would hold the page that long). The worker bounds a line as VS Code does, at 20,000 characters and 500 ms. Where the page passes a line over in the lines it paints first, its own states for the lines after it stand — Monaco keeps a state it computed over one sent later — so an edit there is drawn from them until the worker's tokens arrive. A highlighter inkstone did not build, or a host that routes no worker for `textmate`, leaves every model to Monaco's own pass on the page's thread, under the same bound.
+- **Which regex engine** — the worker builds the engine kind the shared highlighter was configured with: Oniguruma wherever an engine was configured, the strict JavaScript engine where none was, so the page and the worker tokenize alike. The JavaScript engine translates each grammar's patterns at the first tokenization — in the worker for a model, and on the page for the lines it tokenizes itself (a TypeScript file's first sixty lines: about 170 ms) — so an app that opens files in Monaco should configure `onigurumaEngine` from `@coldsmirk/inkstone-core` (`configureHighlighter({ engine: onigurumaEngine })`, before the first `getHighlighter()`): its WASM is inlined into your bundle and compiles the same lines in about 20 ms.
+- **`registerShikiLanguage(monaco, highlighter, language)`** — the lazy path: an app that loads grammars as files are opened calls this after each `highlighter.loadLanguage(...)` and before a model names the language. Registers the language where Monaco lacks it (`toml`, `shellscript`), its tokenization once under the grammar's canonical name (an alias like `ts` collapses), the models of a grammar shiki has built again since (a language it embeds lazily arrived) tokenized again under the new one, and leaves the theme the editor shows alone — the one thing re-running `@shikijs/monaco`'s `shikiToMonaco` cannot promise, which is why the bridge is inkstone's own. Throws for a grammar the highlighter has not loaded.
+- **`installDesignTokens()`** — defines the VS Code design tokens monaco-editor's chrome has read since 0.56 and the standalone editor still never does ([microsoft/monaco-editor#5408](https://github.com/microsoft/monaco-editor/issues/5408)): `--vscode-cornerRadius-{small,medium,large,xLarge,circle}` and `--vscode-shadow-{md,lg,xl}`, on `.monaco-editor` (a host's `overflowWidgetsDomNode` carries the class) and on `.monaco-component` — every context view, the class Monaco writes its own theme colors on: an element's tooltip and, since 0.57 under `fixedOverflowWidgets`, the editor's context menu stand in the node the editor was created in, outside `.monaco-editor`. Without them the context menu, completion, hover, and find widgets ship square and shadowless. 0.57 also took the completion list off `--vscode-shadow-lg` for a literal `0 2px 8px` shadow; it is put back on the token, so it wears the host's shadow as the hover and the context menu do. Each resolves through the host's `--inkstone-radius-*` / `--inkstone-shadow-*` token first (see `@coldsmirk/inkstone-core`), then a fallback scale in the spirit of 0.55's look. It also keeps an input box's focus ring single: the standalone editor outlines both the `.monaco-inputbox` container (its `synthetic-focus` ring, which follows the box's radius) and the inner `<input>` (square, 1px inside), and the inner ring's corners poke out as soon as the box has corners — the container's ring is the one ring. `ensureMonacoHost` installs it; idempotent.
+- **`DEFAULT_MONACO_OPTIONS`** — the inkstone construction-option baseline (no minimap, 14px/1.6 type on the shared `--inkstone-font-family-code` token, embedded-friendly find widget, tuned completion, thin scrollbars). Spread it to extend: `{ ...DEFAULT_MONACO_OPTIONS, wordWrap: "on" }`. Left out on purpose: `fullwidthCharacterWidth: "twoCells"`, which centers each full-width character — CJK text among it — in exactly two cells of a monospace font, so it lines up with the grid the way a terminal draws it. It draws every such character as a span of its own, which costs on text dense with them (a Chinese JSON body, a document of prose), so it is a host's to turn on where the alignment matters more.
 - **`reconcileModelValue` / `modelValueMatches`** — the controlled-value reconcile: diff an external value against a live model (normalized to the model's EOL) and apply one minimal `applyEdits` replace that never lands on the undo stack or moves a caret outside the changed region.
 
 ## Install
@@ -28,7 +29,7 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 pnpm add @coldsmirk/inkstone-monaco monaco-editor
 ```
 
-`monaco-editor` is a **peer dependency** — your app owns the version. Node.js >= 24 for build / SSR hosts. Workers are bundled from dependency-relative `new Worker(new URL(...))` references; Vite and webpack 5 handle this form natively. Plain Rollup does not emit these workers by itself, so configure worker/asset handling that bundles the referenced modules or implement `getWorker` for every label your app can request.
+`monaco-editor` is a **peer dependency** — your app owns the version. Node.js >= 24 for build / SSR hosts. Workers are bundled from dependency-relative `new Worker(new URL(...))` references — monaco's five and inkstone's tokenization worker (`workers/textmate.worker.js`, under the `textmate` label); Vite and webpack 5 handle this form natively. Plain Rollup does not emit these workers by itself, so configure worker/asset handling that bundles the referenced modules or implement `getWorker` for every label your app can request.
 
 ## Quick start
 

package/dist/bounds-CZ_iAUoc.js

@@ -0,0 +1,120 @@
+import { INITIAL } from "@shikijs/vscode-textmate";
+//#region src/tokenization/content.ts
+const LINE_BREAK = /\r\n|\r|\n/;
+function lineCount(text) {
+	let count = 1;
+	for (let index = 0; index < text.length; index += 1) {
+		const char = text.codePointAt(index);
+		if (char === 10) count += 1;
+		else if (char === 13) {
+			count += 1;
+			if (text.codePointAt(index + 1) === 10) index += 1;
+		}
+	}
+	return count;
+}
+function lineSplice({ range, text }) {
+	return {
+		start: range.startLineNumber - 1,
+		removed: range.endLineNumber - range.startLineNumber + 1,
+		inserted: lineCount(text)
+	};
+}
+function applySplice(array, { start, removed, inserted }, fill) {
+	const RUN = 8192;
+	const items = Array.from({ length: inserted }, (_, index) => fill(index));
+	array.splice(start, removed, ...items.slice(0, RUN));
+	for (let from = RUN; from < items.length; from += RUN) array.splice(start + from, 0, ...items.slice(from, from + RUN));
+}
+function applyChange(lines, change) {
+	const { range, text } = change;
+	const written = `${(lines[range.startLineNumber - 1] ?? "").slice(0, range.startColumn - 1)}${text}${(lines[range.endLineNumber - 1] ?? "").slice(range.endColumn - 1)}`.split(LINE_BREAK);
+	applySplice(lines, lineSplice(change), (index) => written[index] ?? "");
+}
+function lineAfter(line, edits) {
+	let at = line;
+	for (const { changes } of edits) for (const change of changes) {
+		const { start, removed, inserted } = lineSplice(change);
+		if (at - 1 >= start + removed) at += inserted - removed;
+		else if (at - 1 >= start) return;
+	}
+	return at;
+}
+//#endregion
+//#region src/tokenization/states.ts
+const Stack = INITIAL.constructor;
+function stateDiff(previous, next) {
+	let pops = 0;
+	const newFrames = [];
+	let first = previous;
+	let second = next;
+	while (first !== second) if (first && (!second || first.depth >= second.depth)) {
+		pops += 1;
+		first = first.parent;
+	} else if (second) {
+		newFrames.push(second.toStateStackFrame());
+		second = second.parent;
+	}
+	return {
+		pops,
+		newFrames: newFrames.toReversed()
+	};
+}
+function applyStateDiff(previous, diff) {
+	let stack = previous;
+	for (let pop = 0; pop < diff.pops; pop += 1) {
+		if (!stack) return null;
+		stack = stack.parent;
+	}
+	for (const frame of diff.newFrames) stack = Stack.pushFrame(stack, frame);
+	return stack;
+}
+//#endregion
+//#region src/tokenization/metadata.ts
+const LANGUAGE_MASK = 255;
+const KEPT_MASK = 32512;
+const FOREGROUND_MASK = 16744448;
+const FOREGROUND_OFFSET = 15;
+const BACKGROUND_OFFSET = 24;
+const DEFAULT_FOREGROUND = 1;
+const DEFAULT_BACKGROUND = 2;
+function hexOf(color) {
+	const hex = color.trim().toLowerCase();
+	return hex.length === 9 && hex.endsWith("ff") ? hex.slice(0, 7) : hex;
+}
+function channel(value) {
+	return Math.round(value).toString(16).padStart(2, "0");
+}
+function hexOfMapColor({ rgba }) {
+	const opaque = `#${channel(rgba.r)}${channel(rgba.g)}${channel(rgba.b)}`;
+	return rgba.a >= 1 ? opaque : `${opaque}${channel(rgba.a * 255)}`;
+}
+function colorIdsFor(highlighterColors, monacoColors) {
+	const monacoIds = /* @__PURE__ */ new Map();
+	const colors = monacoColors ?? [];
+	for (const [id, color] of colors.entries()) if (color && !monacoIds.has(hexOfMapColor(color))) monacoIds.set(hexOfMapColor(color), id);
+	return Uint16Array.from(highlighterColors, (color) => monacoIds.get(hexOf(color ?? "")) ?? DEFAULT_FOREGROUND);
+}
+function monacoMetadata(metadata, languageId, colorIds) {
+	const foreground = colorIds[(metadata & FOREGROUND_MASK) >>> FOREGROUND_OFFSET] ?? DEFAULT_FOREGROUND;
+	return (languageId & LANGUAGE_MASK | metadata & KEPT_MASK | foreground << FOREGROUND_OFFSET | DEFAULT_BACKGROUND << BACKGROUND_OFFSET) >>> 0;
+}
+function defaultMetadata(languageId) {
+	return (languageId & LANGUAGE_MASK | 33587200) >>> 0;
+}
+function toMonacoTokens(tokens, languageId, colorIds) {
+	for (let index = 1; index < tokens.length; index += 2) tokens[index] = monacoMetadata(tokens[index] ?? 0, languageId, colorIds);
+	return tokens;
+}
+function withEndOffsets(tokens, lineLength) {
+	const last = tokens.length - 2;
+	for (let index = 0; index < last; index += 2) tokens[index] = tokens[index + 2] ?? lineLength;
+	if (last >= 0) tokens[last] = lineLength;
+	return tokens;
+}
+//#endregion
+//#region src/tokenization/bounds.ts
+const SYNC_LINE_BOUND = 1e3;
+const WORKER_LINE_BOUND = 2e4;
+//#endregion
+export { toMonacoTokens as a, stateDiff as c, lineAfter as d, lineSplice as f, defaultMetadata as i, applyChange as l, WORKER_LINE_BOUND as n, withEndOffsets as o, colorIdsFor as r, applyStateDiff as s, SYNC_LINE_BOUND as t, applySplice as u };

package/dist/index.d.ts

@@ -33,8 +33,9 @@ interface MonacoHostOptions {
  */
 type MonacoModule = typeof monacoEditor;
 /**
- * Bring up the shared Monaco host: UI locale, then the monaco module, then worker routing —
- * fully offline, no CDN, workers bundled by the consumer's bundler.
+ * Bring up the shared Monaco host: UI locale, then the monaco module and the parts of it the
+ * Shiki tokenization stands on, then worker routing — fully offline, no CDN, workers bundled by
+ * the consumer's bundler.
  *
  * Idempotent and async-safe: every caller shares one promise, and the first call wins the
  * configuration (`globalThis.MonacoEnvironment` is a single global — per-editor configuration
@@ -50,7 +51,7 @@ type MonacoModule = typeof monacoEditor;
  * (both are idempotent), instead of replaying one transient failure — a chunk 404 after a
  * redeploy, a flaky network — for the rest of the session.
  */
-declare function ensureMonacoHost(options?: MonacoHostOptions): Promise<MonacoModule>;
+export declare function ensureMonacoHost(options?: MonacoHostOptions): Promise<MonacoModule>;
 //#endregion
 //#region src/auto-imports.d.ts
 /**
@@ -74,18 +75,19 @@ interface AutoImportCompletionOptions {
  * subclass (a `getWorker` override) is detected on the first answer and the install disarms
  * silently. Install once per app; repeated calls return the first install's disposable.
  */
-declare function installAutoImportCompletions(monaco: MonacoModule, options: AutoImportCompletionOptions): monacoEditor.IDisposable;
+export declare function installAutoImportCompletions(monaco: MonacoModule, options: AutoImportCompletionOptions): monacoEditor.IDisposable;
 //#endregion
 //#region src/design-tokens.d.ts
 /**
  * Define the VS Code design tokens Monaco's chrome reads but the standalone editor lacks —
- * corner radii and widget shadows — on `.monaco-editor`, resolved through the host's
- * `--inkstone-radius-*` / `--inkstone-shadow-*` tokens with inkstone's fallbacks, and keep an
- * input box's focus ring single now that the box has corners (see `INPUTBOX_FOCUS_RULE`).
+ * corner radii and widget shadows — where Monaco writes its theme's colors, resolved
+ * through the host's `--inkstone-radius-*` / `--inkstone-shadow-*` tokens with inkstone's
+ * fallbacks; keep an input box's focus ring single now that the box has corners (see
+ * `INPUTBOX_FOCUS_RULE`), and the completion list on the shadow token (`SUGGEST_SHADOW_RULE`).
  * Installed by `ensureMonacoHost`; idempotent, so a host that brings monaco up on its own can
  * call it too.
  */
-declare function installDesignTokens(): void;
+export declare function installDesignTokens(): void;
 //#endregion
 //#region src/hover-dead-band.d.ts
 /**
@@ -102,7 +104,7 @@ interface HoverDeadBandTargetEditor {
  * on every editor; a monaco whose hover controller no longer matches the pinned shape is
  * left untouched. Dispose to restore the stock test.
  */
-declare function installHoverDeadBandFix(editorInstance: HoverDeadBandTargetEditor): monacoEditor.IDisposable;
+export declare function installHoverDeadBandFix(editorInstance: HoverDeadBandTargetEditor): monacoEditor.IDisposable;
 //#endregion
 //#region src/jsx-tag-closing.d.ts
 /**
@@ -148,7 +150,7 @@ interface JsxTagClosingTargetEditor {
  * detected on its first answer and the install disarms silently. Listener lifetime follows
  * the editor; dispose the returned subscription to detach earlier.
  */
-declare function installJsxTagClosing(monaco: MonacoModule, editorInstance: JsxTagClosingTargetEditor): monacoEditor.IDisposable;
+export declare function installJsxTagClosing(monaco: MonacoModule, editorInstance: JsxTagClosingTargetEditor): monacoEditor.IDisposable;
 //#endregion
 //#region src/keybindings.d.ts
 /**
@@ -163,13 +165,13 @@ interface SwallowedKeybindings {
  * The keybindings `disableFind` / `disableCommandPalette` swallow: the find/replace widget's
  * entries (Ctrl/Cmd+F to find; Ctrl+H and Cmd+Alt+F, the per-platform replace bindings) and
  * the command palette's F1 — Monaco never binds Ctrl+Shift+P (a VS Code binding). Sourced
- * from monaco-editor 0.56.0 (`findController.js`, `standaloneCommandsQuickAccess.js`).
+ * from monaco-editor 0.57.0 (`findController.js`, `standaloneCommandsQuickAccess.js`).
  *
  * Both replace chords are swallowed on every platform rather than branching: `CtrlCmd` already
  * resolves per platform, and the off-platform chord (Ctrl+Alt+F on Windows, Cmd+H on macOS, where
  * the OS takes the key before the page sees it) is one Monaco never bound.
  */
-declare function disabledKeybindings(monaco: MonacoModule, disableFind: boolean, disableCommandPalette: boolean): SwallowedKeybindings[];
+export declare function disabledKeybindings(monaco: MonacoModule, disableFind: boolean, disableCommandPalette: boolean): SwallowedKeybindings[];
 /**
  * The slice of a standalone editor the install touches — a structural type, so the install is
  * testable without evaluating the monaco module (which jsdom cannot do).
@@ -184,7 +186,7 @@ interface KeybindingTargetEditor {
  * the page-global service only the first time any editor of that monaco module asks for it —
  * monaco offers no way to unregister one.
  */
-declare function installDisabledKeybindings(monaco: MonacoModule, editorInstance: KeybindingTargetEditor, groups: readonly SwallowedKeybindings[]): void;
+export declare function installDisabledKeybindings(monaco: MonacoModule, editorInstance: KeybindingTargetEditor, groups: readonly SwallowedKeybindings[]): void;
 //#endregion
 //#region src/languages.d.ts
 /**
@@ -208,7 +210,7 @@ type MonacoBuiltInLanguage = (typeof BUILT_IN_LANGUAGES)[number];
 /**
  * Every built-in Monaco language id, sorted.
  */
-declare const monacoLanguages: readonly MonacoBuiltInLanguage[];
+export declare const monacoLanguages: readonly MonacoBuiltInLanguage[];
 /**
  * The `<MonacoEditor language>` type. Monaco also accepts any custom-registered id (e.g. a
  * language wired through `getWorker` such as monaco-yaml), so the type stays open: the union
@@ -223,7 +225,7 @@ type MonacoLanguage = MonacoBuiltInLanguage | (string & Record<never, never>);
  * tuned for API-assisted editing, thin scrollbars, and an embedded-friendly find widget.
  * Spread these to extend rather than replace: `options={{ ...DEFAULT_MONACO_OPTIONS, wordWrap: "on" }}`.
  */
-declare const DEFAULT_MONACO_OPTIONS: editor.IStandaloneEditorConstructionOptions;
+export declare const DEFAULT_MONACO_OPTIONS: editor.IStandaloneEditorConstructionOptions;
 //#endregion
 //#region src/reconcile.d.ts
 /**
@@ -231,11 +233,11 @@ declare const DEFAULT_MONACO_OPTIONS: editor.IStandaloneEditorConstructionOption
  * documents already match. Fires the model's content-change event synchronously when it
  * edits — callers that must not re-enter their change callback silence it around this call.
  */
-declare function reconcileModelValue(model: editor.ITextModel, externalValue: string): void;
+export declare function reconcileModelValue(model: editor.ITextModel, externalValue: string): void;
 /**
  * Whether an external value already matches the model after applying its EOL convention.
  */
-declare function modelValueMatches(model: editor.ITextModel, externalValue: string): boolean;
+export declare function modelValueMatches(model: editor.ITextModel, externalValue: string): boolean;
 //#endregion
 //#region src/replace.d.ts
 /**
@@ -245,36 +247,46 @@ declare function modelValueMatches(model: editor.ITextModel, externalValue: stri
  * because it is read-only — e.g. locked by a stream in progress; the caller falls back to a
  * state write).
  */
-declare function replaceInEditor(target: editor.ICodeEditor | null, search: string, replaceWith: string, all: boolean): boolean;
+export declare function replaceInEditor(target: editor.ICodeEditor | null, search: string, replaceWith: string, all: boolean): boolean;
 //#endregion
 //#region src/shiki-bridge.d.ts
 /**
- * Register the shared Shiki highlighter's languages, themes, and tokenizers on a monaco module —
- * once per distinct module, whichever caller gets there first — and, on a module already
- * bridged, install the tokenizer of any grammar loaded since. Language services are unaffected:
+ * Register the shared Shiki highlighter's languages, themes, and tokenization on a monaco module
+ * — once per distinct module, whichever caller gets there first — and, on a module already
+ * bridged, the tokenization of any grammar loaded since. Language services are unaffected:
  * Shiki replaces syntax highlighting only. The module is recorded only after every registration
- * succeeds, so a transient failure can retry on a later call.
+ * succeeds, so a transient failure can retry on a later call. The module must be the host's
+ * (`ensureMonacoHost`), which brings up the parts of Monaco the tokenization stands on.
+ *
+ * A model is tokenized whole in a worker of its own, built from what the highlighter was built
+ * from (`@coldsmirk/inkstone-core`'s `highlighterSources`) — VS Code's design, through the same
+ * door in Monaco: the page's thread tokenizes only the lines Monaco asks of it before the worker
+ * reaches them — a model's first lines in view, the lines an edit touches — and of those only a
+ * line it can afford; a longer one waits for the worker's colours (`tokenization/bounds.ts`).
+ * A highlighter inkstone did not build, or a host that routes no worker for the `textmate`
+ * label, leaves every model to Monaco's own pass on the page's thread, under the same bound.
  *
  * Monaco's own `setTheme` is patched to route the Shiki theme ids through the highlighter, so
- * the tokenizers paint in the theme the editor shows; Monaco's built-in themes (vs/vs-dark) and
- * caller-defined themes keep Monaco's original path. `create` is patched so its `theme` option
- * reaches that patch. The shape is `@shikijs/monaco`'s, kept here because that package
+ * the tokenization paints in the theme the editor shows; Monaco's built-in themes (vs/vs-dark)
+ * and caller-defined themes keep Monaco's original path. `create` is patched so its `theme`
+ * option reaches that patch. The shape is `@shikijs/monaco`'s, kept here because that package
  * installs tokenizers only for the grammars loaded at the moment it is called, and re-calling
  * it for a grammar loaded later resets the editor's theme to the first one.
  *
  * A grammar loaded into the highlighter *after* this call is not tokenized until it is
  * registered — pass it to {@link registerShikiLanguage} (or call this again).
  */
-declare function registerShikiHighlighting(monaco: MonacoModule, highlighter: HighlighterCore): void;
+export declare function registerShikiHighlighting(monaco: MonacoModule, highlighter: HighlighterCore): void;
 /**
- * Install the tokenizer of one grammar the highlighter has loaded since the module was bridged
- * — the lazy path: an app that loads grammars as files are opened registers each here, after
- * its `loadLanguage`, and before a model names it. `language` may be an alias (`ts`); it
+ * Register the tokenization of one grammar the highlighter has loaded since the module was
+ * bridged — the lazy path: an app that loads grammars as files are opened registers each here,
+ * after its `loadLanguage`, and before a model names it. `language` may be an alias (`ts`); it
  * collapses to the grammar's canonical name. Bridges the module first where nothing has, and
- * is nothing for a grammar already registered. Throws when the highlighter has not loaded the
- * grammar: there is nothing to tokenize with.
+ * is nothing for a grammar already registered — but a grammar shiki has built again since, as
+ * a language it embeds lazily arrived, has its models tokenized again under the new one. Throws
+ * when the highlighter has not loaded the grammar: there is nothing to tokenize with.
  */
-declare function registerShikiLanguage(monaco: MonacoModule, highlighter: HighlighterCore, language: string): void;
+export declare function registerShikiLanguage(monaco: MonacoModule, highlighter: HighlighterCore, language: string): void;
 //#endregion
 //#region src/streaming.d.ts
 /**
@@ -297,7 +309,7 @@ type StreamTargetEditor = editor.ICodeEditor;
  * accumulate in a buffer that is replayed on {@link attach}, so the final document is
  * complete even if the editor never mounts.
  */
-declare class StreamingEditorController {
+export declare class StreamingEditorController {
   private editor;
   private writing;
   private buffer;
@@ -398,4 +410,4 @@ declare class StreamingEditorController {
   discard(): void;
 }
 //#endregion
-export { type AutoImportCompletionOptions, DEFAULT_MONACO_OPTIONS, type HoverDeadBandTargetEditor, type JsxTagClosingTargetEditor, type JsxTagClosingTargetModel, type KeybindingTargetEditor, type MonacoBuiltInLanguage, type MonacoHostOptions, type MonacoLanguage, type MonacoModule, type MonacoUiLocale, type StreamTargetEditor, StreamingEditorController, type SwallowedKeybindings, disabledKeybindings, ensureMonacoHost, installAutoImportCompletions, installDesignTokens, installDisabledKeybindings, installHoverDeadBandFix, installJsxTagClosing, modelValueMatches, monacoLanguages, reconcileModelValue, registerShikiHighlighting, registerShikiLanguage, replaceInEditor };
+export type { AutoImportCompletionOptions, HoverDeadBandTargetEditor, JsxTagClosingTargetEditor, JsxTagClosingTargetModel, KeybindingTargetEditor, MonacoBuiltInLanguage, MonacoHostOptions, MonacoLanguage, MonacoModule, MonacoUiLocale, StreamTargetEditor, SwallowedKeybindings };

package/dist/index.js

@@ -1,4 +1,5 @@
-import { CODE_FONT_FAMILY, inkstoneRadius, inkstoneShadow } from "@coldsmirk/inkstone-core";
+import { a as toMonacoTokens, d as lineAfter, i as defaultMetadata, r as colorIdsFor, s as applyStateDiff } from "./bounds-CZ_iAUoc.js";
+import { CODE_FONT_FAMILY, highlighterSources, inkstoneRadius, inkstoneShadow } from "@coldsmirk/inkstone-core";
 import { EncodedTokenMetadata, INITIAL } from "@shikijs/vscode-textmate";
 //#region src/auto-imports.ts
 const MISSING_LOOKUP_REJECTION$1 = "Missing requestHandler or method: getAutoImportCompletionsAtPosition";
@@ -130,16 +131,18 @@ const DESIGN_TOKENS = [
 	`--vscode-cornerRadius-medium: ${inkstoneRadius("medium", "4px")}`,
 	`--vscode-cornerRadius-large: ${inkstoneRadius("large", "6px")}`,
 	`--vscode-cornerRadius-xLarge: ${inkstoneRadius("xlarge", "8px")}`,
+	"--vscode-cornerRadius-circle: 9999px",
 	`--vscode-shadow-md: ${inkstoneShadow("medium", "0 1px 4px var(--vscode-widget-shadow)")}`,
 	`--vscode-shadow-lg: ${inkstoneShadow("large", "0 2px 8px var(--vscode-widget-shadow)")}`,
 	`--vscode-shadow-xl: ${inkstoneShadow("xlarge", "0 4px 16px var(--vscode-widget-shadow)")}`
 ];
 const INPUTBOX_FOCUS_RULE = ".monaco-editor .monaco-inputbox > .ibwrapper > .input:focus { outline: none; }";
+const SUGGEST_SHADOW_RULE = ".monaco-editor:not(.hc-black):not(.hc-light) .suggest-widget { box-shadow: var(--vscode-shadow-lg); }";
 function installDesignTokens() {
 	if (document.querySelector(DESIGN_TOKENS_STYLE_SELECTOR)) return;
 	const style = document.createElement("style");
 	style.dataset.inkstone = "monaco-design-tokens";
-	style.textContent = `.monaco-editor { ${DESIGN_TOKENS.join("; ")}; }\n${INPUTBOX_FOCUS_RULE}`;
+	style.textContent = `.monaco-editor, .monaco-diff-editor, .monaco-component { ${DESIGN_TOKENS.join("; ")}; }\n${INPUTBOX_FOCUS_RULE}\n${SUGGEST_SHADOW_RULE}`;
 	document.head.append(style);
 }
 //#endregion
@@ -180,6 +183,14 @@ function localizeActionDescriptions(editorInstance) {
 	}
 }
 //#endregion
+//#region src/context-menu-fix.ts
+const COMPONENT_CLASS = "monaco-component";
+function installContextMenuThemeFix(monaco) {
+	monaco.editor.onDidCreateEditor((editorInstance) => {
+		editorInstance.getContainerDomNode().classList.add(COMPONENT_CLASS);
+	});
+}
+//#endregion
 //#region src/geometry-fixes.ts
 const HOVER_GEOMETRY_STYLE_SELECTOR = "style[data-inkstone=\"monaco-hover-geometry\"]";
 function installHoverGeometryFix() {
@@ -197,14 +208,6 @@ function installSuggestGeometryFix() {
 	style.textContent = ".monaco-editor .suggest-widget .monaco-list .monaco-list-row > .contents > .main > .right > span.readMore.codicon.codicon-suggest-more-info::before { position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); }";
 	document.head.append(style);
 }
-const QUICK_INPUT_GEOMETRY_STYLE_SELECTOR = "style[data-inkstone=\"monaco-quick-input-geometry\"]";
-function installQuickInputGeometryFix() {
-	if (document.querySelector(QUICK_INPUT_GEOMETRY_STYLE_SELECTOR)) return;
-	const style = document.createElement("style");
-	style.dataset.inkstone = "monaco-quick-input-geometry";
-	style.textContent = ".monaco-editor .quick-input-list { position: relative; }";
-	document.head.append(style);
-}
 const HOVER_FOCUS_STYLE_SELECTOR = "style[data-inkstone=\"monaco-hover-focus\"]";
 function installHoverFocusFix() {
 	if (document.querySelector(HOVER_FOCUS_STYLE_SELECTOR)) return;
@@ -214,6 +217,33 @@ function installHoverFocusFix() {
 	document.head.append(style);
 }
 //#endregion
+//#region src/tokenization/internals.ts
+async function loadMonacoInternals() {
+	const [languages, tokens] = await Promise.all([import("monaco-editor/editor/common/languages.js"), import("monaco-editor/editor/common/tokens/contiguousMultilineTokensBuilder.js")]);
+	return {
+		registry: languages.TokenizationRegistry,
+		encoded: (encoded, endState) => new languages.EncodedTokenizationResult(encoded, [], endState),
+		classic: (classic, language, endState) => new languages.TokenizationResult(classic.map((token) => new languages.Token(token.offset, token.type, language)), endState),
+		multiline: (lines) => {
+			const builder = new tokens.ContiguousMultilineTokensBuilder();
+			for (const [lineNumber, lineTokens] of lines) builder.add(lineNumber, lineTokens);
+			return builder.finalize();
+		}
+	};
+}
+const INTERNALS_KEY = Symbol.for("coldsmirk.inkstone.monaco.internals");
+function ledger() {
+	const host = globalThis;
+	host[INTERNALS_KEY] ??= /* @__PURE__ */ new WeakMap();
+	return host[INTERNALS_KEY];
+}
+function rememberInternals(monaco, internals) {
+	ledger().set(monaco, internals);
+}
+function internalsOf(monaco) {
+	return ledger().get(monaco);
+}
+//#endregion
 //#region src/host.ts
 const HOST_PROMISE_KEY = Symbol.for("coldsmirk.inkstone.monaco.hostPromise");
 const NLS_REGISTRY_KEY = Symbol.for("coldsmirk.inkstone.monaco.nlsRegistry");
@@ -254,11 +284,13 @@ async function applyZhCnCatalog() {
 function editorWorker() {
 	return new Worker(new URL("workers/editor.worker.js", import.meta.url), { type: "module" });
 }
+const TEXTMATE_WORKER_LABEL = "textmate";
 const WORKER_BY_LABEL = {
 	typescript: () => new Worker(new URL("workers/ts.worker.js", import.meta.url), { type: "module" }),
 	json: () => new Worker(new URL("workers/json.worker.js", import.meta.url), { type: "module" }),
 	css: () => new Worker(new URL("workers/css.worker.js", import.meta.url), { type: "module" }),
-	html: () => new Worker(new URL("workers/html.worker.js", import.meta.url), { type: "module" })
+	html: () => new Worker(new URL("workers/html.worker.js", import.meta.url), { type: "module" }),
+	[TEXTMATE_WORKER_LABEL]: () => new Worker(new URL("workers/textmate.worker.js", import.meta.url), { type: "module" })
 };
 const WORKER_ALIASES = {
 	javascript: "typescript",
@@ -283,10 +315,11 @@ function ensureMonacoHost(options = {}) {
 			installDesignTokens();
 			installHoverGeometryFix();
 			installSuggestGeometryFix();
-			installQuickInputGeometryFix();
 			installHoverFocusFix();
 			const monaco = await import("monaco-editor");
+			rememberInternals(monaco, await loadMonacoInternals());
 			globalThis.MonacoEnvironment = { getWorker: (workerId, label) => getWorker?.(workerId, label) ?? builtinGetWorker(label) };
+			installContextMenuThemeFix(monaco);
 			if (locale === "zh-cn") monaco.editor.onDidCreateEditor(localizeActionDescriptions);
 			return monaco;
 		} catch (error) {
@@ -600,6 +633,280 @@ function replaceInEditor(target, search, replaceWith, all) {
 	return target.executeEdits("inkstone-replace", edits);
 }
 //#endregion
+//#region src/tokenization/controller.ts
+var WorkerTokenizerController = class {
+	model;
+	store;
+	internals;
+	channel;
+	attachment;
+	session;
+	disposed = false;
+	language;
+	constructor(options) {
+		this.language = options.language;
+		this.model = options.model;
+		this.store = options.store;
+		this.internals = options.internals;
+		this.channel = options.channel;
+		this.attachment = this.model.onDidChangeAttached(() => {
+			this.follow();
+		});
+		queueMicrotask(() => {
+			this.follow();
+		});
+	}
+	follow() {
+		if (this.disposed) return;
+		const attached = this.model.isAttachedToEditor();
+		if (attached && !this.session) this.connect();
+		else if (!attached && this.session) this.disconnect();
+	}
+	connect() {
+		const id = this.channel.open(this, this.model.getLinesContent(), this.model.getVersionId());
+		const pending = [];
+		const listener = this.model.onDidChangeContent((event) => {
+			const changes = event.changes.map(({ range, text }) => {
+				return {
+					range: {
+						startLineNumber: range.startLineNumber,
+						startColumn: range.startColumn,
+						endLineNumber: range.endLineNumber,
+						endColumn: range.endColumn
+					},
+					text
+				};
+			});
+			pending.push({
+				version: event.versionId,
+				changes
+			});
+			this.channel.send({
+				kind: "change",
+				model: id,
+				version: event.versionId,
+				changes
+			});
+		});
+		this.session = {
+			id,
+			pending,
+			listener
+		};
+	}
+	disconnect() {
+		if (!this.session) return;
+		this.session.listener.dispose();
+		this.channel.close(this.session.id);
+		this.session = void 0;
+	}
+	keepStates(message) {
+		let previous = null;
+		for (const [index, line] of message.lines.entries()) {
+			const diff = message.states[index];
+			const follows = index > 0 && message.lines[index - 1] === line - 1;
+			let state = null;
+			if (diff && (!follows || previous)) try {
+				state = applyStateDiff(follows ? previous : null, diff);
+			} catch {
+				state = null;
+			}
+			if (state) this.store.setEndState(line, state);
+			previous = state;
+		}
+	}
+	receiveTokens(message) {
+		const { session } = this;
+		if (session?.id !== message.model) return;
+		const { pending } = session;
+		while (pending[0] && pending[0].version <= message.version) pending.shift();
+		const lines = [];
+		let offset = 0;
+		for (const [index, sent] of message.lines.entries()) {
+			const length = message.lengths[index] ?? 0;
+			const tokens = message.tokens.subarray(offset, offset + length);
+			const line = lineAfter(sent, pending);
+			offset += length;
+			if (line !== void 0) lines.push([line, tokens]);
+		}
+		if (lines.length > 0) this.store.setTokens(this.internals.multiline(lines));
+		if (pending.length === 0) this.keepStates(message);
+	}
+	finished(model, version) {
+		const { session } = this;
+		if (session?.id === model && session.pending.length === 0 && version === this.model.getVersionId()) this.store.backgroundTokenizationFinished();
+	}
+	requestTokens(startLineNumber, endLineNumberExclusive) {
+		if (this.session) this.channel.send({
+			kind: "retokenize",
+			model: this.session.id,
+			start: startLineNumber,
+			end: endLineNumberExclusive
+		});
+	}
+	dispose() {
+		if (this.disposed) return;
+		this.disposed = true;
+		this.attachment.dispose();
+		this.disconnect();
+	}
+};
+//#endregion
+//#region src/tokenization/client.ts
+var TokenizationClient = class {
+	options;
+	controllers = /* @__PURE__ */ new Map();
+	worker;
+	sentLanguages = 0;
+	nextModel = 1;
+	dead = false;
+	constructor(options) {
+		this.options = options;
+	}
+	start() {
+		if (this.dead) return false;
+		if (this.worker) return true;
+		let created;
+		try {
+			created = this.options.createWorker();
+		} catch {
+			this.dead = true;
+			return false;
+		}
+		this.worker = Promise.resolve(created);
+		this.worker.then((worker) => {
+			worker.addEventListener("message", (event) => {
+				this.receive(event.data);
+			});
+			worker.addEventListener("error", () => {
+				this.fail();
+			});
+		}, () => {
+			this.fail();
+		});
+		const { sources } = this.options;
+		this.sentLanguages = sources.languages.length;
+		this.post({
+			kind: "build",
+			engine: sources.engine,
+			themes: this.options.themes(),
+			languages: [...sources.languages]
+		});
+		return true;
+	}
+	sendLanguages() {
+		const { languages } = this.options.sources;
+		if (languages.length > this.sentLanguages) {
+			this.post({
+				kind: "languages",
+				languages: languages.slice(this.sentLanguages)
+			});
+			this.sentLanguages = languages.length;
+		}
+	}
+	post(message) {
+		this.worker?.then((worker) => {
+			worker.postMessage(message);
+		});
+	}
+	receive(message) {
+		switch (message.kind) {
+			case "tokens":
+				this.controllers.get(message.model)?.receiveTokens(message);
+				break;
+			case "finished":
+				this.controllers.get(message.model)?.finished(message.model, message.version);
+				break;
+			case "unavailable": {
+				const controller = this.controllers.get(message.model);
+				if (controller) this.options.unavailable([controller.language]);
+				break;
+			}
+			case "failed": this.fail();
+		}
+	}
+	fail() {
+		if (this.dead) return;
+		this.dead = true;
+		this.worker?.then((worker) => {
+			worker.terminate();
+		});
+		this.worker = void 0;
+		this.options.unavailable("all");
+	}
+	open(model, store, tokenization) {
+		if (!this.start()) return;
+		return new WorkerTokenizerController({
+			language: tokenization.language,
+			model,
+			store,
+			internals: this.options.internals,
+			channel: {
+				open: (controller, lines, version) => {
+					const id = this.nextModel;
+					this.nextModel += 1;
+					this.sendLanguages();
+					this.controllers.set(id, controller);
+					this.post({
+						kind: "open",
+						model: id,
+						...tokenization,
+						version,
+						lines
+					});
+					return id;
+				},
+				send: (message) => {
+					this.post(message);
+				},
+				close: (id) => {
+					this.controllers.delete(id);
+					this.post({
+						kind: "close",
+						model: id
+					});
+				}
+			}
+		});
+	}
+};
+//#endregion
+//#region src/tokenization/support.ts
+function shikiTokenization(options) {
+	const { internals, highlighter, language, languageId, theme, colorIds, scopeOf, background } = options;
+	const tokenizeOnPage = (line, state) => {
+		if (line.length >= 1e3) return;
+		try {
+			highlighter.setTheme(theme());
+			const result = highlighter.getLanguage(language).tokenizeLine2(line, state, 50);
+			return result.stoppedEarly ? void 0 : result;
+		} catch {
+			return;
+		}
+	};
+	return {
+		getInitialState: () => INITIAL,
+		tokenizeEncoded: (line, _hasEOL, state) => {
+			const result = tokenizeOnPage(line, state);
+			return result ? internals.encoded(toMonacoTokens(result.tokens, languageId, colorIds()), result.ruleStack) : internals.encoded(Uint32Array.of(0, defaultMetadata(languageId)), state);
+		},
+		tokenize: (line, _hasEOL, state) => {
+			const result = tokenizeOnPage(line, state);
+			if (!result) return internals.classic([{
+				offset: 0,
+				type: ""
+			}], language, state);
+			const tokens = [];
+			for (let index = 0; index < result.tokens.length; index += 2) tokens.push({
+				offset: result.tokens[index] ?? 0,
+				type: scopeOf(result.tokens[index + 1] ?? 0)
+			});
+			return internals.classic(tokens, language, result.ruleStack);
+		},
+		createBackgroundTokenizer: background
+	};
+}
+//#endregion
 //#region src/shiki-bridge.ts
 const SHIKI_REGISTRY_KEY = Symbol.for("coldsmirk.inkstone.monaco.shikiRegistry");
 function bridges() {
@@ -610,8 +917,6 @@ function bridges() {
 	host[SHIKI_REGISTRY_KEY] = created;
 	return created;
 }
-const TOKENIZE_MAX_LINE_LENGTH = 2e4;
-const TOKENIZE_TIME_LIMIT_MS = 500;
 const ITALIC = 1;
 const BOLD = 2;
 const UNDERLINE = 4;
@@ -675,6 +980,7 @@ function monacoThemeOf(theme) {
 function adoptTheme(bridge, themeName) {
 	const { colorMap } = bridge.highlighter.setTheme(themeName);
 	const theme = bridge.themes.get(themeName);
+	bridge.theme = themeName;
 	bridge.colorMap.length = colorMap.length;
 	for (const [index, color] of colorMap.entries()) bridge.colorMap[index] = color;
 	bridge.scopeOf.clear();
@@ -686,58 +992,78 @@ function adoptTheme(bridge, themeName) {
 		if (!bridge.scopeOf.has(key)) bridge.scopeOf.set(key, rule.token);
 	}
 }
-var TokenizerState = class TokenizerState {
-	ruleStack;
-	constructor(ruleStack) {
-		this.ruleStack = ruleStack;
-	}
-	clone() {
-		return new TokenizerState(this.ruleStack);
-	}
-	equals(other) {
-		return other instanceof TokenizerState && other.ruleStack === this.ruleStack;
-	}
-};
-function tokensProvider(bridge, lang) {
-	return {
-		getInitialState() {
-			return new TokenizerState(INITIAL);
-		},
-		tokenize(line, state) {
-			if (line.length >= TOKENIZE_MAX_LINE_LENGTH) return {
-				endState: state,
-				tokens: [{
-					startIndex: 0,
-					scopes: ""
-				}]
-			};
-			const ruleStack = state instanceof TokenizerState ? state.ruleStack : INITIAL;
-			const result = bridge.highlighter.getLanguage(lang).tokenizeLine2(line, ruleStack, TOKENIZE_TIME_LIMIT_MS);
-			if (result.stoppedEarly) console.warn(`inkstone: time limit reached when tokenizing a line: ${line.slice(0, 100)}`);
-			const tokens = [];
-			for (let index = 0; index < result.tokens.length; index += 2) {
-				const startIndex = result.tokens[index] ?? 0;
-				const metadata = result.tokens[index + 1] ?? 0;
-				const color = normalizeColor(bridge.colorMap[EncodedTokenMetadata.getForeground(metadata)]);
-				const fontStyle = normalizeFontStyleBits(EncodedTokenMetadata.getFontStyle(metadata));
-				const scopes = color ? bridge.scopeOf.get(colorStyleKey(color, fontStyle)) ?? "" : "";
-				tokens.push({
-					startIndex,
-					scopes
-				});
-			}
-			return {
-				endState: new TokenizerState(result.ruleStack),
-				tokens
-			};
-		}
+function scopeOfMetadata(bridge, metadata) {
+	const color = normalizeColor(bridge.colorMap[EncodedTokenMetadata.getForeground(metadata)]);
+	const fontStyle = normalizeFontStyleBits(EncodedTokenMetadata.getFontStyle(metadata));
+	return color ? bridge.scopeOf.get(colorStyleKey(color, fontStyle)) ?? "" : "";
+}
+function colorIds(bridge) {
+	const monacoColors = bridge.internals.registry.getColorMap();
+	if (bridge.colorIds?.theme !== bridge.theme || bridge.colorIds.monacoColors !== monacoColors) bridge.colorIds = {
+		theme: bridge.theme,
+		monacoColors,
+		ids: colorIdsFor(bridge.colorMap, monacoColors)
 	};
+	return bridge.colorIds.ids;
+}
+function clientOf(bridge) {
+	if (bridge.client !== void 0) return bridge.client;
+	const sources = highlighterSources(bridge.highlighter);
+	const getWorker = globalThis.MonacoEnvironment?.getWorker;
+	bridge.client = sources && getWorker ? new TokenizationClient({
+		internals: bridge.internals,
+		sources,
+		themes: () => bridge.highlighter.getLoadedThemes().map((id) => bridge.highlighter.getTheme(id)),
+		createWorker: () => getWorker("inkstone-textmate", TEXTMATE_WORKER_LABEL),
+		unavailable: (languages) => {
+			if (languages === "all") bridge.client = null;
+			else for (const language of languages) bridge.workerless.add(language);
+			bridge.internals.registry.handleChange(languages === "all" ? bridge.registered.keys().toArray() : [...languages]);
+		}
+	}) : null;
+	return bridge.client;
+}
+function quietBuiltInTokenizer(monaco, language) {
+	if (language !== "json") return;
+	const { jsonDefaults } = monaco.json;
+	if (jsonDefaults.modeConfiguration.tokens !== false) jsonDefaults.setModeConfiguration({
+		...jsonDefaults.modeConfiguration,
+		tokens: false
+	});
 }
 function install(monaco, bridge, canonical) {
 	if (bridge.registered.has(canonical)) return;
 	if (monaco.languages.getLanguages().every((language) => language.id !== canonical)) monaco.languages.register({ id: canonical });
-	monaco.languages.setTokensProvider(canonical, tokensProvider(bridge, canonical));
-	bridge.registered.add(canonical);
+	quietBuiltInTokenizer(monaco, canonical);
+	bridge.registered.set(canonical, bridge.highlighter.getLanguage(canonical));
+	bridge.internals.registry.register(canonical, shikiTokenization({
+		internals: bridge.internals,
+		highlighter: bridge.highlighter,
+		language: canonical,
+		languageId: monaco.languages.getEncodedLanguageId(canonical),
+		theme: () => bridge.theme,
+		colorIds: () => colorIds(bridge),
+		scopeOf: (metadata) => scopeOfMetadata(bridge, metadata),
+		background: (model, store) => {
+			return (bridge.workerless.has(canonical) ? null : clientOf(bridge))?.open(model, store, {
+				language: canonical,
+				languageId: monaco.languages.getEncodedLanguageId(canonical),
+				theme: bridge.theme,
+				colorIds: colorIds(bridge)
+			});
+		}
+	}));
+}
+function retokenizeRebuilt(bridge) {
+	const rebuilt = [];
+	for (const [language, grammar] of bridge.registered) {
+		const current = bridge.highlighter.getLanguage(language);
+		if (current !== grammar) {
+			bridge.registered.set(language, current);
+			rebuilt.push(language);
+		}
+	}
+	if (rebuilt.length > 0) bridge.internals.registry.handleChange(rebuilt);
 }
 function canonicalLanguages(highlighter) {
 	return new Set(highlighter.getLoadedLanguages().map((id) => highlighter.getLanguage(id).name));
@@ -747,14 +1073,22 @@ function registerShikiHighlighting(monaco, highlighter) {
 	const bridged = registry.get(monaco);
 	if (bridged) {
 		for (const canonical of canonicalLanguages(highlighter)) install(monaco, bridged, canonical);
+		retokenizeRebuilt(bridged);
 		return;
 	}
+	const internals = internalsOf(monaco);
+	if (!internals) throw new Error("inkstone: Shiki highlighting needs the monaco module ensureMonacoHost brought up");
 	const bridge = {
+		internals,
 		highlighter,
 		themes: /* @__PURE__ */ new Map(),
+		theme: "",
 		colorMap: [],
+		colorIds: void 0,
 		scopeOf: /* @__PURE__ */ new Map(),
-		registered: /* @__PURE__ */ new Set()
+		registered: /* @__PURE__ */ new Map(),
+		client: void 0,
+		workerless: /* @__PURE__ */ new Set()
 	};
 	const originalCreate = monaco.editor.create;
 	const originalSetTheme = monaco.editor.setTheme;
@@ -791,6 +1125,7 @@ function registerShikiLanguage(monaco, highlighter, language) {
 	}
 	if (!bridge) throw new Error("inkstone: the Shiki bridge is not installed on this monaco module");
 	install(monaco, bridge, highlighter.getLanguage(language).name);
+	retokenizeRebuilt(bridge);
 }
 //#endregion
 //#region src/streaming.ts

package/dist/vite.d.ts

@@ -17,6 +17,5 @@ import { Plugin } from "vite";
  * time and leaves the file untouched — the duplicate chunk returns (dead weight, nothing
  * breaks) until the pattern catches up.
  */
-declare function dropMonacoWorkerFallbacks(): Plugin;
-//#endregion
-export { dropMonacoWorkerFallbacks };
+export declare function dropMonacoWorkerFallbacks(): Plugin;
+//#endregion

package/dist/workers/textmate.worker.js

@@ -0,0 +1,296 @@
+import { a as toMonacoTokens, c as stateDiff, f as lineSplice, i as defaultMetadata, l as applyChange, o as withEndOffsets, u as applySplice } from "../bounds-CZ_iAUoc.js";
+import { INITIAL } from "@shikijs/vscode-textmate";
+import { createHighlighterCore } from "shiki/core";
+//#region src/tokenization/invalid-lines.ts
+var InvalidLines = class {
+	runs = [];
+	constructor(lineCount = 0) {
+		this.add(0, lineCount);
+	}
+	first() {
+		return this.runs[0]?.start;
+	}
+	has(line) {
+		return this.runs.some((run) => run.start <= line && line < run.end);
+	}
+	add(start, end) {
+		if (end <= start) return;
+		let from = start;
+		let to = end;
+		const kept = this.runs.filter((run) => {
+			if (run.end < from || to < run.start) return true;
+			from = Math.min(from, run.start);
+			to = Math.max(to, run.end);
+			return false;
+		});
+		kept.push({
+			start: from,
+			end: to
+		});
+		kept.sort((one, other) => one.start - other.start);
+		this.runs.splice(0, this.runs.length, ...kept);
+	}
+	delete(line) {
+		const at = this.runs.findIndex((run) => run.start <= line && line < run.end);
+		const run = this.runs[at];
+		if (!run) return;
+		const pieces = [{
+			start: run.start,
+			end: line
+		}, {
+			start: line + 1,
+			end: run.end
+		}].filter((piece) => piece.start < piece.end);
+		this.runs.splice(at, 1, ...pieces);
+	}
+	splice({ start, removed, inserted }) {
+		const shift = inserted - removed;
+		const moved = this.runs.flatMap((run) => {
+			return [{
+				start: run.start,
+				end: Math.min(run.end, start)
+			}, {
+				start: Math.max(run.start, start + removed) + shift,
+				end: run.end + shift
+			}].filter((piece) => piece.start < piece.end);
+		});
+		this.runs.splice(0, this.runs.length, ...moved);
+		this.add(start, start + inserted);
+	}
+};
+//#endregion
+//#region src/tokenization/worker-runtime.ts
+function hasWork(model) {
+	return model.requested.length > 0 || model.invalid.first() !== void 0;
+}
+async function engineOf(kind) {
+	if (kind === "oniguruma") {
+		const { createOnigurumaEngine } = await import("shiki/engine/oniguruma");
+		return createOnigurumaEngine(import("shiki/wasm"));
+	}
+	const { createJavaScriptRegexEngine } = await import("shiki/engine/javascript");
+	return createJavaScriptRegexEngine();
+}
+function tokenizeLine(grammar, model, line, state) {
+	if (line.length >= 2e4) return {
+		tokens: Uint32Array.of(line.length, defaultMetadata(model.languageId)),
+		endState: state
+	};
+	const result = grammar.tokenizeLine2(line, state, 500);
+	return {
+		tokens: withEndOffsets(toMonacoTokens(result.tokens, model.languageId, model.colorIds), line.length),
+		endState: result.stoppedEarly ? state : result.ruleStack
+	};
+}
+var Batch = class {
+	lines = [];
+	lengths = [];
+	parts = [];
+	states = [];
+	last;
+	get empty() {
+		return this.lines.length === 0;
+	}
+	add(line, tokens, state) {
+		const previous = this.last?.line === line - 1 ? this.last.state : null;
+		this.lines.push(line + 1);
+		this.lengths.push(tokens.length);
+		this.parts.push(tokens);
+		this.states.push(stateDiff(previous, state));
+		this.last = {
+			line,
+			state
+		};
+	}
+	post(post, model, version) {
+		const tokens = new Uint32Array(this.lengths.reduce((sum, length) => sum + length, 0));
+		let offset = 0;
+		for (const part of this.parts) {
+			tokens.set(part, offset);
+			offset += part.length;
+		}
+		const lines = Uint32Array.from(this.lines);
+		const lengths = Uint32Array.from(this.lengths);
+		post({
+			kind: "tokens",
+			model,
+			version,
+			lines,
+			lengths,
+			tokens,
+			states: this.states
+		}, [
+			tokens.buffer,
+			lines.buffer,
+			lengths.buffer
+		]);
+	}
+};
+var TokenizationWorker = class {
+	send;
+	schedule;
+	now;
+	highlighter;
+	models = /* @__PURE__ */ new Map();
+	scheduled = false;
+	failed = false;
+	constructor(send, schedule = (work) => {
+		setTimeout(() => {
+			work();
+		}, 0);
+	}, now = () => performance.now()) {
+		this.send = send;
+		this.schedule = schedule;
+		this.now = now;
+	}
+	wake() {
+		if (this.scheduled || this.failed || !this.highlighter || this.models.values().every((model) => !hasWork(model))) return;
+		this.scheduled = true;
+		this.schedule(() => this.work());
+	}
+	async work() {
+		try {
+			const highlighter = await this.highlighter;
+			if (highlighter) {
+				const deadline = this.now() + 8;
+				const turns = this.models.values().toArray();
+				for (const model of turns) {
+					if (this.now() >= deadline) break;
+					if (hasWork(model)) {
+						this.tokenizeSlice(highlighter, model, deadline);
+						if (this.models.delete(model.id)) this.models.set(model.id, model);
+					}
+				}
+			}
+		} catch (error) {
+			this.failed = true;
+			this.send({
+				kind: "failed",
+				reason: error instanceof Error ? error.message : String(error)
+			}, []);
+		} finally {
+			this.scheduled = false;
+		}
+		this.wake();
+	}
+	tokenizeSlice(highlighter, model, deadline) {
+		let grammar;
+		try {
+			highlighter.setTheme(model.theme);
+			grammar = highlighter.getLanguage(model.language);
+		} catch {
+			this.models.delete(model.id);
+			this.send({
+				kind: "unavailable",
+				model: model.id
+			}, []);
+			return;
+		}
+		const batch = new Batch();
+		const requested = [...model.requested];
+		model.requested.length = 0;
+		for (const { start, end } of requested) for (let line = start; line < Math.min(end, model.lines.length); line += 1) {
+			const tokens = model.tokens[line];
+			const state = model.states[line];
+			if (tokens && state && !model.invalid.has(line)) batch.add(line, tokens, state);
+		}
+		while (this.now() < deadline) {
+			const line = model.invalid.first();
+			if (line === void 0) break;
+			const start = line === 0 ? INITIAL : model.states[line - 1] ?? INITIAL;
+			const { tokens, endState } = tokenizeLine(grammar, model, model.lines[line] ?? "", start);
+			const previous = model.states[line];
+			model.states[line] = endState;
+			model.tokens[line] = tokens;
+			model.invalid.delete(line);
+			if (line + 1 < model.lines.length && !previous?.equals(endState)) model.invalid.add(line + 1, line + 2);
+			batch.add(line, tokens, endState);
+		}
+		if (!batch.empty) batch.post(this.send, model.id, model.version);
+		if (model.invalid.first() === void 0 && !model.finished) {
+			model.finished = true;
+			this.send({
+				kind: "finished",
+				model: model.id,
+				version: model.version
+			}, []);
+		}
+	}
+	receive(message) {
+		switch (message.kind) {
+			case "build":
+				this.highlighter = (async () => createHighlighterCore({
+					themes: [...message.themes],
+					langs: [...message.languages],
+					engine: await engineOf(message.engine)
+				}))();
+				break;
+			case "languages":
+				this.highlighter = this.highlighter?.then((highlighter) => {
+					try {
+						highlighter.loadLanguageSync([...message.languages]);
+					} catch (error) {
+						console.warn("inkstone: the tokenization worker could not load a grammar", error);
+					}
+					return highlighter;
+				});
+				break;
+			case "open":
+				this.models.set(message.model, {
+					id: message.model,
+					language: message.language,
+					languageId: message.languageId,
+					theme: message.theme,
+					colorIds: message.colorIds,
+					version: message.version,
+					lines: [...message.lines],
+					states: [],
+					tokens: [],
+					invalid: new InvalidLines(message.lines.length),
+					requested: [],
+					finished: false
+				});
+				break;
+			case "change": {
+				const model = this.models.get(message.model);
+				if (model) {
+					for (const change of message.changes) {
+						const splice = lineSplice(change);
+						applyChange(model.lines, change);
+						applySplice(model.states, splice, () => void 0);
+						applySplice(model.tokens, splice, () => void 0);
+						model.invalid.splice(splice);
+					}
+					model.version = message.version;
+					model.finished = false;
+					model.requested.length = 0;
+				}
+				break;
+			}
+			case "retokenize":
+				this.models.get(message.model)?.requested.push({
+					start: message.start - 1,
+					end: message.end - 1
+				});
+				break;
+			case "close": this.models.delete(message.model);
+		}
+		this.wake();
+	}
+};
+//#endregion
+//#region src/tokenization/worker-entry.ts
+function startTokenizationWorker() {
+	const scope = globalThis;
+	const worker = new TokenizationWorker((message, transfer) => {
+		scope.postMessage(message, transfer);
+	});
+	scope.addEventListener("message", (event) => {
+		worker.receive(event.data);
+	});
+}
+//#endregion
+//#region src/workers/textmate.worker.js
+startTokenizationWorker();
+//#endregion
+export {};

package/dist/workers/ts.worker.js

@@ -63,7 +63,7 @@ class JsxTypeScriptWorker extends TypeScriptWorker {
   }
 }
 
-// Monaco's worker handshake is two-staged (`internal/common/initialize.js`, 0.56): a first
+// Monaco's worker handshake is two-staged (`internal/common/initialize.js`, 0.57): a first
 // message merely triggers `initialize`, whose own handler then reads the *second* message as
 // createData. The stock entry import above installed the stage-one handler for its own
 // factory; this assignment must *replace* it with one building the subclass — calling

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@coldsmirk/inkstone-monaco",
-  "version": "0.29.0",
+  "version": "0.31.0",
   "description": "Framework-agnostic Monaco assembly: lazy offline host (all language workers bundled, no CDN), opt-in Simplified-Chinese UI via monaco's official NLS catalog, a streaming editor controller for AI-typed content, and the shared Shiki highlighting bridge.",
   "keywords": [
     "monaco",
@@ -40,14 +40,15 @@
   ],
   "dependencies": {
     "@shikijs/vscode-textmate": "^10.0.2",
-    "@coldsmirk/inkstone-core": "^0.29.0"
+    "shiki": "^4.5.0",
+    "@coldsmirk/inkstone-core": "^0.31.0"
   },
   "devDependencies": {
-    "monaco-editor": "^0.56.0",
-    "vite": "^8.2.2"
+    "monaco-editor": "^0.57.0",
+    "vite": "^8.3.4"
   },
   "peerDependencies": {
-    "monaco-editor": "^0.56.0",
+    "monaco-editor": "^0.57.0",
     "vite": ">=6"
   },
   "peerDependenciesMeta": {