@coldsmirk/inkstone-monaco 0.28.1 → 0.30.0

package/README.md

@@ -6,7 +6,7 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 
 ## What you get
 
-- **`ensureMonacoHost(options?)`** — the process-wide host: resolves the local `monaco-editor` module (never the CDN), routes **all** built-in language workers (base editor, TypeScript/JavaScript, JSON, CSS/scss/less, HTML/handlebars/razor) as bundler-emitted chunks so everything works offline, and applies `locale: "zh-cn"` via Monaco's official NLS catalog when asked — plus, on every editor created under that host, zh-CN descriptions for the six command-palette actions Monaco describes in English literals that no catalog can reach (Fold, Unfold, Find with Arguments, Refactor..., Source Action..., Paste As...), each held to the installed `monaco-editor` by an upgrade-tripwire test. Idempotent — every caller gets the same module; a load failure is not cached, so a fresh call retries.
+- **`ensureMonacoHost(options?)`** — the process-wide host: resolves the local `monaco-editor` module (never the CDN) and the parts of it the Shiki tokenization stands on, routes **all** built-in language workers (base editor, TypeScript/JavaScript, JSON, CSS/scss/less, HTML/handlebars/razor) and inkstone's own tokenization worker (the `textmate` label) as bundler-emitted chunks so everything works offline, and applies `locale: "zh-cn"` via Monaco's official NLS catalog when asked — plus, on every editor created under that host, zh-CN descriptions for the six command-palette actions Monaco describes in English literals that no catalog can reach (Fold, Unfold, Find with Arguments, Refactor..., Source Action..., Paste As...), each held to the installed `monaco-editor` by an upgrade-tripwire test. Idempotent — every caller gets the same module; a load failure is not cached, so a fresh call retries.
 - **`getWorker` extension point** — plug third-party language workers (e.g. monaco-yaml) into the routing; return `undefined` to fall back to the built-in routes.
 - **`monacoLanguages`** — the sorted catalog of every built-in language id the installed Monaco registers (with the `MonacoBuiltInLanguage` union type behind it), held to the installed `monaco-editor` by an upgrade-tripwire test. These are *ids*, not display names (`proto`, `sol`, `aes`); an unknown id silently falls back to `plaintext`.
 - **`disabledKeybindings` / `installDisabledKeybindings`** — swallow the find/replace chords and the command palette's F1 for editors that opt in. Monaco's standalone keybinding service is page-global and its rules are permanent, so the install registers each chord once per monaco module's service (duplicate copies of this package share the ledger; independent monaco bundles register their own) and scopes it with a per-capability context key created on the opted-in editor; the chord table is held to the installed `monaco-editor` by an upgrade-tripwire test.
@@ -15,9 +15,10 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 - **Embedded-editor fixes** — works around [microsoft/monaco-editor#5177](https://github.com/microsoft/monaco-editor/issues/5177) (find-widget button tooltips landing on top of the buttons they describe) with a `width: max-content` hover-layer style tagged `data-inkstone="monaco-hover-geometry"`; deleted the moment upstream fixes it. The tooltip layer positions against the editor's container — give that container `position: relative`. The command palette's row tooltips have the same trouble inside the palette: 0.56 positions them against the palette's list, which sets no position of its own, so they landed above the row they describe (by the height of the filter box); a `position: relative` style on the list, tagged `data-inkstone="monaco-quick-input-geometry"`, puts them on their row until upstream does.
 - **`StreamingEditorController`** — stream generated text into an editor without routing every token through app state: `begin()` / `append(chunk)` / `end()` / `discard()`, `attach(editor)` / `detach(editor)` with buffered replay for late-mounting editors (and for a model swapped onto the attached editor mid-stream, including after the final chunk), and `isStreaming()` for the onChange guard. The stream locks every editor it touches read-only and `end()` / `discard()` restore each one's own setting — by handle, so an editor detached mid-stream is restored too; `discard()` puts every displaced document back into the model it came from, and `end()` does the same for every model except the one keeping the streamed text. Full-buffer writes and restorations preserve each model's UTF-8 BOM. Editors and models the app disposed mid-stream are skipped, never touched — teardown can race the stream without wedging it.
 - **`replaceInEditor(editor, old, new, all)`** — exact find-and-replace on an editor handle via `executeEdits`, preserving undo/cursor/folds; returns whether the edit landed (`false` = no editor / empty search / no match / a read-only editor refused it — e.g. one locked by a stream in progress), so a tool-call handler knows to fall back to a state write.
-- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module, once per distinct module: defines the highlighter's themes as Monaco themes, registers each loaded grammar's canonical name as a Monaco language where Monaco has none and installs a Shiki tokens provider for it, and patches `setTheme` so the Shiki theme ids route through the highlighter while non-Shiki ids (vs/vs-dark, caller-defined) keep Monaco's original path. Called again on a bridged module, it installs the grammars loaded since. A failed registration is restored and retried on the next call.
-- **Which regex engine** — the bridge tokenizes whole files through the shared highlighter, and `@coldsmirk/inkstone-core`'s default JavaScript engine translates each grammar's patterns at the first tokenization, on the main thread (a TypeScript file's first sixty lines: about 170 ms, colour landing a beat after the text on every first open of a language). An app that opens files in Monaco should configure `onigurumaEngine` from `@coldsmirk/inkstone-core` (`configureHighlighter({ engine: onigurumaEngine })`, before the first `getHighlighter()`): its WASM is inlined into your bundle and compiles the same lines in about 20 ms.
-- **`registerShikiLanguage(monaco, highlighter, language)`** — the lazy path: an app that loads grammars as files are opened calls this after each `highlighter.loadLanguage(...)` and before a model names the language. Registers the language where Monaco lacks it (`toml`, `shellscript`), installs its tokens provider once under the grammar's canonical name (an alias like `ts` collapses), and leaves the theme the editor shows alone — the one thing re-running `@shikijs/monaco`'s `shikiToMonaco` cannot promise, which is why the bridge is inkstone's own. Throws for a grammar the highlighter has not loaded.
+- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module (one from `ensureMonacoHost`), once per distinct module: defines the highlighter's themes as Monaco themes, registers each loaded grammar's canonical name as a Monaco language where Monaco has none and its Shiki tokenization in Monaco's registry — over Monaco's own, JSON's included (its mode's tokenizer is turned off through its configuration, its other features left on) — and patches `setTheme` so the Shiki theme ids route through the highlighter while non-Shiki ids (vs/vs-dark, caller-defined) keep Monaco's original path. Called again on a bridged module, it registers the grammars loaded since. A failed registration is restored and retried on the next call.
+- **Tokenization off the page's thread** — VS Code's design, through the same door in Monaco: each model an editor shows is tokenized whole in a worker of its own, built from what the shared highlighter was built from (`highlighterSources`), the tokens and the state each line ends in sent back as they are made and moved past the edits the worker has not yet seen. The page's thread tokenizes only what Monaco still asks of it before the worker reaches it — a model's first lines in view, the lines an edit touches — and of those only a line it can afford: one of 1,000 characters or more, or one the grammar does not finish in 50 ms, is drawn in the default color until the worker's tokens arrive, the line after it read from the state before it (a minified line costs the grammar about 10 ms per 1,000 characters, 370 over 10,000, and one in view would hold the page that long). The worker bounds a line as VS Code does, at 20,000 characters and 500 ms. Where the page passes a line over in the lines it paints first, its own states for the lines after it stand — Monaco keeps a state it computed over one sent later — so an edit there is drawn from them until the worker's tokens arrive. A highlighter inkstone did not build, or a host that routes no worker for `textmate`, leaves every model to Monaco's own pass on the page's thread, under the same bound.
+- **Which regex engine** — the worker builds the engine kind the shared highlighter was configured with: Oniguruma wherever an engine was configured, the strict JavaScript engine where none was, so the page and the worker tokenize alike. The JavaScript engine translates each grammar's patterns at the first tokenization — in the worker for a model, and on the page for the lines it tokenizes itself (a TypeScript file's first sixty lines: about 170 ms) — so an app that opens files in Monaco should configure `onigurumaEngine` from `@coldsmirk/inkstone-core` (`configureHighlighter({ engine: onigurumaEngine })`, before the first `getHighlighter()`): its WASM is inlined into your bundle and compiles the same lines in about 20 ms.
+- **`registerShikiLanguage(monaco, highlighter, language)`** — the lazy path: an app that loads grammars as files are opened calls this after each `highlighter.loadLanguage(...)` and before a model names the language. Registers the language where Monaco lacks it (`toml`, `shellscript`), its tokenization once under the grammar's canonical name (an alias like `ts` collapses), the models of a grammar shiki has built again since (a language it embeds lazily arrived) tokenized again under the new one, and leaves the theme the editor shows alone — the one thing re-running `@shikijs/monaco`'s `shikiToMonaco` cannot promise, which is why the bridge is inkstone's own. Throws for a grammar the highlighter has not loaded.
 - **`installDesignTokens()`** — defines the VS Code design tokens monaco-editor 0.56's chrome reads and the standalone editor never does ([microsoft/monaco-editor#5408](https://github.com/microsoft/monaco-editor/issues/5408)): `--vscode-cornerRadius-{small,medium,large,xLarge}` and `--vscode-shadow-{md,lg,xl}`, on `.monaco-editor` (the context menu's shadow root and a host's `overflowWidgetsDomNode` inherit them). Without them the context menu, completion, hover, and find widgets ship square and shadowless. Each resolves through the host's `--inkstone-radius-*` / `--inkstone-shadow-*` token first (see `@coldsmirk/inkstone-core`), then a fallback scale in the spirit of 0.55's look. It also keeps an input box's focus ring single: the standalone editor outlines both the `.monaco-inputbox` container (its `synthetic-focus` ring, which follows the box's radius) and the inner `<input>` (square, 1px inside), and the inner ring's corners poke out as soon as the box has corners — the container's ring is the one ring. `ensureMonacoHost` installs it; idempotent.
 - **`DEFAULT_MONACO_OPTIONS`** — the inkstone construction-option baseline (no minimap, 14px/1.6 type on the shared `--inkstone-font-family-code` token, embedded-friendly find widget, tuned completion, thin scrollbars). Spread it to extend: `{ ...DEFAULT_MONACO_OPTIONS, wordWrap: "on" }`.
 - **`reconcileModelValue` / `modelValueMatches`** — the controlled-value reconcile: diff an external value against a live model (normalized to the model's EOL) and apply one minimal `applyEdits` replace that never lands on the undo stack or moves a caret outside the changed region.
@@ -28,7 +29,7 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 pnpm add @coldsmirk/inkstone-monaco monaco-editor
 ```
 
-`monaco-editor` is a **peer dependency** — your app owns the version. Node.js >= 24 for build / SSR hosts. Workers are bundled from dependency-relative `new Worker(new URL(...))` references; Vite and webpack 5 handle this form natively. Plain Rollup does not emit these workers by itself, so configure worker/asset handling that bundles the referenced modules or implement `getWorker` for every label your app can request.
+`monaco-editor` is a **peer dependency** — your app owns the version. Node.js >= 24 for build / SSR hosts. Workers are bundled from dependency-relative `new Worker(new URL(...))` references — monaco's five and inkstone's tokenization worker (`workers/textmate.worker.js`, under the `textmate` label); Vite and webpack 5 handle this form natively. Plain Rollup does not emit these workers by itself, so configure worker/asset handling that bundles the referenced modules or implement `getWorker` for every label your app can request.
 
 ## Quick start
 

package/dist/bounds-CZ_iAUoc.js

@@ -0,0 +1,120 @@
+import { INITIAL } from "@shikijs/vscode-textmate";
+//#region src/tokenization/content.ts
+const LINE_BREAK = /\r\n|\r|\n/;
+function lineCount(text) {
+	let count = 1;
+	for (let index = 0; index < text.length; index += 1) {
+		const char = text.codePointAt(index);
+		if (char === 10) count += 1;
+		else if (char === 13) {
+			count += 1;
+			if (text.codePointAt(index + 1) === 10) index += 1;
+		}
+	}
+	return count;
+}
+function lineSplice({ range, text }) {
+	return {
+		start: range.startLineNumber - 1,
+		removed: range.endLineNumber - range.startLineNumber + 1,
+		inserted: lineCount(text)
+	};
+}
+function applySplice(array, { start, removed, inserted }, fill) {
+	const RUN = 8192;
+	const items = Array.from({ length: inserted }, (_, index) => fill(index));
+	array.splice(start, removed, ...items.slice(0, RUN));
+	for (let from = RUN; from < items.length; from += RUN) array.splice(start + from, 0, ...items.slice(from, from + RUN));
+}
+function applyChange(lines, change) {
+	const { range, text } = change;
+	const written = `${(lines[range.startLineNumber - 1] ?? "").slice(0, range.startColumn - 1)}${text}${(lines[range.endLineNumber - 1] ?? "").slice(range.endColumn - 1)}`.split(LINE_BREAK);
+	applySplice(lines, lineSplice(change), (index) => written[index] ?? "");
+}
+function lineAfter(line, edits) {
+	let at = line;
+	for (const { changes } of edits) for (const change of changes) {
+		const { start, removed, inserted } = lineSplice(change);
+		if (at - 1 >= start + removed) at += inserted - removed;
+		else if (at - 1 >= start) return;
+	}
+	return at;
+}
+//#endregion
+//#region src/tokenization/states.ts
+const Stack = INITIAL.constructor;
+function stateDiff(previous, next) {
+	let pops = 0;
+	const newFrames = [];
+	let first = previous;
+	let second = next;
+	while (first !== second) if (first && (!second || first.depth >= second.depth)) {
+		pops += 1;
+		first = first.parent;
+	} else if (second) {
+		newFrames.push(second.toStateStackFrame());
+		second = second.parent;
+	}
+	return {
+		pops,
+		newFrames: newFrames.toReversed()
+	};
+}
+function applyStateDiff(previous, diff) {
+	let stack = previous;
+	for (let pop = 0; pop < diff.pops; pop += 1) {
+		if (!stack) return null;
+		stack = stack.parent;
+	}
+	for (const frame of diff.newFrames) stack = Stack.pushFrame(stack, frame);
+	return stack;
+}
+//#endregion
+//#region src/tokenization/metadata.ts
+const LANGUAGE_MASK = 255;
+const KEPT_MASK = 32512;
+const FOREGROUND_MASK = 16744448;
+const FOREGROUND_OFFSET = 15;
+const BACKGROUND_OFFSET = 24;
+const DEFAULT_FOREGROUND = 1;
+const DEFAULT_BACKGROUND = 2;
+function hexOf(color) {
+	const hex = color.trim().toLowerCase();
+	return hex.length === 9 && hex.endsWith("ff") ? hex.slice(0, 7) : hex;
+}
+function channel(value) {
+	return Math.round(value).toString(16).padStart(2, "0");
+}
+function hexOfMapColor({ rgba }) {
+	const opaque = `#${channel(rgba.r)}${channel(rgba.g)}${channel(rgba.b)}`;
+	return rgba.a >= 1 ? opaque : `${opaque}${channel(rgba.a * 255)}`;
+}
+function colorIdsFor(highlighterColors, monacoColors) {
+	const monacoIds = /* @__PURE__ */ new Map();
+	const colors = monacoColors ?? [];
+	for (const [id, color] of colors.entries()) if (color && !monacoIds.has(hexOfMapColor(color))) monacoIds.set(hexOfMapColor(color), id);
+	return Uint16Array.from(highlighterColors, (color) => monacoIds.get(hexOf(color ?? "")) ?? DEFAULT_FOREGROUND);
+}
+function monacoMetadata(metadata, languageId, colorIds) {
+	const foreground = colorIds[(metadata & FOREGROUND_MASK) >>> FOREGROUND_OFFSET] ?? DEFAULT_FOREGROUND;
+	return (languageId & LANGUAGE_MASK | metadata & KEPT_MASK | foreground << FOREGROUND_OFFSET | DEFAULT_BACKGROUND << BACKGROUND_OFFSET) >>> 0;
+}
+function defaultMetadata(languageId) {
+	return (languageId & LANGUAGE_MASK | 33587200) >>> 0;
+}
+function toMonacoTokens(tokens, languageId, colorIds) {
+	for (let index = 1; index < tokens.length; index += 2) tokens[index] = monacoMetadata(tokens[index] ?? 0, languageId, colorIds);
+	return tokens;
+}
+function withEndOffsets(tokens, lineLength) {
+	const last = tokens.length - 2;
+	for (let index = 0; index < last; index += 2) tokens[index] = tokens[index + 2] ?? lineLength;
+	if (last >= 0) tokens[last] = lineLength;
+	return tokens;
+}
+//#endregion
+//#region src/tokenization/bounds.ts
+const SYNC_LINE_BOUND = 1e3;
+const WORKER_LINE_BOUND = 2e4;
+//#endregion
+export { toMonacoTokens as a, stateDiff as c, lineAfter as d, lineSplice as f, defaultMetadata as i, applyChange as l, WORKER_LINE_BOUND as n, withEndOffsets as o, colorIdsFor as r, applyStateDiff as s, SYNC_LINE_BOUND as t, applySplice as u };

package/dist/index.d.ts

@@ -33,8 +33,9 @@ interface MonacoHostOptions {
  */
 type MonacoModule = typeof monacoEditor;
 /**
- * Bring up the shared Monaco host: UI locale, then the monaco module, then worker routing —
- * fully offline, no CDN, workers bundled by the consumer's bundler.
+ * Bring up the shared Monaco host: UI locale, then the monaco module and the parts of it the
+ * Shiki tokenization stands on, then worker routing — fully offline, no CDN, workers bundled by
+ * the consumer's bundler.
  *
  * Idempotent and async-safe: every caller shares one promise, and the first call wins the
  * configuration (`globalThis.MonacoEnvironment` is a single global — per-editor configuration
@@ -249,16 +250,25 @@ declare function replaceInEditor(target: editor.ICodeEditor | null, search: stri
 //#endregion
 //#region src/shiki-bridge.d.ts
 /**
- * Register the shared Shiki highlighter's languages, themes, and tokenizers on a monaco module —
- * once per distinct module, whichever caller gets there first — and, on a module already
- * bridged, install the tokenizer of any grammar loaded since. Language services are unaffected:
+ * Register the shared Shiki highlighter's languages, themes, and tokenization on a monaco module
+ * — once per distinct module, whichever caller gets there first — and, on a module already
+ * bridged, the tokenization of any grammar loaded since. Language services are unaffected:
  * Shiki replaces syntax highlighting only. The module is recorded only after every registration
- * succeeds, so a transient failure can retry on a later call.
+ * succeeds, so a transient failure can retry on a later call. The module must be the host's
+ * (`ensureMonacoHost`), which brings up the parts of Monaco the tokenization stands on.
+ *
+ * A model is tokenized whole in a worker of its own, built from what the highlighter was built
+ * from (`@coldsmirk/inkstone-core`'s `highlighterSources`) — VS Code's design, through the same
+ * door in Monaco: the page's thread tokenizes only the lines Monaco asks of it before the worker
+ * reaches them — a model's first lines in view, the lines an edit touches — and of those only a
+ * line it can afford; a longer one waits for the worker's colours (`tokenization/bounds.ts`).
+ * A highlighter inkstone did not build, or a host that routes no worker for the `textmate`
+ * label, leaves every model to Monaco's own pass on the page's thread, under the same bound.
  *
  * Monaco's own `setTheme` is patched to route the Shiki theme ids through the highlighter, so
- * the tokenizers paint in the theme the editor shows; Monaco's built-in themes (vs/vs-dark) and
- * caller-defined themes keep Monaco's original path. `create` is patched so its `theme` option
- * reaches that patch. The shape is `@shikijs/monaco`'s, kept here because that package
+ * the tokenization paints in the theme the editor shows; Monaco's built-in themes (vs/vs-dark)
+ * and caller-defined themes keep Monaco's original path. `create` is patched so its `theme`
+ * option reaches that patch. The shape is `@shikijs/monaco`'s, kept here because that package
  * installs tokenizers only for the grammars loaded at the moment it is called, and re-calling
  * it for a grammar loaded later resets the editor's theme to the first one.
  *
@@ -267,12 +277,13 @@ declare function replaceInEditor(target: editor.ICodeEditor | null, search: stri
  */
 declare function registerShikiHighlighting(monaco: MonacoModule, highlighter: HighlighterCore): void;
 /**
- * Install the tokenizer of one grammar the highlighter has loaded since the module was bridged
- * — the lazy path: an app that loads grammars as files are opened registers each here, after
- * its `loadLanguage`, and before a model names it. `language` may be an alias (`ts`); it
+ * Register the tokenization of one grammar the highlighter has loaded since the module was
+ * bridged — the lazy path: an app that loads grammars as files are opened registers each here,
+ * after its `loadLanguage`, and before a model names it. `language` may be an alias (`ts`); it
  * collapses to the grammar's canonical name. Bridges the module first where nothing has, and
- * is nothing for a grammar already registered. Throws when the highlighter has not loaded the
- * grammar: there is nothing to tokenize with.
+ * is nothing for a grammar already registered — but a grammar shiki has built again since, as
+ * a language it embeds lazily arrived, has its models tokenized again under the new one. Throws
+ * when the highlighter has not loaded the grammar: there is nothing to tokenize with.
  */
 declare function registerShikiLanguage(monaco: MonacoModule, highlighter: HighlighterCore, language: string): void;
 //#endregion

package/dist/index.js

@@ -1,4 +1,5 @@
-import { CODE_FONT_FAMILY, inkstoneRadius, inkstoneShadow } from "@coldsmirk/inkstone-core";
+import { a as toMonacoTokens, d as lineAfter, i as defaultMetadata, r as colorIdsFor, s as applyStateDiff } from "./bounds-CZ_iAUoc.js";
+import { CODE_FONT_FAMILY, highlighterSources, inkstoneRadius, inkstoneShadow } from "@coldsmirk/inkstone-core";
 import { EncodedTokenMetadata, INITIAL } from "@shikijs/vscode-textmate";
 //#region src/auto-imports.ts
 const MISSING_LOOKUP_REJECTION$1 = "Missing requestHandler or method: getAutoImportCompletionsAtPosition";
@@ -214,6 +215,33 @@ function installHoverFocusFix() {
 	document.head.append(style);
 }
 //#endregion
+//#region src/tokenization/internals.ts
+async function loadMonacoInternals() {
+	const [languages, tokens] = await Promise.all([import("monaco-editor/editor/common/languages.js"), import("monaco-editor/editor/common/tokens/contiguousMultilineTokensBuilder.js")]);
+	return {
+		registry: languages.TokenizationRegistry,
+		encoded: (encoded, endState) => new languages.EncodedTokenizationResult(encoded, [], endState),
+		classic: (classic, language, endState) => new languages.TokenizationResult(classic.map((token) => new languages.Token(token.offset, token.type, language)), endState),
+		multiline: (lines) => {
+			const builder = new tokens.ContiguousMultilineTokensBuilder();
+			for (const [lineNumber, lineTokens] of lines) builder.add(lineNumber, lineTokens);
+			return builder.finalize();
+		}
+	};
+}
+const INTERNALS_KEY = Symbol.for("coldsmirk.inkstone.monaco.internals");
+function ledger() {
+	const host = globalThis;
+	host[INTERNALS_KEY] ??= /* @__PURE__ */ new WeakMap();
+	return host[INTERNALS_KEY];
+}
+function rememberInternals(monaco, internals) {
+	ledger().set(monaco, internals);
+}
+function internalsOf(monaco) {
+	return ledger().get(monaco);
+}
+//#endregion
 //#region src/host.ts
 const HOST_PROMISE_KEY = Symbol.for("coldsmirk.inkstone.monaco.hostPromise");
 const NLS_REGISTRY_KEY = Symbol.for("coldsmirk.inkstone.monaco.nlsRegistry");
@@ -254,11 +282,13 @@ async function applyZhCnCatalog() {
 function editorWorker() {
 	return new Worker(new URL("workers/editor.worker.js", import.meta.url), { type: "module" });
 }
+const TEXTMATE_WORKER_LABEL = "textmate";
 const WORKER_BY_LABEL = {
 	typescript: () => new Worker(new URL("workers/ts.worker.js", import.meta.url), { type: "module" }),
 	json: () => new Worker(new URL("workers/json.worker.js", import.meta.url), { type: "module" }),
 	css: () => new Worker(new URL("workers/css.worker.js", import.meta.url), { type: "module" }),
-	html: () => new Worker(new URL("workers/html.worker.js", import.meta.url), { type: "module" })
+	html: () => new Worker(new URL("workers/html.worker.js", import.meta.url), { type: "module" }),
+	[TEXTMATE_WORKER_LABEL]: () => new Worker(new URL("workers/textmate.worker.js", import.meta.url), { type: "module" })
 };
 const WORKER_ALIASES = {
 	javascript: "typescript",
@@ -286,6 +316,7 @@ function ensureMonacoHost(options = {}) {
 			installQuickInputGeometryFix();
 			installHoverFocusFix();
 			const monaco = await import("monaco-editor");
+			rememberInternals(monaco, await loadMonacoInternals());
 			globalThis.MonacoEnvironment = { getWorker: (workerId, label) => getWorker?.(workerId, label) ?? builtinGetWorker(label) };
 			if (locale === "zh-cn") monaco.editor.onDidCreateEditor(localizeActionDescriptions);
 			return monaco;
@@ -600,6 +631,280 @@ function replaceInEditor(target, search, replaceWith, all) {
 	return target.executeEdits("inkstone-replace", edits);
 }
 //#endregion
+//#region src/tokenization/controller.ts
+var WorkerTokenizerController = class {
+	model;
+	store;
+	internals;
+	channel;
+	attachment;
+	session;
+	disposed = false;
+	language;
+	constructor(options) {
+		this.language = options.language;
+		this.model = options.model;
+		this.store = options.store;
+		this.internals = options.internals;
+		this.channel = options.channel;
+		this.attachment = this.model.onDidChangeAttached(() => {
+			this.follow();
+		});
+		queueMicrotask(() => {
+			this.follow();
+		});
+	}
+	follow() {
+		if (this.disposed) return;
+		const attached = this.model.isAttachedToEditor();
+		if (attached && !this.session) this.connect();
+		else if (!attached && this.session) this.disconnect();
+	}
+	connect() {
+		const id = this.channel.open(this, this.model.getLinesContent(), this.model.getVersionId());
+		const pending = [];
+		const listener = this.model.onDidChangeContent((event) => {
+			const changes = event.changes.map(({ range, text }) => {
+				return {
+					range: {
+						startLineNumber: range.startLineNumber,
+						startColumn: range.startColumn,
+						endLineNumber: range.endLineNumber,
+						endColumn: range.endColumn
+					},
+					text
+				};
+			});
+			pending.push({
+				version: event.versionId,
+				changes
+			});
+			this.channel.send({
+				kind: "change",
+				model: id,
+				version: event.versionId,
+				changes
+			});
+		});
+		this.session = {
+			id,
+			pending,
+			listener
+		};
+	}
+	disconnect() {
+		if (!this.session) return;
+		this.session.listener.dispose();
+		this.channel.close(this.session.id);
+		this.session = void 0;
+	}
+	keepStates(message) {
+		let previous = null;
+		for (const [index, line] of message.lines.entries()) {
+			const diff = message.states[index];
+			const follows = index > 0 && message.lines[index - 1] === line - 1;
+			let state = null;
+			if (diff && (!follows || previous)) try {
+				state = applyStateDiff(follows ? previous : null, diff);
+			} catch {
+				state = null;
+			}
+			if (state) this.store.setEndState(line, state);
+			previous = state;
+		}
+	}
+	receiveTokens(message) {
+		const { session } = this;
+		if (session?.id !== message.model) return;
+		const { pending } = session;
+		while (pending[0] && pending[0].version <= message.version) pending.shift();
+		const lines = [];
+		let offset = 0;
+		for (const [index, sent] of message.lines.entries()) {
+			const length = message.lengths[index] ?? 0;
+			const tokens = message.tokens.subarray(offset, offset + length);
+			const line = lineAfter(sent, pending);
+			offset += length;
+			if (line !== void 0) lines.push([line, tokens]);
+		}
+		if (lines.length > 0) this.store.setTokens(this.internals.multiline(lines));
+		if (pending.length === 0) this.keepStates(message);
+	}
+	finished(model, version) {
+		const { session } = this;
+		if (session?.id === model && session.pending.length === 0 && version === this.model.getVersionId()) this.store.backgroundTokenizationFinished();
+	}
+	requestTokens(startLineNumber, endLineNumberExclusive) {
+		if (this.session) this.channel.send({
+			kind: "retokenize",
+			model: this.session.id,
+			start: startLineNumber,
+			end: endLineNumberExclusive
+		});
+	}
+	dispose() {
+		if (this.disposed) return;
+		this.disposed = true;
+		this.attachment.dispose();
+		this.disconnect();
+	}
+};
+//#endregion
+//#region src/tokenization/client.ts
+var TokenizationClient = class {
+	options;
+	controllers = /* @__PURE__ */ new Map();
+	worker;
+	sentLanguages = 0;
+	nextModel = 1;
+	dead = false;
+	constructor(options) {
+		this.options = options;
+	}
+	start() {
+		if (this.dead) return false;
+		if (this.worker) return true;
+		let created;
+		try {
+			created = this.options.createWorker();
+		} catch {
+			this.dead = true;
+			return false;
+		}
+		this.worker = Promise.resolve(created);
+		this.worker.then((worker) => {
+			worker.addEventListener("message", (event) => {
+				this.receive(event.data);
+			});
+			worker.addEventListener("error", () => {
+				this.fail();
+			});
+		}, () => {
+			this.fail();
+		});
+		const { sources } = this.options;
+		this.sentLanguages = sources.languages.length;
+		this.post({
+			kind: "build",
+			engine: sources.engine,
+			themes: this.options.themes(),
+			languages: [...sources.languages]
+		});
+		return true;
+	}
+	sendLanguages() {
+		const { languages } = this.options.sources;
+		if (languages.length > this.sentLanguages) {
+			this.post({
+				kind: "languages",
+				languages: languages.slice(this.sentLanguages)
+			});
+			this.sentLanguages = languages.length;
+		}
+	}
+	post(message) {
+		this.worker?.then((worker) => {
+			worker.postMessage(message);
+		});
+	}
+	receive(message) {
+		switch (message.kind) {
+			case "tokens":
+				this.controllers.get(message.model)?.receiveTokens(message);
+				break;
+			case "finished":
+				this.controllers.get(message.model)?.finished(message.model, message.version);
+				break;
+			case "unavailable": {
+				const controller = this.controllers.get(message.model);
+				if (controller) this.options.unavailable([controller.language]);
+				break;
+			}
+			case "failed": this.fail();
+		}
+	}
+	fail() {
+		if (this.dead) return;
+		this.dead = true;
+		this.worker?.then((worker) => {
+			worker.terminate();
+		});
+		this.worker = void 0;
+		this.options.unavailable("all");
+	}
+	open(model, store, tokenization) {
+		if (!this.start()) return;
+		return new WorkerTokenizerController({
+			language: tokenization.language,
+			model,
+			store,
+			internals: this.options.internals,
+			channel: {
+				open: (controller, lines, version) => {
+					const id = this.nextModel;
+					this.nextModel += 1;
+					this.sendLanguages();
+					this.controllers.set(id, controller);
+					this.post({
+						kind: "open",
+						model: id,
+						...tokenization,
+						version,
+						lines
+					});
+					return id;
+				},
+				send: (message) => {
+					this.post(message);
+				},
+				close: (id) => {
+					this.controllers.delete(id);
+					this.post({
+						kind: "close",
+						model: id
+					});
+				}
+			}
+		});
+	}
+};
+//#endregion
+//#region src/tokenization/support.ts
+function shikiTokenization(options) {
+	const { internals, highlighter, language, languageId, theme, colorIds, scopeOf, background } = options;
+	const tokenizeOnPage = (line, state) => {
+		if (line.length >= 1e3) return;
+		try {
+			highlighter.setTheme(theme());
+			const result = highlighter.getLanguage(language).tokenizeLine2(line, state, 50);
+			return result.stoppedEarly ? void 0 : result;
+		} catch {
+			return;
+		}
+	};
+	return {
+		getInitialState: () => INITIAL,
+		tokenizeEncoded: (line, _hasEOL, state) => {
+			const result = tokenizeOnPage(line, state);
+			return result ? internals.encoded(toMonacoTokens(result.tokens, languageId, colorIds()), result.ruleStack) : internals.encoded(Uint32Array.of(0, defaultMetadata(languageId)), state);
+		},
+		tokenize: (line, _hasEOL, state) => {
+			const result = tokenizeOnPage(line, state);
+			if (!result) return internals.classic([{
+				offset: 0,
+				type: ""
+			}], language, state);
+			const tokens = [];
+			for (let index = 0; index < result.tokens.length; index += 2) tokens.push({
+				offset: result.tokens[index] ?? 0,
+				type: scopeOf(result.tokens[index + 1] ?? 0)
+			});
+			return internals.classic(tokens, language, result.ruleStack);
+		},
+		createBackgroundTokenizer: background
+	};
+}
+//#endregion
 //#region src/shiki-bridge.ts
 const SHIKI_REGISTRY_KEY = Symbol.for("coldsmirk.inkstone.monaco.shikiRegistry");
 function bridges() {
@@ -610,8 +915,6 @@ function bridges() {
 	host[SHIKI_REGISTRY_KEY] = created;
 	return created;
 }
-const TOKENIZE_MAX_LINE_LENGTH = 2e4;
-const TOKENIZE_TIME_LIMIT_MS = 500;
 const ITALIC = 1;
 const BOLD = 2;
 const UNDERLINE = 4;
@@ -675,6 +978,7 @@ function monacoThemeOf(theme) {
 function adoptTheme(bridge, themeName) {
 	const { colorMap } = bridge.highlighter.setTheme(themeName);
 	const theme = bridge.themes.get(themeName);
+	bridge.theme = themeName;
 	bridge.colorMap.length = colorMap.length;
 	for (const [index, color] of colorMap.entries()) bridge.colorMap[index] = color;
 	bridge.scopeOf.clear();
@@ -686,58 +990,78 @@ function adoptTheme(bridge, themeName) {
 		if (!bridge.scopeOf.has(key)) bridge.scopeOf.set(key, rule.token);
 	}
 }
-var TokenizerState = class TokenizerState {
-	ruleStack;
-	constructor(ruleStack) {
-		this.ruleStack = ruleStack;
-	}
-	clone() {
-		return new TokenizerState(this.ruleStack);
-	}
-	equals(other) {
-		return other instanceof TokenizerState && other.ruleStack === this.ruleStack;
-	}
-};
-function tokensProvider(bridge, lang) {
-	return {
-		getInitialState() {
-			return new TokenizerState(INITIAL);
-		},
-		tokenize(line, state) {
-			if (line.length >= TOKENIZE_MAX_LINE_LENGTH) return {
-				endState: state,
-				tokens: [{
-					startIndex: 0,
-					scopes: ""
-				}]
-			};
-			const ruleStack = state instanceof TokenizerState ? state.ruleStack : INITIAL;
-			const result = bridge.highlighter.getLanguage(lang).tokenizeLine2(line, ruleStack, TOKENIZE_TIME_LIMIT_MS);
-			if (result.stoppedEarly) console.warn(`inkstone: time limit reached when tokenizing a line: ${line.slice(0, 100)}`);
-			const tokens = [];
-			for (let index = 0; index < result.tokens.length; index += 2) {
-				const startIndex = result.tokens[index] ?? 0;
-				const metadata = result.tokens[index + 1] ?? 0;
-				const color = normalizeColor(bridge.colorMap[EncodedTokenMetadata.getForeground(metadata)]);
-				const fontStyle = normalizeFontStyleBits(EncodedTokenMetadata.getFontStyle(metadata));
-				const scopes = color ? bridge.scopeOf.get(colorStyleKey(color, fontStyle)) ?? "" : "";
-				tokens.push({
-					startIndex,
-					scopes
-				});
-			}
-			return {
-				endState: new TokenizerState(result.ruleStack),
-				tokens
-			};
-		}
+function scopeOfMetadata(bridge, metadata) {
+	const color = normalizeColor(bridge.colorMap[EncodedTokenMetadata.getForeground(metadata)]);
+	const fontStyle = normalizeFontStyleBits(EncodedTokenMetadata.getFontStyle(metadata));
+	return color ? bridge.scopeOf.get(colorStyleKey(color, fontStyle)) ?? "" : "";
+}
+function colorIds(bridge) {
+	const monacoColors = bridge.internals.registry.getColorMap();
+	if (bridge.colorIds?.theme !== bridge.theme || bridge.colorIds.monacoColors !== monacoColors) bridge.colorIds = {
+		theme: bridge.theme,
+		monacoColors,
+		ids: colorIdsFor(bridge.colorMap, monacoColors)
 	};
+	return bridge.colorIds.ids;
+}
+function clientOf(bridge) {
+	if (bridge.client !== void 0) return bridge.client;
+	const sources = highlighterSources(bridge.highlighter);
+	const getWorker = globalThis.MonacoEnvironment?.getWorker;
+	bridge.client = sources && getWorker ? new TokenizationClient({
+		internals: bridge.internals,
+		sources,
+		themes: () => bridge.highlighter.getLoadedThemes().map((id) => bridge.highlighter.getTheme(id)),
+		createWorker: () => getWorker("inkstone-textmate", TEXTMATE_WORKER_LABEL),
+		unavailable: (languages) => {
+			if (languages === "all") bridge.client = null;
+			else for (const language of languages) bridge.workerless.add(language);
+			bridge.internals.registry.handleChange(languages === "all" ? bridge.registered.keys().toArray() : [...languages]);
+		}
+	}) : null;
+	return bridge.client;
+}
+function quietBuiltInTokenizer(monaco, language) {
+	if (language !== "json") return;
+	const { jsonDefaults } = monaco.json;
+	if (jsonDefaults.modeConfiguration.tokens !== false) jsonDefaults.setModeConfiguration({
+		...jsonDefaults.modeConfiguration,
+		tokens: false
+	});
 }
 function install(monaco, bridge, canonical) {
 	if (bridge.registered.has(canonical)) return;
 	if (monaco.languages.getLanguages().every((language) => language.id !== canonical)) monaco.languages.register({ id: canonical });
-	monaco.languages.setTokensProvider(canonical, tokensProvider(bridge, canonical));
-	bridge.registered.add(canonical);
+	quietBuiltInTokenizer(monaco, canonical);
+	bridge.registered.set(canonical, bridge.highlighter.getLanguage(canonical));
+	bridge.internals.registry.register(canonical, shikiTokenization({
+		internals: bridge.internals,
+		highlighter: bridge.highlighter,
+		language: canonical,
+		languageId: monaco.languages.getEncodedLanguageId(canonical),
+		theme: () => bridge.theme,
+		colorIds: () => colorIds(bridge),
+		scopeOf: (metadata) => scopeOfMetadata(bridge, metadata),
+		background: (model, store) => {
+			return (bridge.workerless.has(canonical) ? null : clientOf(bridge))?.open(model, store, {
+				language: canonical,
+				languageId: monaco.languages.getEncodedLanguageId(canonical),
+				theme: bridge.theme,
+				colorIds: colorIds(bridge)
+			});
+		}
+	}));
+}
+function retokenizeRebuilt(bridge) {
+	const rebuilt = [];
+	for (const [language, grammar] of bridge.registered) {
+		const current = bridge.highlighter.getLanguage(language);
+		if (current !== grammar) {
+			bridge.registered.set(language, current);
+			rebuilt.push(language);
+		}
+	}
+	if (rebuilt.length > 0) bridge.internals.registry.handleChange(rebuilt);
 }
 function canonicalLanguages(highlighter) {
 	return new Set(highlighter.getLoadedLanguages().map((id) => highlighter.getLanguage(id).name));
@@ -747,14 +1071,22 @@ function registerShikiHighlighting(monaco, highlighter) {
 	const bridged = registry.get(monaco);
 	if (bridged) {
 		for (const canonical of canonicalLanguages(highlighter)) install(monaco, bridged, canonical);
+		retokenizeRebuilt(bridged);
 		return;
 	}
+	const internals = internalsOf(monaco);
+	if (!internals) throw new Error("inkstone: Shiki highlighting needs the monaco module ensureMonacoHost brought up");
 	const bridge = {
+		internals,
 		highlighter,
 		themes: /* @__PURE__ */ new Map(),
+		theme: "",
 		colorMap: [],
+		colorIds: void 0,
 		scopeOf: /* @__PURE__ */ new Map(),
-		registered: /* @__PURE__ */ new Set()
+		registered: /* @__PURE__ */ new Map(),
+		client: void 0,
+		workerless: /* @__PURE__ */ new Set()
 	};
 	const originalCreate = monaco.editor.create;
 	const originalSetTheme = monaco.editor.setTheme;
@@ -791,6 +1123,7 @@ function registerShikiLanguage(monaco, highlighter, language) {
 	}
 	if (!bridge) throw new Error("inkstone: the Shiki bridge is not installed on this monaco module");
 	install(monaco, bridge, highlighter.getLanguage(language).name);
+	retokenizeRebuilt(bridge);
 }
 //#endregion
 //#region src/streaming.ts

package/dist/workers/textmate.worker.js

@@ -0,0 +1,296 @@
+import { a as toMonacoTokens, c as stateDiff, f as lineSplice, i as defaultMetadata, l as applyChange, o as withEndOffsets, u as applySplice } from "../bounds-CZ_iAUoc.js";
+import { INITIAL } from "@shikijs/vscode-textmate";
+import { createHighlighterCore } from "shiki/core";
+//#region src/tokenization/invalid-lines.ts
+var InvalidLines = class {
+	runs = [];
+	constructor(lineCount = 0) {
+		this.add(0, lineCount);
+	}
+	first() {
+		return this.runs[0]?.start;
+	}
+	has(line) {
+		return this.runs.some((run) => run.start <= line && line < run.end);
+	}
+	add(start, end) {
+		if (end <= start) return;
+		let from = start;
+		let to = end;
+		const kept = this.runs.filter((run) => {
+			if (run.end < from || to < run.start) return true;
+			from = Math.min(from, run.start);
+			to = Math.max(to, run.end);
+			return false;
+		});
+		kept.push({
+			start: from,
+			end: to
+		});
+		kept.sort((one, other) => one.start - other.start);
+		this.runs.splice(0, this.runs.length, ...kept);
+	}
+	delete(line) {
+		const at = this.runs.findIndex((run) => run.start <= line && line < run.end);
+		const run = this.runs[at];
+		if (!run) return;
+		const pieces = [{
+			start: run.start,
+			end: line
+		}, {
+			start: line + 1,
+			end: run.end
+		}].filter((piece) => piece.start < piece.end);
+		this.runs.splice(at, 1, ...pieces);
+	}
+	splice({ start, removed, inserted }) {
+		const shift = inserted - removed;
+		const moved = this.runs.flatMap((run) => {
+			return [{
+				start: run.start,
+				end: Math.min(run.end, start)
+			}, {
+				start: Math.max(run.start, start + removed) + shift,
+				end: run.end + shift
+			}].filter((piece) => piece.start < piece.end);
+		});
+		this.runs.splice(0, this.runs.length, ...moved);
+		this.add(start, start + inserted);
+	}
+};
+//#endregion
+//#region src/tokenization/worker-runtime.ts
+function hasWork(model) {
+	return model.requested.length > 0 || model.invalid.first() !== void 0;
+}
+async function engineOf(kind) {
+	if (kind === "oniguruma") {
+		const { createOnigurumaEngine } = await import("shiki/engine/oniguruma");
+		return createOnigurumaEngine(import("shiki/wasm"));
+	}
+	const { createJavaScriptRegexEngine } = await import("shiki/engine/javascript");
+	return createJavaScriptRegexEngine();
+}
+function tokenizeLine(grammar, model, line, state) {
+	if (line.length >= 2e4) return {
+		tokens: Uint32Array.of(line.length, defaultMetadata(model.languageId)),
+		endState: state
+	};
+	const result = grammar.tokenizeLine2(line, state, 500);
+	return {
+		tokens: withEndOffsets(toMonacoTokens(result.tokens, model.languageId, model.colorIds), line.length),
+		endState: result.stoppedEarly ? state : result.ruleStack
+	};
+}
+var Batch = class {
+	lines = [];
+	lengths = [];
+	parts = [];
+	states = [];
+	last;
+	get empty() {
+		return this.lines.length === 0;
+	}
+	add(line, tokens, state) {
+		const previous = this.last?.line === line - 1 ? this.last.state : null;
+		this.lines.push(line + 1);
+		this.lengths.push(tokens.length);
+		this.parts.push(tokens);
+		this.states.push(stateDiff(previous, state));
+		this.last = {
+			line,
+			state
+		};
+	}
+	post(post, model, version) {
+		const tokens = new Uint32Array(this.lengths.reduce((sum, length) => sum + length, 0));
+		let offset = 0;
+		for (const part of this.parts) {
+			tokens.set(part, offset);
+			offset += part.length;
+		}
+		const lines = Uint32Array.from(this.lines);
+		const lengths = Uint32Array.from(this.lengths);
+		post({
+			kind: "tokens",
+			model,
+			version,
+			lines,
+			lengths,
+			tokens,
+			states: this.states
+		}, [
+			tokens.buffer,
+			lines.buffer,
+			lengths.buffer
+		]);
+	}
+};
+var TokenizationWorker = class {
+	send;
+	schedule;
+	now;
+	highlighter;
+	models = /* @__PURE__ */ new Map();
+	scheduled = false;
+	failed = false;
+	constructor(send, schedule = (work) => {
+		setTimeout(() => {
+			work();
+		}, 0);
+	}, now = () => performance.now()) {
+		this.send = send;
+		this.schedule = schedule;
+		this.now = now;
+	}
+	wake() {
+		if (this.scheduled || this.failed || !this.highlighter || this.models.values().every((model) => !hasWork(model))) return;
+		this.scheduled = true;
+		this.schedule(() => this.work());
+	}
+	async work() {
+		try {
+			const highlighter = await this.highlighter;
+			if (highlighter) {
+				const deadline = this.now() + 8;
+				const turns = this.models.values().toArray();
+				for (const model of turns) {
+					if (this.now() >= deadline) break;
+					if (hasWork(model)) {
+						this.tokenizeSlice(highlighter, model, deadline);
+						if (this.models.delete(model.id)) this.models.set(model.id, model);
+					}
+				}
+			}
+		} catch (error) {
+			this.failed = true;
+			this.send({
+				kind: "failed",
+				reason: error instanceof Error ? error.message : String(error)
+			}, []);
+		} finally {
+			this.scheduled = false;
+		}
+		this.wake();
+	}
+	tokenizeSlice(highlighter, model, deadline) {
+		let grammar;
+		try {
+			highlighter.setTheme(model.theme);
+			grammar = highlighter.getLanguage(model.language);
+		} catch {
+			this.models.delete(model.id);
+			this.send({
+				kind: "unavailable",
+				model: model.id
+			}, []);
+			return;
+		}
+		const batch = new Batch();
+		const requested = [...model.requested];
+		model.requested.length = 0;
+		for (const { start, end } of requested) for (let line = start; line < Math.min(end, model.lines.length); line += 1) {
+			const tokens = model.tokens[line];
+			const state = model.states[line];
+			if (tokens && state && !model.invalid.has(line)) batch.add(line, tokens, state);
+		}
+		while (this.now() < deadline) {
+			const line = model.invalid.first();
+			if (line === void 0) break;
+			const start = line === 0 ? INITIAL : model.states[line - 1] ?? INITIAL;
+			const { tokens, endState } = tokenizeLine(grammar, model, model.lines[line] ?? "", start);
+			const previous = model.states[line];
+			model.states[line] = endState;
+			model.tokens[line] = tokens;
+			model.invalid.delete(line);
+			if (line + 1 < model.lines.length && !previous?.equals(endState)) model.invalid.add(line + 1, line + 2);
+			batch.add(line, tokens, endState);
+		}
+		if (!batch.empty) batch.post(this.send, model.id, model.version);
+		if (model.invalid.first() === void 0 && !model.finished) {
+			model.finished = true;
+			this.send({
+				kind: "finished",
+				model: model.id,
+				version: model.version
+			}, []);
+		}
+	}
+	receive(message) {
+		switch (message.kind) {
+			case "build":
+				this.highlighter = (async () => createHighlighterCore({
+					themes: [...message.themes],
+					langs: [...message.languages],
+					engine: await engineOf(message.engine)
+				}))();
+				break;
+			case "languages":
+				this.highlighter = this.highlighter?.then((highlighter) => {
+					try {
+						highlighter.loadLanguageSync([...message.languages]);
+					} catch (error) {
+						console.warn("inkstone: the tokenization worker could not load a grammar", error);
+					}
+					return highlighter;
+				});
+				break;
+			case "open":
+				this.models.set(message.model, {
+					id: message.model,
+					language: message.language,
+					languageId: message.languageId,
+					theme: message.theme,
+					colorIds: message.colorIds,
+					version: message.version,
+					lines: [...message.lines],
+					states: [],
+					tokens: [],
+					invalid: new InvalidLines(message.lines.length),
+					requested: [],
+					finished: false
+				});
+				break;
+			case "change": {
+				const model = this.models.get(message.model);
+				if (model) {
+					for (const change of message.changes) {
+						const splice = lineSplice(change);
+						applyChange(model.lines, change);
+						applySplice(model.states, splice, () => void 0);
+						applySplice(model.tokens, splice, () => void 0);
+						model.invalid.splice(splice);
+					}
+					model.version = message.version;
+					model.finished = false;
+					model.requested.length = 0;
+				}
+				break;
+			}
+			case "retokenize":
+				this.models.get(message.model)?.requested.push({
+					start: message.start - 1,
+					end: message.end - 1
+				});
+				break;
+			case "close": this.models.delete(message.model);
+		}
+		this.wake();
+	}
+};
+//#endregion
+//#region src/tokenization/worker-entry.ts
+function startTokenizationWorker() {
+	const scope = globalThis;
+	const worker = new TokenizationWorker((message, transfer) => {
+		scope.postMessage(message, transfer);
+	});
+	scope.addEventListener("message", (event) => {
+		worker.receive(event.data);
+	});
+}
+//#endregion
+//#region src/workers/textmate.worker.js
+startTokenizationWorker();
+//#endregion
+export {};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@coldsmirk/inkstone-monaco",
-  "version": "0.28.1",
+  "version": "0.30.0",
   "description": "Framework-agnostic Monaco assembly: lazy offline host (all language workers bundled, no CDN), opt-in Simplified-Chinese UI via monaco's official NLS catalog, a streaming editor controller for AI-typed content, and the shared Shiki highlighting bridge.",
   "keywords": [
     "monaco",
@@ -40,7 +40,8 @@
   ],
   "dependencies": {
     "@shikijs/vscode-textmate": "^10.0.2",
-    "@coldsmirk/inkstone-core": "^0.28.1"
+    "shiki": "^4.4.3",
+    "@coldsmirk/inkstone-core": "^0.30.0"
   },
   "devDependencies": {
     "monaco-editor": "^0.56.0",