@coldsmirk/inkstone-monaco 0.19.2 → 0.20.0

package/README.md

@@ -15,7 +15,8 @@ Part of [inkstone](https://github.com/coldsmirk/inkstone). For a drop-in React c
 - **Embedded-editor fix** — works around [microsoft/monaco-editor#5177](https://github.com/microsoft/monaco-editor/issues/5177) (find-widget button tooltips landing on top of the buttons they describe) with a `width: max-content` hover-layer style tagged `data-inkstone="monaco-hover-geometry"`; deleted the moment upstream fixes it. The tooltip layer positions against the editor's container — give that container `position: relative`.
 - **`StreamingEditorController`** — stream generated text into an editor without routing every token through app state: `begin()` / `append(chunk)` / `end()` / `discard()`, `attach(editor)` / `detach(editor)` with buffered replay for late-mounting editors (and for a model swapped onto the attached editor mid-stream, including after the final chunk), and `isStreaming()` for the onChange guard. The stream locks every editor it touches read-only and `end()` / `discard()` restore each one's own setting — by handle, so an editor detached mid-stream is restored too; `discard()` puts every displaced document back into the model it came from, and `end()` does the same for every model except the one keeping the streamed text. Full-buffer writes and restorations preserve each model's UTF-8 BOM. Editors and models the app disposed mid-stream are skipped, never touched — teardown can race the stream without wedging it.
 - **`replaceInEditor(editor, old, new, all)`** — exact find-and-replace on an editor handle via `executeEdits`, preserving undo/cursor/folds; returns whether the edit landed (`false` = no editor / empty search / no match / a read-only editor refused it — e.g. one locked by a stream in progress), so a tool-call handler knows to fall back to a state write.
-- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module, once per distinct module: registers the canonical grammar names as Monaco languages, applies `@shikijs/monaco`'s tokenizer/themes, and repairs its `setTheme` patch so non-Shiki theme ids (vs/vs-dark, caller-defined) keep Monaco's original path. A failed registration is restored and retried on the next call.
+- **`registerShikiHighlighting(monaco, highlighter)`** — wire `@coldsmirk/inkstone-core`'s shared Shiki highlighter onto a monaco module, once per distinct module: defines the highlighter's themes as Monaco themes, registers each loaded grammar's canonical name as a Monaco language where Monaco has none and installs a Shiki tokens provider for it, and patches `setTheme` so the Shiki theme ids route through the highlighter while non-Shiki ids (vs/vs-dark, caller-defined) keep Monaco's original path. Called again on a bridged module, it installs the grammars loaded since. A failed registration is restored and retried on the next call.
+- **`registerShikiLanguage(monaco, highlighter, language)`** — the lazy path: an app that loads grammars as files are opened calls this after each `highlighter.loadLanguage(...)` and before a model names the language. Registers the language where Monaco lacks it (`toml`, `shellscript`), installs its tokens provider once under the grammar's canonical name (an alias like `ts` collapses), and leaves the theme the editor shows alone — the one thing re-running `@shikijs/monaco`'s `shikiToMonaco` cannot promise, which is why the bridge is inkstone's own. Throws for a grammar the highlighter has not loaded.
 - **`installDesignTokens()`** — defines the VS Code design tokens monaco-editor 0.56's chrome reads and the standalone editor never does ([microsoft/monaco-editor#5408](https://github.com/microsoft/monaco-editor/issues/5408)): `--vscode-cornerRadius-{small,medium,large,xLarge}` and `--vscode-shadow-{md,lg,xl}`, on `.monaco-editor` (the context menu's shadow root and a host's `overflowWidgetsDomNode` inherit them). Without them the context menu, completion, hover, and find widgets ship square and shadowless. Each resolves through the host's `--inkstone-radius-*` / `--inkstone-shadow-*` token first (see `@coldsmirk/inkstone-core`), then a fallback scale in the spirit of 0.55's look. It also keeps an input box's focus ring single: the standalone editor outlines both the `.monaco-inputbox` container (its `synthetic-focus` ring, which follows the box's radius) and the inner `<input>` (square, 1px inside), and the inner ring's corners poke out as soon as the box has corners — the container's ring is the one ring. `ensureMonacoHost` installs it; idempotent.
 - **`DEFAULT_MONACO_OPTIONS`** — the inkstone construction-option baseline (no minimap, 14px/1.6 type on the shared `--inkstone-font-family-code` token, embedded-friendly find widget, tuned completion, thin scrollbars). Spread it to extend: `{ ...DEFAULT_MONACO_OPTIONS, wordWrap: "on" }`.
 - **`reconcileModelValue` / `modelValueMatches`** — the controlled-value reconcile: diff an external value against a live model (normalized to the model's EOL) and apply one minimal `applyEdits` replace that never lands on the undo stack or moves a caret outside the changed region.

package/dist/index.d.ts

@@ -248,12 +248,32 @@ declare function replaceInEditor(target: editor.ICodeEditor | null, search: stri
 //#endregion
 //#region src/shiki-bridge.d.ts
 /**
- * Register the shared Shiki highlighter's languages, themes, and tokenizer on a monaco module —
- * once per distinct module, whichever caller gets there first. Language services are
- * unaffected: Shiki replaces syntax highlighting only. The module is recorded only after every
- * registration succeeds, so a transient failure can retry on a later call.
+ * Register the shared Shiki highlighter's languages, themes, and tokenizers on a monaco module —
+ * once per distinct module, whichever caller gets there first — and, on a module already
+ * bridged, install the tokenizer of any grammar loaded since. Language services are unaffected:
+ * Shiki replaces syntax highlighting only. The module is recorded only after every registration
+ * succeeds, so a transient failure can retry on a later call.
+ *
+ * Monaco's own `setTheme` is patched to route the Shiki theme ids through the highlighter, so
+ * the tokenizers paint in the theme the editor shows; Monaco's built-in themes (vs/vs-dark) and
+ * caller-defined themes keep Monaco's original path. `create` is patched so its `theme` option
+ * reaches that patch. The shape is `@shikijs/monaco`'s, kept here because that package
+ * installs tokenizers only for the grammars loaded at the moment it is called, and re-calling
+ * it for a grammar loaded later resets the editor's theme to the first one.
+ *
+ * A grammar loaded into the highlighter *after* this call is not tokenized until it is
+ * registered — pass it to {@link registerShikiLanguage} (or call this again).
  */
 declare function registerShikiHighlighting(monaco: MonacoModule, highlighter: HighlighterCore): void;
+/**
+ * Install the tokenizer of one grammar the highlighter has loaded since the module was bridged
+ * — the lazy path: an app that loads grammars as files are opened registers each here, after
+ * its `loadLanguage`, and before a model names it. `language` may be an alias (`ts`); it
+ * collapses to the grammar's canonical name. Bridges the module first where nothing has, and
+ * is nothing for a grammar already registered. Throws when the highlighter has not loaded the
+ * grammar: there is nothing to tokenize with.
+ */
+declare function registerShikiLanguage(monaco: MonacoModule, highlighter: HighlighterCore, language: string): void;
 //#endregion
 //#region src/streaming.d.ts
 /**
@@ -377,4 +397,4 @@ declare class StreamingEditorController {
   discard(): void;
 }
 //#endregion
-export { type AutoImportCompletionOptions, DEFAULT_MONACO_OPTIONS, type HoverDeadBandTargetEditor, type JsxTagClosingTargetEditor, type JsxTagClosingTargetModel, type KeybindingTargetEditor, type MonacoBuiltInLanguage, type MonacoHostOptions, type MonacoLanguage, type MonacoModule, type MonacoUiLocale, type StreamTargetEditor, StreamingEditorController, type SwallowedKeybindings, disabledKeybindings, ensureMonacoHost, installAutoImportCompletions, installDesignTokens, installDisabledKeybindings, installHoverDeadBandFix, installJsxTagClosing, modelValueMatches, monacoLanguages, reconcileModelValue, registerShikiHighlighting, replaceInEditor };
+export { type AutoImportCompletionOptions, DEFAULT_MONACO_OPTIONS, type HoverDeadBandTargetEditor, type JsxTagClosingTargetEditor, type JsxTagClosingTargetModel, type KeybindingTargetEditor, type MonacoBuiltInLanguage, type MonacoHostOptions, type MonacoLanguage, type MonacoModule, type MonacoUiLocale, type StreamTargetEditor, StreamingEditorController, type SwallowedKeybindings, disabledKeybindings, ensureMonacoHost, installAutoImportCompletions, installDesignTokens, installDisabledKeybindings, installHoverDeadBandFix, installJsxTagClosing, modelValueMatches, monacoLanguages, reconcileModelValue, registerShikiHighlighting, registerShikiLanguage, replaceInEditor };

package/dist/index.js

@@ -1,5 +1,5 @@
 import { CODE_FONT_FAMILY, inkstoneRadius, inkstoneShadow } from "@coldsmirk/inkstone-core";
-import { shikiToMonaco } from "@shikijs/monaco";
+import { EncodedTokenMetadata, INITIAL } from "@shikijs/vscode-textmate";
 //#region src/auto-imports.ts
 const MISSING_LOOKUP_REJECTION$1 = "Missing requestHandler or method: getAutoImportCompletionsAtPosition";
 const MODE_BOOTING_REJECTIONS$1 = /* @__PURE__ */ new Set(["TypeScript not registered!", "JavaScript not registered!"]);
@@ -546,36 +546,195 @@ function replaceInEditor(target, search, replaceWith, all) {
 //#endregion
 //#region src/shiki-bridge.ts
 const SHIKI_REGISTRY_KEY = Symbol.for("coldsmirk.inkstone.monaco.shikiRegistry");
-function registeredModules() {
+function bridges() {
 	const host = globalThis;
 	const existing = host[SHIKI_REGISTRY_KEY];
 	if (existing) return existing;
-	const created = /* @__PURE__ */ new WeakSet();
+	const created = /* @__PURE__ */ new WeakMap();
 	host[SHIKI_REGISTRY_KEY] = created;
 	return created;
 }
+const TOKENIZE_MAX_LINE_LENGTH = 2e4;
+const TOKENIZE_TIME_LIMIT_MS = 500;
+const ITALIC = 1;
+const BOLD = 2;
+const UNDERLINE = 4;
+const STRIKETHROUGH = 8;
+const RE_FONT_STYLE_SPLIT = /[\s,]+/;
+const FONT_STYLES = [
+	"italic",
+	"bold",
+	"underline",
+	"strikethrough"
+];
+const FONT_STYLE_ALIASES = { "line-through": "strikethrough" };
+function normalizeColor(color) {
+	if (!color) return;
+	let normalized = (color.startsWith("#") ? color.slice(1) : color).toLowerCase();
+	if (normalized.length === 3 || normalized.length === 4) normalized = [...normalized].map((digit) => digit + digit).join("");
+	return normalized;
+}
+function normalizeFontStyleString(fontStyle) {
+	if (!fontStyle) return "";
+	const styles = new Set(fontStyle.split(RE_FONT_STYLE_SPLIT).map((style) => style.trim().toLowerCase()).map((style) => FONT_STYLE_ALIASES[style] ?? style).filter(Boolean));
+	return FONT_STYLES.filter((style) => styles.has(style)).join(" ");
+}
+function normalizeFontStyleBits(fontStyle) {
+	if (fontStyle <= 0) return "";
+	const styles = [];
+	if (fontStyle & ITALIC) styles.push("italic");
+	if (fontStyle & BOLD) styles.push("bold");
+	if (fontStyle & UNDERLINE) styles.push("underline");
+	if (fontStyle & STRIKETHROUGH) styles.push("strikethrough");
+	return styles.join(" ");
+}
+function colorStyleKey(color, fontStyle) {
+	return fontStyle ? `${color}|${fontStyle}` : color;
+}
+function monacoThemeOf(theme) {
+	const rules = [];
+	const tokenSettings = theme.settings ?? theme.tokenColors ?? [];
+	for (const { scope, settings } of tokenSettings) {
+		const { foreground, background, fontStyle } = settings ?? {};
+		if (!foreground && !background && !fontStyle) continue;
+		const scopes = Array.isArray(scope) ? scope : scope ? [scope] : [];
+		const normalizedFontStyle = normalizeFontStyleString(fontStyle);
+		const normalizedForeground = normalizeColor(foreground);
+		const normalizedBackground = normalizeColor(background);
+		for (const token of scopes) rules.push({
+			token,
+			foreground: normalizedForeground,
+			background: normalizedBackground,
+			fontStyle: normalizedFontStyle
+		});
+	}
+	const colors = Object.fromEntries(Object.entries(theme.colors ?? {}).map(([key, value]) => [key, `#${normalizeColor(value) ?? ""}`]));
+	return {
+		base: theme.type === "light" ? "vs" : "vs-dark",
+		inherit: false,
+		colors,
+		rules
+	};
+}
+function adoptTheme(bridge, themeName) {
+	const { colorMap } = bridge.highlighter.setTheme(themeName);
+	const theme = bridge.themes.get(themeName);
+	bridge.colorMap.length = colorMap.length;
+	for (const [index, color] of colorMap.entries()) bridge.colorMap[index] = color;
+	bridge.scopeOf.clear();
+	const rules = theme?.rules ?? [];
+	for (const rule of rules) {
+		const color = normalizeColor(rule.foreground);
+		if (!color) continue;
+		const key = colorStyleKey(color, normalizeFontStyleString(rule.fontStyle));
+		if (!bridge.scopeOf.has(key)) bridge.scopeOf.set(key, rule.token);
+	}
+}
+var TokenizerState = class TokenizerState {
+	ruleStack;
+	constructor(ruleStack) {
+		this.ruleStack = ruleStack;
+	}
+	clone() {
+		return new TokenizerState(this.ruleStack);
+	}
+	equals(other) {
+		return other instanceof TokenizerState && other.ruleStack === this.ruleStack;
+	}
+};
+function tokensProvider(bridge, lang) {
+	return {
+		getInitialState() {
+			return new TokenizerState(INITIAL);
+		},
+		tokenize(line, state) {
+			if (line.length >= TOKENIZE_MAX_LINE_LENGTH) return {
+				endState: state,
+				tokens: [{
+					startIndex: 0,
+					scopes: ""
+				}]
+			};
+			const ruleStack = state instanceof TokenizerState ? state.ruleStack : INITIAL;
+			const result = bridge.highlighter.getLanguage(lang).tokenizeLine2(line, ruleStack, TOKENIZE_TIME_LIMIT_MS);
+			if (result.stoppedEarly) console.warn(`inkstone: time limit reached when tokenizing a line: ${line.slice(0, 100)}`);
+			const tokens = [];
+			for (let index = 0; index < result.tokens.length; index += 2) {
+				const startIndex = result.tokens[index] ?? 0;
+				const metadata = result.tokens[index + 1] ?? 0;
+				const color = normalizeColor(bridge.colorMap[EncodedTokenMetadata.getForeground(metadata)]);
+				const fontStyle = normalizeFontStyleBits(EncodedTokenMetadata.getFontStyle(metadata));
+				const scopes = color ? bridge.scopeOf.get(colorStyleKey(color, fontStyle)) ?? "" : "";
+				tokens.push({
+					startIndex,
+					scopes
+				});
+			}
+			return {
+				endState: new TokenizerState(result.ruleStack),
+				tokens
+			};
+		}
+	};
+}
+function install(monaco, bridge, canonical) {
+	if (bridge.registered.has(canonical)) return;
+	if (monaco.languages.getLanguages().every((language) => language.id !== canonical)) monaco.languages.register({ id: canonical });
+	monaco.languages.setTokensProvider(canonical, tokensProvider(bridge, canonical));
+	bridge.registered.add(canonical);
+}
+function canonicalLanguages(highlighter) {
+	return new Set(highlighter.getLoadedLanguages().map((id) => highlighter.getLanguage(id).name));
+}
 function registerShikiHighlighting(monaco, highlighter) {
-	const registry = registeredModules();
-	if (registry.has(monaco)) return;
-	const canonical = new Set(highlighter.getLoadedLanguages().map((id) => highlighter.getLanguage(id).name));
-	const registered = new Set(monaco.languages.getLanguages().map((registeredLanguage) => registeredLanguage.id));
-	for (const lang of canonical) if (!registered.has(lang)) monaco.languages.register({ id: lang });
-	const shikiThemes = new Set(highlighter.getLoadedThemes());
+	const registry = bridges();
+	const bridged = registry.get(monaco);
+	if (bridged) {
+		for (const canonical of canonicalLanguages(highlighter)) install(monaco, bridged, canonical);
+		return;
+	}
+	const bridge = {
+		highlighter,
+		themes: /* @__PURE__ */ new Map(),
+		colorMap: [],
+		scopeOf: /* @__PURE__ */ new Map(),
+		registered: /* @__PURE__ */ new Set()
+	};
 	const originalCreate = monaco.editor.create;
 	const originalSetTheme = monaco.editor.setTheme;
 	try {
-		shikiToMonaco(highlighter, monaco);
-		const setShikiTheme = monaco.editor.setTheme;
+		for (const themeId of highlighter.getLoadedThemes()) {
+			const theme = monacoThemeOf(highlighter.getTheme(themeId));
+			bridge.themes.set(themeId, theme);
+			monaco.editor.defineTheme(themeId, theme);
+		}
 		monaco.editor.setTheme = (themeName) => {
-			if (shikiThemes.has(themeName)) setShikiTheme.call(monaco.editor, themeName);
-			else originalSetTheme.call(monaco.editor, themeName);
+			if (bridge.themes.has(themeName)) adoptTheme(bridge, themeName);
+			originalSetTheme.call(monaco.editor, themeName);
 		};
+		monaco.editor.create = (element, options, override) => {
+			if (options?.theme) monaco.editor.setTheme(options.theme);
+			return originalCreate.call(monaco.editor, element, options, override);
+		};
+		const [firstTheme] = highlighter.getLoadedThemes();
+		if (firstTheme !== void 0) monaco.editor.setTheme(firstTheme);
+		for (const canonical of canonicalLanguages(highlighter)) install(monaco, bridge, canonical);
 	} catch (error) {
 		monaco.editor.create = originalCreate;
 		monaco.editor.setTheme = originalSetTheme;
 		throw error;
 	}
-	registry.add(monaco);
+	registry.set(monaco, bridge);
+}
+function registerShikiLanguage(monaco, highlighter, language) {
+	const registry = bridges();
+	let bridge = registry.get(monaco);
+	if (!bridge) {
+		registerShikiHighlighting(monaco, highlighter);
+		bridge = registry.get(monaco);
+	}
+	if (!bridge) throw new Error("inkstone: the Shiki bridge is not installed on this monaco module");
+	install(monaco, bridge, highlighter.getLanguage(language).name);
 }
 //#endregion
 //#region src/streaming.ts
@@ -687,4 +846,4 @@ var StreamingEditorController = class {
 	}
 };
 //#endregion
-export { DEFAULT_MONACO_OPTIONS, StreamingEditorController, disabledKeybindings, ensureMonacoHost, installAutoImportCompletions, installDesignTokens, installDisabledKeybindings, installHoverDeadBandFix, installJsxTagClosing, modelValueMatches, monacoLanguages, reconcileModelValue, registerShikiHighlighting, replaceInEditor };
+export { DEFAULT_MONACO_OPTIONS, StreamingEditorController, disabledKeybindings, ensureMonacoHost, installAutoImportCompletions, installDesignTokens, installDisabledKeybindings, installHoverDeadBandFix, installJsxTagClosing, modelValueMatches, monacoLanguages, reconcileModelValue, registerShikiHighlighting, registerShikiLanguage, replaceInEditor };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@coldsmirk/inkstone-monaco",
-  "version": "0.19.2",
+  "version": "0.20.0",
   "description": "Framework-agnostic Monaco assembly: lazy offline host (all language workers bundled, no CDN), opt-in Simplified-Chinese UI via monaco's official NLS catalog, a streaming editor controller for AI-typed content, and the shared Shiki highlighting bridge.",
   "keywords": [
     "monaco",
@@ -39,8 +39,8 @@
     "dist"
   ],
   "dependencies": {
-    "@shikijs/monaco": "^4.4.3",
-    "@coldsmirk/inkstone-core": "^0.19.2"
+    "@shikijs/vscode-textmate": "^10.0.2",
+    "@coldsmirk/inkstone-core": "^0.20.0"
   },
   "devDependencies": {
     "monaco-editor": "^0.56.0",