@docfy/plugin-shiki 0.14.0

package/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Josemar Luedke
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.N

package/README.md

@@ -0,0 +1,221 @@
+# @docfy/plugin-shiki
+
+An **opt-in** [Shiki](https://shiki.style) syntax-highlighting preset for
+Docfy, with first-class support for Ember's `.gjs`/`.gts` file types.
+
+This package is standalone: `shiki` is never a dependency of `@docfy/core` or
+`@docfy/ember`. Consumers who want syntax highlighting install this package
+and spread its return value into their own `rehypePlugins`.
+
+## Why this exists
+
+Wiring up Shiki (or `rehype-highlight`) by hand for a Docfy site normally
+means re-deriving the same handful of settings every time: which themes to
+use, how to map `gjs`/`gts`/`hbs` fence languages onto real grammars, and how
+to enable `{1,3-5}`-style line highlighting. This preset packages all of
+that up so you don't have to.
+
+**Glimmer support is the main reason this package exists.** Shiki bundles
+first-class `glimmer-js` and `glimmer-ts` TextMate grammars (scope
+`source.gts`), so a `.gts`/`.gjs` code fence gets real tokenisation —
+including inside `<template>` tags — instead of silently falling back to
+plain JavaScript (or plain text).
+
+## Installation
+
+```bash
+pnpm add @docfy/plugin-shiki
+```
+
+## Usage
+
+```ts
+import Docfy from '@docfy/core';
+import docfyShiki from '@docfy/plugin-shiki';
+
+const docfy = new Docfy({
+  rehypePlugins: [
+    // ...your other rehype plugins
+    ...docfyShiki(),
+  ],
+});
+```
+
+## Why this preset builds its own highlighter
+
+`@docfy/core` drives its rehype pipeline with `unified().runSync(...)` (see
+`packages/core/src/-private/remark.ts`), never the async `.run()`/`.process()`.
+Shiki's own `@shikijs/rehype` default export is async-only — it lazily boots a
+singleton highlighter the first time the tree transformer runs — so wiring it
+in directly makes `runSync` throw.
+
+`@docfy/ember-cli` (the classic, non-Vite build) also loads a consumer's
+`docfy.config.*` with a **synchronous** `require()`. A module that contains a
+top-level `await` anywhere in its graph is an async ES module, and Node
+throws `ERR_REQUIRE_ASYNC_MODULE` the moment such a config merely `import`s
+it — so this package cannot use a top-level `await` either, even to build its
+highlighter once at startup.
+
+To stay synchronous end-to-end, this preset:
+
+- Loads its grammars and themes via plain, **static** `import` statements
+  (see "Supported languages" and "Supported themes" below) — never a dynamic
+  `import()` or a top-level `await`. This applies just as much to any extra
+  grammar you pass via `langs` (see "Adding a language" below).
+- Builds a Shiki `HighlighterCore` via `createHighlighterCoreSync`, using the
+  pure-JS regex engine (no WASM to load). Calling `docfyShiki()` with no
+  `langs`/`langAlias` reuses one memoized highlighter across calls, built the
+  first time it's needed rather than at `import` time — a consumer that
+  imports this package but never calls `docfyShiki()` doesn't pay for it at
+  all. Passing `langs` and/or `langAlias` builds a dedicated highlighter for
+  that call instead, since `langAlias` can only be set at construction time
+  (see "Language aliases" below).
+- Uses `@shikijs/rehype/core`'s `rehypeShikiFromHighlighter`, which — given an
+  already-built highlighter and no lazy-loaded languages — returns a genuinely
+  synchronous unified transformer.
+
+## Supported languages
+
+Preloading **every** language Shiki bundles (~200 grammars, ~11.6MB of JSON)
+would cost real parse time at `import`, whether or not a given site uses most
+of them. Instead, this preset preloads a curated set covering the glimmer
+grammars that are the point of the package, plus the languages Docfy's own
+docs (and typical Ember app docs) actually fence:
+
+`glimmer-ts`, `glimmer-js`, `handlebars`, `javascript`, `typescript`, `jsx`,
+`tsx`, `json`, `css`, `scss`, `html`, `markdown`, `shellscript`, `diff`,
+`yaml`.
+
+It also registers these language aliases so fences written the way Ember
+docs are actually written resolve to the grammars above:
+
+| Fence language | Resolves to  |
+| -------------- | ------------ |
+| `gts`          | `glimmer-ts` |
+| `gjs`          | `glimmer-js` |
+| `hbs`          | `handlebars` |
+
+You can add to (or override) this table with the `langAlias` option — see
+"Adding a language" below.
+
+### Adding a language
+
+A fence whose language is not in the curated list above (for example
+` ```rust `) is, by default, **not** an error: `@shikijs/rehype` leaves any
+`<pre>` whose language isn't loaded completely untouched — no `.shiki`
+wrapper, no theme classes, no Shiki-applied highlighting — it renders as
+plain, unhighlighted fenced code, exactly as if no highlighter were
+configured for it. A docs build never fails because someone wrote a fence in
+a language this preset doesn't preload.
+
+If you actually want that language highlighted, pass its grammar via the
+`langs` option. Each entry is a statically-imported module from
+`@shikijs/langs` (or any other Shiki-compatible grammar you import
+yourself) — never a dynamic `import()`, since this package has to stay
+synchronous end-to-end (see "Why this preset builds its own highlighter"
+above):
+
+```ts
+import rust from '@shikijs/langs/rust';
+import shiki from '@docfy/plugin-shiki';
+
+export default {
+  rehypePlugins: [autolinkHeadings, ...shiki({ langs: [rust] })],
+};
+```
+
+A ` ```rust ` fence now tokenizes for real. `langs` is additive — it doesn't
+replace the curated set this preset already preloads.
+
+If the language you need is only reachable under a different fence name (for
+example your docs use ` ```rs ` rather than ` ```rust `), pair `langs` with
+`langAlias`, which is merged over this preset's own `gjs`/`gts`/`hbs` table:
+
+```ts
+shiki({ langs: [rust], langAlias: { rs: 'rust' } });
+```
+
+`langAlias` on its own (without a matching `langs` entry) cannot make a new
+language highlight — see "Language aliases" below for why, and why the two
+options work together the way they do.
+
+## Supported themes
+
+This preset statically preloads three themes: `github-light`, `github-dark`
+(the defaults), and `nord`, and — unlike languages — there is currently no
+option to add more; passing an unloaded theme name throws, because the
+underlying highlighter is built synchronously and cannot fetch a theme
+afterwards.
+
+## Overriding themes
+
+```ts
+docfyShiki({
+  themes: { light: 'github-light', dark: 'nord' },
+});
+```
+
+## Language aliases
+
+An earlier version of this package removed a `langAlias` option, because at
+the time the highlighter was built once, synchronously, at *module import*
+time — before `docfyShiki(options)` was ever called. An alias supplied at
+call time could only ever change the `data-language` attribute this preset
+writes on the rendered `<pre>`; it could never register a new alias with
+Shiki's grammar resolver, which was already fixed by then. A caller adding,
+say, `{ svelte: 'html' }` would see their fence mislabelled rather than
+actually highlighted as HTML — an option that appears to work and silently
+does not, which is worse than no option.
+
+`langAlias` is back because that constraint no longer holds: the highlighter
+is now built inside `docfyShiki()` itself (see "Adding a language" above), so
+an alias supplied there is baked into the same construction call as any
+`langs` you pass alongside it, and genuinely participates in grammar
+resolution. `langAlias` merges *over* the package's own `gjs`/`gts`/`hbs`
+table — it can override one of those three, but not remove the other two.
+
+Passing `langAlias` without a matching `langs` entry (or one of the
+languages this preset already preloads) still can't highlight anything new —
+an alias can only point at a grammar that's actually loaded. But it no
+longer looks like it worked when it didn't: a fence whose alias points at an
+unloaded grammar is left completely untouched (see "Adding a language"
+above) — no `data-language` attribute either, exactly like any other
+unsupported language — rather than being mislabelled with a `data-language`
+that implies it highlighted.
+
+## Extra transformers
+
+```ts
+import { transformerNotationDiff } from '@shikijs/transformers';
+
+docfyShiki({
+  transformers: [transformerNotationDiff()],
+});
+```
+
+`transformers` is appended after this preset's own defaults
+(`transformerMetaHighlight`, `transformerMetaWordHighlight`, and two small
+internal transformers that add the `data-language="..."` and
+`data-highlighted` attributes described above).
+
+## Line highlighting
+
+Marking specific lines in a fence, e.g.:
+
+````md
+```ts {2,4-5}
+// ...
+```
+````
+
+works because this preset wires up `transformerMetaHighlight` from
+`@shikijs/transformers`. Line highlighting is **not** a feature of Shiki (or
+of Docfy) on its own — it only works when this preset (or an equivalent
+Shiki configuration that includes `transformerMetaHighlight`) is in use.
+Highlighted lines get both a `highlighted` class and a `data-highlighted`
+attribute, so you can style them with either `.highlighted` or
+`[data-highlighted]` in your own CSS.
+
+## License
+
+This project is licensed under the [MIT License](LICENSE.md).

package/lib/index.d.ts

@@ -0,0 +1,56 @@
+import { type LanguageRegistration } from 'shiki/core';
+export interface DocfyShikiOptions {
+    /**
+     * Light and dark themes, emitted as CSS variables so the page switches
+     * themes without re-highlighting and without any runtime JavaScript.
+     *
+     * Only themes preloaded by this package (currently `github-light`,
+     * `github-dark`, and `nord`) can be used — see "Supported themes" in the
+     * README.
+     */
+    themes?: {
+        light: string;
+        dark: string;
+    };
+    /**
+     * Extra Shiki transformers, appended after the defaults.
+     */
+    transformers?: unknown[];
+    /**
+     * Extra TextMate grammars to register alongside the curated set this
+     * preset preloads (see "Supported languages" below), so a fence in a
+     * language this preset doesn't ship by default can still be highlighted.
+     *
+     * Each entry is a statically-imported grammar module, e.g.:
+     *
+     * ```ts
+     * import rust from '@shikijs/langs/rust';
+     * import shiki from '@docfy/plugin-shiki';
+     *
+     * shiki({ langs: [rust] });
+     * ```
+     *
+     * Import must stay static (never `import()`) — see the module-level
+     * comment above about why a top-level `await` anywhere in this graph
+     * breaks `@docfy/ember-cli`'s synchronous `require()` of a consumer's
+     * `docfy.config.*`.
+     */
+    langs?: LanguageRegistration[];
+    /**
+     * Extra fence-language aliases, merged OVER this preset's built-in
+     * `gjs`/`gts`/`hbs` map (so an entry here can override one of those three,
+     * but cannot remove the other two). Unlike the removed-and-reinstated
+     * option this replaces, this one is genuinely functional: passing
+     * `langs` and `langAlias` together builds a highlighter that knows about
+     * the new grammar from construction, so an alias can resolve to it.
+     *
+     * ```ts
+     * import svelte from '@shikijs/langs/svelte';
+     * import shiki from '@docfy/plugin-shiki';
+     *
+     * shiki({ langs: [svelte], langAlias: { html_svelte: 'svelte' } });
+     * ```
+     */
+    langAlias?: Record<string, string>;
+}
+export default function docfyShiki(options?: DocfyShikiOptions): unknown[];

package/lib/index.js

@@ -0,0 +1,200 @@
+import { createHighlighterCoreSync, } from 'shiki/core';
+import { createJavaScriptRegexEngine } from 'shiki/engine/javascript';
+// Static imports only. `@docfy/ember-cli` loads a consumer's `docfy.config.*`
+// with a synchronous `require()` (see `packages/ember-cli/src/get-config.ts`).
+// A top-level `await` anywhere in this module's graph turns it into an async
+// ES module, and Node throws `ERR_REQUIRE_ASYNC_MODULE` the moment such a
+// config merely `import`s this package. Static imports do not make a module
+// async, so they're the only way to preload grammars/themes and still be
+// usable from the classic (non-Vite) build.
+//
+// Each of these modules is a plain, synchronously-evaluated ES module that
+// exports the parsed grammar/theme object (or, for grammars that embed other
+// languages, an array of them) directly — nothing here is a lazy loader.
+import glimmerTs from '@shikijs/langs/glimmer-ts';
+import glimmerJs from '@shikijs/langs/glimmer-js';
+import handlebars from '@shikijs/langs/handlebars';
+import javascript from '@shikijs/langs/javascript';
+import typescript from '@shikijs/langs/typescript';
+import jsx from '@shikijs/langs/jsx';
+import tsx from '@shikijs/langs/tsx';
+import json from '@shikijs/langs/json';
+import css from '@shikijs/langs/css';
+import scss from '@shikijs/langs/scss';
+import html from '@shikijs/langs/html';
+import markdown from '@shikijs/langs/markdown';
+import shellscript from '@shikijs/langs/shellscript';
+import diff from '@shikijs/langs/diff';
+import yaml from '@shikijs/langs/yaml';
+import githubLight from '@shikijs/themes/github-light';
+import githubDark from '@shikijs/themes/github-dark';
+import nord from '@shikijs/themes/nord';
+import rehypeShikiFromHighlighter from '@shikijs/rehype/core';
+import { transformerMetaHighlight, transformerMetaWordHighlight } from '@shikijs/transformers';
+/**
+ * Shiki bundles first-class `glimmer-js` and `glimmer-ts` TextMate grammars
+ * (scope `source.gts`), so `.gjs`/`.gts` fences get real tokenisation rather
+ * than falling back to plain JavaScript. This is the preset's built-in alias
+ * table; a caller can add to it (or override individual entries) via the
+ * `langAlias` option on `docfyShiki(...)` — see `DocfyShikiOptions.langAlias`
+ * and the README's "Language aliases" section. It is merged with any
+ * caller-supplied aliases and re-baked into a fresh highlighter at
+ * `docfyShiki()` call time (see `getHighlighter` below), which is what makes
+ * a caller-supplied alias able to actually resolve to a caller-supplied
+ * `langs` grammar rather than only relabelling output.
+ */
+const DEFAULT_LANG_ALIAS = {
+    gjs: 'glimmer-js',
+    gts: 'glimmer-ts',
+    hbs: 'handlebars',
+};
+const DEFAULT_THEMES = { light: 'github-light', dark: 'github-dark' };
+/**
+ * The curated set of languages this preset preloads by default. This is
+ * deliberately NOT "every language Shiki bundles" (~200 grammars, ~11.6MB of
+ * JSON) — preloading everything would cost real parse time for every
+ * consumer, whether or not they use most of those languages. This list
+ * covers the glimmer grammars that are the point of this package, plus the
+ * languages Docfy's own docs (and typical Ember app docs) actually fence:
+ * TypeScript/JavaScript and their JSX variants, Handlebars, JSON, CSS/SCSS,
+ * HTML, Markdown, shell, diff, and YAML.
+ *
+ * A fence in a language outside this set (and not covered by a `langs`
+ * option passed to `docfyShiki(...)`) is NOT an error: `rehypeShiki` (see
+ * below) leaves any `<pre>` whose language isn't loaded untouched — no
+ * `.shiki` wrapper, no theme, no crash — so a docs build never dies because
+ * someone wrote a ```rust fence. See the "Unsupported languages" section of
+ * the README, and `DocfyShikiOptions.langs` for how to add it instead.
+ */
+const DEFAULT_LANGS = [
+    glimmerTs,
+    glimmerJs,
+    handlebars,
+    javascript,
+    typescript,
+    jsx,
+    tsx,
+    json,
+    css,
+    scss,
+    html,
+    markdown,
+    shellscript,
+    diff,
+    yaml,
+];
+/**
+ * Builds a `HighlighterCore`. `langAlias` is construction-time-only in Shiki
+ * (it cannot be changed on an already-built highlighter), which is exactly
+ * why `docfyShiki()` builds a fresh highlighter per call rather than reusing
+ * a single module-level instance whenever a caller passes `langs` and/or
+ * `langAlias` — see `getHighlighter` below.
+ */
+function buildHighlighter(extraLangs, langAlias) {
+    return createHighlighterCoreSync({
+        langs: [...DEFAULT_LANGS, ...extraLangs],
+        themes: [githubLight, githubDark, nord],
+        // Shiki mutates the `langAlias` object it's given (accumulating its own
+        // built-in aliases into it), so pass a copy rather than the caller's map.
+        langAlias: Object.assign({}, langAlias),
+        // The pure-JS regex engine avoids loading a WASM binary. It cannot
+        // translate every Oniguruma pattern the way the WASM-backed `oniguruma`
+        // engine can, so `forgiving: true` is required — without it, an
+        // untranslatable pattern in any preloaded grammar throws at construction
+        // time, crashing every consumer's build rather than degrading the one
+        // language affected.
+        engine: createJavaScriptRegexEngine({ forgiving: true }),
+    });
+}
+// Memoized highlighter for the common case: a call to `docfyShiki()` with no
+// `langs`/`langAlias`. Building the highlighter is moved from module load
+// time into `docfyShiki()` (rather than the previous module-level
+// `const highlighter = ...`) so that a consumer who imports this package but
+// never calls `docfyShiki()` doesn't pay the parse cost of ~15 grammars for
+// nothing; this cache keeps the *common* path — calling it once, with no
+// extra options — down to a single construction, same as before.
+let defaultHighlighter;
+function getHighlighter(options) {
+    var _a;
+    const extraLangs = (_a = options.langs) !== null && _a !== void 0 ? _a : [];
+    const langAlias = Object.assign(Object.assign({}, DEFAULT_LANG_ALIAS), options.langAlias);
+    if (extraLangs.length === 0 && !options.langAlias) {
+        if (!defaultHighlighter) {
+            defaultHighlighter = buildHighlighter([], langAlias);
+        }
+        return { highlighter: defaultHighlighter, langAlias };
+    }
+    return { highlighter: buildHighlighter(extraLangs, langAlias), langAlias };
+}
+/**
+ * Records the language actually requested for a fence (after applying this
+ * preset's own alias table) as a `data-language` attribute on the `<pre>`.
+ * This makes the alias resolution independently observable in the rendered
+ * HTML: a `gts` fence that silently fell back to plain text would never
+ * reach this transformer with a real grammar match. It is intentionally a
+ * secondary signal in the test suite — the primary evidence that a language
+ * actually tokenised is Shiki's own per-token `<span style="...">` output,
+ * which this attribute cannot fake.
+ */
+function languageAttributeTransformer(langAlias) {
+    return {
+        name: 'docfy-shiki:language-attribute',
+        pre(node) {
+            var _a;
+            const lang = this.options.lang;
+            node.properties['data-language'] = (_a = langAlias[lang]) !== null && _a !== void 0 ? _a : lang;
+            return node;
+        },
+    };
+}
+/**
+ * `transformerMetaHighlight` (from `@shikijs/transformers`) marks lines named
+ * in a fence's `{1,3-5}` meta by adding a `highlighted` class to the line —
+ * it does not add a `data-highlighted` attribute. Consumers that prefer to
+ * style highlighted lines via `[data-highlighted]` (rather than matching a
+ * class name that could collide with their own CSS) need that attribute too,
+ * so this transformer mirrors the class onto a boolean data attribute. It
+ * must run after `transformerMetaHighlight` in the `transformers` array so
+ * the class has already been applied by the time this sees the line.
+ */
+function dataHighlightedAttributeTransformer() {
+    return {
+        name: 'docfy-shiki:data-highlighted',
+        line(node) {
+            const classes = node.properties.class;
+            const classList = Array.isArray(classes)
+                ? classes
+                : typeof classes === 'string'
+                    ? classes.split(/\s+/)
+                    : [];
+            if (classList.includes('highlighted')) {
+                node.properties['data-highlighted'] = '';
+            }
+            return node;
+        },
+    };
+}
+export default function docfyShiki(options = {}) {
+    const { highlighter, langAlias } = getHighlighter(options);
+    function shikiRehypePlugin() {
+        var _a, _b;
+        const transformers = [
+            transformerMetaHighlight(),
+            transformerMetaWordHighlight(),
+            languageAttributeTransformer(langAlias),
+            dataHighlightedAttributeTransformer(),
+            ...((_a = options.transformers) !== null && _a !== void 0 ? _a : []),
+        ];
+        return rehypeShikiFromHighlighter(highlighter, {
+            themes: (_b = options.themes) !== null && _b !== void 0 ? _b : DEFAULT_THEMES,
+            defaultColor: false,
+            transformers,
+            // A fence whose language was never preloaded (and isn't a "special"
+            // pseudo-language like `text`/`ansi`) is left as plain, unhighlighted
+            // code rather than throwing. `lazy` defaults to `false`, so there is no
+            // attempt to fetch a grammar on demand either — this preset is fully
+            // synchronous end to end.
+        });
+    }
+    return [shikiRehypePlugin];
+}

package/package.json

@@ -0,0 +1,59 @@
+{
+  "name": "@docfy/plugin-shiki",
+  "version": "0.14.0",
+  "private": false,
+  "description": "An opt-in Shiki syntax highlighting preset for Docfy",
+  "repository": "https://github.com/josemarluedke/docfy",
+  "license": "MIT",
+  "author": "Josemar Luedke <josemarluedke@gmail.com>",
+  "main": "lib/index.js",
+  "types": "lib/index.d.ts",
+  "files": [
+    "lib"
+  ],
+  "scripts": {
+    "compile": "tsc",
+    "prepare": "tsc",
+    "test": "vitest run",
+    "lint": "eslint . --cache",
+    "lint:fix": "eslint . --fix",
+    "format": "prettier . --cache --write",
+    "format:check": "prettier . --cache --check"
+  },
+  "dependencies": {
+    "@shikijs/langs": "^3.0.0",
+    "@shikijs/rehype": "^3.0.0",
+    "@shikijs/themes": "^3.0.0",
+    "@shikijs/transformers": "^3.0.0",
+    "shiki": "^3.0.0"
+  },
+  "devDependencies": {
+    "@docfy/core": "^0.13.1",
+    "@eslint/js": "^10.0.1",
+    "@types/mdast": "^4.0.4",
+    "@types/node": "^22.10.5",
+    "eslint": "^10.9.0",
+    "eslint-config-prettier": "^10.1.8",
+    "eslint-plugin-n": "^18.3.0",
+    "globals": "^17.11.0",
+    "prettier": "^3.9.6",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.8.3",
+    "typescript-eslint": "^8.67.0",
+    "vitest": "^4.1.11"
+  },
+  "engines": {
+    "node": ">=22.22.2"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./lib/index.d.ts",
+      "default": "./lib/index.js"
+    }
+  },
+  "gitHead": "d1cd6b90537801c49101a40d429a282aaad39aa7"
+}