@nvl/tree-sitter-sveltex 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 N. V. Lang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,83 @@
+# tree-sitter-sveltex
+
+> [!WARNING]
+> **This package is in alpha.** It is brand new and under active development.
+> Its API, behaviour, and configuration may change at any time, and breaking
+> changes should be expected before version `1.0.0`.
+
+A [tree-sitter] grammar for **SvelTeX** (`.sveltex`) documents.
+
+A `.sveltex` file is a [Svelte](https://svelte.dev) component whose markup is
+written in Markdown, with embedded LaTeX/math, fenced/inline code,
+YAML/TOML/JSON frontmatter and SvelTeX "verbatim" environments. See the
+[`@nvl/sveltex`](https://sveltex.dev) documentation for the language itself.
+
+## Design
+
+This grammar deliberately **does not reimplement Markdown, Svelte or LaTeX**.
+It parses only the `.sveltex` *top-level* structure — the constructs a plain
+Markdown grammar would mis-tokenise — and delegates every embedded language to
+an existing grammar through [tree-sitter injections](queries/injections.scm):
+
+| `.sveltex` construct                         | Grammar node                                  | Injected language |
+| -------------------------------------------- | ---------------------------------------------- | ----------------- |
+| YAML/TOML/JSON frontmatter                   | `frontmatter` / `frontmatter_content`          | `yaml`/`toml`/`json` |
+| Markdown prose (everything not below)        | `markdown_chunk`                               | `markdown` (combined) |
+| Inline/display math `$…$`, `$$…$$`, `\(…\)`, `\[…\]` | `inline_math` / `display_math`         | `latex`           |
+| `<tex>` / `<latex>` / `<tikz>` environments  | `verbatim_environment` → `tex_verbatim_body`   | `latex`           |
+| `<verb>` / `<verbatim>` environments         | `verbatim_environment` → `plain_verbatim_body` | *(none — opaque)* |
+
+The `markdown` grammar that each `markdown_chunk` is delegated to in turn
+injects `markdown_inline`, the fenced-code languages, and `html`/`svelte` — so
+Svelte `<script>` blocks, `{#if}`/`{#each}`/… logic blocks and `{expr}`
+mustache tags are handled downstream, exactly as in a plain `.svelte`/`.md`
+setup.
+
+The split into `frontmatter` / `markdown` / `math` / `verbatim` mirrors the
+`RegionKind`s the SvelTeX language server computes in
+`packages/sveltex-language-server/src/core/regions.ts`.
+
+### External scanner
+
+`src/scanner.c` resolves the constructs an LR grammar cannot express:
+
+- paired `$` / `$$` math fences (the same token opens and closes them);
+- the matching `</tag>` of a verbatim environment, whose body is arbitrary
+  text spanning many lines;
+- the `---` / `+++` fences and body of frontmatter;
+- maximal opaque Markdown runs that stop right before the next
+  `.sveltex`-special token.
+
+Crucially, the Markdown-chunk scanner **skips over fenced code blocks, inline
+code spans, `<script>`/`<style>` blocks and `{ … }` mustache tags**, so a `$`
+inside any of those (e.g. a `$state` rune, or `import x from '$lib/…'`) is
+*not* mistaken for a math delimiter — matching SvelTeX's own escaper.
+
+The scanner keeps no state between tokens, which makes it correct under
+tree-sitter's speculative parsing.
+
+## Layout
+
+```
+grammar.js              the grammar definition
+src/scanner.c           the external scanner
+src/parser.c            the generated parser (committed; run `tree-sitter
+                        generate` to regenerate)
+queries/highlights.scm  syntax highlighting for the structural delimiters
+queries/injections.scm  delegation to the embedded-language grammars
+queries/folds.scm       foldable regions
+test/corpus/            tree-sitter test corpus
+```
+
+## Development
+
+```sh
+npm install            # installs tree-sitter-cli
+npx tree-sitter generate
+npx tree-sitter test
+npx tree-sitter parse path/to/file.sveltex
+```
+
+## License
+
+MIT

package/binding.gyp

@@ -0,0 +1,30 @@
+{
+  "targets": [
+    {
+      "target_name": "tree_sitter_sveltex_binding",
+      "dependencies": [
+        "<!(node -p \"require('node-addon-api').targets\"):node_addon_api_except",
+      ],
+      "include_dirs": [
+        "src",
+      ],
+      "sources": [
+        "bindings/node/binding.cc",
+        "src/parser.c",
+        "src/scanner.c",
+      ],
+      "conditions": [
+        ["OS!='win'", {
+          "cflags_c": [
+            "-std=c11",
+          ],
+        }, {
+          "cflags_c": [
+            "/std:c11",
+            "/utf-8",
+          ],
+        }],
+      ],
+    }
+  ]
+}

package/bindings/node/binding.cc

@@ -0,0 +1,19 @@
+#include <napi.h>
+
+typedef struct TSLanguage TSLanguage;
+
+extern "C" TSLanguage *tree_sitter_sveltex();
+
+// "tree-sitter", "language" hashed with BLAKE2
+const napi_type_tag LANGUAGE_TYPE_TAG = {
+    0x8AF2E5212AD58ABF, 0xD5006CAD83ABBA16};
+
+Napi::Object Init(Napi::Env env, Napi::Object exports) {
+  exports["name"] = Napi::String::New(env, "sveltex");
+  auto language = Napi::External<TSLanguage>::New(env, tree_sitter_sveltex());
+  language.TypeTag(&LANGUAGE_TYPE_TAG);
+  exports["language"] = language;
+  return exports;
+}
+
+NODE_API_MODULE(tree_sitter_sveltex_binding, Init)

package/bindings/node/index.js

@@ -0,0 +1,13 @@
+const root = require("path").join(__dirname, "..", "..");
+
+const binding = require("node-gyp-build")(root);
+
+try {
+  module.exports = require("../../prebuilds/index.js");
+} catch (_) {
+  module.exports = binding;
+}
+
+try {
+  module.exports.nodeTypeInfo = require("../../src/node-types.json");
+} catch (_) {}

package/grammar.js

@@ -0,0 +1,264 @@
+/**
+ * @file Tree-sitter grammar for SvelTeX (`.sveltex`) documents.
+ * @author N. V. Lang
+ * @license MIT
+ *
+ * A `.sveltex` file is a Svelte component whose markup is written in Markdown,
+ * with embedded LaTeX/math, fenced/inline code, YAML/TOML/JSON frontmatter and
+ * SvelTeX "verbatim" environments (`<tex>`, `<verbatim>`, ...).
+ *
+ * This grammar deliberately does NOT reimplement Markdown, Svelte or LaTeX.
+ * Instead it parses only the `.sveltex` top-level structure — the constructs
+ * the plain Markdown grammar would mis-tokenise — and leaves everything else as
+ * opaque spans. `queries/injections.scm` then delegates each span to an
+ * existing grammar:
+ *
+ *   - `markdown` / `markdown_inline` for prose (which in turn injects Svelte
+ *     and fenced-code languages),
+ *   - `latex` for math and TeX verbatim environments,
+ *   - `yaml` / `toml` / `json` for frontmatter.
+ *
+ * The split into `frontmatter`, `verbatim_environment`, `math` and
+ * `markdown_chunk` mirrors the `RegionKind`s the SvelTeX language server
+ * computes in `packages/sveltex-language-server/src/core/regions.ts`.
+ *
+ * Everything that needs to recognise a *closing* delimiter whose body may hold
+ * arbitrary text (paired `$`/`$$` fences, the matching `</tag>` of a verbatim
+ * environment, the `---`/`+++` end of frontmatter, and opaque Markdown runs)
+ * is handled by the external scanner in `src/scanner.c`.
+ */
+
+/* eslint-disable no-undef */
+
+// Verbatim environment tag names whose body is LaTeX. SvelTeX's default
+// `verbatim` config registers `tex`, `latex` and `tikz` as TeX environments
+// (see `defaultConfigSnapshot()` in the language server's `config.ts`).
+const TEX_VERBATIM_TAGS = ['tex', 'latex', 'tikz', 'TeX', 'LaTeX', 'TikZ'];
+
+// Verbatim environment tag names whose body is treated as opaque/escaped text
+// (no embedded language). SvelTeX's defaults register `verb` and `verbatim`.
+const PLAIN_VERBATIM_TAGS = ['verb', 'verbatim', 'Verb', 'Verbatim'];
+
+module.exports = grammar({
+    name: 'sveltex',
+
+    // The external scanner resolves the constructs an LR grammar cannot:
+    // frontmatter fences and body, paired `$`/`$$` math fences, the matching
+    // close tag of a verbatim environment, and opaque Markdown runs that stop
+    // right before the next `.sveltex`-special token.
+    externals: ($) => [
+        $._frontmatter_start, // `---` / `+++` opening a frontmatter block
+        $._frontmatter_end, // `---` / `+++` closing a frontmatter block
+        $._frontmatter_body, // the lines between the frontmatter fences
+        $._verbatim_tex_content, // body of a <tex>/<latex>/<tikz> environment
+        $._verbatim_plain_content, // body of a <verb>/<verbatim> environment
+        $._inline_math_content, // body of `$ ... $`
+        $._display_math_content, // body of `$$ ... $$`
+        $._markdown_chunk, // a run of ordinary Markdown text
+        $._error_sentinel, // tree-sitter's invalid-input sentinel
+    ],
+
+    // `$` participates in math fences and must never be silently skipped; the
+    // scanner owns all whitespace handling, so nothing is `extras`.
+    extras: () => [],
+
+    // Keyword-extraction token. It makes the lexer treat the frontmatter
+    // language keywords (`yaml`/`toml`/`json`) and verbatim tag names as whole
+    // words, so `yamlx` is not tokenised as `yaml` + `x` and `<texx>` is not
+    // mistaken for a `<tex>` environment.
+    word: ($) => $._word,
+
+    conflicts: () => [],
+
+    rules: {
+        // A document is an optional frontmatter block followed by body content.
+        document: ($) => seq(optional($.frontmatter), repeat($._block)),
+
+        // ── Frontmatter ──────────────────────────────────────────────────
+        //
+        // Only valid as the very first thing in the file (the scanner only
+        // emits `_frontmatter_start` at offset 0). The optional language
+        // keyword right after the opening `---` selects the embedded language;
+        // `injections.scm` keys off the `language` child. `+++ … +++` is the
+        // TOML shorthand and carries no keyword.
+        frontmatter: ($) =>
+            seq(
+                field('open', alias($._frontmatter_start, $.frontmatter_fence)),
+                optional(field('language', $.frontmatter_language)),
+                optional(field('content', $.frontmatter_content)),
+                field('close', alias($._frontmatter_end, $.frontmatter_fence)),
+            ),
+
+        // `yaml` is the implicit default when no keyword is given. The keyword
+        // is glued to the opening fence (`---toml`), hence `token.immediate`.
+        frontmatter_language: () =>
+            token.immediate(choice('yaml', 'toml', 'json')),
+
+        // Everything between the fences, produced by the external scanner.
+        frontmatter_content: ($) => $._frontmatter_body,
+
+        // ── Body blocks ──────────────────────────────────────────────────
+        //
+        // Verbatim environments and math are recognised explicitly; anything
+        // else is an opaque `markdown_chunk` for the `markdown` grammar.
+        _block: ($) =>
+            choice(
+                $.verbatim_environment,
+                $.display_math,
+                $.inline_math,
+                $.markdown_chunk,
+            ),
+
+        // ── Verbatim environments ────────────────────────────────────────
+        //
+        // `<tex …>…</tex>`, `<verbatim>…</verbatim>`, etc. The body never
+        // participates in Markdown/Svelte parsing. The external scanner reads
+        // up to (but not including) the matching `</tag>`; for TeX tags the
+        // body is `latex`, otherwise it is opaque text.
+        // Two arms keep the TeX vs. plain distinction: the opening tag's name
+        // token (`_tex_tag_name` / `_plain_tag_name`) is distinct, so the
+        // parser commits to the right arm at the tag, and the body's external
+        // token (`_verbatim_tex_content` / `_verbatim_plain_content`) is
+        // therefore unambiguous.
+        verbatim_environment: ($) =>
+            choice($._verbatim_tex, $._verbatim_plain),
+
+        _verbatim_tex: ($) =>
+            seq(
+                field('open', $.verbatim_tex_open_tag),
+                optional(field('body', $.tex_verbatim_body)),
+                field('close', $.verbatim_close_tag),
+            ),
+
+        _verbatim_plain: ($) =>
+            seq(
+                field('open', $.verbatim_plain_open_tag),
+                optional(field('body', $.plain_verbatim_body)),
+                field('close', $.verbatim_close_tag),
+            ),
+
+        // Opening tags. The tag name is captured so `highlights.scm` can
+        // colour it; attributes are kept as one opaque blob (they may hold
+        // Svelte expressions, but precise attribute parsing is out of scope).
+        verbatim_tex_open_tag: ($) =>
+            seq(
+                '<',
+                field('name', alias($._tex_tag_name, $.tag_name)),
+                optional(field('attributes', $.verbatim_attributes)),
+                token.immediate('>'),
+            ),
+
+        verbatim_plain_open_tag: ($) =>
+            seq(
+                '<',
+                field('name', alias($._plain_tag_name, $.tag_name)),
+                optional(field('attributes', $.verbatim_attributes)),
+                token.immediate('>'),
+            ),
+
+        verbatim_close_tag: ($) =>
+            seq(
+                '</',
+                field('name', alias($._verbatim_tag_name, $.tag_name)),
+                token.immediate('>'),
+            ),
+
+        _tex_tag_name: () =>
+            token.immediate(choiceOfStrings(TEX_VERBATIM_TAGS)),
+        _plain_tag_name: () =>
+            token.immediate(choiceOfStrings(PLAIN_VERBATIM_TAGS)),
+        _verbatim_tag_name: () =>
+            token.immediate(
+                choiceOfStrings([
+                    ...TEX_VERBATIM_TAGS,
+                    ...PLAIN_VERBATIM_TAGS,
+                ]),
+            ),
+
+        // Attributes of a verbatim opening tag — opaque up to the closing `>`.
+        verbatim_attributes: () => token.immediate(/[ \t\r\n][^>]*/),
+
+        tex_verbatim_body: ($) => $._verbatim_tex_content,
+        plain_verbatim_body: ($) => $._verbatim_plain_content,
+
+        // ── Math ─────────────────────────────────────────────────────────
+        //
+        // Four delimiter styles. `$$…$$` and `\[…\]` are display math; `$…$`
+        // and `\(…\)` are inline. Dollar fences need the external scanner
+        // because the same token opens and closes them. The escaped-bracket
+        // styles are plain LR rules. The body is `latex` in every case.
+        //
+        // `$$…$$` is genuinely ambiguous with two empty `$…$` spans wrapping
+        // text, so `display_math` and `inline_math` carry `prec.dynamic`
+        // weights (and appear in `conflicts`): GLR explores both parses and
+        // keeps the higher-weighted display reading.
+        display_math: ($) =>
+            prec.dynamic(
+                2,
+                choice(
+                    seq(
+                        alias($._dollar_dollar, $.math_delimiter),
+                        optional(field('body', $.math_content_display)),
+                        alias($._dollar_dollar, $.math_delimiter),
+                    ),
+                    seq(
+                        alias('\\[', $.math_delimiter),
+                        optional(field('body', $.math_content_bracket)),
+                        alias('\\]', $.math_delimiter),
+                    ),
+                ),
+            ),
+
+        inline_math: ($) =>
+            prec.dynamic(
+                1,
+                choice(
+                    seq(
+                        alias($._dollar, $.math_delimiter),
+                        optional(field('body', $.math_content_inline)),
+                        alias($._dollar, $.math_delimiter),
+                    ),
+                    seq(
+                        alias('\\(', $.math_delimiter),
+                        optional(field('body', $.math_content_paren)),
+                        alias('\\)', $.math_delimiter),
+                    ),
+                ),
+            ),
+
+        _dollar: () => token(prec(1, '$')),
+        _dollar_dollar: () => token(prec(2, '$$')),
+
+        // Dollar-delimited math bodies come from the external scanner, which
+        // stops right before the matching closing fence.
+        math_content_display: ($) => $._display_math_content,
+        math_content_inline: ($) => $._inline_math_content,
+
+        // `\[ ... \]` body: anything up to the literal closing `\]`.
+        math_content_bracket: () => token(prec(-1, /([^\\]|\\[^\]])+/)),
+
+        // `\( ... \)` body: anything up to the literal closing `\)`.
+        math_content_paren: () => token(prec(-1, /([^\\]|\\[^)])+/)),
+
+        // ── Markdown ─────────────────────────────────────────────────────
+        //
+        // An opaque run of ordinary content. The external scanner produces it
+        // greedily, stopping just before the next `.sveltex`-special token
+        // (a verbatim open tag, a `$`/`$$`/`\(`/`\[` math opener, or EOF).
+        // `injections.scm` ships the whole run to the `markdown` grammar.
+        markdown_chunk: ($) => $._markdown_chunk,
+
+        // A bare identifier; only used to satisfy `word`.
+        _word: () => /[A-Za-z][A-Za-z0-9_-]*/,
+    },
+});
+
+/**
+ * Builds a `choice()` of string literals.
+ *
+ * @param {string[]} strings
+ * @returns {ChoiceRule}
+ */
+function choiceOfStrings(strings) {
+    return choice(...strings.map((s) => s));
+}

package/package.json

@@ -0,0 +1,76 @@
+{
+  "name": "@nvl/tree-sitter-sveltex",
+  "version": "0.2.0",
+  "description": "Tree-sitter grammar for SvelTeX (.sveltex) documents",
+  "license": "MIT",
+  "author": {
+    "name": "N. V. Lang",
+    "email": "npm@nvlang.dev",
+    "url": "https://nvlang.dev/"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/nvlang/sveltex.git",
+    "directory": "packages/tree-sitter-sveltex"
+  },
+  "bugs": {
+    "url": "https://github.com/nvlang/sveltex/issues"
+  },
+  "homepage": "https://sveltex.dev",
+  "keywords": [
+    "tree-sitter",
+    "parser",
+    "sveltex",
+    "svelte",
+    "markdown",
+    "latex"
+  ],
+  "main": "bindings/node",
+  "files": [
+    "grammar.js",
+    "tree-sitter.json",
+    "binding.gyp",
+    "queries/*.scm",
+    "src/**",
+    "bindings/node/*"
+  ],
+  "tree-sitter": [
+    {
+      "scope": "source.sveltex",
+      "file-types": [
+        "sveltex"
+      ],
+      "highlights": [
+        "queries/highlights.scm"
+      ],
+      "injections": [
+        "queries/injections.scm"
+      ],
+      "folds": [
+        "queries/folds.scm"
+      ]
+    }
+  ],
+  "dependencies": {
+    "node-addon-api": "^8.0.0",
+    "node-gyp-build": "^4.8.0"
+  },
+  "peerDependencies": {
+    "tree-sitter": "^0.25.0"
+  },
+  "peerDependenciesMeta": {
+    "tree-sitter": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "tree-sitter": "0.25.0",
+    "tree-sitter-cli": "^0.26.8"
+  },
+  "scripts": {
+    "install": "node-gyp-build",
+    "generate": "tree-sitter generate",
+    "test": "tree-sitter generate && tree-sitter test",
+    "parse": "tree-sitter parse"
+  }
+}

package/queries/folds.scm

@@ -0,0 +1,18 @@
+; Code folding for the SvelTeX (`.sveltex`) grammar.
+;
+; Only the multi-line structural blocks the `.sveltex` grammar owns are made
+; foldable here; folding *within* embedded languages is contributed by the
+; injected grammars' own fold queries.
+;
+; The `@fold` capture marks a node as a foldable region (the tree-sitter and
+; Zed convention).
+
+; A frontmatter block folds to its opening fence.
+(frontmatter) @fold
+
+; A verbatim environment (`<tex>…</tex>`, `<verbatim>…</verbatim>`, ...) folds
+; to its opening tag.
+(verbatim_environment) @fold
+
+; A display-math block (`$$…$$` or `\[…\]`) folds to its first line.
+(display_math) @fold

package/queries/highlights.scm

@@ -0,0 +1,51 @@
+; Syntax highlighting for the SvelTeX (`.sveltex`) grammar.
+;
+; This grammar parses only the top-level `.sveltex` structure; the bulk of a
+; document's highlighting comes from the injected grammars (see
+; `injections.scm`). The captures below colour just the structural delimiters
+; that the `.sveltex` grammar itself owns: frontmatter fences, math delimiters
+; and verbatim-environment tags.
+;
+; Capture names follow the standard tree-sitter highlight set so that the
+; query works unchanged across editors (Zed, Neovim, Helix, ...).
+
+; ── Frontmatter ──────────────────────────────────────────────────────────
+
+; The `---` / `+++` fences delimiting a frontmatter block.
+(frontmatter_fence) @punctuation.delimiter
+
+; The optional `yaml` / `toml` / `json` keyword after the opening fence.
+(frontmatter_language) @keyword
+
+; ── Math ─────────────────────────────────────────────────────────────────
+
+; The `$`, `$$`, `\(`, `\)`, `\[`, `\]` math delimiters.
+(math_delimiter) @punctuation.special
+
+; The math body itself is highlighted by the injected `latex` grammar; tag it
+; as embedded so editors without that grammar still render it distinctly.
+(math_content_display) @markup.math
+(math_content_inline) @markup.math
+(math_content_bracket) @markup.math
+(math_content_paren) @markup.math
+
+; ── Verbatim environments ────────────────────────────────────────────────
+
+; The `<` / `>` / `</` punctuation of a verbatim environment's tags.
+(verbatim_tex_open_tag ["<" ">"] @punctuation.bracket)
+(verbatim_plain_open_tag ["<" ">"] @punctuation.bracket)
+(verbatim_close_tag ["</" ">"] @punctuation.bracket)
+
+; The tag name (`tex`, `verbatim`, ...).
+(verbatim_tex_open_tag (tag_name) @tag)
+(verbatim_plain_open_tag (tag_name) @tag)
+(verbatim_close_tag (tag_name) @tag)
+
+; A verbatim opening tag's attributes are kept opaque by the grammar; colour
+; the whole blob as an attribute.
+(verbatim_attributes) @attribute
+
+; The body of a `<verb>` / `<verbatim>` environment is intentionally literal
+; (SvelTeX escapes rather than renders it) and has no injection, so highlight
+; it as plain raw text.
+(plain_verbatim_body) @markup.raw