@willbooster/tree-sitter-embedded-template 0.0.0-stage → 1.0.1

package/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Max Brunsfeld
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,136 @@
-# Temporary Holding Version
+# @willbooster/tree-sitter-embedded-template
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![Test](https://github.com/WillBooster/tree-sitter-embedded-template/actions/workflows/test.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-embedded-template/actions/workflows/test.yml)
+[![Test rust](https://github.com/WillBooster/tree-sitter-embedded-template/actions/workflows/test-rust.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-embedded-template/actions/workflows/test-rust.yml)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![wbfy](https://img.shields.io/badge/wbfy-20.36.11-1e90ff.svg)](https://github.com/WillBooster/shared/tree/main/packages/wbfy)
+[![crates.io](https://img.shields.io/crates/v/willbooster-tree-sitter-embedded-template.svg)](https://crates.io/crates/willbooster-tree-sitter-embedded-template)
+
+Embedded template grammar for [tree-sitter](https://github.com/tree-sitter/tree-sitter), forked from
+[tree-sitter/tree-sitter-embedded-template](https://github.com/tree-sitter/tree-sitter-embedded-template). We are grateful
+to its authors and contributors. This is not an official release of that project.
+
+It parses templating languages such as ERB and EJS, in which scripting code is embedded in text with the delimiters
+`<%` and `%>`. This fork extends upstream's version 0.25.0 grammar with EJS output-tag whitespace slurping.
+It generates the parser with the
+[WillBooster/tree-sitter](https://github.com/WillBooster/tree-sitter) generator. The parser loads only with that fork's runtime, version 2.0.0 or later
+([willbooster-tree-sitter](https://crates.io/crates/willbooster-tree-sitter),
+[@willbooster/web-tree-sitter](https://www.npmjs.com/package/@willbooster/web-tree-sitter)); upstream's runtime rejects it.
+
+## Usage
+
+The npm package ships `tree-sitter-embedded_template.wasm` for
+[@willbooster/web-tree-sitter](https://www.npmjs.com/package/@willbooster/web-tree-sitter) version 2.0.0 or later, which runs in Node.js, Bun, browsers, and Cloudflare Workers. In Node.js and Bun, load the grammar from its path:
+
+```js
+import { fileURLToPath } from 'node:url';
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+
+await Parser.init();
+const parser = new Parser();
+const wasmPath = fileURLToPath(
+  import.meta.resolve('@willbooster/tree-sitter-embedded-template/tree-sitter-embedded_template.wasm')
+);
+parser.setLanguage(await Language.load(wasmPath));
+const tree = parser.parse('<ul><% items.each do |item| %><li><%= item %></li><% end %></ul>\n');
+```
+
+In browsers, serve both `.wasm` files and load them by URL (the example uses Vite's `?url` imports):
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtimeUrl from '@willbooster/web-tree-sitter/web-tree-sitter.wasm?url';
+import grammarUrl from '@willbooster/tree-sitter-embedded-template/tree-sitter-embedded_template.wasm?url';
+
+await Parser.init({ locateFile: () => runtimeUrl });
+const parser = new Parser();
+parser.setLanguage(await Language.load(grammarUrl));
+```
+
+Cloudflare Workers do not allow compiling Wasm at run time, so import both `.wasm` files as modules, with or without
+Node.js compatibility:
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtime from '@willbooster/web-tree-sitter/web-tree-sitter.wasm';
+import grammar from '@willbooster/tree-sitter-embedded-template/tree-sitter-embedded_template.wasm';
+
+await Parser.init({ wasmModule: runtime });
+const parser = new Parser();
+parser.setLanguage(await Language.load(grammar));
+```
+
+The package also ships `grammar.js`, the queries in `queries/`, and the node types in `src/node-types.json`.
+
+In Rust, depend on the [crate](https://crates.io/crates/willbooster-tree-sitter-embedded-template) and on
+[willbooster-tree-sitter](https://crates.io/crates/willbooster-tree-sitter), the runtime this package is tested and
+fuzzed with:
+
+```toml
+[dependencies]
+tree-sitter = { package = "willbooster-tree-sitter", version = "2.0.0" }
+tree-sitter-embedded-template = { package = "willbooster-tree-sitter-embedded-template", version = "1" }
+```
+
+```rust
+let mut parser = tree_sitter::Parser::new();
+parser.set_language(&tree_sitter_embedded_template::LANGUAGE.into())?;
+```
+
+## Development
+
+```sh
+mise install
+bun install --frozen-lockfile
+bun run test/ci-setup # installs Chromium for the browser test
+bun run build/ci
+bun run test
+script/parse-examples
+cargo test
+```
+
+The scripts and tests generate, build, test, and parse with `script/tree-sitter`, the tree-sitter CLI of the
+WillBooster/tree-sitter runtime version locked in `Cargo.lock`, since the generator and the runtime of upstream's CLI are
+not the ones this package ships with. Its first run downloads that CLI from the runtime's GitHub Release, or builds it
+with `cargo` (whose build runs the CMake that `mise.toml` pins) when the download fails or the release has no binary that runs here. Run other CLI commands through it as
+well (e.g. `script/tree-sitter parse --grammar-path . file.erb`).
+
+`bun run generate` records a fresh generation profile from the applicable `test/corpus` cases, then generates
+compact parser tables. It would also profile files tracked in `examples/`, but this repository tracks none: the
+directory holds only the clones `script/parse-examples` makes. The parser also embeds metadata from `tree-sitter.json`.
+After changing a grammar, `tree-sitter.json`, or a corpus case, regenerate and commit `src/`.
+Profiles in `.tmp/generation-profiles/` are temporary and must not be committed. `bun run build-wasm`, `bun run build/ci`,
+and the release build regenerate the parsers before compiling them.
+
+`bun run test` runs:
+
+- the corpus in `test/corpus`, with the native build and with the Wasm build (the first run downloads the WASI SDK);
+- an incremental-parsing check (`test/unit/incremental.test.ts`): `script/fuzz-corpus` runs `tree-sitter fuzz`, which
+  edits each corpus case at random, reparses it, undoes the edits, and reparses again. `TREE_SITTER_SEED`,
+  `TREE_SITTER_ITERATIONS`, and `TREE_SITTER_EDITS` run other or more edits;
+- a check that the real-world ERB and EJS templates in `examples/` fail to parse exactly as listed in
+  `script/known-failures.txt`, which is empty now. The first run clones the repositories. The example repositories are pinned to commits in
+  `script/parse-examples`. After a grammar change or a moved pin alters that list, `script/parse-examples` rewrites it;
+  review its diff before committing;
+- a performance check (`test/unit/performance.test.ts`) that parsing a template whose last directive is still open
+  takes linear time, since consumers parse files while they are being edited. It loads the Wasm build through
+  @willbooster/web-tree-sitter, which `bun run build/ci` rebuilds after regenerating the parser;
+- a check (`test/unit/runtimeVersion.test.ts`) that `@willbooster/web-tree-sitter` in `package.json` and
+  `willbooster-tree-sitter` in `Cargo.lock` are the same version, since the Wasm tests run on the former and the Rust
+  tests, the fuzzer, and the CLI on the latter;
+- tests that load the Wasm build with @willbooster/web-tree-sitter in Chromium (`test/e2e/browser.test.ts`) and in
+  Cloudflare Workers with and without Node.js compatibility (`test/e2e/workers.test.ts`).
+
+The tests and `script/parse-examples` compile the parser into `.tmp/tree-sitter-lib` rather than the CLI's cache shared
+by every checkout; `script/fuzz-corpus` builds a parser of its own in `.tmp/fuzz` for each run and deletes it
+afterwards.
+
+CI also runs these tests on Linux arm64 and macOS, where the CLI compiles the parser natively, and fuzzes the parser
+with libFuzzer and sanitizers (`.github/workflows/robustness.yml`).
+
+### References
+
+- [ERB Documentation](https://docs.ruby-lang.org/en/master/ERB.html)
+- [EJS Documentation](https://ejs.co/#docs)
+- [etlua Documentation](https://github.com/leafo/etlua)
+- [Eta Documentation](https://eta.js.org/docs)

package/grammar.js

@@ -0,0 +1,52 @@
+/**
+ * @file Embedded Template grammar for tree-sitter
+ * @author Max Brunsfeld <maxbrunsfeld@gmail.com>
+ * @license MIT
+ */
+
+/// <reference types="tree-sitter-cli/dsl" />
+// @ts-check
+
+module.exports = grammar({
+  name: 'embedded_template',
+
+  extras: _ => [],
+
+  rules: {
+    template: $ => repeat(choice(
+      $.directive,
+      $.output_directive,
+      $.comment_directive,
+      $.graphql_directive,
+      $.content,
+    )),
+
+    code: _ => repeat1(choice(/[^%=_-]+|[%=_-]/, '%%>')),
+
+    content: _ => prec.right(repeat1(choice(/[^<]+|</, '<%%'))),
+
+    directive: $ => seq(
+      choice('<%', '<%_', '<%|', '<%~'),
+      optional($.code),
+      choice('%>', '-%>', '_%>'),
+    ),
+
+    output_directive: $ => seq(
+      choice('<%=', '<%==', '<%|=', '<%|==', '<%-'),
+      optional($.code),
+      choice('%>', '-%>', '=%>', '_%>'),
+    ),
+
+    comment_directive: $ => seq(
+      '<%#',
+      optional(alias($.code, $.comment)),
+      '%>',
+    ),
+
+    graphql_directive: $ => seq(
+      '<%graphql',
+      optional($.code),
+      '%>',
+    ),
+  },
+});

package/package.json

@@ -1,6 +1,75 @@
 {
   "name": "@willbooster/tree-sitter-embedded-template",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "1.0.1",
+  "description": "Embedded template (ERB, EJS) grammar for tree-sitter",
+  "keywords": [
+    "incremental",
+    "parsing",
+    "tree-sitter",
+    "embedded-template"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/WillBooster/tree-sitter-embedded-template.git"
+  },
+  "license": "MIT",
+  "author": "WillBooster Inc.",
+  "files": [
+    "grammar.js",
+    "queries/*.scm",
+    "src/node-types.json",
+    "tree-sitter.json",
+    "*.wasm"
+  ],
+  "scripts": {
+    "build-wasm": "script/generate --build-wasm",
+    "build/ci": "bun run build-wasm",
+    "cleanup": "bun wb lint --fix --format",
+    "format": "bun wb lint --format",
+    "generate": "script/generate",
+    "lint": "bun wb lint",
+    "lint-fix": "bun wb lint --fix",
+    "prepare": "(bun lefthook install || true) && (test ! -f .lefthook/normalize-bun-lockfile.sh || bash .lefthook/normalize-bun-lockfile.sh)",
+    "release": "wb release",
+    "prestart": "bun run build-wasm",
+    "start": "script/tree-sitter playground",
+    "test": "bun wb test",
+    "test/ci": "bun wb test-on-ci",
+    "test/ci-setup": "playwright install --with-deps chromium",
+    "typecheck": "bun wb typecheck",
+    "verify": "bun wb verify",
+    "verify-full": "bun wb verify --full"
+  },
+  "devDependencies": {
+    "@tsconfig/bun": "1.0.11",
+    "@types/bun": "1.4.2",
+    "@vitest/browser-playwright": "5.0.3",
+    "@willbooster/oxfmt-config": "1.2.2",
+    "@willbooster/oxlint-config": "1.4.8",
+    "@willbooster/wb": "22.24.0",
+    "@willbooster/web-tree-sitter": "2.0.1",
+    "conventional-changelog-conventionalcommits": "10.4.0",
+    "lefthook": "2.1.16",
+    "oxfmt": "0.71.0",
+    "oxlint": "1.86.0",
+    "oxlint-tsgolint": "7.0.2003",
+    "playwright": "1.63.0",
+    "semantic-release": "25.0.9",
+    "sort-package-json": "4.0.0",
+    "typescript": "7.0.2",
+    "vitest": "5.0.3",
+    "wrangler": "4.147.0"
+  },
+  "peerDependencies": {
+    "@willbooster/web-tree-sitter": "^2.0.0"
+  },
+  "peerDependenciesMeta": {
+    "@willbooster/web-tree-sitter": {
+      "optional": true
+    }
+  },
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  }
+}

package/queries/highlights.scm

@@ -0,0 +1,12 @@
+(comment_directive) @comment
+
+[
+  "<%#"
+  "<%"
+  "<%="
+  "<%_"
+  "<%-"
+  "%>"
+  "-%>"
+  "_%>"
+] @keyword

package/queries/injections-ejs.scm

@@ -0,0 +1,7 @@
+((content) @injection.content
+ (#set! injection.language "html")
+ (#set! injection.combined))
+
+((code) @injection.content
+ (#set! injection.language "javascript")
+ (#set! injection.combined))

package/queries/injections-erb.scm

@@ -0,0 +1,7 @@
+((content) @injection.content
+ (#set! injection.language "html")
+ (#set! injection.combined))
+
+((code) @injection.content
+ (#set! injection.language "ruby")
+ (#set! injection.combined))

package/queries/injections-etlua.scm

@@ -0,0 +1,7 @@
+((content) @injection.content
+ (#set! injection.language "html")
+ (#set! injection.combined))
+
+((code) @injection.content
+ (#set! injection.language "lua")
+ (#set! injection.combined))

package/src/node-types.json

@@ -0,0 +1,177 @@
+[
+  {
+    "type": "code",
+    "named": true,
+    "fields": {}
+  },
+  {
+    "type": "comment",
+    "named": true,
+    "fields": {}
+  },
+  {
+    "type": "comment_directive",
+    "named": true,
+    "fields": {},
+    "children": {
+      "multiple": false,
+      "required": false,
+      "types": [
+        {
+          "type": "comment",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "content",
+    "named": true,
+    "fields": {}
+  },
+  {
+    "type": "directive",
+    "named": true,
+    "fields": {},
+    "children": {
+      "multiple": false,
+      "required": false,
+      "types": [
+        {
+          "type": "code",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "graphql_directive",
+    "named": true,
+    "fields": {},
+    "children": {
+      "multiple": false,
+      "required": false,
+      "types": [
+        {
+          "type": "code",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "output_directive",
+    "named": true,
+    "fields": {},
+    "children": {
+      "multiple": false,
+      "required": false,
+      "types": [
+        {
+          "type": "code",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "template",
+    "named": true,
+    "root": true,
+    "fields": {},
+    "children": {
+      "multiple": true,
+      "required": false,
+      "types": [
+        {
+          "type": "comment_directive",
+          "named": true
+        },
+        {
+          "type": "content",
+          "named": true
+        },
+        {
+          "type": "directive",
+          "named": true
+        },
+        {
+          "type": "graphql_directive",
+          "named": true
+        },
+        {
+          "type": "output_directive",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "%%>",
+    "named": false
+  },
+  {
+    "type": "%>",
+    "named": false
+  },
+  {
+    "type": "-%>",
+    "named": false
+  },
+  {
+    "type": "<%",
+    "named": false
+  },
+  {
+    "type": "<%#",
+    "named": false
+  },
+  {
+    "type": "<%%",
+    "named": false
+  },
+  {
+    "type": "<%-",
+    "named": false
+  },
+  {
+    "type": "<%=",
+    "named": false
+  },
+  {
+    "type": "<%==",
+    "named": false
+  },
+  {
+    "type": "<%_",
+    "named": false
+  },
+  {
+    "type": "<%graphql",
+    "named": false
+  },
+  {
+    "type": "<%|",
+    "named": false
+  },
+  {
+    "type": "<%|=",
+    "named": false
+  },
+  {
+    "type": "<%|==",
+    "named": false
+  },
+  {
+    "type": "<%~",
+    "named": false
+  },
+  {
+    "type": "=%>",
+    "named": false
+  },
+  {
+    "type": "_%>",
+    "named": false
+  }
+]

package/tree-sitter.json

@@ -0,0 +1,61 @@
+{
+  "$schema": "https://tree-sitter.github.io/tree-sitter/assets/schemas/config.schema.json",
+  "grammars": [
+    {
+      "name": "embedded-template",
+      "camelcase": "EmbeddedTemplate",
+      "scope": "text.html.ejs",
+      "path": ".",
+      "file-types": [
+        "ejs"
+      ],
+      "injections": "queries/injections-ejs.scm",
+      "injection-regex": "ejs"
+    },
+    {
+      "name": "embedded-template",
+      "camelcase": "EmbeddedTemplate",
+      "scope": "text.html.erb",
+      "path": ".",
+      "file-types": [
+        "erb"
+      ],
+      "injections": "queries/injections-erb.scm",
+      "injection-regex": "erb"
+    },
+    {
+      "name": "embedded-template",
+      "camelcase": "EmbeddedTemplate",
+      "scope": "text.html.etlua",
+      "path": ".",
+      "file-types": [
+        "etlua"
+      ],
+      "injections": "queries/injections-etlua.scm",
+      "injection-regex": "etlua"
+    }
+  ],
+  "metadata": {
+    "version": "1.0.1",
+    "license": "MIT",
+    "description": "Embedded template (ERB, EJS) grammar for tree-sitter",
+    "authors": [
+      {
+        "name": "WillBooster Inc."
+      }
+    ],
+    "links": {
+      "repository": "https://github.com/WillBooster/tree-sitter-embedded-template"
+    }
+  },
+  "bindings": {
+    "c": false,
+    "go": false,
+    "java": false,
+    "node": false,
+    "python": false,
+    "rust": true,
+    "swift": false,
+    "zig": false
+  }
+}