@willbooster/tree-sitter-haskell 0.0.0-placeholder.0 → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Max Brunsfeld
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,156 @@
 # @willbooster/tree-sitter-haskell
 
-This version is a placeholder that reserves the package name so that releases can be published from GitHub Actions with trusted publishing. Use 1.0.0 or later.
+[![npm version](https://img.shields.io/npm/v/@willbooster/tree-sitter-haskell.svg)](https://www.npmjs.com/package/@willbooster/tree-sitter-haskell)
+[![license](https://img.shields.io/npm/l/@willbooster/tree-sitter-haskell.svg)](https://www.npmjs.com/package/@willbooster/tree-sitter-haskell)
+[![Test](https://github.com/WillBooster/tree-sitter-haskell/actions/workflows/test.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-haskell/actions/workflows/test.yml)
+[![Test rust](https://github.com/WillBooster/tree-sitter-haskell/actions/workflows/test-rust.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-haskell/actions/workflows/test-rust.yml)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![wbfy](https://img.shields.io/badge/wbfy-20.36.11-1e90ff.svg)](https://github.com/WillBooster/shared/tree/main/packages/wbfy)
+[![crates.io](https://img.shields.io/crates/v/willbooster-tree-sitter-haskell.svg)](https://crates.io/crates/willbooster-tree-sitter-haskell)
+
+Haskell grammar for [tree-sitter](https://github.com/tree-sitter/tree-sitter), forked from
+[tree-sitter/tree-sitter-haskell](https://github.com/tree-sitter/tree-sitter-haskell). We are grateful
+to its authors and contributors. This is not an official release of that project.
+
+This fork keeps the grammar of upstream's version 0.23.1 and generates the parser with the
+[WillBooster/tree-sitter](https://github.com/WillBooster/tree-sitter) generator. The parser carries that fork's own ABI number, so it loads only with that fork's runtime, version 2.0.0
+or later; upstream's runtime rejects it.
+[Upstream's README at that version](https://github.com/tree-sitter/tree-sitter-haskell/blob/v0.23.1/README.md) lists
+the supported language extensions and, under "Querying", explains how to query the grammar's supertypes; it is the
+readme that the comments in `grammar.js` refer to.
+
+## Differences from upstream
+
+`grammar.js`, the files in `grammar/`, the external scanner (`src/scanner.c`, `src/unicode.h`), `queries/`, and
+upstream's corpus are those of upstream's 0.23.1, except for:
+
+- **The newline token** (`grammar.js`, `grammar/externals.js`): the entry `/\n/` of `extras` and of `externals` is
+  written `token(prec(0, /\n/))` in both places, with a comment line above the one in `grammar.js` that points here. The WillBooster/tree-sitter generator turns an unnamed pattern in
+  `extras` into a lexer separator, which leaves the external token of the same pattern without a symbol, and the
+  generator then fails. Precedence 0 is the default, so the token lexes as before, and the generated parser has the
+  same symbols as upstream's. A later upstream version of these files needs the same two lines until the generator
+  keeps such a pattern as a token ([WillBooster/tree-sitter#129](https://github.com/WillBooster/tree-sitter/issues/129)).
+- **`queries/highlights.scm`**: three patterns that match fields of the supertype `decl` are written through its
+  subtypes (`decl/function`, `decl/bind`, `decl/signature`), as upstream's repository has them since
+  `tree-sitter/tree-sitter-haskell@0b2ebf7`. A parser of ABI 15 or later carries the supertypes, and the runtime then
+  rejects the 0.23.1 patterns as impossible; upstream's own CLI 0.27.0 does so as well.
+- **`test/corpus/null-characters.txt`** is added, because the fuzz job checks that every NUL character of the corpus
+  reaches the fuzz seeds and upstream's corpus has none.
+
+A NUL character inside a string or character literal parses as part of the literal, where upstream's prebuilt parser
+reports an error: the lexers of the WillBooster/tree-sitter generator let character classes match a NUL character.
+
+## Usage
+
+The npm package ships `tree-sitter-haskell.wasm` for
+[@willbooster/web-tree-sitter](https://www.npmjs.com/package/@willbooster/web-tree-sitter) version 2.0.0 or later, which runs in Node.js, Bun, browsers, and Cloudflare Workers. In Node.js and Bun, load the grammar from its path:
+
+```js
+import { fileURLToPath } from 'node:url';
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+
+await Parser.init();
+const parser = new Parser();
+const wasmPath = fileURLToPath(import.meta.resolve('@willbooster/tree-sitter-haskell/tree-sitter-haskell.wasm'));
+parser.setLanguage(await Language.load(wasmPath));
+const tree = parser.parse('main = putStrLn "Hello World!"\n');
+```
+
+In browsers, serve both `.wasm` files and load them by URL (the example uses Vite's `?url` imports):
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtimeUrl from '@willbooster/web-tree-sitter/web-tree-sitter.wasm?url';
+import haskellUrl from '@willbooster/tree-sitter-haskell/tree-sitter-haskell.wasm?url';
+
+await Parser.init({ locateFile: () => runtimeUrl });
+const parser = new Parser();
+parser.setLanguage(await Language.load(haskellUrl));
+```
+
+Cloudflare Workers do not allow compiling Wasm at run time, so import both `.wasm` files as modules, with or without
+Node.js compatibility:
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtime from '@willbooster/web-tree-sitter/web-tree-sitter.wasm';
+import haskell from '@willbooster/tree-sitter-haskell/tree-sitter-haskell.wasm';
+
+await Parser.init({ wasmModule: runtime });
+const parser = new Parser();
+parser.setLanguage(await Language.load(haskell));
+```
+
+The package also ships `grammar.js` with the files in `grammar/` it loads, the queries in `queries/`, and the node types in `src/node-types.json`.
+
+In Rust, depend on the [crate](https://crates.io/crates/willbooster-tree-sitter-haskell) and on
+[willbooster-tree-sitter](https://crates.io/crates/willbooster-tree-sitter), the runtime this package is tested and
+fuzzed with:
+
+```toml
+[dependencies]
+tree-sitter = { package = "willbooster-tree-sitter", version = "2.0.0" }
+tree-sitter-haskell = { package = "willbooster-tree-sitter-haskell", version = "1" }
+```
+
+```rust
+let mut parser = tree_sitter::Parser::new();
+parser.set_language(&tree_sitter_haskell::LANGUAGE.into())?;
+```
+
+## Development
+
+```sh
+mise install
+bun install --frozen-lockfile
+bun run test/ci-setup # installs Chromium for the browser test
+bun run build/ci
+bun run test
+script/parse-examples
+cargo test
+```
+
+The scripts and tests generate, build, test, and parse with `script/tree-sitter`, the tree-sitter CLI of the
+WillBooster/tree-sitter runtime version locked in `Cargo.lock`, since the generator and the runtime of upstream's CLI are
+not the ones this package ships with. Its first run downloads that CLI from the runtime's GitHub Release, or builds it
+with `cargo` (whose build runs the CMake that `mise.toml` pins) when the download fails or the release has no binary that runs here. Run other CLI commands through it as
+well (e.g. `script/tree-sitter parse file.hs`).
+
+`bun run generate` records a fresh generation profile from the applicable `test/corpus` cases and Git-tracked
+files in `examples/`, then generates compact parser tables. The parser also embeds metadata from `tree-sitter.json`. After changing a grammar file,
+`tree-sitter.json`, a corpus case, or a tracked example,
+regenerate and commit `src/`. Stage added or removed examples with `git add -A examples` before generation so the profile uses the intended file list.
+Profiles in `.tmp/generation-profiles/` are temporary and must not be committed. `bun run build-wasm`, `bun run build/ci`,
+and the release build regenerate the parsers before compiling them.
+
+`bun run test` runs:
+
+- the corpus in `test/corpus`, with the native build and with the Wasm build (the first run downloads the WASI SDK);
+- an incremental-parsing check (`test/unit/incremental.test.ts`): `script/fuzz-corpus` runs `tree-sitter fuzz`, which
+  edits each corpus case at random, reparses it, undoes the edits, and reparses again. `TREE_SITTER_SEED`,
+  `TREE_SITTER_ITERATIONS`, and `TREE_SITTER_EDITS` run other or more edits. It skips the eleven cases that
+  upstream's own parser also reparses differently from a fresh parse (see `script/fuzz-corpus`);
+- a check that the real-world Haskell files in `examples/`, the checked-in ones and those of the cloned repositories, fail to
+  parse exactly as listed in `script/known-failures.txt`; upstream's prebuilt parser fails on the same files. The first run clones the repositories. The example repositories are pinned to commits in
+  `script/parse-examples`. After a grammar change or a moved pin alters that list, `script/parse-examples` rewrites it;
+  review its diff before committing;
+- a performance check (`test/unit/performance.test.ts`) that recovering from an error on each line takes linear
+  time, since consumers parse files while they are being edited. It loads the Wasm build through
+  @willbooster/web-tree-sitter, which `bun run build/ci` rebuilds after regenerating the parser;
+- a check (`test/unit/runtimeVersion.test.ts`) that `@willbooster/web-tree-sitter` in `package.json` and
+  `willbooster-tree-sitter` in `Cargo.lock` are the same version, since the Wasm tests run on the former and the Rust
+  tests, the fuzzer, and the CLI on the latter;
+- tests that load the Wasm build with @willbooster/web-tree-sitter in Chromium (`test/e2e/browser.test.ts`) and in
+  Cloudflare Workers with and without Node.js compatibility (`test/e2e/workers.test.ts`).
+
+The tests and `script/parse-examples` compile the parser into `.tmp/tree-sitter-lib` rather than the CLI's cache shared
+by every checkout; `script/fuzz-corpus` builds a parser of its own in `.tmp/fuzz` for each run and deletes it
+afterwards.
+
+CI also runs these tests on Linux arm64 and macOS, where the CLI compiles the parser and the scanner natively, and fuzzes the parser
+with libFuzzer and sanitizers (`.github/workflows/robustness.yml`).
+
+### References
+
+- [Haskell 2010 Language Report: Syntax Reference](https://www.haskell.org/onlinereport/haskell2010/haskellch10.html)
+- [GHC Language Extensions](https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/exts/table.html)

package/grammar/class.js

@@ -0,0 +1,130 @@
+const {
+  sep1,
+  layout,
+  context,
+  forall,
+} = require('./util.js')
+
+module.exports = {
+
+  // ------------------------------------------------------------------------
+  // associated families
+  // ------------------------------------------------------------------------
+
+  /**
+   * In associated family declarations, result type aliasing without injectivity is invalid, since that syntax is taken
+   * by type instance declarations.
+   */
+  _assoc_tyfam: $ => seq(
+    'type',
+    optional('family'),
+    $._type_head,
+    optional(choice(
+      $._kind_annotation,
+      seq(
+        $.type_family_result,
+        $.type_family_injectivity,
+      ),
+    )),
+  ),
+
+  _assoc_tyinst: $ => seq(
+    'type',
+    optional('instance'),
+    forall($),
+    $._cond_assoc_tyinst,
+    $._type_instance_common,
+  ),
+
+  _assoc_datafam: $ => seq(
+    'data',
+    optional('family'),
+    $._datafam,
+  ),
+
+  _assoc_datainst_adt: $ => seq(
+    'data',
+    optional('instance'),
+    $._inst_adt,
+  ),
+
+  _assoc_datainst_newtype: $ => seq(
+    'newtype',
+    optional('instance'),
+    $._inst_newtype,
+  ),
+
+  _assoc_datainst: $ => choice(
+    alias($._assoc_datainst_adt, $.data_type),
+    alias($._assoc_datainst_newtype, $.newtype),
+  ),
+
+  // ------------------------------------------------------------------------
+  // class
+  // ------------------------------------------------------------------------
+
+  default_signature: $ => seq('default', field('signature', $.signature)),
+
+  /**
+   * Classes can have both type families and instances, but only data families.
+   */
+  class_decl: $ => choice(
+    $._local_decl,
+    $.default_signature,
+    alias($._assoc_tyfam, $.type_family),
+    alias($._assoc_tyinst, $.type_instance),
+    alias($._assoc_datafam, $.data_family),
+  ),
+
+  fundep: $ => seq(
+    field('matched', repeat1($.variable)),
+    $._arrow,
+    field('determined', repeat1($.variable)),
+  ),
+
+  fundeps: $ => seq($._bar, sep1(',', field('fundep', $.fundep))),
+
+  class_declarations: $ => layout($, field('declaration', $.class_decl)),
+
+  class: $ => seq(
+    'class',
+    context($),
+    $._type_head,
+    field('fundeps', optional($.fundeps)),
+    optional(seq($._where, optional(field('declarations', $.class_declarations)))),
+  ),
+
+  // ------------------------------------------------------------------------
+  // instance
+  // ------------------------------------------------------------------------
+
+  instance_decl: $ => choice(
+    $.decl,
+    alias($._assoc_datainst, $.data_instance),
+    alias($._assoc_tyinst, $.type_instance),
+  ),
+
+  instance_declarations: $ => layout($, field('declaration', $.instance_decl)),
+
+  /**
+   * instances only allow single foralls and contexts
+   */
+  _instance: $ => seq(
+    'instance',
+    forall($),
+    context($),
+    $._type_instance_head,
+  ),
+
+  instance: $ => seq(
+    $._instance,
+    optional(seq($._where, optional(field('declarations', $.instance_declarations)))),
+  ),
+
+  deriving_instance: $ => seq(
+    optional($._phantom_deriving),
+    'deriving',
+    optional(choice(field('strategy', $.deriving_strategy), field('via', $.via))),
+    $._instance,
+  ),
+}

package/grammar/conflicts.js

@@ -0,0 +1,116 @@
+module.exports = {
+
+  conflicts: $ => [
+
+    /**
+     * For reference in GHC:
+     * - Note [Ambiguous syntactic categories]
+     * - Note [PatBuilder]
+     * - Note [Declaration/signature overlap]
+     * - These correspond to `DisambECP`
+     *
+     * (fun x) y = undefined
+     * (fun x -> y) = undefined
+     * (fun) <> x = undefined
+     *
+     * The first one is a regular function with some redundant parens, where `fun` is the declared name.
+     * The second one is a pattern binder with a view pattern, where `fun` is a free variable.
+     * The third one is an infix pattern binder, where `fun` is a simple varid pattern with redundant parens.
+     *
+     * These conflicts are also relevant for top-level expression splices, which fundamentally conflict with decls, and
+     * since decls start with either `var` or `pat`, they cannot be disambiguated.
+     *
+     * GHC parses functions and binds as expressions and sorts them into the right LHS in a post-processing step.
+     * Since this is not possible in tree-sitter, these conflicts are more function-centric than in GHC.
+     *
+     * function:
+     * func (A a) = a
+     *
+     * bind variable:
+     * a : as = [1, 2, 3]
+     *
+     * pattern bind infix:
+     * a : as = [1, 2, 3]
+     *
+     * pattern bind prefix:
+     * Just 1 = Just 1
+     *
+     * splice:
+     * makeLenses ''A
+     *
+     * Signature and bind:
+     *
+     * fun :: Int
+     * fun :: Int = 5
+     */
+
+    // Function vs bind
+    [$._function_name, $.pattern],
+
+    // Function vs bind vs splice
+    [$._function_name, $.pattern, $.expression],
+
+    // Bind vs splice
+    [$.pattern, $.expression],
+
+    // Signature vs bind
+    [$.signature, $.pattern],
+
+    /**
+     * Unboxed syntax
+     *
+     * The hash in the opening parenthesis of unboxed tuples can be an operator.
+     */
+    [$._operator_hash_head, $._unboxed_open],
+
+    /**
+     * Types conflicting with structures that look like types
+     *
+     * Note: These conflicts have been circumvented by a lookahead mechanism in the scanner.
+     * This comment is preserved for reference.
+     *
+     * `name` and `constructor` use the same terminal symbol, but we cannot reduce `constructor` in prefix data
+     * constructor declarations.
+     *
+     * In GHC, this corresponds to `DisambTD`.
+     *
+     * > data A = Name Int
+     * > data A = Maybe Int :+ Int
+     * > data A = Monoid a => A a
+     *
+     * All of these start with a `name` node.
+     * In the first example, the `name` is a data constructor, which will not be reduced to a `type`.
+     *
+     * In the second example, the `name` is a type constructor applied to another type in the left operand of an infix
+     * data constructor, so it must be reduced to `type`, and then to `apply` with the `Int`.
+     *
+     * In the third example, the `name` is a type constructor applied to a variable resulting in a constraint.
+     * It will be reduced the same way as the second example, but using the class tree, which is mostly identical to the
+     * type tree, but conflicts since we want to distinguish classes from types granularly.
+     *
+     * In GHC, these correspond to `mkHsAppTyHeadPV` and `mkHsAppTyPV`.
+     *
+     * > data A a b = a `C` b => a `A` b
+     * > data A a b = a `A` b
+     *
+     * > data a *** b
+     * > data a +++ b => a *** b
+     *
+     * In GHC, this corresponds to `mkHsOpTyPV`.
+     *
+     * > class A where type a + b = r | r -> a b
+     * > class A where type a + b = (a, b)
+     *
+     * The first one is a type family declaration, the second one an instance.
+     *
+     * These were the conflicts that have been turned into scanner lookahead:
+     *
+     * [$._type_con, $.data_constructor],
+     * [$._type_con, $._type_head_name],
+     * [$._type_variable, $._tyvar],
+     * [$._constructor_ticked, $._tycon_ticked],
+     */
+
+  ],
+
+}

package/grammar/context.js

@@ -0,0 +1,69 @@
+const {
+  parens,
+  sep2,
+} = require('./util.js')
+
+module.exports = {
+
+  // ------------------------------------------------------------------------
+  // context
+  // ------------------------------------------------------------------------
+
+  _class_apply: $ => prec.left('apply', seq(
+    field('constructor', $.constraint),
+    field('argument', $._type_apply_arg),
+  )),
+
+  _class_infix: $ => prec.right('infix', seq(
+    $._cond_infix,
+    field('left_operand', $.type),
+    field('operator', $._tyops),
+    field('right_operand', $.type),
+  )),
+
+  _ctr_parens: $ => parens($, $.constraints),
+
+  _ctr_tuple: $ => parens($, sep2(',', $.constraints)),
+
+  /**
+   * Implicit parameters have an annotation with `::` but bind tighter than `_type_signature`, with the same precedence
+   * as foralls, contexts and arrows.
+   *
+   * > A => ?a :: A   | associates as |   A => (?a :: A)
+   * > ?a :: A -> A   | associates as |   ?a :: (A -> A)
+   */
+  implicit_parameter: $ => prec.left(seq(field('name', $.implicit_variable), $._type_annotation)),
+
+  constraint: $ => choice(
+    $._type_name,
+    alias($._class_infix, $.infix),
+    alias($._class_apply, $.apply),
+    alias($._ctr_parens, $.parens),
+    alias($._ctr_tuple, $.tuple),
+    alias($._type_wildcard, $.wildcard),
+    $._universal,
+  ),
+
+  _ctr_forall: $ => prec('fun', seq($._forall_body, '.', field('constraint', $.constraints))),
+
+  _ctr_context: $ => prec('fun', seq($._context_inline, field('constraint', $.constraints))),
+
+  _ctr_signature: $ => prec('annotated', seq(field('constraint', $.constraints), $._kind_annotation)),
+
+  constraints: $ => choice(
+    $.constraint,
+    alias($._ctr_context, $.context),
+    alias($._ctr_forall, $.forall),
+    $.implicit_parameter,
+    alias($._ctr_signature, $.signature),
+  ),
+
+  _context_inline: $ => seq(
+    $._cond_context,
+    field('context', $.constraint),
+    field('arrow', $._carrow),
+  ),
+
+  context: $ => prec('qtype-single', $._context_inline),
+
+}