@willbooster/tree-sitter-python 0.0.0-stage → 1.0.1

package/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Max Brunsfeld
+Copyright (c) 2026 WillBooster Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,128 @@
-# Temporary Holding Version
+# @willbooster/tree-sitter-python
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![Test](https://github.com/WillBooster/tree-sitter-python/actions/workflows/test.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-python/actions/workflows/test.yml)
+[![Test rust](https://github.com/WillBooster/tree-sitter-python/actions/workflows/test-rust.yml/badge.svg)](https://github.com/WillBooster/tree-sitter-python/actions/workflows/test-rust.yml)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![wbfy](https://img.shields.io/badge/wbfy-20.36.6-1e90ff.svg)](https://github.com/WillBooster/shared/tree/main/packages/wbfy)
+[![crates.io](https://img.shields.io/crates/v/willbooster-tree-sitter-python.svg)](https://crates.io/crates/willbooster-tree-sitter-python)
+
+Python grammar for [tree-sitter](https://github.com/tree-sitter/tree-sitter), forked from
+[tree-sitter/tree-sitter-python](https://github.com/tree-sitter/tree-sitter-python). We are grateful
+to its authors and contributors. This is not an official release of that project.
+
+This fork ships the unmodified grammar of upstream v0.25.0 as a compact ABI 16 parser, generated by the parser generator
+of [WillBooster/tree-sitter](https://github.com/WillBooster/tree-sitter).
+
+## Usage
+
+The npm package ships `tree-sitter-python.wasm` for
+[@willbooster/web-tree-sitter](https://www.npmjs.com/package/@willbooster/web-tree-sitter), which runs in Node.js, Bun,
+browsers, and Cloudflare Workers. The compact ABI 16 parser requires runtime 1.3.0 or later.
+
+In Node.js and Bun, load the grammar from its path:
+
+```js
+import { fileURLToPath } from 'node:url';
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+
+await Parser.init();
+const parser = new Parser();
+const wasmPath = fileURLToPath(import.meta.resolve('@willbooster/tree-sitter-python/tree-sitter-python.wasm'));
+parser.setLanguage(await Language.load(wasmPath));
+const tree = parser.parse('def main():\n    pass\n');
+```
+
+In browsers, load the grammar from its URL. With Vite:
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtimeUrl from '@willbooster/web-tree-sitter/web-tree-sitter.wasm?url';
+import pythonUrl from '@willbooster/tree-sitter-python/tree-sitter-python.wasm?url';
+
+await Parser.init({ locateFile: () => runtimeUrl });
+const parser = new Parser();
+parser.setLanguage(await Language.load(pythonUrl));
+```
+
+In Cloudflare Workers, which do not allow compiling Wasm at run time, import both `.wasm` files as modules:
+
+```js
+import { Language, Parser } from '@willbooster/web-tree-sitter';
+import runtime from '@willbooster/web-tree-sitter/web-tree-sitter.wasm';
+import python from '@willbooster/tree-sitter-python/tree-sitter-python.wasm';
+
+await Parser.init({ wasmModule: runtime });
+const parser = new Parser();
+parser.setLanguage(await Language.load(python));
+```
+
+The package also ships the queries in `queries/` and the node types in `src/node-types.json`.
+
+In Rust, depend on the [crate](https://crates.io/crates/willbooster-tree-sitter-python) and on
+[willbooster-tree-sitter](https://crates.io/crates/willbooster-tree-sitter), the runtime this package is tested and
+fuzzed with. The compact ABI 16 parser requires runtime 1.3.0 or later:
+
+```toml
+[dependencies]
+tree-sitter = { package = "willbooster-tree-sitter", version = "1.3.0" }
+tree-sitter-python = { package = "willbooster-tree-sitter-python", version = "1" }
+```
+
+```rust
+let mut parser = tree_sitter::Parser::new();
+parser.set_language(&tree_sitter_python::LANGUAGE.into())?;
+```
+
+## Development
+
+```sh
+mise install
+bun install --frozen-lockfile
+bun run build/ci
+bun run test
+script/parse-examples
+cargo test
+```
+
+The scripts and tests generate, build, test, and parse with `script/tree-sitter`, the tree-sitter CLI of the
+WillBooster/tree-sitter runtime version locked in `Cargo.lock`, since the generator and the runtime of upstream's CLI are
+not the ones this package ships with. Its first run downloads that CLI from the runtime's GitHub Release, or builds it
+with `cargo` (whose build runs the CMake that `mise.toml` pins) when the download fails or the release has no binary that runs here. Run other CLI
+commands through it as well (e.g. `script/tree-sitter parse file.py`).
+
+`bun run generate` records a fresh ABI 16 generation profile from the applicable `test/corpus` cases and Git-tracked
+Python files in `examples/`, then generates compact parser tables. The parser also embeds metadata from `tree-sitter.json`. After changing a grammar,
+`tree-sitter.json`, a corpus case, or a tracked example,
+regenerate and commit `src/`. Stage added or removed examples with `git add -A examples` before generation so the profile uses the intended file list.
+Profiles in `.tmp/generation-profiles/` are temporary and must not be committed. `bun run build-wasm`, `bun run build/ci`,
+and the release build regenerate the parsers before compiling them.
+
+`bun run test` runs:
+
+- the corpus in `test/corpus`, with the native build and with the Wasm build (the first run downloads the WASI SDK);
+- an incremental-parsing check (`test/unit/incremental.test.ts`): `script/fuzz-corpus` runs `tree-sitter fuzz`, which
+  edits each corpus case at random, reparses it, undoes the edits, and reparses again. `TREE_SITTER_SEED`,
+  `TREE_SITTER_ITERATIONS`, and `TREE_SITTER_EDITS` run other or more edits;
+- a check that the real-world Python files in `examples/`, the checked-in ones and those of the cloned repositories,
+  fail to parse exactly as listed in `script/known-failures.txt`. The first run clones the repositories. The example
+  repositories are pinned to commits in `script/parse-examples`. After a grammar change or a moved pin alters that list,
+  `script/parse-examples` rewrites it; review its diff before committing;
+- a performance check (`test/unit/performance.test.ts`) that recovering from an error on each line takes linear time
+  (ten times the lines take about ten times the CPU time, under a ceiling), since consumers parse files while they are
+  being edited. It loads the Wasm build through @willbooster/web-tree-sitter, which `bun run build/ci` rebuilds after
+  regenerating the parser;
+- a check that `package.json` and `Cargo.lock` test the same runtime version (`test/unit/runtimeVersion.test.ts`);
+- checks that the Wasm build parses in Chromium (`test/unit/browser/`) and in Cloudflare Workers with and without
+  Node.js compatibility (`test/unit/workerd.test.ts`). Run `bun run test/ci-setup` once to install Chromium.
+
+The tests and `script/parse-examples` compile the parser into `.tmp/tree-sitter-lib` rather than the CLI's cache shared
+by every checkout; `script/fuzz-corpus` builds a per-run parser in `.tmp/fuzz` and deletes it afterwards.
+
+CI also runs these tests on Linux arm64 and macOS, where the CLI compiles the parser and the scanner with the platform's
+C compiler as the crate does for its users, and fuzzes the parser with libFuzzer and sanitizers
+(`.github/workflows/robustness.yml`).
+
+### References
+
+- [Python 2 Grammar](https://docs.python.org/2/reference/grammar.html)
+- [Python 3 Grammar](https://docs.python.org/3/reference/grammar.html)