rusteerio 0.0.0-stage → 0.1.0

package/Cargo.toml

@@ -0,0 +1,36 @@
+[package]
+name = "rusteerio"
+version = "0.1.0"
+edition = "2021"
+description = "Rust HTML5 parser and CSS selector engine for the rusteerio Node.js package"
+license = "MIT"
+
+[lib]
+crate-type = ["cdylib", "rlib"]
+path = "native/src/lib.rs"
+
+[dependencies]
+ego-tree = "0.10"
+html5ever = "0.35"
+html5gum = "0.8.4"
+url = "2"
+napi = { version = "2", default-features = false, features = ["napi8", "async"] }
+napi-derive = "2"
+rayon = "1.10"
+scraper = { version = "0.24", default-features = false, features = ["atomic", "deterministic"] }
+serde = { version = "1", features = ["derive"] }
+serde_json = { version = "1", features = ["preserve_order"] }
+
+[build-dependencies]
+napi-build = "2"
+
+[profile.release]
+lto = "thin"
+codegen-units = 1
+
+# Observer hooks retain the upstream HTML5 algorithm while exposing the actual
+# open-element stack to user-supplied parse5-compatible tree adapters.
+[patch.crates-io]
+html5ever = { path = "native/vendor/html5ever-0.35.0" }
+markup5ever = { path = "native/vendor/markup5ever-0.35.0" }
+html5gum = { path = "native/vendor/html5gum-0.8.4" }

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Context.dev and rusteerio contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,253 @@
-# Temporary Holding Version
+# rusteerio
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Heavily inspired by [Cheerio](https://cheerio.js.org/)** and its familiar,
+flexible HTML API. Built by the **[context.dev team](https://context.dev)**.
+
+[![CI](https://github.com/context-dot-dev/rusteerio/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/context-dot-dev/rusteerio/actions/workflows/ci.yml)
+[![Cheerio tests](https://img.shields.io/badge/Cheerio_tests-777_passing-brightgreen)](test/upstream/README.md)
+[![npm version](https://img.shields.io/npm/v/rusteerio)](https://www.npmjs.com/package/rusteerio)
+[![MIT license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+[![Built by context.dev](https://img.shields.io/badge/built_by-context.dev-202020)](https://context.dev)
+[![Runtime dependencies](https://img.shields.io/badge/npm_runtime_dependencies-0-brightgreen)](package.json)
+
+Cheerio's interface, a Rust HTML/XML engine, and configurable parallel workers.
+Rusteerio keeps the DOM, parsing, selectors, mutations and serialization in a
+native addon, with **zero npm runtime dependencies**. TypeScript supplies the
+API, stable node views, user callbacks and Node.js I/O.
+
+## Why this exists
+
+Cheerio makes server-side HTML processing approachable. We wanted to keep that
+experience while making large scraping and extraction workloads faster and
+giving applications a way to move CPU work off the JavaScript main thread.
+
+Rusteerio combines a familiar synchronous API with additional async batch APIs.
+Use the synchronous API for ordinary document work, or set a worker limit when
+processing many documents. The implementation is independent of Cheerio's
+JavaScript engine; Cheerio is a development dependency used to check behavior.
+
+## Installation
+
+The npm package name is **`rusteerio`**:
+
+```sh
+npm install rusteerio
+```
+
+Node.js **20.19 or newer** is required. Release tarballs bundle native binaries
+for macOS arm64/x64, Linux glibc and musl arm64/x64, and Windows arm64/x64. Supported
+platforms need no Rust installation and no install script. For a source checkout,
+follow [Development](#development).
+
+Linux GNU binaries require **glibc 2.28 or newer**. Musl binaries are tested on
+**Alpine 3.24**. Linux also needs `libgcc_s.so.1`; minimal Alpine images can
+install it with `apk add --no-cache libgcc`.
+
+## Familiar Cheerio API
+
+```ts
+import * as rusteerio from 'rusteerio';
+
+const $ = rusteerio.load('<ul><li class="fruit">Apple</li></ul>');
+
+$('li').addClass('fresh').attr('data-id', '1');
+console.log($('.fruit').text()); // Apple
+console.log($.html());
+```
+
+CommonJS works too:
+
+```js
+const rusteerio = require('rusteerio');
+const $ = rusteerio.load('<h1>Hello</h1>');
+console.log($('h1').text());
+```
+
+The API includes selectors and traversal, DOM insertion and removal, cloning,
+attributes and properties, classes, styles, form values and serialization,
+HTML/XML output, extraction, buffers, streams and URL loading. The
+`rusteerio/slim` and `rusteerio/utils` entry points are included. TypeScript
+declarations are self-contained, with ESM and CommonJS support.
+
+XML parsing preserves case and supports CDATA:
+
+```ts
+const $ = rusteerio.load('<Root><Item Id="1"/></Root>', { xml: true });
+console.log($('Item').attr('Id')); // 1
+```
+
+## Parallel extraction and worker limits
+
+```ts
+import { configureWorkers, loadAsync, loadAll, extractAll, workerPoolInfo } from 'rusteerio';
+
+// Set one process-wide CPU budget before starting asynchronous native work.
+configureWorkers({ threads: 4 });
+
+const $ = await loadAsync(largeHtml);
+const documents = await loadAll(htmlPages, { threads: 4 });
+
+const rows = await extractAll(htmlPages, {
+  title: 'h1',
+  headings: ['h2'],
+  links: [{ selector: 'a', value: 'href' }],
+}, { threads: 4 });
+
+console.log(workerPoolInfo()); // configured budget, active work and peak usage
+```
+
+| API | Result | Native work |
+| --- | --- | --- |
+| `loadAsync(content, options?, isDocument?)` | `Promise<CheerioAPI>` | Parses off-thread |
+| `loadAll(strings, options?, isDocument?)` | `Promise<CheerioAPI[]>` | Parses documents in parallel |
+| `extractAll(strings, map, options?)` | Extracted objects in input order | Parses and extracts in parallel |
+
+`configureWorkers({ threads: 4 })` (or `configureWorkers(4)`) caps active native
+CPU work across concurrent batches, single-document async parsing and streams.
+Set it before starting asynchronous native work. You can also set
+`RUSTEERIO_THREADS=4` before starting Node. The default is available CPUs, capped
+at 256. Configuration accepts integers from **1 to 256**; changing the budget
+after work starts throws. Reapplying the same budget is allowed.
+
+The per-batch `threads` option can lower a batch's parallelism within that shared
+budget. Results preserve input order. `{ xml: true, threads: 4 }` works too.
+`workerPoolInfo()` reports `threads`, `activeBatches`, `activeWorkers` and
+`peakActiveWorkers`. The cap measures active CPU work; streams can have additional
+parked threads waiting for input or a JavaScript callback.
+
+Synchronous `load()`, queries, mutations and serialization **block the main
+thread** while they run. The async APIs move heavy native work onto workers;
+input copying, result decoding and user callbacks still run on the JavaScript
+thread. Pure extraction maps, including nested maps, execute entirely in Rust
+before their results are decoded. Callback extraction yields between groups of
+elements, preserving closures and callback order. A single expensive callback
+can still block the main thread.
+
+## Streams and parser extensions
+
+`stringStream()` incrementally parses strings in Rust. `decodeStream()` sniffs a
+bounded encoding prefix and decodes incoming byte chunks; `fromURL()` feeds the
+same parser as the response arrives. Writable backpressure bounds queued input.
+HTTP loading supports redirects, abort signals, header/body timeouts and response
+size limits. A caller can supply an Undici-compatible dispatcher.
+
+Custom HTML5 `treeAdapter` methods participate in tree construction, and
+`onParseError` receives parser diagnostics. The HTML/XML tokenizer's custom
+`Tokenizer` constructor and callbacks are supported as well. User code runs in
+JavaScript. Arbitrary custom tokenizers can refer to earlier source offsets, so
+that path retains the source until completion. Source locations retain extra
+metadata; avoid enabling them when you do not need it.
+
+DOM views expose writable Cheerio-compatible fields and stable identity. Removed
+branches stay usable while their selections or nodes are held; unreachable
+branches can be reclaimed and their Rust storage reused. The optional `dom`
+namespace exports constructors such as `dom.Element`, `dom.Text` and
+`dom.Document`. Nodes imported from another DOM retain their original prototype.
+Fresh Rust-backed nodes have Rusteerio's constructors: an `instanceof` check
+against an independently installed `domhandler.Element` requires importing those
+nodes or supplying a tree adapter that creates that class.
+
+## Benchmarks
+
+These measurements compare **Cheerio 1.2.0** against rusteerio on a Linux x64
+**VM with 8 vCPUs and 16 GB RAM**, running Node 24.18.1. Both engines have the
+same enforced CPU budget. Batch comparisons give Cheerio a reused Node worker
+pool and include each engine's input and result conversion. Values are medians
+of nine samples, with equal output checked for all **810 measured samples**.
+
+Rusteerio was faster in **39 of 45 cases**. Representative results, including
+each kind of measured loss:
+
+| Workload | CPUs | Cheerio | rusteerio | Rust latency |
+| --- | ---: | ---: | ---: | --- |
+| Load/query, 500 articles | 1 | 35.25 ms | 4.08 ms | 8.64× faster |
+| Load/query, 5,000 articles | 1 | 150.57 ms | 41.64 ms | 3.62× faster |
+| HTML mutations, 5,000 articles | 1 | 239.65 ms | 189.49 ms | 1.26× faster |
+| Extract 16 documents × 5,000 articles | 1 | 2,887.61 ms | 511.48 ms | 5.65× faster |
+| Extract 16 documents × 5,000 articles | 8 | 797.55 ms | 114.70 ms | 6.95× faster |
+| Nested callback extraction, 8 × 500 articles | 8 | 54.30 ms | 174.06 ms | 3.21× slower |
+| Cold extraction, 16 × 20 articles | 8 | 169.23 ms | 27.37 ms | 6.18× faster |
+| HTML stream, 50,000 elements | 1 | 1,107.78 ms | 346.21 ms | 3.20× faster |
+| XML stream, 50,000 elements | 8 | 119.85 ms | 162.28 ms | 1.35× slower |
+| Append/update/remove, 4,000 cycles | 1 | 593.28 ms | 614.76 ms | 1.04× slower |
+
+Cold measurements include engine import, pool initialization and the first
+batch. Warm extraction excludes initialization. Large extraction improved
+from 511.48 ms with one Rust worker to 114.70 ms with eight, a **4.46×** scaling
+gain. Concurrent batches also respected the shared native CPU cap.
+
+Callbacks remain JavaScript work. Cheerio's benchmark workers execute callbacks
+on their own threads; rusteerio preserves user closures on the calling thread.
+All four callback workloads were slower than the Cheerio worker pool. Native
+async work also involves main-thread conversion: the large eight CPU extraction
+case had 24.06 ms median maximum timer lag, versus 3.49 ms for Cheerio's workers.
+
+Memory results vary. The HTML stream's median end RSS was 372.6 MB for rusteerio
+and 698.5 MB for Cheerio at one CPU; the XML stream used 275.4 MB versus 228.7 MB
+at eight CPUs. Removed-node storage plateaued after warm-up in the retention
+workload, but RSS still grew initially. RSS includes initialization and resident
+input, and sampled peaks do not establish a universal memory bound.
+
+These deterministic generated fixtures describe this VM and revision. Hardware,
+runtime and input shape affect results. See [all 45 results](bench/results/fair/comparison.json)
+and [the methodology, raw samples and reproduction steps](bench/README.md),
+including regressions against the previous Rust implementation.
+
+## Compatibility and tests
+
+The compatibility target is **Cheerio 1.2.0**. All **777 upstream tests across all
+14 files** run against rusteerio, including parser and utility tests. Assertions
+and fixtures are unchanged; implementation imports are redirected. An integrity
+check verifies the complete inventory and source checksums before each run.
+
+The additional **418 runtime and release checks** cover custom parser hooks,
+streaming encodings and HTTP, detached-node retention, extraction callbacks and
+shared CPU limits. **53 Rust tests**, strict TypeScript consumer checks, and clean
+npm tarball installation checks run as well. CI builds and tests all eight
+native targets
+and checks Node 20.19, 22 and 24. GNU binaries are built and run on glibc 2.28;
+release checks inspect their actual ELF version requirements.
+
+See [the upstream test suite](test/upstream/README.md) and [the latest validation
+record](bench/results/fair/validation.json). Compatibility tests establish
+coverage for that version; they do not establish behavior for every possible
+input or private extension.
+
+## Development
+
+Use Node.js 20.19+ and stable Rust:
+
+```sh
+git clone https://github.com/context-dot-dev/rusteerio.git
+cd rusteerio
+npm ci
+npm run build
+npm test
+npm run test:upstream
+npm run check
+npm run test:types
+npm run test:package
+cargo test
+cargo fmt --all -- --check
+cargo clippy --all-targets -- -D warnings
+```
+
+The Rust engine uses html5ever and scraper for HTML5 parsing and fast queries,
+alongside its mutable arena, HTML/XML tokenizer and selector implementation.
+Rust dependencies are compiled into the addon and pinned in `Cargo.lock`.
+Development dependencies supply build tools and compatibility oracles; package
+consumers install no additional npm packages.
+
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for local
+checks and test conventions, and [RELEASING.md](RELEASING.md) for the npm release
+process.
+
+## Credits and license
+
+Created by the **[context.dev team](https://context.dev)**, with thanks to
+[Cheerio and its contributors](https://github.com/cheeriojs/cheerio) for the API
+and upstream test suite that inspired this project.
+
+Rusteerio's original code is **[MIT licensed](LICENSE)**. Vendored Cheerio tests
+retain their [upstream MIT license](test/upstream/LICENSE). Native dependencies
+retain their own terms, collected in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

package/RELEASING.md

@@ -0,0 +1,138 @@
+# Releasing rusteerio
+
+The public npm package name is **`rusteerio`**, maintained by the context.dev
+team. Releases include all eight native binaries, self-contained TypeScript
+declarations, source files and license notices. Consumers install no additional
+npm packages and need no Rust toolchain on supported platforms.
+
+## First release setup
+
+The repository must be publicly visible for the publishing workflow's
+provenance checks. The package name was unpublished when this release setup was
+prepared; a registry lookup does not reserve ownership.
+
+For the first release, use the workflow's explicit bootstrap mode to establish
+ownership of `rusteerio@0.1.0`. It requires the package to be absent from the npm
+registry before any builds start. The default mode uses OIDC and remains token
+free.
+
+Create a granular npm token with **Read and write (publish and stage)** access,
+**Bypass two-factor authentication**, and a one-day expiration. Select only
+`rusteerio` if npm permits it; an unpublished package may require temporary
+**All Packages** access. Organization-management access is unnecessary. Store
+the value only in the GitHub `npm` environment's `NPM_BOOTSTRAP_TOKEN` secret. Never commit the
+token or put its value in commands or logs. See [npm's token
+documentation](https://docs.npmjs.com/creating-and-viewing-access-tokens/).
+
+Prepare and tag the exact release commit as described below, then dispatch:
+
+```sh
+gh workflow run publish.yml \
+  --repo context-dot-dev/rusteerio \
+  --ref main \
+  -f version=0.1.0 \
+  -f npm_tag=latest \
+  -f bootstrap=true
+```
+
+Bootstrap mode runs the same fresh eight-platform matrix, manifest/source
+checks and clean-consumer tests as OIDC mode. Only its final publish step
+receives the token. Both modes publish the exact tested tarball with GitHub
+Actions provenance. A locally built tarball may contain only one platform's
+binary and is not the release artifact.
+
+After the package exists, configure its GitHub Actions trusted publisher in npm
+package settings:
+
+| Field | Value |
+| --- | --- |
+| Organization | `context-dot-dev` |
+| Repository | `rusteerio` |
+| Workflow filename | `publish.yml` |
+| Environment | `npm` |
+| Allowed action | Direct publishing with `npm publish` |
+
+After configuring the trusted publisher, revoke the bootstrap token in npm and
+delete `NPM_BOOTSTRAP_TOKEN` from the GitHub environment. Leave `bootstrap` false
+for subsequent releases; the registry-existence guard prevents its reuse.
+
+Configure the GitHub `npm` environment for maintainers and permit releases from
+`main`. The workflow uses GitHub-hosted runners, Node 24, npm 11.5.1 or newer,
+and OIDC authentication. A new trusted-publisher configuration must complete
+its first publish within two days. See [npm's trusted-publishing
+documentation](https://docs.npmjs.com/trusted-publishers/) for configuration and
+expiry details.
+
+## Prepare a version
+
+Update `package.json`, `package-lock.json`, `Cargo.toml` and `Cargo.lock` to the
+same semantic version. Regenerate the notices with `npm run notices:generate`
+and merge the release changes to `main`. Use `next` for prerelease versions.
+Confirm the build, tests and packaging checks pass.
+
+Create a version tag on the current `main` commit:
+
+```sh
+git switch main
+git pull --ff-only
+git tag v0.1.0
+git push origin v0.1.0
+```
+
+Replace `0.1.0` with the intended version. Dispatch publication before making
+another commit to `main`: the tag, package version, Cargo version and the
+workflow's `main` commit must identify the same release.
+
+## Publish a subsequent release
+
+Run the **Publish to npm** workflow from `main`, with the exact package version
+and npm dist-tag:
+
+```sh
+gh workflow run publish.yml \
+  --repo context-dot-dev/rusteerio \
+  --ref main \
+  -f version=0.1.0 \
+  -f npm_tag=latest
+```
+
+The workflow validates the public repository and exact release commit, runs the
+complete build/test matrix, assembles one tarball, and checks its source,
+license, metadata, checksum and all eight native binary architectures. It
+installs and tests that exact artifact before publishing it with provenance.
+
+Pushes, pull requests, and ordinary CI runs build and test without publishing.
+Publication requires the separate manual workflow and any configured `npm`
+environment protection. Normal OIDC releases require no stored npm token.
+
+## Package checks
+
+For a local development build:
+
+```sh
+npm run build
+npm run test:package
+npm run release:check -- --local
+```
+
+For a complete release artifact containing all targets:
+
+```sh
+npm run release:check -- --tarball release --commit <release-commit>
+npm run test:package -- release
+```
+
+`release:check` rejects missing binaries, incorrect architectures, accidental
+package files, missing notices, version mismatches and npm runtime dependencies.
+The CI-produced `manifest.json` records the commit and tarball checksum for
+publication verification. Provenance is generated by npm's supported CI
+publishing flow; see [npm's provenance
+documentation](https://docs.npmjs.com/generating-provenance-statements/).
+
+Linux GNU builds run inside pinned glibc 2.28 images on both architectures.
+The binary audit reads ELF symbol-version requirements and rejects binaries
+requiring newer glibc, a C++ runtime, or the wrong libc. Windows releases include
+both x64 and arm64; their PE machine headers are checked. The package includes
+the exact patched html5ever/markup5ever/html5gum sources and original notices alongside
+Rusteerio's source. Regenerate third-party notices after any Cargo dependency
+or compiler change.