@xberg-io/crawlberg-wasm 0.0.1 → 1.0.0-rc.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Kreuzberg, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,169 @@
-# @xberg-io/crawlberg-wasm
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
+    <img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
+  </picture>
+</p>
 
-Name-reservation placeholder (v0.0.1). Real releases come from https://github.com/xberg-io/crawlberg.
+# crawlberg
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
+  <a href="https://github.com/xberg-io/alef">
+    <img src="https://img.shields.io/badge/Bindings-alef%20%D7%90-007ec6" alt="Bindings">
+  </a>
+  <!-- Language Bindings -->
+  <a href="https://crates.io/crates/crawlberg">
+    <img src="https://img.shields.io/crates/v/crawlberg?label=Rust&color=007ec6" alt="Rust">
+  </a>
+  <a href="https://pypi.org/project/crawlberg/">
+    <img src="https://img.shields.io/pypi/v/crawlberg?label=Python&color=007ec6" alt="Python">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg?label=Node.js&color=007ec6" alt="Node.js">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg-wasm">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg-wasm?label=WASM&color=007ec6" alt="WASM">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg/crawlberg?label=Java&color=007ec6" alt="Java">
+  </a>
+  <a href="https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go">
+    <img src="https://img.shields.io/github/v/tag/xberg-io/crawlberg?label=Go&color=007ec6" alt="Go">
+  </a>
+  <a href="https://www.nuget.org/packages/XbergIo.Crawlberg/">
+    <img src="https://img.shields.io/nuget/v/XbergIo.Crawlberg?label=C%23&color=007ec6" alt="C#">
+  </a>
+  <a href="https://packagist.org/packages/xberg-io/crawlberg">
+    <img src="https://img.shields.io/packagist/v/xberg-io/crawlberg?label=PHP&color=007ec6" alt="PHP">
+  </a>
+  <a href="https://rubygems.org/gems/crawlberg">
+    <img src="https://img.shields.io/gem/v/crawlberg?label=Ruby&color=007ec6" alt="Ruby">
+  </a>
+  <a href="https://hex.pm/packages/crawlberg">
+    <img src="https://img.shields.io/hexpm/v/crawlberg?label=Elixir&color=007ec6" alt="Elixir">
+  </a>
+  <a href="https://pub.dev/packages/crawlberg">
+    <img src="https://img.shields.io/pub/v/crawlberg?label=Dart&color=007ec6" alt="Dart">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg.android/crawlberg-android?label=Kotlin&color=007ec6" alt="Kotlin">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/swift">
+    <img src="https://img.shields.io/badge/Swift-SPM-007ec6" alt="Swift">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/zig">
+    <img src="https://img.shields.io/badge/Zig-package-007ec6" alt="Zig">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/releases">
+    <img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg">
+    <img src="https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker&logoColor=white" alt="Docker">
+  </a>
+
+  <!-- Project Info -->
+  <a href="https://github.com/xberg-io/crawlberg/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License">
+  </a>
+  <a href="https://docs.crawlberg.xberg.io">
+    <img src="https://img.shields.io/badge/Docs-crawlberg-007ec6" alt="Documentation">
+  </a>
+</div>
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px;">
+  <a href="https://discord.gg/xt9WY3GnKR">
+    <img height="22" src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white" alt="Join Discord">
+  </a>
+</div>
+
+WebAssembly bindings for **crawlberg** — a high-performance Rust web crawling engine. Built with
+wasm-bindgen for use in browsers and Node.js. Single-threaded; features requiring `std::thread`
+are not available.
+
+## What This Package Provides
+
+- **Same crawler as every binding** — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
+- **Structured scrape output** — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
+- **Crawl controls** — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
+- **Rendering path** — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
+- **WASM package** — browser and edge-compatible binding for environments where native libraries are unavailable.
+
+## Installation
+
+```bash
+pnpm add @xberg-io/crawlberg-wasm
+```
+
+## Agent plugin
+
+The `crawlberg` plugin is available via the `xberg-io/plugins` marketplace.
+
+```text
+/plugin marketplace add xberg-io/plugins
+/plugin install crawlberg@xberg
+```
+
+Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See [the marketplace README](https://github.com/xberg-io/plugins) for harness-specific install instructions.
+
+## Quick Start
+
+```javascript title="WASM"
+import { CrawlConfig, crawl, createEngine, scrape } from "@xberg-io/crawlberg-wasm";
+
+async function main() {
+  // Simplest case: scrape a single page with default settings.
+  const engine = createEngine();
+  const result = await scrape(engine, "https://example.com/");
+  console.log(`Title: ${result.metadata?.title ?? ""}`);
+  console.log(`Status: ${result.statusCode}`);
+  console.log(`Links found: ${result.links?.length ?? 0}`);
+
+  // Crawl from a seed URL, limited to one hop and a handful of pages.
+  const config = new CrawlConfig();
+  config.maxDepth = 1;
+  config.maxPages = 5;
+  const crawlEngine = createEngine(config);
+  const crawlResult = await crawl(crawlEngine, "https://en.wikipedia.org/wiki/Web_scraping");
+  console.log(`Pages crawled: ${crawlResult.pages?.length ?? 0}`);
+}
+
+main().catch((error) => console.error(error));
+```
+
+## API Reference
+
+Full API documentation is available at [docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io).
+
+Key functions:
+
+- `create_engine(config?)` — Create a crawl engine with optional configuration
+- `scrape(engine, url)` — Scrape a single URL
+- `crawl(engine, url)` — Crawl a website following links
+- `map_urls(engine, url)` — Discover all pages on a site
+- `batch_scrape(engine, urls)` — Scrape multiple URLs concurrently
+- `batch_crawl(engine, urls)` — Crawl multiple seed URLs concurrently
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md) for details.
+
+## Part of Xberg.dev
+
+- [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
+- [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
+- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
+- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
+- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers.
+- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
+- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
+- [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
+
+## License
+
+This project is licensed under [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE).
+
+## Links
+
+- [Documentation](https://docs.crawlberg.xberg.io)
+- [GitHub Repository](https://github.com/xberg-io/crawlberg)
+- [Issue Tracker](https://github.com/xberg-io/crawlberg/issues)

package/package.json

@@ -1,10 +1,38 @@
 {
   "name": "@xberg-io/crawlberg-wasm",
-  "version": "0.0.1",
-  "description": "Placeholder to reserve @xberg-io/crawlberg-wasm. Real releases are published from https://github.com/xberg-io/crawlberg.",
+  "version": "1.0.0-rc.2",
+  "private": false,
+  "description": "High-performance web crawling engine",
   "license": "MIT",
-  "repository": { "type": "git", "url": "git+https://github.com/xberg-io/crawlberg.git" },
-  "homepage": "https://xberg.io",
-  "publishConfig": { "access": "public" },
-  "main": "index.js"
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/xberg-io/crawlberg",
+    "directory": "crates/crawlberg-wasm"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "type": "module",
+  "files": [
+    "pkg",
+    "*.wasm",
+    "*.d.ts",
+    "README.md"
+  ],
+  "main": "pkg/nodejs/crawlberg_wasm.js",
+  "module": "pkg/web/crawlberg_wasm.js",
+  "types": "pkg/nodejs/crawlberg_wasm.d.ts",
+  "scripts": {
+    "build": "wasm-pack build --target nodejs --out-dir pkg/nodejs",
+    "build:ci": "wasm-pack build --release --target nodejs --out-dir pkg/nodejs",
+    "build:wasm:web": "wasm-pack build --release --target web --out-dir pkg/web",
+    "build:wasm:bundler": "wasm-pack build --release --target bundler --out-dir pkg/bundler",
+    "build:wasm:nodejs": "wasm-pack build --release --target nodejs --out-dir pkg/nodejs",
+    "build:wasm:deno": "wasm-pack build --release --target deno --out-dir pkg/deno",
+    "build:all": "npm run build:wasm:web && npm run build:wasm:bundler && npm run build:wasm:nodejs && npm run build:wasm:deno && find pkg -name .gitignore -delete",
+    "test": "vitest run",
+    "test:watch": "vitest watch",
+    "test:coverage": "vitest run --coverage",
+    "clean": "rm -rf pkg dist"
+  }
 }

package/pkg/bundler/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Kreuzberg, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/pkg/bundler/README.md

@@ -0,0 +1,169 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
+    <img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
+  </picture>
+</p>
+
+# crawlberg
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
+  <a href="https://github.com/xberg-io/alef">
+    <img src="https://img.shields.io/badge/Bindings-alef%20%D7%90-007ec6" alt="Bindings">
+  </a>
+  <!-- Language Bindings -->
+  <a href="https://crates.io/crates/crawlberg">
+    <img src="https://img.shields.io/crates/v/crawlberg?label=Rust&color=007ec6" alt="Rust">
+  </a>
+  <a href="https://pypi.org/project/crawlberg/">
+    <img src="https://img.shields.io/pypi/v/crawlberg?label=Python&color=007ec6" alt="Python">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg?label=Node.js&color=007ec6" alt="Node.js">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg-wasm">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg-wasm?label=WASM&color=007ec6" alt="WASM">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg/crawlberg?label=Java&color=007ec6" alt="Java">
+  </a>
+  <a href="https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go">
+    <img src="https://img.shields.io/github/v/tag/xberg-io/crawlberg?label=Go&color=007ec6" alt="Go">
+  </a>
+  <a href="https://www.nuget.org/packages/XbergIo.Crawlberg/">
+    <img src="https://img.shields.io/nuget/v/XbergIo.Crawlberg?label=C%23&color=007ec6" alt="C#">
+  </a>
+  <a href="https://packagist.org/packages/xberg-io/crawlberg">
+    <img src="https://img.shields.io/packagist/v/xberg-io/crawlberg?label=PHP&color=007ec6" alt="PHP">
+  </a>
+  <a href="https://rubygems.org/gems/crawlberg">
+    <img src="https://img.shields.io/gem/v/crawlberg?label=Ruby&color=007ec6" alt="Ruby">
+  </a>
+  <a href="https://hex.pm/packages/crawlberg">
+    <img src="https://img.shields.io/hexpm/v/crawlberg?label=Elixir&color=007ec6" alt="Elixir">
+  </a>
+  <a href="https://pub.dev/packages/crawlberg">
+    <img src="https://img.shields.io/pub/v/crawlberg?label=Dart&color=007ec6" alt="Dart">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg.android/crawlberg-android?label=Kotlin&color=007ec6" alt="Kotlin">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/swift">
+    <img src="https://img.shields.io/badge/Swift-SPM-007ec6" alt="Swift">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/zig">
+    <img src="https://img.shields.io/badge/Zig-package-007ec6" alt="Zig">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/releases">
+    <img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg">
+    <img src="https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker&logoColor=white" alt="Docker">
+  </a>
+
+  <!-- Project Info -->
+  <a href="https://github.com/xberg-io/crawlberg/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License">
+  </a>
+  <a href="https://docs.crawlberg.xberg.io">
+    <img src="https://img.shields.io/badge/Docs-crawlberg-007ec6" alt="Documentation">
+  </a>
+</div>
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px;">
+  <a href="https://discord.gg/xt9WY3GnKR">
+    <img height="22" src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white" alt="Join Discord">
+  </a>
+</div>
+
+WebAssembly bindings for **crawlberg** — a high-performance Rust web crawling engine. Built with
+wasm-bindgen for use in browsers and Node.js. Single-threaded; features requiring `std::thread`
+are not available.
+
+## What This Package Provides
+
+- **Same crawler as every binding** — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
+- **Structured scrape output** — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
+- **Crawl controls** — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
+- **Rendering path** — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
+- **WASM package** — browser and edge-compatible binding for environments where native libraries are unavailable.
+
+## Installation
+
+```bash
+pnpm add @xberg-io/crawlberg-wasm
+```
+
+## Agent plugin
+
+The `crawlberg` plugin is available via the `xberg-io/plugins` marketplace.
+
+```text
+/plugin marketplace add xberg-io/plugins
+/plugin install crawlberg@xberg
+```
+
+Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See [the marketplace README](https://github.com/xberg-io/plugins) for harness-specific install instructions.
+
+## Quick Start
+
+```javascript title="WASM"
+import { CrawlConfig, crawl, createEngine, scrape } from "@xberg-io/crawlberg-wasm";
+
+async function main() {
+  // Simplest case: scrape a single page with default settings.
+  const engine = createEngine();
+  const result = await scrape(engine, "https://example.com/");
+  console.log(`Title: ${result.metadata?.title ?? ""}`);
+  console.log(`Status: ${result.statusCode}`);
+  console.log(`Links found: ${result.links?.length ?? 0}`);
+
+  // Crawl from a seed URL, limited to one hop and a handful of pages.
+  const config = new CrawlConfig();
+  config.maxDepth = 1;
+  config.maxPages = 5;
+  const crawlEngine = createEngine(config);
+  const crawlResult = await crawl(crawlEngine, "https://en.wikipedia.org/wiki/Web_scraping");
+  console.log(`Pages crawled: ${crawlResult.pages?.length ?? 0}`);
+}
+
+main().catch((error) => console.error(error));
+```
+
+## API Reference
+
+Full API documentation is available at [docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io).
+
+Key functions:
+
+- `create_engine(config?)` — Create a crawl engine with optional configuration
+- `scrape(engine, url)` — Scrape a single URL
+- `crawl(engine, url)` — Crawl a website following links
+- `map_urls(engine, url)` — Discover all pages on a site
+- `batch_scrape(engine, urls)` — Scrape multiple URLs concurrently
+- `batch_crawl(engine, urls)` — Crawl multiple seed URLs concurrently
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md) for details.
+
+## Part of Xberg.dev
+
+- [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
+- [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
+- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
+- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
+- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers.
+- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
+- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
+- [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
+
+## License
+
+This project is licensed under [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE).
+
+## Links
+
+- [Documentation](https://docs.crawlberg.xberg.io)
+- [GitHub Repository](https://github.com/xberg-io/crawlberg)
+- [Issue Tracker](https://github.com/xberg-io/crawlberg/issues)