@xberg-io/crawlberg 0.0.1 → 1.0.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +167 -2
  3. package/index.d.ts +1079 -0
  4. package/index.js +624 -2
  5. package/package.json +58 -6
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2025-2026 Kreuzberg, Inc.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,168 @@
1
- # @xberg-io/crawlberg
1
+ <p align="center">
2
+ <picture>
3
+ <source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
4
+ <img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
5
+ </picture>
6
+ </p>
2
7
 
3
- Name-reservation placeholder (v0.0.1). Real releases come from https://github.com/xberg-io/crawlberg.
8
+ # crawlberg
9
+
10
+ <div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
11
+ <a href="https://github.com/xberg-io/alef">
12
+ <img src="https://img.shields.io/badge/Bindings-alef%20%D7%90-007ec6" alt="Bindings">
13
+ </a>
14
+ <!-- Language Bindings -->
15
+ <a href="https://crates.io/crates/crawlberg">
16
+ <img src="https://img.shields.io/crates/v/crawlberg?label=Rust&color=007ec6" alt="Rust">
17
+ </a>
18
+ <a href="https://pypi.org/project/crawlberg/">
19
+ <img src="https://img.shields.io/pypi/v/crawlberg?label=Python&color=007ec6" alt="Python">
20
+ </a>
21
+ <a href="https://www.npmjs.com/package/@xberg-io/crawlberg">
22
+ <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg?label=Node.js&color=007ec6" alt="Node.js">
23
+ </a>
24
+ <a href="https://www.npmjs.com/package/@xberg-io/crawlberg-wasm">
25
+ <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg-wasm?label=WASM&color=007ec6" alt="WASM">
26
+ </a>
27
+ <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg">
28
+ <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg/crawlberg?label=Java&color=007ec6" alt="Java">
29
+ </a>
30
+ <a href="https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go">
31
+ <img src="https://img.shields.io/github/v/tag/xberg-io/crawlberg?label=Go&color=007ec6" alt="Go">
32
+ </a>
33
+ <a href="https://www.nuget.org/packages/Crawlberg/">
34
+ <img src="https://img.shields.io/nuget/v/Crawlberg?label=C%23&color=007ec6" alt="C#">
35
+ </a>
36
+ <a href="https://packagist.org/packages/xberg-io/crawlberg">
37
+ <img src="https://img.shields.io/packagist/v/xberg-io/crawlberg?label=PHP&color=007ec6" alt="PHP">
38
+ </a>
39
+ <a href="https://rubygems.org/gems/crawlberg">
40
+ <img src="https://img.shields.io/gem/v/crawlberg?label=Ruby&color=007ec6" alt="Ruby">
41
+ </a>
42
+ <a href="https://hex.pm/packages/crawlberg">
43
+ <img src="https://img.shields.io/hexpm/v/crawlberg?label=Elixir&color=007ec6" alt="Elixir">
44
+ </a>
45
+ <a href="https://pub.dev/packages/crawlberg">
46
+ <img src="https://img.shields.io/pub/v/crawlberg?label=Dart&color=007ec6" alt="Dart">
47
+ </a>
48
+ <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android">
49
+ <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg.android/crawlberg-android?label=Kotlin&color=007ec6" alt="Kotlin">
50
+ </a>
51
+ <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/swift">
52
+ <img src="https://img.shields.io/badge/Swift-SPM-007ec6" alt="Swift">
53
+ </a>
54
+ <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/zig">
55
+ <img src="https://img.shields.io/badge/Zig-package-007ec6" alt="Zig">
56
+ </a>
57
+ <a href="https://github.com/xberg-io/crawlberg/releases">
58
+ <img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI">
59
+ </a>
60
+ <a href="https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg">
61
+ <img src="https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker&logoColor=white" alt="Docker">
62
+ </a>
63
+
64
+ <!-- Project Info -->
65
+ <a href="https://github.com/xberg-io/crawlberg/blob/main/LICENSE">
66
+ <img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License">
67
+ </a>
68
+ <a href="https://docs.crawlberg.xberg.io">
69
+ <img src="https://img.shields.io/badge/Docs-crawlberg-007ec6" alt="Documentation">
70
+ </a>
71
+ </div>
72
+
73
+ <div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px;">
74
+ <a href="https://discord.gg/xt9WY3GnKR">
75
+ <img height="22" src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white" alt="Join Discord">
76
+ </a>
77
+ </div>
78
+
79
+ Node.js bindings for **crawlberg** — a high-performance Rust web crawling engine. Powered by
80
+ NAPI-RS with full TypeScript types, async/await support, and native performance.
81
+
82
+ ## What This Package Provides
83
+
84
+ - **Same crawler as every binding** — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
85
+ - **Structured scrape output** — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
86
+ - **Crawl controls** — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
87
+ - **Rendering path** — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
88
+ - **Node-first TypeScript API** — NAPI-RS package with typed options/results and async/await.
89
+
90
+ ## Installation
91
+
92
+ ```bash
93
+ pnpm add @xberg-io/crawlberg
94
+ ```
95
+
96
+ ## Agent plugin
97
+
98
+ The `crawlberg` plugin is available via the `xberg-io/plugins` marketplace.
99
+
100
+ ```text
101
+ /plugin marketplace add xberg-io/plugins
102
+ /plugin install crawlberg@xberg
103
+ ```
104
+
105
+ Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See [the marketplace README](https://github.com/xberg-io/plugins) for harness-specific install instructions.
106
+
107
+ ## Quick Start
108
+
109
+ ```typescript title="TypeScript"
110
+ import { crawl, createEngine, scrape } from "@xberg-io/crawlberg";
111
+
112
+ async function main(): Promise<void> {
113
+ // Simplest case: scrape a single page with default settings.
114
+ const engine = createEngine();
115
+ const result = await scrape(engine, "https://example.com/");
116
+ console.log(`Title: ${result.metadata?.title ?? ""}`);
117
+ console.log(`Status: ${result.statusCode}`);
118
+ console.log(`Links found: ${result.links?.length ?? 0}`);
119
+
120
+ // Crawl from a seed URL, limited to one hop and a handful of pages.
121
+ const crawlEngine = createEngine({ maxDepth: 1, maxPages: 5 });
122
+ const crawlResult = await crawl(crawlEngine, "https://en.wikipedia.org/wiki/Web_scraping");
123
+ console.log(`Pages crawled: ${crawlResult.pages?.length ?? 0}`);
124
+ }
125
+
126
+ main().catch((error) => {
127
+ console.error(error);
128
+ process.exit(1);
129
+ });
130
+ ```
131
+
132
+ ## API Reference
133
+
134
+ Full API documentation is available at [docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io).
135
+
136
+ Key functions:
137
+
138
+ - `create_engine(config?)` — Create a crawl engine with optional configuration
139
+ - `scrape(engine, url)` — Scrape a single URL
140
+ - `crawl(engine, url)` — Crawl a website following links
141
+ - `map_urls(engine, url)` — Discover all pages on a site
142
+ - `batch_scrape(engine, urls)` — Scrape multiple URLs concurrently
143
+ - `batch_crawl(engine, urls)` — Crawl multiple seed URLs concurrently
144
+
145
+ ## Contributing
146
+
147
+ Contributions are welcome! Please see our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md) for details.
148
+
149
+ ## Part of Xberg.dev
150
+
151
+ - [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
152
+ - [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
153
+ - [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
154
+ - [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
155
+ - [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers.
156
+ - [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
157
+ - [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
158
+ - [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
159
+
160
+ ## License
161
+
162
+ This project is licensed under [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE).
163
+
164
+ ## Links
165
+
166
+ - [Documentation](https://docs.crawlberg.xberg.io)
167
+ - [GitHub Repository](https://github.com/xberg-io/crawlberg)
168
+ - [Issue Tracker](https://github.com/xberg-io/crawlberg/issues)