crawlberg 0.0.1 → 1.0.0.pre.rc.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84d24f5fa9af1d0b4b70254ba190b3cc949c51c6383f4242ec13dbce8fdb6c3b
-  data.tar.gz: 40281a66a14d43b1ea9da7f3142ef35f775b1a97fd69f138fe25b602b5a58ed1
+  metadata.gz: 07722edb64b4714a652ad533eb1594e52ee56e172c5f62e911870a7b7a00cf49
+  data.tar.gz: 312938f63afed7f64edb9765493e72242b186ff473e181ec27ffa570c5c3e86d
 SHA512:
-  metadata.gz: bb5dd17b7ca11213404ed014056027c1aabbef4bf9ced1af345b5ae8ccebe42e75934c07a5d1dc1665f7ca9ed67fe24b2fd4ae126d9239187ba4b7dc69708e0d
-  data.tar.gz: 76515a688aa854dc7514bbd4c1e5716f46614af40ebab1567340a4fbae00cdc137eec9941c58ca7473868bc507b4574e4ea358eed7fa80079a7fce70de58f0bf
+  metadata.gz: 6db2096db0dea5bae3f9d1b211a64fe29350d58a66d4b18474ecb37ac148fcf0c2c63f27cfeaf1254ad33dee7c8f62ad002f6fb933e9ddec74f069ed633aa076
+  data.tar.gz: f3474d90832cab725e874a122a89bace6e2f85df4ffc56bd169a8e57a171f6bc5d9f281657704768570d70f51d25840e29d64548ac591a303e2e55da4a90c804

data/README.md

@@ -1,3 +1,162 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
+    <img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
+  </picture>
+</p>
+
 # crawlberg
 
-Reserved name. See https://github.com/xberg-io/crawlberg
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
+  <a href="https://github.com/xberg-io/alef">
+    <img src="https://img.shields.io/badge/Bindings-alef%20%D7%90-007ec6" alt="Bindings">
+  </a>
+  <!-- Language Bindings -->
+  <a href="https://crates.io/crates/crawlberg">
+    <img src="https://img.shields.io/crates/v/crawlberg?label=Rust&color=007ec6" alt="Rust">
+  </a>
+  <a href="https://pypi.org/project/crawlberg/">
+    <img src="https://img.shields.io/pypi/v/crawlberg?label=Python&color=007ec6" alt="Python">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg?label=Node.js&color=007ec6" alt="Node.js">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg-wasm">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg-wasm?label=WASM&color=007ec6" alt="WASM">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg/crawlberg?label=Java&color=007ec6" alt="Java">
+  </a>
+  <a href="https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go">
+    <img src="https://img.shields.io/github/v/tag/xberg-io/crawlberg?label=Go&color=007ec6" alt="Go">
+  </a>
+  <a href="https://www.nuget.org/packages/XbergIo.Crawlberg/">
+    <img src="https://img.shields.io/nuget/v/XbergIo.Crawlberg?label=C%23&color=007ec6" alt="C#">
+  </a>
+  <a href="https://packagist.org/packages/xberg-io/crawlberg">
+    <img src="https://img.shields.io/packagist/v/xberg-io/crawlberg?label=PHP&color=007ec6" alt="PHP">
+  </a>
+  <a href="https://rubygems.org/gems/crawlberg">
+    <img src="https://img.shields.io/gem/v/crawlberg?label=Ruby&color=007ec6" alt="Ruby">
+  </a>
+  <a href="https://hex.pm/packages/crawlberg">
+    <img src="https://img.shields.io/hexpm/v/crawlberg?label=Elixir&color=007ec6" alt="Elixir">
+  </a>
+  <a href="https://pub.dev/packages/crawlberg">
+    <img src="https://img.shields.io/pub/v/crawlberg?label=Dart&color=007ec6" alt="Dart">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg.android/crawlberg-android?label=Kotlin&color=007ec6" alt="Kotlin">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/swift">
+    <img src="https://img.shields.io/badge/Swift-SPM-007ec6" alt="Swift">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/zig">
+    <img src="https://img.shields.io/badge/Zig-package-007ec6" alt="Zig">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/releases">
+    <img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg">
+    <img src="https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker&logoColor=white" alt="Docker">
+  </a>
+
+  <!-- Project Info -->
+  <a href="https://github.com/xberg-io/crawlberg/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License">
+  </a>
+  <a href="https://docs.crawlberg.xberg.io">
+    <img src="https://img.shields.io/badge/Docs-crawlberg-007ec6" alt="Documentation">
+  </a>
+</div>
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px;">
+  <a href="https://discord.gg/xt9WY3GnKR">
+    <img height="22" src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white" alt="Join Discord">
+  </a>
+</div>
+
+Ruby bindings for **crawlberg** — a high-performance Rust web crawling engine. Powered by
+Magnus with native Ruby objects, full metadata extraction, and Markdown conversion.
+
+## What This Package Provides
+
+- **Same crawler as every binding** — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
+- **Structured scrape output** — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
+- **Crawl controls** — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
+- **Rendering path** — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
+- **Ruby package** — Magnus-backed native extension with Ruby objects for crawl results.
+
+## Installation
+
+```bash
+gem install crawlberg
+```
+
+## Agent plugin
+
+The `crawlberg` plugin is available via the `xberg-io/plugins` marketplace.
+
+```text
+/plugin marketplace add xberg-io/plugins
+/plugin install crawlberg@xberg
+```
+
+Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See [the marketplace README](https://github.com/xberg-io/plugins) for harness-specific install instructions.
+
+## Quick Start
+
+```ruby title="Ruby"
+require "crawlberg"
+
+# Simplest case: scrape a single page with default settings.
+engine = Crawlberg.create_engine
+result = Crawlberg.scrape(engine, "https://example.com/")
+puts "Title: #{result.metadata.title}"
+puts "Status: #{result.status_code}"
+puts "Links found: #{result.links.length}"
+
+# Crawl from a seed URL, limited to one hop and a handful of pages.
+config = Crawlberg::CrawlConfig.new(max_depth: 1, max_pages: 5)
+crawl_engine = Crawlberg.create_engine(config)
+crawl_result = Crawlberg.crawl(crawl_engine, "https://en.wikipedia.org/wiki/Web_scraping")
+puts "Pages crawled: #{crawl_result.pages.length}"
+```
+
+## API Reference
+
+Full API documentation is available at [docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io).
+
+Key functions:
+
+- `create_engine(config?)` — Create a crawl engine with optional configuration
+- `scrape(engine, url)` — Scrape a single URL
+- `crawl(engine, url)` — Crawl a website following links
+- `map_urls(engine, url)` — Discover all pages on a site
+- `batch_scrape(engine, urls)` — Scrape multiple URLs concurrently
+- `batch_crawl(engine, urls)` — Crawl multiple seed URLs concurrently
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md) for details.
+
+## Part of Xberg.dev
+
+- [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
+- [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
+- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
+- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
+- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers.
+- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
+- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
+- [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
+
+## License
+
+This project is licensed under [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE).
+
+## Links
+
+- [Documentation](https://docs.crawlberg.xberg.io)
+- [GitHub Repository](https://github.com/xberg-io/crawlberg)
+- [Issue Tracker](https://github.com/xberg-io/crawlberg/issues)

data/Steepfile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+target :lib do
+  signature "sig"
+  check "lib"
+  # The generated `lib/crawlberg/native.rb` carries inline Sorbet
+  # `sig { ... }` blocks on tagged-enum variant Data classes. Sorbet's runtime
+  # provides those via `extend T::Sig`, but Steep does not understand the
+  # extension (it relies on RBS, not Sorbet sigs) and reports
+  # `Type `self` does not have method `sig`` on every block. RBS coverage
+  # for the same surface lives in `sig/types.rbs`, so we steer Steep to the
+  # RBS file by ignoring the .rb.
+  ignore "lib/crawlberg/native.rb"
+end