crawlberg 1.11.0-arm64-darwin

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2e2330884c50bfcf445600f5820b231c54533148e99917622c88126786b6b8f2
+  data.tar.gz: b53631853b37281f99de76a82cf7c22ebd31c8262232245ccb141243c7f60343
+SHA512:
+  metadata.gz: a9fa55edd67c4deca984fdcfbece10248c4aea62c7911fc04c05207e8cc95b85d924acd587b3dcb901f078a449fec655549ff31d114d42217d54400e9e3005cd
+  data.tar.gz: 30eada1bffc5550df3081cbcef7a8c99cc852ab88a7ba11a793b439a5765f125c948f9e73df20c70240de212918d8ae94e75d37a4a4d0ea98843c5cd1a092b49

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Kreuzberg, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,168 @@
+<!-- This file is auto-generated by alef — DO NOT EDIT. -->
+<!-- alef:hash:fcb23a304253e2719260285609af11e93960b21fb56072d6816533f452223e9c -->
+<!-- To regenerate: alef readme -->
+<!-- To verify freshness: alef verify -->
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
+    <img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
+  </picture>
+</p>
+
+# crawlberg
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;">
+  <a href="https://github.com/xberg-io/alef">
+    <img src="https://img.shields.io/badge/built%20with-alef%20%D7%90-007ec6" alt="Built with alef">
+  </a>
+  <!-- Language Bindings -->
+  <a href="https://crates.io/crates/crawlberg">
+    <img src="https://img.shields.io/crates/v/crawlberg?label=Rust&color=007ec6" alt="Rust">
+  </a>
+  <a href="https://pypi.org/project/crawlberg/">
+    <img src="https://img.shields.io/pypi/v/crawlberg?label=Python&color=007ec6" alt="Python">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg?label=Node.js&color=007ec6" alt="Node.js">
+  </a>
+  <a href="https://www.npmjs.com/package/@xberg-io/crawlberg-wasm">
+    <img src="https://img.shields.io/npm/v/@xberg-io/crawlberg-wasm?label=WASM&color=007ec6" alt="WASM">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg/crawlberg">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg/crawlberg?label=Java&color=007ec6" alt="Java">
+  </a>
+  <a href="https://pkg.go.dev/github.com/xberg-io/crawlberg/packages/go">
+    <img src="https://img.shields.io/github/v/tag/xberg-io/crawlberg?label=Go&color=007ec6" alt="Go">
+  </a>
+  <a href="https://www.nuget.org/packages/XbergIo.Crawlberg/">
+    <img src="https://img.shields.io/nuget/v/XbergIo.Crawlberg?label=C%23&color=007ec6" alt="C#">
+  </a>
+  <a href="https://packagist.org/packages/xberg-io/crawlberg">
+    <img src="https://img.shields.io/packagist/v/xberg-io/crawlberg?label=PHP&color=007ec6" alt="PHP">
+  </a>
+  <a href="https://rubygems.org/gems/crawlberg">
+    <img src="https://img.shields.io/gem/v/crawlberg?label=Ruby&color=007ec6" alt="Ruby">
+  </a>
+  <a href="https://hex.pm/packages/crawlberg">
+    <img src="https://img.shields.io/hexpm/v/crawlberg?label=Elixir&color=007ec6" alt="Elixir">
+  </a>
+  <a href="https://pub.dev/packages/crawlberg">
+    <img src="https://img.shields.io/pub/v/crawlberg?label=Dart&color=007ec6" alt="Dart">
+  </a>
+  <a href="https://central.sonatype.com/artifact/io.xberg.crawlberg.android/crawlberg-android">
+    <img src="https://img.shields.io/maven-central/v/io.xberg.crawlberg.android/crawlberg-android?label=Kotlin&color=007ec6" alt="Kotlin">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/swift">
+    <img src="https://img.shields.io/badge/Swift-SPM-007ec6" alt="Swift">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/tree/main/packages/zig">
+    <img src="https://img.shields.io/badge/Zig-package-007ec6" alt="Zig">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/releases">
+    <img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI">
+  </a>
+  <a href="https://github.com/xberg-io/crawlberg/pkgs/container/crawlberg">
+    <img src="https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker&logoColor=white" alt="Docker">
+  </a>
+
+  <!-- Project Info -->
+  <a href="https://github.com/xberg-io/crawlberg/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License">
+  </a>
+  <a href="https://docs.crawlberg.xberg.io">
+    <img src="https://img.shields.io/badge/Docs-crawlberg-007ec6" alt="Documentation">
+  </a>
+</div>
+
+<div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px;">
+  <a href="https://discord.gg/xt9WY3GnKR">
+    <img height="22" src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white" alt="Join Discord">
+  </a>
+</div>
+
+Ruby bindings for **crawlberg** — a high-performance Rust web crawling engine. Powered by
+Magnus with native Ruby objects, full metadata extraction, and Markdown conversion.
+
+## What This Package Provides
+
+- **Same crawler as every binding** — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
+- **Structured scrape output** — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
+- **Crawl controls** — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
+- **Rendering path** — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
+- **Ruby package** — Magnus-backed native extension with Ruby objects for crawl results.
+
+## Installation
+
+```bash
+gem install crawlberg
+```
+
+## Agent plugin
+
+The `crawlberg` plugin is available via the `xberg-io/crawlberg` marketplace.
+
+```text
+/plugin marketplace add xberg-io/crawlberg
+/plugin install crawlberg@crawlberg
+```
+
+See [the plugin guide](https://github.com/xberg-io/crawlberg/blob/main/plugin/README.md) for all supported runtimes, CLI prerequisites and standalone public skills.
+
+## Quick Start
+
+```ruby title="Ruby"
+require "crawlberg"
+
+# Simplest case: scrape a single page with default settings.
+engine = Crawlberg.create_engine
+result = Crawlberg.scrape(engine, "https://example.com/")
+puts "Title: #{result.metadata.title}"
+puts "Status: #{result.status_code}"
+puts "Links found: #{result.links.length}"
+
+# Crawl from a seed URL, limited to one hop and a handful of pages.
+config = Crawlberg::CrawlConfig.new(max_depth: 1, max_pages: 5)
+crawl_engine = Crawlberg.create_engine(config)
+crawl_result = Crawlberg.crawl(crawl_engine, "https://en.wikipedia.org/wiki/Web_scraping")
+puts "Pages crawled: #{crawl_result.pages.length}"
+```
+
+## API Reference
+
+Full API documentation is available at [docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io).
+
+Key functions:
+
+- `create_engine(config?)` — Create a crawl engine with optional configuration
+- `scrape(engine, url)` — Scrape a single URL
+- `crawl(engine, url)` — Crawl a website following links
+- `map_urls(engine, url)` — Discover all pages on a site
+- `batch_scrape(engine, urls)` — Scrape multiple URLs concurrently
+- `batch_crawl(engine, urls)` — Crawl multiple seed URLs concurrently
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md) for details.
+
+## Part of Xberg.io
+
+- [Xberg](https://github.com/xberg-io/xberg) — the open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence. MIT.
+- [Xberg Pro](https://xberg.io) — a complete self-hosted content-intelligence backend in a single container. Commercial.
+- [Xberg Enterprise](https://xberg.io) — the distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.
+- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
+- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
+- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 165 providers.
+- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
+- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
+- [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
+
+## License
+
+This project is licensed under [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE).
+
+## Links
+
+- [Documentation](https://docs.crawlberg.xberg.io)
+- [GitHub Repository](https://github.com/xberg-io/crawlberg)
+- [Issue Tracker](https://github.com/xberg-io/crawlberg/issues)

data/Steepfile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+target :lib do
+  signature "sig"
+  check "lib"
+  # The generated `lib/crawlberg/native.rb` carries inline Sorbet
+  # `sig { ... }` blocks on tagged-enum variant Data classes. Sorbet's runtime
+  # provides those via `extend T::Sig`, but Steep does not understand the
+  # extension (it relies on RBS, not Sorbet sigs) and reports
+  # `Type `self` does not have method `sig`` on every block. RBS coverage
+  # for the same surface lives in `sig/types.rbs`, so we steer Steep to the
+  # RBS file by ignoring the .rb.
+  ignore "lib/crawlberg/native.rb"
+end