bytecrawl 1.0.0__tar.gz

bytecrawl-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abraham Pérez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bytecrawl-1.0.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.4
+Name: bytecrawl
+Version: 1.0.0
+Summary: Focused crawling for LLM data collection: Shark-Search and OPIC crawlers plus a single scraping API (static HTML, dynamic JS, APIs, session login, Markdown for LLMs) on a 3-dependency core.
+Author-email: abrahamperz <aperez@adokmx.com>
+License: MIT
+Project-URL: Homepage, https://github.com/abrahamperz/ByteCrawl
+Project-URL: Repository, https://github.com/abrahamperz/ByteCrawl
+Keywords: scraping,crawler,focused-crawling,shark-search,opic,pagerank,beautifulsoup,playwright,markdown,llm
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Intended Audience :: Developers
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28
+Requires-Dist: beautifulsoup4>=4.11
+Requires-Dist: lxml>=4.9
+Provides-Extra: browser
+Requires-Dist: playwright>=1.40; extra == "browser"
+Provides-Extra: llm
+Requires-Dist: markdownify>=0.11; extra == "llm"
+Requires-Dist: trafilatura>=1.6; extra == "llm"
+Provides-Extra: mcp
+Requires-Dist: mcp>=2; python_version >= "3.10" and extra == "mcp"
+Requires-Dist: markdownify>=0.11; extra == "mcp"
+Requires-Dist: trafilatura>=1.6; extra == "mcp"
+Provides-Extra: all
+Requires-Dist: playwright>=1.40; extra == "all"
+Requires-Dist: markdownify>=0.11; extra == "all"
+Requires-Dist: trafilatura>=1.6; extra == "all"
+Requires-Dist: mcp>=2; python_version >= "3.10" and extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Dynamic: license-file
+
+# ByteCrawl
+
+[![CI](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml/badge.svg)](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/bytecrawl)](https://pypi.org/project/bytecrawl/)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/bytecrawl/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+**Give your AI agent focused web crawling.** ByteCrawl is an MCP server (and a
+small Python library) that doesn't just scrape a page — it crawls a whole site
+and returns the pages *most relevant* to your topic first, using Shark-Search
+and OPIC in pure Python.
+
+- **Webpage**: https://bytecrawl.vercel.app/
+- **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp
+
+## Quick start (MCP — nothing to install)
+
+Point any MCP-capable agent (Claude Code, Claude Desktop, Cursor...) at the
+hosted endpoint:
+
+```bash
+claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp
+```
+
+Now the agent has four tools:
+
+| Tool | What it does |
+|---|---|
+| `focused_crawl` | Crawl a site, rank pages by relevance to a query (Shark-Search / OPIC / BFS) |
+| `fetch_markdown` | One page → clean Markdown (5–10× fewer tokens than raw HTML) |
+| `extract` | Structured records via CSS selectors |
+| `fetch_json_api` | Hit a hidden JSON API |
+
+The hosted server is static-only, rate-limited per IP, caps crawls at 10 pages,
+and refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites,
+run it locally:
+
+```bash
+pip install bytecrawl[mcp]
+claude mcp add bytecrawl -- bytecrawl-mcp                 # full power, on your machine
+pip install bytecrawl[browser] && playwright install chromium   # + JS rendering
+```
+
+## Why focused crawling?
+
+Most crawlers visit pages in whatever order they find them. With a limited
+request budget, order is everything — Shark-Search chases the branches that
+smell like your query and lets the rest decay, so 100 requests get you the 100
+*most useful* pages, not the 100 closest to the seed.
+
+```python
+from bytecrawl import SharkSearch
+
+result = SharkSearch(query="vector databases").crawl(
+    "https://example.com", max_pages=100)
+
+for page in result.top(10):
+    print(f'{page["relevance"]:.3f}  {page["url"]}')
+```
+
+- **BFS** — level by level, closest to the seed first.
+- **Shark-Search** (Hersovici et al., 1998) — topical best-first; links inherit
+  their parent's relevance with decay.
+- **OPIC** (Abiteboul et al., 2003) — live PageRank via "cash" flow, no full
+  graph needed (a `pagerank()` implementation is included to compare against).
+
+Versus the alternatives: Scrapy is a framework you wire up yourself, Firecrawl
+is a paid SaaS — ByteCrawl is a plain library with a 3-package core and these
+frontier strategies built in.
+
+## Library API
+
+```python
+from bytecrawl import Scraper
+
+bot = Scraper()
+page = bot.fetch("https://books.toscrape.com")   # auto: static, browser fallback
+books = page.extract("article.product_pod",
+                     {"title": "h3 a::attr(title)", "price": "p.price_color::text"})
+page.markdown()   # clean Markdown for LLMs   ·   page.tokens()   # token estimate
+```
+
+```python
+bot.static(url)                                   # plain HTML
+bot.api(url, params={...})                        # hidden JSON API
+bot.browser(url, wait="div.results")              # JS via Playwright
+bot.crawl(url, item="article", fields={...},
+          next_page="li.next a::attr(href)")      # pagination
+bot.session().login(url, data, csrf_field="csrf_token")   # authenticated
+```
+
+## Install
+
+```bash
+pip install bytecrawl            # slim core (requests + beautifulsoup4 + lxml)
+pip install bytecrawl[llm]       # + Markdown for LLMs
+pip install bytecrawl[browser]   # + Playwright
+pip install bytecrawl[mcp]       # + local MCP server
+pip install bytecrawl[all]
+```
+
+## Learn each scraping technique
+
+A guided walkthrough with a runnable example against a practice site:
+[static HTML](docs/01-html-estatico.md) ·
+[dynamic JS](docs/02-js-dinamico.md) ·
+[hidden APIs](docs/03-api-oculta.md) ·
+[pagination](docs/04-crawling-paginacion.md) ·
+[login](docs/05-login-sesion.md) ·
+[graph crawling](docs/06-crawling-grafos.md) ·
+[Markdown for LLMs](docs/markdown-llms.md) ·
+[ethics](docs/nota-etica.md)
+
+## Contributing
+
+```bash
+pip install -e ".[llm,dev,mcp]"
+pytest              # 109 tests, no network required
+pytest -m live      # + live browser tests (needs the browser extra)
+ruff check bytecrawl tests
+```
+
+Scrape responsibly: respect `robots.txt`, terms of service and rate limits.
+ByteCrawl ships with a configurable delay between requests.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

bytecrawl-1.0.0/README.md

@@ -0,0 +1,128 @@
+# ByteCrawl
+
+[![CI](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml/badge.svg)](https://github.com/abrahamperz/ByteCrawl/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/bytecrawl)](https://pypi.org/project/bytecrawl/)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/bytecrawl/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+**Give your AI agent focused web crawling.** ByteCrawl is an MCP server (and a
+small Python library) that doesn't just scrape a page — it crawls a whole site
+and returns the pages *most relevant* to your topic first, using Shark-Search
+and OPIC in pure Python.
+
+- **Webpage**: https://bytecrawl.vercel.app/
+- **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp
+
+## Quick start (MCP — nothing to install)
+
+Point any MCP-capable agent (Claude Code, Claude Desktop, Cursor...) at the
+hosted endpoint:
+
+```bash
+claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp
+```
+
+Now the agent has four tools:
+
+| Tool | What it does |
+|---|---|
+| `focused_crawl` | Crawl a site, rank pages by relevance to a query (Shark-Search / OPIC / BFS) |
+| `fetch_markdown` | One page → clean Markdown (5–10× fewer tokens than raw HTML) |
+| `extract` | Structured records via CSS selectors |
+| `fetch_json_api` | Hit a hidden JSON API |
+
+The hosted server is static-only, rate-limited per IP, caps crawls at 10 pages,
+and refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites,
+run it locally:
+
+```bash
+pip install bytecrawl[mcp]
+claude mcp add bytecrawl -- bytecrawl-mcp                 # full power, on your machine
+pip install bytecrawl[browser] && playwright install chromium   # + JS rendering
+```
+
+## Why focused crawling?
+
+Most crawlers visit pages in whatever order they find them. With a limited
+request budget, order is everything — Shark-Search chases the branches that
+smell like your query and lets the rest decay, so 100 requests get you the 100
+*most useful* pages, not the 100 closest to the seed.
+
+```python
+from bytecrawl import SharkSearch
+
+result = SharkSearch(query="vector databases").crawl(
+    "https://example.com", max_pages=100)
+
+for page in result.top(10):
+    print(f'{page["relevance"]:.3f}  {page["url"]}')
+```
+
+- **BFS** — level by level, closest to the seed first.
+- **Shark-Search** (Hersovici et al., 1998) — topical best-first; links inherit
+  their parent's relevance with decay.
+- **OPIC** (Abiteboul et al., 2003) — live PageRank via "cash" flow, no full
+  graph needed (a `pagerank()` implementation is included to compare against).
+
+Versus the alternatives: Scrapy is a framework you wire up yourself, Firecrawl
+is a paid SaaS — ByteCrawl is a plain library with a 3-package core and these
+frontier strategies built in.
+
+## Library API
+
+```python
+from bytecrawl import Scraper
+
+bot = Scraper()
+page = bot.fetch("https://books.toscrape.com")   # auto: static, browser fallback
+books = page.extract("article.product_pod",
+                     {"title": "h3 a::attr(title)", "price": "p.price_color::text"})
+page.markdown()   # clean Markdown for LLMs   ·   page.tokens()   # token estimate
+```
+
+```python
+bot.static(url)                                   # plain HTML
+bot.api(url, params={...})                        # hidden JSON API
+bot.browser(url, wait="div.results")              # JS via Playwright
+bot.crawl(url, item="article", fields={...},
+          next_page="li.next a::attr(href)")      # pagination
+bot.session().login(url, data, csrf_field="csrf_token")   # authenticated
+```
+
+## Install
+
+```bash
+pip install bytecrawl            # slim core (requests + beautifulsoup4 + lxml)
+pip install bytecrawl[llm]       # + Markdown for LLMs
+pip install bytecrawl[browser]   # + Playwright
+pip install bytecrawl[mcp]       # + local MCP server
+pip install bytecrawl[all]
+```
+
+## Learn each scraping technique
+
+A guided walkthrough with a runnable example against a practice site:
+[static HTML](docs/01-html-estatico.md) ·
+[dynamic JS](docs/02-js-dinamico.md) ·
+[hidden APIs](docs/03-api-oculta.md) ·
+[pagination](docs/04-crawling-paginacion.md) ·
+[login](docs/05-login-sesion.md) ·
+[graph crawling](docs/06-crawling-grafos.md) ·
+[Markdown for LLMs](docs/markdown-llms.md) ·
+[ethics](docs/nota-etica.md)
+
+## Contributing
+
+```bash
+pip install -e ".[llm,dev,mcp]"
+pytest              # 109 tests, no network required
+pytest -m live      # + live browser tests (needs the browser extra)
+ruff check bytecrawl tests
+```
+
+Scrape responsibly: respect `robots.txt`, terms of service and rate limits.
+ByteCrawl ships with a configurable delay between requests.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

bytecrawl-1.0.0/bytecrawl/__init__.py

@@ -0,0 +1,32 @@
+"""
+ByteCrawl — Universal scraping layer
+===================================
+A single API to scrape any site: static HTML, dynamic JS, hidden APIs,
+session login, crawling at scale and Markdown conversion for LLMs.
+
+Quickstart:
+
+    from bytecrawl import Scraper
+
+    bot = Scraper()
+    page = bot.fetch("https://books.toscrape.com")
+    books = page.extract("article.product_pod", {
+        "title": "h3 a::attr(title)",
+        "price": "p.price_color::text",
+    })
+
+Explicit strategies:  bot.static(url) · bot.api(url) · bot.browser(url)
+Crawling:             bot.crawl(url, item=..., fields=..., next_page=...)
+Graph crawling:       SharkSearch(query=...).crawl(url) · BFS · OPIC
+LLM:                  page.markdown() · page.tokens()
+"""
+
+from .core import Page, Scraper, Session
+from .crawler import BFS, OPIC, Crawler, CrawlResult, SharkSearch, cosine, pagerank
+
+__all__ = [
+    "Scraper", "Page", "Session",
+    "Crawler", "BFS", "SharkSearch", "OPIC", "CrawlResult",
+    "pagerank", "cosine",
+]
+__version__ = "1.0.0"