onecrawler 0.1.0__tar.gz

onecrawler-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sayed Shaun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

onecrawler-0.1.0/PKG-INFO

@@ -0,0 +1,22 @@
+Metadata-Version: 2.4
+Name: onecrawler
+Version: 0.1.0
+Summary: A feature-rich web crawler package built with Python.
+Author: Sayed Shaun
+Requires-Python: >=3.10
+License-File: LICENSE
+Requires-Dist: aiohttp
+Requires-Dist: curl_cffi
+Requires-Dist: tqdm
+Requires-Dist: playwright
+Requires-Dist: pydantic
+Requires-Dist: trafilatura
+Requires-Dist: lxml
+Provides-Extra: genai
+Requires-Dist: langchain; extra == "genai"
+Requires-Dist: langgraph; extra == "genai"
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: isort; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Dynamic: license-file

onecrawler-0.1.0/README.md

@@ -0,0 +1,309 @@
+<div align="center">
+
+<img src="https://raw.githubusercontent.com/sayedshaun/onecrawler/refs/heads/main/docs/static/onecrawl_logo.png" alt="Onecrawler" width="200"/>
+
+# Onecrawler
+
+**An async Python toolkit for sitemap discovery, browser crawling, and structured content extraction.**
+
+[![CI](https://github.com/sayedshaun/onecrawler/actions/workflows/ci.yml/badge.svg)](https://github.com/sayedshaun/onecrawler/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![PyPI](https://img.shields.io/badge/pypi-onecrawler-2ea44f.svg)](https://pypi.org/project/onecrawler/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/imports-isort-1674b1.svg)](https://pycqa.github.io/isort/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
+[Installation](#installation) · [Quick Start](#quick-start) · [Documentation](#documentation) · [Development](#development)
+
+</div>
+
+---
+
+## Overview
+
+Onecrawler helps you build maintainable crawling and extraction workflows without
+turning every project into a custom scraping script. It gives you a shared
+settingsuration model, async execution, sitemap discovery, browser-backed link
+extraction, heuristic content extraction, and optional GenAI extraction for typed
+outputs.
+
+The recommended workflow is:
+
+1. Use sitemaps first whenever possible.
+2. Fall back to browser link extraction when sitemap coverage is missing or dynamic.
+3. Scrape the final URL list with heuristic extraction by default.
+4. Use GenAI extraction when you need structured output in a Pydantic schema.
+
+```python
+sitemap = UniversalSiteMap(settings)
+urls = await sitemap.run("https://example.com")
+
+async with ScraperEngine(settings) as scraper:
+    records = await scraper.run(urls)
+```
+
+---
+
+## Features
+
+| Capability | Details |
+| --- | --- |
+| **Sitemap discovery** | Resolves `robots.txt`, common sitemap paths, nested indexes, `.xml.gz`, feeds, and HTML fallback |
+| **Browser link extraction** | Shallow and deep Playwright-backed discovery for JavaScript-rendered or sitemap-poor sites |
+| **URL filtering** | Wildcard path filters with `include_link_patterns` |
+| **Async performance** | Tunable concurrency, retries, timeouts, and crawl limits |
+| **Content extraction** | Heuristic extraction with `trafilatura` for fast article-like content extraction |
+| **GenAI extraction** | Optional model-assisted extraction for strongly typed Pydantic outputs |
+| **Output formats** | `markdown`, `json`, `csv`, `html`, `python`, `txt`, `xml`, `xmltei` |
+| **Proxy support** | Single proxy or rotating proxy pools for browser and sitemap workflows |
+| **Browser controls** | Viewport, user agent, locale, timezone, storage state, and runtime settings |
+
+---
+
+## When To Use What
+
+| Need | Use | Why |
+| --- | --- | --- |
+| Fast URL discovery from a public site | `UniversalSiteMap` | It is usually the simplest, fastest, and least expensive way to collect URLs |
+| Links from one listing page | Shallow `LinkExtractionEngine` | It reads direct same-site links from the page |
+| Recursive discovery through navigation | Deep `LinkExtractionEngine` | It follows internal links until your settingsured limit |
+| Bulk article or page text extraction | Heuristic `ScraperEngine` | It is deterministic and avoids model cost |
+| Typed fields or semantic normalization | GenAI extraction | It can produce schema-shaped output for downstream systems |
+
+---
+
+## Installation
+
+```bash
+pip install onecrawler
+```
+
+Install Playwright browser binaries when you use browser-backed crawling or scraping:
+
+```bash
+python -m playwright install chromium
+```
+
+Install optional GenAI dependencies when you use model-assisted extraction:
+
+```bash
+pip install "onecrawler[genai]"
+```
+
+For local development:
+
+```bash
+git clone https://github.com/sayedshaun/onecrawler.git
+cd onecrawler
+python -m pip install -e ".[dev]"
+python -m playwright install chromium
+```
+
+---
+
+## Quick Start
+
+This example uses the production-friendly path: discover URLs from the sitemap, then
+scrape them.
+
+```python
+import json
+import asyncio
+from onecrawler import CrawlerSettings, ScraperEngine, UniversalSiteMap
+
+
+async def main():
+    settings = CrawlerSettings(
+        link_extraction_limit=100,
+        include_link_patterns=["/articles/*"],
+        scraping_strategy="heuristic",
+        scraping_output_format="json",
+        concurrency=8,
+        request_timeout=15,
+        max_retries=3,
+    )
+
+    sitemap = UniversalSiteMap(settings)
+    urls = await sitemap.run("https://example.com")
+
+    async with ScraperEngine(settings) as scraper:
+        records = await scraper.run(urls)
+
+    with open("articles.json", "w", encoding="utf-8") as f:
+        json.dump(records, f, indent=2, ensure_ascii=False)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### Browser Link Extraction
+
+Use browser extraction when sitemaps are incomplete, unavailable, or unable to expose
+JavaScript-rendered links.
+
+```python
+import asyncio
+from onecrawler import CrawlerSettings, LinkExtractionEngine
+
+
+async def main():
+    settings = CrawlerSettings(
+        link_extraction_strategy="deep",
+        link_extraction_limit=250,
+        include_link_patterns=["/news/*"],
+        concurrency=5,
+    )
+
+    async with LinkExtractionEngine(settings) as engine:
+        links = await engine.run("https://example.com/news")
+
+    print(f"Collected {len(links)} links")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+### GenAI Extraction With A Schema
+
+Use GenAI extraction when you need a strongly typed response shape instead of plain
+content. This requires installing the GenAI dependencies:
+
+```bash
+pip install "onecrawler[genai]"
+```
+
+```python
+import asyncio
+from typing import Optional
+from pydantic import BaseModel
+from onecrawler import CrawlerSettings, GenerativeAISettings, ScraperEngine
+
+
+class ArticleSummary(BaseModel):
+    title: str
+    author: Optional[str] = None
+    published_at: Optional[str] = None
+    summary: str
+    topics: list[str]
+
+
+async def main():
+    settings = CrawlerSettings(
+        scraping_strategy="genai",  # Required for GenAI extraction
+        scraping_output_format="json",  # GenAI only supports JSON
+        genai=GenerativeAISettings(
+            provider="openai",  # Options: "openai", "google", "ollama"
+            model_name="gpt-4o-mini",
+            api_key="YOUR_API_KEY",  # Required for OpenAI/Google, optional for Ollama
+            output_schema=ArticleSummary,  # Pydantic model for structured output
+            # Optional: base_url for custom endpoints (e.g., Ollama)
+            # base_url="https://your-ollama-instance.com/",
+        ),
+        concurrency=2,  # Lower concurrency recommended for GenAI
+        request_timeout=30,  # Increase timeout for model responses
+    )
+
+    async with ScraperEngine(settings) as scraper:
+        result = await scraper.run("https://example.com/articles/story")
+
+    # Convert Pydantic model to dict for JSON serialization
+    print(result.model_dump() if hasattr(result, 'model_dump') else result)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+#### Supported Providers
+
+- **OpenAI**: Requires `api_key`, supports GPT models
+- **Google**: Requires `api_key`, supports Gemini models  
+- **Ollama**: No API key needed, requires `base_url`, supports local models
+
+#### Ollama Example
+
+```python
+settings = CrawlerSettings(
+    scraping_strategy="genai",
+    genai=GenerativeAISettings(
+        provider="ollama",
+        model_name="llama3:8b",
+        base_url="http://localhost:11434/",  # Your Ollama instance
+        output_schema=ArticleSummary,
+    ),
+)
+```
+
+### Proxy Support
+
+Attach one proxy or a rotating proxy pool directly to `CrawlerSettings`.
+
+```python
+from onecrawler import CrawlerSettings, ProxySettings
+
+
+settings = CrawlerSettings(
+    proxies=[
+        ProxySettings(server="http://proxy-1.example:8080"),
+        ProxySettings(
+            server="http://proxy-2.example:8080",
+            username="user",
+            password="pass",
+        ),
+    ],
+    proxy_rotation="round_robin",
+)
+```
+
+Use `proxy=ProxySettings(...)` for one proxy, or `proxies=[...]` for rotation.
+
+---
+
+## Documentation
+
+The README is the project overview. The full documentation in [`docs/`](docs/index.md)
+contains production guidance, caveats, performance notes, and copy-paste examples.
+
+| Topic | Guide |
+| --- | --- |
+| Install the package | [Installation](docs/installation.md) |
+| Run your first crawl | [Quick start](docs/quick-start.md) |
+| Tune crawler settings | [settings](docs/settings.md) |
+| Discover URLs from sitemaps | [Sitemap discovery](docs/sitemap-discovery.md) |
+| Extract and filter links | [Link extraction](docs/link-extraction.md) |
+| Scrape page content | [Scraping](docs/scraping.md) |
+| Public classes and exports | [API reference](docs/api-reference.md) |
+| Common fixes | [Troubleshooting](docs/troubleshooting.md) |
+| Contribute locally | [Contributing](docs/contributing.md) |
+| Work on the project | [Development](docs/development.md) |
+
+See [Contributing](docs/contributing.md) for how to improve the docs.
+
+---
+
+## Production Tips
+
+- Prefer `UniversalSiteMap` before browser crawling.
+- Always set `link_extraction_limit` for broad jobs.
+- Use `include_link_patterns` to keep discovery focused.
+- Start with moderate `concurrency`, then increase gradually.
+- Use heuristic scraping for bulk content extraction.
+- Use GenAI extraction for schema-shaped output, summaries, classification, or field
+  normalization.
+- Split discovery and scraping into separate steps for easier retries.
+
+---
+
+## License
+
+Released under the [MIT License](LICENSE). See `LICENSE` for full terms.
+
+---
+
+<div align="center">
+
+Built by [sayedshaun](https://github.com/sayedshaun)
+
+</div>

onecrawler-0.1.0/onecrawler/__init__.py

@@ -0,0 +1,7 @@
+from .crawler.link.classifier import LinkClassifierPipeline
+from .crawler.link.engine import LinkExtractionEngine
+from .crawler.scraper.engine import ScraperEngine
+from .map.sitemap import SiteMap, SitemapStats, UniversalSiteMap
+from .settings import *
+
+__version__ = "0.1.0"

onecrawler-0.1.0/onecrawler/browser.py

@@ -0,0 +1,93 @@
+from contextlib import suppress
+
+from playwright.async_api import async_playwright
+
+from .settings.browser import BrowserSettings
+
+
+class GoogleChrome:
+    def __init__(self, settings: BrowserSettings):
+        self.settings = settings
+        self.playwright = None
+        self.browser = None
+        self.context = None
+        self._started = False
+        self._closed = False
+
+    async def start(self):
+        if self._started:
+            return
+
+        self.playwright = await async_playwright().start()
+
+        launch = self.settings.launch
+        context = self.settings.context
+
+        self.browser = await self.playwright.chromium.launch(
+            headless=launch.headless,
+            slow_mo=launch.slow_mo,
+            args=launch.args,
+            executable_path=launch.executable_path,
+            channel=launch.channel,
+            env=launch.env,
+        )
+
+        self.context = await self.browser.new_context(
+            viewport=context.viewport,
+            screen=context.screen,
+            no_viewport=context.no_viewport,
+            locale=context.locale,
+            timezone_id=context.timezone_id,
+            user_agent=context.user_agent,
+            java_script_enabled=context.java_script_enabled,
+            bypass_csp=context.bypass_csp,
+            ignore_https_errors=context.ignore_https_errors,
+            extra_http_headers=context.extra_http_headers,
+            offline=context.offline,
+            geolocation=context.geolocation,
+            permissions=context.permissions,
+            storage_state=context.storage_state,
+            base_url=context.base_url,
+            proxy=self.settings.proxy.as_playwright() if self.settings.proxy else None,
+        )
+
+        self._started = True
+        self._closed = False
+
+    async def new_page(self):
+        if not self._started:
+            await self.start()
+
+        page = await self.context.new_page()
+
+        runtime = self.settings.runtime
+        page.set_default_timeout(runtime.action_timeout)
+        page.set_default_navigation_timeout(runtime.navigation_timeout)
+
+        return page
+
+    async def close(self):
+        if self._closed:
+            return
+
+        self._closed = True
+
+        # Close context safely
+        if self.context:
+            with suppress(Exception):
+                await self.context.close()
+            self.context = None
+
+        # Close browser (YOU WERE MISSING THIS)
+        if self.browser:
+            with suppress(Exception):
+                await self.browser.close()
+            self.browser = None
+
+        # Stop playwright
+        if self.playwright:
+            with suppress(Exception):
+                await self.playwright.stop()
+            self.playwright = None
+
+        self._started = False

onecrawler-0.1.0/onecrawler/crawler/base.py

@@ -0,0 +1,54 @@
+import logging
+from abc import ABC, abstractmethod
+from typing import Optional, Type
+
+
+class BaseEngine(ABC):
+    def __init__(self):
+        self._closed: bool = True
+        self.logger = logging.getLogger(self.__class__.__name__)
+
+    # ===== Context Manager =====
+    async def __aenter__(self):
+        self._closed = False
+        await self.start()
+        self.logger.debug("Engine started")
+        return self
+
+    async def __aexit__(
+        self,
+        exc_type: Optional[Type[BaseException]],
+        exc: Optional[BaseException],
+        tb,
+    ):
+        try:
+            await self.close()
+        finally:
+            self._closed = True
+            self.logger.debug("Engine closed")
+
+    # ===== Lifecycle Hooks =====
+    async def start(self):
+        """Override to initialize resources."""
+        pass
+
+    async def close(self):
+        """Override to cleanup resources."""
+        pass
+
+    # ===== REQUIRED API =====
+    @abstractmethod
+    async def run(self, *args, **kwargs):
+        """Main execution method for the engine."""
+        raise NotImplementedError
+
+    # ===== Safety =====
+    def _ensure_open(self):
+        if self._closed:
+            raise RuntimeError(
+                f"{self.__class__.__name__} is closed. Use 'async with'."
+            )
+
+    @property
+    def is_closed(self) -> bool:
+        return self._closed

onecrawler-0.1.0/onecrawler/crawler/link/classifier.py

@@ -0,0 +1,116 @@
+import asyncio
+import warnings
+from functools import lru_cache
+from typing import List
+from urllib.parse import unquote
+
+CLASSIFIER_AVAILABLE = True
+try:
+    import torch
+    from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
+except ImportError:
+    CLASSIFIER_AVAILABLE = False
+
+
+if CLASSIFIER_AVAILABLE:
+    repo_id = "SayedShaun/distilbert-link-type-classifier"
+
+    torch.set_grad_enabled(False)
+
+    model = AutoModelForSequenceClassification.from_pretrained(repo_id)
+    tokenizer = AutoTokenizer.from_pretrained(repo_id)
+
+    def get_classifier(device: str = "cpu"):
+        device_id = 0 if device == "cuda" else -1
+        return pipeline(
+            "text-classification",
+            model=model,
+            tokenizer=tokenizer,
+            device=device_id,
+        )
+
+    clf = get_classifier("cuda" if torch.cuda.is_available() else "cpu")
+
+
+def cheap_filter(url: str) -> bool:
+    if not url:
+        return False
+
+    bad_patterns = (
+        "javascript:",
+        "mailto:",
+        "tel:",
+        "#",
+    )
+
+    return not any(p in url for p in bad_patterns)
+
+
+@lru_cache(maxsize=10000)
+def _cached_single_prediction(url: str) -> str:
+    result = clf(url, truncation=True, max_length=128)[0]
+    return result["label"]
+
+
+class LinkClassifierPipeline:
+    def __init__(
+        self,
+        confidence_threshold: float = 0.8,
+    ):
+        self.threshold = confidence_threshold
+
+        if not CLASSIFIER_AVAILABLE:
+            warnings.warn(
+                "Classifier enabled but transformers/torch not installed. Disabling."
+            )
+            self.available = False
+        else:
+            self.available = True
+
+    async def classify_batch(self, urls: List[str]) -> List[bool]:
+        if not self.available:
+            return [True] * len(urls)
+
+        filtered_urls = []
+        index_map = []
+
+        for i, url in enumerate(urls):
+            if cheap_filter(url):
+                filtered_urls.append(unquote(url))
+                index_map.append(i)
+
+        results = [False] * len(urls)
+
+        if not filtered_urls:
+            return results
+
+        try:
+            predictions = await asyncio.to_thread(
+                lambda: clf(filtered_urls, truncation=True, max_length=128)
+            )
+        except Exception:
+            return [True] * len(urls)
+
+        for idx, pred in zip(index_map, predictions):
+            label = pred["label"]
+            score = pred["score"]
+
+            if label == "content" and score >= self.threshold:
+                results[idx] = True
+            else:
+                results[idx] = False
+
+        return results
+
+    async def is_valid(self, url: str) -> bool:
+        if not self.available:
+            return True
+
+        if not cheap_filter(url):
+            return False
+
+        try:
+            label = await asyncio.to_thread(_cached_single_prediction, unquote(url))
+            return label != "section"
+        except Exception:
+            return True