pagefetch 0.1.0__tar.gz

pagefetch-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 PageFetch contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

pagefetch-0.1.0/PKG-INFO

@@ -0,0 +1,423 @@
+Metadata-Version: 2.4
+Name: pagefetch
+Version: 0.1.0
+Summary: Asynchronous HTTP-first web fetching with conservative content extraction and browser fallback
+Author: PageFetch contributors
+License-Expression: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiosqlite<1,>=0.20
+Requires-Dist: beautifulsoup4<5,>=4.12
+Requires-Dist: brotlicffi<2,>=1.0
+Requires-Dist: camoufox<1,>=0.4
+Requires-Dist: httpx[http2,socks]<1,>=0.27
+Requires-Dist: lxml<6,>=5.2
+Requires-Dist: platformdirs<5,>=4.2
+Requires-Dist: pypdf<6,>=5.0
+Requires-Dist: pyyaml<7,>=6.0
+Requires-Dist: tldextract<6,>=5.1
+Provides-Extra: test
+Requires-Dist: pytest<9,>=8.0; extra == "test"
+Requires-Dist: pytest-asyncio<1,>=0.23; extra == "test"
+Requires-Dist: ruff<1,>=0.6; extra == "test"
+Dynamic: license-file
+
+# PageFetch
+
+**Async-first web scraping and content extraction library for Python 3.11+ — with
+intelligent browser fallback, built-in caching, and structured Markdown output.**
+
+<p align="center">
+  <img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python 3.11+">
+  <img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License">
+  <img src="https://img.shields.io/badge/code%20style-ruff-261230" alt="Ruff">
+</p>
+
+PageFetch is a modern, asynchronous Python library for **fetching web pages** and
+extracting clean, structured content. It attempts a pooled **HTTP request** first,
+scores the returned HTML using multiple **completeness signals**, and lazily falls
+back to a **Camoufox stealth browser** when the page looks blocked, empty, or
+dependent on client-side JavaScript rendering. The result is always a consistent
+`FetchResult` object carrying **Markdown**, raw HTML, extracted links, images,
+metadata, and more — regardless of which method succeeded.
+
+Perfect for **web scraping**, **content aggregation**, **LLM data pipelines**,
+**SEO analysis**, **archiving**, and any workflow that needs reliable page
+content without fighting bot detection.
+
+---
+
+## Table of Contents
+
+- [Quick Start](#quick-start)
+- [Why PageFetch](#why-pagefetch)
+- [How It Works](#how-it-works)
+- [Installation](#installation)
+- [API Reference](#api-reference)
+- [Output Model](#output-model)
+- [CLI Usage](#cli-usage)
+- [Caching](#caching)
+- [Proxy Support](#proxy-support)
+- [Non-HTML Content](#non-html-content)
+- [Configuration Reference](#configuration-reference)
+- [Development](#development)
+- [License](#license)
+
+---
+
+## Quick Start
+
+```python
+import asyncio
+
+from pagefetch import PageFetch
+
+
+async def main() -> None:
+    async with PageFetch(mode="auto", cache_ttl="24h") as client:
+        result = await client.fetch("https://example.com")
+        if result.success:
+            print(f"Title: {result.title}")
+            print(f"Markdown length: {len(result.markdown or '')} chars")
+            print(f"Links found: {len(result.links or [])}")
+            print(f"Images found: {len(result.images or [])}")
+        else:
+            print(f"Failed: {result.error.message}")
+
+
+asyncio.run(main())
+```
+
+### Fetch Multiple URLs in Parallel
+
+```python
+async with PageFetch(mode="auto") as client:
+    results = await client.fetch_many([
+        "https://example.com",
+        "https://example.org",
+        "https://httpbin.org/html",
+    ])
+    for r in results:
+        status = "✓" if r.success else "✗"
+        print(f"{status} {r.url} ({r.fetch_method}, {r.duration_ms:.0f}ms)")
+```
+
+---
+
+## Why PageFetch
+
+| Challenge | PageFetch Solution |
+|---|---|
+| **Bot detection & blocking** | Camoufox stealth browser with automatic fallback when HTTP returns empty, blocked (403/429), or JavaScript-dependent pages |
+| **Over-fetching with heavy browsers** | HTTP-first strategy — only ~5–15% of pages need the browser in auto mode |
+| **Inconsistent output formats** | Single `FetchResult` model: always get `.markdown`, `.html`, `.text`, `.links`, `.images`, `.metadata` |
+| **Managing concurrency** | Built-in semaphores for HTTP (default 10) and browser (default 4) — safe for hundreds of URLs |
+| **Repeated requests waste bandwidth** | SQLite disk cache with configurable TTL, shared across runs |
+| **Proxy rotation complexity** | Native Decodo and DataImpulse integration — configure via env vars |
+| **Content that isn't HTML** | PDFs auto-detected and extracted; XML documents parsed; plain text preserved |
+| **Dependency management friction** | Automatic bootstrap installs missing Python packages and the Camoufox binary at first startup |
+
+---
+
+## How It Works
+
+1. **Normalize** the URL (scheme, encoding, fragments).
+2. **Check cache** — if a valid SQLite entry exists, return instantly.
+3. **HTTP fetch** using `httpx` with HTTP/2, connection pooling, and configurable retries.  
+   Non-HTML responses (PDF, XML, plain text) are handled directly without browser overhead.
+4. **Content analysis** — the `confidence` score evaluates HTML completeness using  
+   text density, structural markup, heading presence, link counts, and common blocking signals  
+   (captcha walls, empty bodies, access-denied patterns).
+5. **Browser fallback** (auto mode only) — if confidence is below the threshold (default 0.80),  
+   Camoufox takes over: renders JavaScript, scrolls for lazy-loaded content, waits for  
+   network stability, and re-extracts.
+6. **Processing pipeline** — cleaned HTML → extracted links, images, metadata →  
+   converted to Markdown via a custom converter that preserves tables, code blocks,  
+   and nested lists.
+7. **Cache & return** — the structured `FetchResult` is persisted to SQLite and returned.
+
+---
+
+## Installation
+
+### Prerequisites
+
+- **Python** ≥ 3.11
+- **Camoufox** browser binary — *downloaded automatically* at first startup
+
+### Install
+
+```bash
+pip install .
+```
+
+Automatic bootstrap is **enabled by default**. At first import, PageFetch will:
+
+1. Install any missing Python dependencies via `pip`.
+2. Download the Camoufox browser binary.
+
+Subsequent starts perform quick availability checks only — the browser is launched
+only when `mode="browser"` is selected or auto mode detects incomplete HTTP content.
+
+### Disable Auto-Install (Managed / Offline Environments)
+
+```bash
+# Windows
+set PAGEFETCH_AUTO_INSTALL=0
+
+# Linux / macOS
+export PAGEFETCH_AUTO_INSTALL=0
+```
+
+With auto-install disabled, provision both components manually:
+
+```bash
+pip install .
+python -m camoufox fetch
+```
+
+---
+
+## API Reference
+
+### `PageFetch` Client
+
+The main entry point. Use as an async context manager for automatic cleanup.
+
+```python
+from pagefetch import PageFetch
+
+client = PageFetch(
+    mode="auto",              # "auto" | "http" | "browser"
+    proxy="none",             # "none" | "decodo" | "dataimpulse"
+    http_concurrency=10,      # Max parallel HTTP requests
+    browser_concurrency=4,    # Max parallel browser instances
+    cache_enabled=True,       # Enable SQLite disk cache
+    cache_ttl="24h",          # TTL: "30m", "2h", "7d", or seconds as int
+    cache_path=None,          # Custom SQLite cache path (None = platform default)
+    http_timeout=20.0,        # Per-request HTTP timeout (seconds)
+    browser_timeout=45.0,     # Per-page browser timeout (seconds)
+    retries_http=3,           # Retry on 429/5xx for HTTP
+    retries_browser=2,        # Retry on browser failure
+    max_redirects=10,         # Maximum redirect chain
+    max_content_size=25 * 1024 * 1024,  # Max response body bytes (25 MiB)
+    confidence_threshold=0.80,    # Min confidence before browser fallback
+    raise_on_error=False,     # Raise PageFetchError instead of returning error result
+)
+```
+
+### Modes
+
+| Mode | Behavior |
+|---|---|
+| `"auto"` | HTTP first; falls back to Camoufox if confidence < threshold (default) |
+| `"http"` | Pure HTTP/2 fetching — no browser, no confidence scoring |
+| `"browser"` | Camoufox stealth browser for every request |
+
+### Methods
+
+**`fetch(url, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → FetchResult`**
+
+Fetch a single URL. All keyword arguments override the client-level defaults
+for this individual request only.
+
+**`fetch_many(urls, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → list[FetchResult]`**
+
+Fetch multiple URLs concurrently. Deduplicates identical inputs internally,
+preserves the original input order, and isolates individual failures — one
+bad URL never affects the others.
+
+---
+
+## Output Model
+
+Every fetch returns a `FetchResult` dataclass:
+
+```python
+from dataclasses import dataclass
+
+@dataclass
+class FetchResult:
+    url: str                    # Normalized request URL
+    final_url: str | None       # URL after all redirects
+    status_code: int | None     # HTTP status code
+    success: bool               # Did the fetch succeed?
+    content_type: str | None    # e.g. "text/html", "application/pdf"
+    encoding: str | None        # Detected charset
+    title: str | None           # Page <title> or PDF title
+    markdown: str | None        # Cleaned Markdown body
+    html: str | None            # Raw HTML (excluded from JSON by default)
+    text: str | None            # Plain-text body fallback
+    metadata: dict              # OpenGraph, Twitter Cards, meta tags
+    links: list[LinkInfo]       # All <a> tags with text, URL, rel
+    images: list[ImageInfo]     # All <img> tags with src, alt, title
+    fetch_method: str | None    # "http" or "browser"
+    proxy_provider: str         # "none", "decodo", or "dataimpulse"
+    content_confidence: float | None  # 0–1 completeness score (None for browser mode)
+    from_cache: bool            # Was this served from cache?
+    duration_ms: float | None   # Total fetch duration
+    fetched_at: datetime | None # ISO 8601 timestamp
+    warnings: list[str]         # Non-fatal issues (cache skip, etc.)
+    error: FetchErrorInfo | None  # Error details when success=False
+```
+
+### Serialization
+
+```python
+# JSON output (HTML excluded by default for compactness)
+print(result.json(indent=2))
+print(result.json(include_html=True))   # Include raw HTML
+
+# Python dict
+data = result.to_dict()
+data = result.to_dict(include_html=True)
+
+# Reconstruct from cached JSON
+reconstructed = FetchResult.from_dict(data)
+```
+
+---
+
+## CLI Usage
+
+PageFetch ships with a command-line interface accessible via `pagefetch`:
+
+```bash
+# Fetch and print Markdown
+pagefetch https://example.com --format markdown
+
+# Fetch from a list and output JSON
+pagefetch urls.txt --format json --mode auto
+
+# Save output to a file
+pagefetch https://example.com --mode browser -o output.md
+
+# Structured JSON with raw HTML included
+pagefetch https://example.com --format json --include-html
+
+# Multiple URLs from a file (one URL per line)
+pagefetch urls.txt --format json --mode auto
+```
+
+CLI arguments map directly to the Python API — `--mode`, `--proxy`, `--timeout`,
+`--http-concurrency`, and `--browser-concurrency` are all supported.
+
+---
+
+## Caching
+
+PageFetch uses a **SQLite-backed disk cache** (`platformdirs` user cache directory
+by default). Cache entries are keyed by normalized URL + mode + proxy + relevant
+fetch settings, so switching from `"auto"` to `"browser"` mode produces a
+different cache key.
+
+- **Default TTL**: 24 hours (configurable: `"30m"`, `"2h"`, `"7d"`, or integer seconds)
+- **Automatic**: cache hits skip all network and browser work
+- **Graceful degradation**: cache read/write failures never crash a fetch — they produce warnings
+
+```python
+# Disable caching for a single request
+result = await client.fetch("https://example.com", use_cache=False)
+
+# Override TTL per-request
+result = await client.fetch("https://example.com", cache_ttl="1h")
+
+# Use a custom cache location
+client = PageFetch(cache_path="/path/to/custom_cache.sqlite3")
+```
+
+---
+
+## Proxy Support
+
+PageFetch natively supports **rotating residential proxy** providers:
+
+| Provider | Env Var (Full URL) | Env Vars (Components) |
+|---|---|---|
+| **Decodo** | `DECODO_PROXY_URL` | `DECODO_HOST`, `DECODO_PORT`, `DECODO_USERNAME`, `DECODO_PASSWORD` |
+| **DataImpulse** | `DATAIMPULSE_PROXY_URL` | `DATAIMPULSE_HOST`, `DATAIMPULSE_PORT`, `DATAIMPULSE_USERNAME`, `DATAIMPULSE_PASSWORD` |
+
+```python
+# Use a proxy provider
+async with PageFetch(proxy="decodo") as client:
+    result = await client.fetch("https://example.com")
+```
+
+Credentials are **never** included in results, logs, or cache keys. Configure
+either a full proxy URL or the individual components — PageFetch validates
+both forms automatically.
+
+---
+
+## Non-HTML Content
+
+PageFetch handles content types beyond HTML natively:
+
+| Content Type | Detection | Extraction |
+|---|---|---|
+| **PDF** | Magic bytes + `Content-Type` | Text via `pypdf` |
+| **XML** | `Content-Type` matching `+xml` or `application/xml` | Preserved as `.text` |
+| **Plain text** | Fallback when no structured type matches | Served as `.text` directly |
+
+No browser overhead is incurred for non-HTML content — detection happens
+at the HTTP response level before any processing pipeline runs.
+
+---
+
+## Configuration Reference
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `mode` | `str` | `"auto"` | Fetch strategy: `"auto"`, `"http"`, or `"browser"` |
+| `proxy` | `str` | `"none"` | Proxy provider: `"none"`, `"decodo"`, or `"dataimpulse"` |
+| `http_concurrency` | `int` | `10` | Maximum concurrent HTTP connections |
+| `browser_concurrency` | `int` | `4` | Maximum concurrent browser instances |
+| `cache_enabled` | `bool` | `True` | Enable SQLite disk cache |
+| `cache_ttl` | `str \| int` | `"24h"` | Cache time-to-live |
+| `cache_path` | `str \| Path` | *platform default* | Custom SQLite cache file path |
+| `http_timeout` | `float` | `20.0` | HTTP request timeout in seconds |
+| `browser_timeout` | `float` | `45.0` | Browser page load timeout in seconds |
+| `retries_http` | `int` | `3` | Automatic retries on retryable HTTP errors |
+| `retries_browser` | `int` | `2` | Automatic retries on browser failures |
+| `max_redirects` | `int` | `10` | Maximum redirect chain to follow |
+| `max_content_size` | `int` | `25 MiB` | Maximum response body in bytes |
+| `confidence_threshold` | `float` | `0.80` | Threshold for browser fallback in auto mode |
+| `raise_on_error` | `bool` | `False` | Raise `PageFetchError` on failure instead of returning error result |
+
+---
+
+## Development
+
+```bash
+# Clone and set up
+git clone <repo-url> && cd pagefetch
+
+# Install with test dependencies
+pip install -e ".[test]"
+
+# Run the test suite (browser integration tests are opt-in)
+pytest
+
+# Lint
+ruff check .
+```
+
+Browser integration requires the separately downloaded Camoufox binary and is
+therefore kept optional in deterministic test environments. The test suite is
+designed to run fully offline — URLs are served via local fixtures.
+
+---
+
+## License
+
+PageFetch is released under the [MIT License](LICENSE).
+
+---
+
+<p align="center">
+  <sub>
+    Built with ❤️ for developers who need reliable, structured web content
+    without fighting bot detection.
+  </sub>
+</p>