veilrender 0.1.0__tar.gz

veilrender-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: veilrender
+Version: 0.1.0
+Summary: Headless browser rendering API — self-hostable on HF Spaces, Docker, or bare metal
+Author: Peng Ding
+License-Expression: MIT
+Project-URL: Repository, https://github.com/Oaklight/veilrender
+Project-URL: Issues, https://github.com/Oaklight/veilrender/issues
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: cloakbrowser>=0.3.0
+Provides-Extra: dev
+Requires-Dist: ruff>=0.11.0; extra == "dev"
+Requires-Dist: ty>=0.0.1a0; extra == "dev"
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+
+# VeilRender
+
+[![PyPI version](https://img.shields.io/pypi/v/veilrender?color=green)](https://pypi.org/project/veilrender/)
+[![GitHub release](https://img.shields.io/github/v/release/Oaklight/veilrender?color=green)](https://github.com/Oaklight/veilrender/releases/latest)
+[![CI](https://github.com/Oaklight/veilrender/actions/workflows/ci.yml/badge.svg)](https://github.com/Oaklight/veilrender/actions/workflows/ci.yml)
+[![Docker Pulls](https://img.shields.io/docker/pulls/oaklight/veilrender)](https://hub.docker.com/r/oaklight/veilrender)
+[![Docker Image](https://img.shields.io/docker/v/oaklight/veilrender?sort=semver&label=docker)](https://hub.docker.com/r/oaklight/veilrender)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![HF Spaces](https://img.shields.io/badge/%F0%9F%A4%97-Spaces-blue)](https://huggingface.co/spaces/oaklight/veilrender)
+
+[中文](README_zh.md) | **English**
+
+Headless browser rendering API — self-hostable on HF Spaces, Docker, or bare metal.
+
+VeilRender accepts a URL and returns the fully rendered page content (HTML, Markdown, readability-extracted article) using a headless Chromium browser. Designed as a fallback for fetch tools that fail on JavaScript-rendered pages.
+
+## Quick Start
+
+### Docker
+
+```bash
+docker run -p 7860:7860 -e VEILRENDER_API_TOKEN=your-secret ghcr.io/oaklight/veilrender
+```
+
+### Local Development
+
+```bash
+pip install -e ".[dev]"
+playwright install chromium
+python -m veilrender
+```
+
+## API
+
+### GET /health
+
+Returns `{"status": "ok"}` if the service is running.
+
+### POST /render
+
+Render a URL and return the page content.
+
+```bash
+curl -X POST http://localhost:7860/render \
+  -H "Authorization: Bearer your-secret" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com"}'
+```
+
+Response:
+
+```json
+{
+  "content": {
+    "html": "...",
+    "markdown": "...",
+    "readability": "..."
+  },
+  "metadata": {
+    "title": "Example Domain",
+    "url": "https://example.com",
+    "status_code": 200
+  },
+  "links": [{"url": "https://www.iana.org/domains/example", "text": "More information..."}]
+}
+```
+
+### POST /screenshot
+
+Capture a screenshot of a URL.
+
+```bash
+curl -X POST http://localhost:7860/screenshot \
+  -H "Authorization: Bearer your-secret" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com"}' \
+  -o screenshot.png
+```
+
+## Configuration
+
+All settings are configured via environment variables with the `VEILRENDER_` prefix:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VEILRENDER_API_TOKEN` | *(none)* | API token for authentication. If unset, auth is disabled. |
+| `VEILRENDER_PORT` | `7860` | Server port |
+| `VEILRENDER_HOST` | `0.0.0.0` | Server bind address |
+| `VEILRENDER_TIMEOUT` | `30000` | Browser navigation timeout (ms) |
+| `VEILRENDER_VIEWPORT_WIDTH` | `1280` | Browser viewport width |
+| `VEILRENDER_VIEWPORT_HEIGHT` | `720` | Browser viewport height |
+| `VEILRENDER_MAX_CONCURRENT` | `3` | Max concurrent browser contexts |
+
+## Benchmark
+
+Tested on HF Spaces (free tier, 2 vCPU) and a self-hosted VPS (3 vCPU, 1 GB container). **100% success rate** across all 46 requests per target.
+
+| Test | HF Spaces | Self-hosted |
+|------|-----------|-------------|
+| Sequential × 5 (mixed URLs) | 8.72 s total | 11.81 s total |
+| Concurrent × 10 (mixed URLs) | 1.40 – 9.37 s | 1.29 – 13.45 s |
+| Rapid-fire × 20 (sequential) | 0.885 s avg | 1.029 s avg |
+| Peak container memory | — | 614 MiB / 1 GB |
+
+Full results: [BENCHMARK.md](BENCHMARK.md)
+
+## License
+
+MIT

veilrender-0.1.0/README.md

@@ -0,0 +1,109 @@
+# VeilRender
+
+[![PyPI version](https://img.shields.io/pypi/v/veilrender?color=green)](https://pypi.org/project/veilrender/)
+[![GitHub release](https://img.shields.io/github/v/release/Oaklight/veilrender?color=green)](https://github.com/Oaklight/veilrender/releases/latest)
+[![CI](https://github.com/Oaklight/veilrender/actions/workflows/ci.yml/badge.svg)](https://github.com/Oaklight/veilrender/actions/workflows/ci.yml)
+[![Docker Pulls](https://img.shields.io/docker/pulls/oaklight/veilrender)](https://hub.docker.com/r/oaklight/veilrender)
+[![Docker Image](https://img.shields.io/docker/v/oaklight/veilrender?sort=semver&label=docker)](https://hub.docker.com/r/oaklight/veilrender)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![HF Spaces](https://img.shields.io/badge/%F0%9F%A4%97-Spaces-blue)](https://huggingface.co/spaces/oaklight/veilrender)
+
+[中文](README_zh.md) | **English**
+
+Headless browser rendering API — self-hostable on HF Spaces, Docker, or bare metal.
+
+VeilRender accepts a URL and returns the fully rendered page content (HTML, Markdown, readability-extracted article) using a headless Chromium browser. Designed as a fallback for fetch tools that fail on JavaScript-rendered pages.
+
+## Quick Start
+
+### Docker
+
+```bash
+docker run -p 7860:7860 -e VEILRENDER_API_TOKEN=your-secret ghcr.io/oaklight/veilrender
+```
+
+### Local Development
+
+```bash
+pip install -e ".[dev]"
+playwright install chromium
+python -m veilrender
+```
+
+## API
+
+### GET /health
+
+Returns `{"status": "ok"}` if the service is running.
+
+### POST /render
+
+Render a URL and return the page content.
+
+```bash
+curl -X POST http://localhost:7860/render \
+  -H "Authorization: Bearer your-secret" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com"}'
+```
+
+Response:
+
+```json
+{
+  "content": {
+    "html": "...",
+    "markdown": "...",
+    "readability": "..."
+  },
+  "metadata": {
+    "title": "Example Domain",
+    "url": "https://example.com",
+    "status_code": 200
+  },
+  "links": [{"url": "https://www.iana.org/domains/example", "text": "More information..."}]
+}
+```
+
+### POST /screenshot
+
+Capture a screenshot of a URL.
+
+```bash
+curl -X POST http://localhost:7860/screenshot \
+  -H "Authorization: Bearer your-secret" \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com"}' \
+  -o screenshot.png
+```
+
+## Configuration
+
+All settings are configured via environment variables with the `VEILRENDER_` prefix:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VEILRENDER_API_TOKEN` | *(none)* | API token for authentication. If unset, auth is disabled. |
+| `VEILRENDER_PORT` | `7860` | Server port |
+| `VEILRENDER_HOST` | `0.0.0.0` | Server bind address |
+| `VEILRENDER_TIMEOUT` | `30000` | Browser navigation timeout (ms) |
+| `VEILRENDER_VIEWPORT_WIDTH` | `1280` | Browser viewport width |
+| `VEILRENDER_VIEWPORT_HEIGHT` | `720` | Browser viewport height |
+| `VEILRENDER_MAX_CONCURRENT` | `3` | Max concurrent browser contexts |
+
+## Benchmark
+
+Tested on HF Spaces (free tier, 2 vCPU) and a self-hosted VPS (3 vCPU, 1 GB container). **100% success rate** across all 46 requests per target.
+
+| Test | HF Spaces | Self-hosted |
+|------|-----------|-------------|
+| Sequential × 5 (mixed URLs) | 8.72 s total | 11.81 s total |
+| Concurrent × 10 (mixed URLs) | 1.40 – 9.37 s | 1.29 – 13.45 s |
+| Rapid-fire × 20 (sequential) | 0.885 s avg | 1.029 s avg |
+| Peak container memory | — | 614 MiB / 1 GB |
+
+Full results: [BENCHMARK.md](BENCHMARK.md)
+
+## License
+
+MIT

veilrender-0.1.0/pyproject.toml

@@ -0,0 +1,61 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "veilrender"
+dynamic = ["version"]
+description = "Headless browser rendering API — self-hostable on HF Spaces, Docker, or bare metal"
+authors = [{ name = "Peng Ding" }]
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+classifiers = [
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Internet :: WWW/HTTP",
+]
+
+dependencies = [
+    "cloakbrowser>=0.3.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "ruff>=0.11.0",
+    "ty>=0.0.1a0",
+    "pytest>=7.0.0",
+    "pytest-asyncio>=0.21.0",
+]
+
+[project.scripts]
+veilrender = "veilrender.app:main"
+
+[project.urls]
+Repository = "https://github.com/Oaklight/veilrender"
+Issues = "https://github.com/Oaklight/veilrender/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.dynamic]
+version = { attr = "veilrender.__version__" }
+
+[tool.setuptools.package-data]
+"veilrender" = ["py.typed"]
+
+[tool.ruff]
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "UP"]
+ignore = ["UP007", "E501"]
+
+[tool.ty.environment]
+python-version = "3.10"
+
+[tool.ty.src]
+exclude = ["src/veilrender/_vendor/**"]
+
+[tool.ty.rules]
+unresolved-import = "ignore"

veilrender-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

veilrender-0.1.0/src/veilrender/__init__.py

@@ -0,0 +1,3 @@
+"""VeilRender — headless browser rendering API."""
+
+__version__ = "0.1.0"

veilrender-0.1.0/src/veilrender/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for ``python -m veilrender``."""
+
+from veilrender.app import main
+
+if __name__ == "__main__":
+    main()

veilrender-0.1.0/src/veilrender/_vendor/__init__.py

@@ -0,0 +1,2 @@
+# Vendored zerodep modules — do not edit manually.
+# Update via: make vendor

veilrender-0.1.0/src/veilrender/_vendor/benchmark_compare.py

@@ -0,0 +1,323 @@
+#!/usr/bin/env python3
+"""Three-way readability benchmark: zerodep vs readability-lxml vs Mozilla JS.
+
+Runs each implementation on Mozilla's test fixtures and prints a comparison
+table.  JS timing is measured internally by bench_mozilla.js (no subprocess
+overhead in the numbers).
+
+Usage:
+    python benchmark_compare.py              # all fixtures
+    python benchmark_compare.py 001 bbc-1    # specific fixtures
+    python benchmark_compare.py --rounds 20  # more rounds
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import timeit
+
+# ── Setup paths ──────────────────────────────────────────────────────────────
+
+_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
+_TEST_PAGES_DIR = os.path.join(_THIS_DIR, "test-pages")
+_BENCH_JS = os.path.join(_THIS_DIR, "bench_mozilla.js")
+
+sys.path.insert(0, _THIS_DIR)
+
+
+# ── Discover fixtures ────────────────────────────────────────────────────────
+
+
+def discover_fixtures() -> list[str]:
+    """Return sorted list of available fixture names."""
+    if not os.path.isdir(_TEST_PAGES_DIR):
+        return []
+    return sorted(
+        d
+        for d in os.listdir(_TEST_PAGES_DIR)
+        if os.path.isdir(os.path.join(_TEST_PAGES_DIR, d))
+        and os.path.isfile(os.path.join(_TEST_PAGES_DIR, d, "source.html"))
+    )
+
+
+def load_source(name: str) -> str:
+    """Load source HTML for a fixture."""
+    path = os.path.join(_TEST_PAGES_DIR, name, "source.html")
+    with open(path, encoding="utf-8") as f:
+        return f.read()
+
+
+# ── Python: zerodep readability ──────────────────────────────────────────────
+
+
+def bench_zerodep(html: str, rounds: int) -> dict:
+    """Benchmark our readability.extract() and return timing dict."""
+    from readability import extract
+
+    # Warm-up.
+    result = extract(html)
+
+    times = []
+    for _ in range(rounds):
+        t0 = timeit.default_timer()
+        extract(html)
+        t1 = timeit.default_timer()
+        times.append((t1 - t0) * 1000)  # ms
+
+    return {
+        "times_ms": times,
+        "min_ms": min(times),
+        "mean_ms": sum(times) / len(times),
+        "max_ms": max(times),
+        "title": result.title,
+        "length": result.length,
+    }
+
+
+# ── Python: readability-lxml ────────────────────────────────────────────────
+
+
+def _load_readability_lxml():
+    """Load readability-lxml's Document class, working around name clash."""
+    import importlib
+    import importlib.metadata
+
+    try:
+        importlib.metadata.version("readability-lxml")
+    except importlib.metadata.PackageNotFoundError:
+        return None
+
+    saved_path = sys.path[:]
+    saved_modules = {
+        k: sys.modules.pop(k)
+        for k in list(sys.modules)
+        if k == "readability" or k.startswith("readability.")
+    }
+    try:
+        sys.path = [
+            p for p in sys.path if os.path.abspath(p) != os.path.abspath(_THIS_DIR)
+        ]
+        mod = importlib.import_module("readability")
+        return mod.Document
+    finally:
+        sys.path = saved_path
+        for k in list(sys.modules):
+            if k == "readability" or k.startswith("readability."):
+                del sys.modules[k]
+        sys.modules.update(saved_modules)
+
+
+_RefDocument = _load_readability_lxml()
+
+
+def bench_readability_lxml(html: str, rounds: int) -> dict | None:
+    """Benchmark readability-lxml and return timing dict, or None."""
+    if _RefDocument is None:
+        return None
+
+    # Warm-up.
+    doc = _RefDocument(html)
+    summary = doc.summary()
+
+    times = []
+    for _ in range(rounds):
+        t0 = timeit.default_timer()
+        doc = _RefDocument(html)
+        doc.summary()
+        t1 = timeit.default_timer()
+        times.append((t1 - t0) * 1000)
+
+    # Extract title from summary HTML (basic).
+    title = doc.short_title() if hasattr(doc, "short_title") else ""
+    length = len(summary) if summary else 0
+
+    return {
+        "times_ms": times,
+        "min_ms": min(times),
+        "mean_ms": sum(times) / len(times),
+        "max_ms": max(times),
+        "title": title,
+        "length": length,
+    }
+
+
+# ── JavaScript: Mozilla Readability.js ───────────────────────────────────────
+
+
+def bench_mozilla_js(fixture_name: str, rounds: int) -> dict | None:
+    """Benchmark Mozilla Readability.js via Node.js subprocess.
+
+    Timing is measured internally by bench_mozilla.js — no subprocess
+    overhead in the reported numbers.
+    """
+    if not shutil.which("node"):
+        return None
+    if not os.path.isfile(_BENCH_JS):
+        return None
+
+    source_path = os.path.join(_TEST_PAGES_DIR, fixture_name, "source.html")
+    try:
+        result = subprocess.run(
+            ["node", _BENCH_JS, source_path, str(rounds)],
+            capture_output=True,
+            text=True,
+            timeout=120,
+            cwd=_THIS_DIR,
+        )
+        if result.returncode != 0:
+            print(f"  [JS error: {result.stderr.strip()[:100]}]", file=sys.stderr)
+            return None
+        return json.loads(result.stdout)
+    except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError):
+        return None
+
+
+# ── Output formatting ────────────────────────────────────────────────────────
+
+# ANSI colors (disabled if not a terminal).
+if sys.stdout.isatty():
+    _BOLD = "\033[1m"
+    _GREEN = "\033[32m"
+    _YELLOW = "\033[33m"
+    _CYAN = "\033[36m"
+    _RESET = "\033[0m"
+    _DIM = "\033[2m"
+else:
+    _BOLD = _GREEN = _YELLOW = _CYAN = _RESET = _DIM = ""
+
+
+def _fmt_ms(ms: float) -> str:
+    """Format milliseconds with appropriate unit."""
+    if ms < 1:
+        return f"{ms * 1000:.0f} µs"
+    if ms < 1000:
+        return f"{ms:.1f} ms"
+    return f"{ms / 1000:.2f} s"
+
+
+def _ratio_str(ms: float, baseline: float) -> str:
+    """Format a ratio relative to baseline."""
+    if baseline <= 0:
+        return ""
+    ratio = ms / baseline
+    if ratio < 1.05:
+        return f"{_GREEN}1.00x{_RESET}"
+    return f"{_YELLOW}{ratio:.2f}x{_RESET}"
+
+
+def print_results(
+    fixture_name: str,
+    html_size: int,
+    zd: dict,
+    lxml: dict | None,
+    js: dict | None,
+) -> None:
+    """Print a single fixture's results as a formatted row."""
+    baseline = zd["mean_ms"]
+
+    cols = [
+        f"  {_BOLD}{fixture_name:<28s}{_RESET}",
+        f"{_DIM}{html_size / 1024:>7.1f} KB{_RESET}",
+        f"{_CYAN}zerodep{_RESET}  {_fmt_ms(zd['mean_ms']):>10s}"
+        f" {_ratio_str(zd['mean_ms'], baseline)}",
+    ]
+
+    if lxml is not None:
+        cols.append(
+            f"{_CYAN}lxml{_RESET}     {_fmt_ms(lxml['mean_ms']):>10s}"
+            f" {_ratio_str(lxml['mean_ms'], baseline)}"
+        )
+    else:
+        cols.append(f"{_DIM}lxml     {'n/a':>10s}{_RESET}")
+
+    if js is not None:
+        cols.append(
+            f"{_CYAN}mozilla{_RESET}  {_fmt_ms(js['mean_ms']):>10s}"
+            f" {_ratio_str(js['mean_ms'], baseline)}"
+        )
+    else:
+        cols.append(f"{_DIM}mozilla  {'n/a':>10s}{_RESET}")
+
+    print("  ".join(cols))
+
+
+# ── Main ─────────────────────────────────────────────────────────────────────
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Three-way readability benchmark comparison."
+    )
+    parser.add_argument(
+        "fixtures",
+        nargs="*",
+        help="Fixture names to benchmark (default: all).",
+    )
+    parser.add_argument(
+        "--rounds",
+        type=int,
+        default=10,
+        help="Number of timing rounds per fixture (default: 10).",
+    )
+    args = parser.parse_args()
+
+    all_fixtures = discover_fixtures()
+    if not all_fixtures:
+        print("No test fixtures found in test-pages/", file=sys.stderr)
+        sys.exit(1)
+
+    fixtures = args.fixtures if args.fixtures else all_fixtures
+    # Validate fixture names.
+    for name in fixtures:
+        if name not in all_fixtures:
+            print(f"Unknown fixture: {name}", file=sys.stderr)
+            print(f"Available: {', '.join(all_fixtures)}", file=sys.stderr)
+            sys.exit(1)
+
+    rounds = args.rounds
+
+    # Header.
+    print()
+    print(f"{_BOLD}Readability Benchmark ({rounds} rounds per fixture){_RESET}")
+    has_node = shutil.which("node") is not None
+    has_lxml = _RefDocument is not None
+    status = []
+    status.append(f"zerodep: {_GREEN}yes{_RESET}")
+    lxml_status = _GREEN + "yes" + _RESET if has_lxml else _DIM + "no" + _RESET
+    status.append(f"readability-lxml: {lxml_status}")
+    status.append(
+        f"mozilla js: {_GREEN + 'yes' + _RESET if has_node else _DIM + 'no' + _RESET}"
+    )
+    print(f"  Implementations: {' | '.join(status)}")
+    print(f"  {_DIM}Times shown are mean. Ratios relative to zerodep.{_RESET}")
+    print()
+
+    # Column headers.
+    print(
+        f"  {'Fixture':<28s}  {'Size':>9s}  "
+        f"{'zerodep':>19s}  {'readability-lxml':>19s}  "
+        f"{'mozilla js':>19s}"
+    )
+    print("  " + "─" * 110)
+
+    for name in fixtures:
+        html = load_source(name)
+        html_size = len(html.encode("utf-8"))
+
+        # Benchmark all three.
+        zd = bench_zerodep(html, rounds)
+        lxml = bench_readability_lxml(html, rounds)
+        js = bench_mozilla_js(name, rounds)
+
+        print_results(name, html_size, zd, lxml, js)
+
+    print()
+
+
+if __name__ == "__main__":
+    main()