pg2md 1.2.0__tar.gz

pg2md-1.2.0/.gitignore

@@ -0,0 +1,82 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# Linters / formatters
+.ruff_cache/
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Jupyter
+.ipynb_checkpoints
+
+# Project specific
+*.md
+!README.md
+data.json
+proxies.txt

pg2md-1.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 lemantorus
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pg2md-1.2.0/PKG-INFO

@@ -0,0 +1,273 @@
+Metadata-Version: 2.4
+Name: pg2md
+Version: 1.2.0
+Summary: Page to Markdown converter with JS rendering support via Playwright
+Project-URL: Homepage, https://github.com/lemantorus/pg2md
+Project-URL: Documentation, https://github.com/lemantorus/pg2md#readme
+Project-URL: Repository, https://github.com/lemantorus/pg2md
+Project-URL: Issues, https://github.com/lemantorus/pg2md/issues
+Author: lemantorus
+License-Expression: MIT
+License-File: LICENSE
+Keywords: converter,html,js-rendering,markdown,parser,playwright,scraper,web-scraping
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: html-to-markdown>=2.0.0
+Requires-Dist: playwright>=1.40.0
+Provides-Extra: dev
+Requires-Dist: mypy>=1.0.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# pg2md
+
+[![PyPI version](https://badge.fury.io/py/pg2md.svg)](https://badge.fury.io/py/pg2md)
+[![Python](https://img.shields.io/pypi/pyversions/pg2md.svg)](https://pypi.org/project/pg2md/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Downloads](https://static.pepy.tech/badge/pg2md)](https://pepy.tech/project/pg2md)
+[![GitHub stars](https://img.shields.io/github/stars/lemantorus/pg2md.svg?style=social)](https://github.com/lemantorus/pg2md/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/lemantorus/pg2md.svg)](https://github.com/lemantorus/pg2md/issues)
+[![GitHub forks](https://img.shields.io/github/forks/lemantorus/pg2md.svg?style=social)](https://github.com/lemantorus/pg2md/network/members)
+
+**P**a**g**e to **M**ark**d**own — fast HTML-to-Markdown converter with JavaScript rendering support.
+
+Converts any web page to clean Markdown using Playwright for JS rendering and Rust-based `html-to-markdown` for conversion.
+
+## Features
+
+- **Auto Lightpanda** — automatically downloads and starts Lightpanda browser
+- **JavaScript Rendering** — handles SPA, React, Vue, dynamic content
+- **Fast Conversion** — Rust-based `html-to-markdown` core
+- **Clean Output** — strips scripts, styles, navigation, forms
+- **Proxy Support** — HTTP/HTTPS/SOCKS5 with auth
+- **Custom User-Agents** — includes Googlebot, Bingbot, etc.
+- **Async & Sync API** — `parse()` and `async_parse()`
+- **Batch Processing** — `async_parse_many()` for parallel requests
+- **Configurable** — images, links, headers, timeouts
+
+## Quick Start
+
+```python
+from pg2md import PageParser
+
+# Auto-downloads Lightpanda, starts it, and parses page
+# If Lightpanda fails, falls back to Chromium automatically
+parser = PageParser()
+markdown = parser.parse("https://example.com")
+print(markdown)
+```
+
+First run will download Lightpanda (~50MB) to `~/.cache/pg2md/`.
+
+**Note:** Lightpanda is in beta and may not support all Playwright features. If it fails, pg2md automatically falls back to Chromium.
+
+## Installation
+
+```bash
+pip install pg2md
+playwright install chromium  # fallback browser if Lightpanda unsupported
+```
+
+## Usage Examples
+
+### Basic Usage
+
+```python
+from pg2md import PageParser
+
+parser = PageParser()
+result = parser.parse("https://example.com")
+print(result)
+```
+
+### Without Images and Links
+
+```python
+from pg2md import PageParser
+
+parser = PageParser(with_image=False, with_link=False)
+result = parser.parse("https://example.com")
+```
+
+### With Proxy
+
+```python
+from pg2md import PageParser, ProxyConfig
+
+proxy = ProxyConfig(
+    server="http://proxy.example.com:8080",
+    username="user",
+    password="pass"
+)
+
+parser = PageParser()
+result = parser.parse("https://example.com", proxy=proxy)
+```
+
+### SOCKS5 Proxy
+
+```python
+from pg2md import PageParser, ProxyConfig
+
+proxy = ProxyConfig(server="socks5://127.0.0.1:1080")
+parser = PageParser()
+result = parser.parse("https://example.com", proxy=proxy)
+```
+
+### Custom User-Agent
+
+```python
+from pg2md import PageParser, BrowserConfig, UserAgents
+
+config = BrowserConfig(
+    cdp_url=None,
+    user_agent=UserAgents.GOOGLEBOT_DESKTOP,
+    extra_headers={"Accept-Language": "en-US,en;q=0.9"}
+)
+
+parser = PageParser(browser_config=config)
+result = parser.parse("https://example.com")
+```
+
+### Async API
+
+```python
+import asyncio
+from pg2md import PageParser
+
+async def main():
+    parser = PageParser()
+    result = await parser.async_parse("https://example.com")
+    print(result)
+
+asyncio.run(main())
+```
+
+### Batch Processing
+
+```python
+import asyncio
+from pg2md import PageParser
+
+async def main():
+    parser = PageParser()
+    urls = [
+        "https://example.com",
+        "https://example.org",
+        "https://example.net",
+    ]
+    results = await parser.async_parse_many(urls)
+    
+    for url, result in results.items():
+        if isinstance(result, Exception):
+            print(f"Error {url}: {result}")
+        else:
+            print(f"{url}: {len(result)} chars")
+
+asyncio.run(main())
+```
+
+### Using Lightpanda
+
+```python
+from pg2md import PageParser, BrowserConfig
+
+# Start Lightpanda manually:
+# ./lightpanda serve --host 127.0.0.1 --port 9222
+
+config = BrowserConfig(cdp_url="ws://127.0.0.1:9222")
+parser = PageParser(browser_config=config)
+result = parser.parse("https://example.com")
+```
+
+## Configuration
+
+### BrowserConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `cdp_url` | `str \| None` | `"auto"` | `"auto"` = auto Lightpanda, `None` = Chromium, `"ws://..."` = custom CDP |
+| `lightpanda_bin` | `str \| None` | `None` | Path to Lightpanda binary (for manual start) |
+| `navigation_timeout` | `int` | `60000` | Navigation timeout (ms) |
+| `wait_until` | `str` | `"domcontentloaded"` | Wait event (`"load"`, `"domcontentloaded"`, `"networkidle"`) |
+| `default_proxy` | `ProxyConfig \| None` | `None` | Default proxy for all requests |
+| `user_agent` | `str \| None` | Chrome Desktop | User-Agent string |
+| `extra_headers` | `dict \| None` | `None` | Additional HTTP headers |
+| `viewport` | `dict \| None` | `{"width": 1920, "height": 1080}` | Browser viewport size |
+
+### ProxyConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `server` | `str` | required | Proxy URL |
+| `username` | `str \| None` | `None` | Username |
+| `password` | `str \| None` | `None` | Password |
+| `bypass` | `str \| None` | `None` | Hosts to bypass |
+
+### UserAgents
+
+Available presets:
+
+- `CHROME_DESKTOP`, `CHROME_MAC`, `CHROME_LINUX`
+- `FIREFOX_DESKTOP`
+- `SAFARI_MAC`
+- `EDGE`
+- `GOOGLEBOT_DESKTOP`, `GOOGLEBOT_MOBILE`, `GOOGLEBOT_VIDEO`
+- `BINGBOT`, `BINGBOT_MOBILE`
+- `YANDEXBOT`
+- `DUCKBOT`
+- `APPLEBOT`
+
+## API Reference
+
+### PageParser
+
+```python
+PageParser(
+    with_image: bool = False,
+    with_link: bool = True,
+    browser_config: BrowserConfig | None = None
+)
+```
+
+#### Methods
+
+| Method | Description |
+|--------|-------------|
+| `parse(url, proxy=None)` | Sync parse, returns Markdown string |
+| `async_parse(url, proxy=None)` | Async parse, returns Markdown string |
+| `async_parse_many(urls, proxy=None)` | Batch async parse, returns dict |
+| `stop_lightpanda()` | Stop Lightpanda if started |
+
+## Development
+
+```bash
+git clone https://github.com/lemantorus/pg2md.git
+cd pg2md
+python -m venv venv
+source venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium
+```
+
+## License
+
+[MIT](LICENSE)
+
+## Credits
+
+- [Playwright](https://playwright.dev/python/) — browser automation
+- [html-to-markdown](https://pypi.org/project/html-to-markdown/) — Rust-based HTML to Markdown
+- [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing

pg2md-1.2.0/README.md

@@ -0,0 +1,240 @@
+# pg2md
+
+[![PyPI version](https://badge.fury.io/py/pg2md.svg)](https://badge.fury.io/py/pg2md)
+[![Python](https://img.shields.io/pypi/pyversions/pg2md.svg)](https://pypi.org/project/pg2md/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Downloads](https://static.pepy.tech/badge/pg2md)](https://pepy.tech/project/pg2md)
+[![GitHub stars](https://img.shields.io/github/stars/lemantorus/pg2md.svg?style=social)](https://github.com/lemantorus/pg2md/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/lemantorus/pg2md.svg)](https://github.com/lemantorus/pg2md/issues)
+[![GitHub forks](https://img.shields.io/github/forks/lemantorus/pg2md.svg?style=social)](https://github.com/lemantorus/pg2md/network/members)
+
+**P**a**g**e to **M**ark**d**own — fast HTML-to-Markdown converter with JavaScript rendering support.
+
+Converts any web page to clean Markdown using Playwright for JS rendering and Rust-based `html-to-markdown` for conversion.
+
+## Features
+
+- **Auto Lightpanda** — automatically downloads and starts Lightpanda browser
+- **JavaScript Rendering** — handles SPA, React, Vue, dynamic content
+- **Fast Conversion** — Rust-based `html-to-markdown` core
+- **Clean Output** — strips scripts, styles, navigation, forms
+- **Proxy Support** — HTTP/HTTPS/SOCKS5 with auth
+- **Custom User-Agents** — includes Googlebot, Bingbot, etc.
+- **Async & Sync API** — `parse()` and `async_parse()`
+- **Batch Processing** — `async_parse_many()` for parallel requests
+- **Configurable** — images, links, headers, timeouts
+
+## Quick Start
+
+```python
+from pg2md import PageParser
+
+# Auto-downloads Lightpanda, starts it, and parses page
+# If Lightpanda fails, falls back to Chromium automatically
+parser = PageParser()
+markdown = parser.parse("https://example.com")
+print(markdown)
+```
+
+First run will download Lightpanda (~50MB) to `~/.cache/pg2md/`.
+
+**Note:** Lightpanda is in beta and may not support all Playwright features. If it fails, pg2md automatically falls back to Chromium.
+
+## Installation
+
+```bash
+pip install pg2md
+playwright install chromium  # fallback browser if Lightpanda unsupported
+```
+
+## Usage Examples
+
+### Basic Usage
+
+```python
+from pg2md import PageParser
+
+parser = PageParser()
+result = parser.parse("https://example.com")
+print(result)
+```
+
+### Without Images and Links
+
+```python
+from pg2md import PageParser
+
+parser = PageParser(with_image=False, with_link=False)
+result = parser.parse("https://example.com")
+```
+
+### With Proxy
+
+```python
+from pg2md import PageParser, ProxyConfig
+
+proxy = ProxyConfig(
+    server="http://proxy.example.com:8080",
+    username="user",
+    password="pass"
+)
+
+parser = PageParser()
+result = parser.parse("https://example.com", proxy=proxy)
+```
+
+### SOCKS5 Proxy
+
+```python
+from pg2md import PageParser, ProxyConfig
+
+proxy = ProxyConfig(server="socks5://127.0.0.1:1080")
+parser = PageParser()
+result = parser.parse("https://example.com", proxy=proxy)
+```
+
+### Custom User-Agent
+
+```python
+from pg2md import PageParser, BrowserConfig, UserAgents
+
+config = BrowserConfig(
+    cdp_url=None,
+    user_agent=UserAgents.GOOGLEBOT_DESKTOP,
+    extra_headers={"Accept-Language": "en-US,en;q=0.9"}
+)
+
+parser = PageParser(browser_config=config)
+result = parser.parse("https://example.com")
+```
+
+### Async API
+
+```python
+import asyncio
+from pg2md import PageParser
+
+async def main():
+    parser = PageParser()
+    result = await parser.async_parse("https://example.com")
+    print(result)
+
+asyncio.run(main())
+```
+
+### Batch Processing
+
+```python
+import asyncio
+from pg2md import PageParser
+
+async def main():
+    parser = PageParser()
+    urls = [
+        "https://example.com",
+        "https://example.org",
+        "https://example.net",
+    ]
+    results = await parser.async_parse_many(urls)
+    
+    for url, result in results.items():
+        if isinstance(result, Exception):
+            print(f"Error {url}: {result}")
+        else:
+            print(f"{url}: {len(result)} chars")
+
+asyncio.run(main())
+```
+
+### Using Lightpanda
+
+```python
+from pg2md import PageParser, BrowserConfig
+
+# Start Lightpanda manually:
+# ./lightpanda serve --host 127.0.0.1 --port 9222
+
+config = BrowserConfig(cdp_url="ws://127.0.0.1:9222")
+parser = PageParser(browser_config=config)
+result = parser.parse("https://example.com")
+```
+
+## Configuration
+
+### BrowserConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `cdp_url` | `str \| None` | `"auto"` | `"auto"` = auto Lightpanda, `None` = Chromium, `"ws://..."` = custom CDP |
+| `lightpanda_bin` | `str \| None` | `None` | Path to Lightpanda binary (for manual start) |
+| `navigation_timeout` | `int` | `60000` | Navigation timeout (ms) |
+| `wait_until` | `str` | `"domcontentloaded"` | Wait event (`"load"`, `"domcontentloaded"`, `"networkidle"`) |
+| `default_proxy` | `ProxyConfig \| None` | `None` | Default proxy for all requests |
+| `user_agent` | `str \| None` | Chrome Desktop | User-Agent string |
+| `extra_headers` | `dict \| None` | `None` | Additional HTTP headers |
+| `viewport` | `dict \| None` | `{"width": 1920, "height": 1080}` | Browser viewport size |
+
+### ProxyConfig
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `server` | `str` | required | Proxy URL |
+| `username` | `str \| None` | `None` | Username |
+| `password` | `str \| None` | `None` | Password |
+| `bypass` | `str \| None` | `None` | Hosts to bypass |
+
+### UserAgents
+
+Available presets:
+
+- `CHROME_DESKTOP`, `CHROME_MAC`, `CHROME_LINUX`
+- `FIREFOX_DESKTOP`
+- `SAFARI_MAC`
+- `EDGE`
+- `GOOGLEBOT_DESKTOP`, `GOOGLEBOT_MOBILE`, `GOOGLEBOT_VIDEO`
+- `BINGBOT`, `BINGBOT_MOBILE`
+- `YANDEXBOT`
+- `DUCKBOT`
+- `APPLEBOT`
+
+## API Reference
+
+### PageParser
+
+```python
+PageParser(
+    with_image: bool = False,
+    with_link: bool = True,
+    browser_config: BrowserConfig | None = None
+)
+```
+
+#### Methods
+
+| Method | Description |
+|--------|-------------|
+| `parse(url, proxy=None)` | Sync parse, returns Markdown string |
+| `async_parse(url, proxy=None)` | Async parse, returns Markdown string |
+| `async_parse_many(urls, proxy=None)` | Batch async parse, returns dict |
+| `stop_lightpanda()` | Stop Lightpanda if started |
+
+## Development
+
+```bash
+git clone https://github.com/lemantorus/pg2md.git
+cd pg2md
+python -m venv venv
+source venv/bin/activate
+pip install -e ".[dev]"
+playwright install chromium
+```
+
+## License
+
+[MIT](LICENSE)
+
+## Credits
+
+- [Playwright](https://playwright.dev/python/) — browser automation
+- [html-to-markdown](https://pypi.org/project/html-to-markdown/) — Rust-based HTML to Markdown
+- [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing

pg2md-1.2.0/pyproject.toml

@@ -0,0 +1,64 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pg2md"
+version = "1.2.0"
+description = "Page to Markdown converter with JS rendering support via Playwright"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [{ name = "lemantorus" }]
+keywords = [
+    "markdown",
+    "html",
+    "converter",
+    "playwright",
+    "scraper",
+    "parser",
+    "web-scraping",
+    "js-rendering",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: Text Processing :: Markup :: Markdown",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "playwright>=1.40.0",
+    "html-to-markdown>=2.0.0",
+    "beautifulsoup4>=4.12.0",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0.0", "ruff>=0.1.0", "mypy>=1.0.0"]
+
+[project.urls]
+Homepage = "https://github.com/lemantorus/pg2md"
+Documentation = "https://github.com/lemantorus/pg2md#readme"
+Repository = "https://github.com/lemantorus/pg2md"
+Issues = "https://github.com/lemantorus/pg2md/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pg2md"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true

pg2md-1.2.0/src/pg2md/__init__.py

@@ -0,0 +1,49 @@
+"""
+pg2md — Page to Markdown converter with JS rendering support.
+
+A fast, clean HTML-to-Markdown converter that uses Playwright for
+JavaScript rendering and html-to-markdown (Rust-based) for conversion.
+
+Features:
+    - Auto-download and start Lightpanda browser (default)
+    - JavaScript rendering via Playwright/Lightpanda
+    - Fast Rust-based HTML-to-Markdown conversion
+    - Proxy support (HTTP/HTTPS/SOCKS5)
+    - Custom User-Agents (Googlebot, Bingbot, etc.)
+
+Example:
+    from pg2md import PageParser
+
+    # Auto-downloads Lightpanda and parses page
+    parser = PageParser()
+    markdown = parser.parse("https://example.com")
+    print(markdown)
+
+    # With proxy
+    from pg2md import ProxyConfig
+    proxy = ProxyConfig(server="socks5://1.2.3.4:1080")
+    markdown = parser.parse("https://example.com", proxy=proxy)
+"""
+
+from pg2md.parser import (
+    PageParser,
+    BrowserConfig,
+    ProxyConfig,
+    UserAgents,
+    HtmlCleaner,
+    MarkdownCleaner,
+)
+from pg2md import lightpanda
+
+__version__ = "1.2.0"
+__author__ = "lemantorus"
+__all__ = [
+    "PageParser",
+    "BrowserConfig",
+    "ProxyConfig",
+    "UserAgents",
+    "HtmlCleaner",
+    "MarkdownCleaner",
+    "lightpanda",
+    "__version__",
+]

pg2md-1.2.0/src/pg2md/lightpanda.py

@@ -0,0 +1,163 @@
+"""
+Lightpanda browser auto-downloader and launcher.
+
+Automatically downloads Lightpanda binary from GitHub releases
+and manages the CDP server process.
+
+Supported platforms:
+    - Linux x86_64
+    - macOS arm64 (Apple Silicon)
+"""
+
+import platform
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+
+LIGHTPANDA_VERSION = "nightly"
+
+DOWNLOAD_URLS = {
+    (
+        "Linux",
+        "x86_64",
+    ): "https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-x86_64-linux",
+    (
+        "Darwin",
+        "arm64",
+    ): "https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-aarch64-macos",
+}
+
+
+def get_cache_dir() -> Path:
+    """Returns the cache directory for Lightpanda binary."""
+    xdg_cache = Path.home() / ".cache" / "pg2md"
+    return xdg_cache
+
+
+def get_binary_path() -> Path:
+    """Returns the path to the Lightpanda binary."""
+    return get_cache_dir() / "lightpanda"
+
+
+def get_platform_key() -> tuple[str, str]:
+    """Returns (system, machine) tuple for current platform."""
+    return (platform.system(), platform.machine())
+
+
+def is_supported() -> bool:
+    """Check if Lightpanda is available for current platform."""
+    return get_platform_key() in DOWNLOAD_URLS
+
+
+def is_downloaded() -> bool:
+    """Check if Lightpanda binary is already downloaded."""
+    return get_binary_path().exists()
+
+
+def download(force: bool = False) -> Path:
+    """
+    Download Lightpanda binary if not already present.
+
+    Args:
+        force: Re-download even if binary exists
+
+    Returns:
+        Path to the binary
+
+    Raises:
+        RuntimeError: If platform is not supported
+    """
+    binary_path = get_binary_path()
+
+    if binary_path.exists() and not force:
+        return binary_path
+
+    key = get_platform_key()
+    if key not in DOWNLOAD_URLS:
+        raise RuntimeError(
+            f"Lightpanda not available for {key[0]} {key[1]}. "
+            f"Supported: {list(DOWNLOAD_URLS.keys())}"
+        )
+
+    url = DOWNLOAD_URLS[key]
+    binary_path.parent.mkdir(parents=True, exist_ok=True)
+
+    print(f"[pg2md] Downloading Lightpanda {LIGHTPANDA_VERSION}...")
+    urllib.request.urlretrieve(url, binary_path)
+    binary_path.chmod(0o755)
+    print(f"[pg2md] Saved to {binary_path}")
+
+    return binary_path
+
+
+def start(
+    host: str = "127.0.0.1",
+    port: int = 9222,
+    download_if_missing: bool = True,
+) -> subprocess.Popen:
+    """
+    Start Lightpanda CDP server.
+
+    Args:
+        host: Host to bind to
+        port: Port to bind to
+        download_if_missing: Download binary if not present
+
+    Returns:
+        subprocess.Popen object
+    """
+    if download_if_missing:
+        binary_path = download()
+    else:
+        binary_path = get_binary_path()
+        if not binary_path.exists():
+            raise FileNotFoundError(
+                f"Lightpanda binary not found at {binary_path}. "
+                "Set download_if_missing=True to auto-download."
+            )
+
+    proc = subprocess.Popen(
+        [str(binary_path), "serve", "--host", host, "--port", str(port)],
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    )
+
+    cdp_url = f"ws://{host}:{port}"
+    _wait_for_cdp(cdp_url, timeout=10)
+
+    return proc
+
+
+def _wait_for_cdp(cdp_url: str, timeout: float = 10.0) -> bool:
+    """Wait for CDP server to be ready."""
+    import socket
+
+    host = cdp_url.split("://")[1].split(":")[0]
+    port = int(cdp_url.split(":")[-1])
+
+    start_time = time.time()
+    while time.time() - start_time < timeout:
+        try:
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            sock.settimeout(1)
+            result = sock.connect_ex((host, port))
+            sock.close()
+            if result == 0:
+                time.sleep(0.5)
+                return True
+        except Exception:
+            pass
+        time.sleep(0.2)
+
+    raise TimeoutError(f"Lightpanda CDP server not ready at {cdp_url}")
+
+
+def stop(proc: subprocess.Popen) -> None:
+    """Stop Lightpanda process."""
+    if proc and proc.poll() is None:
+        proc.terminate()
+        try:
+            proc.wait(timeout=5)
+        except subprocess.TimeoutExpired:
+            proc.kill()

pg2md-1.2.0/src/pg2md/parser.py

@@ -0,0 +1,469 @@
+"""
+PageParser — Parser for web pages with JS rendering via Lightpanda/Playwright
+and conversion to clean Markdown.
+
+Dependencies:
+    pip install playwright html-to-markdown beautifulsoup4
+    playwright install chromium  # if using regular Chrome, not Lightpanda
+
+Running Lightpanda (optional, instead of Chrome):
+    ./lightpanda serve --host 127.0.0.1 --port 9222
+
+Usage:
+    parser = PageParser(with_image=False, with_link=False)
+
+    # Without proxy
+    result = parser.parse("https://example.com")
+
+    # With proxy for a specific request
+    proxy = ProxyConfig(server="http://1.2.3.4:8080", username="user", password="pass")
+    result = parser.parse("https://example.com", proxy=proxy)
+
+    print(result)
+"""
+
+import re
+import subprocess
+import time
+import asyncio
+from dataclasses import dataclass
+from typing import Optional, Literal
+
+from bs4 import BeautifulSoup
+from html_to_markdown import convert, ConversionOptions, PreprocessingOptions
+from playwright.async_api import async_playwright, ProxySettings, ViewportSize
+
+from pg2md import lightpanda
+
+
+@dataclass
+class ProxyConfig:
+    """
+    Proxy settings for a single request.
+
+    Supported server formats:
+        http://host:port
+        https://host:port
+        socks5://host:port
+
+    Args:
+        server   : proxy address (required)
+        username : login (optional)
+        password : password (optional)
+        bypass   : comma-separated list of hosts to bypass proxy
+                   (e.g. "localhost,127.0.0.1")
+    """
+
+    server: str
+    username: Optional[str] = None
+    password: Optional[str] = None
+    bypass: Optional[str] = None
+
+    def to_playwright(self) -> ProxySettings:
+        """Converts to Playwright ProxySettings format."""
+        settings: ProxySettings = {"server": self.server}
+        if self.username:
+            settings["username"] = self.username
+        if self.password:
+            settings["password"] = self.password
+        if self.bypass:
+            settings["bypass"] = self.bypass
+        return settings
+
+
+class UserAgents:
+    """Popular User-Agent strings for bypassing blocks."""
+
+    CHROME_DESKTOP = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36"
+    )
+
+    CHROME_MAC = (
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36"
+    )
+
+    FIREFOX_DESKTOP = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"
+    )
+
+    SAFARI_MAC = (
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) "
+        "Version/17.2 Safari/605.1.15"
+    )
+
+    EDGE = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0"
+    )
+
+    GOOGLEBOT_DESKTOP = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+
+    GOOGLEBOT_MOBILE = (
+        "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.71 MobileSafari/537.36 "
+        "(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+    )
+
+    GOOGLEBOT_VIDEO = (
+        "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.71 MobileSafari/537.36 "
+        "Googlebot/2.1"
+    )
+
+    BINGBOT = "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
+
+    BINGBOT_MOBILE = (
+        "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 "
+        "(KHTML, like Gecko) Version/16.6 Mobile/15E148 BingWeb/7.15.13.7055 (advisor; +http://www.bing.com/bingbot.htm)"
+    )
+
+    YANDEXBOT = "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
+
+    DUCKBOT = "DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)"
+
+    APPLEBOT = (
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) "
+        "Version/17.0 Safari/605.1.15 Applebot/0.1"
+    )
+
+    CHROME_LINUX = (
+        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36"
+    )
+
+
+@dataclass
+class BrowserConfig:
+    """
+    Browser connection settings.
+
+    cdp_url options:
+        "auto"  - Auto-download and start Lightpanda (default)
+        None    - Use built-in Chromium via Playwright
+        "ws://host:port" - Connect to existing CDP server (Lightpanda/Chrome)
+    """
+
+    cdp_url: Optional[str] = "auto"
+    lightpanda_bin: Optional[str] = None
+    navigation_timeout: int = 60_000
+    wait_until: Literal["load", "domcontentloaded", "networkidle"] = "domcontentloaded"
+    default_proxy: Optional[ProxyConfig] = None
+    user_agent: Optional[str] = UserAgents.CHROME_DESKTOP
+    extra_headers: Optional[dict[str, str]] = None
+    viewport: Optional[ViewportSize] = None
+
+    def __post_init__(self):
+        if self.viewport is None:
+            self.viewport = {"width": 1920, "height": 1080}
+
+
+class HtmlCleaner:
+    """
+    Cleans HTML before converting to Markdown:
+      - removes <script>, <style>, <noscript>, <svg>, <canvas>, <video>, <audio>
+      - removes <img> (optional)
+      - removes href/src with data:, blob: (base64 junk)
+      - strips links, keeping only text (optional)
+    """
+
+    ALWAYS_STRIP_TAGS = [
+        "script",
+        "style",
+        "noscript",
+        "svg",
+        "canvas",
+        "video",
+        "audio",
+        "iframe",
+        "object",
+        "embed",
+        "head",
+    ]
+
+    def __init__(self, with_image: bool = False, with_link: bool = True):
+        self.with_image = with_image
+        self.with_link = with_link
+
+    def clean(self, html: str) -> str:
+        soup = BeautifulSoup(html, "html.parser")
+
+        for tag in self.ALWAYS_STRIP_TAGS:
+            for el in soup.find_all(tag):
+                el.decompose()
+
+        if not self.with_image:
+            for el in soup.find_all("img"):
+                el.decompose()
+        else:
+            for el in soup.find_all("img"):
+                src = el.get("src", "")
+                if isinstance(src, str) and (src.startswith("data:") or src.startswith("blob:")):
+                    el.decompose()
+
+        if not self.with_link:
+            for el in soup.find_all("a"):
+                el.replace_with(el.get_text())
+        else:
+            for el in soup.find_all("a"):
+                href = el.get("href", "")
+                if isinstance(href, str) and (href.startswith("data:") or href.startswith("blob:")):
+                    el["href"] = ""
+
+        for el in soup.find_all(True):
+            for attr in ("src", "href", "srcset", "poster", "background"):
+                val = el.get(attr, "")
+                if isinstance(val, str) and (val.startswith("data:") or val.startswith("blob:")):
+                    del el[attr]
+
+        return str(soup)
+
+
+class MarkdownCleaner:
+    """Final cleanup of ready Markdown text."""
+
+    _BASE64_LINE = re.compile(r"^[A-Za-z0-9+/=]{40,}\s*$", re.MULTILINE)
+    _BINARY_GARBAGE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]")
+    _EXCESS_NEWLINES = re.compile(r"\n{3,}")
+    _MD_IMAGE = re.compile(r"!\[.*?\]\(.*?\)")
+
+    def clean(self, text: str, strip_images: bool = False) -> str:
+        text = self._BINARY_GARBAGE.sub("", text)
+        text = self._BASE64_LINE.sub("", text)
+
+        if strip_images:
+            text = self._MD_IMAGE.sub("", text)
+
+        text = self._EXCESS_NEWLINES.sub("\n\n", text)
+
+        return text.strip()
+
+
+class PageParser:
+    """
+    Parses web pages with JS rendering and returns clean Markdown.
+
+    Args:
+        with_image     (bool)         : Include images in output. Default False.
+        with_link      (bool)         : Include links (href). Default True.
+                                        False — links are replaced with their text.
+        browser_config (BrowserConfig): Browser connection settings.
+                                        Can set default_proxy for all requests.
+
+    Proxy is passed to parse() / async_parse() per-request:
+        proxy = ProxyConfig(server="socks5://1.2.3.4:1080")
+        result = parser.parse("https://example.com", proxy=proxy)
+    """
+
+    def __init__(
+        self,
+        with_image: bool = False,
+        with_link: bool = True,
+        browser_config: Optional[BrowserConfig] = None,
+    ):
+        self.with_image = with_image
+        self.with_link = with_link
+        self.config = browser_config or BrowserConfig()
+
+        self._html_cleaner = HtmlCleaner(with_image=with_image, with_link=with_link)
+        self._md_cleaner = MarkdownCleaner()
+
+        self._lightpanda_proc: Optional[subprocess.Popen] = None
+        self._auto_lightpanda_proc: Optional[subprocess.Popen] = None
+        self._auto_cdp_url: Optional[str] = None
+
+    def parse(self, url: str, proxy: Optional[ProxyConfig] = None) -> str:
+        """
+        Synchronous wrapper over async_parse.
+
+        Args:
+            url   : page to parse
+            proxy : proxy for this specific request (overrides default_proxy)
+        """
+        return asyncio.run(self.async_parse(url, proxy=proxy))
+
+    async def async_parse(self, url: str, proxy: Optional[ProxyConfig] = None) -> str:
+        """
+        Loads page, renders JS, returns clean Markdown.
+
+        Args:
+            url   : page to parse
+            proxy : proxy for this specific request (overrides default_proxy)
+        """
+        html = await self._fetch_html(url, proxy=proxy)
+        return self._html_to_markdown(html)
+
+    async def async_parse_many(
+        self,
+        urls: list[str],
+        proxy: Optional[ProxyConfig] = None,
+    ) -> dict[str, str | BaseException]:
+        """
+        Parses multiple URLs in parallel.
+
+        Args:
+            urls  : list of pages
+            proxy : one proxy for all requests (or None)
+        """
+        tasks = [self.async_parse(url, proxy=proxy) for url in urls]
+        results = await asyncio.gather(*tasks, return_exceptions=True)
+        return {url: res for url, res in zip(urls, results)}
+
+    async def _fetch_html(self, url: str, proxy: Optional[ProxyConfig] = None) -> str:
+        """Opens browser, loads page, returns HTML."""
+        self._maybe_start_lightpanda()
+
+        effective_proxy = proxy or self.config.default_proxy
+        proxy_settings = effective_proxy.to_playwright() if effective_proxy else None
+
+        headers = {}
+        cdp_url = self._resolve_cdp_url()
+        use_cdp = cdp_url is not None
+        if use_cdp and self.config.user_agent:
+            headers["User-Agent"] = self.config.user_agent
+        if self.config.extra_headers:
+            headers.update(self.config.extra_headers)
+
+        async with async_playwright() as pw:
+            fallback_to_chromium = False
+
+            while True:
+                try:
+                    if use_cdp and not fallback_to_chromium:
+                        browser = await pw.chromium.connect_over_cdp(cdp_url)
+                        context = await browser.new_context(
+                            proxy=proxy_settings,
+                            extra_http_headers=headers if headers else None,
+                            viewport=self.config.viewport,
+                        )
+                    else:
+                        browser = await pw.chromium.launch(headless=True)
+                        context = await browser.new_context(
+                            proxy=proxy_settings,
+                            user_agent=self.config.user_agent,
+                            extra_http_headers=headers if headers else None,
+                            viewport=self.config.viewport,
+                        )
+
+                    page = await context.new_page()
+
+                    try:
+                        await page.goto(
+                            url,
+                            timeout=self.config.navigation_timeout,
+                            wait_until=self.config.wait_until,
+                        )
+                        await page.wait_for_timeout(1500)
+                        html = await page.content()
+                    finally:
+                        await page.close()
+                        await context.close()
+                        await browser.close()
+
+                    return html
+
+                except Exception as e:
+                    if use_cdp and not fallback_to_chromium and self.config.cdp_url == "auto":
+                        print(f"[pg2md] Lightpanda failed: {e}")
+                        print("[pg2md] Falling back to Chromium...")
+                        self._stop_auto_lightpanda()
+                        fallback_to_chromium = True
+                        continue
+                    else:
+                        raise
+
+    def _resolve_cdp_url(self) -> Optional[str]:
+        """Resolve CDP URL based on config."""
+        if self.config.cdp_url == "auto":
+            if self._auto_cdp_url:
+                return self._auto_cdp_url
+
+            if lightpanda.is_supported():
+                self._auto_lightpanda_proc = lightpanda.start()
+                self._auto_cdp_url = "ws://127.0.0.1:9222"
+                return self._auto_cdp_url
+            else:
+                return None
+        return self.config.cdp_url
+
+    def _html_to_markdown(self, html: str) -> str:
+        """Cleans HTML and converts to Markdown."""
+
+        clean_html = self._html_cleaner.clean(html)
+
+        options = ConversionOptions(
+            heading_style="atx",
+            strong_em_symbol="*",
+            bullets="*",
+            escape_asterisks=False,
+        )
+        preprocessing = PreprocessingOptions(
+            enabled=True,
+            preset="aggressive",
+            remove_navigation=True,
+            remove_forms=True,
+        )
+        markdown = convert(clean_html, options, preprocessing)
+
+        markdown = self._md_cleaner.clean(markdown, strip_images=not self.with_image)
+
+        return markdown
+
+    def _maybe_start_lightpanda(self) -> None:
+        """If binary path is set and process not running — start it."""
+        if not self.config.lightpanda_bin:
+            return
+        if self._lightpanda_proc and self._lightpanda_proc.poll() is None:
+            return
+
+        self._lightpanda_proc = subprocess.Popen(
+            [
+                self.config.lightpanda_bin,
+                "serve",
+                "--host",
+                "127.0.0.1",
+                "--port",
+                "9222",
+            ],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        time.sleep(1.5)
+
+    def stop_lightpanda(self) -> None:
+        """Explicitly stops Lightpanda if started by us."""
+        if self._lightpanda_proc:
+            self._lightpanda_proc.terminate()
+            self._lightpanda_proc = None
+
+    def _stop_auto_lightpanda(self) -> None:
+        """Stop auto-launched Lightpanda process."""
+        if self._auto_lightpanda_proc:
+            lightpanda.stop(self._auto_lightpanda_proc)
+            self._auto_lightpanda_proc = None
+            self._auto_cdp_url = None
+
+    def stop(self) -> None:
+        """Stop all browser processes started by this parser."""
+        self.stop_lightpanda()
+        self._stop_auto_lightpanda()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *_):
+        self.stop()
+
+
+if __name__ == "__main__":
+    import sys
+
+    url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com"
+
+    parser = PageParser(with_image=False, with_link=False)
+
+    with parser:
+        result = parser.parse(url)
+
+    print(result)

pg2md-1.2.0/venv/lib/python3.12/site-packages/playwright/driver/README.md

@@ -0,0 +1 @@
+manylinux1_x86_64.whl driver package

pg2md-1.2.0/venv/lib/python3.12/site-packages/playwright/driver/package/README.md

@@ -0,0 +1,3 @@
+# playwright-core
+
+This package contains the no-browser flavor of [Playwright](http://github.com/microsoft/playwright).