scraper-rust 0.4.3__tar.gz → 0.5.0__tar.gz

{scraper_rust-0.4.3 → scraper_rust-0.5.0}/.gitignore

@@ -23,6 +23,7 @@ env/
 .ruff_cache/
 .pytest_cache/
 .hypothesis/
+.uv-cache/
 .coverage
 .coverage.*
 htmlcov/

{scraper_rust-0.4.3 → scraper_rust-0.5.0}/AGENTS.md

@@ -17,10 +17,11 @@ This file contains essential information for Large Language Models (LLMs) to eff
 
 ### Core Technologies
 - **Rust** (2024 edition)
-  - `scraper` v0.25 - HTML parsing and CSS selection
+  - `scraper` v0.27.0 from `rust-scraper/scraper` with the `atomic` feature - HTML parsing and CSS selection
   - `xee-xpath` v0.1.5 - XPath support
-  - `pyo3` v0.27 with `extension-module` and `abi3-py310` features
-  - `pyo3-async-runtimes` v0.27 with `tokio-runtime` feature
+  - `html5ever` v0.39.0 - parse-tree dictionary helpers
+  - `pyo3` v0.28.3 with `abi3-py310`; the `extension-module` feature is enabled through the package build config
+  - `pyo3-async-runtimes` v0.28 with `tokio-runtime` feature
   - `tokio` v1 (rt, macros) for async wrappers
 - **Python** 3.10+
   - Type annotations and stub files (`scraper_rs.pyi`, `py.typed`)
@@ -40,13 +41,25 @@ This file contains essential information for Large Language Models (LLMs) to eff
 ```
 scraper-rs/
 ├── src/
-│   └── lib.rs              # Main Rust source (PyO3 bindings)
+│   ├── lib.rs              # PyO3 module entry point
+│   ├── document.rs         # Document class
+│   ├── element.rs          # Element class and snapshots
+│   ├── functions.rs        # Top-level functions
+│   ├── selectors.rs        # CSS selector helpers/cache
+│   ├── xpath.rs            # XPath helpers/cache
+│   ├── limits.rs           # Size limits and truncation
+│   ├── prettify.rs         # Pretty-print helpers
+│   ├── html5_dict.rs       # parse_document/parse_fragment dictionaries
+│   └── async_core.rs       # Extension-level async core helpers
 ├── tests/
 │   ├── test_scraper.py     # Synchronous API tests
-│   └── test_asyncio.py     # Asynchronous API tests
+│   ├── test_asyncio.py     # Asynchronous API tests
+│   └── test_performance.py # Performance behavior tests
 ├── examples/
 │   ├── demo.py             # Sync usage examples
-│   └── demo_asyncio.py     # Async usage examples
+│   ├── demo_asyncio.py     # Async usage examples
+│   ├── demo_async_document.py
+│   └── demo_prettify_url.py
 ├── scraper_rs/             # Python package directory (async wrappers + import glue)
 ├── scraper_rs.pyi          # Type stub file
 ├── py.typed                # PEP 561 marker for type support
@@ -59,18 +72,17 @@ scraper-rs/
 
 ### Key Files
 
-- **`src/lib.rs`**: Single Rust source file containing all PyO3 bindings
-  - `Document` class: Main HTML document wrapper
-  - `Element` class: Snapshot of HTML element with tag, text, HTML, and attributes
-  - Helper functions: `parse()`, `select()`, `select_first()`, `first()`, `xpath()`, `xpath_first()`
-  - Size limit enforcement (default 1 GiB, configurable via `max_size_bytes`)
-  - Truncation support (`truncate_on_limit` parameter)
+- **`src/lib.rs`**: PyO3 module initializer that registers classes, helper functions, metadata, and extension-level async helpers
+- **`src/document.rs`**: `Document` class and persistent DOM/XPath state
+- **`src/element.rs`**: `Element` class, owned element snapshots, lazy field computation
+- **`src/functions.rs`**: Top-level helpers: `parse()`, `parse_document()`, `parse_fragment()`, `prettify()`, `select()`, `select_first()`, `first()`, `xpath()`, `xpath_first()`
+- **`src/selectors.rs`**, **`src/xpath.rs`**, **`src/limits.rs`**: selector caches, XPath conversion, size limit enforcement, and UTF-8-safe truncation
 
 - **`scraper_rs.pyi`**: Type stubs for IDE support and type checking
 
 - **`Cargo.toml`**: Rust crate configuration
   - `crate-type = ["cdylib"]` for Python extension module
-  - Release optimizations: LTO, `codegen-units = 1`, `opt-level = "z"`
+  - Release optimizations: fat LTO, `codegen-units = 1`, `opt-level = 3`, `panic = "abort"`, `strip = true`
 
 - **`pyproject.toml`**: Python package metadata and build configuration
   - Maturin build backend
@@ -170,7 +182,7 @@ pytest tests/ -v
 ### CI/CD
 - **GitHub Actions**: `.github/workflows/tests.yml`
   - Runs on Ubuntu latest
-  - Tests against Python 3.10, 3.11, 3.12, 3.13, 3.14
+  - Tests against Python 3.10, 3.11, 3.12, 3.13, 3.13t, 3.14, 3.14t
   - Steps: checkout → setup Python/Rust → install deps → build with maturin → run pytest
 
 ## API Design
@@ -201,6 +213,9 @@ pytest tests/ -v
 
 ### Top-Level Functions
 - `parse(html, max_size_bytes, truncate_on_limit) -> Document`
+- `parse_document(html, max_size_bytes, truncate_on_limit) -> dict`
+- `parse_fragment(html, max_size_bytes, truncate_on_limit) -> dict`
+- `prettify(html, max_size_bytes, truncate_on_limit) -> str`
 - `select(html, css, max_size_bytes, truncate_on_limit) -> list[Element]`
 - `select_first(html, css, max_size_bytes, truncate_on_limit) -> Element | None`
 - `first(html, css, max_size_bytes, truncate_on_limit) -> Element | None`
@@ -211,12 +226,12 @@ pytest tests/ -v
 Module: `scraper_rs.asyncio`
 - `AsyncDocument` and `AsyncElement` wrappers mirror the sync API
 - `parse()` yields to the event loop once before constructing the document
-- `select()` and `xpath()` run in a thread pool via `pyo3-async-runtimes` + `tokio::task::spawn_blocking`
+- The public `scraper_rs.asyncio` Python wrappers yield once and then call the sync helpers. They provide coroutine API consistency, but do not make CPU-bound parsing parallel on their own.
 - Same keyword arguments as sync versions
 
 ## Code Conventions
 
-### Rust Code (`src/lib.rs`)
+### Rust Code (`src/`)
 - Use `#[pyclass]` for Python-exposed classes
 - Use `#[pymethods]` for Python-exposed methods
 - Use `#[getter]` for property accessors
@@ -242,7 +257,7 @@ Module: `scraper_rs.asyncio`
 ## Common Development Tasks
 
 ### Adding a New Method to Document
-1. Add Rust implementation in `src/lib.rs` under `impl Document`
+1. Add Rust implementation in `src/document.rs` under `impl Document`
 2. Add `#[pymethods]` annotation if needed
 3. Update `scraper_rs.pyi` with type signature
 4. Add tests in `tests/test_scraper.py`
@@ -252,7 +267,7 @@ Module: `scraper_rs.asyncio`
 8. Run `pytest tests/` to verify
 
 ### Adding a New Top-Level Function
-1. Add Rust function in `src/lib.rs` with `#[pyfunction]`
+1. Add Rust function in `src/functions.rs` with `#[pyfunction]`
 2. Register with `m.add_function(wrap_pyfunction!(function_name, m)?)?` in `scraper_rs` module
 3. Export in `__all__` if applicable
 4. Update `scraper_rs.pyi`
@@ -270,7 +285,8 @@ Module: `scraper_rs.asyncio`
 ### Release Process
 See `.github/workflows/release.yml` and `.github/workflows/bump-version.yml`
 - Version is specified in both `Cargo.toml` and `pyproject.toml` (must match)
-- Manylinux wheels built via Docker with official maturin image
+- Release workflow builds abi3 wheels for Linux, Windows, and macOS, plus free-threaded wheels for Python 3.13t and 3.14t
+- Local manylinux builds are available via Docker with `just build_manylinux`
 - Published to PyPI via `maturin publish`
 
 ## Important Notes for LLMs
@@ -279,8 +295,8 @@ See `.github/workflows/release.yml` and `.github/workflows/bump-version.yml`
 2. **PyO3 limitations**: Not all Rust types can cross the FFI boundary - use owned data
 3. **Size limits**: Always respect `max_size_bytes` to prevent memory issues
 4. **UTF-8 safety**: Truncation must respect character boundaries
-5. **Async wrappers**: Rust exposes async helpers via `pyo3-async-runtimes` + `tokio::task::spawn_blocking`; parsing/selectors remain synchronous
-6. **Single source file**: All Rust code is in `src/lib.rs` (keep it organized)
+5. **Async wrappers**: Public async wrappers live in `scraper_rs/asyncio.py`; keep their stubs and tests aligned with sync behavior
+6. **Rust modules are split by responsibility**: `src/lib.rs` is the module entry point; add core behavior to the focused module that owns it
 7. **Type stubs are critical**: Always update `scraper_rs.pyi` when changing APIs
 8. **Tests are comprehensive**: Follow existing test patterns in `tests/`
 9. **Build artifacts**: `target/` and `.venv/` should not be committed
@@ -289,11 +305,12 @@ See `.github/workflows/release.yml` and `.github/workflows/bump-version.yml`
 ## Dependencies
 
 ### Rust Dependencies (Cargo.toml)
-- `pyo3 = { version = "0.27", features = ["extension-module", "abi3-py310"] }`
-- `pyo3-async-runtimes = { version = "0.27", features = ["tokio-runtime"] }`
+- `pyo3 = { version = "0.28.3", features = ["abi3-py310"] }`
+- `pyo3-async-runtimes = { version = "0.28", features = ["tokio-runtime"] }`
 - `tokio = { version = "1", features = ["rt", "macros"] }`
-- `scraper = "0.25"`
+- `scraper = { version = "0.27.0", git = "https://github.com/rust-scraper/scraper", features = ["atomic"], rev = "9c1eff304e45a8bccb463968268ce6758703e821" }`
 - `xee-xpath = "0.1.5"`
+- `html5ever = "0.39.0"`
 
 ### Python Dependencies (pyproject.toml)
 **Runtime**: None (self-contained binary wheel)