htmlsift 0.1.0__tar.gz → 0.2.0__tar.gz

htmlsift-0.2.0/.github/workflows/publish.yml

@@ -0,0 +1,39 @@
+name: Publish
+
+# Publishes to PyPI via Trusted Publishing (OIDC) -- no API token or secret.
+# Fires when a GitHub release is published; the release tag should match the
+# version in src/htmlsift/__init__.py (e.g. tag v0.1.1 for version 0.1.1).
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade build
+          python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write          # required: lets PyPI mint a short-lived OIDC token
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

{htmlsift-0.1.0 → htmlsift-0.2.0}/.gitignore

@@ -44,5 +44,3 @@ desktop.ini
 htmlcov/
 .tox/
 .nox/
-
-.env

htmlsift-0.2.0/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to htmlsift are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-11-02
+
+### Added
+- `extract()` accepts `bytes` and decodes them: byte order mark, then `encoding=`, then the
+  page's `<meta>` declaration, then utf-8, falling back to cp1252. Legacy charset labels map
+  the way browsers read them (`latin1` → cp1252, `shift_jis` → cp932, `gb2312` → gb18030).
+- Keyword-only `encoding=` on `Extractor.extract` and `extract`, for a charset known from
+  an HTTP or WARC header.
+- Dependency: `w3lib` (BSD-3-Clause) for charset sniffing.
+- Keyword-only `base_url=` on `Extractor.extract` and `extract`: links and image URLs in
+  `html` and `markdown` output are made absolute against it. The page's first
+  `<base href>` is resolved against `base_url`; malformed links are left as they are.
+  `text` output and `with_blocks` are unaffected.
+- `htmlsift extract` command: files, quoted globs or stdin in; the content of a single
+  input, or one JSON line per file (`--jsonl` to force), out. Flags `--output`,
+  `--mode`, `--device`, `--base-url`, `--encoding`. Failed files are reported as
+  records and the run exits with 1.
+
+### Changed
+- Input other than `str` or `bytes` raises `TypeError`; `extract(None)` previously
+  returned `""`.
+- `encoding=` with `str` input raises `TypeError`, since a `str` is already decoded.
+- A page with an absolute `<base href>` gets absolute links in `html` and `markdown` output
+  even without `base_url`. Previously `<base>` was dropped with `<head>`, leaving its links
+  relative.
+
+### Fixed
+- Non-utf-8 `bytes` input crashed with `UnicodeDecodeError`.
+- `html` and `markdown` output could drop images and other inline content inside a kept
+  block, varying from run to run on the same page. `text` output was unaffected.
+
+## [0.1.1] - 2026-09-23
+
+Documentation and packaging metadata only — no code or model changes since 0.1.0.
+
+### Added
+- Project URLs in package metadata (research repo, Hugging Face weights, issue tracker).
+- PyPI Trusted Publishing workflow (`.github/workflows/publish.yml`).
+- Cross-benchmark accuracy table (`base`/`mini` vs trafilatura, readability, resiliparse on
+  WMB, WCXB, DAnIEL) and an "Annotation is a policy" section in the README.
+- GPU-on-Windows install note (CUDA torch from the PyTorch index).
+
+### Changed
+- Stated the model-weight license (Apache-2.0) and added WebMainBench attribution.
+- Cited the WebMainBench source paper (Liu et al., *Dripper*, arXiv:2511.23119).
+- Clarified model shorthand and tightened copy in the README and the PyPI description.
+
+## [0.1.0] - 2026-09-22
+
+### Added
+- Initial release.
+- `mini` model (int8 ONNX, runs on CPU) and `base` model (311M encoder, GPU-capable),
+  installed via `pip install htmlsift` and `pip install htmlsift[full]` respectively.
+- `Extractor(mode, *, bundle_dir, threads, device)` with
+  `.extract(html, output=..., with_blocks=...)`, plus the module-level `extract(...)`.
+- Output formats: plain text, HTML, and markdown.
+- Device selection for `base`: `device="cuda"` / `"cpu"`, auto-detecting a GPU and
+  warning on CPU fallback.
+- Model bundles fetched from the Hugging Face Hub, pinned to a fixed commit revision.
+
+[Unreleased]: https://github.com/koivualeksi/htmlsift/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/koivualeksi/htmlsift/compare/v0.1.1...v0.2.0
+[0.1.1]: https://github.com/koivualeksi/htmlsift/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/koivualeksi/htmlsift/releases/tag/v0.1.0

htmlsift-0.2.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.5
+Name: htmlsift
+Version: 0.2.0
+Summary: Main-content extraction for web pages
+Project-URL: Repository, https://github.com/koivualeksi/htmlsift
+Project-URL: Research, https://github.com/koivualeksi/htmlsift-research
+Project-URL: Model weights, https://huggingface.co/koivualeksi/htmlsift-release
+Project-URL: Issues, https://github.com/koivualeksi/htmlsift/issues
+Author-email: Aleksi Koivu <koivu.aleksi@gmail.com>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: boilerplate-removal,content-extraction,html,main-content,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Requires-Python: >=3.10
+Requires-Dist: huggingface-hub
+Requires-Dist: lxml
+Requires-Dist: numpy
+Requires-Dist: onnxruntime
+Requires-Dist: tokenizers
+Requires-Dist: w3lib<3,>=2.5
+Provides-Extra: full
+Requires-Dist: torch; extra == 'full'
+Requires-Dist: transformers; extra == 'full'
+Description-Content-Type: text/markdown
+
+# htmlsift
+
+Web main-content extraction: HTML in, clean text / HTML / markdown out. Trained on
+[WebMainBench](https://github.com/opendatalab/WebMainBench), scores each rendered line of a
+page as main content or boilerplate (nav, ads, cookie banners, and footers) and returns just
+the content.
+
+Apache-2.0. Full documentation, diagrams, and benchmarks:
+https://github.com/koivualeksi/htmlsift
+
+## Results
+
+| model | desc | WMB test F1 | seeds | CPU (pages/s) | GPU (pages/s) | install |
+|---|---|---|---|---|---|---|
+| `base` | 311M, 10 layers | **0.9311** | 3 | 0.4 | 24.6 | `htmlsift[full]` |
+| unpublished | 97M, 6 layers, int8 QAT | 0.9256 | 3 | 0.9 | 30.2 | — |
+| `mini` | int8 embedding table | 0.9010 | 3 | **27.7** | — | `htmlsift` |
+
+GPU = RTX 4090, CPU = Ryzen 5 3600 single thread; WMB test = 544 pages (one of the 545 has
+no gold reference). On WMB (in-domain), `mini` scores ~15 F1 points above trafilatura at
+comparable CPU speed.
+
+### Cross-benchmark accuracy (zero-shot)
+
+Trained on WMB, scored on benchmarks the model never saw. Each column uses that benchmark's own metric.
+
+| model | WMB (ROUGE-5) | WCXB (word-F1) | DAnIEL (ROUGE-L) |
+|---|---|---|---|
+| `base` | 0.9311 | 0.8633 | 0.9175 |
+| `mini` | 0.9010 | 0.8474 | 0.8797 |
+| trafilatura | 0.7525 | 0.8584 | 0.8265 |
+| readability | 0.8016 | 0.7653 | 0.8925 |
+| resiliparse | 0.7135 | 0.7909 | 0.7094 |
+
+Model rows are 3-seed test means; heuristics are single deterministic runs. "Main content" is a labeling policy: htmlsift leads in-domain and stays competitive off-policy — retrain on your own labels if your definition differs. Details in the [research repo](https://github.com/koivualeksi/htmlsift-research).
+
+## Install
+
+```bash
+pip install htmlsift          # default: the mini model (CPU, ONNX)
+pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
+```
+
+- `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
+- `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
+  downloads ~1 GB.
+
+Requires Python 3.10+. Model files download from the Hugging Face Hub on first use and are
+cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
+cache only.
+
+## Usage
+
+```python
+from htmlsift import Extractor
+
+ex = Extractor()                     # defaults to "mini"
+ex.extract(html)                     # -> str (plain text)
+ex.extract(html, output="html")      # structure, links, images kept
+ex.extract(html, output="markdown")  # clean markdown: [text](url), ![alt](src)
+ex.extract(html, with_blocks=True)   # -> (blocks, probs)
+
+import htmlsift
+htmlsift.extract(html)               # module-level convenience
+```
+
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
+The larger model, from `htmlsift[full]`:
+
+```python
+ex = Extractor("base")               # 311M encoder; GPU when present, else CPU (with a warning)
+ex = Extractor("base", device="cuda")  # require the GPU
+```
+
+`mini` is CPU-only; only `base` runs on the GPU.
+
+## Output modes
+
+| `output` | what you get |
+|---|---|
+| `text` (default) | selected content as plain text |
+| `html` | selected DOM, with structure / links / images kept |
+| `markdown` | clean markdown, links and images preserved |
+
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
+## License
+
+Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and
+are trained on [WebMainBench](https://github.com/opendatalab/WebMainBench) (Apache-2.0); please
+attribute WebMainBench when you use them.

{htmlsift-0.1.0 → htmlsift-0.2.0}/README.md

@@ -7,7 +7,7 @@ Main-content extraction for web pages: hand it a raw HTML string and get back th
 Two models shipped, both trained on [WebMainBench][wmb] (WMB) — 7,825 labelled pages, custom split
 6,548 train / 732 val / 545 test:
 
-- **`mini`** (default) - a small int8-ONNX model that runs on CPU. `pip install htmlsift`
+- **`mini`** (default) - a fast int8-ONNX model that runs on CPU. `pip install htmlsift`
 - **`base`** - a 311M encoder, most accurate and GPU-capable. `pip install htmlsift[full]`
 
 Each model is fully re-trainable with your own annotated data and the whole research suite is available at [htmlsift-research](https://github.com/koivualeksi/htmlsift-research)
@@ -18,12 +18,13 @@ Each model is fully re-trainable with your own annotated data and the whole rese
 
 | model | desc | WMB test F1 | seeds | CPU (pages/s) | GPU (pages/s) | install |
 |---|---|---|---|---|---|---|
-| `base` | 311m-10 | **0.9311** | 3 | 0.4 | 24.6 | `htmlsift[full]` |
-| unpublished | 97m-6, int8 QAT | 0.9256 | 3 | 0.9 | 30.2 | — |
-| `mini` | int8 table | 0.9010 | 3 | **27.7** | — | `htmlsift` |
+| `base` | 311M, 10 layers | **0.9311** | 3 | 0.4 | 24.6 | `htmlsift[full]` |
+| unpublished | 97M, 6 layers, int8 QAT | 0.9256 | 3 | 0.9 | 30.2 | — |
+| `mini` | int8 embedding table | 0.9010 | 3 | **27.7** | — | `htmlsift` |
 
 * GPU = RTX 4090
 * CPU = Ryzen 5 3600 single thread
+* WMB test = 544 pages: one of the 545 test pages has no gold reference and is skipped in scoring.
 
 ### On CPU
 
@@ -36,21 +37,25 @@ per-page time (36 vs 31 ms median) and scores ~15 F1 points higher. readability
 
 ![htmlsift base sustains 21x the throughput of MinerU-HTML at matched F1 on an RTX 4090](docs/assets/gpu_speed_accuracy_lite.svg)
 
-RTX 4090, WMB test (544 pages). `base` with band attention sustains 24.6 pages/s against
+RTX 4090, WMB test (544 pages). `base` sustains 24.6 pages/s against
 MinerU-HTML v1.1's 1.16. A 21x gap at matched accuracy (F1 0.9311 vs 0.9306).
 
 ### Cross benchmark accuracy
 
 The shipped models are trained on WMB, so WMB is in-domain. The other two columns are
 **zero-shot**, the model never saw WCXB or the multilingual DAnIEL in training. Each column is
-scored with that benchmark's own metric.
+scored with that benchmark's own metric. `base` leads every heuristic here; `mini` lands just
+behind the best on each board — see **Annotation is a policy** under Concept.
 
 | model | WMB (ROUGE-5) | WCXB (word-F1, zero-shot) | DAnIEL (ROUGE-L, zero-shot) |
 |---|---|---|---|
-| `base` (311m-10) | 0.9311 | 0.8633 | 0.9175 |
-| `mini` (int8) | 0.9010 | 0.8474 | 0.8797 |
+| `base` | 0.9311 | 0.8633 | 0.9175 |
+| `mini` | 0.9010 | 0.8474 | 0.8797 |
+| trafilatura | 0.7525 | 0.8584 | 0.8265 |
+| readability | 0.8016 | 0.7653 | 0.8925 |
+| resiliparse | 0.7135 | 0.7909 | 0.7094 |
 
-All results are 3-seed test means.
+Model rows are 3-seed test means; heuristics are single deterministic runs.
 
 ## Installation
 
@@ -59,8 +64,8 @@ pip install htmlsift          # default: the mini model (CPU, ONNX)
 pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
-- **`htmlsift`** — the `mini` model only; a small install (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+- **`htmlsift`** — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - **`htmlsift[full]`** — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -87,6 +92,36 @@ import htmlsift
 htmlsift.extract(html)               # default mini
 ```
 
+### Bytes and encodings
+
+Pass raw bytes from a file, an HTTP response or a WARC record, and htmlsift decodes them:
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+The charset is taken from, in order: a byte order mark, `encoding=`, the page's
+`<meta charset>`; an undeclared page is read as utf-8, falling back to cp1252. Unknown
+labels are skipped and undecodable bytes become `�`. A `str` counts as already decoded, so
+`encoding=` with a `str` raises `TypeError`. Prefer passing bytes over a `str` decoded
+upstream: `requests`' `response.text`, for one, guesses ISO-8859-1 for pages without a
+charset header.
+
+### Absolute links
+
+Pass the page's URL to make links and image URLs absolute in `html` and `markdown` output:
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+# [the report](report)  ->  [the report](https://example.com/blog/report)
+```
+
+The page's first `<base href>` is resolved against `base_url`, as a browser does. Without
+`base_url`, an absolute `<base href>` is used on its own; with neither, links stay as they
+are. Malformed links are left unchanged. `text` output has no links, so it ignores
+`base_url`.
+
 The larger model, from the `[full]` install:
 
 ```python
@@ -131,6 +166,35 @@ Composite image showing the global distribution of photosynthesis, including bot
 | `markdown` | clean markdown, links and images preserved |
 
 
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+htmlsift extract page.html --output markdown --base-url https://example.com/page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" > out.jsonl         # one JSON record per file
+htmlsift extract "crawl/**/*.html" --mode base --device cuda > out.jsonl
+```
+
+One file or `-` (stdin) prints the content itself. Several files or a glob print one JSON
+line per file, `{"path": ..., "content": ...}`, or `{"path": ..., "error": ...}` for a file
+that failed; the run continues and exits with 1 at the end. Pass `--jsonl` to get records
+for a single file too, which scripts should do: bash expands an unquoted `*.html` before
+htmlsift sees it. Quoted globs are expanded by htmlsift itself (`**` recurses), so they
+work the same in every shell; directories are not walked.
+
+| flag | |
+|---|---|
+| `--output text\|html\|markdown` | as `output=` above (default `text`) |
+| `--mode mini\|base`, `--device` | model and torch device, as for `Extractor` |
+| `--base-url URL` | make links absolute; applies to every input |
+| `--encoding ENC` | charset of the input bytes; applies to every input |
+| `--jsonl` | always print JSON records |
+
+Files are read as bytes, so the charset is detected as described above. Output is utf-8.
+Exit codes: 0 ok, 1 a file or the model failed, 2 a usage error. In Windows PowerShell 5.1,
+`>` re-encodes a program's output; redirect from bash, cmd or PowerShell 7 instead.
+
 ## Concept
 
 <details>
@@ -161,6 +225,19 @@ Both tiers share the render, the per-line pooling, and the BiGRU head. They diff
 - base uses granite r2 311m embedder using only first 10 layers requiring GPU for efficient usage 
 
 
+</details>
+
+<details>
+<summary><b>Annotation is a policy</b></summary>
+
+![The same page kept differently under different benchmark annotation policies](docs/assets/annotation-example.svg)
+
+htmlsift learns what counts as main content from a benchmark's labels, so it's most accurate on pages that follow that policy. On WebMainBench (the benchmark the shipped models train on) it beats the common heuristic extractors by a wide margin. Zero-shot on benchmarks with different policies, the gap narrows: the 311M `base` still edges every heuristic tested (trafilatura, readability, resiliparse) on both WCXB and DAnIEL, and `mini` lands just behind the strongest heuristic on each board, ahead of the rest.
+
+Train htmlsift on a different policy and it turns around. Fit it to WCXB instead of WMB and it tops WCXB but loses points on other benchmarks. For example when training `base` model on WCXB data, WMB falls to 0.8479, DAnIEL to 0.8826 while WCXB climbs up to 0.9209. Whichever policy you train on, htmlsift leads there and pays for it elsewhere.
+
+For more information, review the [htmlsift-research](https://github.com/koivualeksi/htmlsift-research) readme.
+
 </details>
 
 <details>
@@ -183,20 +260,25 @@ and its `docs/REPRODUCE.md`.
 A standalone hero image that is its own block renders to no text line, so it can't be selected
 today. Inline images inside a kept block are preserved in `html` and `markdown`.
 
+`base_url` does not rewrite `srcset`, so responsive images in `html` output keep relative
+candidates (`src` is rewritten; `markdown` uses only `src`).
+
 </details>
 
 ## License
 
 Apache-2.0. See [LICENSE](LICENSE).
 
-htmlsift does **not** depend on `html2text`, directly or transitively — rendering and
-serialization are lxml-only, which is what keeps the licence clean. Model weights are
-distributed separately, from the Hugging Face Hub.
+Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and are
+trained on [WebMainBench](https://github.com/opendatalab/WebMainBench) (Apache-2.0); please
+attribute WebMainBench when you use them.
 
 ## References
 
 - **WMB — WebMainBench.** The training benchmark; ROUGE-5 main-content metric.
-  [opendatalab/WebMainBench](https://github.com/opendatalab/WebMainBench)
+  Liu et al., *Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM*.
+  [arXiv:2511.23119](https://arxiv.org/abs/2511.23119) ·
+  [repo](https://github.com/opendatalab/WebMainBench)
 - **WCXB — Web Content Extraction Benchmark.** Foley, *WCXB: A Multi-Type Web Content Extraction
   Benchmark*. [arXiv:2605.21097](https://arxiv.org/abs/2605.21097) ·
   [repo](https://github.com/Murrough-Foley/web-content-extraction-benchmark)

htmlsift-0.2.0/README.pypi.md

@@ -0,0 +1,116 @@
+# htmlsift
+
+Web main-content extraction: HTML in, clean text / HTML / markdown out. Trained on
+[WebMainBench](https://github.com/opendatalab/WebMainBench), scores each rendered line of a
+page as main content or boilerplate (nav, ads, cookie banners, and footers) and returns just
+the content.
+
+Apache-2.0. Full documentation, diagrams, and benchmarks:
+https://github.com/koivualeksi/htmlsift
+
+## Results
+
+| model | desc | WMB test F1 | seeds | CPU (pages/s) | GPU (pages/s) | install |
+|---|---|---|---|---|---|---|
+| `base` | 311M, 10 layers | **0.9311** | 3 | 0.4 | 24.6 | `htmlsift[full]` |
+| unpublished | 97M, 6 layers, int8 QAT | 0.9256 | 3 | 0.9 | 30.2 | — |
+| `mini` | int8 embedding table | 0.9010 | 3 | **27.7** | — | `htmlsift` |
+
+GPU = RTX 4090, CPU = Ryzen 5 3600 single thread; WMB test = 544 pages (one of the 545 has
+no gold reference). On WMB (in-domain), `mini` scores ~15 F1 points above trafilatura at
+comparable CPU speed.
+
+### Cross-benchmark accuracy (zero-shot)
+
+Trained on WMB, scored on benchmarks the model never saw. Each column uses that benchmark's own metric.
+
+| model | WMB (ROUGE-5) | WCXB (word-F1) | DAnIEL (ROUGE-L) |
+|---|---|---|---|
+| `base` | 0.9311 | 0.8633 | 0.9175 |
+| `mini` | 0.9010 | 0.8474 | 0.8797 |
+| trafilatura | 0.7525 | 0.8584 | 0.8265 |
+| readability | 0.8016 | 0.7653 | 0.8925 |
+| resiliparse | 0.7135 | 0.7909 | 0.7094 |
+
+Model rows are 3-seed test means; heuristics are single deterministic runs. "Main content" is a labeling policy: htmlsift leads in-domain and stays competitive off-policy — retrain on your own labels if your definition differs. Details in the [research repo](https://github.com/koivualeksi/htmlsift-research).
+
+## Install
+
+```bash
+pip install htmlsift          # default: the mini model (CPU, ONNX)
+pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
+```
+
+- `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
+- `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
+  downloads ~1 GB.
+
+Requires Python 3.10+. Model files download from the Hugging Face Hub on first use and are
+cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
+cache only.
+
+## Usage
+
+```python
+from htmlsift import Extractor
+
+ex = Extractor()                     # defaults to "mini"
+ex.extract(html)                     # -> str (plain text)
+ex.extract(html, output="html")      # structure, links, images kept
+ex.extract(html, output="markdown")  # clean markdown: [text](url), ![alt](src)
+ex.extract(html, with_blocks=True)   # -> (blocks, probs)
+
+import htmlsift
+htmlsift.extract(html)               # module-level convenience
+```
+
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
+The larger model, from `htmlsift[full]`:
+
+```python
+ex = Extractor("base")               # 311M encoder; GPU when present, else CPU (with a warning)
+ex = Extractor("base", device="cuda")  # require the GPU
+```
+
+`mini` is CPU-only; only `base` runs on the GPU.
+
+## Output modes
+
+| `output` | what you get |
+|---|---|
+| `text` (default) | selected content as plain text |
+| `html` | selected DOM, with structure / links / images kept |
+| `markdown` | clean markdown, links and images preserved |
+
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
+## License
+
+Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and
+are trained on [WebMainBench](https://github.com/opendatalab/WebMainBench) (Apache-2.0); please
+attribute WebMainBench when you use them.