htmlsift 0.1.1__tar.gz → 0.2.1__tar.gz

htmlsift-0.2.1/.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  schedule:
+    - cron: "17 4 * * 1"   # Mondays 04:17 UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install (base deps only, no torch)
+        run: pip install -e . ruff pytest
+      - name: Show installed versions
+        run: pip list
+      - name: Lint
+        run: ruff check src tests
+      - name: Read pinned bundle revision
+        id: rev
+        run: |
+          rev=$(sed -n 's/^REVISION = "\([0-9a-f]\{40\}\)".*/\1/p' src/htmlsift/_artifacts.py)
+          test -n "$rev" || { echo "REVISION not found in _artifacts.py"; exit 1; }
+          echo "rev=$rev" >> "$GITHUB_OUTPUT"
+      - name: Cache mini bundle
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface/hub
+          key: htmlsift-mini-${{ steps.rev.outputs.rev }}
+      - name: Fetch mini bundle (no network on a cache hit)
+        run: |
+          htmlsift download mini
+          dir="$HOME/.cache/huggingface/hub/models--koivualeksi--htmlsift-release/snapshots/${{ steps.rev.outputs.rev }}/mini"
+          test -f "$dir/manifest.json" || { echo "bundle not at $dir"; exit 1; }
+          echo "HTMLSIFT_BUNDLE=$dir" >> "$GITHUB_ENV"
+      - name: Test (model goldens included)
+        run: pytest -q -rs

htmlsift-0.2.1/CHANGELOG.md

@@ -0,0 +1,111 @@
+# Changelog
+
+All notable changes to htmlsift are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.1] - 2026-11-08
+
+### Added
+- `htmlsift --version` prints the htmlsift and Python versions and the installed
+  backends: onnxruntime for `mini`, torch and transformers for `base`.
+- `drop_title=True` in `extract()` and `--drop-title` in the CLI leave out the page title
+  that starts the content: the first selected line, when it comes from an `h1`–`h6` or
+  `p` element holding only that line and matches the page's `<title>` or `og:title`.
+  Applies to `text`, `html` and `markdown`. Off by default.
+
+### Changed
+- With w3lib 2.6 or newer, the five bytes windows-1252 leaves undefined (`0x81`, `0x8D`,
+  `0x8F`, `0x90`, `0x9D`) decode to C1 control characters, as in browsers, instead of
+  `U+FFFD`.
+- `output="text"` (the default) returns plain text, as documented. Heading `#`, emphasis,
+  inline-code and strikethrough markers, `>` quote prefixes, `* * *` rules and code-fence
+  lines are no longer in it; list bullets, ordered-list numbers and table rows stay.
+  Literal `#`, `*`, `_` or `|` in the page text are kept. The selected content is the
+  same as in 0.2.0, and the model input and predictions are unchanged.
+- `text` output renders the page a second time, without markers: about 3 ms per page
+  more on one CPU core (mini, measured on CommonCrawl pages).
+- The lines as the model scored them, markers included, are still available through
+  `with_blocks=True`.
+
+### Fixed
+- `output="text"` no longer contains lines made only of invisible characters (zero-width
+  spaces and joiners, byte order marks, soft hyphens, control characters), which pages
+  use as spacers.
+- `mini` memory on very long pages. The embedding step no longer grows with page length
+  (a 3M-token page peaked at 39 GB, now 1.2 GB), and pages estimated above 100 MB hand
+  their memory back to the system when they finish instead of keeping it for the life
+  of the process. Output is unchanged; model time is about 15 % lower on CommonCrawl
+  pages.
+- `mode="base"` without torch or transformers installed raised a bare
+  `No module named 'torch'`, and only after downloading the ~1 GB base bundle. It now
+  raises `ImportError` before any download, naming the missing packages and
+  `pip install "htmlsift[full]"`. The CLI prints the same message.
+
+## [0.2.0] - 2026-11-02
+
+### Added
+- `extract()` accepts `bytes` and decodes them: byte order mark, then `encoding=`, then the
+  page's `<meta>` declaration, then utf-8, falling back to cp1252. Legacy charset labels map
+  the way browsers read them (`latin1` → cp1252, `shift_jis` → cp932, `gb2312` → gb18030).
+- Keyword-only `encoding=` on `Extractor.extract` and `extract`, for a charset known from
+  an HTTP or WARC header.
+- Dependency: `w3lib` (BSD-3-Clause) for charset sniffing.
+- Keyword-only `base_url=` on `Extractor.extract` and `extract`: links and image URLs in
+  `html` and `markdown` output are made absolute against it. The page's first
+  `<base href>` is resolved against `base_url`; malformed links are left as they are.
+  `text` output and `with_blocks` are unaffected.
+- `htmlsift extract` command: files, quoted globs or stdin in; the content of a single
+  input, or one JSON line per file (`--jsonl` to force), out. Flags `--output`,
+  `--mode`, `--device`, `--base-url`, `--encoding`. Failed files are reported as
+  records and the run exits with 1.
+
+### Changed
+- Input other than `str` or `bytes` raises `TypeError`; `extract(None)` previously
+  returned `""`.
+- `encoding=` with `str` input raises `TypeError`, since a `str` is already decoded.
+- A page with an absolute `<base href>` gets absolute links in `html` and `markdown` output
+  even without `base_url`. Previously `<base>` was dropped with `<head>`, leaving its links
+  relative.
+
+### Fixed
+- Non-utf-8 `bytes` input crashed with `UnicodeDecodeError`.
+- `html` and `markdown` output could drop images and other inline content inside a kept
+  block, varying from run to run on the same page. `text` output was unaffected.
+
+## [0.1.1] - 2026-09-23
+
+Documentation and packaging metadata only — no code or model changes since 0.1.0.
+
+### Added
+- Project URLs in package metadata (research repo, Hugging Face weights, issue tracker).
+- PyPI Trusted Publishing workflow (`.github/workflows/publish.yml`).
+- Cross-benchmark accuracy table (`base`/`mini` vs trafilatura, readability, resiliparse on
+  WMB, WCXB, DAnIEL) and an "Annotation is a policy" section in the README.
+- GPU-on-Windows install note (CUDA torch from the PyTorch index).
+
+### Changed
+- Stated the model-weight license (Apache-2.0) and added WebMainBench attribution.
+- Cited the WebMainBench source paper (Liu et al., *Dripper*, arXiv:2511.23119).
+- Clarified model shorthand and tightened copy in the README and the PyPI description.
+
+## [0.1.0] - 2026-09-22
+
+### Added
+- Initial release.
+- `mini` model (int8 ONNX, runs on CPU) and `base` model (311M encoder, GPU-capable),
+  installed via `pip install htmlsift` and `pip install htmlsift[full]` respectively.
+- `Extractor(mode, *, bundle_dir, threads, device)` with
+  `.extract(html, output=..., with_blocks=...)`, plus the module-level `extract(...)`.
+- Output formats: plain text, HTML, and markdown.
+- Device selection for `base`: `device="cuda"` / `"cpu"`, auto-detecting a GPU and
+  warning on CPU fallback.
+- Model bundles fetched from the Hugging Face Hub, pinned to a fixed commit revision.
+
+[Unreleased]: https://github.com/koivualeksi/htmlsift/compare/v0.2.1...HEAD
+[0.2.1]: https://github.com/koivualeksi/htmlsift/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/koivualeksi/htmlsift/compare/v0.1.1...v0.2.0
+[0.1.1]: https://github.com/koivualeksi/htmlsift/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/koivualeksi/htmlsift/releases/tag/v0.1.0

{htmlsift-0.1.1 → htmlsift-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: htmlsift
-Version: 0.1.1
+Version: 0.2.1
 Summary: Main-content extraction for web pages
 Project-URL: Repository, https://github.com/koivualeksi/htmlsift
 Project-URL: Research, https://github.com/koivualeksi/htmlsift-research
@@ -24,6 +24,7 @@ Requires-Dist: lxml
 Requires-Dist: numpy
 Requires-Dist: onnxruntime
 Requires-Dist: tokenizers
+Requires-Dist: w3lib<3,>=2.5
 Provides-Extra: full
 Requires-Dist: torch; extra == 'full'
 Requires-Dist: transformers; extra == 'full'
@@ -73,7 +74,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -81,6 +82,12 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
 cache only.
 
+### Memory (`mini`)
+
+~340 MB once loaded; 99 % of CommonCrawl pages need under 50 MB more. Memory grows with
+page length (~21 KB per rendered line), so a very large page can exhaust RAM on a 1–2 GB
+machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -96,6 +103,29 @@ import htmlsift
 htmlsift.extract(html)               # module-level convenience
 ```
 
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
+Leave out the page title that starts the content, as pretraining text pipelines usually do
+(the first line, when it is an `h1`–`h6` or `p` matching the page's `<title>` or
+`og:title`; kept by default):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -109,10 +139,32 @@ ex = Extractor("base", device="cuda")  # require the GPU
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+htmlsift --version                                  # htmlsift, Python and backend versions
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--drop-title`,
+`--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
 ## License
 
 Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and

{htmlsift-0.1.1 → htmlsift-0.2.1}/README.md

@@ -2,7 +2,7 @@
 
 Main-content extraction for web pages: hand it a raw HTML string and get back the main content
  - nav, ads, cookie banners, and footers stripped
- - outputs raw text, **HTML**, or **markdown**.
+ - outputs plain text, **HTML**, or **markdown**.
 
 Two models shipped, both trained on [WebMainBench][wmb] (WMB) — 7,825 labelled pages, custom split
 6,548 train / 732 val / 545 test:
@@ -65,7 +65,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - **`htmlsift`** — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - **`htmlsift[full]`** — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -73,6 +73,17 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; prefetch with `htmlsift download mini` (or `base`), or set `HF_HUB_OFFLINE=1` to run
 from cache only.
 
+### Memory (`mini`)
+
+- **Fixed:** ~340 MB once the model is loaded.
+- **Per page:** 99 % of CommonCrawl pages need under 50 MB on top of that.
+- **Long pages:** memory grows with the page, about 21 KB per rendered line plus ~0.2 KB
+  per token. The largest page in an 8,820-page CommonCrawl sample needed ~0.8 GB.
+- **Release:** a page estimated above 100 MB hands its memory back to the system when it
+  finishes, so one large page doesn't leave a long-running worker big.
+- **Small machines:** page size has no built-in limit, so a very large page can exhaust
+  RAM on a 1–2 GB machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -92,6 +103,52 @@ import htmlsift
 htmlsift.extract(html)               # default mini
 ```
 
+### Bytes and encodings
+
+Pass raw bytes from a file, an HTTP response or a WARC record, and htmlsift decodes them:
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+The charset is taken from, in order: a byte order mark, `encoding=`, the page's
+`<meta charset>`; an undeclared page is read as utf-8, falling back to cp1252. Unknown
+labels are skipped and undecodable bytes become `�`. A `str` counts as already decoded, so
+`encoding=` with a `str` raises `TypeError`. Prefer passing bytes over a `str` decoded
+upstream: `requests`' `response.text`, for one, guesses ISO-8859-1 for pages without a
+charset header.
+
+### Absolute links
+
+Pass the page's URL to make links and image URLs absolute in `html` and `markdown` output:
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+# [the report](report)  ->  [the report](https://example.com/blog/report)
+```
+
+The page's first `<base href>` is resolved against `base_url`, as a browser does. Without
+`base_url`, an absolute `<base href>` is used on its own; with neither, links stay as they
+are. Malformed links are left unchanged. `text` output has no links, so it ignores
+`base_url`.
+
+### Page title
+
+Most pages start their main content with the page title. `drop_title=True` leaves it out,
+as text extractors for pretraining corpora usually do (they keep the title as metadata):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
+The first selected line is dropped when it comes from an `h1`–`h6` or `p` element whose
+whole text is that line, and it matches the page's `<title>` or `og:title` (case and
+punctuation ignored, similarity at least 0.8 against the whole title or a part split at
+` | `, ` - `, ` – `, ` : ` and similar separators). Any other first line is kept. It
+applies to `text`, `html` and `markdown`; `with_blocks=True` is unchanged. The title is
+kept by default.
+
 The larger model, from the `[full]` install:
 
 ```python
@@ -131,10 +188,49 @@ Composite image showing the global distribution of photosynthesis, including bot
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
+
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+htmlsift extract page.html --output markdown --base-url https://example.com/page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" > out.jsonl         # one JSON record per file
+htmlsift extract "crawl/**/*.html" --mode base --device cuda > out.jsonl
+htmlsift --version                                  # htmlsift, Python and backend versions
+```
+
+One file or `-` (stdin) prints the content itself. Several files or a glob print one JSON
+line per file, `{"path": ..., "content": ...}`, or `{"path": ..., "error": ...}` for a file
+that failed; the run continues and exits with 1 at the end. Pass `--jsonl` to get records
+for a single file too, which scripts should do: bash expands an unquoted `*.html` before
+htmlsift sees it. Quoted globs are expanded by htmlsift itself (`**` recurses), so they
+work the same in every shell; directories are not walked.
+
+| flag | |
+|---|---|
+| `--output text\|html\|markdown` | as `output=` above (default `text`) |
+| `--mode mini\|base`, `--device` | model and torch device, as for `Extractor` |
+| `--base-url URL` | make links absolute; applies to every input |
+| `--encoding ENC` | charset of the input bytes; applies to every input |
+| `--drop-title` | leave out the leading page title, as `drop_title=True` |
+| `--jsonl` | always print JSON records |
+
+Files are read as bytes, so the charset is detected as described above. Output is utf-8.
+Exit codes: 0 ok, 1 a file or the model failed, 2 a usage error. In Windows PowerShell 5.1,
+`>` re-encodes a program's output; redirect from bash, cmd or PowerShell 7 instead.
 
 ## Concept
 
@@ -201,6 +297,9 @@ and its `docs/REPRODUCE.md`.
 A standalone hero image that is its own block renders to no text line, so it can't be selected
 today. Inline images inside a kept block are preserved in `html` and `markdown`.
 
+`base_url` does not rewrite `srcset`, so responsive images in `html` output keep relative
+candidates (`src` is rewritten; `markdown` uses only `src`).
+
 </details>
 
 ## License

{htmlsift-0.1.1 → htmlsift-0.2.1}/README.pypi.md

@@ -42,7 +42,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -50,6 +50,12 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
 cache only.
 
+### Memory (`mini`)
+
+~340 MB once loaded; 99 % of CommonCrawl pages need under 50 MB more. Memory grows with
+page length (~21 KB per rendered line), so a very large page can exhaust RAM on a 1–2 GB
+machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -65,6 +71,29 @@ import htmlsift
 htmlsift.extract(html)               # module-level convenience
 ```
 
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
+Leave out the page title that starts the content, as pretraining text pipelines usually do
+(the first line, when it is an `h1`–`h6` or `p` matching the page's `<title>` or
+`og:title`; kept by default):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -78,10 +107,32 @@ ex = Extractor("base", device="cuda")  # require the GPU
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+htmlsift --version                                  # htmlsift, Python and backend versions
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--drop-title`,
+`--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
 ## License
 
 Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and

{htmlsift-0.1.1 → htmlsift-0.2.1}/pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
     "lxml",
     "numpy",
     "huggingface-hub",
+    "w3lib>=2.5,<3",
 ]
 
 [project.optional-dependencies]

{htmlsift-0.1.1 → htmlsift-0.2.1}/src/htmlsift/__init__.py

@@ -4,6 +4,6 @@
 """
 from .extractor import Extractor, extract
 
-__version__ = "0.1.1"
+__version__ = "0.2.1"
 
 __all__ = ["Extractor", "extract"]

htmlsift-0.2.1/src/htmlsift/_decode.py

@@ -0,0 +1,45 @@
+"""Decode HTML bytes to str.
+
+Resolution order: BOM, encoding= argument, <meta> declaration, strict utf-8, cp1252.
+Undecodable bytes become U+FFFD.
+"""
+from w3lib.encoding import html_body_declared_encoding, read_bom, resolve_encoding, to_unicode
+
+
+def _utf8_or_cp1252(data):
+    try:
+        data.decode("utf-8")
+    except UnicodeDecodeError:
+        return "cp1252"
+    return "utf-8"
+
+
+def decode_html(data, encoding=None):
+    """Decode HTML bytes to str. encoding= is a charset label, e.g. from an HTTP or WARC
+    Content-Type header. Raises TypeError for non-bytes input."""
+    if not isinstance(data, bytes):
+        raise TypeError(f"decode_html expects bytes, got {type(data).__name__}")
+    bom_enc, bom = read_bom(data)
+    if bom_enc:
+        return to_unicode(data[len(bom):], bom_enc)
+    enc = resolve_encoding(encoding.strip()) if encoding else None
+    if enc is None:
+        enc = html_body_declared_encoding(data)
+        # A <meta> readable as ASCII cannot truthfully declare utf-16/32.
+        if enc and enc.startswith(("utf-16", "utf-32")):
+            enc = "utf-8"
+    if enc is None:
+        enc = _utf8_or_cp1252(data)
+    return to_unicode(data, enc)
+
+
+def prepare_input(html, encoding=None):
+    """Extractor input -> what parse() takes: str unchanged, bytes decoded and re-encoded
+    as utf-8. Raises TypeError for other types, or for encoding= with str input."""
+    if isinstance(html, str):
+        if encoding is not None:
+            raise TypeError("encoding= applies to bytes input; str is already decoded")
+        return html
+    if isinstance(html, bytes):
+        return decode_html(html, encoding).encode("utf-8")
+    raise TypeError(f"html must be str or bytes, got {type(html).__name__}")

htmlsift-0.2.1/src/htmlsift/_links.py

@@ -0,0 +1,52 @@
+"""Make a page's links absolute: resolve the base URL, then rewrite against it."""
+from urllib.parse import urljoin, urlsplit
+
+
+def _is_absolute_http(url):
+    try:
+        parts = urlsplit(url)
+    except ValueError:
+        return False
+    return parts.scheme in ("http", "https") and bool(parts.netloc)
+
+
+def _first_base_href(root):
+    for el in root.iter("base"):
+        href = (el.get("href") or "").strip()
+        if href:
+            return href
+    return None
+
+
+def effective_base(root, base_url=None):
+    """The URL to resolve links against, or None to leave them unchanged.
+
+    The first non-empty <base href> is resolved against base_url; a malformed or
+    non-http(s) result falls back to base_url. Without base_url, the <base href> is
+    used only if it is an absolute http(s) URL. A malformed base_url gives None."""
+    href = _first_base_href(root)
+    if base_url:
+        try:
+            urlsplit(base_url)
+        except ValueError:
+            return None
+        if href is None:
+            return base_url
+        try:
+            joined = urljoin(base_url, href)
+        except ValueError:
+            return base_url
+        return joined if _is_absolute_http(joined) else base_url
+    return href if href is not None and _is_absolute_http(href) else None
+
+
+def make_absolute(root, base):
+    """Rewrite root's links against base in place; malformed links are left as is.
+    <base> tags in root are ignored; srcset is not rewritten."""
+    def absolute(link):
+        try:
+            return urljoin(base, link.strip())
+        except ValueError:
+            return link
+
+    root.rewrite_links(absolute, resolve_base_href=False)

htmlsift-0.2.1/src/htmlsift/_title.py

@@ -0,0 +1,63 @@
+"""Leading page-title detection for drop_title: the first kept line, when it comes from a
+heading or paragraph that matches the page's <title> or og:title."""
+import difflib
+import re
+import unicodedata
+
+TITLE_SEP = re.compile(r"\s[|\-–—:·•»/~]{1,2}\s|\s*[|·•»]\s*")
+SIM = 0.8
+_TAGS = frozenset({"h1", "h2", "h3", "h4", "h5", "h6", "p"})
+
+
+def norm(s):
+    """NFKC, casefold, runs of non-alphanumerics to one space."""
+    s = unicodedata.normalize("NFKC", s).casefold()
+    return re.sub(r"[\W_]+", " ", s).strip()
+
+
+def title_candidates(title, og):
+    """Normalised whole titles and their segments split at TITLE_SEP."""
+    out = set()
+    for t in (title, og):
+        if t:
+            out.add(norm(t))
+            out.update(norm(p) for p in TITLE_SEP.split(t))
+    out.discard("")
+    return out
+
+
+def title_match(line, title, og):
+    """Best SequenceMatcher ratio between the normalised line and any title candidate."""
+    h = norm(line)
+    return max((difflib.SequenceMatcher(None, h, c, autojunk=False).ratio()
+                for c in title_candidates(title, og)), default=0.0)
+
+
+def page_titles(root):
+    """(first <title>, og:title), whitespace collapsed; None where absent."""
+    t = root.xpath("//title")
+    og = root.xpath("//meta[@property='og:title' or @name='og:title']/@content")
+    title = " ".join(t[0].text_content().split()) if t else None
+    return title, (" ".join(og[0].split()) if og else None)
+
+
+def _enclosing(el):
+    """Nearest ancestor-or-self that is h1-h6 or p, else None."""
+    while el is not None:
+        if isinstance(el.tag, str) and el.tag.lower() in _TAGS:
+            return el
+        el = el.getparent()
+    return None
+
+
+def find_title(root, k, line, block_src):
+    """The h1-h6/p element behind block k (text: line) if it is the page title, else None.
+    All of block k's elements must sit in that one element, the element's text must be
+    this line alone, and line must match <title> or og:title."""
+    encl = {_enclosing(e) for e in block_src[k]}
+    if len(encl) != 1 or None in encl:
+        return None
+    el = encl.pop()
+    if norm(el.text_content()) != norm(line):
+        return None
+    return el if title_match(line, *page_titles(root)) >= SIM else None