htmlsift 0.1.1__tar.gz → 0.2.0__tar.gz

htmlsift-0.2.0/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to htmlsift are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-11-02
+
+### Added
+- `extract()` accepts `bytes` and decodes them: byte order mark, then `encoding=`, then the
+  page's `<meta>` declaration, then utf-8, falling back to cp1252. Legacy charset labels map
+  the way browsers read them (`latin1` → cp1252, `shift_jis` → cp932, `gb2312` → gb18030).
+- Keyword-only `encoding=` on `Extractor.extract` and `extract`, for a charset known from
+  an HTTP or WARC header.
+- Dependency: `w3lib` (BSD-3-Clause) for charset sniffing.
+- Keyword-only `base_url=` on `Extractor.extract` and `extract`: links and image URLs in
+  `html` and `markdown` output are made absolute against it. The page's first
+  `<base href>` is resolved against `base_url`; malformed links are left as they are.
+  `text` output and `with_blocks` are unaffected.
+- `htmlsift extract` command: files, quoted globs or stdin in; the content of a single
+  input, or one JSON line per file (`--jsonl` to force), out. Flags `--output`,
+  `--mode`, `--device`, `--base-url`, `--encoding`. Failed files are reported as
+  records and the run exits with 1.
+
+### Changed
+- Input other than `str` or `bytes` raises `TypeError`; `extract(None)` previously
+  returned `""`.
+- `encoding=` with `str` input raises `TypeError`, since a `str` is already decoded.
+- A page with an absolute `<base href>` gets absolute links in `html` and `markdown` output
+  even without `base_url`. Previously `<base>` was dropped with `<head>`, leaving its links
+  relative.
+
+### Fixed
+- Non-utf-8 `bytes` input crashed with `UnicodeDecodeError`.
+- `html` and `markdown` output could drop images and other inline content inside a kept
+  block, varying from run to run on the same page. `text` output was unaffected.
+
+## [0.1.1] - 2026-09-23
+
+Documentation and packaging metadata only — no code or model changes since 0.1.0.
+
+### Added
+- Project URLs in package metadata (research repo, Hugging Face weights, issue tracker).
+- PyPI Trusted Publishing workflow (`.github/workflows/publish.yml`).
+- Cross-benchmark accuracy table (`base`/`mini` vs trafilatura, readability, resiliparse on
+  WMB, WCXB, DAnIEL) and an "Annotation is a policy" section in the README.
+- GPU-on-Windows install note (CUDA torch from the PyTorch index).
+
+### Changed
+- Stated the model-weight license (Apache-2.0) and added WebMainBench attribution.
+- Cited the WebMainBench source paper (Liu et al., *Dripper*, arXiv:2511.23119).
+- Clarified model shorthand and tightened copy in the README and the PyPI description.
+
+## [0.1.0] - 2026-09-22
+
+### Added
+- Initial release.
+- `mini` model (int8 ONNX, runs on CPU) and `base` model (311M encoder, GPU-capable),
+  installed via `pip install htmlsift` and `pip install htmlsift[full]` respectively.
+- `Extractor(mode, *, bundle_dir, threads, device)` with
+  `.extract(html, output=..., with_blocks=...)`, plus the module-level `extract(...)`.
+- Output formats: plain text, HTML, and markdown.
+- Device selection for `base`: `device="cuda"` / `"cpu"`, auto-detecting a GPU and
+  warning on CPU fallback.
+- Model bundles fetched from the Hugging Face Hub, pinned to a fixed commit revision.
+
+[Unreleased]: https://github.com/koivualeksi/htmlsift/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/koivualeksi/htmlsift/compare/v0.1.1...v0.2.0
+[0.1.1]: https://github.com/koivualeksi/htmlsift/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/koivualeksi/htmlsift/releases/tag/v0.1.0

{htmlsift-0.1.1 → htmlsift-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: htmlsift
-Version: 0.1.1
+Version: 0.2.0
 Summary: Main-content extraction for web pages
 Project-URL: Repository, https://github.com/koivualeksi/htmlsift
 Project-URL: Research, https://github.com/koivualeksi/htmlsift-research
@@ -24,6 +24,7 @@ Requires-Dist: lxml
 Requires-Dist: numpy
 Requires-Dist: onnxruntime
 Requires-Dist: tokenizers
+Requires-Dist: w3lib<3,>=2.5
 Provides-Extra: full
 Requires-Dist: torch; extra == 'full'
 Requires-Dist: transformers; extra == 'full'
@@ -73,7 +74,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -96,6 +97,21 @@ import htmlsift
 htmlsift.extract(html)               # module-level convenience
 ```
 
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -113,6 +129,18 @@ ex = Extractor("base", device="cuda")  # require the GPU
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
 ## License
 
 Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and

{htmlsift-0.1.1 → htmlsift-0.2.0}/README.md

@@ -65,7 +65,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - **`htmlsift`** — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - **`htmlsift[full]`** — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -92,6 +92,36 @@ import htmlsift
 htmlsift.extract(html)               # default mini
 ```
 
+### Bytes and encodings
+
+Pass raw bytes from a file, an HTTP response or a WARC record, and htmlsift decodes them:
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+The charset is taken from, in order: a byte order mark, `encoding=`, the page's
+`<meta charset>`; an undeclared page is read as utf-8, falling back to cp1252. Unknown
+labels are skipped and undecodable bytes become `�`. A `str` counts as already decoded, so
+`encoding=` with a `str` raises `TypeError`. Prefer passing bytes over a `str` decoded
+upstream: `requests`' `response.text`, for one, guesses ISO-8859-1 for pages without a
+charset header.
+
+### Absolute links
+
+Pass the page's URL to make links and image URLs absolute in `html` and `markdown` output:
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+# [the report](report)  ->  [the report](https://example.com/blog/report)
+```
+
+The page's first `<base href>` is resolved against `base_url`, as a browser does. Without
+`base_url`, an absolute `<base href>` is used on its own; with neither, links stay as they
+are. Malformed links are left unchanged. `text` output has no links, so it ignores
+`base_url`.
+
 The larger model, from the `[full]` install:
 
 ```python
@@ -136,6 +166,35 @@ Composite image showing the global distribution of photosynthesis, including bot
 | `markdown` | clean markdown, links and images preserved |
 
 
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+htmlsift extract page.html --output markdown --base-url https://example.com/page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" > out.jsonl         # one JSON record per file
+htmlsift extract "crawl/**/*.html" --mode base --device cuda > out.jsonl
+```
+
+One file or `-` (stdin) prints the content itself. Several files or a glob print one JSON
+line per file, `{"path": ..., "content": ...}`, or `{"path": ..., "error": ...}` for a file
+that failed; the run continues and exits with 1 at the end. Pass `--jsonl` to get records
+for a single file too, which scripts should do: bash expands an unquoted `*.html` before
+htmlsift sees it. Quoted globs are expanded by htmlsift itself (`**` recurses), so they
+work the same in every shell; directories are not walked.
+
+| flag | |
+|---|---|
+| `--output text\|html\|markdown` | as `output=` above (default `text`) |
+| `--mode mini\|base`, `--device` | model and torch device, as for `Extractor` |
+| `--base-url URL` | make links absolute; applies to every input |
+| `--encoding ENC` | charset of the input bytes; applies to every input |
+| `--jsonl` | always print JSON records |
+
+Files are read as bytes, so the charset is detected as described above. Output is utf-8.
+Exit codes: 0 ok, 1 a file or the model failed, 2 a usage error. In Windows PowerShell 5.1,
+`>` re-encodes a program's output; redirect from bash, cmd or PowerShell 7 instead.
+
 ## Concept
 
 <details>
@@ -201,6 +260,9 @@ and its `docs/REPRODUCE.md`.
 A standalone hero image that is its own block renders to no text line, so it can't be selected
 today. Inline images inside a kept block are preserved in `html` and `markdown`.
 
+`base_url` does not rewrite `srcset`, so responsive images in `html` output keep relative
+candidates (`src` is rewritten; `markdown` uses only `src`).
+
 </details>
 
 ## License

{htmlsift-0.1.1 → htmlsift-0.2.0}/README.pypi.md

@@ -42,7 +42,7 @@ pip install htmlsift[full]    # adds the base 311M model (torch; GPU-capable)
 ```
 
 - `htmlsift` — the `mini` model only; few dependencies (onnxruntime + tokenizers + lxml +
-  numpy + huggingface-hub). First use downloads ~230 MB of model files.
+  numpy + huggingface-hub + w3lib). First use downloads ~230 MB of model files.
 - `htmlsift[full]` — adds `torch` + `transformers` to run `base`. First use of `base`
   downloads ~1 GB.
 
@@ -65,6 +65,21 @@ import htmlsift
 htmlsift.extract(html)               # module-level convenience
 ```
 
+Raw bytes are decoded for you (byte order mark, then `encoding=`, then `<meta charset>`,
+then utf-8 / cp1252):
+
+```python
+ex.extract(response.content)                       # charset from the page itself
+ex.extract(payload, encoding="windows-1251")       # charset from the HTTP / WARC header
+```
+
+Make links and image URLs absolute in `html` / `markdown` output (the page's `<base href>`
+is honoured):
+
+```python
+ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -82,6 +97,18 @@ ex = Extractor("base", device="cuda")  # require the GPU
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+## Command line
+
+```bash
+htmlsift extract page.html                          # main content of one page
+curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
+htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+```
+
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+even for a single file). Failed files become `{"path", "error"}` records and the exit code
+is 1. Output is utf-8.
+
 ## License
 
 Apache-2.0. Model weights are distributed from the Hugging Face Hub, also under Apache-2.0, and

{htmlsift-0.1.1 → htmlsift-0.2.0}/pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
     "lxml",
     "numpy",
     "huggingface-hub",
+    "w3lib>=2.5,<3",
 ]
 
 [project.optional-dependencies]

{htmlsift-0.1.1 → htmlsift-0.2.0}/src/htmlsift/__init__.py

@@ -4,6 +4,6 @@
 """
 from .extractor import Extractor, extract
 
-__version__ = "0.1.1"
+__version__ = "0.2.0"
 
 __all__ = ["Extractor", "extract"]

htmlsift-0.2.0/src/htmlsift/_decode.py

@@ -0,0 +1,45 @@
+"""Decode HTML bytes to str.
+
+Resolution order: BOM, encoding= argument, <meta> declaration, strict utf-8, cp1252.
+Undecodable bytes become U+FFFD.
+"""
+from w3lib.encoding import html_body_declared_encoding, read_bom, resolve_encoding, to_unicode
+
+
+def _utf8_or_cp1252(data):
+    try:
+        data.decode("utf-8")
+    except UnicodeDecodeError:
+        return "cp1252"
+    return "utf-8"
+
+
+def decode_html(data, encoding=None):
+    """Decode HTML bytes to str. encoding= is a charset label, e.g. from an HTTP or WARC
+    Content-Type header. Raises TypeError for non-bytes input."""
+    if not isinstance(data, bytes):
+        raise TypeError(f"decode_html expects bytes, got {type(data).__name__}")
+    bom_enc, bom = read_bom(data)
+    if bom_enc:
+        return to_unicode(data[len(bom):], bom_enc)
+    enc = resolve_encoding(encoding.strip()) if encoding else None
+    if enc is None:
+        enc = html_body_declared_encoding(data)
+        # A <meta> readable as ASCII cannot truthfully declare utf-16/32.
+        if enc and enc.startswith(("utf-16", "utf-32")):
+            enc = "utf-8"
+    if enc is None:
+        enc = _utf8_or_cp1252(data)
+    return to_unicode(data, enc)
+
+
+def prepare_input(html, encoding=None):
+    """Extractor input -> what parse() takes: str unchanged, bytes decoded and re-encoded
+    as utf-8. Raises TypeError for other types, or for encoding= with str input."""
+    if isinstance(html, str):
+        if encoding is not None:
+            raise TypeError("encoding= applies to bytes input; str is already decoded")
+        return html
+    if isinstance(html, bytes):
+        return decode_html(html, encoding).encode("utf-8")
+    raise TypeError(f"html must be str or bytes, got {type(html).__name__}")

htmlsift-0.2.0/src/htmlsift/_links.py

@@ -0,0 +1,52 @@
+"""Make a page's links absolute: resolve the base URL, then rewrite against it."""
+from urllib.parse import urljoin, urlsplit
+
+
+def _is_absolute_http(url):
+    try:
+        parts = urlsplit(url)
+    except ValueError:
+        return False
+    return parts.scheme in ("http", "https") and bool(parts.netloc)
+
+
+def _first_base_href(root):
+    for el in root.iter("base"):
+        href = (el.get("href") or "").strip()
+        if href:
+            return href
+    return None
+
+
+def effective_base(root, base_url=None):
+    """The URL to resolve links against, or None to leave them unchanged.
+
+    The first non-empty <base href> is resolved against base_url; a malformed or
+    non-http(s) result falls back to base_url. Without base_url, the <base href> is
+    used only if it is an absolute http(s) URL. A malformed base_url gives None."""
+    href = _first_base_href(root)
+    if base_url:
+        try:
+            urlsplit(base_url)
+        except ValueError:
+            return None
+        if href is None:
+            return base_url
+        try:
+            joined = urljoin(base_url, href)
+        except ValueError:
+            return base_url
+        return joined if _is_absolute_http(joined) else base_url
+    return href if href is not None and _is_absolute_http(href) else None
+
+
+def make_absolute(root, base):
+    """Rewrite root's links against base in place; malformed links are left as is.
+    <base> tags in root are ignored; srcset is not rewritten."""
+    def absolute(link):
+        try:
+            return urljoin(base, link.strip())
+        except ValueError:
+            return link
+
+    root.rewrite_links(absolute, resolve_base_href=False)

htmlsift-0.2.0/src/htmlsift/cli.py

@@ -0,0 +1,135 @@
+"""`htmlsift` command line.
+
+    htmlsift extract page.html                      # main content of one page
+    htmlsift extract "crawl/*.html" > out.jsonl     # one JSON record per file
+    curl -s URL | htmlsift extract - --base-url URL
+    htmlsift download mini                          # prefetch a model bundle
+"""
+import argparse
+import errno
+import glob
+import json
+import os
+import sys
+from pathlib import Path
+
+from w3lib.encoding import resolve_encoding
+
+from ._artifacts import BUNDLE_FILES, resolve_bundle
+from .extractor import Extractor
+
+
+def _download(mode):
+    paths = resolve_bundle(mode)          # hf_hub_download each file -> cached
+    print(f"{mode}: cached {len(paths)} files")
+    for name, p in paths.items():
+        print(f"  {name:10} {p}")
+    return 0
+
+
+def _is_glob(arg):
+    """A pattern to expand: not stdin, not an existing path, and has glob characters."""
+    return arg != "-" and not os.path.exists(arg) and any(c in arg for c in "*?[")
+
+
+def _resolve(arg):
+    """One argument -> [(path, error)]; error is None for an input to read."""
+    if os.path.isdir(arg):
+        return [(arg, f'is a directory; use "{arg}/*.html"')]
+    if _is_glob(arg):
+        matches = sorted(p for p in glob.glob(arg, recursive=True) if not os.path.isdir(p))
+        return [(Path(p).as_posix(), None) for p in matches] or [(arg, "no files match")]
+    return [(arg, None)]
+
+
+def _read(path):
+    return sys.stdin.buffer.read() if path == "-" else Path(path).read_bytes()
+
+
+class _PipeClosed(Exception):
+    pass
+
+
+def _write(text):
+    try:
+        sys.stdout.buffer.write(text.encode("utf-8"))
+        sys.stdout.buffer.flush()
+    except OSError as e:
+        # Reader closed the pipe (e.g. `| head`); Windows reports it as EINVAL.
+        if isinstance(e, BrokenPipeError) or e.errno == errno.EINVAL:
+            raise _PipeClosed from e
+        raise
+
+
+def _emit(path, content, error, raw):
+    if raw and error is not None:
+        print(f"htmlsift: {path}: {error}", file=sys.stderr)
+    elif raw:
+        _write(content.rstrip("\n") + "\n")
+    else:
+        rec = {"path": path, "error": error} if error is not None else {"path": path,
+                                                                        "content": content}
+        _write(json.dumps(rec, ensure_ascii=False) + "\n")
+
+
+def _extract(ap, args):
+    if args.inputs.count("-") > 1:
+        ap.error("'-' (stdin) can be given only once")
+    if args.encoding is not None and resolve_encoding(args.encoding.strip()) is None:
+        ap.error(f"unknown encoding: {args.encoding}")
+    ex = Extractor(args.mode, device=args.device)
+    try:
+        _ = ex.model
+    except Exception as e:
+        print(f"htmlsift: cannot load model: {e}", file=sys.stderr)
+        return 1
+    raw = len(args.inputs) == 1 and not _is_glob(args.inputs[0]) and not args.jsonl
+    failed = False
+    try:
+        for arg in args.inputs:
+            for path, error in _resolve(arg):
+                content = None
+                if error is None:
+                    try:
+                        content = ex.extract(_read(path), output=args.output,
+                                             encoding=args.encoding, base_url=args.base_url)
+                    except Exception as e:
+                        error = str(e) or type(e).__name__
+                failed = failed or error is not None
+                _emit(path, content, error, raw)
+    except _PipeClosed:
+        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
+        return 1
+    return 1 if failed else 0
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="htmlsift")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    dl = sub.add_parser("download", help="prefetch a model bundle into the HF cache")
+    dl.add_argument("mode", choices=list(BUNDLE_FILES))
+
+    ex = sub.add_parser(
+        "extract", help="extract main content from HTML files",
+        description="Extract main content. One file or '-' (stdin) prints the content; "
+                    "several files or a glob print one JSON record per file.")
+    ex.add_argument("inputs", nargs="+", metavar="INPUT",
+                    help="HTML file, glob pattern (quote it; ** recurses), or - for stdin")
+    ex.add_argument("--output", choices=["text", "html", "markdown"], default="text")
+    ex.add_argument("--mode", choices=list(BUNDLE_FILES), default="mini")
+    ex.add_argument("--device", help="torch device for --mode base, e.g. cuda or cpu")
+    ex.add_argument("--base-url", help="make links absolute (html/markdown); applies to "
+                                       "every input")
+    ex.add_argument("--encoding", help="charset of the input bytes; applies to every input")
+    ex.add_argument("--jsonl", action="store_true",
+                    help="always print JSON records, even for a single input")
+
+    args = ap.parse_args(argv)
+    if args.cmd == "download":
+        return _download(args.mode)
+    return _extract(ap, args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{htmlsift-0.1.1 → htmlsift-0.2.0}/src/htmlsift/core/prune.py

@@ -11,13 +11,14 @@ from lxml import html as lxml_html
 def keep_set(elements):
     """The nodes to keep: each element, its ancestors, and its descendants. Holding the
     proxies in the set keeps their lxml identity stable for the prune walk."""
-    keep = set()
+    keep, expanded = set(), set()
     for el in elements:
-        if el in keep:
+        if el in expanded:
             continue
-        keep.add(el)
+        sub = {el, *el.iterdescendants()}
+        expanded |= sub
+        keep |= sub
         keep.update(el.iterancestors())
-        keep.update(el.iterdescendants())
     return keep
 
 

{htmlsift-0.1.1 → htmlsift-0.2.0}/src/htmlsift/extractor.py

@@ -4,9 +4,12 @@ over threshold, serialize. Mirrors the research inference front-end (render_bloc
 prep_page + threshold); no sanitize step.
 """
 import numpy as np
+from lxml import html as lxml_html
 
+from ._decode import prepare_input
+from ._links import effective_base, make_absolute
 from .core.prep_page import prep_page
-from .core.prune import prune, to_html
+from .core.prune import prune
 from .core.render import parse, render_tree
 from .loader import load_model
 
@@ -36,33 +39,42 @@ class Extractor:
                                      threads=self._threads, device=self._device)
         return self._model
 
-    def extract(self, html, output="text", with_blocks=False):
-        """Extract main content from an HTML string.
+    def extract(self, html, output="text", with_blocks=False, *, encoding=None, base_url=None):
+        """Extract main content from HTML, given as str or bytes.
 
+        bytes are decoded by BOM, then encoding= (e.g. an HTTP or WARC charset), then the
+        page's <meta> declaration, then utf-8 / cp1252. encoding= with str raises TypeError.
         output="text" joins the selected lines; output="html" prunes the DOM to the
         selected blocks (structure, links, images kept) and serializes. with_blocks=True
-        returns (blocks, probs) -- every block and its probability, unthresholded."""
-        root, blocks, block_src, probs = self._predict(html)
+        returns (blocks, probs) -- every block and its probability, unthresholded.
+        base_url= makes links absolute in html/markdown output; the page's first
+        <base href> is resolved against it, or used alone if absolute."""
+        if base_url is not None and not isinstance(base_url, str):
+            raise TypeError(f"base_url must be str or None, not {type(base_url).__name__}")
+        root, blocks, block_src, probs = self._predict(html, encoding)
         if with_blocks:
             return blocks, probs
         if output == "text":
             return "\n".join(b for b, p in zip(blocks, probs, strict=True) if p > THRESHOLD)
-        if output == "html":
-            selected = _selected_srcs(block_src, probs)
-            return to_html(root, selected) if selected else ""
-        if output == "markdown":
+        if output in ("html", "markdown"):
             selected = _selected_srcs(block_src, probs)
             if not selected:
                 return ""
+            base = effective_base(root, base_url)
             prune(root, selected)
+            if base:
+                make_absolute(root, base)
+            if output == "html":
+                return lxml_html.tostring(root, encoding="unicode")
             lines, _ = render_tree(root, links=True)
             return "\n".join(lines).strip("\n")
         raise ValueError(f"unknown output {output!r}; expected 'text', 'html', or 'markdown'")
 
-    def _predict(self, html):
+    def _predict(self, html, encoding=None):
         """html -> (root, blocks, block_src, probs). blocks are the rstripped non-empty
         render lines; block_src[i] the DOM elements behind block i. Empty page ->
         (None, [], [], empty array)."""
+        html = prepare_input(html, encoding)
         if not html or not html.strip():
             return None, [], [], np.empty(0, dtype=np.float32)
         root = parse(html)
@@ -83,9 +95,11 @@ class Extractor:
 _CACHE = {}
 
 
-def extract(html, mode="mini", output="text", with_blocks=False):
+def extract(html, mode="mini", output="text", with_blocks=False, *, encoding=None,
+            base_url=None):
     """Module-level convenience: extract with a per-mode cached Extractor."""
     ex = _CACHE.get(mode)
     if ex is None:
         ex = _CACHE[mode] = Extractor(mode)
-    return ex.extract(html, output=output, with_blocks=with_blocks)
+    return ex.extract(html, output=output, with_blocks=with_blocks, encoding=encoding,
+                      base_url=base_url)

htmlsift-0.2.0/tests/conftest.py

@@ -0,0 +1,16 @@
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+
+from htmlsift import Extractor
+
+
+@pytest.fixture
+def keep_all(monkeypatch):
+    """An Extractor whose dummy model scores every line 1.0."""
+    monkeypatch.setattr("htmlsift.extractor.prep_page", lambda tok, blocks, *a, **k: blocks)
+    ex = Extractor()
+    ex._model = SimpleNamespace(features=lambda lines, src: None, tok=None, window=0, cap=0,
+                                infer_fn=lambda blocks: np.ones(len(blocks), dtype=np.float32))
+    return ex