htmlsift 0.2.0__tar.gz → 0.2.1__tar.gz

htmlsift-0.2.1/.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  schedule:
+    - cron: "17 4 * * 1"   # Mondays 04:17 UTC
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install (base deps only, no torch)
+        run: pip install -e . ruff pytest
+      - name: Show installed versions
+        run: pip list
+      - name: Lint
+        run: ruff check src tests
+      - name: Read pinned bundle revision
+        id: rev
+        run: |
+          rev=$(sed -n 's/^REVISION = "\([0-9a-f]\{40\}\)".*/\1/p' src/htmlsift/_artifacts.py)
+          test -n "$rev" || { echo "REVISION not found in _artifacts.py"; exit 1; }
+          echo "rev=$rev" >> "$GITHUB_OUTPUT"
+      - name: Cache mini bundle
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface/hub
+          key: htmlsift-mini-${{ steps.rev.outputs.rev }}
+      - name: Fetch mini bundle (no network on a cache hit)
+        run: |
+          htmlsift download mini
+          dir="$HOME/.cache/huggingface/hub/models--koivualeksi--htmlsift-release/snapshots/${{ steps.rev.outputs.rev }}/mini"
+          test -f "$dir/manifest.json" || { echo "bundle not at $dir"; exit 1; }
+          echo "HTMLSIFT_BUNDLE=$dir" >> "$GITHUB_ENV"
+      - name: Test (model goldens included)
+        run: pytest -q -rs

{htmlsift-0.2.0 → htmlsift-0.2.1}/CHANGELOG.md

@@ -6,6 +6,44 @@ All notable changes to htmlsift are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.1] - 2026-11-08
+
+### Added
+- `htmlsift --version` prints the htmlsift and Python versions and the installed
+  backends: onnxruntime for `mini`, torch and transformers for `base`.
+- `drop_title=True` in `extract()` and `--drop-title` in the CLI leave out the page title
+  that starts the content: the first selected line, when it comes from an `h1`–`h6` or
+  `p` element holding only that line and matches the page's `<title>` or `og:title`.
+  Applies to `text`, `html` and `markdown`. Off by default.
+
+### Changed
+- With w3lib 2.6 or newer, the five bytes windows-1252 leaves undefined (`0x81`, `0x8D`,
+  `0x8F`, `0x90`, `0x9D`) decode to C1 control characters, as in browsers, instead of
+  `U+FFFD`.
+- `output="text"` (the default) returns plain text, as documented. Heading `#`, emphasis,
+  inline-code and strikethrough markers, `>` quote prefixes, `* * *` rules and code-fence
+  lines are no longer in it; list bullets, ordered-list numbers and table rows stay.
+  Literal `#`, `*`, `_` or `|` in the page text are kept. The selected content is the
+  same as in 0.2.0, and the model input and predictions are unchanged.
+- `text` output renders the page a second time, without markers: about 3 ms per page
+  more on one CPU core (mini, measured on CommonCrawl pages).
+- The lines as the model scored them, markers included, are still available through
+  `with_blocks=True`.
+
+### Fixed
+- `output="text"` no longer contains lines made only of invisible characters (zero-width
+  spaces and joiners, byte order marks, soft hyphens, control characters), which pages
+  use as spacers.
+- `mini` memory on very long pages. The embedding step no longer grows with page length
+  (a 3M-token page peaked at 39 GB, now 1.2 GB), and pages estimated above 100 MB hand
+  their memory back to the system when they finish instead of keeping it for the life
+  of the process. Output is unchanged; model time is about 15 % lower on CommonCrawl
+  pages.
+- `mode="base"` without torch or transformers installed raised a bare
+  `No module named 'torch'`, and only after downloading the ~1 GB base bundle. It now
+  raises `ImportError` before any download, naming the missing packages and
+  `pip install "htmlsift[full]"`. The CLI prints the same message.
+
 ## [0.2.0] - 2026-11-02
 
 ### Added
@@ -66,7 +104,8 @@ Documentation and packaging metadata only — no code or model changes since 0.1
   warning on CPU fallback.
 - Model bundles fetched from the Hugging Face Hub, pinned to a fixed commit revision.
 
-[Unreleased]: https://github.com/koivualeksi/htmlsift/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/koivualeksi/htmlsift/compare/v0.2.1...HEAD
+[0.2.1]: https://github.com/koivualeksi/htmlsift/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/koivualeksi/htmlsift/compare/v0.1.1...v0.2.0
 [0.1.1]: https://github.com/koivualeksi/htmlsift/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/koivualeksi/htmlsift/releases/tag/v0.1.0

{htmlsift-0.2.0 → htmlsift-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: htmlsift
-Version: 0.2.0
+Version: 0.2.1
 Summary: Main-content extraction for web pages
 Project-URL: Repository, https://github.com/koivualeksi/htmlsift
 Project-URL: Research, https://github.com/koivualeksi/htmlsift-research
@@ -82,6 +82,12 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
 cache only.
 
+### Memory (`mini`)
+
+~340 MB once loaded; 99 % of CommonCrawl pages need under 50 MB more. Memory grows with
+page length (~21 KB per rendered line), so a very large page can exhaust RAM on a 1–2 GB
+machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -112,6 +118,14 @@ is honoured):
 ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
 ```
 
+Leave out the page title that starts the content, as pretraining text pipelines usually do
+(the first line, when it is an `h1`–`h6` or `p` matching the page's `<title>` or
+`og:title`; kept by default):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -125,19 +139,29 @@ ex = Extractor("base", device="cuda")  # require the GPU
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
 ## Command line
 
 ```bash
 htmlsift extract page.html                          # main content of one page
 curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
 htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+htmlsift --version                                  # htmlsift, Python and backend versions
 ```
 
-Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--drop-title`,
+`--jsonl` (records
 even for a single file). Failed files become `{"path", "error"}` records and the exit code
 is 1. Output is utf-8.
 

{htmlsift-0.2.0 → htmlsift-0.2.1}/README.md

@@ -2,7 +2,7 @@
 
 Main-content extraction for web pages: hand it a raw HTML string and get back the main content
  - nav, ads, cookie banners, and footers stripped
- - outputs raw text, **HTML**, or **markdown**.
+ - outputs plain text, **HTML**, or **markdown**.
 
 Two models shipped, both trained on [WebMainBench][wmb] (WMB) — 7,825 labelled pages, custom split
 6,548 train / 732 val / 545 test:
@@ -73,6 +73,17 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; prefetch with `htmlsift download mini` (or `base`), or set `HF_HUB_OFFLINE=1` to run
 from cache only.
 
+### Memory (`mini`)
+
+- **Fixed:** ~340 MB once the model is loaded.
+- **Per page:** 99 % of CommonCrawl pages need under 50 MB on top of that.
+- **Long pages:** memory grows with the page, about 21 KB per rendered line plus ~0.2 KB
+  per token. The largest page in an 8,820-page CommonCrawl sample needed ~0.8 GB.
+- **Release:** a page estimated above 100 MB hands its memory back to the system when it
+  finishes, so one large page doesn't leave a long-running worker big.
+- **Small machines:** page size has no built-in limit, so a very large page can exhaust
+  RAM on a 1–2 GB machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -122,6 +133,22 @@ The page's first `<base href>` is resolved against `base_url`, as a browser does
 are. Malformed links are left unchanged. `text` output has no links, so it ignores
 `base_url`.
 
+### Page title
+
+Most pages start their main content with the page title. `drop_title=True` leaves it out,
+as text extractors for pretraining corpora usually do (they keep the title as metadata):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
+The first selected line is dropped when it comes from an `h1`–`h6` or `p` element whose
+whole text is that line, and it matches the page's `<title>` or `og:title` (case and
+punctuation ignored, similarity at least 0.8 against the whole title or a part split at
+` | `, ` - `, ` – `, ` : ` and similar separators). Any other first line is kept. It
+applies to `text`, `html` and `markdown`; `with_blocks=True` is unchanged. The title is
+kept by default.
+
 The larger model, from the `[full]` install:
 
 ```python
@@ -161,10 +188,18 @@ Composite image showing the global distribution of photosynthesis, including bot
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
 
 ## Command line
 
@@ -174,6 +209,7 @@ htmlsift extract page.html --output markdown --base-url https://example.com/page
 curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
 htmlsift extract "crawl/*.html" > out.jsonl         # one JSON record per file
 htmlsift extract "crawl/**/*.html" --mode base --device cuda > out.jsonl
+htmlsift --version                                  # htmlsift, Python and backend versions
 ```
 
 One file or `-` (stdin) prints the content itself. Several files or a glob print one JSON
@@ -189,6 +225,7 @@ work the same in every shell; directories are not walked.
 | `--mode mini\|base`, `--device` | model and torch device, as for `Extractor` |
 | `--base-url URL` | make links absolute; applies to every input |
 | `--encoding ENC` | charset of the input bytes; applies to every input |
+| `--drop-title` | leave out the leading page title, as `drop_title=True` |
 | `--jsonl` | always print JSON records |
 
 Files are read as bytes, so the charset is detected as described above. Output is utf-8.

{htmlsift-0.2.0 → htmlsift-0.2.1}/README.pypi.md

@@ -50,6 +50,12 @@ Requires Python 3.10+. Model files download from the Hugging Face Hub on first u
 cached; run `htmlsift download <mode>` to prefetch, or set `HF_HUB_OFFLINE=1` to run from
 cache only.
 
+### Memory (`mini`)
+
+~340 MB once loaded; 99 % of CommonCrawl pages need under 50 MB more. Memory grows with
+page length (~21 KB per rendered line), so a very large page can exhaust RAM on a 1–2 GB
+machine. Cap the input size before calling `extract()`.
+
 ## Usage
 
 ```python
@@ -80,6 +86,14 @@ is honoured):
 ex.extract(html, output="markdown", base_url="https://example.com/blog/post")
 ```
 
+Leave out the page title that starts the content, as pretraining text pipelines usually do
+(the first line, when it is an `h1`–`h6` or `p` matching the page's `<title>` or
+`og:title`; kept by default):
+
+```python
+ex.extract(html, drop_title=True)
+```
+
 The larger model, from `htmlsift[full]`:
 
 ```python
@@ -93,19 +107,29 @@ ex = Extractor("base", device="cuda")  # require the GPU
 
 | `output` | what you get |
 |---|---|
-| `text` (default) | selected content as plain text |
+| `text` (default) | selected content as plain text: no heading, emphasis, code, quote, rule or fence markers; list bullets, ordered-list numbers and table rows (`\| a \| b \|`) stay |
 | `html` | selected DOM, with structure / links / images kept |
 | `markdown` | clean markdown, links and images preserved |
 
+The lines exactly as the model scored them, markdown markers included, come from
+`with_blocks=True`:
+
+```python
+blocks, probs = ex.extract(html, with_blocks=True)
+scored = "\n".join(b for b, p in zip(blocks, probs) if p > 0.5)
+```
+
 ## Command line
 
 ```bash
 htmlsift extract page.html                          # main content of one page
 curl -s https://example.com/a | htmlsift extract - --base-url https://example.com/a
 htmlsift extract "crawl/*.html" --output markdown > out.jsonl   # one JSON line per file
+htmlsift --version                                  # htmlsift, Python and backend versions
 ```
 
-Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--jsonl` (records
+Flags: `--output`, `--mode`, `--device`, `--base-url`, `--encoding`, `--drop-title`,
+`--jsonl` (records
 even for a single file). Failed files become `{"path", "error"}` records and the exit code
 is 1. Output is utf-8.
 

{htmlsift-0.2.0 → htmlsift-0.2.1}/src/htmlsift/__init__.py

@@ -4,6 +4,6 @@
 """
 from .extractor import Extractor, extract
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 __all__ = ["Extractor", "extract"]

htmlsift-0.2.1/src/htmlsift/_title.py

@@ -0,0 +1,63 @@
+"""Leading page-title detection for drop_title: the first kept line, when it comes from a
+heading or paragraph that matches the page's <title> or og:title."""
+import difflib
+import re
+import unicodedata
+
+TITLE_SEP = re.compile(r"\s[|\-–—:·•»/~]{1,2}\s|\s*[|·•»]\s*")
+SIM = 0.8
+_TAGS = frozenset({"h1", "h2", "h3", "h4", "h5", "h6", "p"})
+
+
+def norm(s):
+    """NFKC, casefold, runs of non-alphanumerics to one space."""
+    s = unicodedata.normalize("NFKC", s).casefold()
+    return re.sub(r"[\W_]+", " ", s).strip()
+
+
+def title_candidates(title, og):
+    """Normalised whole titles and their segments split at TITLE_SEP."""
+    out = set()
+    for t in (title, og):
+        if t:
+            out.add(norm(t))
+            out.update(norm(p) for p in TITLE_SEP.split(t))
+    out.discard("")
+    return out
+
+
+def title_match(line, title, og):
+    """Best SequenceMatcher ratio between the normalised line and any title candidate."""
+    h = norm(line)
+    return max((difflib.SequenceMatcher(None, h, c, autojunk=False).ratio()
+                for c in title_candidates(title, og)), default=0.0)
+
+
+def page_titles(root):
+    """(first <title>, og:title), whitespace collapsed; None where absent."""
+    t = root.xpath("//title")
+    og = root.xpath("//meta[@property='og:title' or @name='og:title']/@content")
+    title = " ".join(t[0].text_content().split()) if t else None
+    return title, (" ".join(og[0].split()) if og else None)
+
+
+def _enclosing(el):
+    """Nearest ancestor-or-self that is h1-h6 or p, else None."""
+    while el is not None:
+        if isinstance(el.tag, str) and el.tag.lower() in _TAGS:
+            return el
+        el = el.getparent()
+    return None
+
+
+def find_title(root, k, line, block_src):
+    """The h1-h6/p element behind block k (text: line) if it is the page title, else None.
+    All of block k's elements must sit in that one element, the element's text must be
+    this line alone, and line must match <title> or og:title."""
+    encl = {_enclosing(e) for e in block_src[k]}
+    if len(encl) != 1 or None in encl:
+        return None
+    el = encl.pop()
+    if norm(el.text_content()) != norm(line):
+        return None
+    return el if title_match(line, *page_titles(root)) >= SIM else None

{htmlsift-0.2.0 → htmlsift-0.2.1}/src/htmlsift/cli.py

@@ -10,13 +10,45 @@ import errno
 import glob
 import json
 import os
+import platform
 import sys
+from importlib.metadata import PackageNotFoundError, version
 from pathlib import Path
 
 from w3lib.encoding import resolve_encoding
 
+from . import __version__
 from ._artifacts import BUNDLE_FILES, resolve_bundle
 from .extractor import Extractor
+from .loader import BASE_DEPS, FULL_INSTALL
+
+
+def _dist_version(name):
+    try:
+        return version(name)
+    except PackageNotFoundError:
+        return None
+
+
+def _version_text():
+    """htmlsift, Python and backend versions, read from package metadata (no imports)."""
+    def backend(mode, deps):
+        found = {d: _dist_version(d) for d in deps}
+        line = f"{mode}: " + ", ".join(f"{d} {v or 'not installed'}" for d, v in found.items())
+        return line if all(found.values()) else f"{line} ({FULL_INSTALL})"
+    return "\n".join([f"htmlsift {__version__}",
+                      f"python {platform.python_version()}",
+                      backend("mini", ["onnxruntime"]),
+                      backend("base", BASE_DEPS)]) + "\n"
+
+
+class _Version(argparse.Action):
+    def __init__(self, option_strings, dest, **kwargs):
+        super().__init__(option_strings, dest, nargs=0, help="print versions and exit")
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        sys.stdout.write(_version_text())
+        parser.exit()
 
 
 def _download(mode):
@@ -92,7 +124,8 @@ def _extract(ap, args):
                 if error is None:
                     try:
                         content = ex.extract(_read(path), output=args.output,
-                                             encoding=args.encoding, base_url=args.base_url)
+                                             encoding=args.encoding, base_url=args.base_url,
+                                             drop_title=args.drop_title)
                     except Exception as e:
                         error = str(e) or type(e).__name__
                 failed = failed or error is not None
@@ -105,6 +138,7 @@ def _extract(ap, args):
 
 def main(argv=None):
     ap = argparse.ArgumentParser(prog="htmlsift")
+    ap.add_argument("--version", action=_Version)
     sub = ap.add_subparsers(dest="cmd", required=True)
 
     dl = sub.add_parser("download", help="prefetch a model bundle into the HF cache")
@@ -122,6 +156,9 @@ def main(argv=None):
     ex.add_argument("--base-url", help="make links absolute (html/markdown); applies to "
                                        "every input")
     ex.add_argument("--encoding", help="charset of the input bytes; applies to every input")
+    ex.add_argument("--drop-title", action="store_true",
+                    help="leave out a leading line that matches the page's <title> or "
+                         "og:title")
     ex.add_argument("--jsonl", action="store_true",
                     help="always print JSON records, even for a single input")
 

htmlsift-0.2.1/src/htmlsift/core/onnx_infer.py

@@ -0,0 +1,55 @@
+"""Torch-free onnxruntime forward for the mini model (int8 table + BiGRU head).
+
+Embed the page's distinct token ids through the int8 table graph, mean-pool by block
+membership, concat any structural features, run the head graph, sigmoid. numpy +
+onnxruntime only -- no torch. The table is a per-id lookup, so embedding each distinct id
+once gives the same rows as embedding every token, and memory scales with the page's
+vocabulary instead of its length.
+"""
+import numpy as np
+import onnxruntime as ort
+
+# onnxruntime's arena keeps its peak after a run. Pages estimated above SHRINK_MB give
+# the memory back when they finish; smaller pages keep the arena for speed.
+TABLE_KB_PER_ID = 12
+HEAD_KB_PER_BLOCK = 21
+SHRINK_MB = 100
+
+
+def arena_mb(n_unique, n_blocks):
+    """Estimated arena memory (MB) one page needs: table per distinct id + head per block."""
+    return (TABLE_KB_PER_ID * n_unique + HEAD_KB_PER_BLOCK * n_blocks) / 1024
+
+
+def pool(emb_u, inv, members):
+    """Per-block mean of the member tokens' embeddings -> [n, H]. emb_u holds one row
+    per distinct id and emb_u[inv[t]] is token t's. A token-less block pools to zeros."""
+    pooled = np.zeros((len(members), emb_u.shape[1]), dtype=np.float32)
+    for i, mem in enumerate(members):
+        if mem:
+            pooled[i] = emb_u[inv[mem]].mean(0)
+    return pooled
+
+
+def load_mini(table_path, head_path, threads=None):
+    """The two mini graphs -> infer_fn(page) -> per-block probs (np.float32): the
+    (page -> probs) contract the extractor consumes."""
+    opts = ort.SessionOptions()
+    if threads:
+        opts.intra_op_num_threads, opts.inter_op_num_threads = threads, 1
+    table = ort.InferenceSession(str(table_path), opts, providers=["CPUExecutionProvider"])
+    head = ort.InferenceSession(str(head_path), opts, providers=["CPUExecutionProvider"])
+    shrink = ort.RunOptions()
+    shrink.add_run_config_entry("memory.enable_memory_arena_shrinkage", "cpu:0")
+
+    def infer_fn(page):
+        uniq, inv = np.unique(page["ids"], return_inverse=True)
+        members = page["members"]
+        ro = shrink if arena_mb(len(uniq), len(members)) > SHRINK_MB else None
+        emb_u = table.run(["out"], {"ids": uniq.astype(np.int64)}, ro)[0]      # [U, H]
+        x = pool(emb_u, inv, members)                                          # [n, H]
+        if page.get("feats") is not None:
+            x = np.concatenate([x, page["feats"].astype(np.float32)], 1)       # [n, H+K]
+        logits = head.run(["logits"], {"x": x[None]}, ro)[0][0]                # [n]
+        return (1.0 / (1.0 + np.exp(-logits))).astype(np.float32)
+    return infer_fn

{htmlsift-0.2.0 → htmlsift-0.2.1}/src/htmlsift/core/render.py

@@ -100,6 +100,9 @@ def _cell_text(el):
 class _Renderer:
     """Accumulates output lines and, per line, the set of source elements."""
 
+    HR = "* * *"
+    FENCE = "```"
+
     def __init__(self, render_hidden=False, links=False):
         self.render_hidden = render_hidden
         self.links = links
@@ -117,6 +120,15 @@ class _Renderer:
     def _qpfx(self):
         return "> " * self.quote
 
+    def _heading_pfx(self, level):
+        return "#" * level + " "
+
+    def _cell(self, txt):
+        return txt
+
+    def _wrap(self, el, open_s, inner, close_s):
+        return open_s + inner + close_s
+
     def _append(self, line, src):
         self.swallow = False
         if self.pending is not None:
@@ -180,10 +192,10 @@ class _Renderer:
         elif tag == "br":
             self.flush()
         elif tag == "hr":
-            self.blank(); self.emit("* * *", {el}); self.blank()
+            self.blank(); self.emit(self.HR, {el}); self.blank()
         elif tag in HEADING_TAGS:
             self.blank()
-            self.linepfx = "#" * HEADING_TAGS[tag] + " "
+            self.linepfx = self._heading_pfx(HEADING_TAGS[tag])
             self.inner_keeping_lead(el)
             self.blank()
         elif tag in LIST_TAGS:
@@ -210,7 +222,7 @@ class _Renderer:
             open_s, close_s = WRAP_TAGS[tag]
             inner = _sub_inline(el, self.render_hidden, self.links)
             if inner:
-                self.raw(open_s + inner + close_s, el)
+                self.raw(self._wrap(el, open_s, inner, close_s), el)
             else:
                 self.swallow = True
         elif tag == "a" and self.links and el.get("href"):
@@ -256,10 +268,10 @@ class _Renderer:
 
     def pre(self, el):
         self.blank()
-        self._append("```", {el})
+        self._append(self.FENCE, {el})
         for line in el.text_content().split("\n"):
             self._append(line, {el})
-        self._append("```", {el})
+        self._append(self.FENCE, {el})
         self.blank()
 
     def table(self, el):
@@ -288,7 +300,7 @@ class _Renderer:
             texts, src = [], {tr}
             for c in cells:
                 t, s = _cell_text(c)
-                texts.append(t); src |= s
+                texts.append(self._cell(t)); src |= s
             if not any(texts):                 # skip fully-empty rows
                 continue
             self.emit("| " + " | ".join(texts) + " |", src)
@@ -298,12 +310,40 @@ class _Renderer:
         self.blank()
 
 
-def _sub_inline(el, render_hidden=False, links=False):
-    sub = _Renderer(render_hidden, links); sub.children(el); sub.flush()
+class _PlainRenderer(_Renderer):
+    """The same walk and the same lines as _Renderer, without markdown markers.
+    Output only; list bullets, ordered numbers and table pipes stay."""
+
+    HR = ""
+    FENCE = ""
+
+    def _qpfx(self):
+        return ""
+
+    def _heading_pfx(self, level):
+        return ""
+
+    def _cell(self, txt):
+        return txt.replace("\\|", "|")
+
+    def _wrap(self, el, open_s, inner, close_s):
+        # The markdown inner decided that a chunk exists; this one may be "" (<b><hr></b>).
+        return _sub_inline(el, self.render_hidden, self.links, _PlainRenderer)
+
+    def text(self, s, src):
+        lead = self.keep_lead and not self.linepfx and not any(self.buf)
+        super().text(s, src)
+        if lead and self.buf:
+            # Heading lead space; the chunk stays in buf so the line is still emitted.
+            self.buf[-1] = self.buf[-1].lstrip()
+
+
+def _sub_inline(el, render_hidden=False, links=False, cls=_Renderer):
+    sub = cls(render_hidden, links); sub.children(el); sub.flush()
     return " ".join(ln.strip() for ln in sub.lines if ln.strip())
 
 
-def render_tree(root, render_hidden=False, links=False):
+def render_tree(root, render_hidden=False, links=False, plain=False):
     """Walk a parsed tree -> (lines, line_src). line_src[i] is the set of source
     DOM elements that produced line i (empty for structural separators).
 
@@ -314,15 +354,19 @@ def render_tree(root, render_hidden=False, links=False):
 
     links=True renders <a>/<img> as markdown ([text](href), ![alt](src)) -- an
     output-only mode. The model-input render keeps links=False (link-flattened) so
-    training still matches; only serialization sets links=True."""
-    r = _Renderer(render_hidden, links); r.walk(root); r.flush()
+    training still matches; only serialization sets links=True.
+
+    plain=True renders the same lines without markdown markers (headings, emphasis,
+    code, quotes, rules, fences) -- output-only, like links."""
+    cls = _PlainRenderer if plain else _Renderer
+    r = cls(render_hidden, links); r.walk(root); r.flush()
     return r.lines, r.line_src
 
 
-def render(html_str, render_hidden=False, links=False):
+def render(html_str, render_hidden=False, links=False, plain=False):
     """HTML string -> readable markdown text (inference; provenance discarded).
     Does not sanitize — a caller that needs it sanitizes the tree first."""
     if not html_str or not html_str.strip():
         return ""
-    lines, _ = render_tree(parse(html_str), render_hidden, links)
+    lines, _ = render_tree(parse(html_str), render_hidden, links, plain)
     return "\n".join(lines) + "\n"