any2md 1.1.1__tar.gz → 1.2.0__tar.gz

{any2md-1.1.1 → any2md-1.2.0}/CHANGELOG.md

@@ -4,7 +4,83 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
-## [1.1.1] — 2026-05-DD
+## [1.2.0] — 2026-11-09
+
+Drops Python 3.10 to take the Pillow 13 security fixes, and fixes
+missing H1 headings in HTML/URL output.
+
+### Removed
+
+- **Python 3.10 support.** `requires-python` is now `>=3.11`. Pillow
+  13.0.0 is the only release that fixes the advisories below, and it
+  requires Python 3.11. Python 3.10 reached end-of-life in October 2026.
+  pip on 3.10 keeps resolving any2md 1.1.1, which pins the vulnerable
+  pillow 12.x; upgrade the interpreter to get these fixes. The `tomli`
+  fallback dependency is gone with it (stdlib `tomllib` covers 3.11+),
+  and the CI matrix now tests 3.11, 3.12, and 3.13.
+
+### Security
+
+- **Pillow 13.0.0.** The `pillow>=12.2.0,<13` range held installs on a
+  release with 39 published advisories (e.g. CVE-2026-112395), all fixed
+  in 13.0.0. The range is now `pillow>=13.0.0,<14`. Docling's packages
+  accept `pillow<14`, so the `high-fidelity` extra still resolves.
+- **Locked dev-container dependencies.** `.devcontainer/requirements.lock`
+  and `requirements.txt` move certifi, lxml-html-clean, soupsieve, and
+  urllib3 to their fixed releases (lxml follows to 6.1.3). Both
+  lockfiles now audit clean with `pip-audit`.
+
+### Fixed
+
+- **HTML/URL bodies had no H1.** trafilatura keeps a heading only
+  when it sits inside the content node it selects. A page `<h1>` in a
+  hero banner or `<header>` block outside that node was dropped, so
+  nearly every blog page warned `validator: H1 count is 0 (expected 1)`
+  (106 of 106 labs.zenity.io posts, 182 of 214 zenity.io/blog posts).
+  The HTML converter now restores the H1 before the pipeline runs:
+  - The body already has an H1: unchanged.
+  - The first body line repeats the title as plain or bold text: that
+    line is promoted to the H1, so the title appears once.
+  - Otherwise `# <title>` is inserted as the first line.
+  - No title is available: unchanged (the validator still warns).
+
+  The H1 text is the page's own `<h1>` when the page has exactly one
+  non-empty `<h1>`, else trafilatura's metadata title. `content_hash`,
+  `token_estimate`, `word_count`, `recommended_chunk_level`, and
+  `abstract_for_rag` are computed from the body with the H1 in place.
+  PDF, DOCX, and TXT output is unchanged.
+- **HTML/URL heading levels skipped below the H1.** Many CMS templates
+  mark body sections as H3 or H4 directly under the page H1, so the
+  restored H1 surfaced `validator: heading level skip h1 -> h3` (34 of
+  the 320 zenity posts above). Headings are now re-ranked by section
+  depth below the H1: siblings stay siblings, nesting is kept, and only
+  the `#` prefix changes (`H3, H3, H4, H3` becomes `H2, H2, H3, H2`).
+  Bodies without skips are byte-identical. All 320 posts now convert
+  with 0 validator warnings.
+
+### Added
+
+- **`seo_title`** frontmatter extension field (HTML/URL only). Holds
+  trafilatura's metadata title (`og:title` / `<title>`) when it differs
+  from `title`. Omitted when the two match.
+
+### Changed
+
+- **CI pins ruff to 0.15.22.** Ruff 0.16 and later enable a wider default
+  rule set this repo has not adopted, which failed the required lint check
+  on an unchanged `main`. The `dev` extra pins the same version.
+- **Release workflow pins `pypa/gh-action-pypi-publish` v1.14.3.** The
+  previous pin bundled twine 6.1, which rejects the Metadata-Version 2.5
+  that current hatchling writes, so uploads failed before reaching the
+  index. The validate step now pins `build==1.6.1` and `twine==7.0.0`.
+- **HTML/URL `title` follows the visible page heading.** `title` is
+  still derived from the body H1, and the body H1 is now the page's
+  `<h1>`. On pages whose `og:title` differs from their `<h1>`, `title`
+  changes from the SEO title to the visible heading. The previous value
+  moves to `seo_title`. Pin a title with `--meta title=...` if you
+  depend on the old value.
+
+## [1.1.1] — 2026-05-07
 
 Security patch release. Closes 10 findings from a comprehensive scan
 (bandit, snyk-code, gitleaks, pip-audit, trivy, manual review) plus

{any2md-1.1.1 → any2md-1.2.0}/CONTRIBUTING.md

@@ -29,7 +29,7 @@ the test suite gates Docling-dependent tests with
 `@pytest.mark.skipif(not has_docling(), …)` so they're transparently skipped
 without it. Coverage of fallback paths is unaffected.
 
-Python 3.10 or later is required.
+Python 3.11 or later is required.
 
 ## Running tests
 

{any2md-1.1.1 → any2md-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: any2md
-Version: 1.1.1
+Version: 1.2.0
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -11,16 +11,15 @@ Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Text Processing :: Markup :: Markdown
-Requires-Python: >=3.10
+Requires-Python: >=3.11
 Requires-Dist: beautifulsoup4<5,>=4.12.0
 Requires-Dist: defusedxml<1,>=0.7.1
 Requires-Dist: lxml<7,>=6.1.0
 Requires-Dist: mammoth<2,>=1.6.0
 Requires-Dist: markdownify<2,>=0.13.0
-Requires-Dist: pillow<13,>=12.2.0
+Requires-Dist: pillow<14,>=13.0.0
 Requires-Dist: pymupdf4llm<1,>=0.0.17
 Requires-Dist: pymupdf<2,>=1.24.0
-Requires-Dist: tomli>=2.0.0; python_version < '3.11'
 Requires-Dist: trafilatura<3,>=1.12.0
 Requires-Dist: urllib3<3,>=2.6.3
 Provides-Extra: dev
@@ -28,7 +27,7 @@ Requires-Dist: pytest-snapshot>=0.9.0; extra == 'dev'
 Requires-Dist: pytest>=8.0.0; extra == 'dev'
 Requires-Dist: pyyaml>=6.0; extra == 'dev'
 Requires-Dist: reportlab>=4.0.0; extra == 'dev'
-Requires-Dist: ruff>=0.6.0; extra == 'dev'
+Requires-Dist: ruff==0.15.22; extra == 'dev'
 Provides-Extra: high-fidelity
 Requires-Dist: docling>=2.0.0; extra == 'high-fidelity'
 Description-Content-Type: text/markdown

{any2md-1.1.1 → any2md-1.2.0}/any2md/__init__.py

@@ -2,7 +2,7 @@
 
 from any2md._docling_cache import docling_session, release_models
 
-__version__ = "1.1.1"
+__version__ = "1.2.0"
 
 __all__ = [
     "__version__",

{any2md-1.1.1 → any2md-1.2.0}/any2md/config.py

@@ -5,28 +5,17 @@ is treated as frontmatter overrides (deep-merged below ``--meta-file``
 and CLI ``--meta`` arguments). The ``[document_id]`` table provides
 ``publisher_prefix`` / ``type_code`` overrides for ``--auto-id``.
 
-TOML parsing uses stdlib ``tomllib`` on Python 3.11+. On older
-interpreters we fall back to the optional ``tomli`` package; if neither
-is importable, ``load_toml`` returns ``{}`` and config is silently
-disabled (no hard error — config is opt-in convenience).
+TOML parsing uses stdlib ``tomllib`` (any2md requires Python 3.11+).
 """
 
 from __future__ import annotations
 
-import sys
+import tomllib
 from pathlib import Path
 from typing import Any
 
 from any2md import _logging
 
-if sys.version_info >= (3, 11):
-    import tomllib
-else:  # pragma: no cover - exercised only on Python 3.10
-    try:
-        import tomli as tomllib  # type: ignore[import-not-found,no-redef]
-    except ImportError:
-        tomllib = None  # type: ignore[assignment]
-
 
 def load_toml(path: Path) -> dict[str, Any]:
     """Load a TOML file. Returns ``{}`` on any failure.
@@ -35,8 +24,6 @@ def load_toml(path: Path) -> dict[str, Any]:
     not block conversion. Callers that need strict validation should
     parse the file themselves.
     """
-    if tomllib is None:  # pragma: no cover - 3.10 without tomli
-        return {}
     try:
         with open(path, "rb") as f:
             return tomllib.load(f)

{any2md-1.1.1 → any2md-1.2.0}/any2md/converters/html.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import re
 import sys
 from datetime import date
 from email.utils import parsedate_to_datetime
@@ -84,6 +85,122 @@ def _extract(raw_html: str) -> tuple[str, str]:
     return md, "trafilatura+bs4_fallback"
 
 
+_WS_RE = re.compile(r"\s+")
+_ATX_H1_RE = re.compile(r"^#[ \t]+\S")
+_FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
+_WRAPPING_EMPHASIS_RE = re.compile(r"^(\*\*|__|\*|_)(.+?)\1$")
+
+
+def _one_line(text: str) -> str:
+    """Collapse every whitespace run (newlines included) to one space."""
+    return _WS_RE.sub(" ", text).strip()
+
+
+def _visible_h1(raw_html: str) -> str | None:
+    """Return the page's own ``<h1>`` text when the page has exactly one.
+
+    Empty ``<h1>`` wrappers (logo images, icon fonts) are ignored. Two or
+    more non-empty ``<h1>`` elements make the document title ambiguous,
+    so the caller falls back to the metadata title instead of guessing.
+    """
+    soup = BeautifulSoup(raw_html, "lxml")
+    texts = [t for t in (_one_line(h.get_text()) for h in soup.find_all("h1")) if t]
+    return texts[0] if len(texts) == 1 else None
+
+
+def _unwrap_emphasis(text: str) -> str:
+    """Strip emphasis markers that wrap the whole line (``**x**``, ``***x***``)."""
+    for _ in range(3):  # bold-italic nests at most three marker levels deep
+        m = _WRAPPING_EMPHASIS_RE.match(text)
+        if not m:
+            break
+        text = _one_line(m.group(2))
+    return text
+
+
+def _ensure_h1(body: str, title: str | None) -> str:
+    """Guarantee the body opens with one H1 when a title is known.
+
+    trafilatura keeps a heading only when it sits inside the content node
+    it selects, so a page ``<h1>`` in a hero banner or ``<header>`` block
+    never reaches the body. This puts it back so the C7 validator and the
+    README's single-H1 guarantee hold for HTML/URL output.
+
+    - Body already has an ATX H1 outside fenced code: returned unchanged.
+    - No usable title: returned unchanged (the validator still warns).
+    - First non-blank line repeats the title, as plain or emphasized text
+      (case-insensitive): that line is promoted to the H1 so the title
+      does not appear twice.
+    - Otherwise ``# <title>`` and a blank line are prepended.
+    """
+    lines = body.split("\n")
+    in_fence = False
+    for line in lines:
+        if _FENCE_RE.match(line):
+            in_fence = not in_fence
+        elif not in_fence and _ATX_H1_RE.match(line):
+            return body
+
+    heading = _one_line(title or "")
+    if not heading:
+        return body
+
+    first = next((i for i, line in enumerate(lines) if line.strip()), None)
+    if first is None:
+        return f"# {heading}\n"
+
+    candidate = _unwrap_emphasis(_one_line(lines[first]))
+    if candidate.casefold() == heading.casefold():
+        # Keep the page's own wording and casing for the promoted line.
+        lines[first] = f"# {candidate}"
+        return "\n".join(lines)
+
+    return f"# {heading}\n\n" + "\n".join(lines[first:])
+
+
+_ATX_HEADING_RE = re.compile(r"^(#{1,6})(?=[ \t]+\S)")
+
+
+def _normalize_heading_levels(body: str) -> str:
+    """Re-rank body headings so levels never skip below the H1.
+
+    CMS templates often mark sections as H3/H4 directly under the page
+    H1, which the C7 validator reports as an ``h1 -> h3`` skip. Each
+    heading's new level is its depth in a stack of the original levels
+    of the sections still open above it, so siblings stay siblings and
+    nesting is kept: ``H3, H3, H4, H3`` becomes ``H2, H2, H3, H2``.
+
+    Structured-lane S4 (``enforce_heading_hierarchy``) flattens one
+    heading at a time, which turns sibling H3s into H2 + H3 children;
+    that is why this lane uses its own ranking. H1 lines are never
+    moved and nothing is ever promoted to H1. Only the ``#`` prefix of
+    a heading line changes. A body with no skips is returned unchanged.
+    """
+    lines = body.split("\n")
+    stack: list[int] = [1]  # original levels of open sections; H1 at bottom
+    in_fence = False
+    changed = False
+    for i, line in enumerate(lines):
+        if _FENCE_RE.match(line):
+            in_fence = not in_fence
+            continue
+        m = None if in_fence else _ATX_HEADING_RE.match(line)
+        if not m:
+            continue
+        level = len(m.group(1))
+        if level == 1:
+            stack = [1]
+            continue
+        while len(stack) > 1 and stack[-1] >= level:
+            stack.pop()
+        stack.append(level)
+        new_level = len(stack)
+        if new_level != level:
+            lines[i] = "#" * new_level + line[level:]
+            changed = True
+    return "\n".join(lines) if changed else body
+
+
 def _extract_metadata(
     raw_html: str,
 ) -> tuple[str | None, list[str], str | None, str | None, list[str]]:
@@ -167,6 +284,14 @@ def convert_html(
 
         md_text, extracted_via = _extract(raw_html)
         title_hint, authors, org, doc_date, keywords = _extract_metadata(raw_html)
+        # Runs before pipeline.run so every text-lane stage, the C7
+        # validator, and compose() (content_hash, token_estimate, chunk
+        # level, abstract) all see the final body with its H1. The
+        # visible <h1> wins over the metadata title because it is the
+        # heading a reader sees; compose() keeps the metadata title as
+        # seo_title when the two differ.
+        md_text = _ensure_h1(md_text, _visible_h1(raw_html) or title_hint)
+        md_text = _normalize_heading_levels(md_text)
 
         if source_url and not doc_date:
             doc_date = _http_last_modified(source_url)
@@ -189,6 +314,7 @@ def convert_html(
             doc_type="html",
             extracted_via=extracted_via,
             lane="text",
+            seo_title=title_hint,
         )
         full = compose(md_text, meta, options, overrides=options.frontmatter_overrides)
 

{any2md-1.1.1 → any2md-1.2.0}/any2md/frontmatter.py

@@ -128,6 +128,15 @@ class SourceMeta:
     # is the human/legal entity. Default None to keep existing call sites
     # working without churn.
     produced_by: str | None = None
+    # v1.2.0: HTML metadata title (og:title / <title> via trafilatura).
+    # Emitted only when it differs from the final `title`, which follows
+    # the body H1 (the page's visible heading).
+    seo_title: str | None = None
+
+
+def _same_title(a: str, b: str) -> bool:
+    """Case- and whitespace-insensitive title equality."""
+    return " ".join(a.split()).casefold() == " ".join(b.split()).casefold()
 
 
 def estimate_tokens(body: str) -> int:
@@ -318,6 +327,9 @@ def _build_fields(
     fields["extracted_via"] = meta.extracted_via  # v1.0 provenance extension
     if meta.produced_by is not None:
         fields["produced_by"] = meta.produced_by  # v1.0.2 extension (spec §6.1)
+    seo_title = (meta.seo_title or "").strip()
+    if seo_title and not _same_title(seo_title, title):
+        fields["seo_title"] = seo_title  # v1.2.0 extension (HTML only)
     if meta.pages is not None:
         fields["pages"] = meta.pages
     if meta.word_count is not None:

{any2md-1.1.1 → any2md-1.2.0}/docs/architecture.md

@@ -301,6 +301,8 @@ class SourceMeta:
         "trafilatura", "trafilatura+bs4_fallback", "heuristic",
     ]
     lane: Lane
+    seo_title: str | None = None  # NEW v1.2.0 — HTML metadata title; emitted
+                                  # only when it differs from the final title.
 ```
 
 ### Field-by-field
@@ -311,6 +313,7 @@ class SourceMeta:
 | `authors` | PDF (`/Author` parsed), DOCX (`dc:creator`), HTML (`<meta name="author">`), TXT (always `[]`). v1.0.2 adds body-text extraction and an arxiv API enrichment via `heuristics.extract_authors`. | No — empty `[]` is valid |
 | `organization` | PDF/DOCX `Company`, HTML `og:site_name`, otherwise `None`. v1.0.2 routes the PDF `Creator` field and DOCX `<Application>` element through `heuristics.filter_organization` so software values populate `produced_by` instead. | No — empty `""` is valid |
 | `produced_by` | PDF `Creator` field (when it matches a known software pattern); DOCX `<Application>` element of `docProps/app.xml`; otherwise `None`. New in v1.0.2. | No — extension field |
+| `seo_title` | HTML/URL only: trafilatura's metadata title (`og:title` / `<title>`). `compose()` emits it only when it differs from the final `title`, which follows the body H1. Before the pipeline runs, the HTML converter restores that H1 from the page's single `<h1>` (else the metadata title) when trafilatura drops it. New in v1.2.0. | No — extension field |
 | `date` | PDF `creationDate`, DOCX `dcterms:modified`, URL `Last-Modified` HEAD response, file mtime, today | Yes — derived to today if all sources fail |
 | `keywords` | PDF `/Keywords`, DOCX `cp:keywords`, HTML `<meta name="keywords">`, trafilatura `categories`, otherwise `[]` | No — emitted only when non-empty |
 | `pages` | PDF only (page count via `pymupdf.open`); other formats `None` | No — extension field |

{any2md-1.1.1 → any2md-1.2.0}/docs/output-format.md

@@ -54,8 +54,9 @@ optional, then any2md extension fields retained from v0.7 for traceability.
 This is the order they appear in the YAML block.
 
 v1.0.2 added one extension field, `produced_by`, between `extracted_via` and
-`pages`. It's documented in its own subsection below and listed alongside the
-other any2md extension fields.
+`pages`. v1.2.0 added `seo_title` (HTML/URL only) right after it. Both are
+documented in their own subsections below and listed alongside the other any2md
+extension fields.
 
 ## Field-by-field reference
 
@@ -77,6 +78,11 @@ trafilatura), then to the source filename with the extension removed and
 underscores converted to spaces. Never empty — the fallback is guaranteed to
 produce a non-empty string.
 
+For HTML and URL input, the body H1 is the page's own `<h1>` when the page has
+exactly one non-empty `<h1>`, so `title` is the heading a reader sees. When the
+`og:title` / `<title>` metadata differs from it, that value is kept in
+[`seo_title`](#seo_title).
+
 **Example:** `"Quarterly Financial Report"`
 
 **Common mistake:** Editing the H1 in the body after generation invalidates
@@ -385,6 +391,22 @@ get a real organization name (or `""`) and never see software junk there.
 
 **Example empty:** field is omitted from the YAML block.
 
+### `seo_title`
+
+**auto, conditional** | type: `string` (any2md extension field, new in v1.2.0)
+
+**Derivation:** The HTML metadata title that trafilatura reads from
+`og:title` or `<title>`. Emitted only for HTML and URL input, and only when it
+differs from `title` (case and whitespace ignored). On many sites the SEO title
+and the visible `<h1>` are written separately. `title` follows the visible
+heading, so `seo_title` keeps the other one for search and citation use.
+
+**Example:** `title: "A Read-Only Account is All it Takes to Own a LiteLLM
+Server"` alongside `seo_title: "LiteLLM CVE-2026-35029: Exploited One Day After
+Disclosure"`.
+
+**Example empty:** field is omitted from the YAML block.
+
 ### `pages`, `word_count`
 
 **auto, conditional** | type: `integer`
@@ -400,15 +422,31 @@ follows three rules.
 
 ### Single H1
 
-Exactly one H1 (`# Heading`) per document. The pipeline's S4 stage promotes the
-first heading to H1 if no H1 was present in the source, and demotes any
-subsequent H1s to H2. This rule is what makes downstream chunkers and titlers
-work reliably.
+Exactly one H1 (`# Heading`) per document. This rule is what makes downstream
+chunkers and titlers work reliably. How it is enforced depends on the lane:
+
+- **Structured lane (Docling):** the S4 stage promotes the first heading to H1
+  if no H1 was present in the source, and demotes any subsequent H1s to H2.
+- **HTML / URL:** the converter restores the page title as the H1 when
+  trafilatura drops it (a page `<h1>` outside the article container is not
+  extracted). If the first body line already repeats the title as plain or
+  bold text, that line is promoted instead of adding a duplicate.
+- **Other text-lane input (pymupdf4llm, mammoth, TXT):** no rewriting. A body
+  without an H1 emits the validator warning described below.
 
 ### No skipped levels
 
-Heading levels never skip. An H2 followed by an H4 in the source is rewritten
-to H2 → H3 → H4. A skip emits a non-fatal validator warning during conversion
+Heading levels never skip. How skips are repaired depends on the lane:
+
+- **Structured lane (Docling):** the S4 stage rewrites a heading that jumps
+  more than one level to one level below the previous heading.
+- **HTML / URL:** headings are re-ranked by section depth below the H1. A body
+  whose sections are marked H3 or H4 directly under the H1 becomes H2 sections,
+  sibling headings stay siblings, and nesting is kept (`H3, H3, H4, H3` becomes
+  `H2, H2, H3, H2`). Only the `#` prefix of a heading line changes.
+- **Other text-lane input (pymupdf4llm, mammoth, TXT):** no rewriting.
+
+Any skip that survives emits a non-fatal validator warning during conversion
 (visible in stderr; promoted to a non-zero exit by `--strict`).
 
 ### Citations

{any2md-1.1.1 → any2md-1.2.0}/pyproject.toml

@@ -8,7 +8,7 @@ dynamic = ["version"]
 description = "Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter."
 readme = "README.md"
 license = "MIT"
-requires-python = ">=3.10"
+requires-python = ">=3.11"
 authors = [
     { name = "rocklambros", email = "rock@rockcyber.com" },
 ]
@@ -26,10 +26,9 @@ dependencies = [
     "trafilatura>=1.12.0,<3",
     "beautifulsoup4>=4.12.0,<5",
     "lxml>=6.1.0,<7",        # CVE-2026-41066 fix
-    "pillow>=12.2.0,<13",    # CVE-2026-25990, CVE-2026-40192 fix
+    "pillow>=13.0.0,<14",    # 39 advisories fixed in 13.0.0 (e.g. CVE-2026-112395)
     "urllib3>=2.6.3,<3",     # CVE-2026-21441 fix
     "defusedxml>=0.7.1,<1",  # XXE defense-in-depth (Task P2)
-    "tomli>=2.0.0; python_version < '3.11'",  # stdlib tomllib is 3.11+; needed by config.load_toml on 3.10
 ]
 
 [project.optional-dependencies]
@@ -38,7 +37,7 @@ dev = [
     "pytest-snapshot>=0.9.0",
     "pyyaml>=6.0",
     "reportlab>=4.0.0",
-    "ruff>=0.6.0",
+    "ruff==0.15.22",  # keep in step with .github/workflows/ci.yml
 ]
 high-fidelity = [
     "docling>=2.0.0",

{any2md-1.1.1 → any2md-1.2.0}/requirements.txt

@@ -10,7 +10,7 @@ beautifulsoup4==4.14.3
     # via
     #   any2md (pyproject.toml)
     #   markdownify
-certifi==2026.4.22
+certifi==2026.10.23
     # via trafilatura
 charset-normalizer==3.4.7
     # via
@@ -28,20 +28,20 @@ htmldate==1.9.4
     # via trafilatura
 justext==3.0.2
     # via trafilatura
-lxml[html-clean]==6.1.0
+lxml[html-clean]==6.1.3
     # via
     #   any2md (pyproject.toml)
     #   htmldate
     #   justext
     #   lxml-html-clean
     #   trafilatura
-lxml-html-clean==0.4.4
+lxml-html-clean==0.4.6
     # via lxml
 mammoth==1.12.0
     # via any2md (pyproject.toml)
 markdownify==1.2.2
     # via any2md (pyproject.toml)
-pillow==12.2.0
+pillow==13.0.0
     # via any2md (pyproject.toml)
 pymupdf==1.27.2.3
     # via
@@ -61,7 +61,7 @@ six==1.17.0
     # via
     #   markdownify
     #   python-dateutil
-soupsieve==2.8.3
+soupsieve==2.10
     # via beautifulsoup4
 tabulate==0.10.0
     # via pymupdf4llm
@@ -73,7 +73,7 @@ typing-extensions==4.15.0
     # via beautifulsoup4
 tzlocal==5.3.1
     # via dateparser
-urllib3==2.6.3
+urllib3==2.8.0
     # via
     #   any2md (pyproject.toml)
     #   courlan

any2md-1.2.0/tests/fixtures/docs/web_bold_title_first_line.html

@@ -0,0 +1,15 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Contest Announcement</title>
+</head>
+<body>
+  <section class="hero"><div class="heading"><h1>Join the Prompt Jar Contest</h1></div></section>
+  <article>
+    <p><strong>Join the Prompt Jar Contest</strong></p>
+    <p>Attackers increasingly target AI gateways because a single deployment holds provider keys, user records, and spend data for every model behind it. This paragraph is long enough to survive extraction.</p>
+    <p>Participants submit their most creative prompts at the booth, and the winner receives a prize announced at the end of the conference week.</p>
+  </article>
+</body>
+</html>

any2md-1.2.0/tests/fixtures/docs/web_h1_in_hero.html

@@ -0,0 +1,38 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Gateway Escalation Writeup | Example Labs</title>
+  <meta property="og:title" content="Gateway CVE: Exploited One Day After Disclosure">
+  <meta name="author" content="Test Author">
+</head>
+<body>
+  <nav><a href="/">Home</a> <a href="/blog">Blog</a></nav>
+  <main>
+    <section class="hero">
+      <header>
+        <p>Research</p>
+        <h1>A Read-Only Account Is All It Takes</h1>
+      </header>
+    </section>
+    <article>
+      <div class="prose">
+    <p>Attackers increasingly target AI gateways because a single deployment holds provider keys, user records, and spend data for every model behind it. This paragraph is long enough to survive extraction.</p>
+    <h2>Background</h2>
+    <p>The vulnerable endpoint accepted configuration updates from accounts that the documentation described as read-only, which let a low-privilege user rewrite the server configuration.</p>
+    <h2>Impact</h2>
+    <p>An attacker with the lowest role could read every stored provider key and pivot to the upstream model accounts without any further exploitation steps.</p>
+    <p>Detail paragraph 1: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 2: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 3: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 4: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 5: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 6: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 7: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+    <p>Detail paragraph 8: the gateway stored provider credentials in a shared database, and the role check ran only on the user interface, so direct calls to the administrative interface skipped it. Operators who rotated keys after the advisory still exposed the new keys to any account that could reach the endpoint.</p>
+      </div>
+    </article>
+  </main>
+  <footer>Site footer that must be stripped.</footer>
+</body>
+</html>

any2md-1.2.0/tests/fixtures/docs/web_h3_sections.html

@@ -0,0 +1,21 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Agent Recon Notes</title>
+</head>
+<body>
+  <section class="hero"><div class="heading"><h1>Agent Recon Notes</h1></div></section>
+  <article>
+    <p>This section explains one part of the incident in enough words that trafilatura treats the block as main content rather than boilerplate.</p>
+    <h3>Discovery</h3>
+    <p>This section explains one part of the incident in enough words that trafilatura treats the block as main content rather than boilerplate.</p>
+    <h3>Enumeration</h3>
+    <p>This section explains one part of the incident in enough words that trafilatura treats the block as main content rather than boilerplate.</p>
+    <h4>Tooling</h4>
+    <p>This section explains one part of the incident in enough words that trafilatura treats the block as main content rather than boilerplate.</p>
+    <h3>Takeaways</h3>
+    <p>This section explains one part of the incident in enough words that trafilatura treats the block as main content rather than boilerplate.</p>
+  </article>
+</body>
+</html>

any2md-1.2.0/tests/fixtures/docs/web_no_title.html

@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+</head>
+<body>
+  <article>
+    <p>Attackers increasingly target AI gateways because a single deployment holds provider keys, user records, and spend data for every model behind it. This paragraph is long enough to survive extraction.</p>
+    <p>A second paragraph with no headings anywhere on the page and no title metadata in the document head at all.</p>
+  </article>
+</body>
+</html>