any2md 1.0.2__tar.gz → 1.0.4__tar.gz

{any2md-1.0.2 → any2md-1.0.4}/CHANGELOG.md

@@ -4,6 +4,112 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.0.4] — 2026-04-26
+
+Patch release. Two follow-ups deferred from v1.0.3 (#17): the T7
+`dedupe_toc_table` now normalizes leader-dot padding from TOC
+cells before matching against body headings, so Docling-rendered
+TOC tables (e.g., `Purpose..........Page`) get properly stripped;
+and the audit script's `leading-toc-table` check no longer
+false-flags documents whose only GFM table sits at the end (skip
+when the body contains no H2).
+
+### Changed
+- `pipeline/text.py::dedupe_toc_table` strips leader-dot padding
+  (`re.sub(r'\.{3,}.*$', '', cell)`) from each TOC cell before
+  lower-casing and matching against body H2/H3 titles. (The
+  text-block TOC variant already handles this via `_TOC_LINE_RE`'s
+  capture group — only the table-variant site needed the fix.)
+- `scripts/audit-outputs.py` `leading-toc-table` check is now
+  gated on `"\n## "` being present in the body — when no H2
+  exists, no "leading-TOC region" is defined, so the check is
+  skipped instead of matching the entire document.
+
+### Fixes
+- TOC tables with dotted-page-number padding (typical Docling
+  output for SafeBreach-style PDFs — e.g., `Backup.pdf`,
+  `Password.pdf`) now get stripped by T7 in aggressive/maximum
+  profiles. Previously they survived because the cell content
+  `"1.1. Purpose............."` failed string-equality with the
+  body heading `"1.1. Purpose"`, dropping overlap below the 70%
+  threshold.
+- Audit script no longer reports false-positive
+  `leading-toc-table` flags on documents whose only GFM table is
+  at the end and the body contains no `## ` headings.
+
+### Tests
+- New `test_strips_toc_table_with_leader_dot_padding` in
+  `tests/unit/pipeline/test_text_dedupe_toc_table.py` — TOC
+  table whose cells carry leader-dot padding mirroring later H2
+  headings still triggers the strip.
+- New `tests/unit/scripts/test_audit_outputs.py` covering the
+  audit-script `leading-toc-table` check: no-H2-with-trailing-
+  table is NOT flagged; leading-table-before-H2 IS still flagged.
+
+### Other
+- `scripts/audit-outputs.py` is now tracked in version control
+  (was previously local-only).
+
+## [1.0.3] — 2026-04-26
+
+Patch release. Audit on the v1.0.2-regenerated corpus surfaced two
+patterns the previous release did not handle: cover-page-titled
+documents whose first H2 stripped to empty produced an empty
+`title` field, and Docling-lane output retained malformed `|`
+table-row remnants because the lone-punctuation filter was
+text-lane-only. The lane-agnostic body cleanup that v1.0.2's
+text lane already had also now runs on Docling output.
+
+### Added
+- New lane-agnostic stage **`strip_orphan_punctuation`** — drops
+  lines containing only `|` or `>`. Extracted from T10
+  `strip_web_fragments` so it can run on the structured (Docling)
+  lane in addition to the text (trafilatura) lane.
+
+### Changed
+- `heuristics.refine_title` now walks all H2 lines line-by-line
+  and skips any whose content strips to empty after removing
+  markdown emphasis, instead of always returning the first
+  regex match. Also guards the Wikipedia-prefix strip against
+  emitting empty when the candidate is the bare prefix.
+- `pipeline/structured.py` now extends `STAGES` with
+  `strip_repeated_byline` (T9), `dedupe_toc_table` (T7),
+  `strip_cover_artifacts` (T8), and `strip_orphan_punctuation`
+  so Docling output gets the same body-cleanup pass as text-lane
+  output. T10's short-fragment heuristic stays text-lane-only —
+  it would over-strip Docling's deliberate short headings.
+- `pipeline/cleanup.py` `decode_html_entities` (C8) now loops
+  `html.unescape` until output stabilizes (max 5 iterations) so
+  double-encoded entities like `&` → `&` → `&` are
+  fully decoded. Some extractors emit doubly-encoded entities
+  that survived a single pass.
+
+### Fixes
+- **Empty-title regression**: cover-page H1 documents whose first
+  H2 contained only markdown emphasis, NBSP-equivalent unicode,
+  or where the H2 regex's `\s+` class spanned a newline into the
+  next paragraph produced `title: ""`. Walk H2 lines line-by-line
+  with a non-empty guard.
+- **Orphan-punctuation in Docling output**: malformed table-row
+  remnants (lone `|`) survived in structured-lane output because
+  T10 was text-lane-only. New `strip_orphan_punctuation` runs on
+  both lanes.
+- **Double-encoded HTML entities**: `&` left a residual
+  `&` after a single unescape pass. Loop until stable.
+
+### Tests
+- 4 new tests in `tests/unit/test_heuristics.py::TestRefineTitle`
+  covering NBSP-only H2, emphasis-only H2 (skip-and-pick-next),
+  all-empty-H2 fallthrough, and Wikipedia-prefix-only candidate.
+- New `tests/unit/pipeline/test_strip_orphan_punctuation.py`
+  covering aggressive removal, conservative no-op, and preservation
+  of lines containing other content alongside `|`/`>`.
+- New `tests/unit/pipeline/test_structured_body_cleanup_stages.py`
+  asserting the lane-agnostic stages are registered and that the
+  targeted patterns are removed from structured-lane output.
+- New `tests/unit/pipeline/test_cleanup_html_entities_double_encoded.py`
+  covering the C8 loop.
+
 ## [1.0.2] — 2026-04-26
 
 Patch release. Closes issue #15 plus 8 additional quality issues

any2md-1.0.4/CODE_OF_CONDUCT.md

@@ -0,0 +1,37 @@
+# Code of Conduct
+
+This project adopts the **[Contributor Covenant version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/)**
+as its code of conduct. The full text is available at the link above and
+the canonical source is at https://www.contributor-covenant.org/.
+
+## Scope
+
+The covenant applies to all project spaces, including but not limited to:
+
+- The GitHub repository (issues, pull requests, discussions, wiki, code
+  review comments, commit messages).
+- Any community channel that explicitly identifies itself as a project
+  space (mailing list, chat room, project event).
+- Public spaces when an individual is officially representing the project.
+
+## Reporting
+
+Reports of abuse, harassment, or otherwise unacceptable behavior may be
+sent to **rock@rockcyber.com**. Reports are confidential and will be
+reviewed within 5 business days.
+
+If the report concerns the maintainer, please contact a trusted
+intermediary or use a public reporting channel of your choice.
+
+## Enforcement
+
+The maintainer is responsible for enforcement and may take any action
+deemed appropriate, ranging from a private warning to a permanent ban
+from project spaces. Enforcement actions follow the
+[Contributor Covenant Enforcement Guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/#enforcement-guidelines).
+
+## Attribution
+
+Code of conduct adapted from the
+[Contributor Covenant](https://www.contributor-covenant.org), version 2.1,
+available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/.

{any2md-1.0.2 → any2md-1.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: any2md
-Version: 1.0.2
+Version: 1.0.4
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -36,21 +36,16 @@ Description-Content-Type: text/markdown
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
-## What's new in 1.0.2
+## What's new in 1.0.3
 
-A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+A patch release closing two regressions surfaced by audit on the v1.0.2-regenerated corpus, plus a Docling-lane reach for the v1.0.2 body-cleanup stages:
 
-- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
-- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
-- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&`, `<`, `>`, and numeric entities (`—`, `—`) from body text. Code-block aware.
-- Four new text-lane stages (aggressive/maximum profile only):
-  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
-  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
-  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
-  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
-- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+- **Empty-title fix**: cover-page-titled documents (`# INTERNATIONAL STANDARD` etc.) whose first H2 stripped to empty (markdown emphasis only, NBSP-equivalent unicode, regex span crossing into the next paragraph) emitted `title: ""`. `heuristics.refine_title` now walks H2 lines line-by-line and skips any that strip to empty after dropping markdown emphasis. The Wikipedia-prefix strip got the same non-empty guard.
+- **Docling-lane orphan-punctuation**: lone `|` or `>` lines from Docling's malformed table parsing leaked through because T10 `strip_web_fragments` was text-lane-only by design. The orphan-punctuation portion is extracted into a new lane-agnostic stage `strip_orphan_punctuation` that runs on Docling output too. T10's trafilatura-specific short-fragment heuristic stays text-lane-only.
+- **Lane-agnostic body cleanup on Docling lane**: T7 `dedupe_toc_table`, T8 `strip_cover_artifacts`, and T9 `strip_repeated_byline` are now appended to the structured-lane `STAGES` list, so Docling output gets the same body-cleanup pass that text-lane output already had.
+- **Double-encoded HTML entities**: C8 `decode_html_entities` now loops `html.unescape` until output stabilizes (max 5 iterations) so `&` → `&` → `&` is fully decoded.
 
-The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md](docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md). v1.0.2 highlights — `any2md/heuristics.py`, `produced_by`, C8, T7–T10, `--no-arxiv-lookup` — remain available; see [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md).
 
 ## What this is
 
@@ -467,7 +462,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0.2
+                  any2md  v1.0.3
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -498,6 +493,12 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │  S2 table compactor     │
         │  S3 cite normalizer     │
         │  S4 heading hierarchy   │
+        │  T9 strip repeated      │   ◀ NEW v1.0.3
+        │     byline              │
+        │  T7 dedupe TOC table    │   ◀ NEW v1.0.3
+        │  T8 strip cover         │   ◀ NEW v1.0.3
+        │     artifacts           │
+        │  strip_orphan_punct     │   ◀ NEW v1.0.3
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
@@ -512,6 +513,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │  T5 hdr/ftr strip       │
         │  T10 strip web          │   ◀ NEW (D2)
         │      fragments          │
+        │      ↳ strip_orphan_punct  ◀ split out v1.0.3
         │  T6 list/code restore   │
         │  T8 strip cover         │   ◀ NEW (D3)
         │     artifacts           │

{any2md-1.0.2 → any2md-1.0.4}/README.md

@@ -6,21 +6,16 @@
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
-## What's new in 1.0.2
+## What's new in 1.0.3
 
-A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+A patch release closing two regressions surfaced by audit on the v1.0.2-regenerated corpus, plus a Docling-lane reach for the v1.0.2 body-cleanup stages:
 
-- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
-- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
-- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&`, `<`, `>`, and numeric entities (`—`, `—`) from body text. Code-block aware.
-- Four new text-lane stages (aggressive/maximum profile only):
-  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
-  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
-  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
-  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
-- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+- **Empty-title fix**: cover-page-titled documents (`# INTERNATIONAL STANDARD` etc.) whose first H2 stripped to empty (markdown emphasis only, NBSP-equivalent unicode, regex span crossing into the next paragraph) emitted `title: ""`. `heuristics.refine_title` now walks H2 lines line-by-line and skips any that strip to empty after dropping markdown emphasis. The Wikipedia-prefix strip got the same non-empty guard.
+- **Docling-lane orphan-punctuation**: lone `|` or `>` lines from Docling's malformed table parsing leaked through because T10 `strip_web_fragments` was text-lane-only by design. The orphan-punctuation portion is extracted into a new lane-agnostic stage `strip_orphan_punctuation` that runs on Docling output too. T10's trafilatura-specific short-fragment heuristic stays text-lane-only.
+- **Lane-agnostic body cleanup on Docling lane**: T7 `dedupe_toc_table`, T8 `strip_cover_artifacts`, and T9 `strip_repeated_byline` are now appended to the structured-lane `STAGES` list, so Docling output gets the same body-cleanup pass that text-lane output already had.
+- **Double-encoded HTML entities**: C8 `decode_html_entities` now loops `html.unescape` until output stabilizes (max 5 iterations) so `&` → `&` → `&` is fully decoded.
 
-The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md](docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md). v1.0.2 highlights — `any2md/heuristics.py`, `produced_by`, C8, T7–T10, `--no-arxiv-lookup` — remain available; see [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md).
 
 ## What this is
 
@@ -437,7 +432,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0.2
+                  any2md  v1.0.3
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -468,6 +463,12 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │  S2 table compactor     │
         │  S3 cite normalizer     │
         │  S4 heading hierarchy   │
+        │  T9 strip repeated      │   ◀ NEW v1.0.3
+        │     byline              │
+        │  T7 dedupe TOC table    │   ◀ NEW v1.0.3
+        │  T8 strip cover         │   ◀ NEW v1.0.3
+        │     artifacts           │
+        │  strip_orphan_punct     │   ◀ NEW v1.0.3
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
@@ -482,6 +483,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │  T5 hdr/ftr strip       │
         │  T10 strip web          │   ◀ NEW (D2)
         │      fragments          │
+        │      ↳ strip_orphan_punct  ◀ split out v1.0.3
         │  T6 list/code restore   │
         │  T8 strip cover         │   ◀ NEW (D3)
         │     artifacts           │

any2md-1.0.4/SECURITY.md

@@ -0,0 +1,83 @@
+# Security Policy
+
+## Supported versions
+
+| Version | Status |
+|---|---|
+| 1.0.x   | ✅ Supported (current stable). Patch releases for security and quality fixes are published as needed. |
+| 0.7.x   | ⚠️ Receives critical security fixes only until 2026-10-26 (six months after the v1.0.0 GA on 2026-04-26). |
+| < 0.7   | ❌ Not supported. Please upgrade. |
+
+## Reporting a vulnerability
+
+**Please do not file public issues for security vulnerabilities.** Public
+disclosure before a fix is published puts users at risk.
+
+Email security disclosures to **rock@rockcyber.com** with the subject line
+`any2md security disclosure`. Encrypted email is welcome — request a
+public key in your initial message and one will be provided.
+
+Include in your report:
+
+- A description of the issue and the affected component.
+- Affected version(s) — e.g., `pip show any2md` output, or commit SHA.
+- Reproduction steps (smallest possible repro is ideal).
+- Any proof-of-concept (please redact sensitive data).
+- Whether you intend to publish your own write-up; if so, a proposed
+  embargo date so we can coordinate.
+
+### Response timeline
+
+- **Acknowledgement** within 5 business days.
+- **Triage and severity assessment** within 10 business days.
+- **Critical issues** (remote code execution, data exfiltration via
+  crafted input, supply-chain compromise of a published wheel): patched
+  and released within 30 days of confirmation, with a coordinated public
+  advisory.
+- **High** (denial-of-service, information disclosure with limited blast
+  radius): patched within 60 days.
+- **Medium / low** issues are tracked in private until a fix lands in a
+  scheduled release, then disclosed publicly via the GitHub Security
+  Advisories page.
+
+## Scope
+
+In scope:
+
+- The `any2md` library and CLI.
+- The published wheels and sdists on PyPI.
+- The release pipeline (`.github/workflows/publish.yml`) and any
+  side-effects of running the workflow.
+- The default conversion pipeline (PDF / DOCX / HTML / URL / TXT) and
+  the optional Docling backend integration.
+- The arxiv API enrichment path (network call + XML parsing).
+- The SSRF guards in the URL fetcher.
+
+Out of scope:
+
+- Vulnerabilities in third-party dependencies (please file directly with
+  the upstream — Docling, PyMuPDF, mammoth, trafilatura, BeautifulSoup,
+  lxml, etc.). We monitor upstream advisories via Dependabot.
+- Issues that require local filesystem access already granted to the
+  running Python process.
+- Local timing attacks against your own machine.
+- Content correctness issues in converted Markdown (those are
+  conversion-quality reports — please use the
+  [Conversion quality issue template](.github/ISSUE_TEMPLATE/conversion_quality.md)
+  instead).
+
+## Hardening reference
+
+If you operate any2md in a hardened environment:
+
+- Pass `--no-arxiv-lookup` to disable the arxiv API call.
+- Pass `--max-file-size` with a budget appropriate for your workload.
+- Run conversions in a sandbox that cannot reach private networks
+  (the SSRF guard validates resolved IPs, but defense in depth is best).
+- Pin `any2md` and its extras (`any2md[high-fidelity]==<exact-version>`)
+  to a verified version and audit the lockfile.
+
+## Disclosure history
+
+No publicly disclosed vulnerabilities to date. Future advisories will
+appear at https://github.com/rocklambros/any2md/security/advisories.

{any2md-1.0.2 → any2md-1.0.4}/any2md/__init__.py

@@ -1,3 +1,3 @@
 """Convert PDF, DOCX, HTML, and TXT files to LLM-optimized Markdown."""
 
-__version__ = "1.0.2"
+__version__ = "1.0.4"

{any2md-1.0.2 → any2md-1.0.4}/any2md/cli.py

@@ -65,11 +65,7 @@ def _deep_merge_overrides(
     """
     out = dict(base)
     for key, val in overrides.items():
-        if (
-            key in out
-            and isinstance(out[key], dict)
-            and isinstance(val, dict)
-        ):
+        if key in out and isinstance(out[key], dict) and isinstance(val, dict):
             out[key] = _deep_merge_overrides(out[key], val)
         else:
             out[key] = val
@@ -136,9 +132,9 @@ def main():
         "--no-arxiv-lookup",
         action="store_true",
         help="Disable the arxiv API lookup that enriches frontmatter for "
-             "PDFs whose filename matches the arxiv ID pattern (\\d{4}.\\d{4,5}). "
-             "Useful in airgapped envs or when you don't want any2md to make "
-             "network calls. Default: arxiv lookup is enabled.",
+        "PDFs whose filename matches the arxiv ID pattern (\\d{4}.\\d{4,5}). "
+        "Useful in airgapped envs or when you don't want any2md to make "
+        "network calls. Default: arxiv lookup is enabled.",
     )
     parser.add_argument(
         "--backend",
@@ -224,9 +220,7 @@ def main():
 
     if args.meta_file is not None:
         if not args.meta_file.is_file():
-            print(
-                f"Error: --meta-file not found: {args.meta_file}", file=sys.stderr
-            )
+            print(f"Error: --meta-file not found: {args.meta_file}", file=sys.stderr)
             sys.exit(1)
         cfg = load_toml(args.meta_file)
         overrides = _deep_merge_overrides(overrides, extract_meta_overrides(cfg))
@@ -253,6 +247,7 @@ def main():
         or args.backend == "docling"
     ):
         from any2md._docling import has_docling, INSTALL_HINT_MSG
+
         if not has_docling():
             print(
                 f"  ERROR: Docling required for --high-fidelity / --ocr-figures / --save-images / --backend docling.\n"

{any2md-1.0.2 → any2md-1.0.4}/any2md/converters/__init__.py

@@ -70,17 +70,19 @@ def convert_file(
     ext = file_path.suffix.lower()
     if ext == ".pdf":
         from any2md.converters.pdf import convert_pdf
+
         return convert_pdf(file_path, output_dir, options=options, force=force)
     if ext == ".docx":
         from any2md.converters.docx import convert_docx
+
         return convert_docx(file_path, output_dir, options=options, force=force)
     if ext in (".html", ".htm"):
         from any2md.converters.html import convert_html
+
         return convert_html(file_path, output_dir, options=options, force=force)
     if ext == ".txt":
         from any2md.converters.txt import convert_txt
+
         return convert_txt(file_path, output_dir, options=options, force=force)
-    print(
-        f"  UNSUPPORTED: {file_path.name} (no converter for {ext})", file=sys.stderr
-    )
+    print(f"  UNSUPPORTED: {file_path.name} (no converter for {ext})", file=sys.stderr)
     return False

{any2md-1.0.2 → any2md-1.0.4}/any2md/converters/docx.py

@@ -99,9 +99,7 @@ def _extract_via_docling(docx_path: Path) -> tuple[str, str]:
     return result.document.export_to_markdown(), "docling"
 
 
-def _extract_via_mammoth(
-    docx_path: Path, options: PipelineOptions
-) -> tuple[str, str]:
+def _extract_via_mammoth(docx_path: Path, options: PipelineOptions) -> tuple[str, str]:
     with open(docx_path, "rb") as f:
         html_result = mammoth.convert_to_html(f)
     md = markdownify.markdownify(
@@ -185,9 +183,7 @@ def convert_docx(
             lane=lane,
             produced_by=props["produced_by"],
         )
-        full = compose(
-            md_text, meta, options, overrides=options.frontmatter_overrides
-        )
+        full = compose(md_text, meta, options, overrides=options.frontmatter_overrides)
 
         output_dir.mkdir(parents=True, exist_ok=True)
         out_path.write_text(full, encoding="utf-8", newline="\n")

{any2md-1.0.2 → any2md-1.0.4}/any2md/converters/html.py

@@ -91,6 +91,7 @@ def _http_last_modified(url: str) -> str | None:
             lm = resp.headers.get("Last-Modified")
             if lm:
                 from email.utils import parsedate_to_datetime
+
                 return parsedate_to_datetime(lm).date().isoformat()
     except Exception:  # noqa: BLE001
         pass
@@ -139,8 +140,10 @@ def _extract_metadata(
     elif isinstance(bare, dict):
         data = bare
     else:
-        data = {k: getattr(bare, k, None) for k in
-                ("title", "author", "sitename", "date", "categories")}
+        data = {
+            k: getattr(bare, k, None)
+            for k in ("title", "author", "sitename", "date", "categories")
+        }
     title = data.get("title")
     authors_raw = data.get("author") or ""
     authors = [a.strip() for a in authors_raw.split(",") if a.strip()]
@@ -221,9 +224,7 @@ def convert_html(
             extracted_via=extracted_via,
             lane="text",
         )
-        full = compose(
-            md_text, meta, options, overrides=options.frontmatter_overrides
-        )
+        full = compose(md_text, meta, options, overrides=options.frontmatter_overrides)
 
         output_dir.mkdir(parents=True, exist_ok=True)
         out_path.write_text(full, encoding="utf-8", newline="\n")

{any2md-1.0.2 → any2md-1.0.4}/any2md/converters/pdf.py

@@ -262,17 +262,13 @@ def convert_pdf(
             lane=lane,
             produced_by=props["produced_by"],
         )
-        full = compose(
-            md_text, meta, options, overrides=options.frontmatter_overrides
-        )
+        full = compose(md_text, meta, options, overrides=options.frontmatter_overrides)
 
         output_dir.mkdir(parents=True, exist_ok=True)
         out_path.write_text(full, encoding="utf-8", newline="\n")
         suffix = f", {len(warnings)} warning(s)" if warnings else ""
         if not is_quiet():
-            print(
-                f"  OK: {out_name} ({page_count} pages, via {extracted_via}{suffix})"
-            )
+            print(f"  OK: {out_name} ({page_count} pages, via {extracted_via}{suffix})")
         return True
 
     except (OSError, ValueError, RuntimeError) as e:

{any2md-1.0.2 → any2md-1.0.4}/any2md/converters/txt.py

@@ -27,8 +27,22 @@ def _is_title_case(line: str) -> bool:
     if len(words) < 2:
         return False
     skip = {
-        "a", "an", "the", "and", "or", "but", "in", "on", "at", "to",
-        "for", "of", "with", "by", "from", "is",
+        "a",
+        "an",
+        "the",
+        "and",
+        "or",
+        "but",
+        "in",
+        "on",
+        "at",
+        "to",
+        "for",
+        "of",
+        "with",
+        "by",
+        "from",
+        "is",
     }
     caps = sum(1 for w in words if w[0].isupper() or w.lower() in skip)
     return caps >= len(words) * 0.7
@@ -182,9 +196,7 @@ def convert_txt(
         md_text, warnings = pipeline.run(md_text, "text", options)
         add_warnings(warnings)
         meta = _build_meta(txt_path, md_text)
-        full = compose(
-            md_text, meta, options, overrides=options.frontmatter_overrides
-        )
+        full = compose(md_text, meta, options, overrides=options.frontmatter_overrides)
 
         output_dir.mkdir(parents=True, exist_ok=True)
         out_path.write_text(full, encoding="utf-8", newline="\n")

{any2md-1.0.2 → any2md-1.0.4}/any2md/frontmatter.py

@@ -51,16 +51,20 @@ class SourceMeta:
     title_hint: str | None
     authors: list[str]
     organization: str | None
-    date: str | None              # ISO-8601 YYYY-MM-DD
+    date: str | None  # ISO-8601 YYYY-MM-DD
     keywords: list[str]
-    pages: int | None             # PDFs only
-    word_count: int | None        # DOCX/HTML/TXT (post-cleanup)
+    pages: int | None  # PDFs only
+    word_count: int | None  # DOCX/HTML/TXT (post-cleanup)
     source_file: str | None
     source_url: str | None
-    doc_type: Literal["pdf", "docx", "html", "txt"]   # v0.7-compat extension
+    doc_type: Literal["pdf", "docx", "html", "txt"]  # v0.7-compat extension
     extracted_via: Literal[
-        "docling", "pymupdf4llm", "mammoth+markdownify",
-        "trafilatura", "trafilatura+bs4_fallback", "heuristic",
+        "docling",
+        "pymupdf4llm",
+        "mammoth+markdownify",
+        "trafilatura",
+        "trafilatura+bs4_fallback",
+        "heuristic",
     ]
     lane: Lane
     # v1.0.2: software/tool that produced the source (PDF Creator field for
@@ -105,7 +109,7 @@ def extract_abstract(body: str) -> str | None:
         return None
 
     # Walk paragraphs after the H1 (split on blank lines).
-    after = body[h1.end():]
+    after = body[h1.end() :]
     paragraphs = [p.strip() for p in re.split(r"\n\s*\n", after)]
     for para in paragraphs:
         if not para:
@@ -145,9 +149,9 @@ def derive_title(body: str, title_hint: str | None, fallback: str) -> str:
 def _yaml_escape(value: str) -> str:
     return (
         value.replace("\\", "\\\\")
-             .replace('"', '\\"')
-             .replace("\n", "\\n")
-             .replace("\r", "\\r")
+        .replace('"', '\\"')
+        .replace("\n", "\\n")
+        .replace("\r", "\\r")
     )
 
 
@@ -194,7 +198,8 @@ def _build_fields(
     title_raw = derive_title(body, meta.title_hint, fallback)
     arxiv_id = heuristics.is_arxiv_filename(meta.source_file or "")
     title = heuristics.refine_title(
-        title_raw, body,
+        title_raw,
+        body,
         source_url=meta.source_url,
         profile=options.profile,
     )
@@ -253,7 +258,7 @@ def _build_fields(
         fields["source_file"] = meta.source_file
     if meta.source_url:
         fields["source_url"] = meta.source_url
-    fields["type"] = meta.doc_type           # v0.7-compat field (spec §3.2)
+    fields["type"] = meta.doc_type  # v0.7-compat field (spec §3.2)
     fields["extracted_via"] = meta.extracted_via  # v1.0 provenance extension
     if meta.produced_by is not None:
         fields["produced_by"] = meta.produced_by  # v1.0.2 extension (spec §6.1)
@@ -273,11 +278,7 @@ def _deep_merge(base: dict[str, Any], overrides: dict[str, Any]) -> dict[str, An
     """
     out = dict(base)
     for key, val in overrides.items():
-        if (
-            key in out
-            and isinstance(out[key], dict)
-            and isinstance(val, dict)
-        ):
+        if key in out and isinstance(out[key], dict) and isinstance(val, dict):
             out[key] = _deep_merge(out[key], val)
         else:
             out[key] = val