any2md 1.0.0__tar.gz → 1.0.2__tar.gz

{any2md-1.0.0 → any2md-1.0.2}/.gitignore

@@ -38,6 +38,7 @@ input/
 template/
 test_docs/
 .worktrees/
+logo/
 
 # ZERG / dev tooling
 .gsd/

{any2md-1.0.0 → any2md-1.0.2}/CHANGELOG.md

@@ -4,6 +4,89 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.0.2] — 2026-04-26
+
+Patch release. Closes issue #15 plus 8 additional quality issues
+discovered during deep investigation against four real-world inputs:
+arxiv academic paper, ISO/IEC 27002 standard, COMP4441 academic
+DOCX, and a Wikipedia article via URL.
+
+### Added
+- New `any2md/heuristics.py` module — pure functions for frontmatter
+  field refinement: `refine_title`, `refine_abstract`, `extract_authors`,
+  `filter_organization`, `arxiv_lookup`, `is_arxiv_filename`. Called
+  from `frontmatter.compose()` and from converter modules.
+- New `produced_by` extension field on `SourceMeta` and frontmatter
+  (between `extracted_via` and `pages`). Records the software that
+  produced the source file (PDF `Creator`, DOCX `Application`).
+  Distinct from `extracted_via` which records the any2md backend
+  that produced the markdown.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal
+  removal of `&`, `>`, `<`, numeric entities (`—`,
+  `—`) from body. Code-block aware (skips fenced ` ``` ` blocks).
+- New text-lane stages T7-T10 (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that T4's
+    text-formatted-TOC heuristic doesn't catch. Common in academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code
+    blurbs, version stamps like "Third edition 2022-02") in the first
+    ~30 lines, before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact
+    Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction
+    fragments (orphan `|` / `>` lines, short incomplete sentences
+    surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup` to disable the arxiv API metadata
+  enrichment. Arxiv enrichment is on by default for filenames matching
+  `\d{4}\.\d{4,5}` (e.g., `2501.17755v1.pdf`); SSRF-guarded with 5s
+  timeout; failures emit non-blocking warnings.
+- Project logo and brand assets under `assets/`.
+
+### Fixes
+The eleven quality issues — see issue #15 and
+`docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md`:
+- A1: authors not extracted from PDF body byline (academic PDFs).
+- A2: abstract picked the byline / cover blurb / TOC line.
+- A3: organization populated with PDF Creator software junk.
+- B1: HTML entities leaked to body universally.
+- C1: ISO/TR titles detected as "INTERNATIONAL STANDARD" cover header.
+- C3: DOCX titles concatenated course code + project title.
+- C4: Wikipedia titles kept "Wikipedia:" namespace prefix.
+- D1: TOCs dumped as markdown tables in academic PDFs.
+- D2: trafilatura fragments leaked into web outputs.
+- D3: ISO cover-page QR-code blurb leaked into body.
+- E1: "Author's Contact Information:" line duplicated byline.
+
+### Changed
+- `frontmatter.compose()` now consults `heuristics.py` to refine
+  title / abstract / authors before YAML emission.
+- `--profile conservative` now also gates the new heuristic
+  aggressiveness (skip-lists active; speculative inferences off).
+- PDF `Creator` software values (LaTeX, acmart, Adobe InDesign,
+  Microsoft Word, etc.) no longer populate `organization`. They
+  go to the new `produced_by` field instead.
+
+## [1.0.1] — 2026-04-26
+
+Patch release. Adds an explicit backend-selection CLI flag.
+
+### Added
+- `--backend {docling,pymupdf4llm,mammoth}` CLI flag and corresponding
+  `PipelineOptions.backend` field. Lets users override the automatic
+  backend selection. Useful when Docling extracts incorrectly on a
+  specific input and the user wants to fall back to pymupdf4llm
+  without uninstalling Docling. Mismatched format/backend combinations
+  (e.g., `--backend pymupdf4llm` on a DOCX) error out per file.
+
+### Fixes (workaround)
+- Refs #13 — Docling drops list items on certain academic PDFs with
+  the "Parent element of the list item is not a ListGroup" warning.
+  No upstream fix yet; users hitting this can now `any2md --backend
+  pymupdf4llm doc.pdf` to use the lightweight backend instead.
+
+### Note
+- `--backend docling` is equivalent to the existing `--high-fidelity`
+  / `-H` flag (both kept). Future releases may consolidate.
+
 ## [1.0.0] — 2026-04-26
 
 First stable release of any2md v1.0. Validated end-to-end against

{any2md-1.0.0 → any2md-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: any2md
-Version: 1.0.0
+Version: 1.0.2
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -28,10 +28,30 @@ Provides-Extra: high-fidelity
 Requires-Dist: docling>=2.0.0; extra == 'high-fidelity'
 Description-Content-Type: text/markdown
 
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.2
+
+A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+
+- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
+- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&amp;`, `&lt;`, `&gt;`, and numeric entities (`&#x2014;`, `&#8212;`) from body text. Code-block aware.
+- Four new text-lane stages (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -144,6 +164,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -226,7 +247,7 @@ You'd run this when your corpus has multi-column PDFs, complex tables, or scanne
 pip install "any2md[high-fidelity]"
 ```
 
-After install, the CLI auto-detects Docling and uses it for PDF and DOCX. To force it explicitly (and fail fast when it isn't installed), pass `-H` / `--high-fidelity`. To require it (treat fallback as an error), combine with `--strict`.
+After install, the CLI auto-detects Docling and uses it for PDF and DOCX. To force it explicitly (and fail fast when it isn't installed), pass `-H` / `--high-fidelity`. To require it (treat fallback as an error), combine with `--strict`. Use `--backend pymupdf4llm` to force the lightweight backend even when Docling is installed (useful when Docling produces extraction artifacts on a specific input).
 
 ### When do I need high-fidelity?
 
@@ -408,6 +429,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -442,7 +467,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.2
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -459,43 +484,66 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -504,7 +552,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

{any2md-1.0.0 → any2md-1.0.2}/README.md

@@ -1,7 +1,27 @@
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.2
+
+A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+
+- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
+- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&amp;`, `&lt;`, `&gt;`, and numeric entities (`&#x2014;`, `&#8212;`) from body text. Code-block aware.
+- Four new text-lane stages (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -114,6 +134,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -196,7 +217,7 @@ You'd run this when your corpus has multi-column PDFs, complex tables, or scanne
 pip install "any2md[high-fidelity]"
 ```
 
-After install, the CLI auto-detects Docling and uses it for PDF and DOCX. To force it explicitly (and fail fast when it isn't installed), pass `-H` / `--high-fidelity`. To require it (treat fallback as an error), combine with `--strict`.
+After install, the CLI auto-detects Docling and uses it for PDF and DOCX. To force it explicitly (and fail fast when it isn't installed), pass `-H` / `--high-fidelity`. To require it (treat fallback as an error), combine with `--strict`. Use `--backend pymupdf4llm` to force the lightweight backend even when Docling is installed (useful when Docling produces extraction artifacts on a specific input).
 
 ### When do I need high-fidelity?
 
@@ -378,6 +399,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -412,7 +437,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.2
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -429,43 +454,66 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -474,7 +522,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

{any2md-1.0.0 → any2md-1.0.2}/any2md/__init__.py

@@ -1,3 +1,3 @@
 """Convert PDF, DOCX, HTML, and TXT files to LLM-optimized Markdown."""
 
-__version__ = "1.0.0"
+__version__ = "1.0.2"

{any2md-1.0.0 → any2md-1.0.2}/any2md/cli.py

@@ -132,6 +132,24 @@ def main():
         action="store_true",
         help="Force the Docling backend (PDF/DOCX). Exit 1 if not installed.",
     )
+    parser.add_argument(
+        "--no-arxiv-lookup",
+        action="store_true",
+        help="Disable the arxiv API lookup that enriches frontmatter for "
+             "PDFs whose filename matches the arxiv ID pattern (\\d{4}.\\d{4,5}). "
+             "Useful in airgapped envs or when you don't want any2md to make "
+             "network calls. Default: arxiv lookup is enabled.",
+    )
+    parser.add_argument(
+        "--backend",
+        choices=("docling", "pymupdf4llm", "mammoth"),
+        default=None,
+        help="Force a specific extraction backend. By default any2md auto-selects "
+        "(prefers Docling when installed). Use 'docling' (equivalent to --high-fidelity), "
+        "'pymupdf4llm' to force the lightweight PDF fallback even when Docling is installed, "
+        "or 'mammoth' to force the lightweight DOCX fallback. Mismatched format/backend "
+        "combinations (e.g., --backend pymupdf4llm on a DOCX) error out per file.",
+    )
     parser.add_argument(
         "--ocr-figures",
         action="store_true",
@@ -228,11 +246,16 @@ def main():
         sys.exit(1)
     overrides = _deep_merge_overrides(overrides, cli_meta)
 
-    if args.high_fidelity or args.ocr_figures or args.save_images:
+    if (
+        args.high_fidelity
+        or args.ocr_figures
+        or args.save_images
+        or args.backend == "docling"
+    ):
         from any2md._docling import has_docling, INSTALL_HINT_MSG
         if not has_docling():
             print(
-                f"  ERROR: Docling required for --high-fidelity / --ocr-figures / --save-images.\n"
+                f"  ERROR: Docling required for --high-fidelity / --ocr-figures / --save-images / --backend docling.\n"
                 f"  {INSTALL_HINT_MSG}",
                 file=sys.stderr,
             )
@@ -252,6 +275,8 @@ def main():
         auto_id_prefix=auto_id_prefix,
         auto_id_type_code=auto_id_type_code,
         frontmatter_overrides=overrides or None,
+        backend=args.backend,
+        arxiv_lookup=not args.no_arxiv_lookup,
     )
 
     # CLI-only output controls (not part of PipelineOptions).

{any2md-1.0.0 → any2md-1.0.2}/any2md/converters/docx.py

@@ -22,6 +22,7 @@ from any2md import pipeline
 from any2md._docling import has_docling
 from any2md.converters import add_warnings, is_quiet
 from any2md.frontmatter import SourceMeta, compose
+from any2md.heuristics import filter_organization
 from any2md.pipeline import PipelineOptions
 from any2md.utils import sanitize_filename
 
@@ -40,6 +41,7 @@ def _read_docx_metadata(docx_path: Path) -> dict[str, object]:
         "title_hint": None,
         "authors": [],
         "organization": None,
+        "produced_by": None,
         "date": None,
         "keywords": [],
     }
@@ -65,8 +67,22 @@ def _read_docx_metadata(docx_path: Path) -> dict[str, object]:
                 with z.open("docProps/app.xml") as f:
                     root = ET.parse(f).getroot()
                 company = root.findtext("ext:Company", namespaces=_NS_APP)
-                if company:
+                application = root.findtext("ext:Application", namespaces=_NS_APP)
+                # v1.0.2: Company takes priority for `organization` (real
+                # legal entity); Application is software → goes to
+                # `produced_by` via filter_organization. When Company is
+                # absent we route Application through filter_organization
+                # for both fields so a real-org Application name still
+                # populates `organization`.
+                if company and company.strip():
                     out["organization"] = company.strip()
+                    if application and application.strip():
+                        app_result = filter_organization(application.strip())
+                        out["produced_by"] = app_result.produced_by
+                elif application and application.strip():
+                    app_result = filter_organization(application.strip())
+                    out["organization"] = app_result.organization
+                    out["produced_by"] = app_result.produced_by
             except KeyError:
                 pass
     except (zipfile.BadZipFile, ET.ParseError):
@@ -114,7 +130,24 @@ def convert_docx(
         return True
 
     try:
-        if has_docling():
+        # Backend override (PipelineOptions.backend). When set, honor the
+        # caller's choice instead of the auto-select. ``pymupdf4llm`` is
+        # invalid for DOCX — fail the file with a clear error.
+        if options.backend == "pymupdf4llm":
+            print(
+                f"  FAIL: {docx_path.name} -- backend 'pymupdf4llm' is not "
+                f"valid for DOCX input (pymupdf4llm processes PDF only).",
+                file=sys.stderr,
+            )
+            return False
+
+        if options.backend == "mammoth":
+            md_text, extracted_via = _extract_via_mammoth(docx_path, options)
+            lane = "text"
+        elif options.backend == "docling":
+            md_text, extracted_via = _extract_via_docling(docx_path)
+            lane = "structured"
+        elif has_docling():
             try:
                 md_text, extracted_via = _extract_via_docling(docx_path)
                 lane = "structured"
@@ -150,6 +183,7 @@ def convert_docx(
             doc_type="docx",
             extracted_via=extracted_via,
             lane=lane,
+            produced_by=props["produced_by"],
         )
         full = compose(
             md_text, meta, options, overrides=options.frontmatter_overrides

{any2md-1.0.0 → any2md-1.0.2}/any2md/converters/pdf.py

@@ -20,6 +20,7 @@ from any2md import pipeline
 from any2md._docling import has_docling, install_hint
 from any2md.converters import add_warnings, is_quiet
 from any2md.frontmatter import SourceMeta, compose
+from any2md.heuristics import filter_organization
 from any2md.pipeline import PipelineOptions
 from any2md.utils import sanitize_filename
 
@@ -47,10 +48,13 @@ def _parse_pdf_date(raw: str | None) -> str | None:
 
 def _parse_pdf_metadata(doc: "pymupdf.Document") -> dict[str, object]:
     meta = doc.metadata or {}
+    creator_raw = (meta.get("creator") or "").strip() or None
+    org_result = filter_organization(creator_raw)
     return {
         "title_hint": (meta.get("title") or "").strip() or None,
         "authors": _parse_pdf_authors(meta.get("author")),
-        "organization": (meta.get("creator") or "").strip() or None,
+        "organization": org_result.organization,
+        "produced_by": org_result.produced_by,
         "date": _parse_pdf_date(meta.get("creationDate")),
         "keywords": [
             k.strip() for k in (meta.get("keywords") or "").split(",") if k.strip()
@@ -182,9 +186,31 @@ def convert_pdf(
         return True
 
     try:
-        use_docling = has_docling()
+        # Backend override (PipelineOptions.backend). When set, honor the
+        # caller's choice instead of the auto-select. ``mammoth`` is
+        # invalid for PDFs — fail the file with a clear error.
+        if options.backend == "mammoth":
+            print(
+                f"  FAIL: {pdf_path.name} -- backend 'mammoth' is not "
+                f"valid for PDF input (mammoth processes DOCX only).",
+                file=sys.stderr,
+            )
+            return False
+
+        if options.backend == "pymupdf4llm":
+            use_docling = False
+        elif options.backend == "docling":
+            # Same install-required behavior as --high-fidelity. If
+            # Docling isn't importable, fall through to the existing
+            # auto-select path which will error via _extract_via_docling
+            # below if Docling truly isn't usable. The CLI layer
+            # pre-checks install for both --high-fidelity and
+            # --backend docling, so this branch normally has Docling.
+            use_docling = has_docling()
+        else:
+            use_docling = has_docling()
 
-        if not use_docling and pdf_looks_complex(pdf_path):
+        if not use_docling and options.backend is None and pdf_looks_complex(pdf_path):
             install_hint()
 
         # Always extract metadata via PyMuPDF — independent of which
@@ -234,6 +260,7 @@ def convert_pdf(
             doc_type="pdf",
             extracted_via=extracted_via,
             lane=lane,
+            produced_by=props["produced_by"],
         )
         full = compose(
             md_text, meta, options, overrides=options.frontmatter_overrides