any2md 1.0.1__tar.gz → 1.0.2__tar.gz

{any2md-1.0.1 → any2md-1.0.2}/.gitignore

@@ -38,6 +38,7 @@ input/
 template/
 test_docs/
 .worktrees/
+logo/
 
 # ZERG / dev tooling
 .gsd/

{any2md-1.0.1 → any2md-1.0.2}/CHANGELOG.md

@@ -4,6 +4,67 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.0.2] — 2026-04-26
+
+Patch release. Closes issue #15 plus 8 additional quality issues
+discovered during deep investigation against four real-world inputs:
+arxiv academic paper, ISO/IEC 27002 standard, COMP4441 academic
+DOCX, and a Wikipedia article via URL.
+
+### Added
+- New `any2md/heuristics.py` module — pure functions for frontmatter
+  field refinement: `refine_title`, `refine_abstract`, `extract_authors`,
+  `filter_organization`, `arxiv_lookup`, `is_arxiv_filename`. Called
+  from `frontmatter.compose()` and from converter modules.
+- New `produced_by` extension field on `SourceMeta` and frontmatter
+  (between `extracted_via` and `pages`). Records the software that
+  produced the source file (PDF `Creator`, DOCX `Application`).
+  Distinct from `extracted_via` which records the any2md backend
+  that produced the markdown.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal
+  removal of `&`, `>`, `<`, numeric entities (`—`,
+  `—`) from body. Code-block aware (skips fenced ` ``` ` blocks).
+- New text-lane stages T7-T10 (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that T4's
+    text-formatted-TOC heuristic doesn't catch. Common in academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code
+    blurbs, version stamps like "Third edition 2022-02") in the first
+    ~30 lines, before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact
+    Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction
+    fragments (orphan `|` / `>` lines, short incomplete sentences
+    surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup` to disable the arxiv API metadata
+  enrichment. Arxiv enrichment is on by default for filenames matching
+  `\d{4}\.\d{4,5}` (e.g., `2501.17755v1.pdf`); SSRF-guarded with 5s
+  timeout; failures emit non-blocking warnings.
+- Project logo and brand assets under `assets/`.
+
+### Fixes
+The eleven quality issues — see issue #15 and
+`docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md`:
+- A1: authors not extracted from PDF body byline (academic PDFs).
+- A2: abstract picked the byline / cover blurb / TOC line.
+- A3: organization populated with PDF Creator software junk.
+- B1: HTML entities leaked to body universally.
+- C1: ISO/TR titles detected as "INTERNATIONAL STANDARD" cover header.
+- C3: DOCX titles concatenated course code + project title.
+- C4: Wikipedia titles kept "Wikipedia:" namespace prefix.
+- D1: TOCs dumped as markdown tables in academic PDFs.
+- D2: trafilatura fragments leaked into web outputs.
+- D3: ISO cover-page QR-code blurb leaked into body.
+- E1: "Author's Contact Information:" line duplicated byline.
+
+### Changed
+- `frontmatter.compose()` now consults `heuristics.py` to refine
+  title / abstract / authors before YAML emission.
+- `--profile conservative` now also gates the new heuristic
+  aggressiveness (skip-lists active; speculative inferences off).
+- PDF `Creator` software values (LaTeX, acmart, Adobe InDesign,
+  Microsoft Word, etc.) no longer populate `organization`. They
+  go to the new `produced_by` field instead.
+
 ## [1.0.1] — 2026-04-26
 
 Patch release. Adds an explicit backend-selection CLI flag.

{any2md-1.0.1 → any2md-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: any2md
-Version: 1.0.1
+Version: 1.0.2
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -28,10 +28,30 @@ Provides-Extra: high-fidelity
 Requires-Dist: docling>=2.0.0; extra == 'high-fidelity'
 Description-Content-Type: text/markdown
 
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.2
+
+A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+
+- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
+- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&amp;`, `&lt;`, `&gt;`, and numeric entities (`&#x2014;`, `&#8212;`) from body text. Code-block aware.
+- Four new text-lane stages (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -144,6 +164,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -408,6 +429,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -442,7 +467,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.2
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -459,43 +484,66 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -504,7 +552,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

{any2md-1.0.1 → any2md-1.0.2}/README.md

@@ -1,7 +1,27 @@
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.2
+
+A patch release focused on frontmatter and body-content quality, validated against four representative inputs (an arxiv academic PDF, an ISO/IEC standard, an academic DOCX, a Wikipedia URL). Eleven distinct artifacts are addressed:
+
+- New `any2md/heuristics.py` module — pure-functions library that `frontmatter.compose()` consults to refine `title`, `abstract_for_rag`, `authors`, and `organization` before emission.
+- New `produced_by` extension field, sitting between `extracted_via` and `pages`. Records the software that produced the source file (PDF `Creator`, DOCX `Application`). Distinct from `extracted_via`, which records the any2md backend.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal removal of `&amp;`, `&lt;`, `&gt;`, and numeric entities (`&#x2014;`, `&#8212;`) from body text. Code-block aware.
+- Four new text-lane stages (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that the existing T4 text-formatted heuristic doesn't catch. Common on academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code blurbs, "Third edition 2022-02"-style version stamps) that appears before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction fragments (orphan `|` and `>` lines, short incomplete sentences surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup`. Arxiv enrichment for filenames matching `\d{4}\.\d{4,5}` is **on by default**; this flag disables it for airgapped environments.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -114,6 +134,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -378,6 +399,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -412,7 +437,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.2
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -429,43 +454,66 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -474,7 +522,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

{any2md-1.0.1 → any2md-1.0.2}/any2md/__init__.py

@@ -1,3 +1,3 @@
 """Convert PDF, DOCX, HTML, and TXT files to LLM-optimized Markdown."""
 
-__version__ = "1.0.1"
+__version__ = "1.0.2"

{any2md-1.0.1 → any2md-1.0.2}/any2md/cli.py

@@ -132,6 +132,14 @@ def main():
         action="store_true",
         help="Force the Docling backend (PDF/DOCX). Exit 1 if not installed.",
     )
+    parser.add_argument(
+        "--no-arxiv-lookup",
+        action="store_true",
+        help="Disable the arxiv API lookup that enriches frontmatter for "
+             "PDFs whose filename matches the arxiv ID pattern (\\d{4}.\\d{4,5}). "
+             "Useful in airgapped envs or when you don't want any2md to make "
+             "network calls. Default: arxiv lookup is enabled.",
+    )
     parser.add_argument(
         "--backend",
         choices=("docling", "pymupdf4llm", "mammoth"),
@@ -268,6 +276,7 @@ def main():
         auto_id_type_code=auto_id_type_code,
         frontmatter_overrides=overrides or None,
         backend=args.backend,
+        arxiv_lookup=not args.no_arxiv_lookup,
     )
 
     # CLI-only output controls (not part of PipelineOptions).

{any2md-1.0.1 → any2md-1.0.2}/any2md/converters/docx.py

@@ -22,6 +22,7 @@ from any2md import pipeline
 from any2md._docling import has_docling
 from any2md.converters import add_warnings, is_quiet
 from any2md.frontmatter import SourceMeta, compose
+from any2md.heuristics import filter_organization
 from any2md.pipeline import PipelineOptions
 from any2md.utils import sanitize_filename
 
@@ -40,6 +41,7 @@ def _read_docx_metadata(docx_path: Path) -> dict[str, object]:
         "title_hint": None,
         "authors": [],
         "organization": None,
+        "produced_by": None,
         "date": None,
         "keywords": [],
     }
@@ -65,8 +67,22 @@ def _read_docx_metadata(docx_path: Path) -> dict[str, object]:
                 with z.open("docProps/app.xml") as f:
                     root = ET.parse(f).getroot()
                 company = root.findtext("ext:Company", namespaces=_NS_APP)
-                if company:
+                application = root.findtext("ext:Application", namespaces=_NS_APP)
+                # v1.0.2: Company takes priority for `organization` (real
+                # legal entity); Application is software → goes to
+                # `produced_by` via filter_organization. When Company is
+                # absent we route Application through filter_organization
+                # for both fields so a real-org Application name still
+                # populates `organization`.
+                if company and company.strip():
                     out["organization"] = company.strip()
+                    if application and application.strip():
+                        app_result = filter_organization(application.strip())
+                        out["produced_by"] = app_result.produced_by
+                elif application and application.strip():
+                    app_result = filter_organization(application.strip())
+                    out["organization"] = app_result.organization
+                    out["produced_by"] = app_result.produced_by
             except KeyError:
                 pass
     except (zipfile.BadZipFile, ET.ParseError):
@@ -167,6 +183,7 @@ def convert_docx(
             doc_type="docx",
             extracted_via=extracted_via,
             lane=lane,
+            produced_by=props["produced_by"],
         )
         full = compose(
             md_text, meta, options, overrides=options.frontmatter_overrides

{any2md-1.0.1 → any2md-1.0.2}/any2md/converters/pdf.py

@@ -20,6 +20,7 @@ from any2md import pipeline
 from any2md._docling import has_docling, install_hint
 from any2md.converters import add_warnings, is_quiet
 from any2md.frontmatter import SourceMeta, compose
+from any2md.heuristics import filter_organization
 from any2md.pipeline import PipelineOptions
 from any2md.utils import sanitize_filename
 
@@ -47,10 +48,13 @@ def _parse_pdf_date(raw: str | None) -> str | None:
 
 def _parse_pdf_metadata(doc: "pymupdf.Document") -> dict[str, object]:
     meta = doc.metadata or {}
+    creator_raw = (meta.get("creator") or "").strip() or None
+    org_result = filter_organization(creator_raw)
     return {
         "title_hint": (meta.get("title") or "").strip() or None,
         "authors": _parse_pdf_authors(meta.get("author")),
-        "organization": (meta.get("creator") or "").strip() or None,
+        "organization": org_result.organization,
+        "produced_by": org_result.produced_by,
         "date": _parse_pdf_date(meta.get("creationDate")),
         "keywords": [
             k.strip() for k in (meta.get("keywords") or "").split(",") if k.strip()
@@ -256,6 +260,7 @@ def convert_pdf(
             doc_type="pdf",
             extracted_via=extracted_via,
             lane=lane,
+            produced_by=props["produced_by"],
         )
         full = compose(
             md_text, meta, options, overrides=options.frontmatter_overrides

{any2md-1.0.1 → any2md-1.0.2}/any2md/frontmatter.py

@@ -15,6 +15,7 @@ from dataclasses import dataclass
 from datetime import date as _date_cls
 from typing import Any, Literal
 
+from any2md import heuristics
 from any2md.pipeline import Lane, PipelineOptions
 
 
@@ -62,6 +63,11 @@ class SourceMeta:
         "trafilatura", "trafilatura+bs4_fallback", "heuristic",
     ]
     lane: Lane
+    # v1.0.2: software/tool that produced the source (PDF Creator field for
+    # PDFs, Application field for DOCX). Distinct from `organization`, which
+    # is the human/legal entity. Default None to keep existing call sites
+    # working without churn.
+    produced_by: str | None = None
 
 
 def estimate_tokens(body: str) -> int:
@@ -185,11 +191,33 @@ def _build_fields(
     can tell "absent" apart from "blank".
     """
     fallback = meta.source_file or meta.source_url or "untitled"
-    title = derive_title(body, meta.title_hint, fallback)
+    title_raw = derive_title(body, meta.title_hint, fallback)
+    arxiv_id = heuristics.is_arxiv_filename(meta.source_file or "")
+    title = heuristics.refine_title(
+        title_raw, body,
+        source_url=meta.source_url,
+        profile=options.profile,
+    )
     content_hash = compute_content_hash(body)
     token_est = estimate_tokens(body)
     chunk_level = recommend_chunk_level(body)
-    abstract = extract_abstract(body) if token_est >= 500 else None
+    abstract_raw = extract_abstract(body) if token_est >= 500 else None
+    abstract = (
+        heuristics.refine_abstract(abstract_raw, body, profile=options.profile)
+        if token_est >= 500
+        else None
+    )
+    # Author chain — only invoke heuristics when meta.authors is empty.
+    # Caller-supplied authors (e.g. from extractor metadata) take priority.
+    authors = list(meta.authors)
+    if not authors:
+        authors = heuristics.extract_authors(
+            body,
+            title_hint=meta.title_hint,
+            arxiv_id=arxiv_id,
+            arxiv_lookup_enabled=options.arxiv_lookup,
+            profile=options.profile,
+        )
     today = _date_cls.today().isoformat()
     fm_date = meta.date or today
 
@@ -210,7 +238,7 @@ def _build_fields(
     fields["status"] = "draft"
     fields["document_type"] = ""
     fields["content_domain"] = []
-    fields["authors"] = list(meta.authors)
+    fields["authors"] = authors
     fields["organization"] = meta.organization or ""
     fields["generation_metadata"] = {"authored_by": "unknown"}
     fields["content_hash"] = content_hash
@@ -227,6 +255,8 @@ def _build_fields(
         fields["source_url"] = meta.source_url
     fields["type"] = meta.doc_type           # v0.7-compat field (spec §3.2)
     fields["extracted_via"] = meta.extracted_via  # v1.0 provenance extension
+    if meta.produced_by is not None:
+        fields["produced_by"] = meta.produced_by  # v1.0.2 extension (spec §6.1)
     if meta.pages is not None:
         fields["pages"] = meta.pages
     if meta.word_count is not None: