any2md 1.0.1__tar.gz → 1.0.3__tar.gz

{any2md-1.0.1 → any2md-1.0.3}/.gitignore

@@ -38,6 +38,7 @@ input/
 template/
 test_docs/
 .worktrees/
+logo/
 
 # ZERG / dev tooling
 .gsd/

{any2md-1.0.1 → any2md-1.0.3}/CHANGELOG.md

@@ -4,6 +4,127 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.0.3] — 2026-04-26
+
+Patch release. Audit on the v1.0.2-regenerated corpus surfaced two
+patterns the previous release did not handle: cover-page-titled
+documents whose first H2 stripped to empty produced an empty
+`title` field, and Docling-lane output retained malformed `|`
+table-row remnants because the lone-punctuation filter was
+text-lane-only. The lane-agnostic body cleanup that v1.0.2's
+text lane already had also now runs on Docling output.
+
+### Added
+- New lane-agnostic stage **`strip_orphan_punctuation`** — drops
+  lines containing only `|` or `>`. Extracted from T10
+  `strip_web_fragments` so it can run on the structured (Docling)
+  lane in addition to the text (trafilatura) lane.
+
+### Changed
+- `heuristics.refine_title` now walks all H2 lines line-by-line
+  and skips any whose content strips to empty after removing
+  markdown emphasis, instead of always returning the first
+  regex match. Also guards the Wikipedia-prefix strip against
+  emitting empty when the candidate is the bare prefix.
+- `pipeline/structured.py` now extends `STAGES` with
+  `strip_repeated_byline` (T9), `dedupe_toc_table` (T7),
+  `strip_cover_artifacts` (T8), and `strip_orphan_punctuation`
+  so Docling output gets the same body-cleanup pass as text-lane
+  output. T10's short-fragment heuristic stays text-lane-only —
+  it would over-strip Docling's deliberate short headings.
+- `pipeline/cleanup.py` `decode_html_entities` (C8) now loops
+  `html.unescape` until output stabilizes (max 5 iterations) so
+  double-encoded entities like `&` → `&` → `&` are
+  fully decoded. Some extractors emit doubly-encoded entities
+  that survived a single pass.
+
+### Fixes
+- **Empty-title regression**: cover-page H1 documents whose first
+  H2 contained only markdown emphasis, NBSP-equivalent unicode,
+  or where the H2 regex's `\s+` class spanned a newline into the
+  next paragraph produced `title: ""`. Walk H2 lines line-by-line
+  with a non-empty guard.
+- **Orphan-punctuation in Docling output**: malformed table-row
+  remnants (lone `|`) survived in structured-lane output because
+  T10 was text-lane-only. New `strip_orphan_punctuation` runs on
+  both lanes.
+- **Double-encoded HTML entities**: `&` left a residual
+  `&` after a single unescape pass. Loop until stable.
+
+### Tests
+- 4 new tests in `tests/unit/test_heuristics.py::TestRefineTitle`
+  covering NBSP-only H2, emphasis-only H2 (skip-and-pick-next),
+  all-empty-H2 fallthrough, and Wikipedia-prefix-only candidate.
+- New `tests/unit/pipeline/test_strip_orphan_punctuation.py`
+  covering aggressive removal, conservative no-op, and preservation
+  of lines containing other content alongside `|`/`>`.
+- New `tests/unit/pipeline/test_structured_body_cleanup_stages.py`
+  asserting the lane-agnostic stages are registered and that the
+  targeted patterns are removed from structured-lane output.
+- New `tests/unit/pipeline/test_cleanup_html_entities_double_encoded.py`
+  covering the C8 loop.
+
+## [1.0.2] — 2026-04-26
+
+Patch release. Closes issue #15 plus 8 additional quality issues
+discovered during deep investigation against four real-world inputs:
+arxiv academic paper, ISO/IEC 27002 standard, COMP4441 academic
+DOCX, and a Wikipedia article via URL.
+
+### Added
+- New `any2md/heuristics.py` module — pure functions for frontmatter
+  field refinement: `refine_title`, `refine_abstract`, `extract_authors`,
+  `filter_organization`, `arxiv_lookup`, `is_arxiv_filename`. Called
+  from `frontmatter.compose()` and from converter modules.
+- New `produced_by` extension field on `SourceMeta` and frontmatter
+  (between `extracted_via` and `pages`). Records the software that
+  produced the source file (PDF `Creator`, DOCX `Application`).
+  Distinct from `extracted_via` which records the any2md backend
+  that produced the markdown.
+- New shared cleanup stage **C8 `decode_html_entities`** — universal
+  removal of `&`, `>`, `<`, numeric entities (`—`,
+  `—`) from body. Code-block aware (skips fenced ` ``` ` blocks).
+- New text-lane stages T7-T10 (aggressive/maximum profile only):
+  - **T7 `dedupe_toc_table`** — strips table-formatted TOCs that T4's
+    text-formatted-TOC heuristic doesn't catch. Common in academic PDFs.
+  - **T8 `strip_cover_artifacts`** — drops cover-page noise (QR-code
+    blurbs, version stamps like "Third edition 2022-02") in the first
+    ~30 lines, before the first H2.
+  - **T9 `strip_repeated_byline`** — removes "Author's Contact
+    Information:" and similar lines that duplicate a byline.
+  - **T10 `strip_web_fragments`** — drops trafilatura extraction
+    fragments (orphan `|` / `>` lines, short incomplete sentences
+    surrounded by blank lines).
+- New CLI flag `--no-arxiv-lookup` to disable the arxiv API metadata
+  enrichment. Arxiv enrichment is on by default for filenames matching
+  `\d{4}\.\d{4,5}` (e.g., `2501.17755v1.pdf`); SSRF-guarded with 5s
+  timeout; failures emit non-blocking warnings.
+- Project logo and brand assets under `assets/`.
+
+### Fixes
+The eleven quality issues — see issue #15 and
+`docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md`:
+- A1: authors not extracted from PDF body byline (academic PDFs).
+- A2: abstract picked the byline / cover blurb / TOC line.
+- A3: organization populated with PDF Creator software junk.
+- B1: HTML entities leaked to body universally.
+- C1: ISO/TR titles detected as "INTERNATIONAL STANDARD" cover header.
+- C3: DOCX titles concatenated course code + project title.
+- C4: Wikipedia titles kept "Wikipedia:" namespace prefix.
+- D1: TOCs dumped as markdown tables in academic PDFs.
+- D2: trafilatura fragments leaked into web outputs.
+- D3: ISO cover-page QR-code blurb leaked into body.
+- E1: "Author's Contact Information:" line duplicated byline.
+
+### Changed
+- `frontmatter.compose()` now consults `heuristics.py` to refine
+  title / abstract / authors before YAML emission.
+- `--profile conservative` now also gates the new heuristic
+  aggressiveness (skip-lists active; speculative inferences off).
+- PDF `Creator` software values (LaTeX, acmart, Adobe InDesign,
+  Microsoft Word, etc.) no longer populate `organization`. They
+  go to the new `produced_by` field instead.
+
 ## [1.0.1] — 2026-04-26
 
 Patch release. Adds an explicit backend-selection CLI flag.

any2md-1.0.3/CODE_OF_CONDUCT.md

@@ -0,0 +1,37 @@
+# Code of Conduct
+
+This project adopts the **[Contributor Covenant version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/)**
+as its code of conduct. The full text is available at the link above and
+the canonical source is at https://www.contributor-covenant.org/.
+
+## Scope
+
+The covenant applies to all project spaces, including but not limited to:
+
+- The GitHub repository (issues, pull requests, discussions, wiki, code
+  review comments, commit messages).
+- Any community channel that explicitly identifies itself as a project
+  space (mailing list, chat room, project event).
+- Public spaces when an individual is officially representing the project.
+
+## Reporting
+
+Reports of abuse, harassment, or otherwise unacceptable behavior may be
+sent to **rock@rockcyber.com**. Reports are confidential and will be
+reviewed within 5 business days.
+
+If the report concerns the maintainer, please contact a trusted
+intermediary or use a public reporting channel of your choice.
+
+## Enforcement
+
+The maintainer is responsible for enforcement and may take any action
+deemed appropriate, ranging from a private warning to a permanent ban
+from project spaces. Enforcement actions follow the
+[Contributor Covenant Enforcement Guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/#enforcement-guidelines).
+
+## Attribution
+
+Code of conduct adapted from the
+[Contributor Covenant](https://www.contributor-covenant.org), version 2.1,
+available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/.

{any2md-1.0.1 → any2md-1.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: any2md
-Version: 1.0.1
+Version: 1.0.3
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -28,10 +28,25 @@ Provides-Extra: high-fidelity
 Requires-Dist: docling>=2.0.0; extra == 'high-fidelity'
 Description-Content-Type: text/markdown
 
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.3
+
+A patch release closing two regressions surfaced by audit on the v1.0.2-regenerated corpus, plus a Docling-lane reach for the v1.0.2 body-cleanup stages:
+
+- **Empty-title fix**: cover-page-titled documents (`# INTERNATIONAL STANDARD` etc.) whose first H2 stripped to empty (markdown emphasis only, NBSP-equivalent unicode, regex span crossing into the next paragraph) emitted `title: ""`. `heuristics.refine_title` now walks H2 lines line-by-line and skips any that strip to empty after dropping markdown emphasis. The Wikipedia-prefix strip got the same non-empty guard.
+- **Docling-lane orphan-punctuation**: lone `|` or `>` lines from Docling's malformed table parsing leaked through because T10 `strip_web_fragments` was text-lane-only by design. The orphan-punctuation portion is extracted into a new lane-agnostic stage `strip_orphan_punctuation` that runs on Docling output too. T10's trafilatura-specific short-fragment heuristic stays text-lane-only.
+- **Lane-agnostic body cleanup on Docling lane**: T7 `dedupe_toc_table`, T8 `strip_cover_artifacts`, and T9 `strip_repeated_byline` are now appended to the structured-lane `STAGES` list, so Docling output gets the same body-cleanup pass that text-lane output already had.
+- **Double-encoded HTML entities**: C8 `decode_html_entities` now loops `html.unescape` until output stabilizes (max 5 iterations) so `&amp;amp;` → `&amp;` → `&` is fully decoded.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md](docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md). v1.0.2 highlights — `any2md/heuristics.py`, `produced_by`, C8, T7–T10, `--no-arxiv-lookup` — remain available; see [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -144,6 +159,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -408,6 +424,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -442,7 +462,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.3
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -459,43 +479,73 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
+        │  T9 strip repeated      │   ◀ NEW v1.0.3
+        │     byline              │
+        │  T7 dedupe TOC table    │   ◀ NEW v1.0.3
+        │  T8 strip cover         │   ◀ NEW v1.0.3
+        │     artifacts           │
+        │  strip_orphan_punct     │   ◀ NEW v1.0.3
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │      ↳ strip_orphan_punct  ◀ split out v1.0.3
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -504,7 +554,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

{any2md-1.0.1 → any2md-1.0.3}/README.md

@@ -1,7 +1,22 @@
+<p align="center">
+  <img src="assets/logo.png" alt="any2md" width="500"/>
+</p>
+
 # any2md
 
 Convert PDFs, DOCX, HTML, URLs, and plain text into structured, machine-consumable Markdown for downstream RAG pipelines.
 
+## What's new in 1.0.3
+
+A patch release closing two regressions surfaced by audit on the v1.0.2-regenerated corpus, plus a Docling-lane reach for the v1.0.2 body-cleanup stages:
+
+- **Empty-title fix**: cover-page-titled documents (`# INTERNATIONAL STANDARD` etc.) whose first H2 stripped to empty (markdown emphasis only, NBSP-equivalent unicode, regex span crossing into the next paragraph) emitted `title: ""`. `heuristics.refine_title` now walks H2 lines line-by-line and skips any that strip to empty after dropping markdown emphasis. The Wikipedia-prefix strip got the same non-empty guard.
+- **Docling-lane orphan-punctuation**: lone `|` or `>` lines from Docling's malformed table parsing leaked through because T10 `strip_web_fragments` was text-lane-only by design. The orphan-punctuation portion is extracted into a new lane-agnostic stage `strip_orphan_punctuation` that runs on Docling output too. T10's trafilatura-specific short-fragment heuristic stays text-lane-only.
+- **Lane-agnostic body cleanup on Docling lane**: T7 `dedupe_toc_table`, T8 `strip_cover_artifacts`, and T9 `strip_repeated_byline` are now appended to the structured-lane `STAGES` list, so Docling output gets the same body-cleanup pass that text-lane output already had.
+- **Double-encoded HTML entities**: C8 `decode_html_entities` now loops `html.unescape` until output stabilizes (max 5 iterations) so `&amp;amp;` → `&amp;` → `&` is fully decoded.
+
+The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md](docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md). Per-fix detail is in the [CHANGELOG](CHANGELOG.md). v1.0.2 highlights — `any2md/heuristics.py`, `produced_by`, C8, T7–T10, `--no-arxiv-lookup` — remain available; see [docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md](docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md).
+
 ## What this is
 
 any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
@@ -114,6 +129,7 @@ source_file: "safety-alignment.docx"               # original filename (any2md e
 type: "docx"                                       # source format (any2md extension field)
 word_count: 14289                                  # any2md extension field
 extracted_via: "docling"                           # which backend produced the markdown
+produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the source (new in v1.0.2)
 ---
 ```
 
@@ -378,6 +394,10 @@ type_code = "GD"
 
 With this file in place and `--auto-id` passed, generated `document_id` values look like `CSA-2026-GD-a3f1c91d`. The resolution order, highest priority first: explicit `--meta KEY=VAL` arguments, then `--meta-file PATH` (when given), then auto-discovered `.any2md.toml`, then the tool's defaults. Full flag and config reference in [docs/cli-reference.md](docs/cli-reference.md).
 
+### Network behavior
+
+For PDFs whose filename matches the arxiv ID pattern (`\d{4}\.\d{4,5}`, e.g. `2501.17755v1.pdf`), any2md by default queries `https://export.arxiv.org/api/query` to enrich `authors`, `abstract_for_rag`, and `date` from the official arxiv metadata. The lookup is SSRF-guarded, has a 5-second timeout, and emits a non-blocking warning on any failure — conversion proceeds either way. Pass `--no-arxiv-lookup` to disable this for airgapped environments or when you want any2md to make no outbound calls. See [docs/cli-reference.md](docs/cli-reference.md#--no-arxiv-lookup) for details.
+
 ### Output verbosity and exit codes
 
 The CLI has three verbosity flags that pair with a four-value exit-code contract:
@@ -412,7 +432,7 @@ The full symptom-cause-fix triage guide, plus deeper diagnosis steps and follow-
 any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy backends (Docling), one for backends that need heavier text repair (pymupdf4llm, trafilatura, mammoth, the TXT heuristic). Both lanes converge on a shared cleanup pass, and the same `frontmatter.py` module emits YAML for every input format.
 
 ```
-                  any2md  v1.0
+                  any2md  v1.0.3
                        │
                   ┌────┴────┐
               CLI │ cli.py  │  parse args, classify URL/file/dir,
@@ -429,43 +449,73 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
         │ pymupdf4llm│ mammoth+md │       │
         └────┬───────┴────┬───────┴───┬───┘
              │            │           │
+             │  SourceMeta carries:
+             │    organization, produced_by  ◀ NEW in v1.0.2
+             │    (split via heuristics.filter_organization)
+             │
        structured-md  structured-md   text-md
              │            │           │
              └─────┬──────┴───────────┘
                    ▼
         ┌─────────────────────────┐
         │  pipeline/structured.py │   structured lane
-        │  - figure caption pass  │   (trusts layout)
-        │  - table compactor      │
-        │  - cite normalizer      │
-        │  - heading hierarchy    │
+        │  S1 figure caption pass │   (trusts layout)
+        │  S2 table compactor     │
+        │  S3 cite normalizer     │
+        │  S4 heading hierarchy   │
+        │  T9 strip repeated      │   ◀ NEW v1.0.3
+        │     byline              │
+        │  T7 dedupe TOC table    │   ◀ NEW v1.0.3
+        │  T8 strip cover         │   ◀ NEW v1.0.3
+        │     artifacts           │
+        │  strip_orphan_punct     │   ◀ NEW v1.0.3
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │     pipeline/text.py    │   text lane
-        │  - line-wrap repair     │
-        │  - dehyphenate          │
-        │  - duplicate paragraph  │
-        │  - TOC dedupe           │
-        │  - hdr/ftr strip        │
-        │  - list/code restore    │
+        │  T1 line-wrap repair    │
+        │  T2 dehyphenate         │
+        │  T9 strip repeated      │   ◀ NEW (E1)
+        │     byline              │
+        │  T3 dedupe paragraphs   │
+        │  T4 dedupe TOC block    │
+        │  T7 dedupe TOC table    │   ◀ NEW (D1)
+        │  T5 hdr/ftr strip       │
+        │  T10 strip web          │   ◀ NEW (D2)
+        │      fragments          │
+        │      ↳ strip_orphan_punct  ◀ split out v1.0.3
+        │  T6 list/code restore   │
+        │  T8 strip cover         │   ◀ NEW (D3)
+        │     artifacts           │
         └────────────┬────────────┘
                      │
         ┌────────────┴────────────┐
         │   pipeline/cleanup.py   │   shared lane
-        │  - NFC normalize        │   (always last)
-        │  - soft hyphen strip    │
-        │  - ligature normalize   │
-        │  - quote/dash normalize │
-        │  - whitespace collapse  │
-        │  - footnote marker strip│
-        │  - validate             │
+        │  C1 NFC normalize       │   (always last)
+        │  C2 soft hyphen strip   │
+        │  C3 ligature normalize  │
+        │  C4 quote/dash normalize│
+        │  C5 whitespace collapse │
+        │  C8 decode HTML entities│   ◀ NEW (B1, all profiles)
+        │  C6 footnote marker strip│
+        │  C7 validate            │
         └────────────┬────────────┘
                      │
-        ┌────────────┴────────────┐
-        │     frontmatter.py      │   SSRM-compatible
-        │  - merge user overrides │   YAML emitter
+                     │           ┌──────────────────────────┐
+                     │           │   any2md/heuristics.py   │  ◀ NEW v1.0.2
+                     │           │  - refine_title          │
+                     │           │  - refine_abstract       │
+                     │           │  - extract_authors       │
+                     │           │    (+ optional arxiv API)│
+                     │           │  - filter_organization   │
+                     │           └────────────┬─────────────┘
+                     │                        │
+                     ▼                        │
+        ┌─────────────────────────┐           │
+        │     frontmatter.py      │ ◀─────────┘   (consults heuristics
+        │  - merge user overrides │                before YAML emission)
         │  - derive title/date/…  │
+        │  - emit produced_by     │   ◀ NEW v1.0.2
         │  - compute content_hash │
         │  - emit YAML            │
         └────────────┬────────────┘
@@ -474,7 +524,7 @@ any2md uses a two-lane post-processing pipeline: one lane for layout-trustworthy
               <output>.md   ← frontmatter + cleaned body
 ```
 
-The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
+The two-lane design exists because Docling output is layout-correct (running aggressive line-wrap repair on it would corrupt tables), while pymupdf4llm/trafilatura/TXT output benefits from heavier regex repair. The shared cleanup pass runs identically for both lanes — that's the determinism boundary that makes `content_hash` reproducible. v1.0.2 added a leaf `heuristics.py` module that `frontmatter.compose()` consults to refine the title, abstract, authors, and organization fields before YAML emission; new pipeline stages C8 and T7–T10 address concrete artifacts found in academic and standards-document PDFs. Full stage-by-stage contract reference, plus contributor guidance for adding new converters or pipeline stages, lives in [docs/architecture.md](docs/architecture.md).
 
 ## Migrating from v0.7
 

any2md-1.0.3/SECURITY.md

@@ -0,0 +1,83 @@
+# Security Policy
+
+## Supported versions
+
+| Version | Status |
+|---|---|
+| 1.0.x   | ✅ Supported (current stable). Patch releases for security and quality fixes are published as needed. |
+| 0.7.x   | ⚠️ Receives critical security fixes only until 2026-10-26 (six months after the v1.0.0 GA on 2026-04-26). |
+| < 0.7   | ❌ Not supported. Please upgrade. |
+
+## Reporting a vulnerability
+
+**Please do not file public issues for security vulnerabilities.** Public
+disclosure before a fix is published puts users at risk.
+
+Email security disclosures to **rock@rockcyber.com** with the subject line
+`any2md security disclosure`. Encrypted email is welcome — request a
+public key in your initial message and one will be provided.
+
+Include in your report:
+
+- A description of the issue and the affected component.
+- Affected version(s) — e.g., `pip show any2md` output, or commit SHA.
+- Reproduction steps (smallest possible repro is ideal).
+- Any proof-of-concept (please redact sensitive data).
+- Whether you intend to publish your own write-up; if so, a proposed
+  embargo date so we can coordinate.
+
+### Response timeline
+
+- **Acknowledgement** within 5 business days.
+- **Triage and severity assessment** within 10 business days.
+- **Critical issues** (remote code execution, data exfiltration via
+  crafted input, supply-chain compromise of a published wheel): patched
+  and released within 30 days of confirmation, with a coordinated public
+  advisory.
+- **High** (denial-of-service, information disclosure with limited blast
+  radius): patched within 60 days.
+- **Medium / low** issues are tracked in private until a fix lands in a
+  scheduled release, then disclosed publicly via the GitHub Security
+  Advisories page.
+
+## Scope
+
+In scope:
+
+- The `any2md` library and CLI.
+- The published wheels and sdists on PyPI.
+- The release pipeline (`.github/workflows/publish.yml`) and any
+  side-effects of running the workflow.
+- The default conversion pipeline (PDF / DOCX / HTML / URL / TXT) and
+  the optional Docling backend integration.
+- The arxiv API enrichment path (network call + XML parsing).
+- The SSRF guards in the URL fetcher.
+
+Out of scope:
+
+- Vulnerabilities in third-party dependencies (please file directly with
+  the upstream — Docling, PyMuPDF, mammoth, trafilatura, BeautifulSoup,
+  lxml, etc.). We monitor upstream advisories via Dependabot.
+- Issues that require local filesystem access already granted to the
+  running Python process.
+- Local timing attacks against your own machine.
+- Content correctness issues in converted Markdown (those are
+  conversion-quality reports — please use the
+  [Conversion quality issue template](.github/ISSUE_TEMPLATE/conversion_quality.md)
+  instead).
+
+## Hardening reference
+
+If you operate any2md in a hardened environment:
+
+- Pass `--no-arxiv-lookup` to disable the arxiv API call.
+- Pass `--max-file-size` with a budget appropriate for your workload.
+- Run conversions in a sandbox that cannot reach private networks
+  (the SSRF guard validates resolved IPs, but defense in depth is best).
+- Pin `any2md` and its extras (`any2md[high-fidelity]==<exact-version>`)
+  to a verified version and audit the lockfile.
+
+## Disclosure history
+
+No publicly disclosed vulnerabilities to date. Future advisories will
+appear at https://github.com/rocklambros/any2md/security/advisories.

{any2md-1.0.1 → any2md-1.0.3}/any2md/__init__.py

@@ -1,3 +1,3 @@
 """Convert PDF, DOCX, HTML, and TXT files to LLM-optimized Markdown."""
 
-__version__ = "1.0.1"
+__version__ = "1.0.3"

{any2md-1.0.1 → any2md-1.0.3}/any2md/cli.py

@@ -65,11 +65,7 @@ def _deep_merge_overrides(
     """
     out = dict(base)
     for key, val in overrides.items():
-        if (
-            key in out
-            and isinstance(out[key], dict)
-            and isinstance(val, dict)
-        ):
+        if key in out and isinstance(out[key], dict) and isinstance(val, dict):
             out[key] = _deep_merge_overrides(out[key], val)
         else:
             out[key] = val
@@ -132,6 +128,14 @@ def main():
         action="store_true",
         help="Force the Docling backend (PDF/DOCX). Exit 1 if not installed.",
     )
+    parser.add_argument(
+        "--no-arxiv-lookup",
+        action="store_true",
+        help="Disable the arxiv API lookup that enriches frontmatter for "
+        "PDFs whose filename matches the arxiv ID pattern (\\d{4}.\\d{4,5}). "
+        "Useful in airgapped envs or when you don't want any2md to make "
+        "network calls. Default: arxiv lookup is enabled.",
+    )
     parser.add_argument(
         "--backend",
         choices=("docling", "pymupdf4llm", "mammoth"),
@@ -216,9 +220,7 @@ def main():
 
     if args.meta_file is not None:
         if not args.meta_file.is_file():
-            print(
-                f"Error: --meta-file not found: {args.meta_file}", file=sys.stderr
-            )
+            print(f"Error: --meta-file not found: {args.meta_file}", file=sys.stderr)
             sys.exit(1)
         cfg = load_toml(args.meta_file)
         overrides = _deep_merge_overrides(overrides, extract_meta_overrides(cfg))
@@ -245,6 +247,7 @@ def main():
         or args.backend == "docling"
     ):
         from any2md._docling import has_docling, INSTALL_HINT_MSG
+
         if not has_docling():
             print(
                 f"  ERROR: Docling required for --high-fidelity / --ocr-figures / --save-images / --backend docling.\n"
@@ -268,6 +271,7 @@ def main():
         auto_id_type_code=auto_id_type_code,
         frontmatter_overrides=overrides or None,
         backend=args.backend,
+        arxiv_lookup=not args.no_arxiv_lookup,
     )
 
     # CLI-only output controls (not part of PipelineOptions).