any2md 1.0.6__tar.gz → 1.1.0__tar.gz

{any2md-1.0.6 → any2md-1.1.0}/CHANGELOG.md

@@ -4,6 +4,108 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [1.1.0] — 2026-05-04
+
+Performance and library-mode hygiene release.
+
+### Added
+
+- `any2md.release_models()` — imperative escape hatch to free
+  cached Docling `DocumentConverter` instances. Stability:
+  experimental for v1.1.0.
+- `any2md.docling_session()` — contextmanager (preferred shape
+  for library use). Stability: experimental for v1.1.0.
+- `ANY2MD_DOCLING_CACHE=0` env-var to disable the persistent
+  converter cache (per-call construction, equivalent to v1.0.x
+  behavior).
+- `__all__` declared on `any2md/__init__.py` (formalizes the
+  public surface).
+- New benchmark `scripts/bench_docling_cache.py` for maintainer
+  pre-release verification.
+
+### Changed
+
+- `DocumentConverter` is now constructed once per process and
+  reused across all files in a batch (was: once per file). New
+  one-time stderr line `Loading Docling models (one-time)...`
+  printed on first PDF/DOCX in non-quiet TTY runs.
+- `.github/workflows/ci.yml` now runs `pytest tests/unit` on each
+  push and PR (was: lint + import smoke + pip-audit only).
+
+### Performance
+
+- Measured: CPU first-call ~2.47s (model load), warm calls ~0.87s
+  (vs prior ~2.47s per file in v1.0.x). Mac MPS first-call ~2.77s,
+  warm ~0.14s. RSS floor +535 MB sustained for the process lifetime
+  (see `docs/troubleshooting.md` for `release_models()` mitigation).
+
+### Migration
+
+- None required for CLI users. The `Loading Docling models...`
+  stderr line is the only user-visible default-output change.
+- Library/embedder users holding any2md across long-running
+  processes should call `any2md.release_models()` or use
+  `with any2md.docling_session():` to free model state between
+  workloads.
+- Test/instrumentation code that monkey-patches
+  `docling.document_converter.DocumentConverter` AFTER any2md's
+  first call will see no effect on subsequent calls (cache returns
+  the pre-patch instance). Workaround: call `release_models()`
+  after applying the patch, or set `ANY2MD_DOCLING_CACHE=0` in the
+  test environment.
+
+### Stability notice
+
+The new public APIs (`release_models`, `docling_session`) are
+marked experimental for v1.1.0; their shape may change in
+subsequent v1.x releases without a major bump until promoted to
+stable in a later release.
+
+### Known limitations
+
+- HF Hub outages during cold-start now cause N retries instead of
+  one silent fallback (the v1.0.x failure-cache behavior was
+  rejected on quality grounds). Bounded retry-with-backoff
+  deferred to a future hardening PR.
+- Forking a process (`multiprocessing` with `fork` start method)
+  after the first `convert()` call inherits torch/CUDA state
+  unsafely regardless of cache. Use
+  `multiprocessing.set_start_method("spawn")` or call
+  `release_models()` before fork.
+- `os.register_at_fork` is POSIX-only; on Windows the fork-reset
+  path is dead code (no fork on Windows). Cache works normally
+  on Windows.
+
+## [1.0.7] — 2026-04-29
+
+Branding/rebrand release. The frontmatter contract that any2md emits is
+now described as "SAGE-compatible" (Security Analysis and Guidance
+Exchange) — the upstream specification was renamed from SSRM (Structured
+Security Reasoning Markdown) to SAGE. No functional or schema changes:
+field names, ordering, controlled vocabularies, and `content_hash`
+derivation are byte-identical to v1.0.6. Existing v1.0.6 outputs remain
+valid SAGE-compatible documents without modification.
+
+### Changed
+
+- README, CHANGELOG, and all of `docs/` updated to reference SAGE /
+  Security Analysis and Guidance Exchange.
+- `--auto-id` CLI help text now reads "SAGE-conformant document_id"
+  (previously "SSRM-conformant"). Behavior unchanged.
+- Module/function docstrings in `any2md/frontmatter.py`,
+  `any2md/converters/txt.py`, and `any2md/pipeline/cleanup.py` updated
+  to reference SAGE §3 / §5.1 (same section numbers as the upstream
+  SSRM v1.0-RC1 spec).
+- Two test functions renamed for naming consistency
+  (`test_txt_end_to_end_writes_sage_compat_output`,
+  `test_compose_emits_required_sage_fields`). Pytest auto-discovery is
+  unaffected; assertions unchanged.
+
+### Migration
+
+None required. Public API, CLI flags, frontmatter shape, and
+`content_hash` outputs are all preserved bit-for-bit.
+
 ## [1.0.6] — 2026-04-27
 
 Security hardening release. Closes 8 actionable findings from the
@@ -318,7 +420,7 @@ Patch release. Adds an explicit backend-selection CLI flag.
 First stable release of any2md v1.0. Validated end-to-end against
 real-world documents (a 164-page PDF technical standard via Docling,
 a multi-page academic DOCX, and a Wikipedia article via trafilatura).
-The output contract — SSRM-compatible frontmatter, deterministic
+The output contract — SAGE-compatible frontmatter, deterministic
 `content_hash`, NFC + LF body normalization — is now stable. Downstream
 consumers can rely on this shape.
 
@@ -340,7 +442,7 @@ Phase 4: configuration, polish, and the v1.0 documentation set.
 - `--meta-file PATH` flag plus auto-discovery of `.any2md.toml` (walks up
   from cwd). The `[meta]` table supplies frontmatter overrides; the
   `[document_id]` table supplies `--auto-id` prefix and type code.
-- `--auto-id` flag — generates an SSRM-conformant `document_id` as
+- `--auto-id` flag — generates an SAGE-conformant `document_id` as
   `{PREFIX}-{YYYY}-{TYPE}-{SHA8}`. Defaults to `LOCAL`/`DOC`; override via
   the `[document_id]` table in `.any2md.toml`.
 - `--strict` flag — promotes pipeline validation warnings (heading
@@ -359,7 +461,7 @@ Phase 4: configuration, polish, and the v1.0 documentation set.
 - Comprehensive documentation set:
   - `README.md` rewritten with educational tone — every command preceded
     by why you'd run it and followed by what you'll see.
-  - `docs/output-format.md` — SSRM-compatible field reference and the
+  - `docs/output-format.md` — SAGE-compatible field reference and the
     `content_hash` recomputation recipe.
   - `docs/cli-reference.md` — flag-by-flag reference with use cases and a
     worked-example matrix.
@@ -462,14 +564,14 @@ any2md transparently falls back to pymupdf4llm (PDF) and mammoth (DOCX).
 
 First prerelease of any2md v1.0. Phase 1 of 5: foundation only — no Docling
 backend yet, no new CLI flags. Output frontmatter has been rewritten to be
-SSRM-compatible (Structured Security Reasoning Markdown) — this is a
+SAGE-compatible (Security Analysis and Guidance Exchange) — this is a
 breaking change for downstream consumers parsing v0.7 output. See
 `docs/superpowers/specs/2026-04-26-any2md-v1-design.md` for the full
 v1.0 design and `docs/superpowers/plans/2026-04-26-any2md-v1-phase1-foundation.md`
 for this phase's task plan.
 
 ### Added
-- New `any2md/frontmatter.py` module — SSRM-compatible YAML emitter with the
+- New `any2md/frontmatter.py` module — SAGE-compatible YAML emitter with the
   `SourceMeta` dataclass, deterministic `compute_content_hash`,
   `estimate_tokens`, `recommend_chunk_level`, `extract_abstract`, and
   `derive_title` helpers, plus the central `compose()` function (the only
@@ -495,7 +597,7 @@ for this phase's task plan.
   `reportlab`, `ruff`.
 
 ### Changed
-- **BREAKING:** Output frontmatter shape is now SSRM-compatible. New required
+- **BREAKING:** Output frontmatter shape is now SAGE-compatible. New required
   fields: `document_id` (empty string by default — opt-in via future
   `--auto-id`), `version`, `date`, `status: "draft"`, `document_type` (empty
   for non-security documents), `content_domain` (empty array),

{any2md-1.0.6 → any2md-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: any2md
-Version: 1.0.6
+Version: 1.1.0
 Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
 Project-URL: Homepage, https://github.com/rocklambros/any2md
 Project-URL: Issues, https://github.com/rocklambros/any2md/issues
@@ -25,6 +25,7 @@ Requires-Dist: urllib3<3,>=2.6.3
 Provides-Extra: dev
 Requires-Dist: pytest-snapshot>=0.9.0; extra == 'dev'
 Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: pyyaml>=6.0; extra == 'dev'
 Requires-Dist: reportlab>=4.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.6.0; extra == 'dev'
 Provides-Extra: high-fidelity
@@ -52,7 +53,7 @@ The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-emp
 
 ## What this is
 
-any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
+any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SAGE-compatible (Security Analysis and Guidance Exchange — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
 
 ## Quick start
 
@@ -144,8 +145,8 @@ document_id: ""                                    # empty unless you pass --aut
 version: "1"                                       # "1" for source files without an embedded version
 date: "2026-03-15"                                 # docx core props "modified" → ISO-8601
 status: "draft"                                    # always "draft" for converted documents
-document_type: ""                                  # SSRM controlled vocab — empty for non-security docs
-content_domain: []                                 # SSRM controlled vocab — empty array when unknown
+document_type: ""                                  # SAGE controlled vocab — empty for non-security docs
+content_domain: []                                 # SAGE controlled vocab — empty array when unknown
 authors:                                           # extracted from docProps/core.xml dc:creator
   - "Rock Lambros"
 organization: ""                                   # docProps/app.xml Company; empty when absent
@@ -166,7 +167,7 @@ produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the
 ---
 ```
 
-Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SSRM contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
+Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SAGE contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
 
 ### Before / after
 
@@ -360,9 +361,18 @@ any2md -f -r ./corpus/
 
 You'll see the OK lines as if the conversion were fresh; previously-written outputs are overwritten.
 
+> **v1.1.0+:** when processing PDFs or DOCX files via the Docling
+> backend, the converter (and its loaded model weights) is constructed
+> once on the first PDF/DOCX in an invocation and reused for all
+> subsequent files; subsequent files extract in ~0.5–1s rather than
+> re-paying the ~2.5s cold-load tax. Library users embedding any2md
+> in long-running processes should call `any2md.release_models()` or
+> use `with any2md.docling_session():` — see
+> `docs/troubleshooting.md`.
+
 ## The output format
 
-Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SSRM-compatible** — it matches the field names, types, and ordering of the Structured Security Reasoning Markdown specification, but values aren't required to match SSRM's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
+Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SAGE-compatible** — it matches the field names, types, and ordering of the Security Analysis and Guidance Exchange specification, but values aren't required to match SAGE's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
 
 ### Field auto-fill summary
 
@@ -407,7 +417,7 @@ You'll see the OK line as usual. The frontmatter of `Text/paper.md` will have `o
 
 ### `.any2md.toml`
 
-You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SSRM-conforming outputs:
+You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SAGE-conforming outputs:
 
 ```toml
 [meta]
@@ -561,7 +571,7 @@ The two-lane design exists because Docling output is layout-correct (running agg
 
 ## Migrating from v0.7
 
-v1.0 emits SSRM-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
+v1.0 emits SAGE-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
 
 ## Security
 

{any2md-1.0.6 → any2md-1.1.0}/README.md

@@ -19,7 +19,7 @@ The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-emp
 
 ## What this is
 
-any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SSRM-compatible (Structured Security Reasoning Markdown — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
+any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SAGE-compatible (Security Analysis and Guidance Exchange — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
 
 ## Quick start
 
@@ -111,8 +111,8 @@ document_id: ""                                    # empty unless you pass --aut
 version: "1"                                       # "1" for source files without an embedded version
 date: "2026-03-15"                                 # docx core props "modified" → ISO-8601
 status: "draft"                                    # always "draft" for converted documents
-document_type: ""                                  # SSRM controlled vocab — empty for non-security docs
-content_domain: []                                 # SSRM controlled vocab — empty array when unknown
+document_type: ""                                  # SAGE controlled vocab — empty for non-security docs
+content_domain: []                                 # SAGE controlled vocab — empty array when unknown
 authors:                                           # extracted from docProps/core.xml dc:creator
   - "Rock Lambros"
 organization: ""                                   # docProps/app.xml Company; empty when absent
@@ -133,7 +133,7 @@ produced_by: "Microsoft® Word for Microsoft 365"   # software that produced the
 ---
 ```
 
-Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SSRM contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
+Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SAGE contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
 
 ### Before / after
 
@@ -327,9 +327,18 @@ any2md -f -r ./corpus/
 
 You'll see the OK lines as if the conversion were fresh; previously-written outputs are overwritten.
 
+> **v1.1.0+:** when processing PDFs or DOCX files via the Docling
+> backend, the converter (and its loaded model weights) is constructed
+> once on the first PDF/DOCX in an invocation and reused for all
+> subsequent files; subsequent files extract in ~0.5–1s rather than
+> re-paying the ~2.5s cold-load tax. Library users embedding any2md
+> in long-running processes should call `any2md.release_models()` or
+> use `with any2md.docling_session():` — see
+> `docs/troubleshooting.md`.
+
 ## The output format
 
-Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SSRM-compatible** — it matches the field names, types, and ordering of the Structured Security Reasoning Markdown specification, but values aren't required to match SSRM's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
+Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SAGE-compatible** — it matches the field names, types, and ordering of the Security Analysis and Guidance Exchange specification, but values aren't required to match SAGE's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
 
 ### Field auto-fill summary
 
@@ -374,7 +383,7 @@ You'll see the OK line as usual. The frontmatter of `Text/paper.md` will have `o
 
 ### `.any2md.toml`
 
-You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SSRM-conforming outputs:
+You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SAGE-conforming outputs:
 
 ```toml
 [meta]
@@ -528,7 +537,7 @@ The two-lane design exists because Docling output is layout-correct (running agg
 
 ## Migrating from v0.7
 
-v1.0 emits SSRM-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
+v1.0 emits SAGE-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
 
 ## Security
 

any2md-1.1.0/any2md/__init__.py

@@ -0,0 +1,11 @@
+"""any2md — convert documents to LLM-friendly Markdown."""
+
+from any2md._docling_cache import docling_session, release_models
+
+__version__ = "1.1.0"
+
+__all__ = [
+    "__version__",
+    "docling_session",
+    "release_models",
+]