any2md 1.0.6__tar.gz → 1.0.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {any2md-1.0.6 → any2md-1.0.7}/CHANGELOG.md +36 -6
- {any2md-1.0.6 → any2md-1.0.7}/PKG-INFO +8 -8
- {any2md-1.0.6 → any2md-1.0.7}/README.md +7 -7
- {any2md-1.0.6 → any2md-1.0.7}/any2md/__init__.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/any2md/cli.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/any2md/converters/txt.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/any2md/frontmatter.py +6 -6
- {any2md-1.0.6 → any2md-1.0.7}/any2md/pipeline/cleanup.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/docs/architecture.md +5 -5
- {any2md-1.0.6 → any2md-1.0.7}/docs/cli-reference.md +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/docs/output-format.md +19 -19
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1-phase1-foundation.md +19 -19
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1-phase4-polish-config-docs.md +11 -11
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/specs/2026-04-26-any2md-v1-design.md +25 -25
- {any2md-1.0.6 → any2md-1.0.7}/docs/upgrading-from-0.7.md +9 -9
- {any2md-1.0.6 → any2md-1.0.7}/scripts/validate-phase1.sh +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_txt.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_content_hash.py +1 -1
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_frontmatter_compose.py +2 -2
- {any2md-1.0.6 → any2md-1.0.7}/.gitignore +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/CODE_OF_CONDUCT.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/CONTRIBUTING.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/LICENSE +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/SECURITY.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/__main__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/_docling.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/_http.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/config.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/converters/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/converters/docx.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/converters/html.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/converters/pdf.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/heuristics.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/pipeline/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/pipeline/structured.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/pipeline/text.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/utils.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/any2md/validators.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/assets/logo-original.jpeg +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/assets/logo.png +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/assets/social-preview.jpg +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1-phase2-docling-backend.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1-phase3-figures-ocr-text-lane.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1.0.2-implementation.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-issue-17-toc-followups.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-27-v1.0.6-security-hardening.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/specs/2026-04-26-any2md-v1.0.2-design.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/specs/2026-04-26-issue-17-toc-followups-design.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/specs/2026-04-26-v1.0.3-empty-title-orphan-punct-design.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/specs/2026-04-27-v1.0.6-security-hardening-design.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/docs/troubleshooting.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/pyproject.toml +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/requirements.txt +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/scripts/audit-outputs.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_backend.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_existing_flags.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_exit_codes.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_high_fidelity.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_meta.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_no_arxiv_lookup.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_ocr_save_images.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_quiet_verbose.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/cli/test_cli_strict.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/conftest.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/docs/ligatures_and_softhyphens.txt +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/docs/multi_column.pdf +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/docs/table_heavy.docx +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/docs/web_page.html +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/make_fixtures.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/ligatures_and_softhyphens.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/multi_column.docling.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/multi_column.fallback.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/table_heavy.docling.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/table_heavy.fallback.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/fixtures/snapshots/web_page.md +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_docx_backend_override.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_docx_docling.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_docx_fallback_on_warn.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_docx_mammoth_fallback.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_html_trafilatura.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_pdf_backend_override.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_pdf_docling.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_pdf_pymupdf_fallback.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_save_images.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/integration/test_snapshots.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_footnote_markers.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_html_entities.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_html_entities_double_encoded.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_ligatures.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_nfc.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_quotes_dashes.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_soft_hyphens.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_validate.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_cleanup_whitespace.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_runner.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_strip_orphan_punctuation.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_structured_body_cleanup_stages.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_structured_compact_tables.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_structured_heading_hierarchy.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_structured_lift_figures.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_structured_normalize_cites.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_dedupe_paragraphs.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_dedupe_toc.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_dedupe_toc_table.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_dehyphenate.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_repair_line_wraps.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_restore_lists_code.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_strip_cover_artifacts.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_strip_headers_footers.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_strip_repeated_byline.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/pipeline/test_text_strip_web_fragments.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/scripts/__init__.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/scripts/test_audit_outputs.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_atomic_write.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_auto_id.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_config.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_config_walkup.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_docling_helper.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_frontmatter_helpers.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_heuristics.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_http_safe.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_log_sanitizer.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_meta_overrides.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_meta_reserved_keys.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_pdf_complexity.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_safe_dir_name.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_validators.py +0 -0
- {any2md-1.0.6 → any2md-1.0.7}/tests/unit/test_zipbomb_guard.py +0 -0
|
@@ -4,6 +4,36 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
6
6
|
|
|
7
|
+
## [1.0.7] — 2026-04-29
|
|
8
|
+
|
|
9
|
+
Branding/rebrand release. The frontmatter contract that any2md emits is
|
|
10
|
+
now described as "SAGE-compatible" (Security Analysis and Guidance
|
|
11
|
+
Exchange) — the upstream specification was renamed from SSRM (Structured
|
|
12
|
+
Security Reasoning Markdown) to SAGE. No functional or schema changes:
|
|
13
|
+
field names, ordering, controlled vocabularies, and `content_hash`
|
|
14
|
+
derivation are byte-identical to v1.0.6. Existing v1.0.6 outputs remain
|
|
15
|
+
valid SAGE-compatible documents without modification.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
|
|
19
|
+
- README, CHANGELOG, and all of `docs/` updated to reference SAGE /
|
|
20
|
+
Security Analysis and Guidance Exchange.
|
|
21
|
+
- `--auto-id` CLI help text now reads "SAGE-conformant document_id"
|
|
22
|
+
(previously "SSRM-conformant"). Behavior unchanged.
|
|
23
|
+
- Module/function docstrings in `any2md/frontmatter.py`,
|
|
24
|
+
`any2md/converters/txt.py`, and `any2md/pipeline/cleanup.py` updated
|
|
25
|
+
to reference SAGE §3 / §5.1 (same section numbers as the upstream
|
|
26
|
+
SSRM v1.0-RC1 spec).
|
|
27
|
+
- Two test functions renamed for naming consistency
|
|
28
|
+
(`test_txt_end_to_end_writes_sage_compat_output`,
|
|
29
|
+
`test_compose_emits_required_sage_fields`). Pytest auto-discovery is
|
|
30
|
+
unaffected; assertions unchanged.
|
|
31
|
+
|
|
32
|
+
### Migration
|
|
33
|
+
|
|
34
|
+
None required. Public API, CLI flags, frontmatter shape, and
|
|
35
|
+
`content_hash` outputs are all preserved bit-for-bit.
|
|
36
|
+
|
|
7
37
|
## [1.0.6] — 2026-04-27
|
|
8
38
|
|
|
9
39
|
Security hardening release. Closes 8 actionable findings from the
|
|
@@ -318,7 +348,7 @@ Patch release. Adds an explicit backend-selection CLI flag.
|
|
|
318
348
|
First stable release of any2md v1.0. Validated end-to-end against
|
|
319
349
|
real-world documents (a 164-page PDF technical standard via Docling,
|
|
320
350
|
a multi-page academic DOCX, and a Wikipedia article via trafilatura).
|
|
321
|
-
The output contract —
|
|
351
|
+
The output contract — SAGE-compatible frontmatter, deterministic
|
|
322
352
|
`content_hash`, NFC + LF body normalization — is now stable. Downstream
|
|
323
353
|
consumers can rely on this shape.
|
|
324
354
|
|
|
@@ -340,7 +370,7 @@ Phase 4: configuration, polish, and the v1.0 documentation set.
|
|
|
340
370
|
- `--meta-file PATH` flag plus auto-discovery of `.any2md.toml` (walks up
|
|
341
371
|
from cwd). The `[meta]` table supplies frontmatter overrides; the
|
|
342
372
|
`[document_id]` table supplies `--auto-id` prefix and type code.
|
|
343
|
-
- `--auto-id` flag — generates an
|
|
373
|
+
- `--auto-id` flag — generates an SAGE-conformant `document_id` as
|
|
344
374
|
`{PREFIX}-{YYYY}-{TYPE}-{SHA8}`. Defaults to `LOCAL`/`DOC`; override via
|
|
345
375
|
the `[document_id]` table in `.any2md.toml`.
|
|
346
376
|
- `--strict` flag — promotes pipeline validation warnings (heading
|
|
@@ -359,7 +389,7 @@ Phase 4: configuration, polish, and the v1.0 documentation set.
|
|
|
359
389
|
- Comprehensive documentation set:
|
|
360
390
|
- `README.md` rewritten with educational tone — every command preceded
|
|
361
391
|
by why you'd run it and followed by what you'll see.
|
|
362
|
-
- `docs/output-format.md` —
|
|
392
|
+
- `docs/output-format.md` — SAGE-compatible field reference and the
|
|
363
393
|
`content_hash` recomputation recipe.
|
|
364
394
|
- `docs/cli-reference.md` — flag-by-flag reference with use cases and a
|
|
365
395
|
worked-example matrix.
|
|
@@ -462,14 +492,14 @@ any2md transparently falls back to pymupdf4llm (PDF) and mammoth (DOCX).
|
|
|
462
492
|
|
|
463
493
|
First prerelease of any2md v1.0. Phase 1 of 5: foundation only — no Docling
|
|
464
494
|
backend yet, no new CLI flags. Output frontmatter has been rewritten to be
|
|
465
|
-
|
|
495
|
+
SAGE-compatible (Security Analysis and Guidance Exchange) — this is a
|
|
466
496
|
breaking change for downstream consumers parsing v0.7 output. See
|
|
467
497
|
`docs/superpowers/specs/2026-04-26-any2md-v1-design.md` for the full
|
|
468
498
|
v1.0 design and `docs/superpowers/plans/2026-04-26-any2md-v1-phase1-foundation.md`
|
|
469
499
|
for this phase's task plan.
|
|
470
500
|
|
|
471
501
|
### Added
|
|
472
|
-
- New `any2md/frontmatter.py` module —
|
|
502
|
+
- New `any2md/frontmatter.py` module — SAGE-compatible YAML emitter with the
|
|
473
503
|
`SourceMeta` dataclass, deterministic `compute_content_hash`,
|
|
474
504
|
`estimate_tokens`, `recommend_chunk_level`, `extract_abstract`, and
|
|
475
505
|
`derive_title` helpers, plus the central `compose()` function (the only
|
|
@@ -495,7 +525,7 @@ for this phase's task plan.
|
|
|
495
525
|
`reportlab`, `ruff`.
|
|
496
526
|
|
|
497
527
|
### Changed
|
|
498
|
-
- **BREAKING:** Output frontmatter shape is now
|
|
528
|
+
- **BREAKING:** Output frontmatter shape is now SAGE-compatible. New required
|
|
499
529
|
fields: `document_id` (empty string by default — opt-in via future
|
|
500
530
|
`--auto-id`), `version`, `date`, `status: "draft"`, `document_type` (empty
|
|
501
531
|
for non-security documents), `content_domain` (empty array),
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: any2md
|
|
3
|
-
Version: 1.0.
|
|
3
|
+
Version: 1.0.7
|
|
4
4
|
Summary: Convert PDF, DOCX, HTML, and TXT files — or web pages by URL — to clean, LLM-optimized Markdown with YAML frontmatter.
|
|
5
5
|
Project-URL: Homepage, https://github.com/rocklambros/any2md
|
|
6
6
|
Project-URL: Issues, https://github.com/rocklambros/any2md/issues
|
|
@@ -52,7 +52,7 @@ The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-emp
|
|
|
52
52
|
|
|
53
53
|
## What this is
|
|
54
54
|
|
|
55
|
-
any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is
|
|
55
|
+
any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SAGE-compatible (Security Analysis and Guidance Exchange — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
|
|
56
56
|
|
|
57
57
|
## Quick start
|
|
58
58
|
|
|
@@ -144,8 +144,8 @@ document_id: "" # empty unless you pass --aut
|
|
|
144
144
|
version: "1" # "1" for source files without an embedded version
|
|
145
145
|
date: "2026-03-15" # docx core props "modified" → ISO-8601
|
|
146
146
|
status: "draft" # always "draft" for converted documents
|
|
147
|
-
document_type: "" #
|
|
148
|
-
content_domain: [] #
|
|
147
|
+
document_type: "" # SAGE controlled vocab — empty for non-security docs
|
|
148
|
+
content_domain: [] # SAGE controlled vocab — empty array when unknown
|
|
149
149
|
authors: # extracted from docProps/core.xml dc:creator
|
|
150
150
|
- "Rock Lambros"
|
|
151
151
|
organization: "" # docProps/app.xml Company; empty when absent
|
|
@@ -166,7 +166,7 @@ produced_by: "Microsoft® Word for Microsoft 365" # software that produced the
|
|
|
166
166
|
---
|
|
167
167
|
```
|
|
168
168
|
|
|
169
|
-
Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the
|
|
169
|
+
Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SAGE contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
|
|
170
170
|
|
|
171
171
|
### Before / after
|
|
172
172
|
|
|
@@ -362,7 +362,7 @@ You'll see the OK lines as if the conversion were fresh; previously-written outp
|
|
|
362
362
|
|
|
363
363
|
## The output format
|
|
364
364
|
|
|
365
|
-
Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **
|
|
365
|
+
Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SAGE-compatible** — it matches the field names, types, and ordering of the Security Analysis and Guidance Exchange specification, but values aren't required to match SAGE's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
|
|
366
366
|
|
|
367
367
|
### Field auto-fill summary
|
|
368
368
|
|
|
@@ -407,7 +407,7 @@ You'll see the OK line as usual. The frontmatter of `Text/paper.md` will have `o
|
|
|
407
407
|
|
|
408
408
|
### `.any2md.toml`
|
|
409
409
|
|
|
410
|
-
You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing
|
|
410
|
+
You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SAGE-conforming outputs:
|
|
411
411
|
|
|
412
412
|
```toml
|
|
413
413
|
[meta]
|
|
@@ -561,7 +561,7 @@ The two-lane design exists because Docling output is layout-correct (running agg
|
|
|
561
561
|
|
|
562
562
|
## Migrating from v0.7
|
|
563
563
|
|
|
564
|
-
v1.0 emits
|
|
564
|
+
v1.0 emits SAGE-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
|
|
565
565
|
|
|
566
566
|
## Security
|
|
567
567
|
|
|
@@ -19,7 +19,7 @@ The full design rationale lives in [docs/superpowers/specs/2026-04-26-v1.0.3-emp
|
|
|
19
19
|
|
|
20
20
|
## What this is
|
|
21
21
|
|
|
22
|
-
any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is
|
|
22
|
+
any2md ingests heterogeneous source documents and emits Markdown with a fixed YAML frontmatter contract. The frontmatter shape is SAGE-compatible (Security Analysis and Guidance Exchange — a documented schema for LLM-consumable documents), so retrieval pipelines, embeddings jobs, and chunking utilities downstream can rely on field names, types, and a deterministic `content_hash` for cache invalidation. The body is NFC-normalized with LF line endings, and the heading hierarchy is guaranteed to start at H1 and not skip levels. The goal is one stable shape, regardless of whether a document started life as a scanned PDF, a Word doc, a Wikipedia page, or a plain text dump.
|
|
23
23
|
|
|
24
24
|
## Quick start
|
|
25
25
|
|
|
@@ -111,8 +111,8 @@ document_id: "" # empty unless you pass --aut
|
|
|
111
111
|
version: "1" # "1" for source files without an embedded version
|
|
112
112
|
date: "2026-03-15" # docx core props "modified" → ISO-8601
|
|
113
113
|
status: "draft" # always "draft" for converted documents
|
|
114
|
-
document_type: "" #
|
|
115
|
-
content_domain: [] #
|
|
114
|
+
document_type: "" # SAGE controlled vocab — empty for non-security docs
|
|
115
|
+
content_domain: [] # SAGE controlled vocab — empty array when unknown
|
|
116
116
|
authors: # extracted from docProps/core.xml dc:creator
|
|
117
117
|
- "Rock Lambros"
|
|
118
118
|
organization: "" # docProps/app.xml Company; empty when absent
|
|
@@ -133,7 +133,7 @@ produced_by: "Microsoft® Word for Microsoft 365" # software that produced the
|
|
|
133
133
|
---
|
|
134
134
|
```
|
|
135
135
|
|
|
136
|
-
Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the
|
|
136
|
+
Fields under `# any2md extension field` are retained from v0.7 for traceability and observability — they're not part of the SAGE contract proper, but they live in the same frontmatter block. See [docs/output-format.md](docs/output-format.md) for the full field reference.
|
|
137
137
|
|
|
138
138
|
### Before / after
|
|
139
139
|
|
|
@@ -329,7 +329,7 @@ You'll see the OK lines as if the conversion were fresh; previously-written outp
|
|
|
329
329
|
|
|
330
330
|
## The output format
|
|
331
331
|
|
|
332
|
-
Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **
|
|
332
|
+
Every file converted by any2md has the same frontmatter shape, regardless of source format. The shape is **SAGE-compatible** — it matches the field names, types, and ordering of the Security Analysis and Guidance Exchange specification, but values aren't required to match SAGE's controlled vocabularies (most converted documents aren't security research). Fields that require a controlled vocabulary (`document_type`, `content_domain`, `tlp`) are emitted empty unless you supply them via `--meta` or `.any2md.toml`.
|
|
333
333
|
|
|
334
334
|
### Field auto-fill summary
|
|
335
335
|
|
|
@@ -374,7 +374,7 @@ You'll see the OK line as usual. The frontmatter of `Text/paper.md` will have `o
|
|
|
374
374
|
|
|
375
375
|
### `.any2md.toml`
|
|
376
376
|
|
|
377
|
-
You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing
|
|
377
|
+
You'd use this when you have a stable set of frontmatter defaults that should apply to every conversion in a project. The file is auto-discovered by walking up from the current working directory, so you can drop one at the root of your corpus repo and forget about it. Worked example for a security-research org producing SAGE-conforming outputs:
|
|
378
378
|
|
|
379
379
|
```toml
|
|
380
380
|
[meta]
|
|
@@ -528,7 +528,7 @@ The two-lane design exists because Docling output is layout-correct (running agg
|
|
|
528
528
|
|
|
529
529
|
## Migrating from v0.7
|
|
530
530
|
|
|
531
|
-
v1.0 emits
|
|
531
|
+
v1.0 emits SAGE-compatible frontmatter — that's the one intended breaking change. v0.7's small frontmatter (`title`, `source_file`, `pages`, `type`, `word_count`) becomes a fuller block with `document_id`, `version`, `date`, `status`, `content_hash`, `token_estimate`, `authors`, and several other fields. The v0.7 fields are retained as any2md extension fields in the same block, so traceability is preserved, but downstream parsers that expected the old fixed shape will need to update. The body is now NFC-normalized with LF line endings, which means `content_hash` is reproducible. If you need bit-for-bit v0.7 output, pin `any2md==0.7.0`. The full field-by-field migration guide and behavior-change list lives in [docs/upgrading-from-0.7.md](docs/upgrading-from-0.7.md).
|
|
532
532
|
|
|
533
533
|
## Security
|
|
534
534
|
|
|
@@ -200,7 +200,7 @@ def main():
|
|
|
200
200
|
parser.add_argument(
|
|
201
201
|
"--auto-id",
|
|
202
202
|
action="store_true",
|
|
203
|
-
help="Generate an
|
|
203
|
+
help="Generate an SAGE-conformant document_id of the form "
|
|
204
204
|
"{PREFIX}-{YYYY}-{TYPE}-{SHA8}. Defaults: PREFIX=LOCAL, TYPE=DOC. "
|
|
205
205
|
"Override via .any2md.toml [document_id] (publisher_prefix, type_code).",
|
|
206
206
|
)
|
|
@@ -176,7 +176,7 @@ def convert_txt(
|
|
|
176
176
|
force: bool = False,
|
|
177
177
|
strip_links_flag: bool = False,
|
|
178
178
|
) -> bool:
|
|
179
|
-
"""Convert a plain-text file to v1.0
|
|
179
|
+
"""Convert a plain-text file to v1.0 SAGE-compatible Markdown."""
|
|
180
180
|
if options is None:
|
|
181
181
|
options = PipelineOptions(strip_links=strip_links_flag)
|
|
182
182
|
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
"""
|
|
1
|
+
"""SAGE-compatible YAML frontmatter emitter.
|
|
2
2
|
|
|
3
3
|
See spec §3 (frontmatter contract) and §5.0 (SourceMeta dataclass).
|
|
4
4
|
This module is the single producer of the YAML block — converters
|
|
@@ -60,7 +60,7 @@ def filter_reserved_overrides(
|
|
|
60
60
|
|
|
61
61
|
|
|
62
62
|
def compute_content_hash(body: str) -> str:
|
|
63
|
-
"""SHA-256 of NFC-normalized, LF-line-ended body.
|
|
63
|
+
"""SHA-256 of NFC-normalized, LF-line-ended body. SAGE §5.1.
|
|
64
64
|
|
|
65
65
|
The body MUST be the post-pipeline output (after C1-C5). This function
|
|
66
66
|
re-applies NFC and LF normalization defensively so callers can pass any
|
|
@@ -74,7 +74,7 @@ def compute_content_hash(body: str) -> str:
|
|
|
74
74
|
def generate_document_id(
|
|
75
75
|
body: str, prefix: str = "LOCAL", type_code: str = "DOC"
|
|
76
76
|
) -> str:
|
|
77
|
-
"""Generate an
|
|
77
|
+
"""Generate an SAGE-conformant ``document_id`` from body content.
|
|
78
78
|
|
|
79
79
|
Pattern: ``{PREFIX}-{YYYY}-{TYPE}-{SHA8}``.
|
|
80
80
|
|
|
@@ -227,7 +227,7 @@ def _normalize_body(body: str) -> str:
|
|
|
227
227
|
def _build_fields(
|
|
228
228
|
body: str, meta: SourceMeta, options: PipelineOptions
|
|
229
229
|
) -> dict[str, Any]:
|
|
230
|
-
"""Build the ordered field map for the
|
|
230
|
+
"""Build the ordered field map for the SAGE frontmatter block.
|
|
231
231
|
|
|
232
232
|
Returns an insertion-ordered dict matching spec §3.2-3.4. Only fields
|
|
233
233
|
that should be emitted are present; conditional fields (e.g. ``pages``,
|
|
@@ -333,7 +333,7 @@ def _emit_field(key: str, value: Any, lines: list[str]) -> None:
|
|
|
333
333
|
lines.append(f" {subkey}: {_emit_value(subval)}")
|
|
334
334
|
return
|
|
335
335
|
if isinstance(value, list):
|
|
336
|
-
# Only string-lists are supported for now (matches
|
|
336
|
+
# Only string-lists are supported for now (matches SAGE §3 fields).
|
|
337
337
|
lines.append(f"{key}: {_emit_array([str(v) for v in value])}")
|
|
338
338
|
return
|
|
339
339
|
if isinstance(value, (int, float)) and not isinstance(value, bool):
|
|
@@ -359,7 +359,7 @@ def compose(
|
|
|
359
359
|
options: PipelineOptions,
|
|
360
360
|
overrides: dict[str, Any] | None = None,
|
|
361
361
|
) -> str:
|
|
362
|
-
"""Build a complete
|
|
362
|
+
"""Build a complete SAGE-compatible Markdown document.
|
|
363
363
|
|
|
364
364
|
Steps:
|
|
365
365
|
1. Normalize body to NFC + LF endings (matches content_hash invariant).
|
|
@@ -14,7 +14,7 @@ Stage = Callable[[str, "PipelineOptions"], str]
|
|
|
14
14
|
|
|
15
15
|
|
|
16
16
|
def nfc_normalize(text: str, _options: "PipelineOptions") -> str:
|
|
17
|
-
"""C1: NFC unicode normalization. Required by
|
|
17
|
+
"""C1: NFC unicode normalization. Required by SAGE §5.1 for content_hash."""
|
|
18
18
|
return unicodedata.normalize("NFC", text)
|
|
19
19
|
|
|
20
20
|
|
|
@@ -167,7 +167,7 @@ normalizations that any well-formed markdown can absorb without harm:
|
|
|
167
167
|
Centralizing these means `content_hash` is byte-deterministic: the same
|
|
168
168
|
post-cleanup body produces the same hash regardless of which lane ran first,
|
|
169
169
|
because the cleanup is the last thing that touched the bytes. This is the
|
|
170
|
-
determinism boundary the
|
|
170
|
+
determinism boundary the SAGE contract relies on.
|
|
171
171
|
|
|
172
172
|
## Stage catalog
|
|
173
173
|
|
|
@@ -244,7 +244,7 @@ order.
|
|
|
244
244
|
|
|
245
245
|
| Run order | ID | Name | Lane | Input shape | Output shape | No-op cases | Edge cases |
|
|
246
246
|
|---|---|---|---|---|---|---|---|
|
|
247
|
-
| 1 | C1 | `nfc_normalize` | shared | Any unicode text | NFC-normalized text | Already-NFC text | Required by
|
|
247
|
+
| 1 | C1 | `nfc_normalize` | shared | Any unicode text | NFC-normalized text | Already-NFC text | Required by SAGE `content_hash` invariant |
|
|
248
248
|
| 2 | C2 | `strip_soft_hyphens` | shared | Text containing U+00AD (``) | Text without U+00AD | No soft hyphens | Soft hyphens are invisible but token-costly |
|
|
249
249
|
| 3 | C3 | `normalize_ligatures` | shared | Text containing presentation-form ligatures (`fi`, `fl`, `ffi`, `ffl`, `ſt`, `st`, `ff`) and NBSP | Text with whitelist-expanded ligatures and regular spaces | None of the whitelist characters present | Whitelist-only — does not run blanket NFKC, which would fold superscripts and CJK compatibility characters |
|
|
250
250
|
| 4 | C4 | `normalize_quotes_dashes` | shared | Smart quotes (`“”‘’`) and ellipsis (`…`) | Straight quotes (`""''`) and three-dot ellipsis | No smart quotes or ellipsis | En-dash and em-dash are preserved (semantic) |
|
|
@@ -305,7 +305,7 @@ class SourceMeta:
|
|
|
305
305
|
|
|
306
306
|
### Field-by-field
|
|
307
307
|
|
|
308
|
-
| Field | Filled by | Required for
|
|
308
|
+
| Field | Filled by | Required for SAGE contract |
|
|
309
309
|
|---|---|---|
|
|
310
310
|
| `title_hint` | All converters when source metadata has a title (PDF `/Title`, DOCX `dc:title`, HTML `<title>`, otherwise `None`) | No — `frontmatter.derive_title` falls back to first H1, then filename |
|
|
311
311
|
| `authors` | PDF (`/Author` parsed), DOCX (`dc:creator`), HTML (`<meta name="author">`), TXT (always `[]`). v1.0.2 adds body-text extraction and an arxiv API enrichment via `heuristics.extract_authors`. | No — empty `[]` is valid |
|
|
@@ -318,11 +318,11 @@ class SourceMeta:
|
|
|
318
318
|
| `source_file` | All file inputs; URL inputs `None` | No — extension field |
|
|
319
319
|
| `source_url` | URL inputs only; file inputs `None` | No — extension field |
|
|
320
320
|
| `doc_type` | All converters | Yes — emitted as `type` in the YAML |
|
|
321
|
-
| `extracted_via` | All converters | Yes — non-
|
|
321
|
+
| `extracted_via` | All converters | Yes — non-SAGE extension but required for traceability |
|
|
322
322
|
| `lane` | All converters | Yes — drives pipeline routing |
|
|
323
323
|
|
|
324
324
|
The four "required" fields above are required for the v1.0 contract to be
|
|
325
|
-
fillable, not for
|
|
325
|
+
fillable, not for SAGE strict conformance. Strict SAGE has a different
|
|
326
326
|
required-fields list — see [output-format.md](output-format.md) for that
|
|
327
327
|
distinction.
|
|
328
328
|
|
|
@@ -268,7 +268,7 @@ that defines what each key means.
|
|
|
268
268
|
|
|
269
269
|
### `--auto-id`
|
|
270
270
|
|
|
271
|
-
Generate an
|
|
271
|
+
Generate an SAGE-conformant `document_id` of the form
|
|
272
272
|
`{PREFIX}-{YYYY}-{TYPE}-{SHA8}`. Defaults: `PREFIX=LOCAL`, `TYPE=DOC`. Override
|
|
273
273
|
via `[document_id]` in `.any2md.toml` (`publisher_prefix`, `type_code`).
|
|
274
274
|
|
|
@@ -31,22 +31,22 @@ non-empty, single-line, and YAML-escaped. None of those guarantees come free —
|
|
|
31
31
|
they exist because every any2md output goes through the same final emitter and
|
|
32
32
|
because every field in this document has a derivation rule that's tested.
|
|
33
33
|
|
|
34
|
-
## The
|
|
34
|
+
## The SAGE connection
|
|
35
35
|
|
|
36
|
-
any2md's frontmatter is **
|
|
37
|
-
|
|
36
|
+
any2md's frontmatter is **SAGE-compatible**. SAGE (Security Analysis
|
|
37
|
+
and Guidance Exchange) is a documented schema for LLM-consumable security
|
|
38
38
|
research documents — its primary audience is producers of threat intelligence
|
|
39
39
|
and security guidance who want a stable shape for downstream RAG and LLM
|
|
40
|
-
analysis.
|
|
40
|
+
analysis. SAGE v1.0-RC1 is the version this contract tracks.
|
|
41
41
|
|
|
42
42
|
"Compatible" rather than "strict" because most documents any2md converts are
|
|
43
|
-
not security research.
|
|
43
|
+
not security research. SAGE has fields with controlled vocabularies —
|
|
44
44
|
`document_type`, `content_domain`, `tlp`, `frameworks_referenced` — that don't
|
|
45
|
-
have plausible defaults for an arbitrary PDF or web page. Strict
|
|
45
|
+
have plausible defaults for an arbitrary PDF or web page. Strict SAGE
|
|
46
46
|
validators reject empty or unknown values for those fields. any2md's choice is
|
|
47
47
|
to emit the fields with empty defaults (`""` or `[]`) and `status: "draft"`, so
|
|
48
|
-
the output is machine-readable in the
|
|
49
|
-
finished
|
|
48
|
+
the output is machine-readable in the SAGE shape but doesn't claim to be a
|
|
49
|
+
finished SAGE document. If you produce real SAGE-conforming security research,
|
|
50
50
|
populate those fields explicitly via [`--meta` or `.any2md.toml`](cli-reference.md#configuration-meta-and-meta-file).
|
|
51
51
|
|
|
52
52
|
The fields below are grouped: identity, classification, provenance, integrity,
|
|
@@ -134,15 +134,15 @@ records when any2md wrote it).
|
|
|
134
134
|
|
|
135
135
|
**auto** | type: `string` (controlled vocabulary)
|
|
136
136
|
|
|
137
|
-
**Derivation:** Always `"draft"` for converted documents.
|
|
137
|
+
**Derivation:** Always `"draft"` for converted documents. SAGE allows
|
|
138
138
|
non-controlled-vocab values for other fields when `status` is `"draft"`, which
|
|
139
139
|
is why this is the default — it's how an any2md output declares "I have an
|
|
140
|
-
|
|
140
|
+
SAGE-compatible shape but I have not been authored as SAGE."
|
|
141
141
|
|
|
142
142
|
**Example:** `"draft"`
|
|
143
143
|
|
|
144
144
|
**Common mistake:** Overriding to `"published"` via `--meta` for documents
|
|
145
|
-
that haven't actually been reviewed.
|
|
145
|
+
that haven't actually been reviewed. SAGE consumers may treat
|
|
146
146
|
`status: "published"` as a signal that the controlled-vocabulary fields are
|
|
147
147
|
trustworthy. Don't claim a status the content doesn't earn.
|
|
148
148
|
|
|
@@ -150,7 +150,7 @@ trustworthy. Don't claim a status the content doesn't earn.
|
|
|
150
150
|
|
|
151
151
|
**user** | type: `string` (controlled vocabulary or empty)
|
|
152
152
|
|
|
153
|
-
**Derivation:** Empty `""` by default.
|
|
153
|
+
**Derivation:** Empty `""` by default. SAGE defines a controlled vocabulary
|
|
154
154
|
(e.g. `"guidance"`, `"vuln_advisory"`, `"threat_report"`) but any2md cannot
|
|
155
155
|
infer it from a generic document.
|
|
156
156
|
|
|
@@ -161,7 +161,7 @@ infer it from a generic document.
|
|
|
161
161
|
|
|
162
162
|
**user** | type: `array<string>`
|
|
163
163
|
|
|
164
|
-
**Derivation:** Empty `[]` by default.
|
|
164
|
+
**Derivation:** Empty `[]` by default. SAGE controlled vocabulary
|
|
165
165
|
(e.g. `"ai_security"`, `"cloud"`, `"identity"`).
|
|
166
166
|
|
|
167
167
|
**Example empty:** `[]`
|
|
@@ -199,7 +199,7 @@ otherwise empty. User overrides commonly set this for corpus-wide attribution.
|
|
|
199
199
|
**auto** | type: `object` (`{authored_by: string}` minimum)
|
|
200
200
|
|
|
201
201
|
**Derivation:** Always emitted with `authored_by: "unknown"` because any2md
|
|
202
|
-
converts content; it does not author it.
|
|
202
|
+
converts content; it does not author it. SAGE's `generation_metadata` block
|
|
203
203
|
exists to record who or what produced the document (a human, a model, a
|
|
204
204
|
collaboration). If you ran any2md as part of an authoring workflow, override
|
|
205
205
|
the value via `--meta generation_metadata.authored_by=human`.
|
|
@@ -218,8 +218,8 @@ generation_metadata:
|
|
|
218
218
|
```
|
|
219
219
|
|
|
220
220
|
**Common mistake:** Adding fields under `generation_metadata` and expecting
|
|
221
|
-
|
|
222
|
-
shape — consult the
|
|
221
|
+
SAGE validators to accept them. The SAGE schema for this block has its own
|
|
222
|
+
shape — consult the SAGE specification before adding non-standard subfields.
|
|
223
223
|
|
|
224
224
|
### `content_hash`
|
|
225
225
|
|
|
@@ -300,7 +300,7 @@ that.
|
|
|
300
300
|
|
|
301
301
|
**user** | type: `array<string>` and `string`
|
|
302
302
|
|
|
303
|
-
**Derivation:** Empty by default.
|
|
303
|
+
**Derivation:** Empty by default. SAGE-specific fields (security frameworks
|
|
304
304
|
the document references; traffic-light-protocol marking). User-supplied via
|
|
305
305
|
`--meta` or `.any2md.toml`.
|
|
306
306
|
|
|
@@ -660,7 +660,7 @@ tampered with after generation.
|
|
|
660
660
|
This schema describes the v1.0 frontmatter block. Conditional fields
|
|
661
661
|
(`keywords`, `abstract_for_rag`, `pages`, `word_count`, `source_file`,
|
|
662
662
|
`source_url`, `produced_by`) are documented here as optional; they're omitted
|
|
663
|
-
from the frontmatter when not set.
|
|
663
|
+
from the frontmatter when not set. SAGE-only fields a user might add (`tlp`,
|
|
664
664
|
`frameworks_referenced`, etc.) are listed as additional valid fields.
|
|
665
665
|
|
|
666
666
|
```json
|
|
@@ -799,7 +799,7 @@ from the frontmatter when not set. SSRM-only fields a user might add (`tlp`,
|
|
|
799
799
|
|
|
800
800
|
The schema sets `additionalProperties: true` because users add fields via
|
|
801
801
|
`--meta` and `.any2md.toml` that are not part of the v1.0 contract. A stricter
|
|
802
|
-
schema for
|
|
802
|
+
schema for SAGE-conforming outputs would set this to `false` and add the SAGE
|
|
803
803
|
controlled-vocabulary constraints to `document_type`, `content_domain`, and
|
|
804
804
|
related fields. Use this schema as a starting point for downstream validation
|
|
805
805
|
and tighten it for your environment.
|
{any2md-1.0.6 → any2md-1.0.7}/docs/superpowers/plans/2026-04-26-any2md-v1-phase1-foundation.md
RENAMED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
4
|
|
|
5
|
-
**Goal:** Restructure any2md into the v1.0 architecture (centralized frontmatter + pipeline + SourceMeta) and rewire all 4 existing converters through it without changing extraction backends. End state: every output file carries
|
|
5
|
+
**Goal:** Restructure any2md into the v1.0 architecture (centralized frontmatter + pipeline + SourceMeta) and rewire all 4 existing converters through it without changing extraction backends. End state: every output file carries SAGE-compatible frontmatter with deterministic `content_hash`, body is NFC-normalized + LF-ended, and a `1.0.0a1` prerelease lands on TestPyPI.
|
|
6
6
|
|
|
7
7
|
**Architecture:** Two-lane post-processing pipeline (structured / text), shared cleanup stages always last, single YAML emitter (`frontmatter.py`). In Phase 1 the structured lane is empty (no Docling yet) and the text lane is empty (T1–T6 land in Phase 3); only the shared cleanup C1–C7 runs.
|
|
8
8
|
|
|
@@ -736,7 +736,7 @@ Expected: FAIL with `ModuleNotFoundError: any2md.frontmatter`.
|
|
|
736
736
|
- [ ] **Step 3: Create `any2md/frontmatter.py` skeleton**
|
|
737
737
|
|
|
738
738
|
```python
|
|
739
|
-
"""
|
|
739
|
+
"""SAGE-compatible YAML frontmatter emitter.
|
|
740
740
|
|
|
741
741
|
See spec §3 (frontmatter contract) and §5.0 (SourceMeta dataclass).
|
|
742
742
|
This module is the single producer of the YAML block — converters
|
|
@@ -842,7 +842,7 @@ Stage = Callable[[str, "PipelineOptions"], str]
|
|
|
842
842
|
|
|
843
843
|
|
|
844
844
|
def nfc_normalize(text: str, _options: "PipelineOptions") -> str:
|
|
845
|
-
"""C1: NFC unicode normalization. Required by
|
|
845
|
+
"""C1: NFC unicode normalization. Required by SAGE §5.1 for content_hash."""
|
|
846
846
|
return unicodedata.normalize("NFC", text)
|
|
847
847
|
|
|
848
848
|
|
|
@@ -1502,7 +1502,7 @@ git commit -m "feat(pipeline): C7 validate (read-only, emits warnings)"
|
|
|
1502
1502
|
|
|
1503
1503
|
```python
|
|
1504
1504
|
# tests/unit/test_content_hash.py
|
|
1505
|
-
"""Tests for content_hash determinism (
|
|
1505
|
+
"""Tests for content_hash determinism (SAGE §5.1)."""
|
|
1506
1506
|
|
|
1507
1507
|
from any2md.frontmatter import compute_content_hash
|
|
1508
1508
|
|
|
@@ -1557,7 +1557,7 @@ import unicodedata
|
|
|
1557
1557
|
|
|
1558
1558
|
|
|
1559
1559
|
def compute_content_hash(body: str) -> str:
|
|
1560
|
-
"""SHA-256 of NFC-normalized, LF-line-ended body.
|
|
1560
|
+
"""SHA-256 of NFC-normalized, LF-line-ended body. SAGE §5.1.
|
|
1561
1561
|
|
|
1562
1562
|
The body MUST be the post-pipeline output (after C1–C5). This function
|
|
1563
1563
|
re-applies NFC and LF normalization defensively so callers can pass any
|
|
@@ -1898,7 +1898,7 @@ git commit -m "feat(frontmatter): derive_title with H1/hint/filename fallback ch
|
|
|
1898
1898
|
|
|
1899
1899
|
---
|
|
1900
1900
|
|
|
1901
|
-
## Task 21: Frontmatter `compose()` — full
|
|
1901
|
+
## Task 21: Frontmatter `compose()` — full SAGE-compatible YAML emitter
|
|
1902
1902
|
|
|
1903
1903
|
**Files:**
|
|
1904
1904
|
- Modify: `any2md/frontmatter.py`
|
|
@@ -1908,7 +1908,7 @@ git commit -m "feat(frontmatter): derive_title with H1/hint/filename fallback ch
|
|
|
1908
1908
|
|
|
1909
1909
|
```python
|
|
1910
1910
|
# tests/unit/test_frontmatter_compose.py
|
|
1911
|
-
"""Tests for frontmatter.compose() —
|
|
1911
|
+
"""Tests for frontmatter.compose() — SAGE-compatible output."""
|
|
1912
1912
|
|
|
1913
1913
|
from datetime import date
|
|
1914
1914
|
|
|
@@ -1945,7 +1945,7 @@ def _meta(**overrides) -> SourceMeta:
|
|
|
1945
1945
|
return SourceMeta(**base)
|
|
1946
1946
|
|
|
1947
1947
|
|
|
1948
|
-
def
|
|
1948
|
+
def test_compose_emits_required_sage_fields():
|
|
1949
1949
|
body = "# Title\n\nbody\n"
|
|
1950
1950
|
out = compose(body, _meta(), PipelineOptions())
|
|
1951
1951
|
fm, _ = _split_frontmatter(out)
|
|
@@ -2092,7 +2092,7 @@ def _emit_array(values: list[str]) -> str:
|
|
|
2092
2092
|
|
|
2093
2093
|
|
|
2094
2094
|
def compose(body: str, meta: SourceMeta, options: PipelineOptions) -> str:
|
|
2095
|
-
"""Build a complete
|
|
2095
|
+
"""Build a complete SAGE-compatible Markdown document.
|
|
2096
2096
|
|
|
2097
2097
|
Steps:
|
|
2098
2098
|
1. Normalize body to NFC + LF endings (matches content_hash invariant).
|
|
@@ -2117,7 +2117,7 @@ def compose(body: str, meta: SourceMeta, options: PipelineOptions) -> str:
|
|
|
2117
2117
|
today = _date_cls.today().isoformat()
|
|
2118
2118
|
fm_date = meta.date or today
|
|
2119
2119
|
|
|
2120
|
-
# 3. Emit YAML in
|
|
2120
|
+
# 3. Emit YAML in SAGE-defined order
|
|
2121
2121
|
lines: list[str] = ["---"]
|
|
2122
2122
|
lines.append(f"title: {_emit_value(title)}")
|
|
2123
2123
|
lines.append('document_id: ""')
|
|
@@ -2163,7 +2163,7 @@ Expected: all PASS.
|
|
|
2163
2163
|
|
|
2164
2164
|
```bash
|
|
2165
2165
|
git add any2md/frontmatter.py tests/unit/test_frontmatter_compose.py
|
|
2166
|
-
git commit -m "feat(frontmatter): compose()
|
|
2166
|
+
git commit -m "feat(frontmatter): compose() SAGE-compatible YAML emitter"
|
|
2167
2167
|
```
|
|
2168
2168
|
|
|
2169
2169
|
---
|
|
@@ -2283,7 +2283,7 @@ from any2md.cli import main
|
|
|
2283
2283
|
from any2md.pipeline import PipelineOptions
|
|
2284
2284
|
|
|
2285
2285
|
|
|
2286
|
-
def
|
|
2286
|
+
def test_txt_end_to_end_writes_sage_compat_output(fixture_dir, tmp_output_dir, monkeypatch):
|
|
2287
2287
|
monkeypatch.setattr(
|
|
2288
2288
|
"sys.argv",
|
|
2289
2289
|
["any2md", "-o", str(tmp_output_dir), str(fixture_dir / "ligatures_and_softhyphens.txt")],
|
|
@@ -2478,7 +2478,7 @@ def convert_txt(
|
|
|
2478
2478
|
force: bool = False,
|
|
2479
2479
|
strip_links_flag: bool = False,
|
|
2480
2480
|
) -> bool:
|
|
2481
|
-
"""Convert a plain-text file to v1.0
|
|
2481
|
+
"""Convert a plain-text file to v1.0 SAGE-compatible Markdown."""
|
|
2482
2482
|
if options is None:
|
|
2483
2483
|
options = PipelineOptions(strip_links=strip_links_flag)
|
|
2484
2484
|
|
|
@@ -3467,7 +3467,7 @@ echo
|
|
|
3467
3467
|
echo "Outputs:"
|
|
3468
3468
|
ls -la "$OUT_DIR"
|
|
3469
3469
|
echo
|
|
3470
|
-
echo "Inspect each for:
|
|
3470
|
+
echo "Inspect each for: SAGE frontmatter, content_hash present, body NFC + LF, no garbled chars."
|
|
3471
3471
|
```
|
|
3472
3472
|
|
|
3473
3473
|
- [ ] **Step 2: Make executable + run**
|
|
@@ -3600,12 +3600,12 @@ Replace the `## [Unreleased]` heading and contents with:
|
|
|
3600
3600
|
|
|
3601
3601
|
This is the first prerelease of any2md v1.0. Phase 1 of 5: foundation only.
|
|
3602
3602
|
No Docling, no new CLI flags. Output frontmatter has been rewritten to be
|
|
3603
|
-
|
|
3603
|
+
SAGE-compatible — this is a breaking change for downstream consumers
|
|
3604
3604
|
parsing v0.7 output. See `docs/superpowers/specs/2026-04-26-any2md-v1-design.md`
|
|
3605
3605
|
for the full v1.0 design.
|
|
3606
3606
|
|
|
3607
3607
|
### Added
|
|
3608
|
-
- New `any2md/frontmatter.py` module (
|
|
3608
|
+
- New `any2md/frontmatter.py` module (SAGE-compatible YAML emitter, `SourceMeta`).
|
|
3609
3609
|
- New `any2md/pipeline/` package with shared cleanup stages C1–C7
|
|
3610
3610
|
(NFC normalization, soft-hyphen strip, ligature normalization,
|
|
3611
3611
|
quote/dash normalization, whitespace collapse, footnote-marker strip,
|
|
@@ -3615,7 +3615,7 @@ for the full v1.0 design.
|
|
|
3615
3615
|
- pytest test suite covering pipeline, frontmatter, validators, and per-format integration.
|
|
3616
3616
|
|
|
3617
3617
|
### Changed
|
|
3618
|
-
- **BREAKING:** Output frontmatter shape is
|
|
3618
|
+
- **BREAKING:** Output frontmatter shape is SAGE-compatible. New required fields:
|
|
3619
3619
|
`document_id` (empty), `version`, `date`, `status: "draft"`, `document_type` (empty),
|
|
3620
3620
|
`content_domain`, `authors`, `organization`, `generation_metadata`, `content_hash`.
|
|
3621
3621
|
`source_file`, `pages`, `word_count`, and `type` are retained as any2md
|
|
@@ -3715,7 +3715,7 @@ PYTHONPATH=/tmp/any2md_test_install python -m any2md \
|
|
|
3715
3715
|
head -20 /tmp/any2md_smoke/web_page.md
|
|
3716
3716
|
```
|
|
3717
3717
|
|
|
3718
|
-
Expected: help text shows; web_page.md has
|
|
3718
|
+
Expected: help text shows; web_page.md has SAGE-compat frontmatter.
|
|
3719
3719
|
|
|
3720
3720
|
- [ ] **Step 8: Final commit (none needed — everything already committed)**
|
|
3721
3721
|
|
|
@@ -3763,7 +3763,7 @@ Approximate critical path: A → C → D/E → F → G → H → I = 12 sequenti
|
|
|
3763
3763
|
## Self-review summary
|
|
3764
3764
|
|
|
3765
3765
|
Spec coverage:
|
|
3766
|
-
- §3 Frontmatter contract → Tasks 8, 16–22 (full
|
|
3766
|
+
- §3 Frontmatter contract → Tasks 8, 16–22 (full SAGE-compat shape, content_hash, all derivation helpers).
|
|
3767
3767
|
- §4 Pipeline (C1–C7) → Tasks 9–15. Text/structured lanes registered empty in Task 7 (Phase 2/3 fill them).
|
|
3768
3768
|
- §5 Per-format converters → Tasks 23–26 (TXT, HTML, DOCX, PDF). Phase 1 PDF is pymupdf4llm only.
|
|
3769
3769
|
- §6 CLI → Task 27 preserves v0.7 flags routing through new pipeline. New flags are Phase 4.
|