bindery-cli 0.31.0__tar.gz → 0.32.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (74) hide show
  1. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/CLAUDE.md +15 -4
  2. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/PKG-INFO +7 -7
  3. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/README.md +6 -6
  4. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/patchnotes.md +109 -0
  5. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/pyproject.toml +1 -1
  6. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/roadmap.md +115 -11
  7. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/spec.md +43 -7
  8. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/__init__.py +1 -1
  9. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/audit.py +13 -3
  10. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/cli.py +25 -0
  11. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/epub.py +135 -34
  12. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/library.py +60 -3
  13. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/pagination.py +16 -4
  14. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/reserialize.py +18 -4
  15. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/transforms.py +122 -32
  16. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/watermark.py +67 -17
  17. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_audit.py +12 -0
  18. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_cli.py +34 -1
  19. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_epub.py +194 -1
  20. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_library.py +50 -0
  21. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_pagination.py +52 -0
  22. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_reserialize.py +13 -0
  23. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_transforms.py +126 -0
  24. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_watermark.py +100 -17
  25. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/uv.lock +1 -1
  26. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/.github/workflows/ci.yml +0 -0
  27. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/.github/workflows/publish.yml +0 -0
  28. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/.gitignore +0 -0
  29. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/AGENTS.md +0 -0
  30. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/LICENSE +0 -0
  31. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/docs/screenshots/library-sweep.png +0 -0
  32. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/logo.svg +0 -0
  33. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/run_tests.sh +0 -0
  34. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/FastSweep.java +0 -0
  35. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/FastSweepExtract.java +0 -0
  36. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/fast_sweep.py +0 -0
  37. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_block_in_inline.py +0 -0
  38. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_css_illegal_tags.py +0 -0
  39. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_duplicate_playorder.py +0 -0
  40. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_empty_body.py +0 -0
  41. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_illegal_tags.py +0 -0
  42. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_invalid_ids.py +0 -0
  43. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_invalid_st_tag.py +0 -0
  44. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_invalid_value_attr.py +0 -0
  45. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_missing_images.py +0 -0
  46. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/find_missing_title.py +0 -0
  47. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/scripts/sweep.sh +0 -0
  48. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/__main__.py +0 -0
  49. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/src/bindery/validate.py +0 -0
  50. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/agda.epub +0 -0
  51. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/agda_fixed.epub +0 -0
  52. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/last_man_out.epub +0 -0
  53. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/last_man_out_fixed.epub +0 -0
  54. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/leaves_of_grass.epub +0 -0
  55. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/leaves_of_grass_fixed.epub +0 -0
  56. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/leaves_of_grass_st_fixed.epub +0 -0
  57. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/leaves_of_grass_tags_fixed.epub +0 -0
  58. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/muse_of_nightmares.epub +0 -0
  59. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/muse_of_nightmares_fixed.epub +0 -0
  60. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/poor_mans_fight.epub +0 -0
  61. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/poor_mans_fight_fixed.epub +0 -0
  62. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_block_in_inline.py +0 -0
  63. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_duplicate_playorder.py +0 -0
  64. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_empty_body.py +0 -0
  65. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_illegal_tags.py +0 -0
  66. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_invalid_ids.py +0 -0
  67. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_invalid_value.py +0 -0
  68. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/test_missing_title.py +0 -0
  69. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/unworthy_gods.epub +0 -0
  70. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/test_facility/unworthy_gods_fixed.epub +0 -0
  71. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/testing_facility/books_to_fix/.gitkeep +0 -0
  72. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/testing_facility/top500candidates/REPORT.md +0 -0
  73. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_validate.py +0 -0
  74. {bindery_cli-0.31.0 → bindery_cli-0.32.0}/tests/test_version.py +0 -0
@@ -34,11 +34,22 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
34
34
  minimal content. v0.14–v0.16 ran these unconditionally, which broke this rule;
35
35
  v0.17.0 restored it. `--unwrap-illegal-tags` additionally protects any illegal-tag
36
36
  name that an EPUB stylesheet styles as an element selector (`css_protected_tags`,
37
- book-wide, inline `<style>` blocks included).
37
+ book-wide, inline `<style>` blocks included). The two EPUB2-targeted fixes
38
+ (`--strip-epub3-attrs`, `--downgrade-epub3-tags`) are additionally gated on the
39
+ package version in the OPF (`package_version()`): inert on EPUB 3 books and when
40
+ no version can be read, since their target defects only exist below EPUB 3; the
41
+ attribute scrub is anchored to real start tags via
42
+ `transforms.strip_attrs_in_start_tags` (prose mentioning the attributes and
43
+ CDATA/comment content are preserved), and `strip_invalid_value` matches the
44
+ attribute name with a `(?<![\w:.-])value` lookbehind so `data-value` survives.
38
45
  * **Lossy strips** (`--strip-pagination`, `--strip-broken-tags`, `--strip-watermarks`;
39
46
  pagination.py, watermark.py): remove only what a converter injected (page numbers,
40
47
  running headers, leaked tags, watermarks), fenced behind character conservation, tag
41
- balance, and the epubcheck no-regression bar.
48
+ balance, and the epubcheck no-regression bar. The watermark anchored pass's
49
+ whole-match fallback only fires when the match holds nothing but the stamp; a
50
+ larger match (an unclosed stamp anchor that swallowed prose) is refused, counted
51
+ in `RepairReport.watermark_refusals`, and surfaced as a `manual_watermark_repair`
52
+ decision by `run phase1` on both the read-only and apply paths.
42
53
  Do not let any NEW fix touch content without its own flag; if a candidate repair cannot
43
54
  be made deterministically safe, it does not belong here — report it for manual repair
44
55
  instead.
@@ -58,14 +69,14 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
58
69
 
59
70
  ## Layout
60
71
 
61
- - `src/bindery/transforms.py`: pure `str -> (str, int)` text transforms (including `strip_broken_tags`).
72
+ - `src/bindery/transforms.py`: pure `str -> (str, int)` text transforms (including `strip_broken_tags`). Since v0.32.0: `strip_attrs_in_start_tags` anchors attribute edits on the quote-aware start-tag matcher, `_outside_protected` forwards arguments (context-carrying transforms can be decorated), `fix_id_colons` matches only the bare `id` attribute and only internal fragments (`fix_ncx_src_fragments` carries renames into the NCX), `fix_ncx_playorder` is anchored to `<navPoint>` start tags, and the css selector boundary covers namespaced and functional selector forms.
62
73
  - `src/bindery/pagination.py`: the opt-in lossy page-number strip (runhead detection, page-layer decision, block-centric removal/merge, safety nets).
63
74
  - `src/bindery/watermark.py`: the opt-in lossy watermark strip (anchored and anchorless signature removal).
64
75
  - `src/bindery/reserialize.py`: structural repair via `html5lib`.
65
76
  - `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, and `monolithic` since v0.21.0, `--max-doc-chars N`) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story).
66
77
  - `src/bindery/epub.py`: archive rewrite, NCX uid sync, RepairReport, mismatch detection, and the opt-in structural-repair plumbing (including the CSS precondition scan).
67
78
  - `src/bindery/validate.py`: epubcheck wrapper, the `gate` (improvement) and `no_worse` (no-regression, for the lossy strips) acceptance bars.
68
- - `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching) — the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (remove+add in one `batch()`; `add_format` refuses duplicates); the external `calibredb` CLI is gone. Never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
79
+ - `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching) — the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (since v0.31.0 via `set_format`, cquarry 1.17's sanctioned remove+add in one transaction); the external `calibredb` CLI is gone. Since v0.32.0 a guessed id drives a row update only when metadata.db corroborates it (`_verify_guess`: books row exists, file in the book's own directory, catalogued `data.name` when a row exists); anything else saves in place and leaves the catalog untouched — never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
69
80
  - `src/bindery/cli.py`: `repair` and `library` subcommands, including `--all` and `--install-to-calibre`; since v0.29.0 also the `run` verbs: `phase1` (pre-import vetting over loose files: the audit battery composed with one gated repair sweep; read-only until `--apply-lossy`, which IS the lossy-strip consent) and `phase3` (the scoped post-import apply step: `library --id --sweep --only all --apply --all --install-to-calibre` plus a pre/post summary; mechanically refuses unscoped sweeps with exit 2). Both take `--non-interactive` and surface open questions as `decisions_needed` in their JSON, never prompts. They drive the shipped `run_library` through the real parser, so no flag can drift between the verb and the subcommand it wraps. `library --workers N` (v0.30.0) parallelizes only the sweep's candidate pass, in windows of N consumed in order so `--limit` stays lazy; the repair phase is never parallel (shared workdir, atomic-replacement contract).
70
81
  - `tests/`: transforms, end-to-end repair, atomic replace, pagination, watermarks, the audit analyzers (content/pagenumbers/emptytext/ocr/monolithic, archive integrity, spine classification), the library sweep and `CalibreIdResolver`, validate, reserialize, CLI wiring, and the version pin.
71
82
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: bindery-cli
3
- Version: 0.31.0
3
+ Version: 0.32.0
4
4
  Summary: Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library.
5
5
  Author: vrnvctss
6
6
  License: MIT
@@ -51,14 +51,14 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
51
51
  - **`--reserialize`**: rebuild content documents that are still malformed by re-parsing them with html5lib and re-emitting XHTML, closing unclosed `<p>`/`<div>`/`<span>`/`<blockquote>` that the regex transforms cannot. Runs only on documents that are not already well-formed, so good files are untouched.
52
52
  - **`--strip-bad-attrs`**: drop attributes that are invalid XML (a name starting with a digit, or a namespaced name whose prefix is never declared, like Office VML `v:shapes`). Surgical and a no-op on well-formed files.
53
53
  - **`--escape-unknown-entities`**: escape entity names that are not in the HTML5 table (`&foo;` becomes `&amp;foo;`), which renders exactly as browsers already render an unknown entity. Documents whose DOCTYPE carries an internal subset are skipped wholesale, since a subset can declare custom entities.
54
- - **`--fix-id-colons`**: translates illegal colons in `id="X:Y"` and their matching `#X:Y` fragment references to valid underscores.
54
+ - **`--fix-id-colons`**: translates illegal colons in `id="X:Y"` and their matching internal `#X:Y` fragment references to valid underscores. External URL fragments are left verbatim (they name positions in other documents), `data-id` values are never touched, and the NCX's `content src` fragments follow the rename so the table of contents never dangles against the ids it references.
55
55
  - **`--fix-empty-body`**: appends a non-breaking space `&nbsp;` to strictly empty body tags to satisfy parser requirements.
56
56
  - **`--fix-missing-title`**: injects a `<title>Unknown</title>` fallback in the `<head>` if missing, handling both empty `<title/>` self-closing tags and entirely absent tags.
57
57
  - **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
58
58
  - **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
59
59
  - **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
60
- - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive).
61
- - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed.
60
+ - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
61
+ - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
62
62
  - **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
63
63
  - **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
64
64
  - **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
@@ -67,9 +67,9 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
67
67
 
68
68
  Three opt-in fixes are **lossy** and stand apart from the semantics-preserving rest:
69
69
 
70
- - **`--strip-pagination`**: remove print page numbers and running headers that a PDF/OCR conversion baked into the body text as literal paragraphs (so they reflow into the middle of a sentence: "where the hay cart **16** was taking him"). It removes only that injected furniture, never the author's prose: where a number split a sentence it rejoins the two paragraphs (closing up a word split like `compli-` / `mentary`), and it preserves roman chapter numbers, page-list nav anchors, and years. A book is only treated as paginated when it has both a dense run of arabic numbers and several confident mid-sentence interrupts, so a merely chapter-numbered book is left alone. Three safety nets guard every edit (character conservation, tag balance, and an epubcheck no-regression check); any failure leaves the document untouched.
70
+ - **`--strip-pagination`**: remove print page numbers and running headers that a PDF/OCR conversion baked into the body text as literal paragraphs (so they reflow into the middle of a sentence: "where the hay cart **16** was taking him"). It removes only that injected furniture, never the author's prose: where a number split a sentence it rejoins the two paragraphs (closing up a word split like `compli-` / `mentary`), and it preserves roman chapter numbers, page-list nav anchors, and years. A roman numeral counts as a page number only when it is well-formed and under 100, so ordinary words (`mid`, `mix`, `civil`) are never mistaken for pages. A book is only treated as paginated when it has both a dense run of arabic numbers and several confident mid-sentence interrupts, so a merely chapter-numbered book is left alone. Three safety nets guard every edit (character conservation, tag balance, and an epubcheck no-regression check); any failure leaves the document untouched.
71
71
  - **`--strip-broken-tags`**: remove leaked HTML closing tags missing their open brackets (e.g. `</p>`) that render as raw text.
72
- - **`--strip-watermarks`**: remove known producer and distributor watermarks (e.g. OceanofPDF, ABC Amber LIT Converter) and stray marker files. It locates the stamp and deletes the outermost wrapper whose entire visible text is the watermark, ensuring prose that merely mentions the URL is preserved.
72
+ - **`--strip-watermarks`**: remove known producer and distributor watermarks (e.g. OceanofPDF, ABC Amber LIT Converter) and stray marker files. It locates the stamp and deletes the outermost wrapper whose entire visible text is the watermark, ensuring prose that merely mentions the URL is preserved. An inline stamp link is deleted only when it holds nothing but the stamp; a match too large to be safe (an unclosed stamp anchor that swallowed prose) is refused and reported for manual repair rather than deleted.
73
73
 
74
74
  All lossy edits are invisible to epubcheck, so they are accepted when the result is *no worse* rather than measurably better.
75
75
 
@@ -200,7 +200,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
200
200
  - `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
201
201
  - `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
202
202
  - `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
203
- - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning.
203
+ - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
204
204
  - `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
205
205
  - Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
206
206
  - A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
@@ -29,14 +29,14 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
29
29
  - **`--reserialize`**: rebuild content documents that are still malformed by re-parsing them with html5lib and re-emitting XHTML, closing unclosed `<p>`/`<div>`/`<span>`/`<blockquote>` that the regex transforms cannot. Runs only on documents that are not already well-formed, so good files are untouched.
30
30
  - **`--strip-bad-attrs`**: drop attributes that are invalid XML (a name starting with a digit, or a namespaced name whose prefix is never declared, like Office VML `v:shapes`). Surgical and a no-op on well-formed files.
31
31
  - **`--escape-unknown-entities`**: escape entity names that are not in the HTML5 table (`&foo;` becomes `&amp;foo;`), which renders exactly as browsers already render an unknown entity. Documents whose DOCTYPE carries an internal subset are skipped wholesale, since a subset can declare custom entities.
32
- - **`--fix-id-colons`**: translates illegal colons in `id="X:Y"` and their matching `#X:Y` fragment references to valid underscores.
32
+ - **`--fix-id-colons`**: translates illegal colons in `id="X:Y"` and their matching internal `#X:Y` fragment references to valid underscores. External URL fragments are left verbatim (they name positions in other documents), `data-id` values are never touched, and the NCX's `content src` fragments follow the rename so the table of contents never dangles against the ids it references.
33
33
  - **`--fix-empty-body`**: appends a non-breaking space `&nbsp;` to strictly empty body tags to satisfy parser requirements.
34
34
  - **`--fix-missing-title`**: injects a `<title>Unknown</title>` fallback in the `<head>` if missing, handling both empty `<title/>` self-closing tags and entirely absent tags.
35
35
  - **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
36
36
  - **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
37
37
  - **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
38
- - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive).
39
- - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed.
38
+ - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
39
+ - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
40
40
  - **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
41
41
  - **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
42
42
  - **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
@@ -45,9 +45,9 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
45
45
 
46
46
  Three opt-in fixes are **lossy** and stand apart from the semantics-preserving rest:
47
47
 
48
- - **`--strip-pagination`**: remove print page numbers and running headers that a PDF/OCR conversion baked into the body text as literal paragraphs (so they reflow into the middle of a sentence: "where the hay cart **16** was taking him"). It removes only that injected furniture, never the author's prose: where a number split a sentence it rejoins the two paragraphs (closing up a word split like `compli-` / `mentary`), and it preserves roman chapter numbers, page-list nav anchors, and years. A book is only treated as paginated when it has both a dense run of arabic numbers and several confident mid-sentence interrupts, so a merely chapter-numbered book is left alone. Three safety nets guard every edit (character conservation, tag balance, and an epubcheck no-regression check); any failure leaves the document untouched.
48
+ - **`--strip-pagination`**: remove print page numbers and running headers that a PDF/OCR conversion baked into the body text as literal paragraphs (so they reflow into the middle of a sentence: "where the hay cart **16** was taking him"). It removes only that injected furniture, never the author's prose: where a number split a sentence it rejoins the two paragraphs (closing up a word split like `compli-` / `mentary`), and it preserves roman chapter numbers, page-list nav anchors, and years. A roman numeral counts as a page number only when it is well-formed and under 100, so ordinary words (`mid`, `mix`, `civil`) are never mistaken for pages. A book is only treated as paginated when it has both a dense run of arabic numbers and several confident mid-sentence interrupts, so a merely chapter-numbered book is left alone. Three safety nets guard every edit (character conservation, tag balance, and an epubcheck no-regression check); any failure leaves the document untouched.
49
49
  - **`--strip-broken-tags`**: remove leaked HTML closing tags missing their open brackets (e.g. `</p>`) that render as raw text.
50
- - **`--strip-watermarks`**: remove known producer and distributor watermarks (e.g. OceanofPDF, ABC Amber LIT Converter) and stray marker files. It locates the stamp and deletes the outermost wrapper whose entire visible text is the watermark, ensuring prose that merely mentions the URL is preserved.
50
+ - **`--strip-watermarks`**: remove known producer and distributor watermarks (e.g. OceanofPDF, ABC Amber LIT Converter) and stray marker files. It locates the stamp and deletes the outermost wrapper whose entire visible text is the watermark, ensuring prose that merely mentions the URL is preserved. An inline stamp link is deleted only when it holds nothing but the stamp; a match too large to be safe (an unclosed stamp anchor that swallowed prose) is refused and reported for manual repair rather than deleted.
51
51
 
52
52
  All lossy edits are invisible to epubcheck, so they are accepted when the result is *no worse* rather than measurably better.
53
53
 
@@ -178,7 +178,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
178
178
  - `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
179
179
  - `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
180
180
  - `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
181
- - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning.
181
+ - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
182
182
  - `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
183
183
  - Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
184
184
  - A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
@@ -1,4 +1,113 @@
1
1
  # bindery-cli Patch Notes
2
+ ## v0.32.0 (2026-09-10)
3
+
4
+ ### The four Bug Reports fixes (2026-09-08 sweep) and the anchored-attribute cluster
5
+
6
+ - **`--strip-watermarks` refuses anchored matches larger than the stamp**
7
+ (the sweep demonstrated two paragraphs of real prose vanishing with
8
+ count 1). The anchored stamp regex is DOTALL with no tag budget, so an
9
+ unclosed stamp `<a>` matched through real prose to the next unrelated
10
+ `</a>`, and the whole-match fallback deleted everything in between;
11
+ epubcheck cannot see deleted text, so the no-regression bar accepted the
12
+ loss. The fallback now fires only when the match demonstrably holds
13
+ nothing but the stamp (a tag-free body within stamp length, or visible
14
+ text exactly the watermark), a second `<a>` opening inside the match
15
+ refuses it outright, and anything larger stays in place, counted in the
16
+ new `RepairReport.watermark_refusals` and surfaced by `run phase1` as a
17
+ `manual_watermark_repair` decision on both the read-only and apply paths.
18
+ Safe shapes (pure wrappers, inline stamp-only links) are byte-identical
19
+ to before.
20
+ - **Roman page-number detection reads well-formed numerals only**, in both
21
+ `--strip-pagination` and the audit pagenumbers analyzer. The old
22
+ `[ivxlcdm]{2,7}` character set read ordinary English words as page
23
+ numbers (`mid` = 1499), and a standalone `<p>mid</p>` before a
24
+ lowercase-starting paragraph is exactly the confident interrupt shape,
25
+ so real words were deleted and merged away. A roman counts only when
26
+ well-formed (explicit subtractive pairs) and valued under 100, where
27
+ front-matter page numbers live and words do not (`mix` = M+IX = 1009).
28
+ Lowercase front-matter romans keep working; the audit copy's deliberately
29
+ different year handling is untouched.
30
+ - **`install_format` verifies directory-guessed ids against metadata.db.**
31
+ In directory mode, an uncatalogued stray `.epub` inside a book directory
32
+ made the `(id)` directory guess fire, and the row update re-registered
33
+ the catalogued format name with the stray file's size: the catalogued
34
+ 999-byte file untouched on disk while `data` recorded 8123 bytes. A
35
+ stale `(N)` directory for a deleted book crashed the sweep instead. A
36
+ guessed id now drives a row update only when metadata.db corroborates it
37
+ (the books row exists, the file sits in that book's own directory, and
38
+ when an EPUB row exists it carries the row's stored name); anything else
39
+ saves the repair in place with a warning and leaves the catalog
40
+ untouched. The resolver path needs no verification: its id comes from
41
+ the catalog's own path map.
42
+ - **The EPUB2-targeted fixes are gated on the package version.** Neither
43
+ `--strip-epub3-attrs` nor `--downgrade-epub3-tags` checked what the
44
+ package declared, so under `--all` every EPUB3 book in a sweep took the
45
+ EPUB2 cure: a repairable EPUB3 book gained a net-new error from a
46
+ stripped nav `epub:type` (whole repair rejected), and a legal
47
+ `epub:type="chapter"` was silently removed because the gate only sees
48
+ epubcheck counts. Both fixes now fire only on EPUB 2 packages (major
49
+ version 1 or 2) and are inert on EPUB 3 packages and when no version can
50
+ be read, where their target defects do not exist.
51
+ - **The anchored-attribute regex cluster closes with the gate** (the same
52
+ sweep's sharpest repair-pipeline finding): the attribute scrub runs
53
+ anchored on real start tags through the new
54
+ `transforms.strip_attrs_in_start_tags`, with CDATA sections and comments
55
+ protected, so prose mentioning `epub:type="chapter"` is no longer
56
+ deleted down to `Use here`; `strip_invalid_value` matches its attribute
57
+ name behind a `(?<![\w:.-])value` lookbehind, so `data-value="42"`
58
+ survives instead of becoming a malformed `<span data->`.
59
+
60
+ ### Repair-pipeline correctness (Phase 14, the 2026-09-08 sweep's backlog)
61
+
62
+ - **`--fix-id-colons` is consistent and honest.** Only the bare `id`
63
+ attribute is in scope (a lookbehind keeps `data-id` and `xml:id` values
64
+ untouched), the fragment of an external URL survives verbatim (it names a
65
+ position in another document), the NCX's `content src` fragments follow
66
+ the rename through the new `fix_ncx_src_fragments` so a ToC never dangles
67
+ against the ids it references, and only actual colons replaced are
68
+ counted (byte-identical reruns report 0 and no longer trigger the
69
+ re-encode the untouched-file guard exists to prevent).
70
+ - **The anchor pass runs last.** `strip_broken_anchors`' id snapshot used
71
+ to predate the unwrap fixes, so an id they deleted still counted as
72
+ present and fragments pointing at it survived dangling. The anchor pass
73
+ now runs after every content fix, against a snapshot that replicates
74
+ each id-moving fix ahead of it.
75
+ - **CDATA sections and comments are protected everywhere.**
76
+ `strip_broken_tags` and `unwrap_illegal_tags` joined the module's
77
+ protected-span machinery (the wrapper now forwards arguments, so the
78
+ protected_tags-carrying unwrap could be decorated); a leaked-looking tag
79
+ inside either is literal rendered text, not markup.
80
+ - **Non-UTF-8 documents are skipped verbatim.** A fix firing on a
81
+ windows-1252 or UTF-16 document used to decode with U+FFFD substitutes
82
+ and re-encode UTF-8 under a declaration still naming the old encoding:
83
+ usually well-formed, so the gate could not see the corruption. The write
84
+ loop strict-decodes first; failures are copied byte-for-byte, counted in
85
+ the new `non_utf8_docs_skipped` entry, and left for manual repair.
86
+ - **`--reserialize` requires an html root.** A broken non-HTML `.xml`
87
+ sidecar used to come back html/body-wrapped with `ns0:` prefixes,
88
+ structurally rewritten while staying well-formed; it is now returned
89
+ untouched.
90
+ - **Seven papercuts.** `fix_ncx_playorder` is anchored to `<navPoint>`
91
+ start tags (nav-label text is untouchable); mimetype fixes increment
92
+ `files_changed`; `fix_manifest_ids` and its reference rewrites are
93
+ quote-agnostic (a single-quoting OPF used to be a silent count-0 no-op;
94
+ the two shadowed `_IDREF_ATTR_RE` definitions collapsed into one);
95
+ the css selector boundary recognizes `svg|st`, `:is(st, w)`, and
96
+ `:where(...)` forms and the stylesheet scan collects `.xpgt` page
97
+ templates, so styled tags keep their unwrap protection; watermark
98
+ normalization decodes entities (`OceanofPDF.com&nbsp;` stamps now
99
+ match); the prolog and duplicate-xmlns fixes run under the
100
+ protected-span wrapper; and the archive comment field rides the rewrite.
101
+
102
+ Dry-run validation for this release: the 14 staged top500candidates plus
103
+ the 3 books_to_fix fixtures swept clean (9 accepted, 2 gate-refused
104
+ regressions, 0 errors), and a 200-candidate sample of the real library
105
+ (5,228 EPUBs) under `--all --workers 4` matched the documented verdict
106
+ mix (94 accepted, 84 equal, 16 gate-refused, 0 partial, 0 epubcheck
107
+ errors, 0 unreadable) with zero crashes. Read-only throughout; the
108
+ gate-refused books are the safety contract declining marginal repairs,
109
+ left untouched for manual work.
110
+
2
111
  ## v0.31.0 (2026-09-09)
3
112
 
4
113
  ### Cascade: install through cquarry 1.17's set_format
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "bindery-cli"
7
- version = "0.31.0"
7
+ version = "0.32.0"
8
8
  description = "Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library."
9
9
  readme = "README.md"
10
10
  classifiers = [
@@ -619,7 +619,17 @@ real content vanish with count 1; the output is well-formed with equal or
619
619
  fewer fatals, so the `no_worse` bar accepts it. `watermark.py:207-213`.
620
620
  **Fix:** require the anchored match to be tag-free (or stamp-length visible
621
621
  text) before falling back to whole-match deletion; anything larger goes to
622
- `decisions_needed` for manual repair. *(Unfixed as of the 2026-09-08 sweep.)*
622
+ `decisions_needed` for manual repair. *(Fixed in v0.32.0: the fallback now
623
+ fires only when the match holds nothing but the stamp, a tag-free body within
624
+ stamp length or a body whose visible text is exactly the watermark, and a
625
+ second `<a>` opening inside the match refuses it outright (the close belongs
626
+ to that other anchor). Larger matches are left in place and counted in
627
+ `RepairReport.watermark_refusals`, surfaced in every sweep summary and as a
628
+ `manual_watermark_repair` decision by `run phase1` on both the read-only and
629
+ apply paths. The anchored pass now decides all matches left to right and
630
+ applies deletions right to left, so a refusal can no longer be re-counted on
631
+ rescan. Regression tests pin the reported two-paragraph loss, the tag-free
632
+ long swallow, and the safe inline shapes.)*
623
633
 
624
634
  ### roman-numeral detector reads ordinary English words as page numbers (2026-09-08 sweep)
625
635
  **Bug in `--strip-pagination`:** `_ROMAN_RE = [ivxlcdm]{2,7}\Z` (IGNORECASE)
@@ -629,8 +639,15 @@ paragraph is a confident page-number hit and is deleted and merged away.
629
639
  epubcheck is blind to deleted text, so `no_worse` passes. `pagination.py:44`,
630
640
  `70-78`, `180-184`.
631
641
  **Fix:** require uppercase romans, or validate with a strict roman grammar
632
- (`M?(D?C{0,3}|C[MD])(X?L{0,3}|L?X{0,3})(I?V{0,3}|V?I{0,3})`). *(Unfixed as of
633
- the 2026-09-08 sweep.)*
642
+ (`M?(D?C{0,3}|C[MD])(X?L{0,3}|L?X{0,3})(I?V{0,3}|V?I{0,3})`). *(Fixed in
643
+ v0.32.0, taking the grammar option to keep lowercase front-matter romans
644
+ working, with one addition the demonstrated words forced: the grammar alone
645
+ still accepts `mix` (M+IX = 1009), so a roman is a page number only when it is
646
+ well-formed AND its value stays under 100, where real roman page numbers live
647
+ and words do not. The audit pagenumbers analyzer's verbatim-twin detector got
648
+ the same fix; the audit copy's deliberately different year handling is
649
+ untouched (still its own question). Tests pin the word list, the surviving
650
+ numerals, and the cap.)*
634
651
 
635
652
  ### install_format can write a wrong size into metadata.db for stray files (2026-09-08 sweep)
636
653
  **Bug in `library --apply --install-to-calibre` without `--id`:**
@@ -645,7 +662,18 @@ book instead raises an uncaught `TypeError` at the `book["path"]` access.
645
662
  **Fix:** when the id came from the directory guess, verify the target
646
663
  matches the DB's book path and format name before the batch; otherwise save
647
664
  in place and warn without touching the row, and catch the `TypeError` path.
648
- *(Unfixed as of the 2026-09-08 sweep.)*
665
+ *(Fixed in v0.32.0: a guessed id now drives a row update only when
666
+ metadata.db corroborates it — the books row exists, the file sits in that
667
+ book's catalogued directory, and when an EPUB row exists it carries that
668
+ row's stored name. Any other guess (a stray file in a book directory, a
669
+ stale `(N)` directory whose book is gone, the resolver's id gone from
670
+ `books` mid-run) saves the repair in place with a warning and leaves the
671
+ catalog untouched; the `TypeError` on the missing books row is gone. The
672
+ existing verified-guess fresh-placement and wrong-directory-degrades tests
673
+ are unchanged in behavior. Also rode along here: spec.md's install section
674
+ still described the v0.31.0-retired remove+add `batch()`; it now describes
675
+ `set_format`. Tests pin the stray-size incident shape and the stale-directory
676
+ save.)*
649
677
 
650
678
  ### EPUB2-targeted structural fixes fire on EPUB3 books under --all (2026-09-08 sweep)
651
679
  **Bug in `--strip-epub3-attrs` / `--downgrade-epub3-tags` via the run verbs
@@ -659,7 +687,14 @@ epubcheck counts. phase1 and phase3 always pass `--all`.
659
687
  `epub.py:251-260`, `903-937`, `957-967`; `cli.py:830`, `976-987`.
660
688
  **Fix:** gate both fixes on the package version carried in `opf_text`, or
661
689
  make them finding-driven (only strip what the before-pass epubcheck flagged).
662
- *(Unfixed as of the 2026-09-08 sweep.)*
690
+ *(Fixed in v0.32.0, taking the version-gate option: both fixes are licensed
691
+ by `package_version(opf_text)` and are inert on EPUB 3 packages and when no
692
+ version can be read. This kills both demonstrated outcomes: the repairable
693
+ EPUB3 book no longer gains a net-new error from a stripped nav `epub:type`,
694
+ and the legal count-neutral `epub:type="chapter"` survives. The finding-
695
+ driven alternative was rejected because it would still silently mutate legal
696
+ markup whenever an unrelated finding existed; the gate keeps EPUB3 books
697
+ byte-identical.)*
663
698
 
664
699
  ## Phase 11: Migrate install-to-calibre to native cquarry API (proposed 2026-08-31)
665
700
 
@@ -837,7 +872,7 @@ can be net-neutral and ships silently.*
837
872
 
838
873
  ### Repair-pipeline correctness
839
874
 
840
- - [ ] **Anchor attribute-name regexes to real start tags.** Three findings
875
+ - [x] **Anchor attribute-name regexes to real start tags.** Three findings
841
876
  share one root cause: bare attribute-name regexes that are not
842
877
  quote-aware and not tag-anchored. `strip_invalid_value` matches the
843
878
  `value` in `data-value="42"` (the `\b` matches between `-` and `v`)
@@ -848,7 +883,14 @@ can be net-neutral and ships silently.*
848
883
  contradicting that fix's own docstring. Reuse the quote-aware
849
884
  start-tag matcher + protected-span machinery the module already has,
850
885
  and a `(?<![\w:.-])value` lookbehind.
851
- - [ ] **Make `fix_id_colons` consistent and honest.** It rewrites fragments
886
+ *(Done in v0.32.0: `strip_epub3_attributes` now runs through the new
887
+ `transforms.strip_attrs_in_start_tags`, which anchors the edit on the
888
+ module's quote-aware `_START_TAG_RE` and never touches CDATA/comments;
889
+ `strip_invalid_value` gained the `(?<![\w:.-])value` lookbehind, so
890
+ `data-value`, `xvalue`, and `xml:value` all survive untouched. Tests
891
+ pin the prose-mention case, CDATA/comment protection, multi-line start
892
+ tags, and the lookalike attribute names.)*
893
+ - [x] **Make `fix_id_colons` consistent and honest.** It rewrites fragments
852
894
  of external URLs (`http://example.com/page#sec:1` becomes `#sec_1`,
853
895
  breaking the link; `transforms.py:524`, `536`, docstring at 509 claims
854
896
  otherwise) and rewrites colon-bearing `data-id`/`xml:id`; it never
@@ -857,25 +899,55 @@ can be net-neutral and ships silently.*
857
899
  (`epub.py:976-979` vs the NCX branch at `873-901`); and it counts
858
900
  every matched `id` attribute even when no colon was replaced, so
859
901
  byte-identical reruns report phantom changes (`transforms.py:513-516`).
860
- - [ ] **Run anchor stripping last and protect CDATA/comments everywhere.**
902
+ *(Done in v0.32.0: only the bare `id` attribute is in scope (a
903
+ `(?<![\w:.-])` lookbehind keeps `data-id` and `xml:id` untouched), a
904
+ fragment is translated only when the target is not an absolute URI or
905
+ protocol-relative reference (`_EXTERNAL_TARGET_RE`), the new
906
+ `fix_ncx_src_fragments` runs in the NCX branch under `--fix-id-colons`
907
+ so ToC `content src` fragments follow the rename, and the id replacer
908
+ counts only actual colons replaced, so byte-identical reruns report 0
909
+ and the untouched-file guard holds. Live evidence for this box showed
910
+ up in the v0.32.0 dry run itself: two staged books were gate-REJECTED
911
+ on `fix_id_colons` regressions manufactured the old way. Tests pin
912
+ the external-URL, data-id/xml:id, phantom-count, and NCX-follow
913
+ shapes.)*
914
+ - [x] **Run anchor stripping last and protect CDATA/comments everywhere.**
861
915
  `strip_broken_anchors`' id snapshot predates `unwrap_block_in_inline`
862
916
  and `unwrap_illegal_tags`, which can delete ids the snapshot thinks
863
917
  exist (`epub.py:980-997`); `strip_broken_tags` and
864
918
  `unwrap_illegal_tags` are not wrapped in the module's own
865
919
  protected-span machinery, so they edit inside CDATA (rendered text)
866
920
  and comments (`transforms.py:444-450`, `619-623`).
867
- - [ ] **Stop re-encoding non-UTF-8 documents.** Any fix that fires on a
921
+ *(Done in v0.32.0: the anchor pass now runs last in the content
922
+ pipeline, and the id snapshot pre-pass replicates every id-moving fix
923
+ ahead of it (reserialize, id-colon renames, both unwraps, the
924
+ resource prune), so a fragment whose target an earlier fix deleted is
925
+ stripped instead of left dangling; `_outside_protected` now forwards
926
+ arguments, so `strip_broken_tags` and `unwrap_illegal_tags` (with its
927
+ protected_tags set) are both decorated and never edit inside CDATA or
928
+ comments. Tests pin the dangling-fragment shape and the protected
929
+ spans for both transforms.)*
930
+ - [x] **Stop re-encoding non-UTF-8 documents.** Any fix that fires on a
868
931
  windows-1252 or UTF-16 document decodes with `replace` and re-encodes
869
932
  UTF-8, materializing mojibake under an XML declaration that still
870
933
  names the old encoding; usually well-formed, so the gate cannot see
871
934
  it (`epub.py:874`, `902`, `939`, `1027`). Detect and skip with a
872
935
  report entry, per the project's manual-repair philosophy.
873
- - [ ] **Scope `--reserialize` to HTML roots.** A broken non-HTML `.xml`
936
+ *(Done in v0.32.0: the write loop strict-decodes each NCX, OPF, and
937
+ content document before any fix runs; a document that is not valid
938
+ UTF-8 is copied byte-for-byte, counted in the new
939
+ `non_utf8_docs_skipped` report entry, and left for manual repair.
940
+ UTF-8 siblings in the same book are still repaired.)*
941
+ - [x] **Scope `--reserialize` to HTML roots.** A broken non-HTML `.xml`
874
942
  sidecar gets html5lib's HTML algorithm and comes back
875
943
  html/body-wrapped with `ns0:` prefixes, structurally rewritten while
876
944
  staying well-formed (`epub.py:938`, `reserialize.py:39-42`). Require
877
945
  an `html` root (or manifest media-type) before rebuilding.
878
- - [ ] **Smaller repair papercuts:** `fix_ncx_playorder` can rewrite
946
+ *(Done in v0.32.0, taking the html-root check: `reserialize_if_broken`
947
+ requires an `<html` root before it will rebuild, so a broken page
948
+ template or other XML sidecar is returned untouched instead of
949
+ structurally rewritten. Tests pin the sidecar shape.)*
950
+ - [x] **Smaller repair papercuts:** `fix_ncx_playorder` can rewrite
879
951
  `playOrder="N"`-shaped text inside nav labels (unanchored pattern,
880
952
  `transforms.py:326-328`); mimetype fixes land in `report.fixes`
881
953
  without incrementing `files_changed` (`epub.py:837-840`);
@@ -886,6 +958,17 @@ can be net-neutral and ships silently.*
886
958
  watermark `_norm` doesn't unescape entities so `OceanofPDF.com&nbsp;`
887
959
  stamps are missed; drop_duplicate_xmlns/strip_prolog_junk edit inside
888
960
  comments; the archive comment field is dropped by the rewrite.
961
+ *(Done in v0.32.0, all seven: playorder resequencing is anchored to
962
+ `<navPoint>` start tags so nav-label text is untouchable; mimetype
963
+ fixes increment `files_changed`; `fix_manifest_ids` and its reference
964
+ rewrites are quote-agnostic (the shadowed `_IDREF_ATTR_RE` definitions
965
+ were unified into one quote-aware regex); the css selector boundary
966
+ recognizes `svg|st`, `:is(st, w)`, and `:where(...)` forms, and the
967
+ stylesheet scan collects `.xpgt` templates too; watermark `_norm`
968
+ decodes entities so `OceanofPDF.com&nbsp;` pure-matches; both
969
+ prolog/xmlns fixes run under the protected-span wrapper; and the
970
+ archive comment rides the rewrite (`zout.comment = zin.comment`).
971
+ Tests pin every one.)*
889
972
 
890
973
  ### audit.py correctness
891
974
 
@@ -1065,3 +1148,24 @@ wired into every mode it promises, and the apply path's failure modes (crash
1065
1148
  mid-run, backup clobber, daemon stall) are unpadded. Fix the four Bug
1066
1149
  Reports entries and the anchored-regex cluster before any further features;
1067
1150
  everything else is backlog, not danger.*
1151
+
1152
+ ## Phase 15: completeness spot-check analyzer (proposed 2026-09-10, from the phase-1 skill's own doctrine)
1153
+
1154
+ *The phase-1 skill's step 2 has a judgment step with no tool owner: the
1155
+ completeness spot-check ("sample early, middle, and late pages ... read the
1156
+ LAST content page to confirm it reaches real back matter rather than cutting
1157
+ off mid-chapter"). Two consecutive phase-1 runs (2026-09-08 Strauss batch,
1158
+ 2026-09-10 Redwall batch) used a hand-rolled zipfile sampler in /tmp for it,
1159
+ which is exactly the "hand-rolled substitute becomes permanent" pattern the
1160
+ skill forbids; the skill allows only two inline checks, both with filed
1161
+ phases here. The sampler also already needed one repair (href unquoting)
1162
+ that the shipped audit gets for free. Proposal: an `audit completeness`
1163
+ analyzer that, in the audit's existing single decompression pass, reports
1164
+ per book: spine doc count, prose-doc count (>400 visible chars), the first /
1165
+ middle / last prose doc's opening and closing text, trailing-ToC detection
1166
+ (a final spine doc that is mostly chapter-heading lines), and the fraction of
1167
+ unreadable docs. The 2026-09-10 Redwall run is the fixture set: Lord
1168
+ Brocktree (real Epilogue doc before a trailing 520-char ToC), Mattimeo
1169
+ (Chapter 50 verified as a real heading with prose after, inside a 174k-char
1170
+ split doc), The Long Patrol (percent-encoded hrefs defeated the hand-rolled
1171
+ reader; 55/57 prose docs).*
@@ -91,8 +91,12 @@ part of the default pipeline:
91
91
  incomplete"). Adds visible content, hence opt-in like `--add-img-alt`.
92
92
  - **`--fix-missing-title`**: inject `<title>Unknown</title>` when the head has no usable
93
93
  title.
94
- - **`--fix-id-colons`**: illegal colons in `id="X:Y"` and their matching `#X:Y` fragment
95
- references become `_`; word-bounded, so external URLs are untouched.
94
+ - **`--fix-id-colons`**: illegal colons in `id="X:Y"` and their matching internal
95
+ `#X:Y` fragment references become `_`. Only the bare `id` attribute is in scope
96
+ (`data-id` values are arbitrary data and are never touched), the fragment of an
97
+ external URL names a position in that other document and survives verbatim, and the
98
+ NCX's `content src` fragments follow the rename (`fix_ncx_src_fragments`), so a ToC
99
+ never dangles against the ids it references.
96
100
  - **`--unwrap-block-in-inline`**: drop a `<span>` that illegally wraps a
97
101
  `<div>/<p>/<blockquote>`, keeping the block element and its text.
98
102
  - **`--strip-invalid-value`**: remove misplaced `value="..."` attributes from non-form
@@ -105,7 +109,9 @@ part of the default pipeline:
105
109
  - **`--strip-epub3-attrs`**: scrub the EPUB3-only attributes epubcheck rejects on an
106
110
  EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed,
107
111
  documented set; extend only with a named epubcheck finding). Rendering is unchanged,
108
- and reader-legitimate lookalikes (`type`, the wider aria family) survive.
112
+ and reader-legitimate lookalikes (`type`, the wider aria family) survive. The edit is
113
+ anchored to real start tags and never touches CDATA sections or comments, so prose
114
+ that merely mentions `epub:type="chapter"` is preserved.
109
115
  - **`--downgrade-epub3-tags`**: downgrade EPUB3/HTML5 semantic elements to their EPUB2
110
116
  equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing
111
117
  classes and appending the semantic name (`class="figure"`) as the styling hook.
@@ -113,6 +119,12 @@ part of the default pipeline:
113
119
  (`css_protected_tags`/`style_block_tags` are parameterized over the tag set), so
114
120
  styled formatting can never be silently destroyed; a protected book keeps its
115
121
  RSC-005 findings, which is the honest outcome.
122
+
123
+ Both EPUB2-targeted fixes are gated on the package version carried in the OPF: they
124
+ fire on EPUB 2 packages (major version 2 or 1) and are inert on EPUB 3 packages and
125
+ when no version can be read, where their target defects do not exist and firing them
126
+ would strip legal attributes and downgrade legal elements (the 2026-09-08 sweep found
127
+ every EPUB3 book in an `--all` sweep taking exactly that damage).
116
128
  - **`--unwrap-illegal-tags`**: delete `<st> <sentence> <o> <w> <pagebreak>` tags outright,
117
129
  inner text preserved. Its CSS precondition is enforced by the library itself:
118
130
  `transforms.css_protected_tags` scans every stylesheet entry in the book (nested at-rules
@@ -167,6 +179,14 @@ Since this removes visible text from the reading experience, it is lossy by desi
167
179
  ### Opt-in, lossy: watermark strip (`--strip-watermarks`)
168
180
  Strips known producer and redistributor watermarks out of EPUBs (e.g. OceanofPDF.com, ABC Amber LIT Converter). The removal is a balanced-element surgery rather than regex slicing: it locates the stamp and deletes the outermost wrapper whose *entire visible text* is the watermark, ensuring prose that merely mentions the URL is preserved. Also drops known zero-byte marker files. Like other lossy operations, this is verified via `no_worse`.
169
181
 
182
+ When the stamp link sits inline (no clean wrapper), the bare `<a>` element may be
183
+ deleted only when the match demonstrably holds nothing but the stamp: a tag-free
184
+ body no longer than a stamp, or a body whose visible text is exactly the
185
+ watermark. An anchored match larger than that (an unclosed stamp `<a>` that
186
+ swallowed prose out to the next unrelated `</a>`) is refused: nothing is removed,
187
+ the refusal is counted (`watermark_refusals`) and reported for manual repair, and
188
+ `run phase1` surfaces it as a `manual_watermark_repair` decision.
189
+
170
190
  ## Archive rewrite
171
191
 
172
192
  Entries are copied one at a time; `mimetype` is written first and `ZIP_STORED`. Its
@@ -180,7 +200,12 @@ a fresh `ZipInfo` rather than the source one because OCF requires the mimetype e
180
200
  carry no extra field. Content documents and the NCX get the transforms above; every other
181
201
  entry is copied verbatim with its original compression. An eligible entry that no
182
202
  transform changed is also copied byte-for-byte, never decoded and re-encoded, so a
183
- clean non-UTF-8 file cannot be silently mangled. A `RepairReport` records
203
+ clean non-UTF-8 file cannot be silently mangled. A document that is not valid UTF-8
204
+ at all (windows-1252, UTF-16) is never decoded-with-replacement and re-encoded: it is
205
+ copied byte-for-byte, counted in the report (`non_utf8_docs_skipped`), and left for
206
+ manual repair. The anchor pass (`--strip-broken-anchors`) runs after every other
207
+ content fix, against an id snapshot that replicates each id-moving fix ahead of it. A
208
+ `RepairReport` records
184
209
  per-transform counts and whether the NCX uid was synced.
185
210
 
186
211
  ## The epubcheck gate
@@ -219,8 +244,10 @@ Scope is `<p>` elements (where the defect is carried). For each book:
219
244
  - A **page-number layer** is confirmed only when BOTH hold: >= 20 standalone arabic
220
245
  numbers AND >= 3 confident interrupts. A chapter-numbered book has zero interrupts, so
221
246
  its chapter numbers are never touched. Roman numerals are removed only as a confident
222
- interrupt, preserving roman chapter and front-matter numbering. Year-range values
223
- (1500-2099) are never page numbers.
247
+ interrupt, preserving roman chapter and front-matter numbering; a roman counts as a
248
+ number only when it is well-formed (explicit subtractive pairs) and under 100, so
249
+ ordinary words built from roman letters (`mid`, `mix`, `civil`) are never read as
250
+ pages. Year-range values (1500-2099) are never page numbers.
224
251
  - **Merging:** only a confident interrupt (lowercase continuation or word split) rejoins
225
252
  the two surrounding prose paragraphs (a word split closes up its hyphen). Every other
226
253
  removal is delete-only, leaving the existing paragraph break.
@@ -274,7 +301,16 @@ For a Calibre library (`Author/Title (id)/Title - Author.epub`):
274
301
  atomic-replacement contract live there).
275
302
 
276
303
  ### Native format installation (`--install-to-calibre`)
277
- Optionally, bindery-cli installs the repaired EPUB as the book's format through cquarry's write module (`WritableCalibreDB`): the file is placed atomically — an in-place replace over the catalogued file when one exists (same path, same `data.name`), or a fresh placement under the repaired file's stem otherwise — and the `data` row follows in one `batch()` transaction (`remove_format` + `add_format`, since `add_format` refuses duplicates), keeping the size truthful and queuing the book in `metadata_dirtied` so Calibre regenerates its sidecar .opf. The external `calibredb` CLI is no longer used (the v0.23.1 `--replace` crash class is gone with it). It automatically falls back to atomic filesystem replacement if a valid Calibre ID cannot be extracted, and a database failure degrades to the in-place save with a warning rather than losing the repair.
304
+ Optionally, bindery-cli installs the repaired EPUB as the book's format through cquarry's write module (`WritableCalibreDB`): the file is placed atomically — an in-place replace over the catalogued file when one exists (same path, same `data.name`), or a fresh placement under the repaired file's stem otherwise — and the `data` row follows through `set_format` (cquarry 1.17's sanctioned remove+add in one transaction), keeping the size truthful and queuing the book in `metadata_dirtied` so Calibre regenerates its sidecar .opf. The external `calibredb` CLI is no longer used (the v0.23.1 `--replace` crash class is gone with it). It automatically falls back to atomic filesystem replacement if a valid Calibre ID cannot be extracted, and a database failure degrades to the in-place save with a warning rather than losing the repair.
305
+
306
+ The no-catalog fallback guesses the id from the `(N)` directory fragment. A guessed
307
+ id may drive a row update only when metadata.db corroborates it: the book row exists,
308
+ the file lives in that book's own directory, and (when an EPUB is catalogued) carries
309
+ its stored `data.name`. Anything else is a stray file inside a book directory; the
310
+ repair is saved in place and the row is left untouched (updating the row from a stray
311
+ wrote the stray's size over the catalogued entry, the 2026-09-08 stray-size incident),
312
+ and a stale `(N)` directory whose book no longer exists saves in place instead of
313
+ crashing the sweep.
278
314
 
279
315
  ## Audit subcommand (read-only)
280
316
 
@@ -4,7 +4,7 @@ strip), epubcheck-gated.
4
4
  Single source of truth for the version. pyproject.toml must match VERSION below.
5
5
  """
6
6
 
7
- VERSION = "0.31.0"
7
+ VERSION = "0.32.0"
8
8
  __version__ = VERSION
9
9
 
10
10
  from .epub import RepairReport, repair_epub # noqa: E402