bindery-cli 0.34.0__tar.gz → 0.36.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/CLAUDE.md +3 -3
  2. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/PKG-INFO +11 -6
  3. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/README.md +9 -4
  4. bindery_cli-0.36.0/REPORT-12-Sept.md +119 -0
  5. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/patchnotes.md +85 -0
  6. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/pyproject.toml +2 -2
  7. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/roadmap.md +118 -4
  8. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/spec.md +24 -6
  9. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/__init__.py +1 -1
  10. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/audit.py +183 -5
  11. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/cli.py +13 -4
  12. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/validate.py +48 -16
  13. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_audit.py +283 -0
  14. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_validate.py +100 -0
  15. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/uv.lock +5 -5
  16. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.github/workflows/ci.yml +0 -0
  17. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.github/workflows/publish.yml +0 -0
  18. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.gitignore +0 -0
  19. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/AGENTS.md +0 -0
  20. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/LICENSE +0 -0
  21. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/docs/screenshots/library-sweep.png +0 -0
  22. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/logo.svg +0 -0
  23. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/run_tests.sh +0 -0
  24. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/scripts/FastSweep.java +0 -0
  25. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/scripts/fast_sweep.py +0 -0
  26. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/__main__.py +0 -0
  27. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/epub.py +0 -0
  28. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/library.py +0 -0
  29. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/pagination.py +0 -0
  30. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/reserialize.py +0 -0
  31. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/transforms.py +0 -0
  32. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/watermark.py +0 -0
  33. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/testing_facility/books_to_fix/.gitkeep +0 -0
  34. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/testing_facility/top500candidates/REPORT.md +0 -0
  35. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_cli.py +0 -0
  36. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_epub.py +0 -0
  37. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_library.py +0 -0
  38. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_pagination.py +0 -0
  39. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_reserialize.py +0 -0
  40. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_transforms.py +0 -0
  41. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_version.py +0 -0
  42. {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_watermark.py +0 -0
@@ -77,12 +77,12 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
77
77
  - `src/bindery/pagination.py`: the opt-in lossy page-number strip (runhead detection, page-layer decision, block-centric removal/merge, safety nets).
78
78
  - `src/bindery/watermark.py`: the opt-in lossy watermark strip (anchored and anchorless signature removal).
79
79
  - `src/bindery/reserialize.py`: structural repair via `html5lib`.
80
- - `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, and `monolithic` since v0.21.0, `--max-doc-chars N`) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story).
80
+ - `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, `monolithic` since v0.21.0 with `--max-doc-chars N`, and `completeness` since v0.36.0) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story). Since v0.36.0 the completeness analyzer (Phase 15) reports the phase-1 spot-check per book: spine/prose-doc counts (prose = >= 400 visible chars), first/middle/last prose-doc opening/closing excerpts, trailing-ToC classification (`_trailing_toc`: short link lines dominate, no paragraph-length blocks), and the unreadable fraction. It is ADVISORY by contract (problem always False; ADVISORY on a trailing ToC or unreadable fraction >= 10%), never moves the exit code, and is skipped like emptytext when the archive verdict owns the book.
81
81
  - `src/bindery/epub.py`: archive rewrite, NCX uid sync, RepairReport, mismatch detection, and the opt-in structural-repair plumbing (including the CSS precondition scan).
82
82
  - `src/bindery/validate.py`: epubcheck wrapper, the `gate` (improvement) and `no_worse` (no-regression, for the lossy strips) acceptance bars.
83
- - `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching) — the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (since v0.31.0 via `set_format`, cquarry 1.17's sanctioned remove+add in one transaction); the external `calibredb` CLI is gone. Since v0.32.0 a guessed id drives a row update only when metadata.db corroborates it (`_verify_guess`: books row exists, file in the book's own directory, catalogued `data.name` when a row exists); anything else saves in place and leaves the catalog untouched — never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
83
+ - `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching); the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (since v0.31.0 via `set_format`, cquarry 1.17's sanctioned remove+add in one transaction); the external `calibredb` CLI is gone. Since v0.32.0 a guessed id drives a row update only when metadata.db corroborates it (`_verify_guess`: books row exists, file in the book's own directory, catalogued `data.name` when a row exists); anything else saves in place and leaves the catalog untouched — never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
84
84
  - `src/bindery/cli.py`: `repair` and `library` subcommands, including `--all` and `--install-to-calibre`; since v0.29.0 also the `run` verbs: `phase1` (pre-import vetting over loose files: the audit battery composed with one gated repair sweep; read-only until `--apply-lossy`, which IS the lossy-strip consent) and `phase3` (the scoped post-import apply step: `library --id --sweep --only all --apply --all --install-to-calibre` plus a pre/post summary; mechanically refuses unscoped sweeps with exit 2). Both take `--non-interactive` and surface open questions as `decisions_needed` in their JSON, never prompts. They drive the shipped `run_library` through the real parser, so no flag can drift between the verb and the subcommand it wraps. `library --workers N` (v0.30.0) parallelizes only the sweep's candidate pass, in windows of N consumed in order so `--limit` stays lazy; the repair phase is never parallel (shared workdir, atomic-replacement contract).
85
- - `tests/`: transforms, end-to-end repair, atomic replace, pagination, watermarks, the audit analyzers (content/pagenumbers/emptytext/ocr/monolithic, archive integrity, spine classification), the library sweep and `CalibreIdResolver`, validate, reserialize, CLI wiring, and the version pin.
85
+ - `tests/`: transforms, end-to-end repair, atomic replace, pagination, watermarks, the audit analyzers (content/pagenumbers/emptytext/ocr/monolithic/completeness, archive integrity, spine classification), the library sweep and `CalibreIdResolver`, validate, reserialize, CLI wiring, and the version pin.
86
86
 
87
87
  ## Conventions
88
88
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: bindery-cli
3
- Version: 0.34.0
3
+ Version: 0.36.0
4
4
  Summary: Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library.
5
5
  Author: vrnvctss
6
6
  License: MIT
@@ -13,7 +13,7 @@ Classifier: License :: OSI Approved :: MIT License
13
13
  Classifier: Programming Language :: Python :: 3.14
14
14
  Classifier: Topic :: Utilities
15
15
  Requires-Python: >=3.14
16
- Requires-Dist: cquarry>=1.17.0
16
+ Requires-Dist: cquarry>=1.19.0
17
17
  Requires-Dist: tqdm
18
18
  Requires-Dist: vir-tui>=2.2.0
19
19
  Provides-Extra: reserialize
@@ -57,8 +57,8 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
57
57
  - **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
58
58
  - **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
59
59
  - **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
60
- - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
61
- - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
60
+ - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
61
+ - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
62
62
  - **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
63
63
  - **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
64
64
  - **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
@@ -82,7 +82,7 @@ Every repair is gated by [epubcheck](https://github.com/w3c/epubcheck). The acce
82
82
 
83
83
  Introducing a net-new fatal is always rejected. If epubcheck itself fails to run (crash, timeout, unparsable output), the book is reported as an error and never applied; only an explicit `--no-validate` skips the gate. Originals are never modified except by an explicit, atomic in-place replace (see below), and even then only after the gate accepts the result.
84
84
 
85
- Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a small in-process daemon that compiles `FastDaemon.java` into a temp directory at runtime and drives a warm JVM over a pipe. The daemon is a pool bounded by `--workers` (serial runs use one), its every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good — the two paths must never disagree about a book, and the subprocess oracle is the fallback by design. The daemon is also the scariest under-tested code in the repo and depends on a matched javac/java pair; its status is recorded in the roadmap's Phase 12 postscript and Phase 14 backlog.
85
+ Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a persistent daemon that writes `FastDaemon.java` into a temp directory at runtime, compiles it in memory with the running JVM's own compiler (Java's single-file source launcher: no javac, no javac/java version skew), and drives the warm JVM over a pipe. The daemon reports the counts it reads from the JSON document that epubcheck's own `CheckingReport` serializes, so it measures exactly what the subprocess measures, just ~15x faster. It is a pool bounded by `--workers` (serial runs use one), every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good.
86
86
 
87
87
  The lossy modes (`--strip-pagination`, `--strip-broken-tags`, and `--strip-watermarks`) are the exception to the "must improve" rule. Since they remove visible markup rather than correcting XML schema violations, epubcheck counts often remain unchanged. They are accepted when the result is **no worse** (no net-new fatals or errors), relying on strict programmatic safety nets instead.
88
88
 
@@ -126,10 +126,15 @@ bindery audit ocr ~/docs/Calibre\ Library
126
126
  # Flag books where a single content document is too large for readers to render
127
127
  bindery audit monolithic ~/docs/Calibre\ Library
128
128
 
129
+ # Spot-check completeness: prose coverage, first/middle/last excerpts, trailing ToC
130
+ bindery audit completeness ~/docs/Calibre\ Library
131
+
129
132
  # Run all audits and generate a comprehensive CSV
130
133
  cd ~/docs/Calibre\ Library && bindery audit all
131
134
  ```
132
135
 
136
+ **Completeness spot-check:** the `completeness` audit answers the pre-import question "does this book reach its end?" It reports, per book: the prose-document count against the spine (docs with 400+ visible characters), the opening and closing text of the first/middle/last prose doc, a trailing-ToC classification of the final spine doc (a link-dense chapter list after the real back matter), and the fraction of unreadable docs. A trailing ToC is reported as an advisory, never a failure: the closing excerpt always comes from the real back matter, so a human can confirm the book ends where the author ended it.
137
+
133
138
  Audits can also be run on a directory of loose `.epub` files by passing the path as the second argument:
134
139
  ```sh
135
140
  bindery audit pagenumbers /path/to/loose/epubs
@@ -204,7 +209,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
204
209
  - `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
205
210
  - `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
206
211
  - `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
207
- - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
212
+ - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied`; Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
208
213
  - `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
209
214
  - Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
210
215
  - A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
@@ -35,8 +35,8 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
35
35
  - **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
36
36
  - **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
37
37
  - **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
38
- - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package — `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
39
- - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents — `figure`/`section` to `div`, `figcaption` to `p` — keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
38
+ - **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
39
+ - **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
40
40
  - **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
41
41
  - **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
42
42
  - **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
@@ -60,7 +60,7 @@ Every repair is gated by [epubcheck](https://github.com/w3c/epubcheck). The acce
60
60
 
61
61
  Introducing a net-new fatal is always rejected. If epubcheck itself fails to run (crash, timeout, unparsable output), the book is reported as an error and never applied; only an explicit `--no-validate` skips the gate. Originals are never modified except by an explicit, atomic in-place replace (see below), and even then only after the gate accepts the result.
62
62
 
63
- Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a small in-process daemon that compiles `FastDaemon.java` into a temp directory at runtime and drives a warm JVM over a pipe. The daemon is a pool bounded by `--workers` (serial runs use one), its every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good — the two paths must never disagree about a book, and the subprocess oracle is the fallback by design. The daemon is also the scariest under-tested code in the repo and depends on a matched javac/java pair; its status is recorded in the roadmap's Phase 12 postscript and Phase 14 backlog.
63
+ Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a persistent daemon that writes `FastDaemon.java` into a temp directory at runtime, compiles it in memory with the running JVM's own compiler (Java's single-file source launcher: no javac, no javac/java version skew), and drives the warm JVM over a pipe. The daemon reports the counts it reads from the JSON document that epubcheck's own `CheckingReport` serializes, so it measures exactly what the subprocess measures, just ~15x faster. It is a pool bounded by `--workers` (serial runs use one), every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good.
64
64
 
65
65
  The lossy modes (`--strip-pagination`, `--strip-broken-tags`, and `--strip-watermarks`) are the exception to the "must improve" rule. Since they remove visible markup rather than correcting XML schema violations, epubcheck counts often remain unchanged. They are accepted when the result is **no worse** (no net-new fatals or errors), relying on strict programmatic safety nets instead.
66
66
 
@@ -104,10 +104,15 @@ bindery audit ocr ~/docs/Calibre\ Library
104
104
  # Flag books where a single content document is too large for readers to render
105
105
  bindery audit monolithic ~/docs/Calibre\ Library
106
106
 
107
+ # Spot-check completeness: prose coverage, first/middle/last excerpts, trailing ToC
108
+ bindery audit completeness ~/docs/Calibre\ Library
109
+
107
110
  # Run all audits and generate a comprehensive CSV
108
111
  cd ~/docs/Calibre\ Library && bindery audit all
109
112
  ```
110
113
 
114
+ **Completeness spot-check:** the `completeness` audit answers the pre-import question "does this book reach its end?" It reports, per book: the prose-document count against the spine (docs with 400+ visible characters), the opening and closing text of the first/middle/last prose doc, a trailing-ToC classification of the final spine doc (a link-dense chapter list after the real back matter), and the fraction of unreadable docs. A trailing ToC is reported as an advisory, never a failure: the closing excerpt always comes from the real back matter, so a human can confirm the book ends where the author ended it.
115
+
111
116
  Audits can also be run on a directory of loose `.epub` files by passing the path as the second argument:
112
117
  ```sh
113
118
  bindery audit pagenumbers /path/to/loose/epubs
@@ -182,7 +187,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
182
187
  - `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
183
188
  - `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
184
189
  - `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
185
- - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied` — Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
190
+ - `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied`; Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
186
191
  - `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
187
192
  - Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
188
193
  - A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
@@ -0,0 +1,119 @@
1
+ # REPORT-12-Sept: the repair-landscape research (bindery-cli vs real-world EPUB damage)
2
+
3
+ Research date: 2026-09-10/11. Method: a read-only research pass mapping
4
+ the full taxonomy of real-world EPUB damage — by producer (Calibre
5
+ conversions, Word/Office export, InDesign, legacy OCR, Adobe Digital
6
+ Editions, KindleUnpack, self-pub aggregators, Windows case-manglers) —
7
+ against this repo's shipped repair surface, the Phase 14/15 backlogs,
8
+ and the Bug Reports. References include the upstream clone's format
9
+ readers. Full evidence lives in the agent's findings; this report is
10
+ the distilled verdict and every actionable gap with its routing.
11
+
12
+ ## Verdict
13
+
14
+ Bindery handles roughly **60-65% of real-world damage classes
15
+ end-to-end** and is near-complete for its founding niche: markup
16
+ well-formedness (the conversion-fatals class), NCX basics,
17
+ zip/mimetype container hygiene, dead-reference pruning, watermark and
18
+ page-number furniture, and archive/spine diagnosis (corruption, DRM
19
+ skip, ToC-bloat classification). Phase 14's hardening and all five Bug
20
+ Reports are shipped through v0.35.0; Phase 15 (the completeness
21
+ analyzer) is approved.
22
+
23
+ The missing ~35% is not more markup surgery. It is **package-level
24
+ repairs** — container.xml, manifest media-types, spine attribute
25
+ cleanup, encryption.xml/obfuscation state, zip-level dedupe, href
26
+ case/IRI mismatches — and the finding that matters most: several of
27
+ those classes are **gateway defects**. Epubcheck stays fatal while
28
+ they persist, so no shipped repair can ever be gate-accepted on those
29
+ books. They are locked out of the pipeline entirely.
30
+
31
+ ## A. The damage-taxonomy matrix and the repair gaps
32
+
33
+ 1. **Missing/corrupt `META-INF/container.xml`** (any producer):
34
+ epubcheck fatal; the OPF is unfindable. The locator falls back to
35
+ the first `.opf` but never writes the file. Deterministic fix:
36
+ generate the standard container.xml at the located OPF (upstream
37
+ calibre's `initialize_container` is the model). Effort S; HIGH
38
+ value — a gateway repair that unlocks an entire class of books.
39
+ 2. **Stale font-obfuscation state** (publisher EPUB3 fonts; IDPF
40
+ `2008/embedding` and Adobe `pdf/enc#RC`): post-KindleUnpack/strip
41
+ tools decrypt fonts in place but leave `encryption.xml` declaring
42
+ obfuscation; epubcheck errors and the audit implies DRM where there
43
+ is none. Fix: detect via font magic bytes at offset 0 vs the
44
+ CipherReference; prune stale entries; drop encryption.xml when only
45
+ obfuscation entries remain and the fonts verify. Never
46
+ un-obfuscate. Also gives the audit a real OBFUSCATED-vs-DRM
47
+ distinction. Effort M; needs the magic-byte detection validated on
48
+ real samples first.
49
+ 3. **Wrong/missing manifest `media-type`** (aggregators, ADE): fonts,
50
+ images, and media declared with wrong types. Fix: sniff magic bytes
51
+ or extension map; attribute-only. Effort S.
52
+ 4. **href case/backslash/IRI mismatches** (KindleUnpack, Windows
53
+ producers): `Images/Cover.JPG` vs `images/cover.jpg`, backslash
54
+ separators, `%XX` variants — the file IS present, so
55
+ `--prune-missing-resources` cannot touch the RSC-007/PKG-010 it
56
+ causes. Fix: case-insensitive namemap when exactly one match;
57
+ backslash to slash. Effort M.
58
+ 5. **Dangling reference edges after pruning** (any producer):
59
+ `spine@toc`, `item@media-overlay`, `item@fallback`, and EPUB2 cover
60
+ meta pointing at items the prune pass deleted. Today's prune checks
61
+ only `spine_ids` — it can manufacture the regression the gate then
62
+ rejects. Fix: extend prune to rewrite those edges (the rewrite
63
+ machinery exists in fix_manifest_ids). Effort S/M.
64
+ 6. **Duplicate zip entries** (any producer): last-wins shadowing; the
65
+ audit counts `dup_entries`, the rewrite copies both. Fix: drop
66
+ shadowed duplicates on rewrite. Effort S.
67
+ 7. **Cover wiring** (EPUB2 producers): dangling `<meta name="cover">`,
68
+ missing EPUB3 `properties="cover-image"`, wrong cover media-type —
69
+ calibre and booksellers extract no cover. Narrow, finding-driven
70
+ repair when the cover is identifiable from the existing guide/meta;
71
+ audit-only otherwise. Effort M.
72
+ 8. **Broken CSS from converters**: `url()` to absent assets, malformed
73
+ selectors. Repair is fiddly; ship the audit-only detector first and
74
+ let prevalence data decide. Effort M (repair).
75
+ 9. **Human-facing metadata validity** (self-pub/Word): non-ISO
76
+ `dc:date`, invalid `dc:language`, missing EPUB3
77
+ `dcterms:modified`. Explicit non-goal today; `dcterms:modified`
78
+ also collides with byte-determinism. Needs a written charter
79
+ ruling: carve-out with a determinism-safe timestamp policy, or
80
+ route to cquarry/CalibreQuarry. Effort M + a decision.
81
+ 10. **EPUB3 dual-ToC drift** (nav vs NCX): detection is a
82
+ deterministic structural diff feeding `decisions_needed`; full
83
+ sync is ToC synthesis and stays out of charter permanently.
84
+ Effort L if repair is ever attempted; audit-only recommended.
85
+ 11. **No-repair classes, correctly routed**: OCR garbage, truncated
86
+ re-sources, absent font families (audit + re-source); real DRM
87
+ (clean ENCRYPTED skip — shipped); DB-level format rows
88
+ (cquarry's, done); catalog completeness and human-facing metadata
89
+ quality (CalibreQuarry's audits).
90
+
91
+ ## B. Research-before-boxing
92
+
93
+ 1. **FastSweep prevalence study** — run `fast_sweep.py --mode=extract`
94
+ aggregation over the full library targeting the new classes
95
+ (container.xml presence/validity, encryption.xml algorithms,
96
+ media-type warnings, case-mismatch RSC-007s). The harness exists;
97
+ it sizes every box above with real counts before any is built.
98
+ 2. **The metadata carve-out ruling** (gap 9) — Brandon decides;
99
+ the audit records both options.
100
+ 3. **The obfuscation magic-byte table** (gap 2) — validate IDPF and
101
+ Adobe font prefixes against real samples before the repair ships.
102
+
103
+ ## C. Routing notes
104
+
105
+ Package-structure repairs (items 1-6) fit the existing "structural
106
+ opt-ins + gate" model without reopening the semantics-preserving
107
+ contract. Items 7-10 each need an explicit scope decision first.
108
+ Cross-repo: DB-level size/format-row mismatches route to cquarry
109
+ (set_format, shipped); catalog completeness, duplicate detection, and
110
+ human-facing metadata quality route to CalibreQuarry's audits; OCR,
111
+ truncation, monolithic, and missing-font content stay audit-and-
112
+ re-source permanently.
113
+
114
+ ## D. Phase 16 routing
115
+
116
+ Items 1-6 (the ungated package-structure repairs) are boxed in
117
+ roadmap.md **Phase 16** with the FastSweep prevalence study leading.
118
+ Items 7-10 are recorded as decision-gated boxes. Phase 15 (approved)
119
+ runs first if its fixture work is not yet done.
@@ -1,4 +1,89 @@
1
1
  # bindery-cli Patch Notes
2
+ ## v0.36.0 (2026-09-12)
3
+
4
+ ### The completeness spot-check: `audit completeness` (Phase 15)
5
+
6
+ - **The phase-1 judgment step finally has a tool owner.** The import
7
+ skill's completeness step asks a human question with no tool behind
8
+ it: "sample early, middle, and late pages ... read the LAST content
9
+ page to confirm it reaches real back matter rather than cutting off
10
+ mid-chapter". Two consecutive phase-1 runs answered it with a
11
+ hand-rolled zipfile sampler in /tmp. The new analyzer rides the
12
+ audit's existing single decompression pass (no second decode of any
13
+ book) and reports, per book: spine doc count, prose-doc count (spine
14
+ docs with at least 400 visible characters), the opening and closing
15
+ 120 characters of the first/middle/last prose doc, a trailing-ToC
16
+ classification of the final spine doc, and the fraction of unreadable
17
+ docs (corrupt entries plus unresolved itemrefs).
18
+ - **A trailing ToC never impersonates the ending.** The final spine doc
19
+ is classified as a chapter list when short link lines dominate it and
20
+ no block runs to paragraph length; such a doc is book furniture and
21
+ is excluded from the prose sampling, so the closing excerpt comes
22
+ from the real back matter. Verified against the real 2026-09-10
23
+ Redwall fixtures: Lord Brocktree's trailing ToC is caught and the
24
+ Epilogue is sampled as the last prose doc; The Long Patrol reads
25
+ exactly 55/57 prose docs through the percent-encoded hrefs that
26
+ defeated the hand-rolled sampler; Mattimeo's split doc (chapter
27
+ headings with real prose after every one) stays prose.
28
+ - **Advisory by contract.** The analyzer never flags a book and never
29
+ moves the exit code: the archive, spine, and emptytext verdicts keep
30
+ owning the flags. The verdict status is ADVISORY (not OK) when a
31
+ trailing ToC is present or the unreadable fraction reaches 10%.
32
+ `audit all` and `run phase1` include it through the battery, the CLI
33
+ grows the `completeness` mode, and `--json` records carry its
34
+ verdicts in the same shape as the other analyzers.
35
+
36
+ ### Dependencies
37
+
38
+ - Adopted cquarry 1.18.0 (floor + lock): the set_format repair lane now keeps
39
+ Calibre's FTS index and page counts honest automatically (the sidecar's
40
+ dirty queue and the pages rescan flag ride the write), and the search and
41
+ runtime fixes come along in the shared layer.
42
+ - Adopted cquarry 1.19.0 (floor + lock): the repair lane gains its undo
43
+ primitive -- save_original_format/restore_original_format -- so a format
44
+ repair can put the original bytes back if it goes wrong.
45
+
46
+ ### Repo records
47
+
48
+ - The release-tag backlog is settled forward-only from v0.35.0: the
49
+ nine untagged 0.18.0-0.26.0 entries are exempted by decision of
50
+ record (2026-09-11), and the v0.28.0 retag and the v0.33.0 em-dash
51
+ tag repair stay recorded-only force-pushes (roadmap.md, "Tag
52
+ policy").
53
+
54
+ ## v0.35.0 (2026-09-10)
55
+
56
+ ### The epubcheck daemon, reconciled: oracle-identical counts at ~15x the speed
57
+
58
+ - **FastDaemon v2 measures exactly what the subprocess oracle measures.**
59
+ The daemon's old Java side counted message occurrences from
60
+ `CheckingReport`'s counters, while the gate has always measured the
61
+ counts in epubcheck's own `--json` output, which aggregates identical
62
+ messages (one book: 243 occurrences, 4 aggregated messages). On the
63
+ staged-fixture corpus the two methods diverged on 11 of 14 books, so
64
+ the daemon could never be enabled without changing what the gate
65
+ measures. The new daemon calls `CheckingReport.generate()` and reads
66
+ `nFatal`/`nError`/`nWarning` out of the JSON epubcheck itself
67
+ serializes: identical numbers by construction. Verified against the
68
+ subprocess oracle on all 14 staged books and on a 100-book
69
+ real-library sample: 114/114 identical.
70
+ - **No more javac, no more version skew.** The daemon now launches
71
+ through Java's single-file source launcher (JEP 330): the running JVM
72
+ compiles the daemon in memory with its own compiler, so the
73
+ javac-newer-than-java skew that kept the daemon dead on this machine
74
+ (javac 27-ea, java 25) cannot recur on any toolchain.
75
+ - **Measured, working daemon throughput: ~0.27s/book warm against
76
+ ~4.2s/book for the subprocess oracle** (15.6x). With `--workers N`
77
+ each worker gets its own warm daemon from the bounded pool, and every
78
+ roundtrip stays bounded by the caller's timeout with the fail-safe
79
+ fallback to the subprocess oracle.
80
+ - **The dedup semantics are pinned by a suite test:** a book engineered
81
+ with one defect repeated three times plus one different defect must
82
+ answer 2 errors (the aggregated count), not 4 occurrences (the old
83
+ daemon's answer). Skipped where no epubcheck/JDK exists, like the
84
+ other oracle tests. README and spec's gate sections describe the new
85
+ design.
86
+
2
87
  ## v0.34.0 (2026-09-10)
3
88
 
4
89
  ### Brandon's three calls: partial books are trouble, the contract reconciled, the daemon decision informed
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "bindery-cli"
7
- version = "0.34.0"
7
+ version = "0.36.0"
8
8
  description = "Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library."
9
9
  readme = "README.md"
10
10
  classifiers = [
@@ -27,7 +27,7 @@ authors = [
27
27
  dependencies = [
28
28
  "tqdm",
29
29
  "vir-tui>=2.2.0",
30
- "cquarry>=1.17.0",
30
+ "cquarry>=1.19.0",
31
31
  ]
32
32
 
33
33
  [project.optional-dependencies]
@@ -663,7 +663,7 @@ book instead raises an uncaught `TypeError` at the `book["path"]` access.
663
663
  matches the DB's book path and format name before the batch; otherwise save
664
664
  in place and warn without touching the row, and catch the `TypeError` path.
665
665
  *(Fixed in v0.32.0: a guessed id now drives a row update only when
666
- metadata.db corroborates it — the books row exists, the file sits in that
666
+ metadata.db corroborates it: the books row exists, the file sits in that
667
667
  book's catalogued directory, and when an EPUB row exists it carries that
668
668
  row's stored name. Any other guess (a stray file in a book directory, a
669
669
  stale `(N)` directory whose book is gone, the resolver's id gone from
@@ -996,7 +996,7 @@ can be net-neutral and ships silently.*
996
996
  v0.22.0 "its own verdict in every mode" promise (`audit.py:1565`,
997
997
  `1783-1803`, `1520-1558`).
998
998
  *(Done in v0.33.0: the archive and spine sections run unconditionally
999
- after the selected content sections — they are always-on verdicts and
999
+ after the selected content sections; they are always-on verdicts and
1000
1000
  were already scanned and recorded in the JSON; only the console/rc
1001
1001
  path was dead. A CRC-corrupt book now prints its CORRUPT section and
1002
1002
  fails library mode with exit 1, matching directory mode. Test pins
@@ -1156,7 +1156,24 @@ can be net-neutral and ships silently.*
1156
1156
  javac at all), but the daemon should stay off until its counting
1157
1157
  matches the JSON oracle; retire-or-reconcile remains the open
1158
1158
  decision. The daemon already self-disables safely, and --workers
1159
- parallelism works via subprocesses.)**
1159
+ parallelism works via subprocesses.)
1160
+ *(DECIDED 2026-09-10: Brandon chose RECONCILE over retire. The
1161
+ counting divergence turned out to run deeper than the daemon: the
1162
+ CLI's human summary counts message occurrences (243 on one book)
1163
+ while epubcheck's own --json output counts aggregated messages
1164
+ (4 on the same book), and the gate has always measured the
1165
+ aggregated scale. The reconciliation is therefore by construction:
1166
+ FastDaemon v2 calls CheckingReport.generate() and reads nFatal/
1167
+ nError/nWarning from the JSON epubcheck itself serializes, so the
1168
+ daemon measures the subprocess oracle's numbers exactly. Verified
1169
+ against the subprocess oracle on all 14 staged books (the same
1170
+ corpus that diverged 11/14 under the old daemon) and on a
1171
+ 100-book real-library sample. The launch switched to Java's
1172
+ single-file source launcher, removing javac and the version skew
1173
+ permanently; the dedup semantics are pinned by a suite test
1174
+ (skipped where no epubcheck/JDK exists). Daemon warm throughput
1175
+ measured at ~0.27s/book vs ~4.2s/book subprocess. Shipped in
1176
+ v0.35.0.)**
1160
1177
  - [x] **Smaller apply papercuts:** `--limit -1` and a missing
1161
1178
  `--audit` file raise tracebacks instead of usage errors
1162
1179
  (`cli.py:434`, `373`); the fresh-format branch of `install_format`
@@ -1269,7 +1286,7 @@ can be net-neutral and ships silently.*
1269
1286
 
1270
1287
  ### Documentation (drift is real but narrow)
1271
1288
 
1272
- - [ ] **Reconcile spec.md with the shipped opt-ins.** The contract's
1289
+ - [x] **Reconcile spec.md with the shipped opt-ins.** The contract's
1273
1290
  non-goals forbid exactly what `--reserialize` (v0.3.0, the lone
1274
1291
  html5lib dependency) and `--strip-bad-attrs` (v0.4.0) do, and neither
1275
1292
  has a spec section; the OPF half of `--fix-ids` is missing;
@@ -1277,6 +1294,10 @@ can be net-neutral and ships silently.*
1277
1294
  NCX sentence at spec.md:52-53 is garbled (a dangling "to the OPF
1278
1295
  unique identifier" tail). A contributor reading spec.md first would
1279
1296
  conclude two shipped flags violate the charter.
1297
+ *(APPLIED 2026-09-10, commit 01ad173, on Brandon's verbatim go
1298
+ recorded in the audit sheet's evening answers; shipped with
1299
+ v0.34.0. Ticked 2026-09-11 in the main-thread double-check after
1300
+ the verifier caught the work shipping unticked.)*
1280
1301
  **GATED on Brandon: spec.md is the contract file. Draft proposal,
1281
1302
  ready to apply on a go (do not apply without one):**
1282
1303
  (1) Non-goals bullet 2 ("Repairing genuinely mangled structure
@@ -1351,6 +1372,14 @@ everything else is backlog, not danger.*
1351
1372
 
1352
1373
  ## Phase 15: completeness spot-check analyzer (proposed 2026-09-10, from the phase-1 skill's own doctrine)
1353
1374
 
1375
+ *(SHIPPED 2026-09-12 as v0.36.0: `audit completeness` rides the existing
1376
+ single decompression pass; the Redwall-shaped regression tests landed
1377
+ first (4021ce7), the analyzer second (e6156c0). Verified against the
1378
+ real fixtures: Lord Brocktree's trailing ToC is caught with the Epilogue
1379
+ sampled as the last prose doc, The Long Patrol reads exactly 55/57
1380
+ prose docs through its percent-encoded hrefs, and Mattimeo's split doc
1381
+ stays prose. Advisory by contract; `all` and `run phase1` include it.)*
1382
+
1354
1383
  *(RAISED with Brandon 2026-09-10, batched with the session-start raises;
1355
1384
  recommendation: APPROVE. Execution is gated on that go; the proposal
1356
1385
  below is filed and costed. The Redwall run it names is a ready-made
@@ -1375,3 +1404,88 @@ Brocktree (real Epilogue doc before a trailing 520-char ToC), Mattimeo
1375
1404
  (Chapter 50 verified as a real heading with prose after, inside a 174k-char
1376
1405
  split doc), The Long Patrol (percent-encoded hrefs defeated the hand-rolled
1377
1406
  reader; 55/57 prose docs).*
1407
+
1408
+ ## Phase 16: package-structure repairs (proposed 2026-09-12, from REPORT-12-Sept.md)
1409
+
1410
+ A research pass mapped the full real-world EPUB damage taxonomy against
1411
+ the shipped repair surface: bindery handles ~60-65% of damage classes
1412
+ end-to-end, and the missing slice is package-level structure — several
1413
+ classes of which are GATEWAY DEFECTS (epubcheck stays fatal, so no
1414
+ shipped repair can ever be gate-accepted on those books). Full evidence,
1415
+ producer classes, and the cross-repo routing live in
1416
+ REPORT-12-Sept.md. Lead with the prevalence study; every box below is
1417
+ an opt-in structural repair under the existing gate contract.
1418
+
1419
+ ### A. Research first (boxes cannot be honestly sized without these)
1420
+
1421
+ - [ ] **FastSweep prevalence study**: run `fast_sweep.py
1422
+ --mode=extract` aggregation over the full library for the new
1423
+ classes — container.xml presence/validity, encryption.xml
1424
+ algorithms, manifest media-type warnings, case-mismatch RSC-007s.
1425
+ Each Phase 16 repair box gets its real count before it opens.
1426
+ - [ ] **Obfuscation magic-byte validation**: confirm the IDPF
1427
+ (`2008/embedding`) and Adobe (`pdf/enc#RC`) font prefixes at offset 0
1428
+ against real samples before the encryption-repair box opens.
1429
+
1430
+ ### B. The repairs (each: opt-in flag, dry-run default, gate-accepted)
1431
+
1432
+ - [ ] **`--fix-container`**: generate META-INF/container.xml at the
1433
+ located OPF (model: upstream calibre's initialize_container).
1434
+ Gateway repair, effort S, HIGH value.
1435
+ - [ ] **Obfuscation-aware encryption.xml repair**: prune stale
1436
+ obfuscation entries when font magic bytes prove the fonts are
1437
+ decrypted in place; drop encryption.xml when only obfuscation entries
1438
+ remain and fonts verify. NEVER un-obfuscate. M.
1439
+ - [ ] **Audit distinction: OBFUSCATED vs DRM** in audit.py (today the
1440
+ ENCRYPTED verdict keys on unreadable entries and implies DRM where
1441
+ publisher obfuscation is benign). S, rides the box above.
1442
+ - [ ] **Reference-edge completion for `--prune-missing-resources`**:
1443
+ rewrite `spine@toc`, `item@media-overlay`, `item@fallback`, and EPUB2
1444
+ cover-meta edges when their targets are pruned (machinery exists in
1445
+ fix_manifest_ids). Prevents manufacturing gate regressions. S/M.
1446
+ - [ ] **Manifest media-type normalization**: sniff magic bytes /
1447
+ extension map; attribute-only. S.
1448
+ - [ ] **href case/backslash resolution**: case-insensitive namemap when
1449
+ exactly one zip entry matches; backslash to slash. Kills the
1450
+ present-file RSC-007/PKG-010 slice prune cannot touch. M.
1451
+ - [ ] **Duplicate zip-entry dedupe on rewrite**: drop shadowed
1452
+ duplicates (audit already counts `dup_entries`). S.
1453
+
1454
+ ### C. Decision-gated (recorded boxes; no work without a ruling)
1455
+
1456
+ - [ ] **Cover wiring repair** (dangling `<meta name="cover">`, missing
1457
+ `properties="cover-image"`, wrong cover media-type): narrow,
1458
+ finding-driven repair vs audit-only. Needs the scope decision. M.
1459
+ - [ ] **Metadata validity carve-out ruling** (non-ISO `dc:date`,
1460
+ invalid `dc:language`, missing `dcterms:modified`): bindery opt-in
1461
+ with a determinism-safe timestamp policy, or route to
1462
+ cquarry/CalibreQuarry. Needs a written charter ruling. M.
1463
+ - [ ] **NCX<->nav drift DETECTOR** (audit-only; structural diff feeding
1464
+ `decisions_needed`; ToC synthesis stays out of charter
1465
+ permanently). L if repair is ever attempted.
1466
+ - [ ] **CSS url() pruning** as `--prune-missing-resources` scope
1467
+ extension: only after the prevalence study says it matters. M.
1468
+
1469
+ Ship shape: A leads; then B in value order (container, encryption,
1470
+ prune-edges, media-types); C boxes record their decisions and wait.
1471
+ Each repair ships with a fixture built from the prevalence study's
1472
+ real samples, a regression test written before the fix, and the gate
1473
+ (epubcheck) accepting the result.
1474
+
1475
+ ## Tag policy (decided 2026-09-11, recorded 2026-09-12)
1476
+
1477
+ The release-tag backlog is settled, forward-only:
1478
+
1479
+ - **v0.35.0 is the anchor.** Every release from v0.36.0 on is tagged at
1480
+ its release commit under auto-tag promotion (a fully qualified release
1481
+ is tagged and pushed without asking; the blitz rulebook's decision 22).
1482
+ - **The nine untagged 0.18.0-0.26.0 entries are exempted** by decision
1483
+ of record (2026-09-11): no backfill tags will be cut for them. Their
1484
+ release notes live in patchnotes.md and their wheels are on PyPI; the
1485
+ missing git tags are accepted history.
1486
+ - **Two recorded-only repairs stay force-pushes, undecided:** the
1487
+ v0.28.0 retag (the tag points at a red commit) and the v0.33.0
1488
+ em-dash tag message (the pushed tag carries two em-dashes the
1489
+ proofread missed; the source patchnotes entry was fixed immediately in
1490
+ 1c374c1). Either would need publish.yml disabled around the push and
1491
+ Brandon's explicit go; neither is ever auto-executed.
@@ -239,10 +239,16 @@ the gate has not accepted anything, so nothing is applied or written. With
239
239
  `--no-validate`, the gate is skipped and repairs are trusted on the RepairReport alone.
240
240
 
241
241
  The oracle itself runs two ways: the `epubcheck --json -` subprocess, or a persistent
242
- daemon (`_EpubcheckDaemon`, a bounded pool sized by `--workers`) that compiles
243
- `FastDaemon.java` at runtime and drives a warm JVM over a pipe. Every daemon roundtrip
244
- is bounded by the caller's timeout, and any daemon failure tears it down and falls back
245
- to the subprocess for good, so the two paths can never disagree about a book.
242
+ daemon (`_EpubcheckDaemon`, a bounded pool sized by `--workers`) that writes
243
+ `FastDaemon.java` at runtime and launches it through Java's single-file source
244
+ launcher (in-memory compilation by the running JVM: no javac, no version skew),
245
+ driving the warm JVM over a pipe. The daemon answers with the counts it reads from
246
+ the JSON that epubcheck's own `CheckingReport.generate()` serializes (the same
247
+ document `--json` produces), so both paths measure identically by construction.
248
+ (Epubcheck's JSON counts aggregated messages, not occurrences: a defect repeated
249
+ three times is one error. The gate has always measured that scale.) Every daemon
250
+ roundtrip is bounded by the caller's timeout, and any daemon failure tears it down
251
+ and falls back to the subprocess for good.
246
252
 
247
253
  ## Page-number strip (opt-in, lossy)
248
254
 
@@ -337,8 +343,8 @@ crashing the sweep.
337
343
 
338
344
  ## Audit subcommand (read-only)
339
345
 
340
- `bindery audit {content,pagenumbers,emptytext,ocr,monolithic,all} [PATH] [--max-doc-chars N]
341
- [--tag TAG] [--id IDs]` (v0.15.0, `audit.py`; `--tag` since v0.18.0; `monolithic` since v0.21.0; `--id` since v0.19.0, comma-lists in v0.23.0) inspects
346
+ `bindery audit {content,pagenumbers,emptytext,ocr,monolithic,completeness,all} [PATH] [--max-doc-chars N]
347
+ [--tag TAG] [--id IDs]` (v0.15.0, `audit.py`; `--tag` since v0.18.0; `monolithic` since v0.21.0, `--max-doc-chars N`; `--id` since v0.19.0, comma-lists in v0.23.0; `completeness` since v0.36.0) inspects
342
348
  EPUB body text for flaws epubcheck cannot see: non-English script blocks, baked-in page-number
343
349
  layers (sliding-window density heuristics), empty or thin books, systemic OCR damage, and
344
350
  single oversized content documents (one spine doc at or above 300k characters — readers refuse
@@ -347,6 +353,18 @@ is fully read for CRC + decompression, reporting CORRUPT rather than EMPTY), and
347
353
  issues. Manifest/NCX references to absent files are classified as either `convention` (ToC is bloated but
348
354
  present documents form a consecutive chapter span) or `fragment` (the span itself is broken).
349
355
 
356
+ The completeness analyzer is the phase-1 spot-check and is advisory by contract: it never
357
+ flags a book and never moves the exit code. Per book it reports the spine doc count, the
358
+ prose-doc count (spine docs carrying at least 400 visible characters), the opening and closing
359
+ 120 characters of the first/middle/last prose doc, a trailing-ToC classification of the final
360
+ spine doc, and the fraction of spine docs that could not be read (corrupt entries plus
361
+ unresolved itemrefs). A trailing ToC (the final doc is dominated by short link lines, at
362
+ least half its lines carry links, and no block runs to paragraph length) is book furniture:
363
+ it is excluded from the prose sampling so the closing excerpt comes from the real back
364
+ matter, and the verdict becomes ADVISORY (as it does at an unreadable fraction of 10% or
365
+ more). Like emptytext it is skipped wholesale when the archive verdict owns the body-text
366
+ story.
367
+
350
368
  In library mode, EPUB files are resolved through `cquarry.db.CalibreDB.get_format_path()` — the
351
369
  storage-layout logic is not duplicated here. The scan itself still writes nothing. The opt-in
352
370
  `--tag TAG` pass applies `TAG` to flagged books via `cquarry.write.WritableCalibreDB`, the
@@ -4,7 +4,7 @@ strip), epubcheck-gated.
4
4
  Single source of truth for the version. pyproject.toml must match VERSION below.
5
5
  """
6
6
 
7
- VERSION = "0.34.0"
7
+ VERSION = "0.36.0"
8
8
  __version__ = VERSION
9
9
 
10
10
  from .epub import RepairReport, repair_epub # noqa: E402