bindery-cli 0.34.0__tar.gz → 0.36.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/CLAUDE.md +3 -3
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/PKG-INFO +11 -6
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/README.md +9 -4
- bindery_cli-0.36.0/REPORT-12-Sept.md +119 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/patchnotes.md +85 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/pyproject.toml +2 -2
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/roadmap.md +118 -4
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/spec.md +24 -6
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/__init__.py +1 -1
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/audit.py +183 -5
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/cli.py +13 -4
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/validate.py +48 -16
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_audit.py +283 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_validate.py +100 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/uv.lock +5 -5
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.github/workflows/ci.yml +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.github/workflows/publish.yml +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/.gitignore +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/AGENTS.md +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/LICENSE +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/docs/screenshots/library-sweep.png +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/logo.svg +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/run_tests.sh +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/scripts/FastSweep.java +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/scripts/fast_sweep.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/__main__.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/epub.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/library.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/pagination.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/reserialize.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/transforms.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/src/bindery/watermark.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/testing_facility/books_to_fix/.gitkeep +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/testing_facility/top500candidates/REPORT.md +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_cli.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_epub.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_library.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_pagination.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_reserialize.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_transforms.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_version.py +0 -0
- {bindery_cli-0.34.0 → bindery_cli-0.36.0}/tests/test_watermark.py +0 -0
|
@@ -77,12 +77,12 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
|
|
|
77
77
|
- `src/bindery/pagination.py`: the opt-in lossy page-number strip (runhead detection, page-layer decision, block-centric removal/merge, safety nets).
|
|
78
78
|
- `src/bindery/watermark.py`: the opt-in lossy watermark strip (anchored and anchorless signature removal).
|
|
79
79
|
- `src/bindery/reserialize.py`: structural repair via `html5lib`.
|
|
80
|
-
- `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`,
|
|
80
|
+
- `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, `monolithic` since v0.21.0 with `--max-doc-chars N`, and `completeness` since v0.36.0) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story). Since v0.36.0 the completeness analyzer (Phase 15) reports the phase-1 spot-check per book: spine/prose-doc counts (prose = >= 400 visible chars), first/middle/last prose-doc opening/closing excerpts, trailing-ToC classification (`_trailing_toc`: short link lines dominate, no paragraph-length blocks), and the unreadable fraction. It is ADVISORY by contract (problem always False; ADVISORY on a trailing ToC or unreadable fraction >= 10%), never moves the exit code, and is skipped like emptytext when the archive verdict owns the book.
|
|
81
81
|
- `src/bindery/epub.py`: archive rewrite, NCX uid sync, RepairReport, mismatch detection, and the opt-in structural-repair plumbing (including the CSS precondition scan).
|
|
82
82
|
- `src/bindery/validate.py`: epubcheck wrapper, the `gate` (improvement) and `no_worse` (no-regression, for the lossy strips) acceptance bars.
|
|
83
|
-
- `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching)
|
|
83
|
+
- `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching); the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (since v0.31.0 via `set_format`, cquarry 1.17's sanctioned remove+add in one transaction); the external `calibredb` CLI is gone. Since v0.32.0 a guessed id drives a row update only when metadata.db corroborates it (`_verify_guess`: books row exists, file in the book's own directory, catalogued `data.name` when a row exists); anything else saves in place and leaves the catalog untouched — never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
|
|
84
84
|
- `src/bindery/cli.py`: `repair` and `library` subcommands, including `--all` and `--install-to-calibre`; since v0.29.0 also the `run` verbs: `phase1` (pre-import vetting over loose files: the audit battery composed with one gated repair sweep; read-only until `--apply-lossy`, which IS the lossy-strip consent) and `phase3` (the scoped post-import apply step: `library --id --sweep --only all --apply --all --install-to-calibre` plus a pre/post summary; mechanically refuses unscoped sweeps with exit 2). Both take `--non-interactive` and surface open questions as `decisions_needed` in their JSON, never prompts. They drive the shipped `run_library` through the real parser, so no flag can drift between the verb and the subcommand it wraps. `library --workers N` (v0.30.0) parallelizes only the sweep's candidate pass, in windows of N consumed in order so `--limit` stays lazy; the repair phase is never parallel (shared workdir, atomic-replacement contract).
|
|
85
|
-
- `tests/`: transforms, end-to-end repair, atomic replace, pagination, watermarks, the audit analyzers (content/pagenumbers/emptytext/ocr/monolithic, archive integrity, spine classification), the library sweep and `CalibreIdResolver`, validate, reserialize, CLI wiring, and the version pin.
|
|
85
|
+
- `tests/`: transforms, end-to-end repair, atomic replace, pagination, watermarks, the audit analyzers (content/pagenumbers/emptytext/ocr/monolithic/completeness, archive integrity, spine classification), the library sweep and `CalibreIdResolver`, validate, reserialize, CLI wiring, and the version pin.
|
|
86
86
|
|
|
87
87
|
## Conventions
|
|
88
88
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: bindery-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.36.0
|
|
4
4
|
Summary: Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library.
|
|
5
5
|
Author: vrnvctss
|
|
6
6
|
License: MIT
|
|
@@ -13,7 +13,7 @@ Classifier: License :: OSI Approved :: MIT License
|
|
|
13
13
|
Classifier: Programming Language :: Python :: 3.14
|
|
14
14
|
Classifier: Topic :: Utilities
|
|
15
15
|
Requires-Python: >=3.14
|
|
16
|
-
Requires-Dist: cquarry>=1.
|
|
16
|
+
Requires-Dist: cquarry>=1.19.0
|
|
17
17
|
Requires-Dist: tqdm
|
|
18
18
|
Requires-Dist: vir-tui>=2.2.0
|
|
19
19
|
Provides-Extra: reserialize
|
|
@@ -57,8 +57,8 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
|
|
|
57
57
|
- **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
|
|
58
58
|
- **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
|
|
59
59
|
- **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
|
|
60
|
-
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package
|
|
61
|
-
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents
|
|
60
|
+
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
|
|
61
|
+
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
|
|
62
62
|
- **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
|
|
63
63
|
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
64
64
|
- **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
|
|
@@ -82,7 +82,7 @@ Every repair is gated by [epubcheck](https://github.com/w3c/epubcheck). The acce
|
|
|
82
82
|
|
|
83
83
|
Introducing a net-new fatal is always rejected. If epubcheck itself fails to run (crash, timeout, unparsable output), the book is reported as an error and never applied; only an explicit `--no-validate` skips the gate. Originals are never modified except by an explicit, atomic in-place replace (see below), and even then only after the gate accepts the result.
|
|
84
84
|
|
|
85
|
-
Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a
|
|
85
|
+
Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a persistent daemon that writes `FastDaemon.java` into a temp directory at runtime, compiles it in memory with the running JVM's own compiler (Java's single-file source launcher: no javac, no javac/java version skew), and drives the warm JVM over a pipe. The daemon reports the counts it reads from the JSON document that epubcheck's own `CheckingReport` serializes, so it measures exactly what the subprocess measures, just ~15x faster. It is a pool bounded by `--workers` (serial runs use one), every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good.
|
|
86
86
|
|
|
87
87
|
The lossy modes (`--strip-pagination`, `--strip-broken-tags`, and `--strip-watermarks`) are the exception to the "must improve" rule. Since they remove visible markup rather than correcting XML schema violations, epubcheck counts often remain unchanged. They are accepted when the result is **no worse** (no net-new fatals or errors), relying on strict programmatic safety nets instead.
|
|
88
88
|
|
|
@@ -126,10 +126,15 @@ bindery audit ocr ~/docs/Calibre\ Library
|
|
|
126
126
|
# Flag books where a single content document is too large for readers to render
|
|
127
127
|
bindery audit monolithic ~/docs/Calibre\ Library
|
|
128
128
|
|
|
129
|
+
# Spot-check completeness: prose coverage, first/middle/last excerpts, trailing ToC
|
|
130
|
+
bindery audit completeness ~/docs/Calibre\ Library
|
|
131
|
+
|
|
129
132
|
# Run all audits and generate a comprehensive CSV
|
|
130
133
|
cd ~/docs/Calibre\ Library && bindery audit all
|
|
131
134
|
```
|
|
132
135
|
|
|
136
|
+
**Completeness spot-check:** the `completeness` audit answers the pre-import question "does this book reach its end?" It reports, per book: the prose-document count against the spine (docs with 400+ visible characters), the opening and closing text of the first/middle/last prose doc, a trailing-ToC classification of the final spine doc (a link-dense chapter list after the real back matter), and the fraction of unreadable docs. A trailing ToC is reported as an advisory, never a failure: the closing excerpt always comes from the real back matter, so a human can confirm the book ends where the author ended it.
|
|
137
|
+
|
|
133
138
|
Audits can also be run on a directory of loose `.epub` files by passing the path as the second argument:
|
|
134
139
|
```sh
|
|
135
140
|
bindery audit pagenumbers /path/to/loose/epubs
|
|
@@ -204,7 +209,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
|
|
|
204
209
|
- `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
|
|
205
210
|
- `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
|
|
206
211
|
- `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
|
|
207
|
-
- `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied
|
|
212
|
+
- `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied`; Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
|
|
208
213
|
- `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
|
|
209
214
|
- Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
|
|
210
215
|
- A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
|
|
@@ -35,8 +35,8 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
|
|
|
35
35
|
- **`--unwrap-block-in-inline`**: safely unwraps `<span>` tags that illegally contain a block-level element (e.g. `<div>` or `<p>`), leaving the block element intact.
|
|
36
36
|
- **`--strip-invalid-value`**: systematically strips invalid `value="..."` attributes from elements like `<div>`, `<span>`, `<p>`, etc.
|
|
37
37
|
- **`--fix-page-map`**: normalizes legacy page-map markup — drops the non-standard `page-map="..."` attribute from the OPF `<spine>` and adds `class="pages"` to classless NCX `<pageList>` elements (epubcheck rejects both on older HarperCollins / Anna's Archive conversions).
|
|
38
|
-
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package
|
|
39
|
-
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents
|
|
38
|
+
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
|
|
39
|
+
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
|
|
40
40
|
- **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
|
|
41
41
|
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
42
42
|
- **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
|
|
@@ -60,7 +60,7 @@ Every repair is gated by [epubcheck](https://github.com/w3c/epubcheck). The acce
|
|
|
60
60
|
|
|
61
61
|
Introducing a net-new fatal is always rejected. If epubcheck itself fails to run (crash, timeout, unparsable output), the book is reported as an error and never applied; only an explicit `--no-validate` skips the gate. Originals are never modified except by an explicit, atomic in-place replace (see below), and even then only after the gate accepts the result.
|
|
62
62
|
|
|
63
|
-
Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a
|
|
63
|
+
Under the hood, the oracle can run two ways: the usual `epubcheck --json -` subprocess, or a persistent daemon that writes `FastDaemon.java` into a temp directory at runtime, compiles it in memory with the running JVM's own compiler (Java's single-file source launcher: no javac, no javac/java version skew), and drives the warm JVM over a pipe. The daemon reports the counts it reads from the JSON document that epubcheck's own `CheckingReport` serializes, so it measures exactly what the subprocess measures, just ~15x faster. It is a pool bounded by `--workers` (serial runs use one), every roundtrip is bounded by the caller's timeout, and any failure tears it down and falls back to the subprocess for good.
|
|
64
64
|
|
|
65
65
|
The lossy modes (`--strip-pagination`, `--strip-broken-tags`, and `--strip-watermarks`) are the exception to the "must improve" rule. Since they remove visible markup rather than correcting XML schema violations, epubcheck counts often remain unchanged. They are accepted when the result is **no worse** (no net-new fatals or errors), relying on strict programmatic safety nets instead.
|
|
66
66
|
|
|
@@ -104,10 +104,15 @@ bindery audit ocr ~/docs/Calibre\ Library
|
|
|
104
104
|
# Flag books where a single content document is too large for readers to render
|
|
105
105
|
bindery audit monolithic ~/docs/Calibre\ Library
|
|
106
106
|
|
|
107
|
+
# Spot-check completeness: prose coverage, first/middle/last excerpts, trailing ToC
|
|
108
|
+
bindery audit completeness ~/docs/Calibre\ Library
|
|
109
|
+
|
|
107
110
|
# Run all audits and generate a comprehensive CSV
|
|
108
111
|
cd ~/docs/Calibre\ Library && bindery audit all
|
|
109
112
|
```
|
|
110
113
|
|
|
114
|
+
**Completeness spot-check:** the `completeness` audit answers the pre-import question "does this book reach its end?" It reports, per book: the prose-document count against the spine (docs with 400+ visible characters), the opening and closing text of the first/middle/last prose doc, a trailing-ToC classification of the final spine doc (a link-dense chapter list after the real back matter), and the fraction of unreadable docs. A trailing ToC is reported as an advisory, never a failure: the closing excerpt always comes from the real back matter, so a human can confirm the book ends where the author ended it.
|
|
115
|
+
|
|
111
116
|
Audits can also be run on a directory of loose `.epub` files by passing the path as the second argument:
|
|
112
117
|
```sh
|
|
113
118
|
bindery audit pagenumbers /path/to/loose/epubs
|
|
@@ -182,7 +187,7 @@ bindery library ~/docs/Calibre\ Library --only all --apply --all --install-to-ca
|
|
|
182
187
|
- `--json FILE` writes a machine-readable report of the whole run (per-book status, before/after counts, applied flag, summary totals). `--manual-list FILE` writes the paths of every book that was not auto-repaired, one per line, ready for manual follow-up.
|
|
183
188
|
- `--apply` is required to write; the default is a dry run. `--backup DIR` mirrors originals before replacing; `--backup-inplace` writes `.epub.bak` beside each file.
|
|
184
189
|
- `--install-to-calibre` resolves the Calibre book id from `metadata.db` via cquarry (one read-only path→id map per run), so a hand-renamed `Author/Title (id)/` directory can never send the repaired file to the wrong book. The directory-name regex is only a no-catalog fallback; with neither, the file is saved atomically in place.
|
|
185
|
-
- `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied
|
|
190
|
+
- `--install-to-calibre` installs the repaired EPUB through cquarry's write module: the file is placed atomically at the catalogued path (same filename, Calibre's layout never changes) and the format row is re-registered in one transaction, so the stored size stays truthful and the book lands in `metadata_dirtied`; Calibre regenerates its sidecar .opf on next start. No external `calibredb` call is involved. It falls back to atomic file replacement if the Calibre database ID cannot be resolved, and a database failure degrades to the in-place save with a warning. A directory-name-guessed id is trusted only when `metadata.db` corroborates it (the book exists and the file is that book's catalogued EPUB); an uncatalogued stray file inside a book directory is saved in place and the catalog row is left untouched.
|
|
186
191
|
- `--all` automatically turns on all opt-in non-fatal fixes and lossy strips (pagination, watermarks, bad attributes, unknown entities, image alt tags, etc.) in a single run.
|
|
187
192
|
- Only the `.epub` is replaced. `metadata.opf`, `cover.jpg`, and `metadata.db` are left for Calibre's Quality Check sync to reconcile.
|
|
188
193
|
- A per-book progress line goes to stderr (stdout stays a clean report); `--quiet` suppresses it. A corrupt or unreadable book is reported and skipped, never aborting the sweep.
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# REPORT-12-Sept: the repair-landscape research (bindery-cli vs real-world EPUB damage)
|
|
2
|
+
|
|
3
|
+
Research date: 2026-09-10/11. Method: a read-only research pass mapping
|
|
4
|
+
the full taxonomy of real-world EPUB damage — by producer (Calibre
|
|
5
|
+
conversions, Word/Office export, InDesign, legacy OCR, Adobe Digital
|
|
6
|
+
Editions, KindleUnpack, self-pub aggregators, Windows case-manglers) —
|
|
7
|
+
against this repo's shipped repair surface, the Phase 14/15 backlogs,
|
|
8
|
+
and the Bug Reports. References include the upstream clone's format
|
|
9
|
+
readers. Full evidence lives in the agent's findings; this report is
|
|
10
|
+
the distilled verdict and every actionable gap with its routing.
|
|
11
|
+
|
|
12
|
+
## Verdict
|
|
13
|
+
|
|
14
|
+
Bindery handles roughly **60-65% of real-world damage classes
|
|
15
|
+
end-to-end** and is near-complete for its founding niche: markup
|
|
16
|
+
well-formedness (the conversion-fatals class), NCX basics,
|
|
17
|
+
zip/mimetype container hygiene, dead-reference pruning, watermark and
|
|
18
|
+
page-number furniture, and archive/spine diagnosis (corruption, DRM
|
|
19
|
+
skip, ToC-bloat classification). Phase 14's hardening and all five Bug
|
|
20
|
+
Reports are shipped through v0.35.0; Phase 15 (the completeness
|
|
21
|
+
analyzer) is approved.
|
|
22
|
+
|
|
23
|
+
The missing ~35% is not more markup surgery. It is **package-level
|
|
24
|
+
repairs** — container.xml, manifest media-types, spine attribute
|
|
25
|
+
cleanup, encryption.xml/obfuscation state, zip-level dedupe, href
|
|
26
|
+
case/IRI mismatches — and the finding that matters most: several of
|
|
27
|
+
those classes are **gateway defects**. Epubcheck stays fatal while
|
|
28
|
+
they persist, so no shipped repair can ever be gate-accepted on those
|
|
29
|
+
books. They are locked out of the pipeline entirely.
|
|
30
|
+
|
|
31
|
+
## A. The damage-taxonomy matrix and the repair gaps
|
|
32
|
+
|
|
33
|
+
1. **Missing/corrupt `META-INF/container.xml`** (any producer):
|
|
34
|
+
epubcheck fatal; the OPF is unfindable. The locator falls back to
|
|
35
|
+
the first `.opf` but never writes the file. Deterministic fix:
|
|
36
|
+
generate the standard container.xml at the located OPF (upstream
|
|
37
|
+
calibre's `initialize_container` is the model). Effort S; HIGH
|
|
38
|
+
value — a gateway repair that unlocks an entire class of books.
|
|
39
|
+
2. **Stale font-obfuscation state** (publisher EPUB3 fonts; IDPF
|
|
40
|
+
`2008/embedding` and Adobe `pdf/enc#RC`): post-KindleUnpack/strip
|
|
41
|
+
tools decrypt fonts in place but leave `encryption.xml` declaring
|
|
42
|
+
obfuscation; epubcheck errors and the audit implies DRM where there
|
|
43
|
+
is none. Fix: detect via font magic bytes at offset 0 vs the
|
|
44
|
+
CipherReference; prune stale entries; drop encryption.xml when only
|
|
45
|
+
obfuscation entries remain and the fonts verify. Never
|
|
46
|
+
un-obfuscate. Also gives the audit a real OBFUSCATED-vs-DRM
|
|
47
|
+
distinction. Effort M; needs the magic-byte detection validated on
|
|
48
|
+
real samples first.
|
|
49
|
+
3. **Wrong/missing manifest `media-type`** (aggregators, ADE): fonts,
|
|
50
|
+
images, and media declared with wrong types. Fix: sniff magic bytes
|
|
51
|
+
or extension map; attribute-only. Effort S.
|
|
52
|
+
4. **href case/backslash/IRI mismatches** (KindleUnpack, Windows
|
|
53
|
+
producers): `Images/Cover.JPG` vs `images/cover.jpg`, backslash
|
|
54
|
+
separators, `%XX` variants — the file IS present, so
|
|
55
|
+
`--prune-missing-resources` cannot touch the RSC-007/PKG-010 it
|
|
56
|
+
causes. Fix: case-insensitive namemap when exactly one match;
|
|
57
|
+
backslash to slash. Effort M.
|
|
58
|
+
5. **Dangling reference edges after pruning** (any producer):
|
|
59
|
+
`spine@toc`, `item@media-overlay`, `item@fallback`, and EPUB2 cover
|
|
60
|
+
meta pointing at items the prune pass deleted. Today's prune checks
|
|
61
|
+
only `spine_ids` — it can manufacture the regression the gate then
|
|
62
|
+
rejects. Fix: extend prune to rewrite those edges (the rewrite
|
|
63
|
+
machinery exists in fix_manifest_ids). Effort S/M.
|
|
64
|
+
6. **Duplicate zip entries** (any producer): last-wins shadowing; the
|
|
65
|
+
audit counts `dup_entries`, the rewrite copies both. Fix: drop
|
|
66
|
+
shadowed duplicates on rewrite. Effort S.
|
|
67
|
+
7. **Cover wiring** (EPUB2 producers): dangling `<meta name="cover">`,
|
|
68
|
+
missing EPUB3 `properties="cover-image"`, wrong cover media-type —
|
|
69
|
+
calibre and booksellers extract no cover. Narrow, finding-driven
|
|
70
|
+
repair when the cover is identifiable from the existing guide/meta;
|
|
71
|
+
audit-only otherwise. Effort M.
|
|
72
|
+
8. **Broken CSS from converters**: `url()` to absent assets, malformed
|
|
73
|
+
selectors. Repair is fiddly; ship the audit-only detector first and
|
|
74
|
+
let prevalence data decide. Effort M (repair).
|
|
75
|
+
9. **Human-facing metadata validity** (self-pub/Word): non-ISO
|
|
76
|
+
`dc:date`, invalid `dc:language`, missing EPUB3
|
|
77
|
+
`dcterms:modified`. Explicit non-goal today; `dcterms:modified`
|
|
78
|
+
also collides with byte-determinism. Needs a written charter
|
|
79
|
+
ruling: carve-out with a determinism-safe timestamp policy, or
|
|
80
|
+
route to cquarry/CalibreQuarry. Effort M + a decision.
|
|
81
|
+
10. **EPUB3 dual-ToC drift** (nav vs NCX): detection is a
|
|
82
|
+
deterministic structural diff feeding `decisions_needed`; full
|
|
83
|
+
sync is ToC synthesis and stays out of charter permanently.
|
|
84
|
+
Effort L if repair is ever attempted; audit-only recommended.
|
|
85
|
+
11. **No-repair classes, correctly routed**: OCR garbage, truncated
|
|
86
|
+
re-sources, absent font families (audit + re-source); real DRM
|
|
87
|
+
(clean ENCRYPTED skip — shipped); DB-level format rows
|
|
88
|
+
(cquarry's, done); catalog completeness and human-facing metadata
|
|
89
|
+
quality (CalibreQuarry's audits).
|
|
90
|
+
|
|
91
|
+
## B. Research-before-boxing
|
|
92
|
+
|
|
93
|
+
1. **FastSweep prevalence study** — run `fast_sweep.py --mode=extract`
|
|
94
|
+
aggregation over the full library targeting the new classes
|
|
95
|
+
(container.xml presence/validity, encryption.xml algorithms,
|
|
96
|
+
media-type warnings, case-mismatch RSC-007s). The harness exists;
|
|
97
|
+
it sizes every box above with real counts before any is built.
|
|
98
|
+
2. **The metadata carve-out ruling** (gap 9) — Brandon decides;
|
|
99
|
+
the audit records both options.
|
|
100
|
+
3. **The obfuscation magic-byte table** (gap 2) — validate IDPF and
|
|
101
|
+
Adobe font prefixes against real samples before the repair ships.
|
|
102
|
+
|
|
103
|
+
## C. Routing notes
|
|
104
|
+
|
|
105
|
+
Package-structure repairs (items 1-6) fit the existing "structural
|
|
106
|
+
opt-ins + gate" model without reopening the semantics-preserving
|
|
107
|
+
contract. Items 7-10 each need an explicit scope decision first.
|
|
108
|
+
Cross-repo: DB-level size/format-row mismatches route to cquarry
|
|
109
|
+
(set_format, shipped); catalog completeness, duplicate detection, and
|
|
110
|
+
human-facing metadata quality route to CalibreQuarry's audits; OCR,
|
|
111
|
+
truncation, monolithic, and missing-font content stay audit-and-
|
|
112
|
+
re-source permanently.
|
|
113
|
+
|
|
114
|
+
## D. Phase 16 routing
|
|
115
|
+
|
|
116
|
+
Items 1-6 (the ungated package-structure repairs) are boxed in
|
|
117
|
+
roadmap.md **Phase 16** with the FastSweep prevalence study leading.
|
|
118
|
+
Items 7-10 are recorded as decision-gated boxes. Phase 15 (approved)
|
|
119
|
+
runs first if its fixture work is not yet done.
|
|
@@ -1,4 +1,89 @@
|
|
|
1
1
|
# bindery-cli Patch Notes
|
|
2
|
+
## v0.36.0 (2026-09-12)
|
|
3
|
+
|
|
4
|
+
### The completeness spot-check: `audit completeness` (Phase 15)
|
|
5
|
+
|
|
6
|
+
- **The phase-1 judgment step finally has a tool owner.** The import
|
|
7
|
+
skill's completeness step asks a human question with no tool behind
|
|
8
|
+
it: "sample early, middle, and late pages ... read the LAST content
|
|
9
|
+
page to confirm it reaches real back matter rather than cutting off
|
|
10
|
+
mid-chapter". Two consecutive phase-1 runs answered it with a
|
|
11
|
+
hand-rolled zipfile sampler in /tmp. The new analyzer rides the
|
|
12
|
+
audit's existing single decompression pass (no second decode of any
|
|
13
|
+
book) and reports, per book: spine doc count, prose-doc count (spine
|
|
14
|
+
docs with at least 400 visible characters), the opening and closing
|
|
15
|
+
120 characters of the first/middle/last prose doc, a trailing-ToC
|
|
16
|
+
classification of the final spine doc, and the fraction of unreadable
|
|
17
|
+
docs (corrupt entries plus unresolved itemrefs).
|
|
18
|
+
- **A trailing ToC never impersonates the ending.** The final spine doc
|
|
19
|
+
is classified as a chapter list when short link lines dominate it and
|
|
20
|
+
no block runs to paragraph length; such a doc is book furniture and
|
|
21
|
+
is excluded from the prose sampling, so the closing excerpt comes
|
|
22
|
+
from the real back matter. Verified against the real 2026-09-10
|
|
23
|
+
Redwall fixtures: Lord Brocktree's trailing ToC is caught and the
|
|
24
|
+
Epilogue is sampled as the last prose doc; The Long Patrol reads
|
|
25
|
+
exactly 55/57 prose docs through the percent-encoded hrefs that
|
|
26
|
+
defeated the hand-rolled sampler; Mattimeo's split doc (chapter
|
|
27
|
+
headings with real prose after every one) stays prose.
|
|
28
|
+
- **Advisory by contract.** The analyzer never flags a book and never
|
|
29
|
+
moves the exit code: the archive, spine, and emptytext verdicts keep
|
|
30
|
+
owning the flags. The verdict status is ADVISORY (not OK) when a
|
|
31
|
+
trailing ToC is present or the unreadable fraction reaches 10%.
|
|
32
|
+
`audit all` and `run phase1` include it through the battery, the CLI
|
|
33
|
+
grows the `completeness` mode, and `--json` records carry its
|
|
34
|
+
verdicts in the same shape as the other analyzers.
|
|
35
|
+
|
|
36
|
+
### Dependencies
|
|
37
|
+
|
|
38
|
+
- Adopted cquarry 1.18.0 (floor + lock): the set_format repair lane now keeps
|
|
39
|
+
Calibre's FTS index and page counts honest automatically (the sidecar's
|
|
40
|
+
dirty queue and the pages rescan flag ride the write), and the search and
|
|
41
|
+
runtime fixes come along in the shared layer.
|
|
42
|
+
- Adopted cquarry 1.19.0 (floor + lock): the repair lane gains its undo
|
|
43
|
+
primitive -- save_original_format/restore_original_format -- so a format
|
|
44
|
+
repair can put the original bytes back if it goes wrong.
|
|
45
|
+
|
|
46
|
+
### Repo records
|
|
47
|
+
|
|
48
|
+
- The release-tag backlog is settled forward-only from v0.35.0: the
|
|
49
|
+
nine untagged 0.18.0-0.26.0 entries are exempted by decision of
|
|
50
|
+
record (2026-09-11), and the v0.28.0 retag and the v0.33.0 em-dash
|
|
51
|
+
tag repair stay recorded-only force-pushes (roadmap.md, "Tag
|
|
52
|
+
policy").
|
|
53
|
+
|
|
54
|
+
## v0.35.0 (2026-09-10)
|
|
55
|
+
|
|
56
|
+
### The epubcheck daemon, reconciled: oracle-identical counts at ~15x the speed
|
|
57
|
+
|
|
58
|
+
- **FastDaemon v2 measures exactly what the subprocess oracle measures.**
|
|
59
|
+
The daemon's old Java side counted message occurrences from
|
|
60
|
+
`CheckingReport`'s counters, while the gate has always measured the
|
|
61
|
+
counts in epubcheck's own `--json` output, which aggregates identical
|
|
62
|
+
messages (one book: 243 occurrences, 4 aggregated messages). On the
|
|
63
|
+
staged-fixture corpus the two methods diverged on 11 of 14 books, so
|
|
64
|
+
the daemon could never be enabled without changing what the gate
|
|
65
|
+
measures. The new daemon calls `CheckingReport.generate()` and reads
|
|
66
|
+
`nFatal`/`nError`/`nWarning` out of the JSON epubcheck itself
|
|
67
|
+
serializes: identical numbers by construction. Verified against the
|
|
68
|
+
subprocess oracle on all 14 staged books and on a 100-book
|
|
69
|
+
real-library sample: 114/114 identical.
|
|
70
|
+
- **No more javac, no more version skew.** The daemon now launches
|
|
71
|
+
through Java's single-file source launcher (JEP 330): the running JVM
|
|
72
|
+
compiles the daemon in memory with its own compiler, so the
|
|
73
|
+
javac-newer-than-java skew that kept the daemon dead on this machine
|
|
74
|
+
(javac 27-ea, java 25) cannot recur on any toolchain.
|
|
75
|
+
- **Measured, working daemon throughput: ~0.27s/book warm against
|
|
76
|
+
~4.2s/book for the subprocess oracle** (15.6x). With `--workers N`
|
|
77
|
+
each worker gets its own warm daemon from the bounded pool, and every
|
|
78
|
+
roundtrip stays bounded by the caller's timeout with the fail-safe
|
|
79
|
+
fallback to the subprocess oracle.
|
|
80
|
+
- **The dedup semantics are pinned by a suite test:** a book engineered
|
|
81
|
+
with one defect repeated three times plus one different defect must
|
|
82
|
+
answer 2 errors (the aggregated count), not 4 occurrences (the old
|
|
83
|
+
daemon's answer). Skipped where no epubcheck/JDK exists, like the
|
|
84
|
+
other oracle tests. README and spec's gate sections describe the new
|
|
85
|
+
design.
|
|
86
|
+
|
|
2
87
|
## v0.34.0 (2026-09-10)
|
|
3
88
|
|
|
4
89
|
### Brandon's three calls: partial books are trouble, the contract reconciled, the daemon decision informed
|
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "bindery-cli"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.36.0"
|
|
8
8
|
description = "Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
classifiers = [
|
|
@@ -27,7 +27,7 @@ authors = [
|
|
|
27
27
|
dependencies = [
|
|
28
28
|
"tqdm",
|
|
29
29
|
"vir-tui>=2.2.0",
|
|
30
|
-
"cquarry>=1.
|
|
30
|
+
"cquarry>=1.19.0",
|
|
31
31
|
]
|
|
32
32
|
|
|
33
33
|
[project.optional-dependencies]
|
|
@@ -663,7 +663,7 @@ book instead raises an uncaught `TypeError` at the `book["path"]` access.
|
|
|
663
663
|
matches the DB's book path and format name before the batch; otherwise save
|
|
664
664
|
in place and warn without touching the row, and catch the `TypeError` path.
|
|
665
665
|
*(Fixed in v0.32.0: a guessed id now drives a row update only when
|
|
666
|
-
metadata.db corroborates it
|
|
666
|
+
metadata.db corroborates it: the books row exists, the file sits in that
|
|
667
667
|
book's catalogued directory, and when an EPUB row exists it carries that
|
|
668
668
|
row's stored name. Any other guess (a stray file in a book directory, a
|
|
669
669
|
stale `(N)` directory whose book is gone, the resolver's id gone from
|
|
@@ -996,7 +996,7 @@ can be net-neutral and ships silently.*
|
|
|
996
996
|
v0.22.0 "its own verdict in every mode" promise (`audit.py:1565`,
|
|
997
997
|
`1783-1803`, `1520-1558`).
|
|
998
998
|
*(Done in v0.33.0: the archive and spine sections run unconditionally
|
|
999
|
-
after the selected content sections
|
|
999
|
+
after the selected content sections; they are always-on verdicts and
|
|
1000
1000
|
were already scanned and recorded in the JSON; only the console/rc
|
|
1001
1001
|
path was dead. A CRC-corrupt book now prints its CORRUPT section and
|
|
1002
1002
|
fails library mode with exit 1, matching directory mode. Test pins
|
|
@@ -1156,7 +1156,24 @@ can be net-neutral and ships silently.*
|
|
|
1156
1156
|
javac at all), but the daemon should stay off until its counting
|
|
1157
1157
|
matches the JSON oracle; retire-or-reconcile remains the open
|
|
1158
1158
|
decision. The daemon already self-disables safely, and --workers
|
|
1159
|
-
parallelism works via subprocesses.)
|
|
1159
|
+
parallelism works via subprocesses.)
|
|
1160
|
+
*(DECIDED 2026-09-10: Brandon chose RECONCILE over retire. The
|
|
1161
|
+
counting divergence turned out to run deeper than the daemon: the
|
|
1162
|
+
CLI's human summary counts message occurrences (243 on one book)
|
|
1163
|
+
while epubcheck's own --json output counts aggregated messages
|
|
1164
|
+
(4 on the same book), and the gate has always measured the
|
|
1165
|
+
aggregated scale. The reconciliation is therefore by construction:
|
|
1166
|
+
FastDaemon v2 calls CheckingReport.generate() and reads nFatal/
|
|
1167
|
+
nError/nWarning from the JSON epubcheck itself serializes, so the
|
|
1168
|
+
daemon measures the subprocess oracle's numbers exactly. Verified
|
|
1169
|
+
against the subprocess oracle on all 14 staged books (the same
|
|
1170
|
+
corpus that diverged 11/14 under the old daemon) and on a
|
|
1171
|
+
100-book real-library sample. The launch switched to Java's
|
|
1172
|
+
single-file source launcher, removing javac and the version skew
|
|
1173
|
+
permanently; the dedup semantics are pinned by a suite test
|
|
1174
|
+
(skipped where no epubcheck/JDK exists). Daemon warm throughput
|
|
1175
|
+
measured at ~0.27s/book vs ~4.2s/book subprocess. Shipped in
|
|
1176
|
+
v0.35.0.)**
|
|
1160
1177
|
- [x] **Smaller apply papercuts:** `--limit -1` and a missing
|
|
1161
1178
|
`--audit` file raise tracebacks instead of usage errors
|
|
1162
1179
|
(`cli.py:434`, `373`); the fresh-format branch of `install_format`
|
|
@@ -1269,7 +1286,7 @@ can be net-neutral and ships silently.*
|
|
|
1269
1286
|
|
|
1270
1287
|
### Documentation (drift is real but narrow)
|
|
1271
1288
|
|
|
1272
|
-
- [
|
|
1289
|
+
- [x] **Reconcile spec.md with the shipped opt-ins.** The contract's
|
|
1273
1290
|
non-goals forbid exactly what `--reserialize` (v0.3.0, the lone
|
|
1274
1291
|
html5lib dependency) and `--strip-bad-attrs` (v0.4.0) do, and neither
|
|
1275
1292
|
has a spec section; the OPF half of `--fix-ids` is missing;
|
|
@@ -1277,6 +1294,10 @@ can be net-neutral and ships silently.*
|
|
|
1277
1294
|
NCX sentence at spec.md:52-53 is garbled (a dangling "to the OPF
|
|
1278
1295
|
unique identifier" tail). A contributor reading spec.md first would
|
|
1279
1296
|
conclude two shipped flags violate the charter.
|
|
1297
|
+
*(APPLIED 2026-09-10, commit 01ad173, on Brandon's verbatim go
|
|
1298
|
+
recorded in the audit sheet's evening answers; shipped with
|
|
1299
|
+
v0.34.0. Ticked 2026-09-11 in the main-thread double-check after
|
|
1300
|
+
the verifier caught the work shipping unticked.)*
|
|
1280
1301
|
**GATED on Brandon: spec.md is the contract file. Draft proposal,
|
|
1281
1302
|
ready to apply on a go (do not apply without one):**
|
|
1282
1303
|
(1) Non-goals bullet 2 ("Repairing genuinely mangled structure
|
|
@@ -1351,6 +1372,14 @@ everything else is backlog, not danger.*
|
|
|
1351
1372
|
|
|
1352
1373
|
## Phase 15: completeness spot-check analyzer (proposed 2026-09-10, from the phase-1 skill's own doctrine)
|
|
1353
1374
|
|
|
1375
|
+
*(SHIPPED 2026-09-12 as v0.36.0: `audit completeness` rides the existing
|
|
1376
|
+
single decompression pass; the Redwall-shaped regression tests landed
|
|
1377
|
+
first (4021ce7), the analyzer second (e6156c0). Verified against the
|
|
1378
|
+
real fixtures: Lord Brocktree's trailing ToC is caught with the Epilogue
|
|
1379
|
+
sampled as the last prose doc, The Long Patrol reads exactly 55/57
|
|
1380
|
+
prose docs through its percent-encoded hrefs, and Mattimeo's split doc
|
|
1381
|
+
stays prose. Advisory by contract; `all` and `run phase1` include it.)*
|
|
1382
|
+
|
|
1354
1383
|
*(RAISED with Brandon 2026-09-10, batched with the session-start raises;
|
|
1355
1384
|
recommendation: APPROVE. Execution is gated on that go; the proposal
|
|
1356
1385
|
below is filed and costed. The Redwall run it names is a ready-made
|
|
@@ -1375,3 +1404,88 @@ Brocktree (real Epilogue doc before a trailing 520-char ToC), Mattimeo
|
|
|
1375
1404
|
(Chapter 50 verified as a real heading with prose after, inside a 174k-char
|
|
1376
1405
|
split doc), The Long Patrol (percent-encoded hrefs defeated the hand-rolled
|
|
1377
1406
|
reader; 55/57 prose docs).*
|
|
1407
|
+
|
|
1408
|
+
## Phase 16: package-structure repairs (proposed 2026-09-12, from REPORT-12-Sept.md)
|
|
1409
|
+
|
|
1410
|
+
A research pass mapped the full real-world EPUB damage taxonomy against
|
|
1411
|
+
the shipped repair surface: bindery handles ~60-65% of damage classes
|
|
1412
|
+
end-to-end, and the missing slice is package-level structure — several
|
|
1413
|
+
classes of which are GATEWAY DEFECTS (epubcheck stays fatal, so no
|
|
1414
|
+
shipped repair can ever be gate-accepted on those books). Full evidence,
|
|
1415
|
+
producer classes, and the cross-repo routing live in
|
|
1416
|
+
REPORT-12-Sept.md. Lead with the prevalence study; every box below is
|
|
1417
|
+
an opt-in structural repair under the existing gate contract.
|
|
1418
|
+
|
|
1419
|
+
### A. Research first (boxes cannot be honestly sized without these)
|
|
1420
|
+
|
|
1421
|
+
- [ ] **FastSweep prevalence study**: run `fast_sweep.py
|
|
1422
|
+
--mode=extract` aggregation over the full library for the new
|
|
1423
|
+
classes — container.xml presence/validity, encryption.xml
|
|
1424
|
+
algorithms, manifest media-type warnings, case-mismatch RSC-007s.
|
|
1425
|
+
Each Phase 16 repair box gets its real count before it opens.
|
|
1426
|
+
- [ ] **Obfuscation magic-byte validation**: confirm the IDPF
|
|
1427
|
+
(`2008/embedding`) and Adobe (`pdf/enc#RC`) font prefixes at offset 0
|
|
1428
|
+
against real samples before the encryption-repair box opens.
|
|
1429
|
+
|
|
1430
|
+
### B. The repairs (each: opt-in flag, dry-run default, gate-accepted)
|
|
1431
|
+
|
|
1432
|
+
- [ ] **`--fix-container`**: generate META-INF/container.xml at the
|
|
1433
|
+
located OPF (model: upstream calibre's initialize_container).
|
|
1434
|
+
Gateway repair, effort S, HIGH value.
|
|
1435
|
+
- [ ] **Obfuscation-aware encryption.xml repair**: prune stale
|
|
1436
|
+
obfuscation entries when font magic bytes prove the fonts are
|
|
1437
|
+
decrypted in place; drop encryption.xml when only obfuscation entries
|
|
1438
|
+
remain and fonts verify. NEVER un-obfuscate. M.
|
|
1439
|
+
- [ ] **Audit distinction: OBFUSCATED vs DRM** in audit.py (today the
|
|
1440
|
+
ENCRYPTED verdict keys on unreadable entries and implies DRM where
|
|
1441
|
+
publisher obfuscation is benign). S, rides the box above.
|
|
1442
|
+
- [ ] **Reference-edge completion for `--prune-missing-resources`**:
|
|
1443
|
+
rewrite `spine@toc`, `item@media-overlay`, `item@fallback`, and EPUB2
|
|
1444
|
+
cover-meta edges when their targets are pruned (machinery exists in
|
|
1445
|
+
fix_manifest_ids). Prevents manufacturing gate regressions. S/M.
|
|
1446
|
+
- [ ] **Manifest media-type normalization**: sniff magic bytes /
|
|
1447
|
+
extension map; attribute-only. S.
|
|
1448
|
+
- [ ] **href case/backslash resolution**: case-insensitive namemap when
|
|
1449
|
+
exactly one zip entry matches; backslash to slash. Kills the
|
|
1450
|
+
present-file RSC-007/PKG-010 slice prune cannot touch. M.
|
|
1451
|
+
- [ ] **Duplicate zip-entry dedupe on rewrite**: drop shadowed
|
|
1452
|
+
duplicates (audit already counts `dup_entries`). S.
|
|
1453
|
+
|
|
1454
|
+
### C. Decision-gated (recorded boxes; no work without a ruling)
|
|
1455
|
+
|
|
1456
|
+
- [ ] **Cover wiring repair** (dangling `<meta name="cover">`, missing
|
|
1457
|
+
`properties="cover-image"`, wrong cover media-type): narrow,
|
|
1458
|
+
finding-driven repair vs audit-only. Needs the scope decision. M.
|
|
1459
|
+
- [ ] **Metadata validity carve-out ruling** (non-ISO `dc:date`,
|
|
1460
|
+
invalid `dc:language`, missing `dcterms:modified`): bindery opt-in
|
|
1461
|
+
with a determinism-safe timestamp policy, or route to
|
|
1462
|
+
cquarry/CalibreQuarry. Needs a written charter ruling. M.
|
|
1463
|
+
- [ ] **NCX<->nav drift DETECTOR** (audit-only; structural diff feeding
|
|
1464
|
+
`decisions_needed`; ToC synthesis stays out of charter
|
|
1465
|
+
permanently). L if repair is ever attempted.
|
|
1466
|
+
- [ ] **CSS url() pruning** as `--prune-missing-resources` scope
|
|
1467
|
+
extension: only after the prevalence study says it matters. M.
|
|
1468
|
+
|
|
1469
|
+
Ship shape: A leads; then B in value order (container, encryption,
|
|
1470
|
+
prune-edges, media-types); C boxes record their decisions and wait.
|
|
1471
|
+
Each repair ships with a fixture built from the prevalence study's
|
|
1472
|
+
real samples, a regression test written before the fix, and the gate
|
|
1473
|
+
(epubcheck) accepting the result.
|
|
1474
|
+
|
|
1475
|
+
## Tag policy (decided 2026-09-11, recorded 2026-09-12)
|
|
1476
|
+
|
|
1477
|
+
The release-tag backlog is settled, forward-only:
|
|
1478
|
+
|
|
1479
|
+
- **v0.35.0 is the anchor.** Every release from v0.36.0 on is tagged at
|
|
1480
|
+
its release commit under auto-tag promotion (a fully qualified release
|
|
1481
|
+
is tagged and pushed without asking; the blitz rulebook's decision 22).
|
|
1482
|
+
- **The nine untagged 0.18.0-0.26.0 entries are exempted** by decision
|
|
1483
|
+
of record (2026-09-11): no backfill tags will be cut for them. Their
|
|
1484
|
+
release notes live in patchnotes.md and their wheels are on PyPI; the
|
|
1485
|
+
missing git tags are accepted history.
|
|
1486
|
+
- **Two recorded-only repairs stay force-pushes, undecided:** the
|
|
1487
|
+
v0.28.0 retag (the tag points at a red commit) and the v0.33.0
|
|
1488
|
+
em-dash tag message (the pushed tag carries two em-dashes the
|
|
1489
|
+
proofread missed; the source patchnotes entry was fixed immediately in
|
|
1490
|
+
1c374c1). Either would need publish.yml disabled around the push and
|
|
1491
|
+
Brandon's explicit go; neither is ever auto-executed.
|
|
@@ -239,10 +239,16 @@ the gate has not accepted anything, so nothing is applied or written. With
|
|
|
239
239
|
`--no-validate`, the gate is skipped and repairs are trusted on the RepairReport alone.
|
|
240
240
|
|
|
241
241
|
The oracle itself runs two ways: the `epubcheck --json -` subprocess, or a persistent
|
|
242
|
-
daemon (`_EpubcheckDaemon`, a bounded pool sized by `--workers`) that
|
|
243
|
-
`FastDaemon.java` at runtime and
|
|
244
|
-
|
|
245
|
-
|
|
242
|
+
daemon (`_EpubcheckDaemon`, a bounded pool sized by `--workers`) that writes
|
|
243
|
+
`FastDaemon.java` at runtime and launches it through Java's single-file source
|
|
244
|
+
launcher (in-memory compilation by the running JVM: no javac, no version skew),
|
|
245
|
+
driving the warm JVM over a pipe. The daemon answers with the counts it reads from
|
|
246
|
+
the JSON that epubcheck's own `CheckingReport.generate()` serializes (the same
|
|
247
|
+
document `--json` produces), so both paths measure identically by construction.
|
|
248
|
+
(Epubcheck's JSON counts aggregated messages, not occurrences: a defect repeated
|
|
249
|
+
three times is one error. The gate has always measured that scale.) Every daemon
|
|
250
|
+
roundtrip is bounded by the caller's timeout, and any daemon failure tears it down
|
|
251
|
+
and falls back to the subprocess for good.
|
|
246
252
|
|
|
247
253
|
## Page-number strip (opt-in, lossy)
|
|
248
254
|
|
|
@@ -337,8 +343,8 @@ crashing the sweep.
|
|
|
337
343
|
|
|
338
344
|
## Audit subcommand (read-only)
|
|
339
345
|
|
|
340
|
-
`bindery audit {content,pagenumbers,emptytext,ocr,monolithic,all} [PATH] [--max-doc-chars N]
|
|
341
|
-
[--tag TAG] [--id IDs]` (v0.15.0, `audit.py`; `--tag` since v0.18.0; `monolithic` since v0.21.0
|
|
346
|
+
`bindery audit {content,pagenumbers,emptytext,ocr,monolithic,completeness,all} [PATH] [--max-doc-chars N]
|
|
347
|
+
[--tag TAG] [--id IDs]` (v0.15.0, `audit.py`; `--tag` since v0.18.0; `monolithic` since v0.21.0, `--max-doc-chars N`; `--id` since v0.19.0, comma-lists in v0.23.0; `completeness` since v0.36.0) inspects
|
|
342
348
|
EPUB body text for flaws epubcheck cannot see: non-English script blocks, baked-in page-number
|
|
343
349
|
layers (sliding-window density heuristics), empty or thin books, systemic OCR damage, and
|
|
344
350
|
single oversized content documents (one spine doc at or above 300k characters — readers refuse
|
|
@@ -347,6 +353,18 @@ is fully read for CRC + decompression, reporting CORRUPT rather than EMPTY), and
|
|
|
347
353
|
issues. Manifest/NCX references to absent files are classified as either `convention` (ToC is bloated but
|
|
348
354
|
present documents form a consecutive chapter span) or `fragment` (the span itself is broken).
|
|
349
355
|
|
|
356
|
+
The completeness analyzer is the phase-1 spot-check and is advisory by contract: it never
|
|
357
|
+
flags a book and never moves the exit code. Per book it reports the spine doc count, the
|
|
358
|
+
prose-doc count (spine docs carrying at least 400 visible characters), the opening and closing
|
|
359
|
+
120 characters of the first/middle/last prose doc, a trailing-ToC classification of the final
|
|
360
|
+
spine doc, and the fraction of spine docs that could not be read (corrupt entries plus
|
|
361
|
+
unresolved itemrefs). A trailing ToC (the final doc is dominated by short link lines, at
|
|
362
|
+
least half its lines carry links, and no block runs to paragraph length) is book furniture:
|
|
363
|
+
it is excluded from the prose sampling so the closing excerpt comes from the real back
|
|
364
|
+
matter, and the verdict becomes ADVISORY (as it does at an unreadable fraction of 10% or
|
|
365
|
+
more). Like emptytext it is skipped wholesale when the archive verdict owns the body-text
|
|
366
|
+
story.
|
|
367
|
+
|
|
350
368
|
In library mode, EPUB files are resolved through `cquarry.db.CalibreDB.get_format_path()` — the
|
|
351
369
|
storage-layout logic is not duplicated here. The scan itself still writes nothing. The opt-in
|
|
352
370
|
`--tag TAG` pass applies `TAG` to flagged books via `cquarry.write.WritableCalibreDB`, the
|