bindery-cli 0.37.0__tar.gz → 0.38.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/.github/workflows/publish.yml +2 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/CLAUDE.md +8 -6
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/PKG-INFO +4 -2
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/README.md +3 -1
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/patchnotes.md +57 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/pyproject.toml +1 -1
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/roadmap.md +58 -5
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/spec.md +25 -4
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/__init__.py +1 -1
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/audit.py +133 -41
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/cli.py +25 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/epub.py +195 -5
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_audit.py +140 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_epub.py +288 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/.github/workflows/ci.yml +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/.gitignore +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/AGENTS.md +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/LICENSE +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/REPORT-12-Sept.md +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/docs/screenshots/library-sweep.png +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/logo.svg +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/plugin/__init__.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/run_tests.sh +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/scripts/FastSweep.java +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/scripts/build_plugin.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/scripts/fast_sweep.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/__main__.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/library.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/pagination.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/reserialize.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/transforms.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/validate.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/src/bindery/watermark.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/testing_facility/books_to_fix/.gitkeep +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/testing_facility/top500candidates/REPORT.md +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_cli.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_library.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_pagination.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_plugin.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_reserialize.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_transforms.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_validate.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_version.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/tests/test_watermark.py +0 -0
- {bindery_cli-0.37.0 → bindery_cli-0.38.0}/uv.lock +0 -0
|
@@ -29,9 +29,9 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
|
|
|
29
29
|
* **Structural repairs** (`--fix-empty-body`, `--fix-missing-title`, `--fix-id-colons`,
|
|
30
30
|
`--fix-page-map`, `--strip-epub3-attrs`, `--downgrade-epub3-tags`,
|
|
31
31
|
`--unwrap-block-in-inline`, `--strip-invalid-value`, `--unwrap-illegal-tags`,
|
|
32
|
-
`--prune-missing-resources`, `--strip-broken-anchors`, `--encode-url-spaces
|
|
33
|
-
transforms.py, threaded through epub.py):
|
|
34
|
-
minimal content. v0.14–v0.16 ran these unconditionally, which broke this rule;
|
|
32
|
+
`--prune-missing-resources`, `--strip-broken-anchors`, `--encode-url-spaces`,
|
|
33
|
+
`--fix-container`, `--fix-media-types`; transforms.py, threaded through epub.py):
|
|
34
|
+
they alter markup structure or fabricate minimal content. v0.14–v0.16 ran these unconditionally, which broke this rule;
|
|
35
35
|
v0.17.0 restored it. `--unwrap-illegal-tags` additionally protects any illegal-tag
|
|
36
36
|
name that an EPUB stylesheet styles as an element selector (`css_protected_tags`,
|
|
37
37
|
book-wide, inline `<style>` blocks included). The two EPUB2-targeted fixes
|
|
@@ -56,7 +56,9 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
|
|
|
56
56
|
the safe opt-ins (`--fix-ids`, `--add-img-alt`, `--strip-bad-attrs`,
|
|
57
57
|
`--escape-unknown-entities`) plus `--encode-url-spaces`, which repair without
|
|
58
58
|
altering visible markup; the "twelve structural repairs and three lossy strips"
|
|
59
|
-
phrasing elsewhere in these docs was never the complete inventory
|
|
59
|
+
phrasing elsewhere in these docs was never the complete inventory, and the
|
|
60
|
+
Phase 16 package repairs (`--fix-container`, `--fix-media-types`) took the
|
|
61
|
+
structural count to fourteen (v0.38.0).
|
|
60
62
|
- **The gate is the safety contract.** Never apply a repair epubcheck has not accepted.
|
|
61
63
|
Respect the two-mode logic in `validate.gate` (fatal-fixing tolerates error unmasking;
|
|
62
64
|
error-cleanup does not). The lossy strips (`--strip-pagination`, `--strip-broken-tags`,
|
|
@@ -77,8 +79,8 @@ Born from the 2026 library audit (see the user memory `calibre-library-epubcheck
|
|
|
77
79
|
- `src/bindery/pagination.py`: the opt-in lossy page-number strip (runhead detection, page-layer decision, block-centric removal/merge, safety nets).
|
|
78
80
|
- `src/bindery/watermark.py`: the opt-in lossy watermark strip (anchored and anchorless signature removal).
|
|
79
81
|
- `src/bindery/reserialize.py`: structural repair via `html5lib`.
|
|
80
|
-
- `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, `monolithic` since v0.21.0 with `--max-doc-chars N`, and `completeness` since v0.36.0) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story). Since v0.36.0 the completeness analyzer (Phase 15) reports the phase-1 spot-check per book: spine/prose-doc counts (prose = >= 400 visible chars), first/middle/last prose-doc opening/closing excerpts, trailing-ToC classification (`_trailing_toc`: short link lines dominate, no paragraph-length blocks), and the unreadable fraction. It is ADVISORY by contract (problem always False; ADVISORY on a trailing ToC or unreadable fraction >= 10%), never moves the exit code, and is skipped like emptytext when the archive verdict owns the book.
|
|
81
|
-
- `src/bindery/epub.py`: archive rewrite, NCX uid sync, RepairReport, mismatch detection, and the opt-in structural-repair plumbing (including the CSS precondition scan).
|
|
82
|
+
- `src/bindery/audit.py`: read-only body-text audits (`content`, `pagenumbers`, `emptytext`, `ocr`, `monolithic` since v0.21.0 with `--max-doc-chars N`, and `completeness` since v0.36.0) behind the `audit` subcommand (v0.15.0). Since v0.18.0 library mode resolves EPUB paths through `cquarry.get_format_path()` and can apply a tag to flagged books via `cquarry.write.WritableCalibreDB` (only with explicit `--tag`; the only sanctioned write path). Since v0.19.0 `audit --id BOOK_ID` audits a single library book through cquarry's single-entity `get_book()` fetch (no library-wide cache; supports `--tag`; incompatible with the directory argument; comma lists since v0.23.0). Since v0.22.0 every archive entry is fully read (CRC + decompression) before analysis — a damaged entry reports its own CORRUPT verdict instead of feeding emptytext — and manifest/NCX references to absent files classify as `convention` (bloated ToC, consecutive span) or `fragment` (broken span). Hazard: the scan loop reuses `tag` as its per-book display column, so the `--tag` argument is captured as `audit_tag` at function entry — do not collapse them again. Since v0.29.0 `audit --json FILE` writes per-file analyzer verdicts in the `library --json` shape (directory, library, and single-book modes; with `--id`, exactly one id; emptytext omitted from a record when the archive verdict owns the body-text story). Since v0.36.0 the completeness analyzer (Phase 15) reports the phase-1 spot-check per book: spine/prose-doc counts (prose = >= 400 visible chars), first/middle/last prose-doc opening/closing excerpts, trailing-ToC classification (`_trailing_toc`: short link lines dominate, no paragraph-length blocks), and the unreadable fraction. It is ADVISORY by contract (problem always False; ADVISORY on a trailing ToC or unreadable fraction >= 10%), never moves the exit code, and is skipped like emptytext when the archive verdict owns the book. Since v0.38.0 the archive verdict distinguishes font obfuscation from DRM (`OBFUSCATION_ALGOS`: IDPF 2008/embedding + both Adobe URIs; readable obfuscated entries are the benign OBFUSCATED advisory, an unreadable one is CORRUPT, and only non-obfuscation algorithms give ENCRYPTED).
|
|
83
|
+
- `src/bindery/epub.py`: archive rewrite, NCX uid sync, RepairReport, mismatch detection, and the opt-in structural-repair plumbing (including the CSS precondition scan). Since v0.38.0 (Phase 16): `generate_container` + the `--fix-container` gateway (insert after mimetype when missing, replace in-loop when stale; constant-epoch timestamp keeps bytes deterministic), `prune_dangling_edges` (prune_missing_manifest_items returns the pruned ids and the edges that pointed at them — spine@toc, media-overlay, fallback, EPUB2 cover meta — are rewritten so pruning cannot manufacture the regression the gate rejects), and `fix_manifest_media_types` (extension map + magic-byte confirmation via a `peek` callable over the open source zip; jpg/png/gif only).
|
|
82
84
|
- `src/bindery/validate.py`: epubcheck wrapper, the `gate` (improvement) and `no_worse` (no-regression, for the lossy strips) acceptance bars.
|
|
83
85
|
- `src/bindery/library.py`: Calibre walk, atomic replace, backups, and native format installation. Since v0.19.0 `CalibreIdResolver` resolves the book id from `metadata.db` through cquarry (one lazy path→id map per run; since v0.20.0 the map comes from `CalibreDB.format_path_index()`, re-normalized with `resolve().lower()` for the resolver's symlinked-directory and case-insensitive matching); the `(id)` directory-name regex is only the no-catalog fallback. Since v0.24.0 `install_format()` places the repaired file atomically and updates the `data` row through `cquarry.write.WritableCalibreDB` (since v0.31.0 via `set_format`, cquarry 1.17's sanctioned remove+add in one transaction); the external `calibredb` CLI is gone. Since v0.32.0 a guessed id drives a row update only when metadata.db corroborates it (`_verify_guess`: books row exists, file in the book's own directory, catalogued `data.name` when a row exists); anything else saves in place and leaves the catalog untouched — never reintroduce directory-name guessing as the primary source: renamed/mismatched directories would replace the wrong book.
|
|
84
86
|
- `src/bindery/cli.py`: `repair` and `library` subcommands, including `--all` and `--install-to-calibre`; since v0.29.0 also the `run` verbs: `phase1` (pre-import vetting over loose files: the audit battery composed with one gated repair sweep; read-only until `--apply-lossy`, which IS the lossy-strip consent) and `phase3` (the scoped post-import apply step: `library --id --sweep --only all --apply --all --install-to-calibre` plus a pre/post summary; mechanically refuses unscoped sweeps with exit 2). Both take `--non-interactive` and surface open questions as `decisions_needed` in their JSON, never prompts. They drive the shipped `run_library` through the real parser, so no flag can drift between the verb and the subcommand it wraps. `library --workers N` (v0.30.0) parallelizes only the sweep's candidate pass, in windows of N consumed in order so `--limit` stays lazy; the repair phase is never parallel (shared workdir, atomic-replacement contract).
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: bindery-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.38.0
|
|
4
4
|
Summary: Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library.
|
|
5
5
|
Author: vrnvctss
|
|
6
6
|
License: MIT
|
|
@@ -60,9 +60,11 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
|
|
|
60
60
|
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
|
|
61
61
|
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
|
|
62
62
|
- **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
|
|
63
|
-
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
63
|
+
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items (with every package edge that pointed at them rewritten: `spine@toc`, `item@media-overlay`, `item@fallback`, and the EPUB2 cover meta). Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
64
64
|
- **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
|
|
65
65
|
- **`--encode-url-spaces`**: percent-encodes raw spaces in `src`/`href` attribute values across the package (OPF manifest, NCX, content documents). A URL with a literal space is not a valid URL (RSC-020) and unresolvable on strict readers; the encoded form denotes the same file.
|
|
66
|
+
- **`--fix-container`**: generates `META-INF/container.xml` at the located OPF when the container is missing or names a file the archive does not contain. This is the gateway defect: epubcheck stays fatal while the OPF is unfindable, so no repair can be gate-accepted on such a book until the container exists.
|
|
67
|
+
- **`--fix-media-types`**: normalizes wrong manifest media-type declarations (OPF-029, e.g. a jpg stamped `image/png` by an aggregator). Attribute-only and quote-preserving; it fires only when the file's magic bytes confirm the extension, so an honestly-declared misnamed file is never made worse.
|
|
66
68
|
|
|
67
69
|
|
|
68
70
|
Three opt-in fixes are **lossy** and stand apart from the semantics-preserving rest:
|
|
@@ -38,9 +38,11 @@ Everything below is off until its flag is passed (or all at once via `--all`). T
|
|
|
38
38
|
- **`--strip-epub3-attrs`**: scrubs the EPUB3-only attributes epubcheck rejects on an EPUB2 package: `page-progression-direction`, `epub:type`, `aria-label` (a fixed, documented set; rendering is unchanged, and lookalikes like `type` or the wider aria family survive). The edit is anchored to real start tags, so prose that merely mentions `epub:type="chapter"` is preserved.
|
|
39
39
|
- **`--downgrade-epub3-tags`**: downgrades EPUB3/HTML5 semantic elements to their EPUB2 equivalents (`figure`/`section` to `div`, `figcaption` to `p`), keeping existing classes and appending the semantic name (`class="figure"`) as the styling hook. Tag names a stylesheet styles as an element selector are protected book-wide, so styled formatting is never destroyed. Both EPUB2-targeted fixes are gated on the package version declared in the OPF: on an EPUB 3 book they do nothing, because their target defects only exist below EPUB 3.
|
|
40
40
|
- **`--unwrap-illegal-tags`**: strips completely invalid or deprecated HTML tags that break EPUB3 validation (`<st>`, `<sentence>`, `<o>`, `<w>`, `<pagebreak>`) while retaining their inner text. Any of those names styled as an *element selector* by an EPUB stylesheet (`.css` entries and inline `<style>` blocks alike; class/id selectors like `.st`/`#w` don't count) is protected for the whole book, guaranteeing format preservation.
|
|
41
|
-
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items. Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
41
|
+
- **`--prune-missing-resources`**: removes references to files the archive does not contain (RSC-007/PKG-010): dead `<link>` elements, anchors' `href` to absent files (anchor text preserved), absent `<img>` sources (replaced by their alt text when they carry one), and orphaned non-spine OPF manifest items (with every package edge that pointed at them rewritten: `spine@toc`, `item@media-overlay`, `item@fallback`, and the EPUB2 cover meta). Spine documents are never pruned: a missing spine document is reported as a damaged fragment, not silently dropped.
|
|
42
42
|
- **`--strip-broken-anchors`**: strips href attributes that cannot resolve, keeping the anchor text byte-for-byte: a `#fragment` the target document does not define (RSC-020/RSC-012; NCX navTargets fall back to the document target, so chapter navigation survives) and unresolvable URI schemes (`kindle:`, `file:`).
|
|
43
43
|
- **`--encode-url-spaces`**: percent-encodes raw spaces in `src`/`href` attribute values across the package (OPF manifest, NCX, content documents). A URL with a literal space is not a valid URL (RSC-020) and unresolvable on strict readers; the encoded form denotes the same file.
|
|
44
|
+
- **`--fix-container`**: generates `META-INF/container.xml` at the located OPF when the container is missing or names a file the archive does not contain. This is the gateway defect: epubcheck stays fatal while the OPF is unfindable, so no repair can be gate-accepted on such a book until the container exists.
|
|
45
|
+
- **`--fix-media-types`**: normalizes wrong manifest media-type declarations (OPF-029, e.g. a jpg stamped `image/png` by an aggregator). Attribute-only and quote-preserving; it fires only when the file's magic bytes confirm the extension, so an honestly-declared misnamed file is never made worse.
|
|
44
46
|
|
|
45
47
|
|
|
46
48
|
Three opt-in fixes are **lossy** and stand apart from the semantics-preserving rest:
|
|
@@ -1,4 +1,61 @@
|
|
|
1
1
|
# bindery-cli Patch Notes
|
|
2
|
+
## v0.38.0 (2026-09-12)
|
|
3
|
+
|
|
4
|
+
### Phase 16: package-structure repairs
|
|
5
|
+
|
|
6
|
+
- **`--fix-container`, the gateway repair.** A book whose
|
|
7
|
+
META-INF/container.xml is missing, unparseable, or names a file the
|
|
8
|
+
archive does not contain is locked out of every repair: epubcheck
|
|
9
|
+
stays fatal while the OPF is unfindable. The new flag generates the
|
|
10
|
+
standard container at the OPF the locator already finds by fallback
|
|
11
|
+
(a byte-deterministic template with the constant epoch timestamp).
|
|
12
|
+
Oracle-verified on a real-shaped fixture: fatal before, gate-accepted
|
|
13
|
+
after. The Phase 16 prevalence study found 0 such books in-library
|
|
14
|
+
(5,228/5,228 have valid containers); the repair ships for the
|
|
15
|
+
acquisition phase, where the class actually lives.
|
|
16
|
+
- **`--prune-missing-resources` rewrites what it drops.** Removing a
|
|
17
|
+
manifest item whose file is absent used to leave every package edge
|
|
18
|
+
pointing at it dangling, manufacturing the regression the gate then
|
|
19
|
+
rejects. The prune now carries the pruned ids into a rewrite of the
|
|
20
|
+
four edge classes: `spine@toc`, `item@media-overlay`, `item@fallback`,
|
|
21
|
+
and the EPUB2 cover meta. Optional edges of dead targets are removed;
|
|
22
|
+
edges to live ids are untouched.
|
|
23
|
+
- **`--fix-media-types`.** Wrong manifest media-type declarations
|
|
24
|
+
(OPF-029; 275 occurrences in 15 library books, the class being a jpg
|
|
25
|
+
stamped `image/png` by an aggregator) are normalized attribute-only,
|
|
26
|
+
preserving quote style, and only when the file's magic bytes confirm
|
|
27
|
+
the extension: a PNG renamed .jpg keeps its wrong-but-honest
|
|
28
|
+
declaration instead of a worse one. On the real Arthur sample the run
|
|
29
|
+
goes from nochange to gate-accepted with 9 declarations fixed.
|
|
30
|
+
- **The audit distinguishes font obfuscation from DRM.** encryption.xml
|
|
31
|
+
entries under the obfuscation algorithms (the IDPF URI and both Adobe
|
|
32
|
+
forms, including the ns.adobe.com URI real-world files actually
|
|
33
|
+
carry, which the prevalence study caught) that read fine are the new
|
|
34
|
+
OBFUSCATED advisory: publisher font embedding, benign, problem-False,
|
|
35
|
+
exit-neutral. An unreadable obfuscation entry is CORRUPT (a broken
|
|
36
|
+
font, re-source), and only non-obfuscation algorithms give the
|
|
37
|
+
ENCRYPTED skip. 86 of the library's 5,228 books carry
|
|
38
|
+
obfuscation-only encryption.xml; the library holds no real DRM at all.
|
|
39
|
+
- **Boxed, not built, with real counts.** The obfuscation-aware
|
|
40
|
+
encryption repair (the magic-byte precondition validated on 424 real
|
|
41
|
+
fonts: all genuinely scrambled, zero stale entries in-library), href
|
|
42
|
+
case/backslash resolution (1 href in 1 book; 0 backslashes), and
|
|
43
|
+
duplicate zip-entry dedupe (0 books) are recorded in the roadmap with
|
|
44
|
+
their prevalence numbers and reopen conditions. The decision-gated
|
|
45
|
+
Phase 16 C boxes (cover wiring, the metadata carve-out, the NCX<->nav
|
|
46
|
+
drift detector, CSS url() pruning) carry recorded options and wait
|
|
47
|
+
for rulings.
|
|
48
|
+
|
|
49
|
+
### The prevalence study (Phase 16 A)
|
|
50
|
+
|
|
51
|
+
- 5,228 library books through the FastSweep epubcheck harness plus a
|
|
52
|
+
direct zip-level pass: container.xml valid on every book; RSC-007 at
|
|
53
|
+
14,156 occurrences in 600 books and PKG-010 in 280 (the prune
|
|
54
|
+
surface, 1,538 truly-absent manifest hrefs); OPF-029 at 275 in 15;
|
|
55
|
+
case-mismatch hrefs exactly 1; encryption.xml in 86 books, all of it
|
|
56
|
+
font obfuscation. Every Phase 16 repair box now carries its real
|
|
57
|
+
count.
|
|
58
|
+
|
|
2
59
|
## v0.37.0 (2026-09-12)
|
|
3
60
|
|
|
4
61
|
### The Bindery Repair Calibre plugin (Phase 3)
|
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "bindery-cli"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.38.0"
|
|
8
8
|
description = "Audit EPUB body text and repair broken EPUBs with deterministic, epubcheck-gated fixes; atomic in-place replacement in a Calibre library."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
classifiers = [
|
|
@@ -1483,48 +1483,101 @@ an opt-in structural repair under the existing gate contract.
|
|
|
1483
1483
|
|
|
1484
1484
|
### B. The repairs (each: opt-in flag, dry-run default, gate-accepted)
|
|
1485
1485
|
|
|
1486
|
-
- [
|
|
1486
|
+
- [x] **`--fix-container`**: generate META-INF/container.xml at the
|
|
1487
1487
|
located OPF (model: upstream calibre's initialize_container).
|
|
1488
1488
|
Gateway repair, effort S, HIGH value.
|
|
1489
|
+
*(shipped v0.38.0, dac7835: fires when the container is missing,
|
|
1490
|
+
unparseable, or names a file the archive does not contain; the
|
|
1491
|
+
generated entry carries the constant epoch so repairs stay
|
|
1492
|
+
byte-deterministic. Prevalence: 0 books in-library (5,228/5,228 have
|
|
1493
|
+
valid containers) — shipped anyway as the acquisition-phase gateway;
|
|
1494
|
+
oracle-verified on a real-shaped fixture (fatal before, accept
|
|
1495
|
+
after).)*
|
|
1489
1496
|
- [ ] **Obfuscation-aware encryption.xml repair**: prune stale
|
|
1490
1497
|
obfuscation entries when font magic bytes prove the fonts are
|
|
1491
1498
|
decrypted in place; drop encryption.xml when only obfuscation entries
|
|
1492
1499
|
remain and fonts verify. NEVER un-obfuscate. M.
|
|
1493
|
-
|
|
1500
|
+
*(BOXED with real counts, 2026-09-12: the magic-byte validation
|
|
1501
|
+
proved 424/424 readable obfuscation fonts genuinely scrambled — zero
|
|
1502
|
+
stale entries in-library, so the repair has nothing to do here; its
|
|
1503
|
+
precondition is validated and recognition carries both Adobe URIs.
|
|
1504
|
+
The audit half of this box SHIPPED (see below). Opens only if
|
|
1505
|
+
acquisition ever brings a decrypted-in-place book.)*
|
|
1506
|
+
- [x] **Audit distinction: OBFUSCATED vs DRM** in audit.py (today the
|
|
1494
1507
|
ENCRYPTED verdict keys on unreadable entries and implies DRM where
|
|
1495
1508
|
publisher obfuscation is benign). S, rides the box above.
|
|
1496
|
-
|
|
1509
|
+
*(shipped v0.38.0, dac7835: readable obfuscation entries are the
|
|
1510
|
+
OBFUSCATED advisory (problem-False, exit-neutral; 86 library books);
|
|
1511
|
+
an unreadable obfuscation entry is CORRUPT (broken font, re-source),
|
|
1512
|
+
and only non-obfuscation algorithms give the ENCRYPTED skip.)*
|
|
1513
|
+
- [x] **Reference-edge completion for `--prune-missing-resources`**:
|
|
1497
1514
|
rewrite `spine@toc`, `item@media-overlay`, `item@fallback`, and EPUB2
|
|
1498
1515
|
cover-meta edges when their targets are pruned (machinery exists in
|
|
1499
1516
|
fix_manifest_ids). Prevents manufacturing gate regressions. S/M.
|
|
1500
|
-
|
|
1517
|
+
*(shipped v0.38.0, dac7835: prune_missing_manifest_items returns the
|
|
1518
|
+
pruned ids; prune_dangling_edges removes the optional attributes and
|
|
1519
|
+
a cover meta whose payload died. RSC-007 backs it at 14,156
|
|
1520
|
+
occurrences in 600 books.)*
|
|
1521
|
+
- [x] **Manifest media-type normalization**: sniff magic bytes /
|
|
1501
1522
|
extension map; attribute-only. S.
|
|
1523
|
+
*(shipped v0.38.0, dac7835 as `--fix-media-types`: OPF-029 class,
|
|
1524
|
+
275 occurrences in 15 books; jpg/png/gif only, fired only when the
|
|
1525
|
+
magic bytes confirm the extension. Oracle-verified on the real
|
|
1526
|
+
Arthur sample: nochange without the flag, accept with 9 normalized.)*
|
|
1502
1527
|
- [ ] **href case/backslash resolution**: case-insensitive namemap when
|
|
1503
1528
|
exactly one zip entry matches; backslash to slash. Kills the
|
|
1504
1529
|
present-file RSC-007/PKG-010 slice prune cannot touch. M.
|
|
1530
|
+
*(BOXED with real counts, 2026-09-12: exactly 1 case-mismatch href in
|
|
1531
|
+
1 book and 0 backslash hrefs across the library — below the shipping
|
|
1532
|
+
bar; reopen if acquisition changes the mix.)*
|
|
1505
1533
|
- [ ] **Duplicate zip-entry dedupe on rewrite**: drop shadowed
|
|
1506
1534
|
duplicates (audit already counts `dup_entries`). S.
|
|
1535
|
+
*(BOXED with real counts, 2026-09-12: 0 books with duplicate entries
|
|
1536
|
+
in-library.)*
|
|
1507
1537
|
|
|
1508
1538
|
### C. Decision-gated (recorded boxes; no work without a ruling)
|
|
1509
1539
|
|
|
1510
1540
|
- [ ] **Cover wiring repair** (dangling `<meta name="cover">`, missing
|
|
1511
1541
|
`properties="cover-image"`, wrong cover media-type): narrow,
|
|
1512
1542
|
finding-driven repair vs audit-only. Needs the scope decision. M.
|
|
1543
|
+
*(Recorded options, 2026-09-12: (a) narrow, finding-driven repair —
|
|
1544
|
+
re-point the cover meta / add `properties="cover-image"` only when
|
|
1545
|
+
the cover item is unambiguously identifiable from the existing guide
|
|
1546
|
+
or meta (no content fabrication, deterministic); (b) audit-only — an
|
|
1547
|
+
audit detector reporting unwired covers as `decisions_needed`. The
|
|
1548
|
+
research leans (a) for EPUB2 meta re-pointing and (b) for everything
|
|
1549
|
+
else, since guessing "which image is the cover" from names is exactly
|
|
1550
|
+
the non-determinism the charter forbids.)*
|
|
1513
1551
|
- [ ] **Metadata validity carve-out ruling** (non-ISO `dc:date`,
|
|
1514
1552
|
invalid `dc:language`, missing `dcterms:modified`): bindery opt-in
|
|
1515
1553
|
with a determinism-safe timestamp policy, or route to
|
|
1516
1554
|
cquarry/CalibreQuarry. Needs a written charter ruling. M.
|
|
1555
|
+
*(Recorded options, 2026-09-12: (a) bindery opt-in — non-ISO dates
|
|
1556
|
+
and language tags are deterministic normalizations (OPF-007/OPF-012
|
|
1557
|
+
class), but `dcterms:modified` needs a timestamp policy that breaks
|
|
1558
|
+
byte-determinism unless pinned to the file's own zip timestamps;
|
|
1559
|
+
(b) route to cquarry/CalibreQuarry — metadata quality is catalog
|
|
1560
|
+
territory and the library's metadata writes already live there.
|
|
1561
|
+
(b) is the cleaner charter fit; (a) only if epubcheck fatals on such
|
|
1562
|
+
books show up in acquisition.)*
|
|
1517
1563
|
- [ ] **NCX<->nav drift DETECTOR** (audit-only; structural diff feeding
|
|
1518
1564
|
`decisions_needed`; ToC synthesis stays out of charter
|
|
1519
1565
|
permanently). L if repair is ever attempted.
|
|
1520
1566
|
- [ ] **CSS url() pruning** as `--prune-missing-resources` scope
|
|
1521
|
-
extension: only after the prevalence study says it matters.
|
|
1567
|
+
extension: only after the prevalence study says it matters.
|
|
1568
|
+
*(Prevalence read, 2026-09-12: CSS-007/CSS-008 appear at 56/30 books
|
|
1569
|
+
but are selector damage, not url() pruning candidates; no url()-class
|
|
1570
|
+
count surfaced in the extract codes. Stays closed until a named
|
|
1571
|
+
epubcheck finding demands it.)*
|
|
1522
1572
|
|
|
1523
1573
|
Ship shape: A leads; then B in value order (container, encryption,
|
|
1524
1574
|
prune-edges, media-types); C boxes record their decisions and wait.
|
|
1525
1575
|
Each repair ships with a fixture built from the prevalence study's
|
|
1526
1576
|
real samples, a regression test written before the fix, and the gate
|
|
1527
1577
|
(epubcheck) accepting the result.
|
|
1578
|
+
*(2026-09-12 disposition: B shipped the four boxes with real backing
|
|
1579
|
+
and boxed the two zero-prevalence classes with their counts; C carries
|
|
1580
|
+
recorded options and waits for Brandon.)*
|
|
1528
1581
|
|
|
1529
1582
|
## Tag policy (decided 2026-09-11, recorded 2026-09-12)
|
|
1530
1583
|
|
|
@@ -14,10 +14,11 @@ schema (RSC-005) violations, which are usually harmless to readers and not safel
|
|
|
14
14
|
mechanizable.
|
|
15
15
|
|
|
16
16
|
The deliberate exceptions to "semantics-preserving" come in two kinds, both strictly
|
|
17
|
-
opt-in. The
|
|
17
|
+
opt-in. The fourteen **structural repairs** (`--fix-empty-body`, `--fix-missing-title`,
|
|
18
18
|
`--fix-id-colons`, `--fix-page-map`, `--strip-epub3-attrs`, `--downgrade-epub3-tags`,
|
|
19
19
|
`--unwrap-block-in-inline`, `--strip-invalid-value`, `--unwrap-illegal-tags`,
|
|
20
|
-
`--prune-missing-resources`, `--strip-broken-anchors`, `--encode-url-spaces
|
|
20
|
+
`--prune-missing-resources`, `--strip-broken-anchors`, `--encode-url-spaces`,
|
|
21
|
+
`--fix-container`, `--fix-media-types`) alter
|
|
21
22
|
markup structure or fabricate minimal content; the three
|
|
22
23
|
**lossy modes** (`--strip-pagination`, `--strip-broken-tags`, `--strip-watermarks`) remove
|
|
23
24
|
content a converter injected rather than content the author wrote. The default pass runs
|
|
@@ -91,7 +92,7 @@ normal gate applies.
|
|
|
91
92
|
|
|
92
93
|
### Opt-in: structural repairs
|
|
93
94
|
|
|
94
|
-
|
|
95
|
+
Fourteen repairs go past well-formedness and therefore require their own flag; none is ever
|
|
95
96
|
part of the default pipeline:
|
|
96
97
|
|
|
97
98
|
- **`--fix-empty-body`**: ` ` inside a strictly empty `<body></body>` ("body
|
|
@@ -168,7 +169,20 @@ every EPUB3 book in an `--all` sweep taking exactly that damage).
|
|
|
168
169
|
strict readers; the encoded form denotes the same file and renders identically. Scope
|
|
169
170
|
is fixed to the space character: extend only with a named epubcheck finding.
|
|
170
171
|
|
|
171
|
-
|
|
172
|
+
- **`--fix-container`**: generate `META-INF/container.xml` at the located OPF when the
|
|
173
|
+
container is missing, unparseable, or names a file the archive does not contain. This is the
|
|
174
|
+
gateway defect: epubcheck stays fatal while the OPF is unfindable, so no other repair can be
|
|
175
|
+
gate-accepted on such a book. The generated file is byte-deterministic (a fixed template plus
|
|
176
|
+
the OPF path; the entry's timestamp is the constant epoch), and a healthy container is never
|
|
177
|
+
touched.
|
|
178
|
+
- **`--fix-media-types`**: normalize wrong manifest `media-type` declarations (OPF-029: a
|
|
179
|
+
file's bytes do not match the declared type). The expected type comes from the extension, and
|
|
180
|
+
the rewrite fires only when the file's magic bytes at offset 0 confirm the extension (jpg,
|
|
181
|
+
jpeg, png, gif): a PNG renamed `.jpg` keeps its wrong-but-honest declaration rather than
|
|
182
|
+
gaining a worse one. Attribute-only, quote-style preserved; files absent from the archive
|
|
183
|
+
belong to `--prune-missing-resources`.
|
|
184
|
+
|
|
185
|
+
All fourteen are evaluated by the normal `gate`: unlike the lossy strips, their benefit is
|
|
172
186
|
visible to epubcheck (they clear errors), so a run with no measurable improvement is a
|
|
173
187
|
noop and nothing is applied. CDATA sections and comments are never rewritten, as
|
|
174
188
|
everywhere else.
|
|
@@ -353,6 +367,13 @@ is fully read for CRC + decompression, reporting CORRUPT rather than EMPTY), and
|
|
|
353
367
|
issues. Manifest/NCX references to absent files are classified as either `convention` (ToC is bloated but
|
|
354
368
|
present documents form a consecutive chapter span) or `fragment` (the span itself is broken).
|
|
355
369
|
|
|
370
|
+
The archive verdict distinguishes font obfuscation from DRM (Phase 16): entries
|
|
371
|
+
`encryption.xml` declares under a font-obfuscation algorithm (the IDPF `2008/embedding`
|
|
372
|
+
URI and both Adobe forms, including the `ns.adobe.com` URI real-world files carry) that
|
|
373
|
+
read fine are the OBFUSCATED advisory — publisher embedding, benign, never a failure; an
|
|
374
|
+
unreadable obfuscation entry is CORRUPT (a broken font, not a business model), and only
|
|
375
|
+
non-obfuscation algorithms give the ENCRYPTED verdict with its DRM skip advice.
|
|
376
|
+
|
|
356
377
|
The completeness analyzer is the phase-1 spot-check and is advisory by contract: it never
|
|
357
378
|
flags a book and never moves the exit code. Per book it reports the spine doc count, the
|
|
358
379
|
prose-doc count (spine docs carrying at least 400 visible characters), the opening and closing
|