epub-optimizer 0.0.0-stage → 3.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +1434 -2
- package/dist/src/batch.d.ts +34 -0
- package/dist/src/batch.js +137 -0
- package/dist/src/cli.d.ts +7 -0
- package/dist/src/cli.js +275 -0
- package/dist/src/index.d.ts +23 -0
- package/dist/src/index.js +24 -0
- package/dist/src/optimization.d.ts +7 -0
- package/dist/src/optimization.js +207 -0
- package/dist/src/pipeline.d.ts +2 -0
- package/dist/src/pipeline.js +48 -0
- package/dist/src/processors/archive-processor.d.ts +26 -0
- package/dist/src/processors/archive-processor.js +170 -0
- package/dist/src/processors/epub-processor.d.ts +17 -0
- package/dist/src/processors/epub-processor.js +133 -0
- package/dist/src/processors/font-characters.d.ts +2 -0
- package/dist/src/processors/font-characters.js +227 -0
- package/dist/src/processors/font-engine.d.ts +6 -0
- package/dist/src/processors/font-engine.js +101 -0
- package/dist/src/processors/font-limits.d.ts +3 -0
- package/dist/src/processors/font-limits.js +3 -0
- package/dist/src/processors/font-processor.d.ts +4 -0
- package/dist/src/processors/font-processor.js +121 -0
- package/dist/src/processors/font-runner.d.ts +8 -0
- package/dist/src/processors/font-runner.js +94 -0
- package/dist/src/processors/font-worker.d.ts +1 -0
- package/dist/src/processors/font-worker.js +12 -0
- package/dist/src/processors/html-processor.d.ts +22 -0
- package/dist/src/processors/html-processor.js +190 -0
- package/dist/src/processors/image-converter.d.ts +14 -0
- package/dist/src/processors/image-converter.js +481 -0
- package/dist/src/processors/image-processor.d.ts +24 -0
- package/dist/src/processors/image-processor.js +165 -0
- package/dist/src/processors/js-processor.d.ts +7 -0
- package/dist/src/processors/js-processor.js +59 -0
- package/dist/src/processors/lazy-img.d.ts +5 -0
- package/dist/src/processors/lazy-img.js +48 -0
- package/dist/src/processors/svg-optimizer.d.ts +5 -0
- package/dist/src/processors/svg-optimizer.js +49 -0
- package/dist/src/scripts/create-epub.d.ts +2 -0
- package/dist/src/scripts/create-epub.js +29 -0
- package/dist/src/scripts/fix/fix-span-tags.d.ts +2 -0
- package/dist/src/scripts/fix/fix-span-tags.js +36 -0
- package/dist/src/scripts/fix/fix-xml.d.ts +3 -0
- package/dist/src/scripts/fix/fix-xml.js +59 -0
- package/dist/src/scripts/fix/index.d.ts +5 -0
- package/dist/src/scripts/fix/index.js +20 -0
- package/dist/src/scripts/fix/remove-empty-styles.d.ts +10 -0
- package/dist/src/scripts/fix/remove-empty-styles.js +53 -0
- package/dist/src/scripts/fix/xhtml-files.d.ts +2 -0
- package/dist/src/scripts/fix/xhtml-files.js +34 -0
- package/dist/src/scripts/ops/add-chapter-sections-to-toc.d.ts +5 -0
- package/dist/src/scripts/ops/add-chapter-sections-to-toc.js +242 -0
- package/dist/src/scripts/ops/add-cover-image-property.d.ts +2 -0
- package/dist/src/scripts/ops/add-cover-image-property.js +27 -0
- package/dist/src/scripts/ops/ncx.d.ts +6 -0
- package/dist/src/scripts/ops/ncx.js +19 -0
- package/dist/src/scripts/ops/update-cover-linear.d.ts +11 -0
- package/dist/src/scripts/ops/update-cover-linear.js +45 -0
- package/dist/src/scripts/ops/update-structure.d.ts +5 -0
- package/dist/src/scripts/ops/update-structure.js +22 -0
- package/dist/src/scripts/ops/update-summary-page.d.ts +2 -0
- package/dist/src/scripts/ops/update-summary-page.js +53 -0
- package/dist/src/scripts/ops/update-toc-with-cover.d.ts +2 -0
- package/dist/src/scripts/ops/update-toc-with-cover.js +140 -0
- package/dist/src/scripts/utils.d.ts +53 -0
- package/dist/src/scripts/utils.js +92 -0
- package/dist/src/scripts/validate-epub.d.ts +12 -0
- package/dist/src/scripts/validate-epub.js +185 -0
- package/dist/src/storage.d.ts +30 -0
- package/dist/src/storage.js +286 -0
- package/dist/src/types.d.ts +51 -0
- package/dist/src/types.js +1 -0
- package/dist/src/utils/author-workflow-config.d.ts +17 -0
- package/dist/src/utils/author-workflow-config.js +95 -0
- package/dist/src/utils/buffered-process.d.ts +8 -0
- package/dist/src/utils/buffered-process.js +36 -0
- package/dist/src/utils/cli-config.d.ts +4 -0
- package/dist/src/utils/cli-config.js +84 -0
- package/dist/src/utils/config.d.ts +49 -0
- package/dist/src/utils/config.js +58 -0
- package/dist/src/utils/diagnostics.d.ts +11 -0
- package/dist/src/utils/diagnostics.js +38 -0
- package/dist/src/utils/epub-doctor.d.ts +78 -0
- package/dist/src/utils/epub-doctor.js +301 -0
- package/dist/src/utils/epub-features.d.ts +20 -0
- package/dist/src/utils/epub-features.js +105 -0
- package/dist/src/utils/epub-integrity.d.ts +25 -0
- package/dist/src/utils/epub-integrity.js +197 -0
- package/dist/src/utils/epub-text.d.ts +13 -0
- package/dist/src/utils/epub-text.js +106 -0
- package/dist/src/utils/epub-utils.d.ts +79 -0
- package/dist/src/utils/epub-utils.js +207 -0
- package/dist/src/utils/epubcheck-report.d.ts +35 -0
- package/dist/src/utils/epubcheck-report.js +68 -0
- package/dist/src/utils/file-size.d.ts +13 -0
- package/dist/src/utils/file-size.js +40 -0
- package/dist/src/utils/files.d.ts +14 -0
- package/dist/src/utils/files.js +39 -0
- package/dist/src/utils/html-report.d.ts +17 -0
- package/dist/src/utils/html-report.js +95 -0
- package/dist/src/utils/i18n.d.ts +5 -0
- package/dist/src/utils/i18n.js +9 -0
- package/dist/src/utils/output-transaction.d.ts +8 -0
- package/dist/src/utils/output-transaction.js +146 -0
- package/dist/src/utils/package-root.d.ts +2 -0
- package/dist/src/utils/package-root.js +15 -0
- package/dist/src/utils/path-safety.d.ts +2 -0
- package/dist/src/utils/path-safety.js +30 -0
- package/dist/src/utils/pipeline-options.d.ts +34 -0
- package/dist/src/utils/pipeline-options.js +49 -0
- package/dist/src/utils/png-animation.d.ts +7 -0
- package/dist/src/utils/png-animation.js +44 -0
- package/dist/src/utils/run-report.d.ts +49 -0
- package/dist/src/utils/run-report.js +71 -0
- package/dist/src/utils/temp-dir.d.ts +11 -0
- package/dist/src/utils/temp-dir.js +80 -0
- package/dist/src/utils/watch-lock.d.ts +8 -0
- package/dist/src/utils/watch-lock.js +100 -0
- package/dist/src/utils/watch-summary.d.ts +13 -0
- package/dist/src/utils/watch-summary.js +130 -0
- package/dist/src/utils/xhtml-structure.d.ts +7 -0
- package/dist/src/utils/xhtml-structure.js +53 -0
- package/dist/src/utils/xhtml.d.ts +27 -0
- package/dist/src/utils/xhtml.js +205 -0
- package/dist/src/utils/xml-dtd.d.ts +6 -0
- package/dist/src/utils/xml-dtd.js +133 -0
- package/dist/src/watch.d.ts +21 -0
- package/dist/src/watch.js +328 -0
- package/package.json +99 -3
- package/scripts/install-epubcheck.sh +44 -0
package/README.md
CHANGED
|
@@ -1,3 +1,1435 @@
|
|
|
1
|
-
#
|
|
1
|
+
# EPUB Optimizer
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://github.com/kiki-le-singe/epub-optimizer/actions/workflows/ci.yml)
|
|
4
|
+
[](https://nodejs.org)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
|
|
7
|
+
A Node.js utility to optimize EPUB files by compressing HTML, CSS, images and recompressing the archive. This tool can significantly reduce EPUB file sizes while maintaining compatibility with e-readers and ensuring EPUB specification compliance.
|
|
8
|
+
|
|
9
|
+
## Why Use EPUB Optimizer?
|
|
10
|
+
|
|
11
|
+
- ✅ **Can dramatically reduce file size** - The original author workflow often produces files 70-90% smaller
|
|
12
|
+
- ✅ **Validates before publishing** - EPUBCheck must succeed before the requested output is replaced
|
|
13
|
+
- ✅ **Offers explicit quality trade-offs** - Choose balanced, lossless, or the complete author workflow
|
|
14
|
+
- ✅ **Uses the EPUB package as source of truth** - Discovers modern and legacy content through `container.xml` and the OPF manifest
|
|
15
|
+
- ✅ **Handles tricky EPUBs gracefully** - Refuses DRM-encrypted books, leaves obfuscated/embedded fonts intact, and skips downscaling for fixed-layout titles instead of corrupting them
|
|
16
|
+
- ✅ **Zero setup with Docker** - No need to install Node.js, Java, or other dependencies
|
|
17
|
+
- ✅ **Battle-tested** - Used for real book publishing workflows
|
|
18
|
+
- ✅ **Structured results** - Optional JSON reports, strict mode, and per-step profiling
|
|
19
|
+
- ✅ **Explicit controls** - Generic, lossless, repair, and complete author workflows
|
|
20
|
+
|
|
21
|
+

|
|
22
|
+
|
|
23
|
+
### Results
|
|
24
|
+
|
|
25
|
+
Measured on an image-heavy EPUB with 12 full-page PNGs, with EPUBCheck-clean output. The input is 57 MB as printed by the CLI, which uses 1024-based units (60.0 MB on disk):
|
|
26
|
+
|
|
27
|
+
| Preset | Output | Reduction | EPUBCheck |
|
|
28
|
+
| -------------------- | ---------- | ---------------------------------- | --------------------- |
|
|
29
|
+
| `balanced` (default) | 8.3 MB | **−85.5%** | 0 errors / 0 warnings |
|
|
30
|
+
| `author` | 8.3 MB | **−85.5%** | 0 errors / 0 warnings |
|
|
31
|
+
| `repair` | 8.3 MB | **−85.5%** | 0 errors / 0 warnings |
|
|
32
|
+
| `lossless` | ~unchanged | by design (no lossy image changes) | 0 errors / 0 warnings |
|
|
33
|
+
|
|
34
|
+
The `balanced` row was remeasured with v3.8.0. The `author` and `repair` rows are earlier measurements and were not rerun for v3.8.0; the `lossless` row describes the preset's behavior.
|
|
35
|
+
|
|
36
|
+
Nearly all of the savings come from raster image optimization, so results scale with how image-heavy your book is. In this book, 11 of the 12 PNGs contain transparency: they stay PNG and shrink through palette quantization (`--png-quality`) and downscaling to the default 1600 px maximum dimension. Only the opaque cover is converted to JPEG, because PNG-to-JPEG conversion applies by default only to large PNGs without an alpha channel.
|
|
37
|
+
|
|
38
|
+
Memory was observed, not bounded by design. In local v3.8.0 runs of the `balanced` preset on this book (macOS arm64, Node.js 25, Java 25, default image concurrency), the Node.js process peaked at about 548 MiB (0.53 GiB) of resident memory. EPUBCheck then runs in a separate Java process, about 0.3 GiB here, which that figure excludes. These are single-machine observations, not limits: memory depends mainly on image dimensions and [`--image-concurrency`](#image-concurrency).
|
|
39
|
+
|
|
40
|
+
## Table of Contents
|
|
41
|
+
|
|
42
|
+
- [Quick Start](#quick-start)
|
|
43
|
+
- [Migrating to v3](#migrating-to-v3)
|
|
44
|
+
- [Features](#features)
|
|
45
|
+
- [About This Project](#about-this-project)
|
|
46
|
+
- [Important Note for Apple Pages Users](#️-important-note-for-apple-pages-users)
|
|
47
|
+
- [Requirements](#requirements)
|
|
48
|
+
- [Installation](#installation)
|
|
49
|
+
- [Local Package and Typed Node API](#local-package-and-typed-node-api)
|
|
50
|
+
- [EPUBCheck Setup](#epubcheck-setup)
|
|
51
|
+
- [Docker Alternative](#docker-alternative)
|
|
52
|
+
- [Usage](#usage)
|
|
53
|
+
- [Available Scripts](#available-scripts)
|
|
54
|
+
- [Modern Workflow](#modern-workflow)
|
|
55
|
+
- [Command Line Options](#command-line-options)
|
|
56
|
+
- [Modes and Presets](#modes-and-presets)
|
|
57
|
+
- [Safe Temporary Directories and Validation Deadlines](#safe-temporary-directories-and-validation-deadlines)
|
|
58
|
+
- [Image Concurrency](#image-concurrency)
|
|
59
|
+
- [Doctor / Inspect Mode](#doctor--inspect-mode)
|
|
60
|
+
- [Structured Reports and Profiling](#structured-reports-and-profiling)
|
|
61
|
+
- [Batch Optimization](#batch-optimization)
|
|
62
|
+
- [Safety Model](#safety-model)
|
|
63
|
+
- [Docker Usage](#docker-usage)
|
|
64
|
+
- [Debugging with Temporary Files](#debugging-with-temporary-files)
|
|
65
|
+
- [Project Structure](#project-structure)
|
|
66
|
+
- [Development Information](#development-information)
|
|
67
|
+
- [Source and Build Separation](#source-and-build-separation)
|
|
68
|
+
- [Import Structure](#import-structure)
|
|
69
|
+
- [Testing](#testing)
|
|
70
|
+
- [Production Validation](#production-validation)
|
|
71
|
+
- [Release Validation](#release-validation)
|
|
72
|
+
- [Development and Production](#development-and-production)
|
|
73
|
+
- [Linting and Formatting](#linting-and-formatting)
|
|
74
|
+
- [Minification](#minification)
|
|
75
|
+
- [Modular Fix Scripts](#modular-fix-scripts)
|
|
76
|
+
- [Customizing the Optimization Process](#customizing-the-optimization-process)
|
|
77
|
+
- [Common Issues](#common-issues)
|
|
78
|
+
- [Troubleshooting](#troubleshooting)
|
|
79
|
+
- [Dependencies](#dependencies)
|
|
80
|
+
- [License](#license)
|
|
81
|
+
|
|
82
|
+
## About This Project
|
|
83
|
+
|
|
84
|
+
I use this project to optimize EPUB files that I create using Pages on Mac. My workflow is:
|
|
85
|
+
|
|
86
|
+
- Write (text and images) in Pages.
|
|
87
|
+
- Create a manual summary page with bookmark links to chapters and sections. The author workflow preset can synchronize these subsections from the manual summary page to the system navigation files - both EPUB3 toc.xhtml and EPUB2 NCX - so they appear in e-reader navigation menus.
|
|
88
|
+
- Export my work as an EPUB file.
|
|
89
|
+
- Fill in the required information.
|
|
90
|
+
- For "Cover": check the option "Use the first page as the book cover image".
|
|
91
|
+
- For "Layout": check the reflowable option and "Use table of contents". For "Embed fonts", see the [Important Note for Apple Pages Users](#️-important-note-for-apple-pages-users) section below to decide based on whether preserving your typography/design is important to you.
|
|
92
|
+
|
|
93
|
+
After exporting, my original EPUB file is about 24.4MB. I use the author workflow preset to optimize it (resulting in about 7.3MB). Then I test the result in Apple Books, Kindle Previewer, etc.
|
|
94
|
+
|
|
95
|
+
This project started with this workflow (I don't use any other tools), but anyone who wants to optimize their EPUB file is welcome to try it! The default `pnpm optimize` path is generic. My complete Pages/manual-summary workflow is available through `pnpm optimize:author`, `--preset author`, or `--author-workflow`. If you have any questions or issues, let me know. Enjoy! :)
|
|
96
|
+
|
|
97
|
+
### ⚠️ Important Note for Apple Pages Users
|
|
98
|
+
|
|
99
|
+
**Apple Pages applies DRM encryption to embedded fonts** for copyright protection. This means:
|
|
100
|
+
|
|
101
|
+
- ✅ **Optional font optimization can work** on eligible unencrypted TTF, OTF, WOFF and WOFF2 fonts when `--fonts` is enabled
|
|
102
|
+
- ❌ **Font optimization WILL NOT work** on Pages EPUBs if you check "Embed fonts"
|
|
103
|
+
- 🔒 **Why?** Apple encrypts fonts to protect font vendors from piracy (this is intentional and legal)
|
|
104
|
+
|
|
105
|
+
**Your Decision: To Embed Fonts or Not?**
|
|
106
|
+
|
|
107
|
+
This depends on whether preserving your book's design and typography is important to you.
|
|
108
|
+
|
|
109
|
+
**Option 1: Keep "Embed fonts" checked** (Preserve design - recommended if typography matters)
|
|
110
|
+
|
|
111
|
+
- ✅ Your embedded typography and intended design are preserved more consistently
|
|
112
|
+
- ✅ Consistent appearance across all e-readers
|
|
113
|
+
- ✅ Full control over typography
|
|
114
|
+
- ❌ Fonts remain at their original size (typically 1-3 MB total, can't be optimized due to DRM encryption)
|
|
115
|
+
- 📊 My image-heavy workflow still achieves about 85-90% total reduction from image/HTML/CSS optimization
|
|
116
|
+
|
|
117
|
+
**Option 2: Uncheck "Embed fonts"** (Maximum optimization - recommended if design is not critical)
|
|
118
|
+
|
|
119
|
+
- ✅ **Maximum file size reduction potential** - optimize images/HTML/CSS and optionally subset compatible fonts with `--fonts`
|
|
120
|
+
- ✅ **Better performance** - smaller files load faster
|
|
121
|
+
- ✅ **More compatible** - readers' preferred fonts are used (better accessibility)
|
|
122
|
+
- ✅ **Cleaner output** - no encrypted font files that can't be processed
|
|
123
|
+
- ❌ You lose control over typography - readers will see their default fonts
|
|
124
|
+
- ❌ Your book's appearance becomes unpredictable across devices
|
|
125
|
+
- ❌ Design integrity may be compromised
|
|
126
|
+
|
|
127
|
+
**Choose based on your priority:**
|
|
128
|
+
|
|
129
|
+
- **Professional publications where typography matters** → Embed fonts (accept 1-3 MB overhead)
|
|
130
|
+
- **Maximum file size and performance** → Don't embed fonts (lose design control)
|
|
131
|
+
|
|
132
|
+
## Quick Start
|
|
133
|
+
|
|
134
|
+
> **Note:** Traditional usage accepts absolute or relative file paths. Docker Compose automatically shares the repository directory with the container, so Docker commands also use simple relative paths.
|
|
135
|
+
|
|
136
|
+
**Fastest: run the published image (no clone, no build):**
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
docker run --rm -v "$PWD:/epub-files" ghcr.io/kiki-le-singe/epub-optimizer \
|
|
140
|
+
-i YourBook.epub -o YourBook-optimized.epub
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The image ships Node.js, Java, and EPUBCheck pre-installed and supports amd64 and arm64 (Apple Silicon).
|
|
144
|
+
|
|
145
|
+
**Traditional installation:**
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
pnpm install --frozen-lockfile
|
|
149
|
+
pnpm build
|
|
150
|
+
pnpm optimize -i YourBook.epub -o YourBook-optimized.epub
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
The default `balanced` preset performs generic optimization without running the modifying XHTML repair or author workflow passes.
|
|
154
|
+
|
|
155
|
+
**Or use Docker Compose (recommended if you cloned the repository):**
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
git clone https://github.com/kiki-le-singe/epub-optimizer.git
|
|
159
|
+
cd epub-optimizer
|
|
160
|
+
# Place YourBook.epub in this directory, then run:
|
|
161
|
+
docker compose run --build --rm optimizer \
|
|
162
|
+
-i YourBook.epub -o YourBook-optimized.epub
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
The first run builds the image and optimizes the EPUB. Later runs can omit `--build`.
|
|
166
|
+
|
|
167
|
+
For the complete Pages/manual-summary author workflow:
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
docker compose run --rm optimizer \
|
|
171
|
+
-i YourBook.epub -o YourBook-optimized.epub --preset author
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
## Migrating to v3
|
|
175
|
+
|
|
176
|
+
Version 3 makes the default command safer and less specific to this project's original author workflow:
|
|
177
|
+
|
|
178
|
+
- `pnpm optimize` now uses the generic `balanced` preset. It does not apply XHTML repairs, lazy loading, or author structure changes unless explicitly requested.
|
|
179
|
+
- `pnpm optimize:author`, `--preset author`, and `--author-workflow` run the complete Pages/manual-summary workflow, including repairs, lazy loading, cover navigation, summary synchronization, and chapter sections.
|
|
180
|
+
- For behavior close to the old generic v2 pipeline without the author structure changes, use `pnpm optimize ... --repair --lazy-loading`.
|
|
181
|
+
- Input, output, and JSON report paths must be different. The output is created as a candidate and replaces the requested output only after EPUBCheck succeeds.
|
|
182
|
+
- Use `--strict` to reject warnings, `--profile` to inspect step durations, and `--report-json` for a machine-readable result.
|
|
183
|
+
|
|
184
|
+
## Features
|
|
185
|
+
|
|
186
|
+
- **Discovers the package/content root through `META-INF/container.xml` and resolves navigation/resources through the OPF manifest**
|
|
187
|
+
- HTML/XHTML minification (removes comments and minifies embedded code while preserving text and significant whitespace)
|
|
188
|
+
- CSS optimization (minifies and combines rules)
|
|
189
|
+
- Image compression (JPEG, PNG, WebP, GIF, AVIF, and SVG according to the selected preset and quality settings)
|
|
190
|
+
- JPEG recompression applies EXIF orientation to the pixels before encoding, preserving the displayed rotation and mirroring.
|
|
191
|
+
- PNG to JPEG conversion for non-transparent images (significantly reduces file size)
|
|
192
|
+
- JavaScript minification (reduces script size)
|
|
193
|
+
- **Optional font subsetting** via `--fonts`, using the included HarfBuzz WebAssembly engine with conservative content and font checks
|
|
194
|
+
- **SVG optimization** (minifies SVG files using SVGO)
|
|
195
|
+
- **Image downscaling** (optionally resizes large images to a max dimension for e-reader compatibility)
|
|
196
|
+
- **Lazy loading for images** (adds `loading="lazy"` to all `<img>` tags in XHTML for EPUB3 readers)
|
|
197
|
+
- **Cross-platform archive processing** (pure JavaScript implementation with no external ZIP binaries)
|
|
198
|
+
- Archive recompression (EPUB-compliant ZIP packaging with proper compression settings)
|
|
199
|
+
- EPUB validation against the EPUB specification, with optional structured success and failure reports
|
|
200
|
+
- Transactional output publication after successful EPUBCheck validation
|
|
201
|
+
- **Optional XHTML repair mode** via `--repair`
|
|
202
|
+
- **Optional author workflow preset** (syncs subsections from a manual summary page, updates cover navigation, and applies the project author's structure fixes)
|
|
203
|
+
- **Balanced, lossless, and author presets**
|
|
204
|
+
- Strict failure mode, per-step profiling, and JSON run reports
|
|
205
|
+
- Non-destructive doctor/inspect mode for OPF, navigation, internal reference, image weight, and optimization-risk summaries
|
|
206
|
+
- Archive extraction limits, zip-slip and symlink rejection, and safe OPF/manifest/navigation path resolution
|
|
207
|
+
- **Graceful handling of special EPUBs**: detects `META-INF/encryption.xml` and refuses DRM-encrypted books instead of silently corrupting them, while still optimizing books whose only encrypted resources are fonts (the fonts are left byte-for-byte intact)
|
|
208
|
+
- **Fixed-layout aware**: automatically disables image downscaling for pre-paginated (`rendition:layout`) EPUBs so page images stay aligned with the declared viewport
|
|
209
|
+
- **Animation-safe image processing**: keeps animated PNG (APNG) files byte-for-byte unchanged, including when resizing is requested; preserves animated GIF/WebP/AVIF frames and timing during processing. Recompression that would grow an image is discarded unless its dimensions actually shrink. Image concurrency bounds parallel processing on image-heavy books.
|
|
210
|
+
- Modular fix scripts for EPUB and OPF structure
|
|
211
|
+
- Command-line interface with customizable options
|
|
212
|
+
- File size comparison reporting
|
|
213
|
+
- Automated unit, EPUBCheck E2E, and Docker E2E test suites
|
|
214
|
+
|
|
215
|
+
> **Note:**
|
|
216
|
+
>
|
|
217
|
+
> - **Font subsetting limitation:** fonts listed in encryption metadata remain byte-for-byte intact. `--fonts` is opt-in and uses the included HarfBuzz engine; no separate font package installation is needed. Dynamic CSS/JavaScript text and unsupported font features are conservatively skipped. See [Font subsetting](#font-subsetting) for limits.
|
|
218
|
+
> - Lazy loading is enabled by the author preset or explicitly with `--lazy-loading`.
|
|
219
|
+
|
|
220
|
+
## Requirements
|
|
221
|
+
|
|
222
|
+
**Only required for traditional installation** (skip if using Docker):
|
|
223
|
+
|
|
224
|
+
- Node.js 22.12 or higher (CI validates Node.js 22 and 24; use Node.js 24 for development)
|
|
225
|
+
- Java Runtime Environment (JRE) 17 or higher for EPUBCheck (CI and Docker use OpenJDK 17)
|
|
226
|
+
- pnpm 10.30.3 for repository development and packing, pinned by `packageManager`; the installed CLI/API runs directly on Node.js
|
|
227
|
+
|
|
228
|
+
## Installation
|
|
229
|
+
|
|
230
|
+
```bash
|
|
231
|
+
# Clone the repository
|
|
232
|
+
git clone https://github.com/kiki-le-singe/epub-optimizer.git
|
|
233
|
+
cd epub-optimizer
|
|
234
|
+
|
|
235
|
+
# Install dependencies
|
|
236
|
+
pnpm install --frozen-lockfile
|
|
237
|
+
|
|
238
|
+
# Build the project (required before running optimize commands)
|
|
239
|
+
pnpm build
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
### Local package and typed Node API
|
|
243
|
+
|
|
244
|
+
Registry publication setup and the manual release workflow are described in
|
|
245
|
+
[docs/NPM.md](docs/NPM.md). These instructions do not mean a package is already
|
|
246
|
+
available on npm; until the first publication is confirmed, use the local tarball
|
|
247
|
+
or published Docker image.
|
|
248
|
+
|
|
249
|
+
Build and inspect a local package without publishing it to a registry:
|
|
250
|
+
|
|
251
|
+
```bash
|
|
252
|
+
# From this repository; prepack rebuilds JavaScript and declarations first.
|
|
253
|
+
pnpm pack --out /tmp/epub-optimizer.tgz
|
|
254
|
+
|
|
255
|
+
# From a separate Node.js project:
|
|
256
|
+
pnpm add /tmp/epub-optimizer.tgz
|
|
257
|
+
bash node_modules/epub-optimizer/scripts/install-epubcheck.sh ./epubcheck
|
|
258
|
+
EPUBCHECK_PATH="$PWD/epubcheck/epubcheck.jar" pnpm exec epub-optimizer \
|
|
259
|
+
-i book.epub -o book-optimized.epub --strict
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
The tarball contains compiled JavaScript, TypeScript declarations, README, license and the pinned EPUBCheck installer. It excludes TypeScript implementation sources, tests, development tools, local reports and books. Java and the EPUBCheck JAR are not bundled or downloaded during package installation; configure the JAR explicitly with `EPUBCHECK_PATH` as above. Docker remains the distribution that includes Java and EPUBCheck. Creating a tarball does not publish an npm release.
|
|
263
|
+
|
|
264
|
+
ESM consumers import the API and its types from the package root:
|
|
265
|
+
|
|
266
|
+
```typescript
|
|
267
|
+
import { optimizeEPUB, type Args, type OptimizeOptions } from "epub-optimizer";
|
|
268
|
+
|
|
269
|
+
export async function optimizeBook(args: Args, hooks: OptimizeOptions = {}) {
|
|
270
|
+
return optimizeEPUB(args, hooks);
|
|
271
|
+
}
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
`Args` still describes fully resolved processing options; this does not introduce a partial-options API or change defaults. Exported types also include `Preset`, `ContentProcessingOptions`, `BatchReport`, `BatchItem`, `PipelineReport`, `StepReport`, `StepStatus` and `EpubCheckReport`. Existing root and deep ESM imports remain available. The same Java/EPUBCheck validation and output protection apply to installed API calls.
|
|
275
|
+
|
|
276
|
+
`pnpm test:package` packs the project and installs the tarball with production dependencies into a new temporary consumer outside the repository, without dependency lifecycle scripts. It audits that consumer's production dependencies and verifies strict TypeScript usage (including rejected invalid types), CLI/API execution from another working directory, readable JavaScript stack locations, external EPUBCheck configuration and preservation of an existing output when the validator is missing. It requires Java, the pinned validator and registry access for consumer dependencies; it only uses generated EPUB fixtures. CI runs it on Node 22 and 24.
|
|
277
|
+
|
|
278
|
+
### EPUBCheck Setup
|
|
279
|
+
|
|
280
|
+
This tool requires EPUBCheck to validate EPUB files. Follow these steps:
|
|
281
|
+
|
|
282
|
+
1. Download EPUBCheck from the [official website](https://www.w3.org/publishing/epubcheck/)
|
|
283
|
+
2. Extract the downloaded zip file
|
|
284
|
+
3. Copy the extracted `epubcheck-x.x.x` folder (where x.x.x is the version) to the root of this project
|
|
285
|
+
4. Make sure the folder is named `epubcheck` to match the path in `epubcheckPath` in src/utils/config.ts
|
|
286
|
+
|
|
287
|
+
Or simply run `bash scripts/install-epubcheck.sh` from the project root — it downloads and installs the pinned EPUBCheck version into `epubcheck/` for you.
|
|
288
|
+
|
|
289
|
+
The default JAR is resolved from the installed package's `epubcheck/` directory,
|
|
290
|
+
so the CLI can run from another working directory. Set `EPUBCHECK_PATH` to use a
|
|
291
|
+
different installation, for example:
|
|
292
|
+
|
|
293
|
+
```bash
|
|
294
|
+
EPUBCHECK_PATH=/opt/epubcheck/epubcheck.jar epub-optimizer -i book.epub -o book-opt.epub
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
An explicit relative `EPUBCHECK_PATH` is resolved from the current working
|
|
298
|
+
directory. EPUBCheck messages use English regardless of the JVM's language.
|
|
299
|
+
`--strict` enables EPUBCheck's native rejection of warnings and also rejects a
|
|
300
|
+
missing or unreadable validation totals. Totals come from JSON, with the native
|
|
301
|
+
text summary retained as a compatibility fallback when detailed JSON diagnostics
|
|
302
|
+
are unavailable. A nonzero validator exit or timeout always fails validation.
|
|
303
|
+
|
|
304
|
+
CI and Docker pin EPUBCheck 5.4.0 and its sha256 checksum centrally in `scripts/install-epubcheck.sh` (single source of truth); the download is verified against the checksum before installation.
|
|
305
|
+
|
|
306
|
+
## Docker Alternative
|
|
307
|
+
|
|
308
|
+
You must have [Docker installed](https://docs.docker.com/get-docker/) with Docker Compose v2. Docker Desktop includes both.
|
|
309
|
+
|
|
310
|
+
### Pre-built Image (GHCR)
|
|
311
|
+
|
|
312
|
+
Every release publishes a multi-arch image (amd64 and arm64, including Apple Silicon) to GitHub Container Registry. It is the fastest way to use the optimizer — no clone, no build, no Node.js or Java to install:
|
|
313
|
+
|
|
314
|
+
```bash
|
|
315
|
+
docker run --rm -v "$PWD:/epub-files" ghcr.io/kiki-le-singe/epub-optimizer \
|
|
316
|
+
-i your-book.epub -o your-book-optimized.epub
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
Pin a specific release with a version tag, e.g. `ghcr.io/kiki-le-singe/epub-optimizer:3.5.0` (images are published starting with v3.5.0). Building locally with Compose (below) remains fully supported and is the right choice when modifying the source.
|
|
320
|
+
|
|
321
|
+
Docker Compose provides a containerized environment with all dependencies pre-installed. It automatically mounts the repository directory, so the same relative EPUB paths work on Windows, macOS, and Linux. When working from a clone, Compose builds the image locally.
|
|
322
|
+
|
|
323
|
+
### Docker Requirements
|
|
324
|
+
|
|
325
|
+
- Docker Desktop, or Docker Engine with Docker Compose v2
|
|
326
|
+
|
|
327
|
+
### Docker Installation
|
|
328
|
+
|
|
329
|
+
```bash
|
|
330
|
+
# Clone the repository
|
|
331
|
+
git clone https://github.com/kiki-le-singe/epub-optimizer.git
|
|
332
|
+
cd epub-optimizer
|
|
333
|
+
|
|
334
|
+
# Build the local image (includes all dependencies)
|
|
335
|
+
docker compose build
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
### Docker Quick Start
|
|
339
|
+
|
|
340
|
+
```bash
|
|
341
|
+
# Build when needed, then optimize an EPUB from the repository directory
|
|
342
|
+
docker compose run --build --rm optimizer \
|
|
343
|
+
-i your-book.epub -o your-book-optimized.epub
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
**Benefits of Docker approach:**
|
|
347
|
+
|
|
348
|
+
- ✅ No need to install Node.js, Java, or pnpm
|
|
349
|
+
- ✅ Works consistently across all platforms (Windows, Mac, Linux)
|
|
350
|
+
- ✅ Includes EPUBCheck automatically
|
|
351
|
+
- ✅ Uses the dependency versions tested by the project
|
|
352
|
+
- ✅ Automatically mounts the working directory and accepts relative EPUB paths
|
|
353
|
+
- ✅ Runs non-root with no network, no Linux capabilities, and a read-only container filesystem
|
|
354
|
+
|
|
355
|
+
## Usage
|
|
356
|
+
|
|
357
|
+
### Available Scripts
|
|
358
|
+
|
|
359
|
+
| Script | Description |
|
|
360
|
+
| ------------------- | --------------------------------------------------------------------------------------------- |
|
|
361
|
+
| `build` | Build TypeScript for production (with minification) |
|
|
362
|
+
| `build:dev` | Build TypeScript for development (no minification) |
|
|
363
|
+
| `build:prod` | Build TypeScript with minification for production |
|
|
364
|
+
| `minify:safe` | Optional legacy minification of compiled JavaScript; not part of the production/package build |
|
|
365
|
+
| `optimize` | Run optimizer, keeping temp files |
|
|
366
|
+
| `optimize:author` | Run the complete project-author workflow, including repairs and structure updates |
|
|
367
|
+
| `optimize:repair` | Run generic optimization plus modifying XHTML repair passes |
|
|
368
|
+
| `optimize:lossless` | Avoid lossy image conversion, recompression, and resizing |
|
|
369
|
+
| `optimize:clean` | Run optimizer, removing temp files afterward |
|
|
370
|
+
| `cleanup` | Remove temporary files |
|
|
371
|
+
| `test` | Run tests in watch mode |
|
|
372
|
+
| `test:run` | Run tests once and exit |
|
|
373
|
+
| `test:coverage` | Run tests with coverage report |
|
|
374
|
+
| `test:package` | Install and verify a local tarball, its CLI/API and TypeScript declarations |
|
|
375
|
+
| `test:e2e` | Validate every public pnpm optimization workflow with fixtures and EPUBCheck |
|
|
376
|
+
| `test:docker` | Validate raw Docker and the complete Docker Compose workflow matrix with EPUBCheck |
|
|
377
|
+
| `release:check` | Validate a requested release version against `package.json` and optional tag/ref guards |
|
|
378
|
+
| `release:prepare` | Prepare a version PR, wait for Required CI, then merge into `main` (`--dry-run` to preview) |
|
|
379
|
+
| `lint` | Lint TypeScript files in src and scripts directories |
|
|
380
|
+
| `lint:fix` | Lint and auto-fix TypeScript files in src and scripts |
|
|
381
|
+
| `format` | Auto-format all .ts, .json, and .md files with Prettier |
|
|
382
|
+
| `format:check` | Check formatting of all .ts, .json, and .md files with Prettier |
|
|
383
|
+
|
|
384
|
+
### Modern Workflow
|
|
385
|
+
|
|
386
|
+
```bash
|
|
387
|
+
# Build for development (faster, not minified)
|
|
388
|
+
pnpm build:dev
|
|
389
|
+
|
|
390
|
+
# Build for production (with minification)
|
|
391
|
+
pnpm build
|
|
392
|
+
# or
|
|
393
|
+
pnpm build:prod
|
|
394
|
+
|
|
395
|
+
# Then run the optimizer
|
|
396
|
+
pnpm optimize -i YourBook.epub -o YourBook-optimized.epub
|
|
397
|
+
|
|
398
|
+
# Or use the project author's Pages/manual-summary workflow
|
|
399
|
+
pnpm optimize:author -i YourBook.epub -o YourBook-optimized.epub
|
|
400
|
+
|
|
401
|
+
# Avoid lossy image changes
|
|
402
|
+
pnpm optimize:lossless -i YourBook.epub -o YourBook-optimized.epub
|
|
403
|
+
|
|
404
|
+
# Apply the explicit XHTML repair passes
|
|
405
|
+
pnpm optimize:repair -i YourBook.epub -o YourBook-optimized.epub
|
|
406
|
+
|
|
407
|
+
# Reject warnings, print timings, and write a machine-readable report
|
|
408
|
+
pnpm optimize -i YourBook.epub -o YourBook-optimized.epub \
|
|
409
|
+
--strict --profile --report-json reports/optimization-report.json
|
|
410
|
+
|
|
411
|
+
# Inspect an EPUB without writing an optimized output
|
|
412
|
+
pnpm optimize --doctor -i YourBook.epub \
|
|
413
|
+
--report-json reports/doctor-report.json
|
|
414
|
+
|
|
415
|
+
# Run tests
|
|
416
|
+
pnpm test
|
|
417
|
+
# or run tests once and exit
|
|
418
|
+
pnpm test:run
|
|
419
|
+
# run the EPUBCheck end-to-end fixture after pnpm build
|
|
420
|
+
pnpm test:e2e
|
|
421
|
+
|
|
422
|
+
# after docker compose build, run the raw Docker and Compose fixtures
|
|
423
|
+
pnpm test:docker
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
### Command Line Options
|
|
427
|
+
|
|
428
|
+
```
|
|
429
|
+
Usage: epub-optimizer [options]
|
|
430
|
+
|
|
431
|
+
Options:
|
|
432
|
+
-i, --input Input EPUB file path [string] [default: "mybook.epub"]
|
|
433
|
+
-o, --output Output EPUB file path [string] [default: "mybook_opt.epub"]
|
|
434
|
+
--config Explicit JSON settings; CLI flags override file values
|
|
435
|
+
--jobs Concurrent batch books (1-4; default: 1)
|
|
436
|
+
--storage Inspect a watcher output directory without deleting anything
|
|
437
|
+
--storage-plan Save a new JSON purge preview (requires --storage)
|
|
438
|
+
--apply-storage-plan Reverify and apply a saved purge preview
|
|
439
|
+
--keep-revisions Successful revisions retained per original (minimum 1; default 2)
|
|
440
|
+
--batch Directory of EPUB files to optimize (non-recursive)
|
|
441
|
+
--watch Continuously optimize EPUB files dropped in this directory
|
|
442
|
+
--watch-settle-ms Quiet interval before processing a watched file [default: 3000, minimum: 1000]
|
|
443
|
+
--output-dir Destination directory required with --batch or --watch
|
|
444
|
+
-t, --temp Temporary directory for processing [string] [default: unique temp_epub-<id>]
|
|
445
|
+
--jpg-quality JPEG compression quality (0-100) [number] [default: 70]
|
|
446
|
+
--png-quality PNG compression quality (0-1 scale) [number] [default: 0.6]
|
|
447
|
+
--lang UI language for labels (e.g. fr, en) [string] [default: "fr"]
|
|
448
|
+
--fonts Enable experimental font subsetting [boolean] [default: false]
|
|
449
|
+
--preset Optimization preset: balanced, lossless, author
|
|
450
|
+
--repair Apply potentially modifying XHTML repairs
|
|
451
|
+
--author-workflow Enable the complete author workflow, including repairs
|
|
452
|
+
--author-config JSON overrides for the author workflow's summary, cover, and CSS class mapping
|
|
453
|
+
--convert-png Convert large opaque PNG files to JPEG when safe
|
|
454
|
+
--png-background Opt in to flattening eligible transparent PNGs onto #RRGGBB before JPEG conversion
|
|
455
|
+
--no-convert-png Disable PNG-to-JPEG conversion
|
|
456
|
+
--lazy-loading Add loading="lazy" to XHTML images
|
|
457
|
+
--no-lazy-loading Disable lazy loading
|
|
458
|
+
--max-image-dim Maximum image width/height; 0 disables resizing
|
|
459
|
+
--strict Fail when a step emits warnings or errors
|
|
460
|
+
--validation-timeout
|
|
461
|
+
Maximum EPUBCheck runtime in milliseconds [number] [default: 120000]
|
|
462
|
+
--image-concurrency
|
|
463
|
+
Concurrent image operations, from 1 to 64 [number] [default: CPU parallelism, capped at 8]
|
|
464
|
+
--profile Print execution time for each pipeline step
|
|
465
|
+
--report-json Write a structured pipeline or batch summary report to JSON
|
|
466
|
+
--report-html Write a standalone HTML optimization or batch summary report
|
|
467
|
+
--doctor, --inspect
|
|
468
|
+
Inspect an EPUB without optimizing or writing an output EPUB
|
|
469
|
+
--clean Clean temporary files after processing [boolean] [default: false]
|
|
470
|
+
-h, --help Show help message [boolean]
|
|
471
|
+
-v, --version Show version number [boolean]
|
|
472
|
+
|
|
473
|
+
Examples:
|
|
474
|
+
pnpm optimize -i book.epub -o book-optimized.epub Basic optimization
|
|
475
|
+
pnpm optimize:clean -i book.epub -o book-opt.epub Optimize and clean temp files
|
|
476
|
+
pnpm optimize -i book.epub -o book-opt.epub --jpg-quality 85 Higher JPEG quality (less compression)
|
|
477
|
+
pnpm optimize -i book.epub -o book-opt.epub --png-quality 0.9 Higher PNG quality (less compression)
|
|
478
|
+
pnpm optimize -i book.epub -o book-opt.epub --fonts Enable font subsetting
|
|
479
|
+
pnpm optimize:lossless -i book.epub -o book-opt.epub Avoid lossy image changes
|
|
480
|
+
pnpm optimize:repair -i book.epub -o book-opt.epub Apply XHTML repair passes
|
|
481
|
+
pnpm optimize:author -i book.epub -o book-opt.epub Use the author's Pages/manual-summary workflow
|
|
482
|
+
pnpm optimize -i input.epub -o output.epub --jpg-quality 85 --png-quality 0.8 Custom image settings
|
|
483
|
+
pnpm optimize --doctor -i input.epub Inspect without modifying the EPUB
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
> **Script Differences:**
|
|
487
|
+
>
|
|
488
|
+
> - `pnpm optimize` - Optimizes the EPUB file and keeps temporary files for inspection
|
|
489
|
+
> - `pnpm optimize:lossless` - Skips lossy image conversion, recompression, and resizing
|
|
490
|
+
> - `pnpm optimize:repair` - Same as optimize, plus explicit XHTML repair passes
|
|
491
|
+
> - `pnpm optimize:author` - Complete author workflow: generic optimization, repairs, lazy loading, cover navigation, summary page, and chapter sections
|
|
492
|
+
> - `pnpm optimize:clean` - Same as optimize but removes temporary files afterward
|
|
493
|
+
> - `pnpm cleanup <directory>` - Removes a marked temporary directory from a previous run
|
|
494
|
+
|
|
495
|
+
### JSON configuration
|
|
496
|
+
|
|
497
|
+
Use an explicit JSON file with long, kebab-case option names. There is no automatic
|
|
498
|
+
search for configuration files and no executable JavaScript, includes or environment
|
|
499
|
+
interpolation. Built-in defaults apply first, file settings second, explicit CLI
|
|
500
|
+
flags last (including `--no-fonts` and other negative booleans).
|
|
501
|
+
|
|
502
|
+
```json
|
|
503
|
+
{
|
|
504
|
+
"preset": "balanced",
|
|
505
|
+
"strict": true,
|
|
506
|
+
"clean": true,
|
|
507
|
+
"image-concurrency": 2,
|
|
508
|
+
"watch": "inbox",
|
|
509
|
+
"output-dir": "optimized"
|
|
510
|
+
}
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
```bash
|
|
514
|
+
pnpm optimize --config optimizer.json
|
|
515
|
+
# Override one setting without editing the file:
|
|
516
|
+
pnpm optimize --config optimizer.json --image-concurrency 1
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
Paths from the file are relative to that file's directory; CLI paths are relative
|
|
520
|
+
to the current working directory. Unknown keys and mistyped values fail before
|
|
521
|
+
processing. The file must be regular JSON, at most 64 KiB. Optimization options,
|
|
522
|
+
paths, batch/watch settings and `jobs` are supported; help/version, storage/purge
|
|
523
|
+
commands, nested config and short/camelCase aliases are deliberately excluded.
|
|
524
|
+
Storage maintenance cannot be combined with `--config`. The settings file cannot
|
|
525
|
+
alias an output/report target and must stay outside batch/watch output directories.
|
|
526
|
+
|
|
527
|
+
### Storage inventory and controlled purge
|
|
528
|
+
|
|
529
|
+
Stop the watcher before applying a purge. Preview is read-only; keep two successful
|
|
530
|
+
revisions per original by default, or explicitly choose at least one:
|
|
531
|
+
|
|
532
|
+
```bash
|
|
533
|
+
pnpm optimize --storage watch/optimized --keep-revisions 2 \
|
|
534
|
+
--storage-plan /tmp/epub-storage-plan.json
|
|
535
|
+
# Review that JSON file, then explicitly apply that same preview:
|
|
536
|
+
pnpm optimize --apply-storage-plan /tmp/epub-storage-plan.json
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
A saved plan is created exclusively: an existing file is never overwritten.
|
|
540
|
+
`verifiedRevisionBytes` counts recognized, checksum-verified successful revisions
|
|
541
|
+
only; `candidateBytes` is their proposed removal size. These are apparent file
|
|
542
|
+
bytes, not total filesystem usage or guaranteed reclaimed space. Unknown entries,
|
|
543
|
+
failed runs, unexpected files, symlinks, hard links and altered outputs are listed
|
|
544
|
+
as protected rather than deleted. Revisions are grouped by original absolute path
|
|
545
|
+
and ordered by completion-record modification time. Originals are never read or
|
|
546
|
+
removed by maintenance.
|
|
547
|
+
|
|
548
|
+
Apply acquires the same lock as the watcher and recomputes the saved candidates;
|
|
549
|
+
changed candidates require a new preview. Only recognized revision files are
|
|
550
|
+
unlinked; no recursive deletion follows links. An interruption can leave an
|
|
551
|
+
`.epub-prune-*` recovery directory: it is preserved as an unknown entry, so inspect
|
|
552
|
+
it manually before any cleanup. A purge is irreversible; keep backups of outputs
|
|
553
|
+
you need. Paths in a container must refer to that container's mounted output directory;
|
|
554
|
+
create and apply a plan in the same path namespace.
|
|
555
|
+
|
|
556
|
+
Temporary extraction directories are kept separately. Use `--clean` for future
|
|
557
|
+
successful runs; remove an old, inspected, marked extraction directory explicitly
|
|
558
|
+
with `pnpm cleanup <directory>`. Storage retention does not sweep system temporary
|
|
559
|
+
folders or discard failed-run diagnostics automatically.
|
|
560
|
+
|
|
561
|
+
### Modes and Presets
|
|
562
|
+
|
|
563
|
+
| Behavior | `balanced` | `lossless` | `author` |
|
|
564
|
+
| -------------------------------- | ---------- | ---------- | -------- |
|
|
565
|
+
| HTML/CSS/JavaScript minification | Yes | Yes | Yes |
|
|
566
|
+
| SVG optimization | Yes | Yes | Yes |
|
|
567
|
+
| PNG-to-JPEG conversion | Yes | No | Yes |
|
|
568
|
+
| Raster image resize/re-encode | Yes | No | Yes |
|
|
569
|
+
| Default maximum image dimension | 1600 px | Disabled | 1600 px |
|
|
570
|
+
| Lazy loading | No | No | Yes |
|
|
571
|
+
| XHTML repairs | No | No | Yes |
|
|
572
|
+
| Author structure workflow | No | No | Yes |
|
|
573
|
+
|
|
574
|
+
- `balanced` is the default generic preset. It optimizes content and images without applying XHTML repairs or author structure changes.
|
|
575
|
+
- `lossless` means no lossy raster processing. Text, CSS, JavaScript, and SVG files are still optimized, so the output is not byte-for-byte identical to the input.
|
|
576
|
+
- `author` runs this project's complete Pages/manual-summary workflow and is equivalent to `pnpm optimize:author`.
|
|
577
|
+
- `--repair` explicitly enables the potentially modifying XHTML repair passes.
|
|
578
|
+
- XHTML repair preserves valid scripts and text directly inside `<body>`; it does not remove scripting from an EPUB. XHTML normalization also preserves quoted attributes and markup-like strings inside scripts.
|
|
579
|
+
- All three repair passes discover nested `.xhtml` files case-insensitively and XHTML documents declared in the OPF manifest, including `.html` and other extensions. Manifest paths are resolved relative to the OPF, so chapters may live outside its folder within the EPUB. Repair skips symbolic links and keeps valid microdata metadata, CDATA, and UTF-8 text.
|
|
580
|
+
- `--author-workflow` remains supported and always enables the complete author workflow, including repairs.
|
|
581
|
+
- `--author-config` optionally adapts the author workflow to another summary/cover/class mapping. Without it, the original project-author workflow is unchanged.
|
|
582
|
+
- Font subsetting is opt-in through `--fonts` and is not enabled by any preset.
|
|
583
|
+
|
|
584
|
+
Applicable CLI options override preset defaults. For example, `--preset balanced --no-convert-png` disables PNG-to-JPEG conversion. The `lossless` preset always skips lossy raster processing, and the author workflow always includes repairs and author structure updates.
|
|
585
|
+
|
|
586
|
+
#### Optional background for transparent PNGs
|
|
587
|
+
|
|
588
|
+
By default, PNGs with an alpha channel are kept as PNG. To allow eligible transparent PNGs to be converted to JPEG, explicitly choose an opaque background:
|
|
589
|
+
|
|
590
|
+
```bash
|
|
591
|
+
pnpm optimize -i book.epub -o book-white.epub --png-background '#ffffff'
|
|
592
|
+
```
|
|
593
|
+
|
|
594
|
+
`--png-background` accepts only a six-digit hex color (`#RRGGBB`), including the `#`; quote it in the shell. There is no default background. This option also applies to each book in batch mode and is available to Node API callers as `args.pngBackground`.
|
|
595
|
+
|
|
596
|
+
Flattening permanently replaces transparency, including around captions, with the chosen color. White backgrounds can create visible white rectangles in dark reading modes. JPEG encoding is lossy; inspect text inside images, diagrams and edges in the target readers. EPUBCheck and content/text checks do not certify visual quality.
|
|
597
|
+
|
|
598
|
+
The existing conversion safeguards still apply: PNGs smaller than 200 KiB and animated PNGs are skipped; the JPEG must be smaller than the original PNG, have a manifest entry and have supported, safely rewritable references. Conversion and resizing share one encoding pass; an unsuccessful conversion retains the PNG. This is not a command to recolor every PNG. Opaque PNGs retain their existing conversion behavior. `--png-background` is rejected with `--preset lossless`, `--no-convert-png`, or doctor mode, rather than silently ignored. Existing commands and presets are unchanged when the option is omitted.
|
|
599
|
+
|
|
600
|
+
### Safe temporary directories and validation deadlines
|
|
601
|
+
|
|
602
|
+
Each CLI run uses a unique `temp_epub-<id>` directory by default. An explicit
|
|
603
|
+
`--temp <path>` must not already exist: even an empty or previously used directory
|
|
604
|
+
is refused, so concurrent runs cannot overwrite one another. Existing directories
|
|
605
|
+
are never cleared during extraction. The console and JSON report (`tempDir`) show
|
|
606
|
+
the actual extraction path. `--clean` removes only the current run's marked
|
|
607
|
+
directory after successful publication. For a retained run, use
|
|
608
|
+
`pnpm cleanup <directory>`; unmarked directories and symbolic links are refused.
|
|
609
|
+
The hidden ownership marker is excluded from the output EPUB.
|
|
610
|
+
|
|
611
|
+
EPUBCheck has a 120-second deadline. Use `--validation-timeout 300000` for larger
|
|
612
|
+
books that need up to five minutes. A timeout fails the run, discards the candidate,
|
|
613
|
+
and preserves any previous output and the extracted files.
|
|
614
|
+
|
|
615
|
+
HTML/XHTML text whitespace and SVG identifiers are preserved because external
|
|
616
|
+
stylesheets and references can depend on them. Named character references such as
|
|
617
|
+
` ` or `é` (valid in EPUB 2) are rewritten as the equivalent numeric
|
|
618
|
+
references, and CDATA text outside scripts and styles is rewritten as escaped text:
|
|
619
|
+
the displayed characters are unchanged. XHTML minification keeps existing UTF-8
|
|
620
|
+
text in UTF-8.
|
|
621
|
+
XHTML documents are identified by their OPF media type as well as their extension
|
|
622
|
+
and XML markers. Before accepting minification, the optimizer compares element
|
|
623
|
+
nesting and order, identifiers, namespaces, and the position of character data. If these
|
|
624
|
+
change, it keeps the original document byte for byte and logs the skipped
|
|
625
|
+
optimization. Intentional inline CSS and JavaScript minification is excluded from
|
|
626
|
+
this comparison; final EPUBCheck validation still applies.
|
|
627
|
+
CDATA-wrapped or entity-escaped inline scripts and styles are kept as written.
|
|
628
|
+
CSS `@import` rules are kept instead of being inlined. External stylesheets,
|
|
629
|
+
`<style>` blocks and `style` attributes retain their CSS when the minifier reports
|
|
630
|
+
an error or warning, or offers no smaller result. XHTML styles also retain their
|
|
631
|
+
original CSS if minification would introduce XML delimiters. Raster recompressions
|
|
632
|
+
are committed only when smaller, unless they actually change the dimensions
|
|
633
|
+
requested by the user.
|
|
634
|
+
|
|
635
|
+
### Font subsetting
|
|
636
|
+
|
|
637
|
+
`--fonts` enables HarfBuzz/WASM subsetting of eligible TTF, OTF, WOFF and WOFF2
|
|
638
|
+
fonts, preserving their container format, hinting, layout features, embedding
|
|
639
|
+
permissions and copyright/license text. It is disabled by default for every
|
|
640
|
+
preset. Original bytes are retained when there is no size benefit or processing
|
|
641
|
+
fails. The feature does not remove fonts or change the book's text or CSS.
|
|
642
|
+
|
|
643
|
+
The character pool includes nested XHTML, navigation, SVG, attributes and literal
|
|
644
|
+
stylesheets across the archive, including Unicode NFC/NFD and case equivalents.
|
|
645
|
+
For safety, the entire font step is skipped when it encounters scripts, MathML,
|
|
646
|
+
embedded documents, imported/remote stylesheets, CSS variables, generated content,
|
|
647
|
+
unsupported list markers or text transformations. This deliberately sacrifices
|
|
648
|
+
some possible savings when displayed characters cannot be determined reliably.
|
|
649
|
+
All fonts are preserved when encryption metadata exists. Variable, color, MathML/
|
|
650
|
+
Graphite and embedding/subsetting-restricted fonts are also preserved.
|
|
651
|
+
|
|
652
|
+
Fonts within each book are processed sequentially, with at most 16 fonts per book, 8 MiB per input font,
|
|
653
|
+
16 MiB per expanded font, 16 MiB of text resources, 4,096 resources and 65,536
|
|
654
|
+
collected code points. Each font runs in a disposable child process with a
|
|
655
|
+
10-second timeout and a 128 MiB V8 heap cap. Parent-side RSS sampling every 100 ms
|
|
656
|
+
terminates a worker observed above 256 MiB; this is a sampled budget, **not an OS
|
|
657
|
+
hard memory limit**, and short allocation spikes can exceed it. Monitoring is
|
|
658
|
+
supported on Linux (`/proc`) and macOS (`/bin/ps`); if it is unavailable, fonts
|
|
659
|
+
remain unchanged. Other platforms conservatively skip this optional feature.
|
|
660
|
+
These budgets apply per book/worker; concurrent API calls can run separate workers.
|
|
661
|
+
|
|
662
|
+
Failures and uncertain content produce report warnings; `--strict` can reject
|
|
663
|
+
these warnings while preserving an existing final EPUB. Known unsupported fonts
|
|
664
|
+
and absent size benefits are informational skips. EPUBCheck validates the final
|
|
665
|
+
candidate but does not certify font rendering on every reading system. A candidate
|
|
666
|
+
cleanup or ownership failure aborts optimization even without `--strict`, rather
|
|
667
|
+
than packaging a stray temporary font.
|
|
668
|
+
|
|
669
|
+
### Image Concurrency
|
|
670
|
+
|
|
671
|
+
Use `--image-concurrency 2` on memory-constrained machines, or adjust from 1 to 64.
|
|
672
|
+
The default is the available CPU parallelism capped at 8. The setting applies to
|
|
673
|
+
both PNG-to-JPEG conversion and image recompression. Lower values trade runtime for
|
|
674
|
+
lower peak memory; compare with `--profile` on representative books before tuning.
|
|
675
|
+
|
|
676
|
+
A local Node 24 measurement on the 60 MB example, for the image stage alone, gave:
|
|
677
|
+
|
|
678
|
+
| Concurrent images | Time | Peak Node RSS |
|
|
679
|
+
| ----------------- | ------- | ------------- |
|
|
680
|
+
| 1 | 13.31 s | 210 MiB |
|
|
681
|
+
| 2 | 6.99 s | 307 MiB |
|
|
682
|
+
| 4 | 4.53 s | 417 MiB |
|
|
683
|
+
| 8 | 4.39 s | 429 MiB |
|
|
684
|
+
|
|
685
|
+
These are single measurements on one Mac, not cross-platform guarantees. All four
|
|
686
|
+
runs produced identical raster bytes (SHA-256 comparison). Java memory is excluded.
|
|
687
|
+
|
|
688
|
+
### Doctor / Inspect Mode
|
|
689
|
+
|
|
690
|
+
Use `--doctor` or `--inspect` to analyze an EPUB without publishing a candidate output and without modifying the input archive:
|
|
691
|
+
|
|
692
|
+
```bash
|
|
693
|
+
pnpm optimize --doctor -i book.epub
|
|
694
|
+
pnpm optimize --inspect -i book.epub --report-json reports/book-doctor.json
|
|
695
|
+
```
|
|
696
|
+
|
|
697
|
+
Doctor mode extracts the EPUB to a system temporary directory, reads `container.xml` and the OPF manifest, prints a console summary, writes JSON when `--report-json` is provided, and removes its temporary extraction directory. It does not run the optimizing processors and it does not replace EPUBCheck.
|
|
698
|
+
|
|
699
|
+
For supported books, the report includes OPF path/version, manifest and spine counts, EPUB3/NCX navigation presence, internal reference counts and missing targets, image byte totals and largest images, duplicate image basenames, large raster images, opaque PNG conversion candidates, and other optimization-risk hints.
|
|
700
|
+
|
|
701
|
+
Doctor checks internal DTD subsets and content DRM before interpreting package or content XML. When either blocks optimization, it prints the reason and writes `analysisComplete: false`, `optimization.blocked: true`, and structured blockers in `optimization.blockers` and `risks`. Detailed `opf`, `navigation`, `references`, `images`, and `content` fields are omitted rather than reporting misleading metrics. Font obfuscation alone remains inspectable. Exit code 0 means the diagnosis completed, including a diagnosis that finds a blocker; optimizing a blocked book still fails.
|
|
702
|
+
|
|
703
|
+
For a complete diagnosis, `analysisComplete` is true and `optimization.blocked` is false. These fields do not certify EPUB validity or replace EPUBCheck. Consumers of doctor JSON must check `analysisComplete` before reading detailed metrics.
|
|
704
|
+
|
|
705
|
+
Image reporting distinguishes `images.totalBytes` (uncompressed manifest image file bytes) from `images.compressedBytes` (the compressed ZIP payload of unique image resources). `images.archivePercent` uses compressed image bytes divided by the EPUB archive size, including ZIP overhead. Multiple manifest references to the same resource count once in the compressed share. When inspecting only an extracted directory without ZIP metadata, `compressedBytes` and `archivePercent` are `null`; uncompressed bytes are never presented as a percentage of compressed archive size.
|
|
706
|
+
|
|
707
|
+
#### Optional Author Workflow Configuration
|
|
708
|
+
|
|
709
|
+
`pnpm optimize:author` continues to use the original Pages/manual-summary conventions by default: `chapter-2.xhtml`, `cover.xhtml`, cover spine id `cover`, chapter classes `p6`/`p8`, section class `p7`, and navigation classes `s3`/`s4`.
|
|
710
|
+
|
|
711
|
+
For a different authoring workflow, create a JSON file containing only the values that differ:
|
|
712
|
+
|
|
713
|
+
```json
|
|
714
|
+
{
|
|
715
|
+
"summaryHref": "contents.xhtml",
|
|
716
|
+
"coverSpineId": "front-cover",
|
|
717
|
+
"chapterClasses": ["chapter-link"],
|
|
718
|
+
"sectionClasses": ["section-link"],
|
|
719
|
+
"summaryEntryClass": "chapter-link",
|
|
720
|
+
"coverNavClass": "toc-cover",
|
|
721
|
+
"sectionNavClass": "toc-section"
|
|
722
|
+
}
|
|
723
|
+
```
|
|
724
|
+
|
|
725
|
+
Then pass it to the same author command:
|
|
726
|
+
|
|
727
|
+
```bash
|
|
728
|
+
pnpm optimize:author -i book.epub -o book-optimized.epub \
|
|
729
|
+
--author-config author-workflow.json
|
|
730
|
+
```
|
|
731
|
+
|
|
732
|
+
The configuration is validated before structure updates. Unknown keys, unsafe paths, and invalid class/id values fail the run while preserving the temporary directory for debugging.
|
|
733
|
+
|
|
734
|
+
### Structured Reports and Profiling
|
|
735
|
+
|
|
736
|
+
- `--strict` also fails the run on processing or EPUBCheck warnings; processing and EPUBCheck errors always fail.
|
|
737
|
+
- `--report-html reports/book.html` writes a standalone, offline HTML report with archive size changes, validation/publication status, step diagnostics and a byte/content comparison of extracted resources. It is available for optimization runs; doctor mode continues to use JSON.
|
|
738
|
+
- `--profile` prints each pipeline step's status and duration.
|
|
739
|
+
- `--report-json reports/report.json` writes a structured report on success and on pipeline failures that occur after preflight path validation.
|
|
740
|
+
- The report directory is created automatically. Local reports under `reports/` are ignored by Git.
|
|
741
|
+
|
|
742
|
+
Processing diagnostics are emitted explicitly and collected within each asynchronous step; the optimizer does not replace global `console` methods or infer severity from message text. Step `messages` and strict warning/error handling retain their existing format. Custom API processing hooks should use the exported `reportWarning(...args)` and `reportError(...args)` helpers to participate in reports and strict checks. Ordinary `console` output remains display-only; thrown exceptions still fail the operation. Nested step diagnostics also reach the enclosing operation, while concurrent operations have separate collectors and completed steps stop collecting late callbacks. Batch concurrency is bounded by `--jobs` (one by default).
|
|
743
|
+
|
|
744
|
+
Once EPUBCheck runs, optimization reports include an optional `validation` object with its native `exitCode`, diagnostic `source` (`json`, `text`, or `unavailable`), version when available, `totals` (`fatals`, `errors`, `warnings`, `infos`), and detailed `messages`. Each message contains a `code`, uppercase `severity`, `message`, `locations` (path and available line/column/context), `additionalLocations`, and an optional suggestion. The same details appear in HTML reports and per-book batch reports, including failed runs. These fields are additive; existing step statuses and messages are retained, and doctor mode is unchanged.
|
|
745
|
+
|
|
746
|
+
EPUBCheck groups repeated messages and [defaults to at most 25 occurrences per code/message combination](https://www.w3.org/publishing/epubcheck/docs/cli/); totals may exceed the detailed entries returned. Unknown line/column values are omitted. Publication metadata and unrelated raw validator fields are not copied into the optimization report. JSON diagnostics are read from a unique private temporary file with a 16 MiB limit and removed after the invocation. If they are missing, malformed, unsupported or too large, `diagnosticsError` explains that details are unavailable and the native text totals are used when readable. Strict mode still rejects warnings and unavailable totals; non-strict mode retains its native-exit fallback with an explicit warning when totals cannot be read. Fatal/error diagnostics also block publication even if a validator incorrectly exits with code 0.
|
|
747
|
+
|
|
748
|
+
HTML reports can be used alongside `--report-json`; all report paths must be distinct from the EPUBs and from each other, including symbolic/hard-link aliases. Reports are written on success and on processing/validation failures after preflight. A report write failure makes the command fail, while preserving any already-published validated EPUB. Resource hashing adds disk reads only when HTML reporting is requested. The report contains no external assets or scripts and escapes book-derived filenames and diagnostics. It contains local paths; review it before sharing. Added and removed resources can describe format conversion, and byte comparisons do not certify visual rendering.
|
|
749
|
+
|
|
750
|
+
```bash
|
|
751
|
+
pnpm optimize -i book.epub -o book-optimized.epub --clean \
|
|
752
|
+
--report-html reports/book.html --report-json reports/book.json
|
|
753
|
+
```
|
|
754
|
+
|
|
755
|
+
Reports contain the configured input/output paths, preset, strict mode, timestamps, total duration, size reduction when available, per-step statuses/durations/messages, before/after content metrics, integrity checks, and the final error when a run fails:
|
|
756
|
+
|
|
757
|
+
```json
|
|
758
|
+
{
|
|
759
|
+
"input": "/books/input.epub",
|
|
760
|
+
"output": "/books/output.epub",
|
|
761
|
+
"preset": "balanced",
|
|
762
|
+
"strict": true,
|
|
763
|
+
"success": true,
|
|
764
|
+
"durationMs": 1234,
|
|
765
|
+
"content": {
|
|
766
|
+
"before": {
|
|
767
|
+
"manifestItems": 42,
|
|
768
|
+
"spineItems": 18,
|
|
769
|
+
"contentDocuments": 20,
|
|
770
|
+
"images": 12,
|
|
771
|
+
"navigationEntries": 36,
|
|
772
|
+
"missingReferences": 0
|
|
773
|
+
},
|
|
774
|
+
"after": {
|
|
775
|
+
"manifestItems": 42,
|
|
776
|
+
"spineItems": 18,
|
|
777
|
+
"contentDocuments": 20,
|
|
778
|
+
"images": 12,
|
|
779
|
+
"navigationEntries": 36,
|
|
780
|
+
"missingReferences": 0
|
|
781
|
+
},
|
|
782
|
+
"integrity": {
|
|
783
|
+
"valid": true,
|
|
784
|
+
"newMissingReferences": [],
|
|
785
|
+
"issues": []
|
|
786
|
+
}
|
|
787
|
+
},
|
|
788
|
+
"steps": [
|
|
789
|
+
{
|
|
790
|
+
"name": "Extract EPUB",
|
|
791
|
+
"status": "success",
|
|
792
|
+
"durationMs": 42,
|
|
793
|
+
"messages": []
|
|
794
|
+
}
|
|
795
|
+
]
|
|
796
|
+
}
|
|
797
|
+
```
|
|
798
|
+
|
|
799
|
+
The report path must differ from the input and output EPUB paths and must be absent or a regular file. A report write failure fails the command.
|
|
800
|
+
|
|
801
|
+
### Batch optimization
|
|
802
|
+
|
|
803
|
+
`--jobs 2` enables two books in flight (`1` by default, maximum `4`). Output paths
|
|
804
|
+
are checked for the entire batch before work starts, each book has a unique temp
|
|
805
|
+
directory and report, and summary rows retain input order even when completion
|
|
806
|
+
order differs. A failed book does not stop the others. Each book retains its own
|
|
807
|
+
image concurrency and Java memory requirements; start with `--image-concurrency 1`
|
|
808
|
+
on a constrained NAS. EPUBCheck runs asynchronously so timers and sibling books remain responsive.
|
|
809
|
+
Concurrency is not a guarantee of a proportional speedup. See the reproducible
|
|
810
|
+
[batch measurement](docs/BATCH-PERFORMANCE.md).
|
|
811
|
+
|
|
812
|
+
Optimize the regular `.epub` files directly inside a directory (case-insensitive extension, non-recursive):
|
|
813
|
+
|
|
814
|
+
```bash
|
|
815
|
+
pnpm optimize --batch books --output-dir optimized --preset balanced --clean \
|
|
816
|
+
--report-json reports/batch.json --report-html reports/batch.html
|
|
817
|
+
```
|
|
818
|
+
|
|
819
|
+
Books are processed one at a time in filename order, using the same candidate validation and transactional output publication as single-book runs. Image concurrency still applies within each book. A failure is recorded and the remaining books continue. Exit code 0 means every book succeeded; exit code 1 means one or more books failed, or planning/report writing failed.
|
|
820
|
+
|
|
821
|
+
Outputs are named `<original-stem>-optimized.epub`. Per-book JSON reports are always written under `<output-dir>/reports`; requesting an HTML summary also creates per-book HTML reports there. Without `--report-json`, the JSON summary defaults to `<output-dir>/batch-report.json`. The summary lists each book's result, whether this invocation published a validated output, and any error. A later cleanup/report failure can mark a run failed even though its validated output was already published.
|
|
822
|
+
|
|
823
|
+
The entire path plan is checked before processing: input/output/report aliases, hard links, duplicate target names and file/parent collisions are rejected. EPUB input symlinks and symbolic-link directory roots are refused. Use a distinct output directory. Do not combine `--batch` with explicit `--input`, `--output`, `--temp`, or doctor mode. Empty directories fail without creating outputs. Each book receives a unique system temporary directory; `--clean` removes successful extraction directories while failed runs keep their diagnostics.
|
|
824
|
+
|
|
825
|
+
The exported `runBatch(args)` API returns the structured summary without setting a process exit code. Pass resolved options, including `batch` and `outputDir`; use `optimizeEPUB(args)` for single-book calls.
|
|
826
|
+
|
|
827
|
+
### Background Docker watcher
|
|
828
|
+
|
|
829
|
+
For a NAS or custom host directory, the setup helper creates directories and a
|
|
830
|
+
private env file without starting Docker or changing existing settings:
|
|
831
|
+
|
|
832
|
+
```bash
|
|
833
|
+
sh scripts/setup-watch.sh /absolute/path/to/epub-library .env.watch
|
|
834
|
+
docker compose --env-file .env.watch -f compose.watch.yaml config
|
|
835
|
+
docker compose --env-file .env.watch -f compose.watch.yaml up -d
|
|
836
|
+
```
|
|
837
|
+
|
|
838
|
+
Review `WATCH_UID`/`WATCH_GID` for the NAS account allowed to access the files.
|
|
839
|
+
The helper pins the published 3.15.0 digest; it does not install Docker, elevate
|
|
840
|
+
permissions, change ownership or enable unreleased source features in that image.
|
|
841
|
+
It refuses existing env files and symbolic-link storage directories. Linux NAS
|
|
842
|
+
vendor interfaces and ACLs vary; validate permissions on the target device.
|
|
843
|
+
|
|
844
|
+
One-time setup from the repository root (Docker Compose v2 required):
|
|
845
|
+
|
|
846
|
+
```bash
|
|
847
|
+
mkdir -p watch/inbox watch/optimized
|
|
848
|
+
printf 'WATCH_UID=%s\nWATCH_GID=%s\n' "$(id -u)" "$(id -g)" > .env.watch
|
|
849
|
+
```
|
|
850
|
+
|
|
851
|
+
Start the published, digest-pinned image in the background:
|
|
852
|
+
|
|
853
|
+
```bash
|
|
854
|
+
docker compose --env-file .env.watch -f compose.watch.yaml up -d
|
|
855
|
+
```
|
|
856
|
+
|
|
857
|
+
Drop EPUBs into `watch/inbox`. Results and per-book reports appear in
|
|
858
|
+
`watch/optimized`, in separate revision directories. The inbox is mounted read-only;
|
|
859
|
+
both directories and `.env.watch` are ignored by Git and Docker builds. No server
|
|
860
|
+
or network connection runs inside the container. The initial image download requires
|
|
861
|
+
network access on the host. The existing `compose.yaml` single-book workflow is unchanged.
|
|
862
|
+
|
|
863
|
+
```bash
|
|
864
|
+
# Show progress (Ctrl+C stops following logs, not the watcher)
|
|
865
|
+
docker compose --env-file .env.watch -f compose.watch.yaml logs -f watch
|
|
866
|
+
# Stop and remove the service; books and results remain on the host
|
|
867
|
+
docker compose --env-file .env.watch -f compose.watch.yaml down
|
|
868
|
+
```
|
|
869
|
+
|
|
870
|
+
Docker restarts the service after normal daemon/machine restarts unless explicitly
|
|
871
|
+
stopped. Shutdown allows five minutes for the current book; increase
|
|
872
|
+
`stop_grace_period` for longer operations. **Forced termination or power loss can
|
|
873
|
+
leave a stale output lock:** stop the service, confirm no other watcher is using
|
|
874
|
+
that output, remove `watch/optimized/.epub-watch.lock`, then start again. The service
|
|
875
|
+
does not automatically break locks. An unchanged failed book is retried after restart.
|
|
876
|
+
Logs rotate at 10 MB with three files retained.
|
|
877
|
+
|
|
878
|
+
Newer Linux/Docker builds use a kernel-backed guard and can recover their own
|
|
879
|
+
interrupted watch records automatically after process/container death. The pinned
|
|
880
|
+
3.13.0 Compose image predates this capability: select a release that includes it
|
|
881
|
+
(or build the current sources) before relying on automatic crash recovery.
|
|
882
|
+
|
|
883
|
+
Use `EPUB_WATCH_INBOX` and `EPUB_WATCH_OUTPUT` for other host directories (create them
|
|
884
|
+
first). On a NAS, set `WATCH_UID` and `WATCH_GID` in `.env.watch` to an account allowed
|
|
885
|
+
to read the inbox and write the output. Override `EPUB_OPTIMIZER_DOCKER_IMAGE` only
|
|
886
|
+
with the version/digest you intend to run; it is never updated implicitly by this file.
|
|
887
|
+
|
|
888
|
+
### Watched folder
|
|
889
|
+
|
|
890
|
+
Open `<output-dir>/index.html` in a browser for an offline overview of completed
|
|
891
|
+
and failed processing attempts, size reductions, diagnostics and links to each
|
|
892
|
+
optimized book and JSON/HTML report. The file updates after each recorded attempt;
|
|
893
|
+
refresh the browser to see changes. Results survive restart via `watch-status.json`
|
|
894
|
+
in each job directory. No web server, scripts or external assets are needed. Older
|
|
895
|
+
jobs are also listed when their bounded JSON reports are readable. Invalid/oversized
|
|
896
|
+
records are ignored; original reports remain available.
|
|
897
|
+
|
|
898
|
+
```bash
|
|
899
|
+
mkdir -p inbox optimized
|
|
900
|
+
pnpm optimize --watch inbox --output-dir optimized --strict --clean
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
The process keeps running, scans the top level every second (including existing books),
|
|
904
|
+
and optimizes one book at a time. Drop an EPUB into `inbox`; each successful job gets
|
|
905
|
+
its own directory under `optimized`, containing `optimized.epub`, `report.json`,
|
|
906
|
+
`report.html`, and `completed.json`. Original files remain in the inbox untouched.
|
|
907
|
+
The directory name includes the book name and hashes of its name, contents, tool
|
|
908
|
+
version and settings. Changed books or different optimization settings get new
|
|
909
|
+
outputs; previous successful outputs remain available. Author configuration is
|
|
910
|
+
snapshotted at startup; restart the watcher after editing that configuration. Books with the same contents
|
|
911
|
+
and settings are skipped on restart using the completion record and output checksum.
|
|
912
|
+
Do not edit the completion records. Reports refer to the private input snapshot;
|
|
913
|
+
`completed.json` records the original inbox path.
|
|
914
|
+
|
|
915
|
+
Files must remain unchanged in size, timestamps and identity for at least three
|
|
916
|
+
seconds before processing. Adjust with `--watch-settle-ms 10000` (minimum 1000).
|
|
917
|
+
**A quiet interval cannot prove that a copy has finished.** Prefer copying with a
|
|
918
|
+
name such as `book.epub.part`, then renaming it to `book.epub` once the transfer
|
|
919
|
+
finishes. The watcher ignores other extensions, nested directories, empty files,
|
|
920
|
+
and symbolic links. A private snapshot is used during optimization; changes during
|
|
921
|
+
snapshot copying defer processing until the file settles again.
|
|
922
|
+
|
|
923
|
+
Failed books are logged and do not stop later books. An unchanged failure is tried
|
|
924
|
+
once per session; change the file or restart the watcher to retry it. Existing
|
|
925
|
+
successful output is protected by the same EPUBCheck-before-publication pipeline.
|
|
926
|
+
`--strict`, presets, `--fonts` and image options apply normally. JSON and HTML reports
|
|
927
|
+
are automatic; do not pass `--report-json` or `--report-html`. Input and output
|
|
928
|
+
directories must be separate, non-nested real directories. Watch mode cannot be
|
|
929
|
+
combined with batch, doctor, or explicit input/output/temp paths.
|
|
930
|
+
|
|
931
|
+
For Docker, mount the parent containing both directories (build the current sources first):
|
|
932
|
+
|
|
933
|
+
```bash
|
|
934
|
+
docker build --target runtime -t epub-optimizer .
|
|
935
|
+
docker run -d --name epub-watch --read-only --network none --tmpfs /tmp \
|
|
936
|
+
-v "$PWD:/epub-files" epub-optimizer \
|
|
937
|
+
--watch inbox --output-dir optimized --strict --clean
|
|
938
|
+
docker logs -f epub-watch
|
|
939
|
+
# In another terminal, allow the current book time to finish:
|
|
940
|
+
docker stop --time 300 epub-watch
|
|
941
|
+
docker rm epub-watch
|
|
942
|
+
```
|
|
943
|
+
|
|
944
|
+
The mounted output directory must be writable by the container user. Files and
|
|
945
|
+
reports stay on the host; Java and EPUBCheck are included in the image.
|
|
946
|
+
|
|
947
|
+
Ctrl+C or SIGTERM stops accepting new books and lets the current book finish. Exit
|
|
948
|
+
status is 1 if any book failed during that session, otherwise 0. Polling works on
|
|
949
|
+
mounted folders as well as local disks. Only one watcher may use an output directory;
|
|
950
|
+
an exclusive `.epub-watch.lock` enforces this. With legacy exclusive-file locking, if the process was forcibly killed or
|
|
951
|
+
the machine crashed, first confirm no watcher is running, then remove that stale lock
|
|
952
|
+
before restarting. An output or completion record altered externally is refused;
|
|
953
|
+
choose a fresh output directory to rerun it safely.
|
|
954
|
+
|
|
955
|
+
#### Watch lock recovery
|
|
956
|
+
|
|
957
|
+
On Linux with `/usr/bin/flock` and a mount that retains descriptor locks, the watcher
|
|
958
|
+
retains an exclusive lock on a persistent
|
|
959
|
+
`.epub-watch.guard` file for its entire lifetime. Process/container death releases
|
|
960
|
+
the kernel lock. After reacquiring that same guard, a new watcher can remove its
|
|
961
|
+
own modern interrupted `.epub-watch.lock` record and resume. No timestamp timeout
|
|
962
|
+
or PID guessing is used. **Do not delete or replace the guard file.**
|
|
963
|
+
|
|
964
|
+
Legacy/unknown records, changed guard identity, unsafe targets and lock errors are
|
|
965
|
+
refused. macOS, Windows and Linux without the utility retain the previous exclusive
|
|
966
|
+
file behavior and require manual recovery after a crash. The watcher probes lock
|
|
967
|
+
retention at startup; shared mounts that do not retain descriptor locks also use
|
|
968
|
+
this conservative fallback. For those cases, stop the
|
|
969
|
+
service, confirm no watcher uses the output, then remove the stale record before
|
|
970
|
+
restarting. Never remove another active watcher's lock. Automatic recovery is tested
|
|
971
|
+
on local Linux-backed Docker mounts; remote/network filesystems and their locking
|
|
972
|
+
mount options require separate validation. See [Linux flock semantics](https://man7.org/linux/man-pages/man2/flock.2.html).
|
|
973
|
+
|
|
974
|
+
### Safety Model
|
|
975
|
+
|
|
976
|
+
The exported `optimizeEPUB(args)` Node.js entry point uses the same pipeline as the CLI: content/text preservation checks, optional repair and author passes, candidate EPUBCheck validation, and transactional publication. Java and EPUBCheck must be available for API calls too; `validationTimeout`, `strict`, `reportJson`, and `clean` apply as they do in the CLI. Validation failure or timeout preserves an existing output. The supplied options object is copied before feature detection adjusts processing settings.
|
|
977
|
+
|
|
978
|
+
Advanced `optimizeEPUB(args, { skipPackaging: true })` calls only prepare extracted content. They never create, replace or validate an output EPUB, and leave the extraction directory available even when `clean` is true. Processing hooks apply to the content-processing stage and do not replace final validation.
|
|
979
|
+
|
|
980
|
+
The final EPUB is transactional: the optimizer creates a hidden candidate next to the requested output, validates that candidate with EPUBCheck, and publishes it only after validation succeeds. If optimization, packaging, validation, or publication fails before the commit, the candidate is discarded, the previous output remains untouched, and any temporary processing directory already created is preserved for debugging.
|
|
981
|
+
|
|
982
|
+
Before parsing package metadata or optimizing content, every preset checks for internal DTD subsets in XML resources, regardless of their filename. These are refused with a message identifying the file, even without `--strict`: custom entity declarations can otherwise change displayed text while still passing EPUBCheck. This conservative check also refuses empty or comment-only internal subsets. Ordinary `<!DOCTYPE html>` and standard EPUB 2 external doctypes remain supported. The check does not resolve entities or fetch external DTDs. For affected books, expand the internal declarations to literal text or numeric character references with a tool that supports them before optimizing. A refusal preserves the input, any existing output, and the extracted files for inspection.
|
|
983
|
+
|
|
984
|
+
Before packaging, the optimizer also compares the extracted EPUB before and after processing. Publication is blocked if manifest, spine, document, image, or navigation counts decrease, or if processing introduces a new missing internal reference. Existing input issues remain visible in the report without turning a previously processable EPUB into a false failure.
|
|
985
|
+
|
|
986
|
+
The text of every content document is also compared before and after the optimizing and repair passes (the `Validate text preservation` step). Scripts, styles, comments and markup are ignored and character references are resolved, so re-encoding is accepted, but publication is blocked if any displayed character is lost or altered. The check runs before the author workflow, whose navigation and summary edits are intentional.
|
|
987
|
+
|
|
988
|
+
SVG resources are included: text, titles, accessibility descriptions and foreign object content are checked. Editor metadata and formatting whitespace between graphical elements are ignored. HTML/XHTML documents throughout the archive are checked, including secondary renditions.
|
|
989
|
+
|
|
990
|
+
Cleanup and JSON report writing happen after the validated EPUB is published. If either of these later operations fails, the command exits non-zero but keeps the newly published, EPUBCheck-validated output.
|
|
991
|
+
|
|
992
|
+
Input, output, and report collisions are rejected, including aliases through symbolic links or hard links and case-only aliases on macOS/Windows. Existing output and report targets must be regular files.
|
|
993
|
+
|
|
994
|
+
Temporary-directory cleanup is guarded: the optimizer refuses filesystem roots, the current working directory, the home directory, and any temp directory containing the input or output EPUB. Manual cleanup requires the optimizer's ownership marker and rejects symbolic links. The build cleanup script also refuses paths outside the project or system temp directories.
|
|
995
|
+
|
|
996
|
+
Archive extraction rejects unsafe absolute/traversal paths and symbolic links. It also limits archives to 20,000 entries, 512 MB per file, 2 GB total extracted size, and a maximum compression ratio of 1000:1.
|
|
997
|
+
|
|
998
|
+
`META-INF/container.xml` and the OPF package document are the source of truth for content discovery. OPF and navigation references are resolved safely and cannot escape the extracted EPUB root.
|
|
999
|
+
|
|
1000
|
+
### Docker Usage
|
|
1001
|
+
|
|
1002
|
+
Docker Compose is the recommended cross-platform Docker interface. It automatically builds or reuses the local image, mounts the repository at `/epub-files`, and lets the CLI use normal relative paths.
|
|
1003
|
+
|
|
1004
|
+
**First run or rebuild after updating the repository:**
|
|
1005
|
+
|
|
1006
|
+
```bash
|
|
1007
|
+
docker compose run --build --rm optimizer \
|
|
1008
|
+
-i book.epub -o book-optimized.epub
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
**Later runs:**
|
|
1012
|
+
|
|
1013
|
+
```bash
|
|
1014
|
+
docker compose run --rm optimizer \
|
|
1015
|
+
-i book.epub -o book-optimized.epub
|
|
1016
|
+
```
|
|
1017
|
+
|
|
1018
|
+
**Custom image quality:**
|
|
1019
|
+
|
|
1020
|
+
```bash
|
|
1021
|
+
docker compose run --rm optimizer \
|
|
1022
|
+
-i book.epub -o book-optimized.epub \
|
|
1023
|
+
--jpg-quality 50 --png-quality 0.4
|
|
1024
|
+
```
|
|
1025
|
+
|
|
1026
|
+
**Complete author workflow:**
|
|
1027
|
+
|
|
1028
|
+
```bash
|
|
1029
|
+
docker compose run --rm optimizer \
|
|
1030
|
+
-i book.epub -o book-optimized.epub \
|
|
1031
|
+
--preset author --clean
|
|
1032
|
+
```
|
|
1033
|
+
|
|
1034
|
+
**Author workflow with an optional configuration file:**
|
|
1035
|
+
|
|
1036
|
+
```bash
|
|
1037
|
+
docker compose run --rm optimizer \
|
|
1038
|
+
-i book.epub -o book-optimized.epub \
|
|
1039
|
+
--preset author --author-config author-workflow.json
|
|
1040
|
+
```
|
|
1041
|
+
|
|
1042
|
+
**Compose command breakdown:**
|
|
1043
|
+
|
|
1044
|
+
- `docker compose run` - Run the `optimizer` service defined in `compose.yaml`
|
|
1045
|
+
- `--build` - Build or refresh the image before running; needed on the first run and after code updates
|
|
1046
|
+
- `--rm` - Remove the stopped one-off container after processing
|
|
1047
|
+
- Arguments after `optimizer` are passed directly to the EPUB optimizer
|
|
1048
|
+
|
|
1049
|
+
By default, Compose shares the repository directory with the container. Place EPUB files in that directory and use their normal relative names. Temporary files also appear there unless `--clean` is used.
|
|
1050
|
+
|
|
1051
|
+
<details>
|
|
1052
|
+
<summary>Advanced: raw Docker</summary>
|
|
1053
|
+
|
|
1054
|
+
Raw `docker run` remains supported, but its bind-mount syntax varies by shell. The image now works from `/epub-files`, so the EPUB arguments themselves remain relative:
|
|
1055
|
+
|
|
1056
|
+
```bash
|
|
1057
|
+
docker build -t epub-optimizer .
|
|
1058
|
+
docker run --rm -v "$PWD:/epub-files" epub-optimizer \
|
|
1059
|
+
-i book.epub -o book-optimized.epub
|
|
1060
|
+
```
|
|
1061
|
+
|
|
1062
|
+
</details>
|
|
1063
|
+
|
|
1064
|
+
### Debugging with Temporary Files
|
|
1065
|
+
|
|
1066
|
+
**Traditional usage:**
|
|
1067
|
+
|
|
1068
|
+
```bash
|
|
1069
|
+
# Keep temp files for inspection (default behavior)
|
|
1070
|
+
pnpm optimize -i book.epub -o book-opt.epub
|
|
1071
|
+
# Inspect the temp_epub-<id>/ directory reported by the command
|
|
1072
|
+
|
|
1073
|
+
# Custom temp location
|
|
1074
|
+
pnpm optimize -i book.epub -o book-opt.epub -t my-debug-folder
|
|
1075
|
+
|
|
1076
|
+
# Clean up when done
|
|
1077
|
+
pnpm cleanup my-debug-folder
|
|
1078
|
+
```
|
|
1079
|
+
|
|
1080
|
+
**Docker usage:**
|
|
1081
|
+
|
|
1082
|
+
```bash
|
|
1083
|
+
# Temp files automatically appear in your current directory
|
|
1084
|
+
docker compose run --rm optimizer \
|
|
1085
|
+
-i book.epub -o book-optimized.epub
|
|
1086
|
+
# Inspect the reported temp_epub-<id>/ directory on your host
|
|
1087
|
+
|
|
1088
|
+
# Custom temp location (still visible on host)
|
|
1089
|
+
docker compose run --rm optimizer \
|
|
1090
|
+
-i book.epub -o book-optimized.epub \
|
|
1091
|
+
-t my-debug-folder
|
|
1092
|
+
|
|
1093
|
+
# Clean temp files after processing
|
|
1094
|
+
docker compose run --rm optimizer \
|
|
1095
|
+
-i book.epub -o book-optimized.epub \
|
|
1096
|
+
--clean
|
|
1097
|
+
```
|
|
1098
|
+
|
|
1099
|
+
**What's in the temp directory?**
|
|
1100
|
+
|
|
1101
|
+
- Extracted EPUB structure
|
|
1102
|
+
- Processed HTML/CSS/JavaScript files
|
|
1103
|
+
- Optimized images
|
|
1104
|
+
- Modified fonts only when `--fonts` is enabled
|
|
1105
|
+
- All intermediate processing artifacts
|
|
1106
|
+
|
|
1107
|
+
This is invaluable for debugging optimization issues or understanding what the tool does to your EPUB.
|
|
1108
|
+
|
|
1109
|
+
Failures before successful cleanup preserve temporary files even when `--clean` was requested. This keeps the failed processing state available for debugging. A later report-writing failure can occur after cleanup and publication have already succeeded.
|
|
1110
|
+
|
|
1111
|
+
## Project Structure
|
|
1112
|
+
|
|
1113
|
+
```
|
|
1114
|
+
epub-optimizer/
|
|
1115
|
+
├── .github/workflows/
|
|
1116
|
+
│ ├── ci.yml # Audit, shared quality/Docker checks, and Required CI gate
|
|
1117
|
+
│ ├── docker-check.yml # Shared native AMD64/ARM64 Docker E2E and image scans
|
|
1118
|
+
│ ├── epubcheck-watch.yml # Weekly check for new EPUBCheck releases (opens an issue)
|
|
1119
|
+
│ ├── npm-publish.yml # Manual npm publication of a verified stable release
|
|
1120
|
+
│ ├── prepare-release.yml # Prepare and merge a checked release PR to main
|
|
1121
|
+
│ ├── quality.yml # Shared Node 22/24, coverage, lint, types, and EPUBCheck checks
|
|
1122
|
+
│ ├── release-published.yml # Promote tested image digests and sync main into develop by PR
|
|
1123
|
+
│ ├── release.yml # Validate candidates, tag the source, and create a release draft
|
|
1124
|
+
│ └── security-audit.yml # Required production audit, also available weekly/manually
|
|
1125
|
+
├── compose.yaml # Recommended cross-platform Docker interface
|
|
1126
|
+
├── Dockerfile # Multi-stage production container image
|
|
1127
|
+
├── docker-entrypoint.sh # Docker defaults and CLI entrypoint
|
|
1128
|
+
├── docs/RELEASING.md # Release preparation, publication, diagnostics, and recovery
|
|
1129
|
+
├── dist/ # Compiled JavaScript (production code)
|
|
1130
|
+
├── package.json # Package configuration
|
|
1131
|
+
├── README.md # Documentation
|
|
1132
|
+
├── tsconfig.json # TypeScript configuration
|
|
1133
|
+
├── vitest.config.ts # Test configuration
|
|
1134
|
+
├── epubcheck/ # EPUBCheck for EPUB validation (not included in repo)
|
|
1135
|
+
├── scripts/ # Build and maintenance scripts
|
|
1136
|
+
│ ├── benchmark-batch.ts # Generated-book concurrency and Node RSS comparison
|
|
1137
|
+
│ ├── setup-watch.sh # Non-overwriting NAS/host directory and env setup
|
|
1138
|
+
│ ├── clean-path.ts # Guarded cleanup helper used instead of rm -rf
|
|
1139
|
+
│ ├── docker-e2e.ts # Docker fixture test runner
|
|
1140
|
+
│ ├── e2e-epubcheck.ts # Local fixture test runner with EPUBCheck
|
|
1141
|
+
│ ├── install-epubcheck.sh # Single source of the EPUBCheck version (CI + Docker)
|
|
1142
|
+
│ ├── minify-dist.ts # Optional legacy minification, outside the default build
|
|
1143
|
+
│ ├── package-e2e.ts # Installed tarball, CLI/API and TypeScript checks
|
|
1144
|
+
│ ├── prepare-release.ts # Prepare and validate a release PR (pnpm release:prepare)
|
|
1145
|
+
│ └── validate-release.ts # Release version/tag/ref guards (pnpm release:check)
|
|
1146
|
+
└── src/ # Source code directory
|
|
1147
|
+
├── index.ts # Validated Node.js API entry point
|
|
1148
|
+
├── watch.ts # Stable-file polling, snapshots and durable completion records
|
|
1149
|
+
├── batch.ts # Sequential batch planning, reports and execution
|
|
1150
|
+
├── optimization.ts # Shared API/CLI validation and publication pipeline
|
|
1151
|
+
├── pipeline.ts # CLI entry point for the shared optimization pipeline
|
|
1152
|
+
├── cli.ts # yargs-based argument parser
|
|
1153
|
+
├── types.ts # TypeScript type definitions
|
|
1154
|
+
├── types.d.ts # Additional TypeScript declarations
|
|
1155
|
+
├── processors/ # Content-processing modules
|
|
1156
|
+
│ ├── epub-processor.ts # Extraction and content processing without publication
|
|
1157
|
+
│ ├── archive-processor.ts # EPUB extraction/compression (zip-slip safe)
|
|
1158
|
+
│ ├── html-processor.ts # HTML/CSS processing
|
|
1159
|
+
│ ├── js-processor.ts # JavaScript minification
|
|
1160
|
+
│ ├── svg-optimizer.ts # SVG optimization
|
|
1161
|
+
│ ├── lazy-img.ts # Add loading="lazy" to <img>
|
|
1162
|
+
│ ├── font-processor.ts # Optional font subsetting
|
|
1163
|
+
│ ├── image-converter.ts # PNG → JPEG conversion (parallel)
|
|
1164
|
+
│ └── image-processor.ts # Resize + re-encode (single-pass, parallel)
|
|
1165
|
+
├── scripts/ # Post-processing steps (exported run(opts) fns)
|
|
1166
|
+
│ ├── create-epub.ts # Package the final EPUB archive
|
|
1167
|
+
│ ├── validate-epub.ts # Run EPUBCheck via java
|
|
1168
|
+
│ ├── utils.ts # Shared helpers (RunOpts, isEntryPoint, …)
|
|
1169
|
+
│ ├── fix/ # General XHTML fix scripts (modular)
|
|
1170
|
+
│ │ ├── fix-span-tags.ts
|
|
1171
|
+
│ │ ├── fix-xml.ts
|
|
1172
|
+
│ │ ├── remove-empty-styles.ts
|
|
1173
|
+
│ │ ├── xhtml-files.ts # Shared recursive and manifest-based repair discovery
|
|
1174
|
+
│ │ └── index.ts # runFixes(): call every fix in sequence
|
|
1175
|
+
│ └── ops/ # EPUB structure modifications
|
|
1176
|
+
│ ├── add-cover-image-property.ts
|
|
1177
|
+
│ ├── update-cover-linear.ts
|
|
1178
|
+
│ ├── update-toc-with-cover.ts
|
|
1179
|
+
│ ├── update-summary-page.ts
|
|
1180
|
+
│ ├── add-chapter-sections-to-toc.ts
|
|
1181
|
+
│ └── update-structure.ts # runStructureUpdates(): call every op
|
|
1182
|
+
└── utils/ # Utility modules
|
|
1183
|
+
├── config.ts # Application configuration
|
|
1184
|
+
├── epub-utils.ts # OPF / TOC file discovery
|
|
1185
|
+
├── output-transaction.ts # Candidate output commit/rollback
|
|
1186
|
+
├── author-workflow-config.ts # Optional validated author workflow mapping
|
|
1187
|
+
├── epub-doctor.ts # Non-destructive EPUB inspection and optimization-risk summary
|
|
1188
|
+
├── epub-integrity.ts # Before/after content metrics and regression checks
|
|
1189
|
+
├── epub-text.ts # Before/after text preservation check
|
|
1190
|
+
├── path-safety.ts # Safe path resolution inside the extracted EPUB
|
|
1191
|
+
├── pipeline-options.ts # Preset resolution
|
|
1192
|
+
├── run-report.ts # Structured results and profiling
|
|
1193
|
+
└── i18n.ts # Localized label lookup
|
|
1194
|
+
```
|
|
1195
|
+
|
|
1196
|
+
Note: Test files are colocated with their respective source files but omitted from this structure for clarity.
|
|
1197
|
+
|
|
1198
|
+
## Development Information
|
|
1199
|
+
|
|
1200
|
+
This project is built with TypeScript and uses modern ESM modules. Here's how the development workflow works:
|
|
1201
|
+
|
|
1202
|
+
### Source and Build Separation
|
|
1203
|
+
|
|
1204
|
+
- TypeScript source files are in the `src/` and `scripts/` directories
|
|
1205
|
+
- The compiled JavaScript output goes to the `dist/` directory
|
|
1206
|
+
- The production build contains:
|
|
1207
|
+
- Test files (`*.test.ts`) are completely excluded
|
|
1208
|
+
- TypeScript declaration files (`*.d.ts`) for the API and its referenced types
|
|
1209
|
+
- No source maps are included
|
|
1210
|
+
- Readable JavaScript with comments and function names, so stack locations point to inspectable code
|
|
1211
|
+
- You must run `pnpm build` before running any `optimize` commands
|
|
1212
|
+
|
|
1213
|
+
### Import Structure
|
|
1214
|
+
|
|
1215
|
+
- The project uses TypeScript's `NodeNext` module resolution, which mirrors Node.js ESM semantics exactly
|
|
1216
|
+
- Relative imports must include the explicit `.js` extension in source files (e.g., `import { foo } from './foo.js'`), matching what Node.js resolves at runtime
|
|
1217
|
+
- Type-only imports use `import type` (enforced by `verbatimModuleSyntax`) so they are fully erased at build time
|
|
1218
|
+
|
|
1219
|
+
### Testing
|
|
1220
|
+
|
|
1221
|
+
See the [validation corpus and manual reading-system checklist](docs/VALIDATION.md) for what automated checks cover and what still requires visual review.
|
|
1222
|
+
|
|
1223
|
+
- Test files are colocated with the source files they test (e.g., `src/index.ts` and `src/index.test.ts`)
|
|
1224
|
+
- This "side-by-side" approach has several advantages:
|
|
1225
|
+
- Easy to locate tests for any module
|
|
1226
|
+
- Promotes keeping tests updated when changing implementation
|
|
1227
|
+
- Makes it clear which parts of the codebase are covered by tests
|
|
1228
|
+
- Simplifies relative imports between tests and implementation
|
|
1229
|
+
- Tests are written using Vitest, a modern test framework compatible with Jest syntax
|
|
1230
|
+
- Run tests with `pnpm test` (watch mode) or `pnpm test:run` (single run)
|
|
1231
|
+
- Run tests with coverage using `pnpm test:coverage`
|
|
1232
|
+
- `pnpm test:e2e` runs the public `optimize`, `optimize:clean`, `optimize:lossless`, `optimize:repair`, and `optimize:author` commands, including a configured author workflow
|
|
1233
|
+
- Every pnpm E2E output is checked for content integrity and validated with EPUBCheck; clean/keep-temp behavior and workflow-specific transformations are also verified
|
|
1234
|
+
- The author E2E fixture also checks before/after content integrity, sequential NCX navigation, and a maximum output-size ratio to catch compression regressions
|
|
1235
|
+
- `pnpm test:docker` validates raw Docker balanced/author/fonts plus Docker Compose balanced, lossless, repair, author, and configured-author workflows
|
|
1236
|
+
- Docker E2E verifies reports, content integrity, EPUBCheck output, workflow-specific transformations, and clean/keep-temp behavior
|
|
1237
|
+
- CI validates the project on Node.js 22 and 24, and runs the Docker E2E workflow on Node.js 24
|
|
1238
|
+
- Unit tests run in a Node.js environment and mock external dependencies where appropriate
|
|
1239
|
+
|
|
1240
|
+
### Production Validation
|
|
1241
|
+
|
|
1242
|
+
Use Node 24 LTS for development and release checks (`.node-version`); the minimum
|
|
1243
|
+
supported runtime is Node 22.12. Install with `pnpm install --frozen-lockfile`.
|
|
1244
|
+
CI validates Node 22 and 24, coverage, formatting, EPUBCheck and Docker workflows.
|
|
1245
|
+
The production dependency audit is required by every CI and release run, including
|
|
1246
|
+
release retries; it also runs weekly and can be started manually.
|
|
1247
|
+
Publishing a prerelease promotes its tested image to the version tag without changing `latest`.
|
|
1248
|
+
|
|
1249
|
+
### Release Validation
|
|
1250
|
+
|
|
1251
|
+
Run **Prepare release** from `main` in GitHub Actions with the next version after develop's CI passes. It prepares a PR, waits for **Required CI**, merges through branch rules and dispatches the shared validation and native AMD64/ARM64 image tests. **Release** creates a draft with the tested image digests; publishing that draft promotes the same images to their version tag and, for stable releases, `latest`, then synchronizes develop. Release retries preserve existing notes and reuse recorded images.
|
|
1252
|
+
|
|
1253
|
+
The local `pnpm release:prepare X.Y.Z` command uses the same PR flow with authenticated Git/GitHub CLI access; follow it by dispatching **Release** from main with the printed `expected-sha`. Configure the dedicated repository-scoped release App before using the automated workflows. See [Releases and CI](docs/RELEASING.md) for setup, diagnostics, scan policy, prereleases and recovery.
|
|
1254
|
+
|
|
1255
|
+
Before creating a release manually, run the CI quality gates plus coverage, audit, and local Docker checks:
|
|
1256
|
+
|
|
1257
|
+
```bash
|
|
1258
|
+
pnpm release:check --version "$(node -p 'require("./package.json").version')"
|
|
1259
|
+
pnpm lint
|
|
1260
|
+
pnpm format:check
|
|
1261
|
+
pnpm build
|
|
1262
|
+
pnpm test:run
|
|
1263
|
+
pnpm test:coverage
|
|
1264
|
+
pnpm audit --prod
|
|
1265
|
+
pnpm test:e2e
|
|
1266
|
+
docker compose config --quiet
|
|
1267
|
+
docker compose build
|
|
1268
|
+
pnpm test:docker
|
|
1269
|
+
```
|
|
1270
|
+
|
|
1271
|
+
Also validate a representative real EPUB through both execution paths:
|
|
1272
|
+
|
|
1273
|
+
```bash
|
|
1274
|
+
pnpm optimize:author -i YourBook.epub -o YourBook-pnpm.epub \
|
|
1275
|
+
--strict --report-json reports/YourBook-pnpm-report.json
|
|
1276
|
+
|
|
1277
|
+
docker compose run --rm optimizer \
|
|
1278
|
+
-i YourBook.epub -o YourBook-docker.epub \
|
|
1279
|
+
--preset author --strict --report-json reports/YourBook-docker-report.json
|
|
1280
|
+
```
|
|
1281
|
+
|
|
1282
|
+
A successful release candidate exits with code 0, passes EPUBCheck without errors or warnings in strict mode, and produces reports with `"success": true` and `"content.integrity.valid": true`.
|
|
1283
|
+
|
|
1284
|
+
### Development and Production
|
|
1285
|
+
|
|
1286
|
+
- For development: Make changes to TypeScript files and run `pnpm build:dev`
|
|
1287
|
+
- For production: Run `pnpm build` to create readable JavaScript and API declarations in `dist/`
|
|
1288
|
+
- The `optimize` commands run against the compiled code in `dist/`
|
|
1289
|
+
|
|
1290
|
+
## Linting and Formatting
|
|
1291
|
+
|
|
1292
|
+
- **Linting:**
|
|
1293
|
+
- `pnpm run lint` lints all TypeScript files in the `src` and `scripts` directories.
|
|
1294
|
+
- `pnpm run lint:fix` does the same, but also auto-fixes issues where possible.
|
|
1295
|
+
- **Formatting:**
|
|
1296
|
+
- `pnpm run format` auto-formats all `.ts`, `.json`, and `.md` files in the project using Prettier.
|
|
1297
|
+
- `pnpm run format:check` checks formatting without making changes (useful for CI).
|
|
1298
|
+
|
|
1299
|
+
## Minification
|
|
1300
|
+
|
|
1301
|
+
JavaScript embedded inside EPUBs is still minified with Terser. The optimizer's own Node.js code is compiled without application minification so stack traces retain useful file/line/column locations. Source maps are not needed to read those JavaScript locations and are not shipped by the production build.
|
|
1302
|
+
|
|
1303
|
+
The legacy `pnpm minify:safe` command remains available as an explicit development-tool opt-in; it reduces readability and is not called by `pnpm build`. `pnpm pack` runs a clean build through `prepack`, regenerating readable JavaScript and declarations before creating the tarball.
|
|
1304
|
+
|
|
1305
|
+
## Modular Fix Scripts
|
|
1306
|
+
|
|
1307
|
+
Every processing step is a plain async function. `pipeline.ts` runs them all in-process (no sub-process spawns), forwarding a shared `{ tempDir, lang, output }` down the chain.
|
|
1308
|
+
|
|
1309
|
+
- **Explicit repair passes** (span tags, XML/XHTML sanity, empty styles) live in `src/scripts/fix/`. They run only with `--repair` or the author workflow because they intentionally modify document structure.
|
|
1310
|
+
- **Author workflow structure modifications** (cover linear, TOC, summary page, chapter sections) live in `src/scripts/ops/`. `src/scripts/ops/update-structure.ts` exports `runStructureUpdates(opts)`, and the pipeline runs it only when `--author-workflow` is enabled.
|
|
1311
|
+
- To enable/disable a step, comment or uncomment the corresponding `await …(opts)` call in the matching orchestrator file.
|
|
1312
|
+
- To add a new step, create a script that exports `async run(opts: RunOpts)` and add an `await run(opts)` line in the orchestrator.
|
|
1313
|
+
- Each leaf script also auto-runs when executed directly (`node dist/src/scripts/fix/fix-xml.js`); the `isEntryPoint()` guard keeps that from firing on import.
|
|
1314
|
+
|
|
1315
|
+
### Customizing the Optimization Process
|
|
1316
|
+
|
|
1317
|
+
Use the default generic optimizer for EPUBs that do not need the project author's Pages/manual-summary workflow:
|
|
1318
|
+
|
|
1319
|
+
```bash
|
|
1320
|
+
pnpm optimize -i book.epub -o book-opt.epub
|
|
1321
|
+
```
|
|
1322
|
+
|
|
1323
|
+
If you do want the complete Pages/manual-summary workflow, enable the author preset:
|
|
1324
|
+
|
|
1325
|
+
```bash
|
|
1326
|
+
pnpm optimize:author -i YourBook.epub -o YourBook-optimized.epub
|
|
1327
|
+
# equivalent:
|
|
1328
|
+
pnpm optimize -i YourBook.epub -o YourBook-optimized.epub --preset author
|
|
1329
|
+
```
|
|
1330
|
+
|
|
1331
|
+
The existing project-author conventions remain the defaults. To adapt summary/cover paths or CSS classes without editing source code, use `--author-config author-workflow.json` as documented under [Optional Author Workflow Configuration](#optional-author-workflow-configuration).
|
|
1332
|
+
|
|
1333
|
+
For deeper source-level customization, there are two levels of granularity:
|
|
1334
|
+
|
|
1335
|
+
**Wholesale (in `src/pipeline.ts`)** — comment out an entire step to disable a whole group at once:
|
|
1336
|
+
|
|
1337
|
+
- Omit `--repair` and the author preset to skip **all** modifying XHTML repair passes.
|
|
1338
|
+
- Omit `--author-workflow` / `--preset author` to skip **all** author workflow structure updates.
|
|
1339
|
+
- EPUBCheck validation is required for transactional publication. When a run fails, inspect the preserved temporary directory instead of disabling validation.
|
|
1340
|
+
|
|
1341
|
+
**Granular (in the orchestrator files)** — keep some steps, skip others:
|
|
1342
|
+
|
|
1343
|
+
- **Example: Skip adding cover to TOC** — comment out `await updateTocWithCover(opts)` in `src/scripts/ops/update-structure.ts`
|
|
1344
|
+
- **Example: Skip adding cover to summary page** — comment out `await updateSummaryPage(opts)` in the same file
|
|
1345
|
+
- **Example: Skip setting cover as first page** — comment out `await updateCoverLinear(opts)` in the same file
|
|
1346
|
+
- **Example: Skip chapter sections synchronization** — comment out `await addChapterSectionsToToc(opts)` in the same file (useful if you don't have subsections in your manual summary page)
|
|
1347
|
+
- **Example: Disable specific XML/HTML fixes** — comment out the relevant `await …(opts)` call in `src/scripts/fix/index.ts`
|
|
1348
|
+
|
|
1349
|
+
After making any customizations, rebuild the project with `pnpm build` to apply your changes.
|
|
1350
|
+
|
|
1351
|
+
## Common Issues
|
|
1352
|
+
|
|
1353
|
+
### Docker Issues
|
|
1354
|
+
|
|
1355
|
+
**"docker: command not found"**
|
|
1356
|
+
|
|
1357
|
+
- Install Docker first: [Download Docker](https://docs.docker.com/get-docker/)
|
|
1358
|
+
- Make sure Docker Desktop is running
|
|
1359
|
+
|
|
1360
|
+
**"`docker compose` is not a docker command"**
|
|
1361
|
+
|
|
1362
|
+
- Install Docker Compose v2 or update Docker Desktop
|
|
1363
|
+
- Verify with `docker compose version`
|
|
1364
|
+
|
|
1365
|
+
**"Permission denied" or file access errors**
|
|
1366
|
+
|
|
1367
|
+
- Ensure your EPUB file is in the project root or mounted directory
|
|
1368
|
+
- Check file permissions (especially on Linux/Mac)
|
|
1369
|
+
|
|
1370
|
+
### Traditional Installation Issues
|
|
1371
|
+
|
|
1372
|
+
**"Java not found" or EPUBCheck errors**
|
|
1373
|
+
|
|
1374
|
+
- Install Java JRE 17+ (see [Requirements](#requirements))
|
|
1375
|
+
- Download EPUBCheck manually (see [EPUBCheck Setup](#epubcheck-setup))
|
|
1376
|
+
|
|
1377
|
+
**"pnpm command not found"**
|
|
1378
|
+
|
|
1379
|
+
- Install pnpm: `npm install -g pnpm`
|
|
1380
|
+
|
|
1381
|
+
**Build errors**
|
|
1382
|
+
|
|
1383
|
+
- Run `pnpm build` before using `pnpm optimize`
|
|
1384
|
+
- Try `pnpm install` if dependencies are missing
|
|
1385
|
+
|
|
1386
|
+
### File Issues
|
|
1387
|
+
|
|
1388
|
+
**"Input file not found"**
|
|
1389
|
+
|
|
1390
|
+
- Check your file path and name (case-sensitive)
|
|
1391
|
+
- Make sure the EPUB file is in the correct directory
|
|
1392
|
+
- Use relative paths from the project root
|
|
1393
|
+
|
|
1394
|
+
**"Output file permission errors"**
|
|
1395
|
+
|
|
1396
|
+
- Check write permissions in the output directory
|
|
1397
|
+
- Try a different output filename
|
|
1398
|
+
|
|
1399
|
+
## Troubleshooting
|
|
1400
|
+
|
|
1401
|
+
If you encounter any issues, please check the [GitHub issues page](https://github.com/kiki-le-singe/epub-optimizer/issues) for existing issues or open a new one.
|
|
1402
|
+
|
|
1403
|
+
## Dependencies
|
|
1404
|
+
|
|
1405
|
+
This project uses the following dependencies:
|
|
1406
|
+
|
|
1407
|
+
### System Requirements
|
|
1408
|
+
|
|
1409
|
+
- Node.js 22.12 or higher (Node.js 24 for development)
|
|
1410
|
+
- Java Runtime Environment (JRE) 17 or higher (for EPUBCheck validation)
|
|
1411
|
+
- pnpm 10.30.3
|
|
1412
|
+
|
|
1413
|
+
### Key npm Packages
|
|
1414
|
+
|
|
1415
|
+
- **cheerio** - For robust HTML/XHTML DOM manipulation
|
|
1416
|
+
- **entities** - For resolving HTML named character references in XHTML
|
|
1417
|
+
- **clean-css** - For CSS minification
|
|
1418
|
+
- **fs-extra** - Enhanced file system operations
|
|
1419
|
+
- **html-minifier-terser** - For HTML minification
|
|
1420
|
+
- **sharp** - For image optimization (JPEG, PNG, WebP, GIF, AVIF, SVG)
|
|
1421
|
+
- **unzipper** - For extracting EPUB files
|
|
1422
|
+
- **yazl** - For cross-platform EPUB archive creation (pure JavaScript ZIP implementation)
|
|
1423
|
+
- **yargs** - For command-line argument parsing
|
|
1424
|
+
|
|
1425
|
+
### Development Dependencies
|
|
1426
|
+
|
|
1427
|
+
- **typescript** - For static typing and compilation
|
|
1428
|
+
- **terser** - For JavaScript minification
|
|
1429
|
+
- **vitest** - For testing
|
|
1430
|
+
- **typescript-eslint** - For linting TypeScript code
|
|
1431
|
+
- **prettier** - For code formatting
|
|
1432
|
+
|
|
1433
|
+
## License
|
|
1434
|
+
|
|
1435
|
+
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
|