joven-ebook-annotator 1.0.0b2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- joven_ebook_annotator-1.0.0b2/LICENSE +21 -0
- joven_ebook_annotator-1.0.0b2/PKG-INFO +490 -0
- joven_ebook_annotator-1.0.0b2/README.md +457 -0
- joven_ebook_annotator-1.0.0b2/pyproject.toml +69 -0
- joven_ebook_annotator-1.0.0b2/setup.cfg +4 -0
- joven_ebook_annotator-1.0.0b2/src/joven/__init__.py +8 -0
- joven_ebook_annotator-1.0.0b2/src/joven/cli.py +497 -0
- joven_ebook_annotator-1.0.0b2/src/joven/detect/__init__.py +1 -0
- joven_ebook_annotator-1.0.0b2/src/joven/detect/pipeline.py +292 -0
- joven_ebook_annotator-1.0.0b2/src/joven/detect/segment.py +119 -0
- joven_ebook_annotator-1.0.0b2/src/joven/detect/triage.py +188 -0
- joven_ebook_annotator-1.0.0b2/src/joven/dialogue.py +72 -0
- joven_ebook_annotator-1.0.0b2/src/joven/epub/__init__.py +1 -0
- joven_ebook_annotator-1.0.0b2/src/joven/epub/archive.py +236 -0
- joven_ebook_annotator-1.0.0b2/src/joven/epub/document.py +245 -0
- joven_ebook_annotator-1.0.0b2/src/joven/epub/package.py +79 -0
- joven_ebook_annotator-1.0.0b2/src/joven/kepub.py +68 -0
- joven_ebook_annotator-1.0.0b2/src/joven/model.py +290 -0
- joven_ebook_annotator-1.0.0b2/src/joven/render/__init__.py +108 -0
- joven_ebook_annotator-1.0.0b2/src/joven/render/annotate.py +286 -0
- joven_ebook_annotator-1.0.0b2/src/joven/render/kobo.py +126 -0
- joven_ebook_annotator-1.0.0b2/src/joven/render/upgrade.py +500 -0
- joven_ebook_annotator-1.0.0b2/src/joven/review.py +422 -0
- joven_ebook_annotator-1.0.0b2/src/joven/suspicion.py +109 -0
- joven_ebook_annotator-1.0.0b2/src/joven/trace.py +212 -0
- joven_ebook_annotator-1.0.0b2/src/joven/translate.py +426 -0
- joven_ebook_annotator-1.0.0b2/src/joven/verify.py +422 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/PKG-INFO +490 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/SOURCES.txt +31 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/dependency_links.txt +1 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/entry_points.txt +2 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/requires.txt +8 -0
- joven_ebook_annotator-1.0.0b2/src/joven_ebook_annotator.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Vlad Yanhursky
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,490 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: joven-ebook-annotator
|
|
3
|
+
Version: 1.0.0b2
|
|
4
|
+
Summary: Insert clickable translation footnotes for foreign-language passages in an EPUB
|
|
5
|
+
Author: Vlad Yanhursky
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/vyanhursky/joven
|
|
8
|
+
Project-URL: Repository, https://github.com/vyanhursky/joven
|
|
9
|
+
Project-URL: Issues, https://github.com/vyanhursky/joven/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/vyanhursky/joven/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: epub,kobo,ebook,translation,footnotes,annotation,kepub
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: End Users/Desktop
|
|
15
|
+
Classifier: Operating System :: MacOS
|
|
16
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Text Processing :: Markup :: XML
|
|
21
|
+
Classifier: Topic :: Utilities
|
|
22
|
+
Requires-Python: >=3.11
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Requires-Dist: lxml>=5.2
|
|
26
|
+
Requires-Dist: typer>=0.12
|
|
27
|
+
Requires-Dist: lingua-language-detector>=2.0
|
|
28
|
+
Requires-Dist: httpx>=0.27
|
|
29
|
+
Provides-Extra: dev
|
|
30
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
31
|
+
Requires-Dist: ruff>=0.5; extra == "dev"
|
|
32
|
+
Dynamic: license-file
|
|
33
|
+
|
|
34
|
+
# Joven — a Cormac McCarthy ebook Spanish annotator
|
|
35
|
+
|
|
36
|
+
[](https://github.com/vyanhursky/joven/actions/workflows/ci.yml)
|
|
37
|
+
[](https://www.python.org/downloads/)
|
|
38
|
+
[](LICENSE)
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
:@@# =@@+.
|
|
42
|
+
+@@@@ #@@@@.
|
|
43
|
+
.@@#@@ .*@@=@@.
|
|
44
|
+
#@@@@% #@@=@@#
|
|
45
|
+
@@:@@* :@@+:@@
|
|
46
|
+
@@=@@@ %@@-:@@
|
|
47
|
+
@@%@@@*. .@@=*+@@
|
|
48
|
+
@@@+@@@=. *@%+#%@@
|
|
49
|
+
*@@@@%@@%=. .@@*:-@@@
|
|
50
|
+
@@@====#@%=. .@@@@*@@@.
|
|
51
|
+
:#@@*::--#@@@#.%@@@@@: #@@=+**@@.
|
|
52
|
+
=@@#-*+@@@@@@@@@@@@@@% @@@%*#*@@.
|
|
53
|
+
-@@@@@@@@@@@-%@@@%..*@@@@@@-+@@@@@@%*+@@
|
|
54
|
+
+@@@@@@@@@@@@@@=:: ....:::::%@@@-@#*=:+#**@@@
|
|
55
|
+
=@@@@@@@*:.:::.:.::.:....... :@@@%@:**=:-%@@@+
|
|
56
|
+
.%@@@- :.. ..::: ....::::-=#@@@=:+=**#@%=
|
|
57
|
+
.. %@@@@@@+ :.:...=@@@@@@: .:--:**==%@@+%#%*#*-
|
|
58
|
+
.@@@@@@@@@@@+=- ::: :@@@@@@@+. :.:--#-*:@@@#%@+
|
|
59
|
+
+@@@@@@@@@@@@%. :+-*:. *@@=@@@@@.:=@@%:-*=%@@.
|
|
60
|
+
.#@@@@@@+@@+=--==:==-:: .:=+@@+%@%@@@@@@@=@*%+ @@%.
|
|
61
|
+
+@@@-===@-@==-----::::: .===*@@@@@@@+@%#=*%@%@#@@
|
|
62
|
+
%@@@@@%=-:=%@%:---==.-=-:. ..-++*%@@@@@:=*@@ @@@@@+@@
|
|
63
|
+
...%@@@@@=@#=..:-:=====-: :..:: .:-==+:+@@@*@@*+%@@%@@-@@
|
|
64
|
+
#@@@@%*@@*@*=:-:-**=-: ...:-==: .:=*:*+=+=:%@@@@=@@@@%
|
|
65
|
+
.#@@@@@:@*=@@===:.=+-. :::.:::. *@-+*:.. =@@@@= %@@@=
|
|
66
|
+
*@@@=%==@@@#+=@@@@@@@+.: : *@*%-:..:.@@@
|
|
67
|
+
@@@=*.:+:-@@@@@@@@%@@@%.:: .=@+@@:. :--#@@
|
|
68
|
+
@@@=@@%*@%@*=*@-@@##@@@%=:::: .:==-:::=@*+%-. .:=:.@@
|
|
69
|
+
%@@@@%-%%%@+@@@@@=*=-@@*=#=.:=+-:::.:-**+%+: .:=::@@
|
|
70
|
+
%@@@@@@=@*@@@@@@@@@@@*%=#@%=:-::-:-::*%*:..::::#@@@
|
|
71
|
+
.=@@@@@@+@@@%-@@@@*.:%#*.@=*-:=..:===- ::::-=*@@@@
|
|
72
|
+
#@@@@@@@@@@% :=+@@% :=+- .:-: : .::-+@@@@+
|
|
73
|
+
.#@@@@@@@@@. ..:=@@@@@@@@*
|
|
74
|
+
. %@@@@@@@@@@@%%##@@@@@@@@@@@@@@@%-
|
|
75
|
+
.#@@@@@@@@@@@@@@@@@@@@@#+=:
|
|
76
|
+
-%@@@@@@@+:.
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
> The boy sat down to read the book. He read of horses and of the country to the
|
|
80
|
+
> south and of men who spoke in a tongue he did not have.
|
|
81
|
+
> *Escúchame, joven,* said the old man, and the boy did not know what it was he
|
|
82
|
+
> was meant to hear. So he set down the book and took up the phone and fed the
|
|
83
|
+
> words into Google Translate one at a time like a man counting stones across a
|
|
84
|
+
> river, and when he looked up again the twilight had gone out of the valley and
|
|
85
|
+
> the page had gone cold and the old man was still standing there in the dark
|
|
86
|
+
> holding his counsel like a man holding a lamp for nobody.
|
|
87
|
+
>
|
|
88
|
+
> So the boy wrote a python service. It went through the book and marked out every
|
|
89
|
+
> passage that was not English and put the question to a small model that ran on
|
|
90
|
+
> his own machine and asked nothing of anyone and told no one what it had seen.
|
|
91
|
+
> Where the Spanish had been it set a single asterisk and no more than that, and
|
|
92
|
+
> under the asterisk it laid the English down like a coin under a tongue, and the
|
|
93
|
+
> boy could take it or leave it as he pleased. He read the book through and he
|
|
94
|
+
> never once opened Google Translate. The old man went on speaking in his own
|
|
95
|
+
> language as he had always done and as he would go on doing, and the boy
|
|
96
|
+
> understood him, and the sun went down bloodred over the mesa and he read on by
|
|
97
|
+
> the pale light of the device until the battery gave out.
|
|
98
|
+
|
|
99
|
+
**Joven is a command-line tool that finds the untranslated Spanish in an English
|
|
100
|
+
novel and inserts tappable translation footnotes.** You give it an EPUB you own;
|
|
101
|
+
it gives you back an annotated copy for your e-reader, with the prose byte-for-byte
|
|
102
|
+
unchanged. It was written for Cormac McCarthy's *The Crossing* and works well on
|
|
103
|
+
his other westerns.
|
|
104
|
+
|
|
105
|
+
Everything runs on your machine against a local model: no API key, no per-book
|
|
106
|
+
cost, nothing uploaded.
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## Why this exists
|
|
111
|
+
|
|
112
|
+
*The Crossing* is an English novel with a great deal of Spanish in it, and the book
|
|
113
|
+
never translates or contextualizes it. **726 Spanish sentences and phrases**, mixed
|
|
114
|
+
into English paragraphs. Reaching for a dictionary breaks the trance the prose spent
|
|
115
|
+
forty pages building; skimming past leaves a hole in the page.
|
|
116
|
+
|
|
117
|
+
Your e-reader's own Dictionary and Translate do not rescue you, because McCarthy
|
|
118
|
+
uses no quotation marks — speech and narration run together in one stream, and the
|
|
119
|
+
Spanish arrives in four distinct shapes:
|
|
120
|
+
|
|
121
|
+
```text
|
|
122
|
+
A Vaya con Dios. a whole paragraph, no English at all
|
|
123
|
+
B Cuántos años tienes? the old man said. Spanish speech, English dialogue tag
|
|
124
|
+
C The matríz will not help you, he said. English prose, Spanish loanword
|
|
125
|
+
D Escúchame, joven, the old man wheezed. Spanish opener, then English narration
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
C and D are why per-paragraph language detection fails in *both* directions: C is a
|
|
129
|
+
false positive waiting to happen, D is a guaranteed miss. Solving that is most of
|
|
130
|
+
what this tool is — see [DESIGN.md §2](DESIGN.md) for the measurements.
|
|
131
|
+
|
|
132
|
+
## What you get
|
|
133
|
+
|
|
134
|
+
A copy of your EPUB in which every Spanish passage carries a small `*`. Tap it and
|
|
135
|
+
the translation appears in the reader's own footnote popup; ignore it and the page
|
|
136
|
+
reads exactly as the author set it down.
|
|
137
|
+
|
|
138
|
+
<table>
|
|
139
|
+
<tr>
|
|
140
|
+
<td width="50%"><img src="docs/images/kobo-asterisk-marker.jpg" alt="A page of The Crossing on a Kobo, with a small asterisk after each Spanish line"></td>
|
|
141
|
+
<td width="50%"><img src="docs/images/kobo-footnote-preview.jpg" alt="The same page with Kobo's Footnote preview open, showing the English translation"></td>
|
|
142
|
+
</tr>
|
|
143
|
+
<tr>
|
|
144
|
+
<td><em>The page as McCarthy set it down, plus one asterisk per Spanish passage.</em></td>
|
|
145
|
+
<td><em>The same page one tap later — the Kobo's own <strong>Footnote preview</strong>.</em></td>
|
|
146
|
+
</tr>
|
|
147
|
+
</table>
|
|
148
|
+
|
|
149
|
+
- **Unintrusive by construction.** One asterisk. No inline brackets, no interlinear
|
|
150
|
+
clutter, no colour. The footnote is opt-in, the way McCarthy's silence is opt-in.
|
|
151
|
+
- **The prose is untouched.** Strip the inserted nodes from the output and what
|
|
152
|
+
remains is *byte-identical* to the original — enforced by a test, verified on the
|
|
153
|
+
real book.
|
|
154
|
+
- **It stays on your machine.** A local `qwen3:8b` does the translating — 73
|
|
155
|
+
minutes for the whole novel, $0, and the book never leaves the laptop.
|
|
156
|
+
- **Precision over recall.** A spurious footnote on `Go on.` is worse than a missed
|
|
157
|
+
one, so ambiguous cases escalate to the model rather than guess.
|
|
158
|
+
|
|
159
|
+
## How it works
|
|
160
|
+
|
|
161
|
+
Five commands, two of which do the real work. Detection is **two-tier**: a cheap
|
|
162
|
+
statistical pass judges every sentence, and only the fraction it cannot call is put
|
|
163
|
+
to the local LLM.
|
|
164
|
+
|
|
165
|
+
```mermaid
|
|
166
|
+
flowchart TD
|
|
167
|
+
START(["your book.epub"])
|
|
168
|
+
|
|
169
|
+
subgraph DETECT["joven detect · ~73 min · $0"]
|
|
170
|
+
direction TB
|
|
171
|
+
EXTRACT["extract text units<br>addressable by file, element path, offset"]
|
|
172
|
+
SEG["segment into sentences<br>character offsets preserved"]
|
|
173
|
+
T1{"TIER 1 — lingua triage<br>statistical, instant, free"}
|
|
174
|
+
T2{"TIER 2 — local LLM<br>qwen3:8b via Ollama, with surrounding context"}
|
|
175
|
+
MERGE["merge contiguous Spanish<br>one footnote per paragraph"]
|
|
176
|
+
EXTRACT --> SEG --> T1
|
|
177
|
+
T1 -- "abstains — 21% of segments" --> T2
|
|
178
|
+
T1 -- "confident Spanish" --> MERGE
|
|
179
|
+
T2 -- "yes, and here is the span" --> MERGE
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
SKIP["left alone<br>but still recorded in trace.jsonl"]
|
|
183
|
+
T1 -- "confident English" --> SKIP
|
|
184
|
+
T2 -- "not Spanish" --> SKIP
|
|
185
|
+
|
|
186
|
+
SIDECAR[("annotations.json<br>source of truth, human-editable")]
|
|
187
|
+
REVIEW["joven review<br>approve / edit / reject<br>decisions survive re-detection"]
|
|
188
|
+
MERGE --> SIDECAR
|
|
189
|
+
SIDECAR <--> REVIEW
|
|
190
|
+
|
|
191
|
+
subgraph RENDER["joven render · idempotent"]
|
|
192
|
+
direction TB
|
|
193
|
+
ANN["insert markers and note documents<br>EPUB 2 → 3 package upgrade"]
|
|
194
|
+
EPUB3(["book.annotated.epub"])
|
|
195
|
+
KEP["kepubify"]
|
|
196
|
+
ANN --> EPUB3 --> KEP
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
START --> EXTRACT
|
|
200
|
+
START -. "the original, never edited in place" .-> ANN
|
|
201
|
+
SIDECAR --> ANN
|
|
202
|
+
KEP --> OUT(["book.annotated.kepub.epub<br>copy this to the Kobo"])
|
|
203
|
+
EPUB3 -. "validated by" .-> VERIFY["joven verify<br>12 checks · epubcheck · text-preservation invariant"]
|
|
204
|
+
|
|
205
|
+
classDef artifact fill:#fff4e0,stroke:#b8860b,stroke-width:2px,color:#111
|
|
206
|
+
classDef endpoint fill:#e6f4ea,stroke:#2e7d4f,stroke-width:2px,color:#111
|
|
207
|
+
classDef muted fill:#f2f2f2,stroke:#999,color:#444
|
|
208
|
+
class SIDECAR artifact
|
|
209
|
+
class START,OUT endpoint
|
|
210
|
+
class SKIP muted
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
**The EPUB is never edited in place.** `annotations.json` is the durable artifact,
|
|
214
|
+
and rendering is a pure function of (original EPUB + sidecar). Corrections mean
|
|
215
|
+
editing the sidecar and re-rendering — never re-translating — and re-running
|
|
216
|
+
detection merges into your edits instead of clobbering them.
|
|
217
|
+
|
|
218
|
+
### Inside one Tier-2 call
|
|
219
|
+
|
|
220
|
+
Tier 2 is not a translation step with a question bolted on. It answers three
|
|
221
|
+
things at once — **is this Spanish**, **which part of it is Spanish**, and **what
|
|
222
|
+
does it mean** — which is why it is an instruct model and not a translation
|
|
223
|
+
engine. An NMT engine can only answer the third, and would cheerfully "translate"
|
|
224
|
+
`Yes mam.`
|
|
225
|
+
|
|
226
|
+
```mermaid
|
|
227
|
+
flowchart TB
|
|
228
|
+
subgraph REQUEST["one /api/chat request to Ollama"]
|
|
229
|
+
direction TB
|
|
230
|
+
SYS["SYSTEM<br>the task, the JSON contract, and a warning<br>that McCarthy's dialect English is English"]
|
|
231
|
+
FS["6 FEW-SHOT TURNS<br>one per mixing pattern, plus two dialect traps<br>added after a traced run exposed them"]
|
|
232
|
+
USR["USER<br>up to 400 characters of preceding prose, fenced<br>and marked do-not-translate, then the paragraph"]
|
|
233
|
+
SYS --- FS --- USR
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
USR --> MODEL["qwen3:8b · temperature 0 · thinking off<br>reply constrained to a JSON schema"]
|
|
237
|
+
MODEL --> OUT["is_spanish · spanish_text · translation"]
|
|
238
|
+
|
|
239
|
+
OUT --> G1{"is the translation<br>just the source tidied up?"}
|
|
240
|
+
G1 -- yes --> VETO["vetoed — a footnote<br>that teaches nothing"]
|
|
241
|
+
G1 -- no --> G2{"one Spanish word<br>inside English prose?"}
|
|
242
|
+
G2 -- yes --> LOAN["suppressed —<br>embedded loanword"]
|
|
243
|
+
G2 -- no --> KEEP["annotation, span narrowed<br>to spanish_text"]
|
|
244
|
+
|
|
245
|
+
VETO --> TRACE[("trace.jsonl<br>every outcome, with the raw reply")]
|
|
246
|
+
LOAN --> TRACE
|
|
247
|
+
KEEP --> TRACE
|
|
248
|
+
|
|
249
|
+
classDef keep fill:#e6f4ea,stroke:#2e7d4f,stroke-width:2px,color:#111
|
|
250
|
+
classDef drop fill:#f2f2f2,stroke:#999,color:#444
|
|
251
|
+
classDef store fill:#fff4e0,stroke:#b8860b,stroke-width:2px,color:#111
|
|
252
|
+
class KEEP keep
|
|
253
|
+
class VETO,LOAN drop
|
|
254
|
+
class TRACE store
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
**There are two prompts, not one**, and which runs depends on what Tier 1 already
|
|
258
|
+
decided. The split was forced by a bug in each direction:
|
|
259
|
+
|
|
260
|
+
| Tier 1 said | Prompt | Because |
|
|
261
|
+
|---|---|---|
|
|
262
|
+
| abstained (the band) | **adjudicate** — is this Spanish, and if so which part? | The genuine open question |
|
|
263
|
+
| confident Spanish | **translate only** — this *is* Spanish; do not second-guess it | A combined prompt let the model veto Tier 1's correct calls. It labels `Dieciseis.` English. |
|
|
264
|
+
|
|
265
|
+
Skipping the model entirely for Tier-1 accepts was the first implementation, and
|
|
266
|
+
it shipped annotations with **no translation at all** — blank popups on the
|
|
267
|
+
device.
|
|
268
|
+
|
|
269
|
+
The prompts themselves, the context-bleed problem, a real call from the trace, and
|
|
270
|
+
the three gates in detail: [docs/anatomy-of-a-call.md](docs/anatomy-of-a-call.md).
|
|
271
|
+
|
|
272
|
+
### The components
|
|
273
|
+
|
|
274
|
+
`src/joven/` is about 4,500 lines. The pieces map onto the pipeline above:
|
|
275
|
+
|
|
276
|
+
| Module | Job |
|
|
277
|
+
|---|---|
|
|
278
|
+
| [`epub/`](src/joven/epub) | Byte-level zip surgery — copy every archive entry verbatim, reserialize only the XHTML that actually changed |
|
|
279
|
+
| [`detect/`](src/joven/detect) | `segment.py` splits paragraphs into sentences; `triage.py` is Tier 1 (`lingua` + tag-stripping); `pipeline.py` runs both tiers and merges contiguous Spanish into one footnote per paragraph |
|
|
280
|
+
| [`translate.py`](src/joven/translate.py) | Tier 2 — the Ollama client, the `Translator` protocol, and the similarity veto that suppresses no-op "translations" |
|
|
281
|
+
| [`dialogue.py`](src/joven/dialogue.py) | McCarthy's `, he said.` vocabulary, shared by every gate that must not measure it |
|
|
282
|
+
| [`model.py`](src/joven/model.py) | The `annotations.json` sidecar — content-hash IDs, statuses, merge semantics |
|
|
283
|
+
| [`render/`](src/joven/render) | Marker insertion, the EPUB 2→3 package upgrade, one-note-per-file footnote documents |
|
|
284
|
+
| [`kepub.py`](src/joven/kepub.py) | The `kepubify` hand-off that produces the file the Kobo wants |
|
|
285
|
+
| [`review.py`](src/joven/review.py), [`suspicion.py`](src/joven/suspicion.py) | The localhost review UI, and the heuristic that sorts likely-wrong translations to the top of it |
|
|
286
|
+
| [`verify.py`](src/joven/verify.py) | The 12-check integrity gate, including the text-preservation invariant |
|
|
287
|
+
| [`trace.py`](src/joven/trace.py) | One JSONL record per segment, annotated or not, so any missing footnote can be explained afterwards |
|
|
288
|
+
|
|
289
|
+
## Requirements
|
|
290
|
+
|
|
291
|
+
| | |
|
|
292
|
+
|---|---|
|
|
293
|
+
| **Platform** | **macOS on Apple Silicon** — where this was developed and device-verified. CI also runs the suite on Linux. Windows is untested. |
|
|
294
|
+
| **Memory** | 16 GB, which is what sets the model ceiling at 7–14B parameters |
|
|
295
|
+
| **Disk** | ~6 GB for the model, plus room for the outputs |
|
|
296
|
+
| **Python** | 3.11+ (developed on 3.13) |
|
|
297
|
+
| **Homebrew** | for the three external binaries below |
|
|
298
|
+
| **Java** | `epubcheck` is a JAR and needs a JVM; macOS ships one at `/usr/bin/java` |
|
|
299
|
+
| **Reader** | A **Kobo** (sideloaded over USB) is the verified target — tested on firmware `4.45.23697`. Apple Books renders the footnotes as popups too. |
|
|
300
|
+
|
|
301
|
+
Three external tools do work Python should not:
|
|
302
|
+
|
|
303
|
+
| Tool | Why |
|
|
304
|
+
|---|---|
|
|
305
|
+
| [`ollama`](https://ollama.com) | Runs `qwen3:8b` (5.2 GB) locally. Chosen on a benchmark, not a hunch — [docs/model-selection.md](docs/model-selection.md). |
|
|
306
|
+
| [`kepubify`](https://github.com/pgaskin/kepubify) | Converts the finished EPUB into Kobo's own KEPUB flavour, which renders better on device |
|
|
307
|
+
| [`epubcheck`](https://www.w3.org/publishing/epubcheck/) | The reference EPUB validator — the external gate on our output |
|
|
308
|
+
|
|
309
|
+
**Bring your own EPUB.** Joven reads a file you already have; it does not fetch,
|
|
310
|
+
share, or unlock anything, and it refuses DRM-protected files outright. Books that
|
|
311
|
+
merely *obfuscate* an embedded font are read normally — that is not DRM, and the
|
|
312
|
+
scrambled font is carried through untouched.
|
|
313
|
+
|
|
314
|
+
## Install
|
|
315
|
+
|
|
316
|
+
The three external binaries first:
|
|
317
|
+
|
|
318
|
+
```bash
|
|
319
|
+
brew install epubcheck kepubify ollama
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
Then the tool, into its own isolated environment:
|
|
323
|
+
|
|
324
|
+
```bash
|
|
325
|
+
uv tool install joven-ebook-annotator
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
`pipx install joven-ebook-annotator` does the same thing if you have pipx instead.
|
|
329
|
+
Both pull prebuilt wheels — worth knowing that `lingua` carries its language models
|
|
330
|
+
inside the wheel, so this step moves about 170 MB.
|
|
331
|
+
|
|
332
|
+
Then the model, once:
|
|
333
|
+
|
|
334
|
+
```bash
|
|
335
|
+
ollama serve & # leave running
|
|
336
|
+
ollama pull qwen3:8b # 5.2 GB
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
> There is no Homebrew formula. One dependency (`lingua-language-detector`)
|
|
340
|
+
> publishes no source distribution at all — only per-platform wheels — which a
|
|
341
|
+
> Homebrew Python formula cannot consume without hand-pinned wheel URLs per
|
|
342
|
+
> architecture and per CPython minor version. Two commands that work beat one
|
|
343
|
+
> command that breaks on the next `python@` bump.
|
|
344
|
+
|
|
345
|
+
To work on the code rather than use it, see [Development](#development).
|
|
346
|
+
|
|
347
|
+
## Use
|
|
348
|
+
|
|
349
|
+
One book, five commands, in order:
|
|
350
|
+
|
|
351
|
+
```bash
|
|
352
|
+
joven inspect book.epub # structure, DRM, word counts
|
|
353
|
+
joven detect book.epub -o annotations.json --trace trace.jsonl
|
|
354
|
+
joven review annotations.json --epub book.epub # triage, suspect passages first
|
|
355
|
+
joven render book.epub annotations.json -o out/ # EPUB 3 + KEPUB for the Kobo
|
|
356
|
+
joven verify out/book.annotated.epub --original book.epub
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
**`inspect`** reports what you are holding — EPUB version, DRM, spine, word counts.
|
|
360
|
+
Run it first; it fails fast on a file the rest of the pipeline cannot use.
|
|
361
|
+
|
|
362
|
+
**`detect`** is the slow step: about 73 minutes for a 150,000-word novel, single
|
|
363
|
+
threaded, $0. It writes `annotations.json` (the sidecar you will edit) and, with
|
|
364
|
+
`--trace`, a JSONL record of every segment it looked at. Re-running it is safe —
|
|
365
|
+
your review decisions are sticky.
|
|
366
|
+
|
|
367
|
+
The trace is also the run's recovery log: it is flushed a record at a time, so an
|
|
368
|
+
interrupted run loses nothing that the model already answered.
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
joven detect book.epub -o annotations.json --trace trace.jsonl --resume trace.jsonl
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
`--resume` reuses every model answer the trace already holds and pays only for the
|
|
375
|
+
segments the previous run never reached. Tier 1 and every suppression gate still run
|
|
376
|
+
over the whole book, so a resumed run picks up threshold changes rather than
|
|
377
|
+
replaying stale conclusions; recorded *errors* are retried rather than inherited.
|
|
378
|
+
|
|
379
|
+
**`review`** opens a local page listing every annotation with the surrounding prose
|
|
380
|
+
and the Spanish highlighted. Approve, edit, or reject (`a`/`r`/`e`); each decision
|
|
381
|
+
writes straight to the sidecar the moment you make it, so quitting mid-review loses
|
|
382
|
+
nothing. Suspect annotations sort first, each badged with the reason —
|
|
383
|
+
`untranslated: historic`, `garbled source: Conoc16` — and everything else follows in
|
|
384
|
+
book order.
|
|
385
|
+
|
|
386
|
+
**`render`** applies the sidecar to a fresh copy of the original and emits two
|
|
387
|
+
files: a spec-clean `.epub` (the verifiable intermediate, which is what `epubcheck`
|
|
388
|
+
validates) and a `.kepub.epub` (what you copy to the Kobo). It is idempotent and
|
|
389
|
+
never touches your source file.
|
|
390
|
+
|
|
391
|
+
**`verify`** runs the 12-check integrity gate — the text-preservation invariant,
|
|
392
|
+
`epubcheck`, noteref→footnote resolution, and the rest.
|
|
393
|
+
|
|
394
|
+
Then copy the `.kepub.epub` to the Kobo's root over USB, eject, and the device
|
|
395
|
+
imports it.
|
|
396
|
+
|
|
397
|
+
Missed a passage while reading?
|
|
398
|
+
|
|
399
|
+
```bash
|
|
400
|
+
joven add annotations.json --epub book.epub \
|
|
401
|
+
--find "Bueno pues" --translation "Well then"
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
Added entries are marked `edited`, so re-detection will never overwrite them.
|
|
405
|
+
|
|
406
|
+
For tuning, tracing, and the debug flags, see
|
|
407
|
+
[docs/troubleshooting.md](docs/troubleshooting.md).
|
|
408
|
+
|
|
409
|
+
## Results
|
|
410
|
+
|
|
411
|
+
Working end to end on a real book: lossless round-trip, two-tier detection with a
|
|
412
|
+
full decision trace, EPUB 2→3 upgrade, footnote rendering device-verified on a Kobo,
|
|
413
|
+
KEPUB output, a review pass, and a finished book on the device.
|
|
414
|
+
|
|
415
|
+
Last full run — Knopf's 1994 edition of *The Crossing*, 151,865 words:
|
|
416
|
+
|
|
417
|
+
| | |
|
|
418
|
+
|---|---|
|
|
419
|
+
| segments considered | 12,302 |
|
|
420
|
+
| escalated to the LLM | 2,556 (21%) |
|
|
421
|
+
| footnotes produced | 726 |
|
|
422
|
+
| wall clock / cost | 73 minutes / $0 |
|
|
423
|
+
| integrity checks | 12 of 12 passing |
|
|
424
|
+
|
|
425
|
+
Known limitations, and what the project has and has not proven, are in
|
|
426
|
+
[CHANGELOG.md](CHANGELOG.md).
|
|
427
|
+
|
|
428
|
+
## Development
|
|
429
|
+
|
|
430
|
+
```bash
|
|
431
|
+
git clone https://github.com/vyanhursky/joven && cd joven
|
|
432
|
+
python3.13 -m venv .venv
|
|
433
|
+
./.venv/bin/pip install -e '.[dev]'
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
```bash
|
|
437
|
+
./.venv/bin/pytest # 369 tests, synthetic fixtures only
|
|
438
|
+
./.venv/bin/ruff check src tests tools
|
|
439
|
+
|
|
440
|
+
# opt in to the real-book tests (the book is never committed)
|
|
441
|
+
JOVEN_TEST_EPUB=/path/to/book.epub ./.venv/bin/pytest
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
Tests never hit the network — a `StubTranslator` stands in for the LLM everywhere.
|
|
445
|
+
Two guard scripts run in CI, each of which has caught a real bug that survived a
|
|
446
|
+
careful manual read of the same files minutes earlier:
|
|
447
|
+
|
|
448
|
+
```bash
|
|
449
|
+
python tools/check_docs.py # every documented command and flag exists
|
|
450
|
+
python tools/check_no_book_content.py # no book text in tracked files
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
Model benchmarks are separate tools, not tests, because they need a running Ollama:
|
|
454
|
+
|
|
455
|
+
```bash
|
|
456
|
+
python tools/bench_models.py # the LLM in isolation
|
|
457
|
+
python tools/bench_pipeline.py # the two-tier system that actually ships
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
## Further reading
|
|
461
|
+
|
|
462
|
+
| | |
|
|
463
|
+
|---|---|
|
|
464
|
+
| [DESIGN.md](DESIGN.md) | Why the architecture is shaped this way — the measurements behind every decision, what the device tests overturned, and the work deliberately left undone |
|
|
465
|
+
| [docs/model-selection.md](docs/model-selection.md) | The local-model benchmark: why `qwen3:8b` |
|
|
466
|
+
| [docs/anatomy-of-a-call.md](docs/anatomy-of-a-call.md) | What the local model is asked, what it may answer, and the gates that check it |
|
|
467
|
+
| [docs/troubleshooting.md](docs/troubleshooting.md) | Tracing a missing footnote, improving translation quality, debug flags, and Kobo quirks |
|
|
468
|
+
| [docs/releasing.md](docs/releasing.md) | Cutting a release: the one-time PyPI trusted-publisher setup, and what CI does with a tag |
|
|
469
|
+
| [CHANGELOG.md](CHANGELOG.md) | Release notes and known limitations |
|
|
470
|
+
|
|
471
|
+
## Licence
|
|
472
|
+
|
|
473
|
+
The **code** is MIT — see [LICENSE](LICENSE). Use it, fork it, ship it.
|
|
474
|
+
|
|
475
|
+
The **books are not.** The annotated output is a derivative of a copyrighted work:
|
|
476
|
+
read it, don't distribute it.
|
|
477
|
+
|
|
478
|
+
**The repository deliberately tracks no book content.** The hazard is not the EPUB
|
|
479
|
+
but what the pipeline derives from it — a full decision trace holds every segment's
|
|
480
|
+
source text verbatim, which for a 150,000-word novel is the entire book. `.gitignore`
|
|
481
|
+
excludes those by pattern, and
|
|
482
|
+
[`tools/check_no_book_content.py`](tools/check_no_book_content.py) verifies it by
|
|
483
|
+
*content*, diffing every tracked file against the book itself:
|
|
484
|
+
|
|
485
|
+
```bash
|
|
486
|
+
python tools/check_no_book_content.py path/to/book.epub
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
Docs and tests quote short passages to illustrate the detection problem, and the two
|
|
490
|
+
screenshots above are photographs of the author's own copy on his own device.
|