waraq 0.0.0-stage → 0.2.0

package/LICENSE

@@ -0,0 +1,26 @@
+waraq
+Copyright (c) 2026 Muhammad Ahsan Ayaz. All rights reserved.
+
+waraq is free to use. You may install it and use it, for any purpose, at no cost.
+
+You may not:
+- copy, redistribute, sell or sublicense the software or any part of it;
+- modify it, or make works derived from it;
+- decompile, de-minify or otherwise reverse engineer it, except where the law
+  allows this despite this license.
+
+The agent skill files under agent/ may be copied into your own agent's
+configuration folders (which `waraq setup-agent` does for you) for use with
+waraq.
+
+The fonts under assets/fonts are not covered by this license. They are
+distributed under the SIL Open Font License 1.1; see the OFL files next to
+them.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY ARISING FROM THE
+USE OF THE SOFTWARE OR OF THE BOOKS IT PRODUCES.
+
+You are responsible for having the right to convert the books you convert.

package/README.md

@@ -1,3 +1,201 @@
-# Temporary Holding Version
+<p align="center">
+  <img src="https://cdn.jsdelivr.net/npm/waraq/assets/logo-readme.png" alt="waraq (ورق) logo" width="240" />
+</p>
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# waraq
+
+Turn a scanned Urdu, Arabic or English book into an EPUB that reflows on your e-reader.
+
+If you read Islamic or Urdu books, you probably have a folder of scanned PDFs...where every page is a photo of paper. On a Kindle that means pinch, zoom, scroll, repeat. Calibre and pandoc can't help here, because there is no text inside the file to convert. They wrap the page images and hand them back to you.
+
+I couldn't find a tool that does this properly for Urdu and Arabic..so I built one.
+
+waraq (ورق, "leaf" or "page") has a vision model read each page into structured data, and then builds the book with ordinary, deterministic code:
+
+`scan → page images → transcription → chapters you can edit → EPUB`
+
+Only the transcription step uses a model. Everything after it is plain code you can rerun for free.
+
+And it is made to be driven by your AI agent. A scan always needs judgement calls...is this heading a chapter, does this verse match the scan, which photo is the cover. Those are exactly what Claude Code, Codex or Antigravity are good at, and plain scripts aren't. So waraq ships a skill for your agent, and every command speaks JSON.
+
+## What you get
+
+- Text that reflows, right to left for Urdu and Arabic. Urdu books embed Amiri; Noto Nastaliq Urdu on request, see below
+- Footnotes linked to their references
+- Paragraphs joined back together across page breaks
+- Figures cropped out of the page and placed in the text
+- Chapters as Markdown files, so you can fix anything by hand
+- A `review.html` that puts every scan next to its transcription
+- Program code and prompt templates kept as code, line breaks and indentation intact
+- Quran verses checked against the canonical text, see below
+- An EPUB that is checked with epubcheck before the build is called done
+- An AZW3 for Kindle with real page numbers, taken from the printed ones
+
+## Where it is right now
+
+Pre-1.0 and honest about it. This is what I have actually run:
+
+- One full English book with Arabic quotations...239 pages, 31 chapters, about 62,000 words. It passes epubcheck and reads fine on my Kindle, boxed quotations, footnotes and Arabic lines included.
+- Two full tafsir sets, i.e. English with a lot of Arabic: Tafsir Ibn Kathir in 10 volumes (6,610 pages) and Tafseer as-Sa'di in 10 volumes (4,703 pages). All 20 are on my Kindle.
+- Three books that already had a text layer: a 742-page algorithms textbook with maths and C code, a 482-page book on AI agents that is half Python and prompt templates, and a 113-page card deck. waraq ignores the text layer and reads the page like any other scan, which is slower than copying the text but keeps the layout.
+- 12 pages of a 1914 Urdu grammar. Hand-written Nastaliq lithograph, faint in places, i.e. about as hard as these scans get. The page I compared by eye matched the scan, with the old spellings kept as printed.
+- Kindle and embedded fonts, tested on my Paperwhite: Nastaliq came out as isolated letters and digits on a blank page. Amiri shaped Urdu body text fine. Amiri and Noto Naskh Arabic both failed on Quran verses with harakat inside an English book, while Kindle's own Arabic font rendered them correctly. So `fonts: auto` (the default) embeds Amiri for an Urdu-language book and nothing else. `fonts: naskh`, `fonts: nastaliq` and `fonts: none` override it in `book.yaml`.
+- On Kindle, the end-of-ayah sign ۝ cannot hold a number: you get an empty circle with the digits beside it. waraq writes ayah numbers as (١٢) instead.
+- Only the `claude-cli` and `antigravity-cli` providers have been run for real. `anthropic`, `gemini` and `openai-compatible` are covered by tests only.
+- Tables and figure captions don't carry a language tag yet, so an Urdu table inside an English book loses its font.
+- Figure boxes from the model are approximate. A line of body text that lands inside the box is trimmed off again when it is clearly a text line; a wordmark under a small logo can get trimmed with it.
+- A word that is split across two pages gets rejoined without its hyphen. That is right for "inde-fatigable" and wrong for "Afrikaans-speaking". You get a warning for every one of them.
+
+## Requirements
+
+- Node 24 or later
+- `pdftoppm` (poppler): `sudo apt install poppler-utils` or `brew install poppler`
+- `pandoc`: `sudo apt install pandoc` or `brew install pandoc`
+- `epubcheck` (optional, validates the result): `sudo apt install epubcheck` or `brew install epubcheck`
+- A vision model...an API key, or a signed-in `claude` or `agy` CLI. See Providers below.
+- Calibre (optional, for `waraq kindle`): https://calibre-ebook.com/download
+
+`waraq doctor` checks all of these for you.
+
+## Use
+
+```bash
+npm i -g waraq
+waraq doctor          # what is installed, what is missing, and how to get it
+waraq setup-agent     # installs the waraq skill for Claude Code, Codex, Copilot, Cursor and Antigravity
+```
+
+Then open your agent in the folder with the PDF and ask it:
+
+> Convert book.pdf into a Kindle book with waraq.
+
+It runs a 12-page slice first and tells you the cost before doing the whole book. It checks the Quran verses, fixes the chapters, reviews the pages against the scans, and gives you an AZW3 with page numbers.
+
+I tested exactly that with a fresh agent that knew nothing about waraq except the skill...on a 68-page 1892 dissertation full of Arabic quotations. It got to a valid Kindle book on its own. Every place it got stuck went back into waraq or into the skill.
+
+### By hand
+
+The same steps, if you'd rather drive it yourself:
+
+```bash
+waraq init book.pdf --lang ur        # makes ./book/, rasterizes the pages
+cd book
+waraq transcribe --pages 1-12        # try a slice first; prints token use and an estimate for the whole book
+waraq transcribe                     # the rest; finished pages are cached and cost nothing to rerun
+waraq quran check                    # if the book quotes the Quran
+waraq assemble                       # writes chapters/*.md, images/, book.yaml, review.html
+waraq status                         # where the book stands, and the next command
+waraq build                          # writes book.epub
+```
+
+Or in one step: `waraq convert book.pdf --lang ur`. Fixes go through `waraq page set`, `waraq headings set` and `waraq quran set`, which check every change and log it in `edits.json`. Add `--json` to any command for machine-readable output.
+
+To get it on a Kindle:
+
+```bash
+waraq kindle                         # <title>.azw3 and <title>.apnx, with Calibre
+waraq kindle --to /media/$USER/Kindle/documents   # and copies both onto the Kindle
+```
+
+Why not just Send to Kindle..it strips embedded fonts (no "Publisher font" entry appears) and shows locations, not page numbers. A sideloaded AZW3 keeps the fonts, and the `.apnx` file next to it gives you page numbers that match the printed book. Copying an EPUB over USB does not work; Kindle ignores it.
+
+## Check the transcription
+
+This is the part I'd ask you not to skip.
+
+A language model can "correct" a passage toward the version it remembers. For Quran verses, hadith and quotations that is a real risk. The prompt forbids it, the model lists whatever it was unsure about, and `review.html` highlights every block that contains Arabic or Urdu...but the check is yours to make. Read the highlighted blocks against the scan before you share a book with anyone.
+
+For scale: on the 239-page book the model flagged 269 readings it was unsure about, and 21 footnotes had no reference mark it could find on the page. Nothing is dropped silently, but somebody has to look.
+
+## Providers
+
+Choose with `--provider` on `waraq init` or `waraq convert`. The choice is stored in `waraq.json`.
+
+| Provider | Needs | Default model |
+|---|---|---|
+| `anthropic` (default) | `ANTHROPIC_API_KEY` | `claude-fable-5-1` |
+| `claude-cli` | the `claude` CLI, signed in | `claude-fable-5-1` |
+| `antigravity-cli` | the `agy` CLI, signed in | `gemini-3.1-pro-high` |
+| `gemini` | `GEMINI_API_KEY` | none, pass `--model` |
+| `openai-compatible` | `OPENAI_API_KEY`, optional `WARAQ_OPENAI_BASE_URL` | none, pass `--model` |
+
+The two CLI providers use the CLI's own login, so no API key is needed. Each page is one CLI run, and the CLI adds its own system prompt to every run, so token counts are higher than with a direct API key. The 239-page book took about 3.67M input and 300k output tokens through the `claude` CLI.
+
+`openai-compatible` works with any server that has an OpenAI-style `/chat/completions` endpoint and accepts images, for example Ollama:
+
+```bash
+export WARAQ_OPENAI_BASE_URL=http://localhost:11434/v1
+waraq init book.pdf --provider openai-compatible --model <model-id>
+```
+
+Model, image size limit, output limit and concurrency can be changed in `waraq.json`.
+
+Which model..for clean print, Sonnet is enough and two to three times faster. I compared it with Opus on 12 tafsir pages: the English matched about 98%, each model got one word wrong that the other got right, and neither added a word to a verse. Where Arabic is mostly Quran, `quran-check` corrects the verses either way. I'd keep the larger model for hadith, which nothing can check afterwards.
+
+### When a provider refuses a page
+
+It happens. On that same book, 2 of the 239 pages were blocked by Claude's output filter...ordinary pages, as far as I could tell. You can send only those pages to another provider for one run:
+
+```bash
+waraq transcribe --pages 16,188 --provider antigravity-cli
+```
+
+Pages written this way are kept by later runs. Delete `pages/NNNN.json` to transcribe a page again.
+
+To make this automatic, name a fallback in `waraq.json`; a page the main provider refuses goes straight to it:
+
+```json
+"fallback": { "provider": "antigravity-cli", "model": "gemini-3.1-pro-high" }
+```
+
+One caveat I'd rather say out loud: `antigravity-cli` runs the `agy` agent without a tool restriction, because I couldn't find a read-only mode that still returned correct output. `claude-cli` is limited to reading the page image.
+
+## Quran verses: checked against the canonical text
+
+A tafsir is mostly Quran verses and hadith in Arabic, and a model reading small print can "complete" a verse from memory. For the verses there is a fix that needs no model at all:
+
+```bash
+waraq quran-check                    # after transcribe, before assemble
+```
+
+It downloads the Tanzil Uthmani text once (to `~/.cache/waraq`), compares every Arabic quotation with it, and rewrites the ones that match with the canonical wording, harakat included. Whatever is printed around the verse stays as printed: brackets, an ayah number carried over from the previous page, the "(سورة …)" reference. If a real word sits outside the matched verse, the quotation is not rewritten and goes to the report instead.
+
+It also checks Arabic quoted inside English sentences and footnotes, not only verse blocks. Whatever it leaves for a person shows up in `waraq quran review`, with the scan to open next to each item.
+
+Across all 20 tafsir volumes: 23,434 quotations matched and were rewritten. 138 matched only in part; those stay as printed and are listed in the report. About 3,200 matched nothing, nearly all of them hadith. The report is in `quran-check.json`. Hadith cannot be checked this way; there is no canonical text to check against.
+
+One thing I'd rather say out loud. An earlier version rewrote the whole block with only the matched span. Anything outside it was dropped, and on 20 blocks that was real verse text, for example everything after the 15th ayah of a long passage. An audit of every volume caught it, and the fix keeps everything outside the span. If you ran an older version, run `waraq quran-check` again: it starts over from the original transcription.
+
+## Which books should you convert?
+
+waraq is a format converter. It can't know where your PDF came from, and I'm not going to pretend it can. So this part is on you..and this is how I think about it for my own shelf:
+
+- **Public domain** (old Urdu classics, anything published before 1929): convert it, share it.
+- **Published for free distribution.** A lot of dawah literature carries an explicit permission to reproduce. Check the copyright page of the book in your hand, keep that page in the EPUB, and then convert and share.
+- **A book you own in print**, where the PDF is a convenience scan: convert it for your own reading. Don't pass the EPUB around.
+- **An in-copyright book you didn't pay for:** converting it doesn't make it yours. If a legal edition exists, buy that. If the book is out of print and can't be bought anywhere, that is a grey area and a personal call.
+
+The line I hold for myself...I don't distribute EPUBs made from the last two. A clean, reflowable EPUB is a much better pirated copy than a blurry scan, and the people who lose out are authors and small publishers.
+
+A few things in the tool follow from this:
+
+- `book.yaml` has `rights` and `source` fields. Fill them in; they are written into the EPUB metadata, so the book carries where it came from.
+- The copyright page of the original is transcribed like any other page. Leave it in.
+- There is no "download this book for me" feature, and there won't be one.
+- Every page is sent to the model provider you chose. For a book that is still in copyright, that is one more copy leaving your machine.
+
+I'm not a lawyer, and copyright law differs from country to country.
+
+## What it does not do
+
+MOBI output (use `waraq kindle`), PDF output, and local OCR without a model. A PDF that already has a text layer is read like a scan; waraq does not use the text layer.
+
+## Disclosure
+
+I architected waraq and made the decisions; Claude Code did most of the typing. The two CLI providers exist because that is how I ran it myself, on my own subscriptions.
+
+## License
+
+Free to use, not open source: you can use waraq for anything, but not redistribute or modify it. See `LICENSE`. The bundled fonts (Amiri, Noto Nastaliq Urdu) are under the SIL Open Font License 1.1.
+
+Looking forward for folks to try it on their own shelves :)

package/agent/skills/waraq/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: waraq
+description: Converts scanned or image-only PDF books (Urdu, Arabic, English) into reflowable EPUB and Kindle AZW3 with the waraq CLI, including checking Quran verses against the canonical text. Use when the user wants to turn a PDF book into an ebook, fix or finish a waraq conversion, check the Arabic in a converted book, or put a book on a Kindle.
+license: Proprietary. Free to use; see LICENSE in the waraq package.
+compatibility: Needs the waraq CLI (npm i -g waraq), Node 24+, poppler (pdftoppm) and pandoc. Calibre for Kindle output. A signed-in claude or agy CLI, or an API key, for transcription.
+metadata:
+  version: "0.2.0"
+---
+
+# Converting a book with waraq
+
+waraq does the mechanical work: page images, transcription by a vision model, assembly, EPUB, Kindle. You make the judgement calls a script cannot: whether a heading is a chapter, whether a verse matches the scan, what the cover is. Every change goes through a waraq command, never a hand edit of the JSON files.
+
+Add `--json` to any command when you need to read the result: one envelope `{ok, data}` or `{ok: false, error: {code, message, hint}}` on stdout. Exit codes: 0 ok, 1 error, 2 usage, 3 missing program, 4 pages failed, 5 epubcheck failed. Follow the `hint` of an error.
+
+## Workflow
+
+Copy this checklist and tick it off:
+
+```
+- [ ] 1. waraq doctor
+- [ ] 2. waraq init <book.pdf> --lang <en|ur|ar> [--provider ...] [--model ...]
+- [ ] 3. Trial slice: waraq transcribe --pages 1-12, then tell the user the cost estimate
+- [ ] 4. User agreed: waraq transcribe (repeat until no pages fail)
+- [ ] 5. If the book has Arabic: waraq quran check, then review (see below)
+- [ ] 6. waraq assemble, then read the warnings and review.html
+- [ ] 7. Fix structure: waraq headings, waraq page show / page set
+- [ ] 8. Fill in book.yaml (title, authors, cover, rights, source)
+- [ ] 9. waraq build
+- [ ] 10. waraq kindle [--to <Kindle documents folder>]
+```
+
+`waraq status` tells you where a project stands and gives the next command. Use it whenever you pick up a project.
+
+### 1. Doctor
+
+Run `waraq doctor`. If it reports a problem, tell the user what to install, and stop.
+
+### 2. Init
+
+- `--lang` is the main language of the book: `ur` for an Urdu book, `ar` for Arabic, `en` for English, even when the English has Arabic quotations.
+- Provider: `claude-cli` or `antigravity-cli` when the user has those CLIs signed in; otherwise an API key provider. When a second CLI is installed, init sets it as the fallback for pages the first refuses. Do not turn that off.
+- Model: see [references/models-and-cost.md](references/models-and-cost.md). For clean print in English, use `--model claude-sonnet-5-5`. When Arabic or Urdu is on most pages, the print is small or faint, or the book quotes hadith, use `--model claude-opus-5-5`.
+- A large scan that holds several volumes: `--pages 697-1320` rasterizes one volume.
+- init makes the project in a new folder (`./book` for `book.pdf`; `--out` to choose). Run the rest of the commands there, or pass `--project <dir>`.
+
+### 3–4. Transcribe
+
+- If Arabic or Urdu lines show up on more than about a third of the slice pages and you started on Sonnet, switch: `waraq transcribe --model claude-opus-5-5` for the rest. Pages written that way are kept.
+- Always run a trial slice first and report the estimate (`estimate for all N pages`). A full book costs millions of tokens. Ask the user before you transcribe the rest.
+- Transcription is cached per page; rerunning `waraq transcribe` only does missing or failed pages. Exit code 4 means some pages failed: run it again.
+- A page that both providers refuse: retry it alone with `waraq transcribe --pages N --provider <other>`.
+
+### 5. Quran verses
+
+Only for books with Arabic quotations. Run `waraq quran check`. It checks whole Arabic blocks, and Arabic quoted inside English paragraphs and footnotes (each inline quotation is a numbered `span`). It rewrites every quotation that matches the canonical Tanzil text and leaves the rest for you. Then follow [references/quran-review.md](references/quran-review.md) for every item in `waraq quran review`.
+
+### 6–7. Assemble and fix structure
+
+Run `waraq assemble`. On the first run it writes chapters, images, `book.yaml` and `review.html`. After that, use `waraq assemble --force` to rebuild the chapters from the pages; it does not touch `book.yaml`.
+
+Then:
+
+1. Read every warning. Heading demotions, removed watermarks, footnotes without a reference and joined hyphenated words are all listed there.
+2. Run `waraq headings`. Each level-1 heading becomes a chapter. Fix what is wrong with `waraq headings set <page> <block> <level>`.
+3. Open `review.html` and compare pages with their scans: every page with an `uncertain` entry that matters for meaning, every title or contents page, and at least one page in ten elsewhere. If the book is about its Arabic, check every Arabic-heavy page. Fix a misread with `waraq page show <n>`, then `waraq page set <n> <block> --text-file <file>`. Write the text to a file first: that is safer than shell quoting, above all for Arabic.
+4. Run `waraq assemble --force` after page edits.
+
+Details and common cases: [references/structure-fixes.md](references/structure-fixes.md).
+
+### 8. book.yaml
+
+`book.yaml` is the one file you edit directly. Check `title`, `authors` and `cover`. waraq guesses the cover from a photo of the book in the first pages; if there is no cover, leave the field out. For `source`, give where the PDF came from (for example an archive.org URL). For `rights`, state only what you can support: "Public domain (first published 1892)" when the book itself shows a publication date before 1929. Otherwise ask the user or leave it empty. See [references/rights.md](references/rights.md).
+
+### 9–10. Build and Kindle
+
+- `waraq build` validates with epubcheck. Exit code 5 means fix what it names, then build again.
+- `waraq kindle` writes `<title>.azw3` and `<title>.apnx` (page numbers). `--to <folder>` copies both to a Kindle's `documents` folder. See [references/kindle.md](references/kindle.md).
+
+## Rules
+
+- **Never write a Quran verse yourself, from memory or by typing.** Use `waraq quran set`, which writes canonical text only. For other Arabic (hadith, poetry, a misread word), change as little as you can: copy the correct letters from the existing text or from `quran canon` output, and say in `--note` what you changed and why.
+- **Never "correct" a variant reading** (qira'at). If the scan prints a reading that differs from Hafs, leave it and note it as `variant`.
+- **Hadith cannot be checked against a canonical text.** Compare it with the scan only.
+- **Ask before spending:** report the slice estimate before transcribing a whole book.
+- **Keep the copyright page** of the original in the book.
+- **Change pages only through waraq commands,** so every change is validated and logged in `edits.json`.
+- **Report honestly:** what you checked, what you could not check, and what is left for the user.

package/agent/skills/waraq/references/kindle.md

@@ -0,0 +1,25 @@
+# Kindle
+
+`waraq kindle` writes two files:
+
+- `<title>.azw3`, made with Calibre from book.epub;
+- `<title>.apnx`, the page numbers, taken from the printed page numbers.
+
+`--to <documents folder>` copies the book there, and the page-number file into `<title>.sdr/` beside it. That is where the Kindle looks for it. The user reopens the book to see page numbers.
+
+What we learned on a Paperwhite:
+
+- **Send to Kindle** (email or app) strips embedded fonts and gives no page numbers. A sideloaded AZW3 keeps both.
+- **Copying an EPUB over USB does not work.** Kindle ignores it.
+- **Kindle cannot shape Nastaliq.** Urdu books embed Amiri (Naskh) by default (`fonts: auto`).
+- **Embedded Arabic fonts fail on Quran verses with harakat inside an English book.** Kindle's own font renders them correctly, so English books embed no font. Tell the user to read with Bookerly or another Amazon font, not "Publisher font".
+- **Ayah numbers are written `(١٢)`**, because Kindle cannot draw digits inside ۝.
+
+## Finding the Kindle's documents folder
+
+| System | Path |
+|---|---|
+| Linux | `/media/$USER/Kindle/documents` |
+| macOS | `/Volumes/Kindle/documents` |
+| Windows | `<drive>:\documents` |
+| WSL | the drive may not be mounted; copy with `powershell.exe Copy-Item` instead |

package/agent/skills/waraq/references/models-and-cost.md

@@ -0,0 +1,43 @@
+# Models, cost and refusals
+
+## Which model
+
+Measured on real books:
+
+| Book | Model | Tokens per page (in / out) |
+|---|---|---|
+| English with Arabic quotations, scanned | Opus | about 15k / 1.5k |
+| Clean printed English, code, maths | Sonnet | about 15k / 1.4k |
+
+The CLI providers add their own system prompt to every page, so they use more tokens than an API key.
+
+- **Sonnet:** clean print, English. Comparing it with Opus on 12 tafsir pages:
+  - the English matched about 98%;
+  - each model got one word right that the other got wrong;
+  - neither added a word to a verse;
+  - Sonnet was 2–3 times faster.
+- **Opus:** small or faint print, Urdu Nastaliq, Arabic-heavy pages, and hadith. Nothing can check hadith afterwards.
+- **Quran verses** are corrected by `waraq quran check` with either model.
+
+Model ids:
+
+| Provider | Model ids |
+|---|---|
+| `claude-cli` and `anthropic` | `claude-sonnet-5-5` (Sonnet), `claude-opus-5-5` (Opus); the default is `claude-fable-5-1` |
+| `antigravity-cli` | `gemini-3.1-pro-high` |
+
+Use the full id, not an alias, so the result is reproducible.
+
+Pass `--model` on `init`, or `--provider` / `--model` on one `transcribe` run.
+
+## Cost
+
+Always run a trial slice (`--pages 1-12`), then quote the estimate it prints for the whole book. A 500-page book is roughly 7–8M input tokens. Ask the user before you go on.
+
+## Refusals
+
+About 2% of pages are refused by a provider's output filter, ordinary pages included. The fallback in `waraq.json` takes them over automatically; `init` sets it when a second CLI is installed. Pages written by a different provider are pinned, so later runs keep them.
+
+## Speed and memory
+
+Each CLI page is one process, and `concurrency` in `waraq.json` (default 4) sets how many run at once. About 12 parallel CLI processes ran comfortably in 7.5 GB of RAM. To run several books at once, lower the concurrency of each.

package/agent/skills/waraq/references/quran-review.md

@@ -0,0 +1,47 @@
+# Reviewing Quran quotations
+
+`waraq quran check` compares every Arabic quotation with the Tanzil Uthmani text:
+
+| Status | Meaning | What the check did |
+|---|---|---|
+| exact | Every word found, spelled the same | Rewrote it with canonical text |
+| fuzzy | Every word found, a letter or two differs | Rewrote it, and listed it |
+| partial | Most words found, but not cleanly | Left it as transcribed, for you to review |
+| unmatched | Not found | Left it. Usually hadith, a dua, or a title |
+| short | One or two words | Left it |
+
+Text printed around a verse stays as printed: brackets ﴿ ﴾, an ayah number carried over from the previous page, ۞, and the reference "(سورة البقرة: ١٣)".
+
+## Procedure for each item in `waraq quran review`
+
+Items with a `span` are Arabic quoted inside an English paragraph or footnote. Pass `--span <n>` to `quran set` and `quran note` for them.
+
+1. **Open the scan.** It is the `scan` path in the review list; read the image. Every item, no exceptions. The list is sorted by page, so open each page once and decide all of its items.
+2. Look at what the page prints, then decide:
+   - **set**: it is a Quran passage and the transcription is wrong or incomplete. Run `waraq quran set <page> <block> <ref>`.
+     - For a verse that starts on the previous page, add `--from <word>`: the number of the first printed word, as listed by `waraq quran canon <ref>`. Pause marks belong to their word and ayah numbers are not words.
+     - For one that runs onto the next page, add `--to <word>`.
+     - Use `--dry-run` first and compare its output with the scan.
+   - **variant**: the scan prints a recognised reading that differs from Hafs. Leave the text, and run `waraq quran note <page> <block> variant --note "..."`.
+   - **hadith**, or **other** for a dua, a title or a label: leave it, and note it.
+   - **ok**: the transcription already matches the scan.
+   - **unsure**: you cannot tell. Leave it, note it, and tell the user.
+3. Every item gets a decision (`set` notes itself), so `waraq quran review` shows how many are still open.
+
+Also look at the fuzzy rewrites: `waraq quran review --status fuzzy`. Check each against its scan when there are few; when there are many, check at least ten. A fuzzy match is usually a misread letter, but it can be a variant reading that the check made Hafs.
+
+## Known traps
+
+- **An abridged quotation**, ending in "…" or the printed "الخ" (etc.): note it `ok` and leave it. `quran set` would fill in the rest of the verse.
+- **Printed wording that differs from Hafs** but is not a variant you can name (one word changed, say): leave it as printed, note it `unsure` with the difference, and list it for the user. It may be the author's slip or an older edition; it is not yours to correct either way.
+- **Older European books number verses differently** (Flügel numbering: 18:47 where Hafs has 18:49). waraq reports Hafs numbers; leave the printed references alone.
+- **A quotation split by a footnote mark** becomes two spans. Set each span to its own words with `--from` and `--to`, or the text appears twice.
+- **`--span` with a block's words:** if `quran set --span` refuses because of words not in the verse, check with `--dry-run` that only that span changes, then use `--force`.
+
+- **An old book may spell a verse in an older orthography,** without the Uthmani marks. A matched verse is rewritten in the canonical Uthmani text anyway: that is the point of the check. Tell the user if the book is about the spelling itself. In that case they may prefer `waraq convert --no-quran-check`.
+
+- **A block that is a hadith quoting a verse.** `quran set` refuses it, because it would delete the hadith. Fix only the verse words with `waraq page set --text-file`, copying them from `waraq quran canon` output.
+- **The basmala printed above a surah is not part of its first ayah.** `canon` already leaves it out.
+- **Ayah numbers are written `(١٢)`**, not ۝١٢. Kindle cannot draw digits inside ۝.
+- **A passage longer than 15 ayahs is fine.** Check that the printed range and the ref agree.
+- **Variant readings seen in printed tafsirs** include 2:127, 2:210, 2:213, 11:46, 12:76, 14:46, 17:23, 18:77-80 and 39:53. That list is not complete. Judge by the scan.

package/agent/skills/waraq/references/rights.md

@@ -0,0 +1,15 @@
+# Which books, and what to tell the user
+
+waraq is a format converter. It cannot know where a PDF came from. Apply this, and say it plainly when it matters:
+
+- **Public domain** (for example, published before 1929): convert it and share it.
+- **Published for free distribution** (many dawah books say so on their copyright page): convert it, keep that page, and sharing is fine.
+- **A book the user owns in print:** convert it for their own reading. They should not pass the EPUB around.
+- **An in-copyright book they did not pay for:** converting it does not make it theirs. If a legal edition exists, point them to it.
+
+In practice:
+
+- Fill `rights` and `source` in `book.yaml` from what the book says, or ask the user. Never invent a rights statement.
+- Keep the copyright page of the original in the book.
+- waraq has no "download a book" feature. Do not find or download books for the user.
+- Every page is sent to the model provider the user chose. For a book still in copyright, that is one more copy leaving their machine. Mention it if the user seems unaware.

package/agent/skills/waraq/references/structure-fixes.md

@@ -0,0 +1,41 @@
+# Fixing structure
+
+## Chapters
+
+Each level-1 heading becomes a chapter. `waraq assemble` already turns these into level 2:
+
+- numbered sections ("2.5 …");
+- index letters;
+- a heading that appears in three or more separate places ("References", "Key Takeaways");
+- table-of-contents lines whose chapter comes later;
+- the book title printed again on an inner title page.
+
+`waraq headings` lists every level-1 heading, with the level assemble will use and why (`auto` or `hand`).
+
+What to fix by hand:
+
+- **A real chapter demoted.** For example, the closing "Conclusion" of a book that has one after every chapter: `waraq headings set <page> <block> 1`.
+- **A false chapter that is left**, such as a title-page line ("ALLAH", "VOLUME 1"), a section heading the model guessed wrong, or "Notes": `waraq headings set <page> <block> 2`.
+- **A chapter title with its number on a separate line.** Set the number block's text to the full title with `page set`, and delete the other block with `page set --delete`. Deleting moves later blocks up one index, so run `page show` again before the next edit on that page.
+
+Run `waraq assemble --force` after changes, then `waraq headings` again.
+
+## Text
+
+- **Misread words.** Check every page marked uncertain in review.html against its scan, then `waraq page set <n> <block> --text-file f`.
+- **Watermarks.** assemble removes a bare web address, or a "Downloaded from …" line, that appears on 3 or more pages. Delete any other site footer with `page set --delete`.
+- **Code** is kept as code blocks with their line breaks. Check that indentation survived on one or two pages.
+- **Maths** is written as plain Unicode (n², ∑, ⌈x⌉), not LaTeX.
+
+- **A footnote mark with no footnote** (`^3^` in the text, with an assemble warning "has no footnote") comes from a mark the model saw but whose note it did not find. Look at the scan. Remove the mark with `page set`, or leave it if the note is on another page.
+- **Library stamps, call numbers and handwriting** on scans are not part of the book. Delete them with `page set --delete`. Keep printed seals and title pages.
+- **Hebrew, Greek and other scripts** are tagged as English. waraq has no tag for them; leave them.
+
+## Figures
+
+- An ornament line becomes a plain rule (`* * *`): a thin band, or a wide band the model calls decorative.
+- A figure whose crop takes in body text is cut back automatically when the text sits at its edge. If a crop still holds text, edit the figure's description with `page set` (it replaces the alt text), or leave it and tell the user.
+
+## Cover
+
+`book.yaml` `cover:` can be `pages/NNNN.png` (a whole scan) or `images/pNNNN-k.png` (a cropped figure). waraq picks a photo of the front cover from the first 8 scans. It skips a back cover, a spine or a dust-jacket spread. If the guess is wrong, set it by hand.

package/assets/body-amiri.css

@@ -0,0 +1,3 @@
+/* Whole-book rule for an Arabic or Urdu book. Plain selector on purpose: Calibre and some reader engines
+   skip :lang() and attribute selectors, then drop the font as unused. */
+body { font-family: "Amiri", serif; line-height: 1.9; }

package/assets/body-nastaliq.css

@@ -0,0 +1,2 @@
+/* Whole-book rule for an Urdu book set in Nastaliq. Plain selector on purpose, see body-amiri.css. */
+body { font-family: "Noto Nastaliq Urdu", serif; line-height: 2.2; }

package/assets/epub.css

@@ -0,0 +1,29 @@
+body { font-family: Georgia, "Times New Roman", serif; line-height: 1.6; }
+
+h1, h2, h3 { font-family: Arial, Helvetica, sans-serif; line-height: 1.25; margin: 1.5em 0 0.5em; page-break-after: avoid; }
+h1 { font-size: 1.7em; text-align: center; page-break-before: always; }
+h2 { font-size: 1.3em; }
+h3 { font-size: 1.1em; }
+
+p { margin: 0 0 0.9em; }
+
+/* Boxed quotations, as in the printed books. */
+blockquote { margin: 1em 0; padding: 0.6em 1em; border: 1px solid #555; }
+blockquote p:last-child { margin-bottom: 0; }
+
+figure { margin: 1.2em 0; text-align: center; page-break-inside: avoid; }
+img { max-width: 100%; height: auto; }
+figcaption { font-size: 0.9em; font-style: italic; }
+
+table { border-collapse: collapse; margin: 1em 0; width: 100%; }
+th, td { border: 1px solid #999; padding: 0.3em 0.5em; }
+
+.footnotes { font-size: 0.9em; }
+
+/* Code keeps its line breaks; long lines wrap rather than run off a small screen. */
+pre { font-family: "Courier New", Courier, monospace; font-size: 0.8em; line-height: 1.35; white-space: pre-wrap; margin: 1em 0; padding: 0.5em; border-left: 3px solid #999; text-align: left; }
+code { font-family: "Courier New", Courier, monospace; }
+
+/* Direction comes from the dir attribute: EPUB forbids the CSS direction property. */
+[dir="rtl"] { text-align: right; }
+[dir="ltr"] { text-align: left; }

package/assets/fonts/Amiri-OFL.txt

@@ -0,0 +1,94 @@
+Copyright 2010-2022 The Amiri Project Authors (https://github.com/aliftype/amiri).
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+https://openfontlicense.org/
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded, 
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.
+