nbk2pdf 1.0.0

data/README.md

@@ -0,0 +1,490 @@
+# nbk2pdf
+
+Converts Kindle Scribe notebooks (`nbk` files) into vector PDFs, including the
+ruled or squared background the notes were written on.
+
+```
+nbk2pdf sync ~/Documents/scribe          # straight off a connected Kindle
+nbk2pdf ~/kindle/.notebooks              # or from files you already have
+nbk2pdf -o ~/Documents/scribe --titles titles.yml /media/kindle/.notebooks
+```
+
+It also takes a PDF you wrote on with the pen and produces a copy of it with
+the handwriting on the pages, ready to print. See [Writing on a
+PDF](#writing-on-a-pdf).
+
+## Installation
+
+```
+gem install nbk2pdf
+```
+
+Or add it to a Gemfile:
+
+```ruby
+gem "nbk2pdf"
+```
+
+Requires Ruby 3.1 or newer. The `sqlite3` gem ships a prebuilt native
+extension for common platforms; elsewhere it builds against the system SQLite.
+
+`nbk2pdf sync` additionally needs `gio`, from gvfs, which is already present on
+most Linux desktops. Everything else works without it.
+
+## Syncing from a connected Kindle
+
+```
+nbk2pdf sync             # into the current directory
+nbk2pdf sync ~/notes     # into a directory of your choosing
+```
+
+A Kindle speaks MTP rather than presenting a disk, so it cannot be mounted with
+`mount(8)`. On Linux, gvfs exposes MTP devices as FUSE paths once `gio mount`
+has attached them, and that is what `sync` drives:
+
+1. If a Kindle is already mounted, its notebooks are used as they are.
+2. Otherwise the MTP volume is located with `gio mount -li` and mounted.
+3. Every notebook under `.notebooks/` on the device is converted into the
+   destination directory, which is created if it does not exist.
+
+Notebooks and PDFs written on are kept apart. Notebooks go side by side into
+`notebooks/`. Each PDF goes into `documents/`, under the same folders it sits in
+under `documents/` on the Kindle:
+
+```
+~/notes/
+├── notebooks/
+│   ├── Reading list.pdf
+│   └── 2026-09-22_dee63122-45e3-c7e2-ba57-7824a0bd244b.pdf
+└── documents/
+    └── HoTT/
+        └── workbook.pdf
+```
+
+Only `documents/` is looked in for PDFs, since it is the only folder the
+Kindle's own reader lists, and so the only place a PDF can have been written on
+with the pen.
+
+The device is left mounted afterwards. Only top-level notebook folders are
+converted, so the clipboard scratch buffer is skipped.
+
+If nothing is found, USB is checked directly for Amazon's vendor id (`1949`) so
+the two cases can be told apart:
+
+* **not on the bus at all** -- reported immediately, since waiting will not
+  help: connect the cable and try again.
+* **plugged in but not offered by gvfs** -- usually the screen is locked, or the
+  device is still settling after being attached, so this one is waited on for a
+  few seconds before giving up with a suggestion to unlock the screen or
+  reconnect the cable.
+
+A Kindle that has dropped off the bus does not come back by itself; it needs the
+cable unplugged and reconnected, which is what the message says.
+
+The waits are measured against a monotonic clock rather than by counting sleeps,
+because each probe of gvfs costs about half a second and a naive count made a
+ten second budget take thirty.
+
+### Why notebook names have to be supplied
+
+The Kindle shows a name for each notebook, but it is not in anything the device
+shares over MTP. Checked on a Scribe:
+
+* the `nbk` file does not contain it in UTF-8 or either UTF-16 byte order, and
+  its only metadata category holds just `book_id` and `language`;
+* `book_id` decodes to a random version-4 UUID, so it is an opaque handle with
+  nothing to derive a name from;
+* `.sync/KSDKNoteSyncDB.sqlite` holds notebook ids and sync bookkeeping only,
+  and does not mention the `book_id` either;
+* a sweep of 1115 files and 84 MB across every other directory MTP exposes
+  turned up nothing, in any of the three encodings.
+
+The names live on the system partition, which MTP does not share -- it offers
+only `Internal Storage`, that is `/mnt/us`. So `sync` names files by date and
+uuid, and offers the unnamed ones to `--titles` to fill in once.
+
+## Writing on a PDF
+
+Copy a PDF into `documents/` on the Kindle, open it there, and write on its
+pages. Then:
+
+```
+nbk2pdf sync ~/Documents/scribe                                  # with everything else
+nbk2pdf -o done.pdf "$KINDLE/documents/HoTT/workbook.pdf"        # just the one
+nbk2pdf --annotations -o done.pdf "$KINDLE/documents/HoTT/workbook.pdf"
+```
+
+The result is the original PDF, with its text, links and named destinations
+unchanged and the ink drawn on each page it was written on. By default the ink
+becomes part of the page, so every viewer and printer shows it and the pressure
+taper is kept.
+
+The copy is rebuilt rather than edited in place, and some document-level
+entries do not survive it. Seen on the sample: the catalog's `/OpenAction` and
+`/PageMode`, and the original dates and producer in the document info. From how
+the PDF library works, `/PageLabels`, XMP metadata, the structure tree used for
+accessibility, and optional-content layers (all layers end up shown) would be
+lost too. Page content, links and named destinations are kept, and the library
+carries the outline across as well.
+
+`--annotations` adds each stroke as a PDF ink annotation instead, which a viewer
+can hide, move or delete. Each carries an appearance drawn by the same code, so
+it keeps the taper too, and it is flagged to print. Whether it actually prints
+is up to the viewer: most print dialogs have a setting that leaves annotations
+out. For paper, prefer the default.
+
+A single PDF is written into the current directory unless `-o` says otherwise,
+and the tool refuses to write over the original. `sync` writes it into
+`DIR/documents/`, in the folders it has on the Kindle, under its own name.
+
+The Kindle never changes the PDF itself, which is why copying it back over USB
+gives you the blank one. What it writes instead is described in [6.
+Documents](#6-documents-documentrb-krdsrb). Ink that cannot be placed on a page
+is reported on stderr rather than dropped silently.
+
+## Naming
+
+Standalone Scribe notebooks do not store the name you gave them on the device --
+that lives in the Kindle's own library database, not in the `nbk` file. The
+output name is therefore resolved in this order:
+
+1. an entry in `--titles`, keyed by the notebook's uuid or its book id
+2. the title inside the notebook, when it has one (notebooks attached to a book do)
+3. the notebook's modification date followed by its uuid,
+   e.g. `2025-09-19_29b6da80-211a-ebf9-56a2-ade81624ebfd.pdf`
+
+`--titles` takes a JSON or YAML mapping:
+
+```yaml
+29b6da80-211a-ebf9-56a2-ade81624ebfd: The Humongous Book of Calculus Problems
+bfa1ee49-16e8-a25e-0852-7273f4ef9e43: Reading list
+```
+
+You do not have to write it by hand. Point `--titles` at a file that does not
+exist yet and every notebook without a name is added to it, with its page count
+and date as a comment so you can tell them apart:
+
+```
+$ nbk2pdf sync --titles titles.yml notes
+...
+3 notebooks had no name; added to titles.yml to fill in
+
+$ cat titles.yml
+# 20 pages, last written 2026-09-19
+29b6da80-211a-ebf9-56a2-ade81624ebfd: ""
+```
+
+Fill a name in between the quotes and every later run uses it. An entry left
+blank still counts as answered, so you are not asked about it again.
+
+## Usage
+
+```
+Convert Kindle Scribe notebooks to PDF.
+
+Usage: nbk2pdf [options] PATH...
+       nbk2pdf sync [options] [DIR]
+
+  PATH may be an `nbk` file, a notebook directory, or any directory
+  containing notebooks (they are found recursively). It may also be a
+  PDF on the Kindle, to get a copy with what was written on it.
+
+  sync converts every notebook on a Kindle attached over USB into
+  DIR/notebooks, and every PDF on it that has been written on into
+  DIR/documents, in the folders it has under documents/ on the
+  Kindle. DIR is the current directory when left out. The device is
+  mounted over MTP first if it is not mounted already, and is left
+  mounted afterwards.
+
+Options:
+    -o, --output PATH                Output file, or directory for several notebooks
+    -n, --name NAME                  Override the output file name (single notebook only)
+    -t, --titles FILE                JSON/YAML map of notebook uuid to the name you gave it
+        --[no-]template              Draw the notebook's ruled background (default: yes)
+    -p, --paper SIZE                 Page size: letter (default), a4, native, ...
+    -a, --annotations                Add ink to a PDF as annotations, not drawn into the page
+    -q, --quiet                      Only report errors
+    -h, --help                       Show this message
+    -v, --version                    Show the version
+```
+
+A Scribe's `.notebooks` folder also holds a `clipboard/` scratch buffer, which
+recursive discovery picks up like any other notebook. Pass notebook directories
+explicitly if you would rather skip it.
+
+## The file format
+
+Nothing about `nbk` is documented by Amazon, so here is what the converter does.
+Each layer has its own module under `lib/kindle_notebook/`.
+
+### 1. The signature container (`container.rb`)
+
+An `nbk` file is a SQLite database interleaved with 1024-byte signature blocks:
+one after the first kibibyte, then one after every mebibyte.
+
+```
+[ 1 KiB db ][ 1 KiB sig ][ 1 MiB db ][ 1 KiB sig ][ 1 MiB db ] ...
+```
+
+Strip those and SQLite opens the result cleanly -- `PRAGMA integrity_check`
+passes on every sample. The expected number of blocks is cross-checked against
+the page size and page count in the SQLite header, so a file that does not fit
+the layout is rejected rather than silently mangled.
+
+### 2. Ion fragments (`ion.rb`)
+
+The database keeps content as rows of binary [Amazon
+Ion](https://amazon-ion.github.io/ion-docs/) in `fragments` and
+`local_delta_fragments` (the latter overrides the former, and marks deletions).
+One row, `$ion_symbol_table`, holds the shared symbol table: it imports a
+catalogue named `YJ_symbols` (max id 816) and appends the notebook's own
+`nmdl.*` symbols.
+
+`YJ_symbols` ships no text, so symbols inside the imported range keep their
+numeric spelling -- which is why field names such as `$258` and `$141` appear in
+the decoded structures, exactly as Amazon's own tooling renders them.
+
+Fragment types are the Ion annotation on each row's top-level value:
+
+| Type   | Meaning                                                   |
+|--------|-----------------------------------------------------------|
+| `$258` | reading order: `$169` sections, the `$351` one lists pages |
+| `$260` | a page, or a template                                      |
+| `$259` | a container of child items (`$146`), reached through `$176`|
+| `$490` | book metadata (`book_id`, `language`, sometimes `title`)   |
+| `$538` | document data, including the default template              |
+
+### 3. Pages and strokes (`notebook.rb`, `stroke_decoder.rb`)
+
+A page records its canvas (15624 x 20832 sub-pixels at 2520 ppi on a Scribe) and
+a tree of content items. Walking it through `$176` references and `$146`
+children reaches the leaves, each carrying `nmdl.stroke_points`.
+
+Coordinates are stored as a second-order delta stream -- the change *of the
+change* between samples, which stays near zero for a smooth pen movement:
+
+```
+[0x01 0x01]        magic
+[uint32-LE]        number of values
+[nibble stream]    one 4-bit instruction per value, two per byte
+[operand bytes]    variable-width operands
+
+bits 0-1  operand width: 0 = none, 1..3 = that many little-endian bytes
+bit 2     the increment is the width value itself; no operands consumed
+bit 3     negate the increment
+```
+
+Decoded values are relative to the stroke's `nmdl.stroke_bounds` origin.
+
+Strokes also carry pressure, tilt and per-point thickness factors, decoded by
+the same scheme. Pressure drives the stroke taper (below). Tilt turns out to be
+recorded once per stroke rather than per point, and both adjust factors are a
+flat `100` everywhere in the samples, so neither is used.
+
+### 4. Templates
+
+Each page names the background it was written on through `nmdl.template_id`.
+It can be:
+
+* a reference to a template fragment, e.g. `narrow_rule_margin_1860x2480`
+* a placeholder symbol (`$349`), meaning the page is blank
+* absent, in which case the document's default from `$538` applies
+
+A template draws itself in its own 1860 x 2480 space with a flat op list --
+`0 x y` to move, `1 x y` to draw, `4` to close -- plus an ARGB colour and a line
+width. The converter scales that space onto the page.
+
+### 5. Rendering (`pdf_renderer.rb`)
+
+Every PDF page matches the Scribe canvas exactly (`canvas / ppi * 72` points, so
+446.4 x 595.2 pt), with the Y axis flipped. Strokes become round-capped polyline
+paths; a one-point stroke becomes a filled dot; highlighter strokes (brush type
+1) are drawn at 20% opacity.
+
+#### Paper size
+
+The device records its pages at 446.4 x 595.2 pt, which is 6.20 x 8.27 in or
+157.5 x 210 mm, in a 3:4 ratio. That is the Scribe's screen, and it matches no
+standard paper size, so a print dialog offers to scale it.
+
+Pages therefore go on **US Letter** by default, scaled to fit and centred so
+nothing is stretched or cropped. `--paper` chooses something else:
+
+```
+nbk2pdf --paper a4 notebook/          # A4 instead
+nbk2pdf --paper native notebook/      # exactly as the device recorded it
+nbk2pdf sync --paper a4 ~/notes
+```
+
+Every size Prawn knows is accepted, including `a3`..`a6`, `letter`, `legal`,
+`tabloid` and `executive`. On Letter the canvas fits the height and leaves 9 pt
+at each side; on A4 it fits the width and leaves 24 pt above and below.
+
+Note that the PDF sets no `/ViewerPreferences` and no `/PrintScaling`, so it
+never tells a viewer whether scaling is allowed. If a print dialog greys out its
+scale field, that is the viewer's own setting -- in Evince, the manual scale is
+only editable once *Page Scaling* is set to *None*.
+
+#### Stroke taper
+
+The pen records a 12-bit pressure per point, and it varies a lot inside a single
+stroke -- a median spread of 3071 out of 4095 across the samples -- so drawing
+every stroke at one width loses the thinning at the ends of each pen movement.
+
+Amazon documents no pressure-to-width curve, so it was fitted against the only
+ground truth in the file set: the 600 x 800 thumbnails the device renders for
+its own pages. Rendering each sample page at that size and taking the mean
+absolute difference against the matching thumbnail gives, summed over three
+notebooks:
+
+| Model                                             | difference |
+|---------------------------------------------------|------------|
+| flat width (no taper)                              | 14.8711    |
+| width from *absolute* pressure, best fit           | 14.8414    |
+| width from pressure *relative to the stroke's mean*| **14.7932**|
+
+The third model wins, which says the overall width of a stroke is already
+carried by `nmdl.thickness`, and pressure shapes the taper within it. So the
+width of each segment is
+
+```
+level  = sqrt(pressure / 4095)
+factor = 1 + TAPER_SPAN * (level - mean level of this stroke)
+width  = nominal thickness * factor
+```
+
+with `TAPER_SPAN = 1.15`, the flat part of the optimum (1.1 and 1.2 score
+identically, 1.0 and 1.3 are worse).
+
+Two cautions. The thumbnail cannot resolve taper directly -- the pen is 0.91 px
+wide there and the taper range is entirely sub-pixel -- so what the measurement
+compares is ink coverage, not shape. And a first attempt at fitting scored every
+taper *worse* than flat, because that parameterisation only ever made strokes
+thinner and the metric was reacting to the missing ink rather than to the taper;
+the numbers above come from mean-preserving variants, where a control sweep of
+plain width multipliers confirms the metric has a real optimum rather than just
+rewarding more ink.
+
+Widths are rounded to 8 steps so each stroke becomes a handful of sub-paths
+instead of one per sample point. At 600 dpi (about 8x zoom) 8, 32 and 256 steps
+are indistinguishable, and the runs share their junction points, so no seam
+shows. The highlighter does not taper: its pressure barely moves along a stroke
+while differing widely between strokes, so following it would give each band an
+arbitrary thickness.
+
+### 6. Documents (`document.rb`, `krds.rb`)
+
+Opening a PDF on the device and writing on it leaves the PDF untouched and
+writes this beside it:
+
+```
+documents/HoTT/workbook.pdf
+documents/HoTT/workbook.sdr/<id>/book.kdf                  the device's copy of the layout
+documents/HoTT/workbook.sdr/<id>/book.sdr/book<hash>.yjr   the notes, as annotations
+documents/HoTT/workbook.sdr/kpp_<id>/book.kdf              the same layout again
+.notebooks/<id>!!PDOC!!notebook/nbk                        the ink
+```
+
+Only the ink and the notes are needed.
+
+**The ink** is an ordinary notebook in the format above, one page per PDF page
+written on. Each page records, besides its canvas, the frame the canvas
+occupies on the PDF page as percentages from the top left: `$59` left, `$58`
+top, `$56` width, `$57` height, each with unit `$314`. On a Letter page the
+Scribe's 3:4 canvas is fitted to the width, so it is 103.03% as tall as the
+page and starts 1.515% above it, 12 points past each edge.
+
+**The notes** are in the `.yjr` file, a Kindle reader data store:
+
+```
+[00 00 00 00 00 1A B1 26]   signature
+[long]                      version
+[int]                       number of top-level fields
+fields                      FE, name, typed values and nested fields, FF
+```
+
+Values carry a type byte: `00` boolean (one byte), `01` int (4 bytes), `02`
+long (8 bytes), `03` string (a flag byte, `01` for null, otherwise a 2-byte
+length and UTF-8), `07` byte. All big-endian. Field names are strings without
+the type byte. That is enough to read the whole file without knowing what any
+field means.
+
+Each note is a field `annotation.personal.handwritten_note`, nested inside
+`annotation.cache.object` > `saved.avl.interval.tree`. It holds a start and an
+end position, creation and modification times, a template string, and the id
+of the ink page. The page is in the number after the colon of the position,
+in its upper 16 bits:
+
+| Page written on | Position                | After the colon | >> 16 |
+|-----------------|-------------------------|-----------------|-------|
+| 1               | `AcYEAAAAAAAA:2`        | `0x000002`      | 0     |
+| 2               | `AckEAAAAAAAA:65538`    | `0x010002`      | 1     |
+| 50              | `AVkFAAAAAAAA:3211266`  | `0x310002`      | 49    |
+| 183             | `AegGAAAAAAAA:11927554` | `0xB60002`      | 182   |
+
+The lower 16 bits were 2 every time. What the base64 part encodes is not
+known and is not used. It is not an element of the layout in `book.kdf`: for
+page 1 its bytes happen to contain 4, the id of that page's image there, which
+is what a first attempt keyed on, but the other pages do not follow.
+
+Pages with a `/Rotate` entry have not been seen yet, and their ink is placed
+as if they were upright.
+
+## Library use
+
+```ruby
+require "nbk2pdf"
+
+notebook = KindleNotebook::Notebook.open("/media/kindle/.notebooks/<uuid>")
+notebook.pages.each do |page|
+  puts "#{page.id}: #{page.strokes.size} strokes on #{page.template&.kind || "blank paper"}"
+end
+
+KindleNotebook::Converter.new.convert(notebook.path, output: "out/")
+
+document = KindleNotebook::Document.open("/media/kindle/documents/HoTT/workbook.pdf")
+document.notes.each { |note| puts "page #{note.page_index + 1}: #{note.page.strokes.size} strokes" }
+KindleNotebook::DocumentRenderer.new.render(document, "workbook.pdf")
+```
+
+## Development
+
+```
+bundle install
+rake               # tests and RuboCop
+rake test
+rake rubocop
+bundle exec exe/nbk2pdf sample-notebooks
+```
+
+The test suite builds synthetic notebooks from scratch -- `test/test_helper.rb`
+contains a small binary Ion *writer* and an encoder for the stroke format -- so
+the unit tests need no binary fixtures. It also writes the device's reader data
+store format, so the same goes for PDFs written on.
+`test/samples_test.rb` additionally runs against `sample-notebooks/` and is
+skipped when that directory is absent. Documents go under
+`sample-notebooks/device/`, laid out as on the device: `documents/...` with the
+PDF and its `.sdr`, and `.notebooks/<id>!!PDOC!!notebook/`.
+
+The stroke decoder is checked against values produced by an independent
+implementation of the same encoding, so a round trip through this project's own
+encoder cannot mask a shared mistake.
+
+Rendering is checked by eye against the PNG thumbnails the device itself writes
+to `sample-notebooks/thumbnails/`:
+
+```
+bundle exec exe/nbk2pdf -o out sample-notebooks/<uuid>
+pdftoppm -png -r 97 -f 1 -l 1 out/*.pdf page
+```
+
+## License
+
+GNU General Public License v3.0 or later. See [COPYING](COPYING).
+
+The encoding this converter decodes was described publicly by
+[kindle_notebook_pdf_converter](https://github.com/rudolfstepan/kindle_notebook_pdf_converter),
+which in turn traces its stroke decoder to `kfxlib` from John Howell's KFX Input
+plugin for Calibre -- GPL v3. This project is an independent Ruby implementation
+written from that description of the format rather than a port of either, but it
+is released under the GPL so that its lineage stays unambiguous.

data/exe/nbk2pdf

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# SPDX-License-Identifier: GPL-3.0-or-later
+# SPDX-FileCopyrightText: 2026 Sergey Avseyev
+# frozen_string_literal: true
+
+require "nbk2pdf"
+
+exit KindleNotebook::CLI.start(ARGV)