pdfmd-cli 3.23.17__tar.gz → 3.25.20__tar.gz

{pdfmd_cli-3.23.17/pdfmd_cli.egg-info → pdfmd_cli-3.25.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pdfmd-cli
-Version: 3.23.17
+Version: 3.25.20
 Summary: One command from Markdown to a good-looking PDF: a Pandoc wrapper with smart defaults, project-file discovery and a multi-engine fallback chain
 Author: Ali Perdekhan
 License-Expression: MIT
@@ -28,6 +28,8 @@ Requires-Dist: pypinyin; extra == "translit"
 Requires-Dist: anyascii; extra == "translit"
 Provides-Extra: pandoc
 Requires-Dist: pypandoc_binary; extra == "pandoc"
+Provides-Extra: tui
+Requires-Dist: prompt_toolkit>=3; extra == "tui"
 Dynamic: license-file
 
 # pdfmd
@@ -38,7 +40,9 @@ Dynamic: license-file
 [Pandoc](https://pandoc.org) and fills in everything you would otherwise
 have to remember: sensible fonts and margins, the right Markdown dialect,
 your project's metadata/preamble/filter files, and a fallback chain across
-every PDF engine you have installed.
+every PDF engine you have installed. The same document also builds to a
+Word or OpenDocument file that follows its page setup, fonts and LaTeX (see
+[Word and OpenDocument output](#word-and-opendocument-output)).
 
 ```console
 $ pdfmd lecture
@@ -46,6 +50,11 @@ AUTO: READER TITLE MARGIN MONOFONT. Use --verbose to see in full
 OK    lecture.md
 ```
 
+In a folder with a single Markdown file you do not even need the name: `cd` into the folder and run `pdfmd`. It finds
+the file, its metadata, preamble and filters, and writes the PDF beside it (with several Markdown files it asks for
+one, or for `-b` to convert all of them or `-r` to join them). `pdfmd --help` is one short page; `pdfmd --help all`
+is the complete option list and `pdfmd --help tables` (or `source`, `history`, `output` ...) one subject at a time.
+
 <p align="center">
   <img src="https://raw.githubusercontent.com/aliperdehan/pdfmd/main/docs/lecture.png" width="560" alt="lecture.md rendered to PDF">
 </p>
@@ -74,6 +83,34 @@ turns that knowledge into defaults:
 - **It doesn't give up on the first engine.** If `lualatex` fails or isn't
   installed, it tries the next engine, then the next, and tells you why each
   one failed.
+- **It writes real Word files.** `pdfmd report -o report.docx` reads the front matter the PDF reads (paper size,
+  margins, fonts, line spacing) and writes it into the file's page setup and styles, so the Word file is laid out the
+  way the PDF is. With LaTeX installed, what only LaTeX can say in the document comes along: `\ce{...}` chemistry,
+  `\SI{...}{...}` units and equations become native, editable Word text and equations; a house style's own macros can be
+  given a Word recipe; what is left (tikz, chemfig, math Pandoc cannot convert) is drawn by LaTeX in the document's
+  own preamble and embedded as vector pictures. `pdfmd-options: {office: {latex: auto|images|off}}`, `--no-auto
+  officelatex` and `pdfmd --setup` switch it (see [Word and OpenDocument output](#word-and-opendocument-output)).
+- **It handles every script and emoji.** Text the main font cannot draw (Arabic,
+  Hebrew, Chinese, Japanese, Korean, Greek, Cyrillic with Kazakh letters, Indic and
+  more) is set run by run in an installed font for its script, emoji become colour
+  pictures, and `pdfmd --install fonts` fetches missing fonts without admin rights
+  (see [Other scripts](#other-scripts)).
+- **It finds documents by name, in any script.** `pdfmd glyukoza` opens
+  `Глюкоза.md`; with `--translit` packs, `pdfmd tyche` finds `Τύχη.md` and
+  `pdfmd mingyun` finds `命運.md`.
+- **It reads PDFs too.** `pdfmd paper.pdf` restores a PDF's own embedded source or
+  OCRs the pages to Markdown (see [PDF to Markdown](#pdf-to-markdown)).
+- **Tables can be data.** A `::: {.csv}` block puts a table's data in a CSV file or right in the document (any
+  delimiter, alignment and widths as attributes), and `pdfmd --extract-tables` turns the tables you already have into
+  such blocks after checking that Pandoc reads the same tables back (see [Tables straight from a CSV
+  file](#tables-straight-from-a-csv-file)). A table that is too wide gets its columns sized by their content.
+- **Long code wraps, SVG and web images work.** Code lines wrap at the margin in a LaTeX build, an SVG is converted for
+  LaTeX with whatever is installed, and an image from the web is fetched once and kept.
+- **Finished files are not rewritten.** A `.typ` file goes to Typst and an `.html` file to WeasyPrint or a browser, so
+  the page setup, styles and scripts survive; `--hybrid` makes the PDF open in LibreOffice as an editable document.
+- **It keeps the history.** `pdfmd --history` lists the backups, commits and compiles of a file, `--history-restore`
+  puts a version back (backing up the one it replaces), and the build history can live in a `.hst` file instead of the
+  document (see [Build stamps, history and snapshots](#build-stamps-history-and-snapshots)).
 - **It shows what it did.** Every automatic decision prints an `AUTO` line,
   `-v` shows the exact Pandoc command, and every default can be switched off.
 
@@ -121,8 +158,12 @@ brew install --cask mactex-no-gui           # optional: full LaTeX (large)
 
 ```sh
 sudo apt install pipx pandoc texlive-xetex  # Debian/Ubuntu
+sudo dnf install pipx pandoc texlive-xetex  # Fedora (add Typst with `pdfmd --install typst`)
+sudo pacman -S python-pipx pandoc-cli typst texlive   # Arch
 ```
 
+On any Linux, `pdfmd --install full` (Pandoc and Typst) needs no package manager and no root.
+
 ```powershell
 py -m pip install --user pipx; py -m pipx ensurepath          # Windows
 winget install --id JohnMacFarlane.Pandoc; winget install --id Typst.Typst
@@ -140,8 +181,8 @@ Office files.
 <details>
 <summary>Without pipx</summary>
 
-`pdfmd.py` is a single file that needs Python 3.9+ (with `pdfmd_inkmd/`
-beside it for the no-Pandoc fallback). It also runs
+`pdfmd.py` needs Python 3.9+ and runs straight from a clone, with the folders beside it
+(`pdfmd_inkmd/` for the no-Pandoc fallback, `pdfmd_office/`, `pdfmd_setup/`, `pdfmd_unicode/`). It also runs
 directly, and `pyyaml`/`pypdf` are optional (features that need them are
 skipped with a warning):
 
@@ -175,6 +216,55 @@ OK    quarto (only needed for .qmd files)  (/usr/local/bin/quarto)
 The numbers are the fallback order, and also shortcuts: `-e 6` means
 `-e typst`.
 
+For everything at once, `pdfmd --doctor` is one page on the Markdown-to-PDF tools, fonts
+and emoji, PDF reading (batchocr, Tesseract, Poppler, OCR languages), your config file and
+the cache, ending with the command that fixes each missing piece:
+
+```console
+$ pdfmd --doctor
+...
+Reading PDFs (PDF to Markdown):
+OK    batchocr 1.2.5
+MISS  tesseract: not found
+MISS  poppler: not found
+
+1 thing(s) to fix:
+  1. tesseract (scanned PDFs) and poppler (scanned PDFs)
+       brew install tesseract poppler
+```
+
+### Tab completion
+
+`pdfmd --completion bash|zsh|fish` prints a completion script, generated from the command
+line itself so it never goes out of date:
+
+```sh
+pdfmd --completion zsh > ~/.zfunc/_pdfmd                              # ~/.zfunc in $fpath, then compinit
+pdfmd --completion bash > ~/.local/share/bash-completion/completions/pdfmd
+pdfmd --completion fish > ~/.config/fish/completions/pdfmd.fish
+```
+
+### An older Pandoc, and a machine with no network
+
+The oldest Pandoc `pdfmd` is tested with is **3.1.3** (what the claude.ai sandbox has); the whole test suite runs against
+it. An older one may work, and the first build with it says so once; `pdfmd --doctor` lists what a given Pandoc does
+without. With 3.1.3: Typst PDFs cannot show a picture `pdfmd` drew itself (before 3.2), a numbered equation in a Word file
+sits in a plain table instead of a borderless one (before 3.4), and `--extract-tables` leaves a table whose caption has
+a `{#tbl:id}` (before 3.8.2: Pandoc reads that as words, so the change could not be checked). A `pandoc-crossref` must be
+the release built for the Pandoc you run; one built for another is not run, with a warning, because it would put `??` where
+the numbers go.
+
+Where `pip install pdfmd-cli` cannot reach the network (a sandbox with no DNS that only takes an upload), two ways:
+
+```sh
+python3 scripts/build_zipapp.py          # on a machine with pdfmd's source: dist/pdfmd.pyz, 1.4 MB, PyYAML and pypdf inside
+python3 pdfmd.pyz report.md              # on the other: needs Python 3.9+, Pandoc and a PDF engine; unpacks itself once
+                                         # into ~/.cache/pdfmd/pyz/ because Pandoc has to open the Lua filters as files
+
+pip download pdfmd-cli -d wheelhouse     # or: a folder of wheels, carried over
+pip install --no-index --find-links wheelhouse pdfmd-cli
+```
+
 ### No Pandoc? The built-in fallback
 
 On a machine with no Pandoc, or Pandoc but no PDF engine, `pdfmd` does not
@@ -278,6 +368,28 @@ letters, no tones, vowels left out where the script leaves them out. Names that
 romanize alike (homophones) are listed as ambiguous, never picked between. The
 same spelling rules find a section by its heading (`doc#tyche`).
 
+### A small editor
+
+```sh
+pdfmd --install tui           # once: prompt_toolkit, about 1 MB
+pdfmd --edit lecture          # nano-like full screen; Ctrl-B saves and builds, Ctrl-P opens the PDF
+```
+
+`pdfmd --edit` (alpha) opens the Markdown file in a plain full-screen editor with the build one key away: Ctrl-S
+saves, Ctrl-B saves and builds (what pdfmd said stays at the bottom), Ctrl-P opens the result, Ctrl-W / F3 find,
+Ctrl-G go to a line, Ctrl-K / Ctrl-U cut and paste a line, Ctrl-Z / Ctrl-Y undo and redo, Ctrl-X leaves (asking once
+when there are unsaved changes); F1 lists them. Options given beside it (`pdfmd --edit lecture -e lualatex`) are used
+for the build, and a name that does not exist yet starts a new file. It is deliberately rough: a base for what comes later.
+
+### In VS Code
+
+`pdfmd --init-vscode` writes a `.vscode/tasks.json` here (build, build and open, watch, extract tables; Terminal > Run
+Build Task). The [`vscode/`](https://github.com/aliperdehan/pdfmd/tree/main/vscode) folder of the repository is a small
+extension over the same command line: build (`Cmd/Ctrl+Alt+B`), build and open, build as Word/HTML/Typst/slides, extract
+tables (all, or chosen ones), file history, compare and restore, restore the source a PDF carries (right-click it). Every
+command it runs is written to the *pdfmd* output panel. Install it with `npx @vscode/vsce package` and
+`code --install-extension pdfmd-vscode-*.vsix`.
+
 ### One section
 
 `pdfmd doc#onlyapart` builds just the section whose heading is "Only a Part"
@@ -305,6 +417,111 @@ pdfmd lecture --to typst -o lecture.typ
 pdfmd lecture -o lecture.tex      # a complete, compilable .tex, not a fragment
 ```
 
+### Word and OpenDocument output
+
+Plain Pandoc ignores `papersize`, `geometry`, `mainfont` and `fontsize` when it writes `.docx`
+or `.odt`: you get a Letter page in the default fonts. pdfmd writes them into the *reference
+document* Pandoc builds the file from, so the page and the text follow the same front matter as the PDF:
+
+| Front matter | In the Word file |
+|---|---|
+| `papersize: a4` (also `letter`, `legal`, `a5`, `b5`..., `classoption: [a4paper, landscape]`) | page size and orientation |
+| `geometry:` (a list, `margin=2cm`, `top=1in,left=...`, `paperwidth=...`) or `margin:` | page margins |
+| `mainfont`, `sansfont`, `monofont`, `CJKmainfont`, `fontsize`, `linestretch`, `lang`, `indent` | body, heading and code fonts, sizes, line spacing, language, first-line indent |
+
+A `.docx` carries font *names*, not fonts, and the person opening it has Times New Roman, not
+STIX Two Text. So well-known fonts Word does not ship are mapped to the closest one it does (STIX
+Two Text, Libertinus, Latin Modern to Times New Roman; Helvetica Neue, Inter to Arial; JetBrains
+Mono, Menlo, Fira Code to Consolas), with an `AUTO OFFICE` note; `office: {fonts: exact}` keeps
+your names. Without any of these keys pdfmd's own look applies: black bold headings sized like a LaTeX
+article, a centred title block, booktabs-style table rules, Times New Roman and Consolas.
+
+**Your own template.** A reference document decides the page and fonts itself, and pdfmd then
+changes only what the document's `office:` block names. It is found, in order, as
+`office: {reference-doc: FILE}`, `<name>-reference.docx` or `reference.docx` (`.dotx` works too,
+`.odt`/`.ott` for ODT, `.pptx`/`.potx` for slides) beside the document, in its `metadata/` folder or
+beside its metadata files, and finally in `~/.config/pdfmd/`. Pandoc's own `--reference-doc` is left
+alone. Only builds whose output is `.docx`/`.odt`/`.pptx` look for one. `pdfmd --init-reference`
+writes `reference.docx` (also `odt`, `pptx`): Pandoc's styles with pdfmd's look, to restyle in Word.
+
+A template keeps its own style names. `office: {styles: {Heading1: LRH1, BodyText: LRNormal, Table: TableGrid}}`
+makes the Heading 1, Body Text and table Pandoc writes look like the template's own `LR H1`, `LR Normal` and
+`Table Grid` (a missing Pandoc style is created from the source); `office: {replace: {NULAB-KICKER: "Course
+wording"}}` rewrites text in the template's headers and footers; `office: {title-page: true}` gives the first
+page a page of its own, without header or footer.
+
+**A LaTeX package can tell pdfmd about itself.** Beside a package's `.sty` (or in an `office/` or `pdfmd/` folder there)
+pdfmd looks for `<package>-pdfmd.yaml`, and for its Word support: `<package>-reference.docx`/`.dotx`,
+`<package>-office.lua` (the profile above) and `<package>-office.yaml` (the `office:` defaults, with `by-option:`
+entries for what an option of `\usepackage[opt]{package}` adds). A document that loads the package gets all of it with
+no setting of its own, and a package without these files behaves exactly as before.
+
+`<package>-pdfmd.yaml` holds `latex-keys:`, the front-matter keys the package's macros are filled from
+(`experiment: LabExperiment` turns `experiment: Titration` into `\renewcommand{\LabExperiment}{Titration}` after the
+preamble), and an optional `office:` section, the same as `<package>-office.yaml`, for a package that wants one file:
+
+```yaml
+latex-keys:
+  experiment: LabExperiment
+  group: LabGroup
+office:
+  reference-doc: house.dotx
+```
+
+```yaml
+pdfmd-options:
+  office:
+    reference-doc: ../templates/house.dotx   # or leave it to be found
+    papersize: a4                            # change just this on top of the template
+    fonts: exact                             # keep STIX Two Text and friends as named
+    style: plain                             # no pdfmd look (headings, title block, table rules)
+```
+
+`--no-auto officeref` turns the search off, `--no-auto officestyle` leaves Pandoc's default alone.
+
+**LaTeX in the document.** Pandoc turns `$x^2$` into a Word equation and drops everything else of
+LaTeX. For `.docx` and `.odt` pdfmd adds a filter that makes it native where it can:
+
+- `\ce{Ag+(aq) + Cl-(aq) -> AgCl(s)}` (mhchem) and `\si{\gram\per\mole}`, `\SI{5}{\milli\liter}`,
+  `\num{1.5e-3}` (siunitx) become formatted text with real subscripts, charges and units, and, inside
+  math, Word equation pieces;
+- `\begin{equation}`, `align`, `reaction`... become Word equations with their number at the right;
+- raw `figure`, `tabular`, lists, `\emph`, `\newpage`... are read by Pandoc's own LaTeX reader;
+- `\ref`, `\eqref`, `\cref`, `\Cref`, `\pageref` and the "Figure 3." in front of a caption use the numbers
+  LaTeX itself gave (pdfmd builds the PDF once, in its cache, and reads the `.aux`; reused while the sources
+  are older), with the caption separator (`labelsep=`) and `\crefname`s of your preamble;
+- **what is left is drawn by LaTeX**: tikz, chemfig, a house style's own macros, math Pandoc cannot convert are compiled in
+  *your document's own preamble* (the same fonts, packages and macros as the PDF, one fragment to a page), and the
+  pages go into the Word file as vector pictures: an SVG with a 300 dpi PNG fallback (Word 2016 and later draw the SVG,
+  LibreOffice and older readers the PNG). An inline fragment sits on the text's baseline, a wide one is scaled to the
+  text width, a figure's caption stays text with its number. `\includegraphics{plot.pdf}` and `![](plot.pdf)` (Word cannot
+  show a PDF) are converted the same way. Pictures are cached by content, so a rebuild draws only what changed
+  (`pdfmd --clear-cache` empties it). A fragment that will not compile is named in a warning and left out; the rest of
+  the document is built. Needs `pdftocairo` (Poppler) and a LaTeX engine; without them the fragments are reported only.
+- **a house style's own macros** can be told to pdfmd: `office.lua` (or `<name>-office.lua`) beside the document, in its
+  `metadata/` folder or in `~/.config/pdfmd/`, or `office: {profile: FILE}`, is a small Lua file:
+  `return {ignore = {"\\LabCaptionBegin"}, commands = {prelab = function(args) return {pandoc.Strong{pandoc.Str("[pre-lab " .. args[1] .. "]")}} end}}`
+  says which macros to drop and what the others mean in a Word file (a command may return text, or blocks such as a
+  native table); an optional `pandoc = function(doc, helpers)` hook may restructure the whole document. The helpers
+  include `inlines(tex)`/`blocks(tex)` (LaTeX read with the filter's own math, `\ce` and `\si` applied),
+  `read_file(name)` (a file the document's `\input` would find), `number_caption(table, "table")`, `labels()`,
+  `preamble()`, `meta()`, `warn(text)`; a profile can thus build a house table from the package's own data, as nulabreport
+  does for `\chemicals`.
+- what cannot be done at all is named in a warning (`-v` lists it).
+
+`pdfmd report.md --check-docx` says what a Word build would make native and what it would draw as pictures, naming the
+environment or macro behind each (`the tikzpicture environment`, `the macro \irspectrum`), without drawing or writing anything: a
+quick way to see what is left to give a recipe (see the profile below).
+
+`pdfmd-options: {office: {latex: auto}}` is the default; `latex: images` draws everything LaTeX says as a picture
+(formulas, `\ce` and units too, instead of making them native); `latex: off` or `--no-auto officelatex` skips all of it
+(Pandoc's own behaviour). `office: {labels: off}` skips the PDF build (numbers are then counted).
+
+The same filter serves the other outputs when LaTeX is the problem: **the soffice fallback** (the Word file above, then LibreOffice: `pdfmd -e soffice report.md` is the `.docx` as LibreOffice draws it) now
+keeps the page size, margins and fonts, shows equations and draws what LaTeX can; a **Typst or WeasyPrint run after LaTeX
+failed** gets what the document says in LaTeX as native text or pictures instead of nothing; and HTML, Typst and
+WeasyPrint builds do it on request (`pdfmd-options: {office: {latex: auto}}`).
+
 ### Stopping part-way
 
 `--stop-at` ends the build after a stage; everything before it runs as normal:
@@ -422,12 +639,16 @@ style, a compound library, a shared bibliography and parts.
   The TeX distribution's own packages, fonts and Pandoc are not carried.
 - **Comments.** Notes to yourself do not travel: the attached source loses its
   `<!-- -->` comments, the LaTeX preamble's `%` comments, the text between a
-  `.bib`'s entries and a CSL file's XML comments. Choose per kind:
-  `strip-comments: [preamble, bibliography]` or `{markdown: false}`
-  (`--strip-comments-in`, `--keep-comments-in`). Code blocks, `\verb`, `\url` and
-  verbatim environments are never touched, and a `%` that closes a line keeps a
-  bare `%`, because it swallows the line break. YAML metadata is merged and
-  written again, so its `#` comments are always gone.
+  `.bib`'s entries, a CSL file's XML comments and the `#` comments of YAML
+  files. Choose per kind: `strip-comments: [preamble, bibliography]` or
+  `{markdown: false}` (`--strip-comments-in`, `--keep-comments-in`). Code
+  blocks, `\verb`, `\url` and verbatim environments are never touched, and a
+  `%` that closes a line keeps a bare `%`, because it swallows the line break.
+  With `strip-comments: false` (or `metadata` kept) the attached source gives
+  back your metadata files and the document's front matter byte for byte,
+  comments and quoting included; `--restore` of such a PDF is a full source
+  copy. The data files of `.csv` blocks (`tables/*.csv`) are always stored
+  with the source.
 - **Only the bibliography entries you cite** are attached (and what they
   cross-reference), not a shared file of hundreds; `attach-bibliography: all`
   keeps it whole, and `nocite: '@*'` keeps every entry.
@@ -461,10 +682,32 @@ options of [M1ck4's pdfmd](https://github.com/M1ck4/pdfmd) (`--ocr`, `--lang`,
 work as they did. Install the reader once:
 
 ```sh
-pdfmd --install batchocr         # pip install batchocr[md] from its GitHub release
-brew install tesseract poppler   # for scanned pages (apt: tesseract-ocr poppler-utils)
+pdfmd --install batchocr         # pip install batchocr[md] from its GitHub release,
+                                 # then offers the system programs below
 ```
 
+Scanned pages also need **Tesseract** (OCR) and **Poppler** (`pdftotext`, `pdftoppm`).
+Those are separate programs with their own builds for every platform, so pdfmd does not
+bundle them; `--install batchocr` finds your package manager (Homebrew, apt, dnf, pacman,
+zypper, apk, scoop, Chocolatey or winget), shows the exact command and, on an interactive
+terminal, runs it if you say yes (`PDFMD_NO_PROMPT=1` only prints it). By hand:
+
+```sh
+brew install tesseract poppler                       # macOS (and Linuxbrew)
+sudo apt install tesseract-ocr poppler-utils         # Debian/Ubuntu; dnf: tesseract poppler-utils
+scoop install tesseract poppler                      # Windows (or choco install ...)
+```
+
+OCR languages are packs of their own. `pdfmd --install ocr` lists the Tesseract languages
+pdfmd can fetch, and `pdfmd --install ocr:rus,kaz` (or `ru`, `kazakh`, `zh`, `ja`...) downloads
+them into pdfmd's own folder, with no admin rights and each file checked against a checksum
+pinned in pdfmd (the compact `tessdata_fast` models, Apache-2.0; English and the orientation
+model come along). `pdfmd scan.pdf --lang rus` then uses that folder whenever it has every
+language you asked for, and Tesseract's own languages otherwise (a `TESSDATA_PREFIX` you set
+always wins). `pdfmd --uninstall ocr:rus` removes one. The package managers have them too:
+`brew install tesseract-lang`, `apt install tesseract-ocr-rus`, `dnf install
+tesseract-langpack-rus`. `pdfmd --check-dependencies` reports batchocr, Tesseract and Poppler.
+
 Credits and licences: batchocr's Markdown structure stages are M1ck4's
 (MIT; that project is archived) and are vendored inside batchocr with their
 licence. PyMuPDF, which they use, is AGPL-3.0: it is installed as its own
@@ -491,6 +734,25 @@ comma inside a field). The text is Helvetica, so letters outside Latin-1 become
 their base letter or `?`. `--attach-links` stores each linked file as
 `linked/PATH` and puts a paperclip in the margin beside the link.
 
+### Use it from Python
+
+```python
+import pdfmd
+
+pdfmd.convert_file("report.md", "docx")                      # -> Path("report.docx")
+pdfmd.convert_file("report.md", "pdf", "out/report.pdf", variables={"geometry": "margin=2cm"},
+                   extra_args=["--engine", "typst"])
+html = pdfmd.convert_text("# Hi\n\nSome *text*.\n", "html")   # str for text formats, bytes for pdf/docx/odt/pptx/epub
+rst = pdfmd.convert_text(open("notes.rst").read(), "docx", format="rst")
+```
+
+Each call runs the command-line tool (as `pypandoc` runs Pandoc), so a document builds exactly as `pdfmd` would build it,
+with all the auto-discovery of the document's folder, and nothing in the caller's process is touched. The source is not
+stamped or backed up unless you ask (`stamp=True`, `backup=True`). A failure raises `pdfmd.PdfmdError` (with
+`returncode`, `stdout` and `stderr`); `check=False` returns the path anyway. `metadata=` takes metadata files,
+`variables=` a dict of Pandoc variables, `extra_args=` anything else the command line takes, `cwd=` the folder it runs
+in and `timeout=` seconds.
+
 ### Other command names: `mdpdf` and `inkmd`
 
 pdfmd also installs `mdpdf` and `inkmd` commands that take those tools' own keys
@@ -643,8 +905,31 @@ Measured values:
   <img src="https://raw.githubusercontent.com/aliperdehan/pdfmd/main/docs/results.png" width="480" alt="A CSV file rendered as a table">
 </p>
 
-- The delimiter is detected from the extension (`.tsv` means tab), or set
-  with `delimiter=";"`.
+Or write the data in the document itself, in a code block so that nothing in it is read as Markdown:
+
+````markdown
+::: {.csv align="lrr"}
+```
+Compound,Mass (g),Price
+Water,18.015,1.20
+"Ethanol, dry",46.07,3.45
+```
+:::
+
+: Masses and prices {#tbl:m}
+````
+
+- Cells are read as Pandoc Markdown (`H~2~O`, `$x^2$`, `\ce{...}`, `[@key]`); `reader="gfm"` makes them plain GFM.
+- A caption is the line after the block (`: Caption {#tbl:id}`, or `Table: Caption`), or `caption="Caption"` inside the
+  braces, with the identifier as `#tbl:id` there too. A caption in the attributes wins over a line after the block.
+- A relative `file=` is found beside the document (and along the resource path), wherever Pandoc runs. With both
+  `file=` and data inside the block, the file is used.
+- The delimiter is `delimiter=";"` or its name (`semicolon`, `tab`, `pipe`, `space`, `colon`, `comma`); without one it
+  comes from the extension (`.tsv` means tab) or from the first line (whichever of `,` `;` tab `|` it holds most of).
+- Columns are aligned with `align="lcr"` (or `align="left,center,right"`), or by a second row in the data made of
+  `---`, `:--`, `:-:`, `--:` as in a pipe table; `align=` wins. If such a row is really data, say `separator="none"`.
+  `widths="5,1,1"` sets relative widths, and a second row with unequal dashes does too. Without either, pdfmd picks
+  widths that suit the contents.
 - The first row is the header unless you add `header="false"`.
 - Large files are capped at 10 rows × 7 columns, so a huge CSV can't
   silently fill 40 pages. `rows=all` or `cols=20` raise the cap. When a
@@ -652,6 +937,118 @@ Measured values:
 
 This works for every output format.
 
+#### Turning tables into CSV blocks, and back
+
+```
+pdfmd --extract-tables report.md      # every table becomes a .csv block reading tables/<name>.csv
+pdfmd --expand-tables report.md       # and back to ordinary tables
+```
+
+`--extract-tables` rewrites the document in place, after copying the old one to the backup folder, and says so. Pipe,
+simple, multiline and grid tables are all read (the ones inside code blocks are not tables). The files are named after
+the captions (`Masses and prices` gives `tables/masses-and-prices.csv`), or `table1`, `table2` by position;
+`--table-names a,b,c` names them, and a list that does not match the number of tables is an error that changes nothing.
+`--table-numbers 1,3-4` extracts just those tables (by position in the document) and leaves the rest; the names then go to
+the chosen ones. In a document in parts the data goes to the report's `tables/` folder and an unnamed table is named after
+its part (`discussion-table1.csv`).
+The caption, the alignment and the widths the document chose stay as attributes of the block, and before anything is
+written Pandoc reads the old and the new document and the tables must come out the same. A table Pandoc cannot write
+as CSV (a list inside a cell, a cell spanning rows, no header row) is left as it is, with the reason; so is every
+`.csv` block already in the document, and if the check fails the whole document is left untouched and the command exits
+with an error. A document with no front matter is built as `gfm`, which reads pipe tables only, so only those are
+touched there. `--tables-inline` puts the data
+inside the document (in a fenced block) instead of in files, `--tables-dir DIR` picks the folder, and `--dry-run` shows
+the change without making it. `--expand-tables` is the reverse, every row included.
+
+A `.csv` block that holds its data inside the document stays as it is, unless you pass `--extract-inline-csv`, which moves
+that data to a file as well. To keep one table or block as it is whatever you run, put `<!-- pdfmd: ignore -->` on the
+line above it (or under its caption); the comment survives comment stripping, because pdfmd's own comments are kept.
+A comment between a table and its caption would stop Pandoc from reading the caption as one, so there is no such
+position.
+
+### Code: wrapping and line numbers
+
+With a LaTeX engine a long line of code wraps at the margin instead of running off the page (a long path or option list
+breaks where the line is full), and `{wrap=false}` on one block, `--no-code-wrap` or `code-wrap: false` in
+`pdfmd-options` lets it run on. Line numbers are off until you ask for them:
+
+````markdown
+``` {.python .numberLines startFrom=10}
+def f(x):
+    return x * 2
+```
+````
+
+`--line-numbers` (or `pdfmd-options: {line-numbers: true}`, or the setting in `pdfmd --setup`) numbers every code block,
+`--line-numbers 5` numbers every fifth line, `{.noNumberLines}` leaves one block out, and `step=5` and `numbersep=8pt`
+adjust a single block.
+
+### Images: SVG and remote ones
+
+LaTeX reads PDF, PNG and JPEG, not SVG. In a LaTeX build pdfmd converts each SVG the document names, once, with
+whatever is installed (`rsvg-convert`, Inkscape, `cairosvg`, `svglib`, or LibreOffice, which most machines have), keeps
+the PDF by the file's content, and uses it where the document wrote the SVG: a Markdown image, a raw
+`\includegraphics{figure.svg}`, a raw `\includesvg{figure}`. A raw `\includegraphics` also gets `graphicx` loaded,
+which Pandoc only does for a Markdown image (a PNG in one used to stop the build with "undefined control sequence").
+A Markdown image from the web is fetched once and kept for a week, so the build does not need the network again, an SVG
+badge works, and a Pandoc built without HTTP support (Homebrew's is) no longer drops it; with no network there is one
+warning and the image's description stays. `--no-auto svg` and `--no-auto remoteimages` turn the two off, and
+`pdfmd --doctor` says which converter will be used.
+
+### Editable in LibreOffice, and one file that carries its data
+
+`--hybrid` makes the PDF a LibreOffice *hybrid PDF*: besides its pages it carries an OpenDocument copy of the document
+(`Original.odt`, or `.odp`, `.ods`), and when LibreOffice opens the PDF it opens that copy as a document you can edit,
+instead of the pages in Draw. An Office file (`pdfmd minutes.docx --hybrid`) is converted by LibreOffice, so the copy is
+the file itself; a Markdown file gets an `.odt` built by pdfmd with its metadata, so what LibreOffice shows is the
+Word-route rendering of the source (not the LaTeX pages). `pdfmd --restore report.pdf` writes the ODF file back out.
+`pdfmd-options: {hybrid: true}` or `pdfmd --setup` makes it the default.
+
+A single Markdown file can carry its data too: `pdfmd report.md --assemble-only --embed-metadata files -o report.one.md`
+folds in the images, CSV tables and included files the text points at (text as it is, anything else as base64), and the
+file builds on its own, in an empty folder; `pdfmd --unpack` writes them back byte for byte. It is left out of the default
+kinds because it can be big (`bundle-max-mb` caps it); for a PDF, `--bundle` stores the same files beside the source.
+
+### HTML, LaTeX, Typst and Word pieces inside Markdown
+
+Markdown can carry raw pieces: `<img>`, `<table>`, `<b>` and `<div>` (HTML), `\ce{...}` or a tikz picture (LaTeX), a
+`` ```{=typst} `` block, Word or OpenDocument XML. Pandoc keeps the ones written in the format it is writing and drops
+every other, silently, so a document that mixes them loses pieces in every build. `pdfmd-options.raw` (or `--raw`) says
+which syntaxes take part in each output family, and carries the foreign ones over:
+
+```yaml
+pdfmd-options:
+  raw:
+    tex:    [tex, html, typst]   # LaTeX builds: PDF through lualatex, xelatex ...
+    typst:  [typst, html, tex]   # Typst builds
+    html:   [html, tex, typst]   # HTML, EPUB, PDF through weasyprint and the other HTML engines (-e html)
+    office: [office, html]       # Word and OpenDocument, and PDF through soffice
+```
+
+```sh
+pdfmd report --raw               # everything for every family
+pdfmd report --raw-for html=tex,typst --raw-for tex=html      # or per family, on the command line
+```
+
+A family takes the syntaxes it lists, in this way: its own syntax stays as written; **HTML** is read by Pandoc's HTML
+reader, so an `<img>` becomes the target's image (an SVG then goes through the image conversion below), a `<table>` a
+real table, `<b>...</b>` bold text and a `<div>` around Markdown a division, even when the opening and the closing tag
+sit in separate pieces; **LaTeX** is read by Pandoc's LaTeX reader (what it does not know, a custom macro, stays LaTeX
+and is lost outside LaTeX; Word builds keep their own LaTeX route, see
+[Word and OpenDocument output](#word-and-opendocument-output)); **Typst** is drawn by `typst compile` as a cropped
+vector picture, a PDF for LaTeX, Typst and Word, an SVG for HTML. A syntax a family does not list is dropped, its own
+included, which is odd but a clear way to say "this output has no HTML in it". A family not mentioned is left to Pandoc
+as it always was, `raw: all` lists everything, and `--no-raw` turns it off for a run. Pictures are kept in pdfmd's cache by
+their content. (An HTML file that is not self-contained points at the cache; use `--self-contained` to carry them.)
+Not yet: LaTeX pieces Pandoc cannot read (tikz, chemfig) as pictures in HTML, HTML drawn as a picture, Word XML as
+anything but Word XML.
+
+**PDF images in HTML.** A browser does not show a PDF in `<img>`, an SVG it does. For an HTML build (a web page, an
+EPUB, a PDF through WeasyPrint) every image that is a PDF, `![fig](plot.pdf)` or `<img src="plot.pdf">`, is converted to
+an SVG with poppler's `pdftocairo` (or `mutool`, `pdf2svg`, `inkscape`) and stays vector; the first page is used and the
+SVG is kept by the PDF's content. It is on by default, since it only changes what would not have shown; `--no-auto
+pdfimages` turns it off. LaTeX and Typst read PDFs themselves, and Word has its own route.
+
 ### Not just Markdown
 
 ```sh
@@ -660,25 +1057,52 @@ pdfmd paper.tex          # compiled directly with a LaTeX engine: reruns until
                          # no .aux/.log clutter (--keep-aux keeps them)
 pdfmd minutes.docx       # Word/PowerPoint/Excel/ODF: converted by LibreOffice
 pdfmd analysis.qmd       # handed to Quarto, so code chunks actually run
-pdfmd page.html          # anything else Pandoc can read (give the extension)
+pdfmd page.html          # built by WeasyPrint, or a browser if the page runs scripts: no Pandoc
+pdfmd note.typ           # compiled by Typst itself: the page setup in the file is kept
+pdfmd notes.rst          # anything else Pandoc can read (give the extension)
 ```
 
+A finished `.typ` or `.html` file goes straight to its own engine, because Pandoc would read it into its own model
+and write it out again, and the page size, the styles and the scripts would not survive the trip. For a page that is
+WeasyPrint first (it is quiet and needs no browser); a page with `<script>` goes to Chrome, Chromium, Edge or Brave
+(headless, printing the page), as does one WeasyPrint cannot draw, and wkhtmltopdf, Prince, pagedjs-cli and LibreOffice
+come after. `-e edge` (or `weasyprint`, `chromium`...) picks one. As with an Office file, such a document gets none of
+pdfmd's defaults, since it is finished; `--apply-defaults` (also in `pdfmd --setup`) gives it the title, author, date,
+language, font, font size, paper and margin of its `metadata.yaml`, ahead of its own settings so those win. A `.tex`
+file that no LaTeX engine can compile is tried through Pandoc, with a warning.
+
+The reader comes from the file's extension, as in Pandoc, and a bare name is looked up as `<name>.md`, so give the
+extension for anything else. The automatic decisions about Markdown (`READER`, `TITLE`, `citeproc`, `crossref` below)
+apply to Markdown files only.
+
+**A plain-text file as Markdown (opt-in).** `pdfmd notes.txt --text-to-markdown` reads a `.txt` file the way a person typed it
+and guesses its structure: headings (capitals, `1.2 Title`, `====` underlines), bullet and numbered lists, tables from
+columns separated by runs of spaces, indented blocks as code, hard-wrapped lines joined into paragraphs. The words never
+change (the letters and digits of the result are the input's), and a text that measures as garbled (symbol soup, OCR
+noise, replacement characters) gets paragraphs only, with the rule that fired printed. `MODE` is `auto` (default),
+`force` (guess even in a garbled text) or `paragraphs`. It needs batchocr 1.2.5 (`pdfmd --install batchocr`), writes `notes.md`
+beside the text (never over a file that is there) and builds from it; with `-o x.md` the conversion is all it does.
+Off unless asked: `--text-to-markdown`, or `options: {text-to-markdown: auto}` in the config file (or `pdfmd --setup`).
+
 ## What it does automatically
 
 Most of these print an `AUTO` line, and each can be switched off individually.
 
 | `AUTO` kind | When | What happens |
 |---|---|---|
-| `READER` | no YAML front matter | reads the file as GitHub-flavoured Markdown (content-sized table columns, relaxed blank-line rules) |
-| `TITLE` | no front matter, first line is `# Title` | that heading becomes the document title, and the remaining headings move up one level |
+| `READER` | a Markdown file with no YAML front matter | reads the file as GitHub-flavoured Markdown (content-sized table columns, relaxed blank-line rules) |
+| `TITLE` | a Markdown file with no front matter, first line is `# Title` | that heading becomes the document title, and the remaining headings move up one level |
 | `MARGIN` | no margin or geometry set anywhere | 1-inch margins instead of LaTeX's wide defaults |
 | `MAINFONT` | no `mainfont:` and no `-f` | STIX Two Text (Times New Roman if it isn't installed), or a serif that has the letters when the document is mostly in a script STIX Two Text lacks (Kazakh Cyrillic, say); retried with DejaVu Serif if a glyph is still missing. A font you name is never changed except by `fallback: document` |
 | `UNICODE` | text the main font cannot draw (Arabic, Han, Greek with accents, rare symbols...) | sets each run in an installed font for its script (see *Other scripts*), instead of printing boxes |
 | `MONOFONT` | the document contains code | JetBrains Mono for code (Menlo or another installed monospace font if it isn't installed) |
-| `tablewidth` | a wide pipe table | balances column widths so the table fits the page, and leaves narrow tables at their natural width |
+| `tablewidth` | a pipe table with plain `---` separators and long lines | picks the column widths that give the fewest lines within the text width (a column never narrower than its longest word), and gives a table that fits its natural width back; unequal dashes (`--|-----`) are kept, and `table-widths: keep` in `pdfmd-options` leaves every table as written |
+| `svg`, `remoteimages` | an SVG, or an image from the web, in a LaTeX build | the SVG is converted to a PDF (once), the remote image is fetched and kept (`--no-auto svg`, `--no-auto remoteimages`) |
+| `typstdirect`, `htmldirect` | a `.typ` or `.html` file | built by Typst / WeasyPrint or a browser, not through Pandoc (`--no-auto typstdirect` for the old route) |
+| `codewrap` | code blocks in a LaTeX build | long lines wrap at the margin (`--no-code-wrap` or `--no-auto codewrap` for the old behaviour) |
 | `YAML` / `TEX` / `LUA` | project files found | attaches `metadata.yaml`, `preamble.tex`, `<name>.lua` (see below) |
-| `citeproc` | `@key` / `[@key, p. 90]` citations | adds `--citeproc`, so citations and the reference list render from your `bibliography:` without any flag (`--no-citeproc` turns it off) |
-| `crossref` | `@fig:`/`@tbl:` references | adds the `pandoc-crossref` filter, ahead of citeproc |
+| `citeproc` | `@key` / `[@key, p. 90]` citations (as Pandoc reads them: `@key` in a code span or an e-mail address is none) | adds `--citeproc`, so citations and the reference list render from your `bibliography:` without any flag (`--no-citeproc` turns it off) |
+| `crossref` | `@fig:`/`@tbl:` references or `{#fig:...}` labels | adds the `pandoc-crossref` filter, ahead of citeproc |
 | `papersize` | `pagesize: a4` (a common typo) | converts it to Pandoc's real `papersize:` |
 
 `-v` explains each decision and prints the exact command it runs:
@@ -703,6 +1127,12 @@ The cause is usually one of these automatic decisions, and `-v` names it.
 
 ### Other scripts
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/aliperdehan/pdfmd/main/docs/scripts.png" width="560" alt="examples/scripts.md: Kazakh, Greek, Arabic, Hebrew, Chinese, Japanese, Korean and Hindi text, and emoji, in one PDF">
+</p>
+
+(That is [`examples/scripts.md`](https://github.com/aliperdehan/pdfmd/blob/main/examples/scripts.md), built with no configuration.)
+
 A document that mixes scripts needs more than one font. With lualatex or
 xelatex, `pdfmd` reads the character tables of your installed fonts, finds
 the characters the main font cannot draw and sets each run of them in a font
@@ -744,10 +1174,10 @@ Emoji are set as pictures with lualatex and xelatex (which cannot draw colour
 fonts): each emoji, flag, skin tone, family or keycap is looked up in the colour
 emoji font (`pdfmd --install emoji`, a system Noto Color Emoji, or else Apple Color Emoji on
 a Mac), its picture taken out of the font file and included at the height of the text, in
-inline code and code blocks too. Apple's font is read less completely than Noto's: single
-emoji, skin tones and most joined emoji work, but flags and keycaps, and joined emoji it
-does not list, come out as their first emoji or are reported as undrawable (install Noto
-for those). The pictures
+inline code and code blocks too. Apple's font is read through its glyph names (the same
+way for single emoji, skin tones, flags, keycaps and the joined emoji it has a glyph for,
+such as a woman scientist); the few it composes in tables pdfmd does not read (a family of
+four, say) are reported as undrawable (install Noto for those). The pictures
 are kept in pdfmd's cache folder; a `.tex` written with `--to latex` points
 there.
 
@@ -831,6 +1261,27 @@ options:
   pdf-engine: lualatex
 ```
 
+**`pdfmd --setup`** changes the same defaults from a menu, without editing the files: a numbered list (nothing to
+install, works over ssh), each setting with its choices and what they mean. It covers what a document would otherwise
+say in `pdfmd-options:` or its front matter, for every document:
+
+- *global metadata* (a `metadata.yaml` beside the config file, below anything a document's own folder says): author, language,
+  paper size, font size, page geometry, main and code font, line spacing, coloured links, table of contents, and for
+  citations the bibliography file, the CSL style and linked citations;
+- characters a font lacks or none draws, fonts for other scripts, romanization packs;
+- the citation engine, the PDF engine, the default output format, HTML (one file, math method), LaTeX and reference
+  numbers in Word files;
+- PDF finishing (header, footer, bookmarks, attached links) and the source file (the "Compiled with..." stamp, backups,
+  comment stripping, attaching the source and its files to the PDF);
+- the cache, reading `.txt` as Markdown, and which of pdfmd's automatic behaviours (`--no-auto` kinds) are off.
+
+Choosing 0 (for a text, `-`) gives a setting its built-in default back; `s` saves, `q` quits. The files are rewritten without your
+comments (the old one is kept as `.bak`). The same keys can be written by hand under `options:` of `config.yaml`. A
+full-screen version is optional: `pdfmd --install tui` (prompt_toolkit, about 1 MB), then `pdfmd --setup fancy`, or set
+"This setup screen" to fancy to make it the one `pdfmd --setup` opens. The first time pdfmd finishes a build in a terminal
+with no config file yet, it prints one `TIP` line about `--setup` (never in a pipe, in CI, or from the Python API;
+`PDFMD_NO_PROMPT=1` silences it).
+
 The decisions pdfmd remembers for you (the Lua filters it trusts, an offer you
 dismissed) live in the same folder, not in `~/.cache/pdfmd`, so cleaning the cache
 (`--clear-cache`) cannot lose them; files from an older version are moved there
@@ -864,15 +1315,16 @@ One shared metadata file can also be **symlinked** into many folders. A
 relative `bibliography: refs.bib` inside it then finds the `refs.bib`
 next to the file's real copy, so you never need an absolute path.
 
-## Build stamps and snapshots
+## Build stamps, history and snapshots
 
-Both are off by default.
+All are off by default.
 
 - **`--stamp`** keeps a `BUILD NOTES` HTML comment at the end of the `.md`
   up to date. It records when the document was compiled and with which
   `pdfmd` version, plus the versions of any LaTeX packages you name with
   `--stamp-packages`. The comment is invisible in the PDF. `--stamp-mode history` also keeps a list of past compiles.
-- **`--backup`** saves a timestamped copy of the source into `backup/`
+- **`--backup`** saves a timestamped copy of the source into `.backups/` (a hidden folder; an existing `backup/` is kept
+  using, and pdfmd says where the first copy went)
   after every successful build. A copy is skipped when nothing changed,
   and `keep: 30` limits how many are kept.
 
@@ -881,15 +1333,45 @@ Both can be turned on for a whole folder from `metadata.yaml`:
 ```yaml
 pdfmd-options:
   stamp: true
-  backup: { dir: backup, keep: 30 }
+  backup: { dir: .backups, keep: 30 }
 ```
 
+**History in a file instead of in the document.** `--stamp-store file` (or `stamp: {store: file}`) writes each compile
+to `NAME.hst` in the backup folder, and the document itself is not edited by a build. The file is one line per entry,
+newest first, so it reads in any editor, two of them join with `cat`, and `pdfmd --merge-history a.hst b.hst -o all.hst`
+keeps every entry once:
+
+```
+# pdfmd history of report.md -- one entry per line, newest first: WHEN | KIND | TEXT [| key=value ...]
+2026-10-09 11:22:01 | compiled | with nulabreport v1.27.7, pdfmd v3.25.7 | sha=3fa91c0d4b21 out=report.pdf
+```
+
+`pdfmd --history-to-file report.md` moves the whole `BUILD NOTES` block that pdfmd owns into that file (the compiles as
+entries, anything written by hand inside it as one `note` entry), and `--history-to-notes` builds the block again (both
+back the document up first, `--dry-run` shows). A document may carry several
+`BUILD NOTES` blocks (yours, an AI agent's): pdfmd writes only into the one that already holds its own lines, or the
+only one there is, or starts one of its own when there are several and none is its; the rest is never touched, and a
+line `pdfmd: ignore` in a block keeps pdfmd out of it. The file travels with the document: `--embed-metadata` folds it
+in (`hst`), `--unpack` writes it out, and `--attach-source` stores it in the PDF for `--restore`.
+
+**Going back.** `pdfmd --history report.md` lists every version it can find, newest first: the backups (in `.backups`,
+and in an older `backup/`, `backups/`... beside the document, whatever naming each file has), the git commits that
+changed the file (the repository may be a folder above), and the compiles recorded in its history. In a terminal the
+list is navigable: a number shows what differs, `d 3` the diff, `r 3` restores. Without one, `--history-diff` and
+`--history-restore` take a number, a backup's timestamp (any start of it: `20261003`), a file name, a commit hash,
+`latest`, or `previous` (the newest that is not the file as it is). A restore backs the file up first, so nothing is
+ever lost; it keeps the build notes and history of the file as it is now, and adds a `Restored version ...` entry
+(`stamp: {restored-note: false}` or `--no-restored-note` leave it out; `--dry-run` shows what would change).
+`pdfmd --init-backups report.md` turns all of this on for a document: backups and a history file, set in its front
+matter under an `AUTO GENERATED` comment (`--global` for every document), and a first copy.
+
 Separately, when `pypdf` is installed, every PDF built by `pdfmd` gets two
 hidden metadata keys, `PdfmdVersions` and `PdfmdBuildDate`, which you can
 read with `pdfinfo -meta`. Turn this off with `--no-stamp-pdf-metadata`.
 
 ## Reference
 
+- [`docs/recipes.md`](https://github.com/aliperdehan/pdfmd/blob/main/docs/recipes.md) is the task-by-task manual: fifteen chapters of "how do I..." with the commands (the basics, a house style, long documents, tables, code and images, raw HTML/LaTeX/Typst, other inputs and outputs, the PDF that carries its source, history, fonts and scripts, speed, the editor and VS Code, and what to do when something looks wrong), plus an option reference by task.
 - `pdfmd --help` lists every flag.
 - The docstring at the top of [`pdfmd.py`](https://github.com/aliperdehan/pdfmd/blob/main/pdfmd.py) is the full reference
   for each behaviour and its edge cases.