pdfmd-cli 3.21.5__tar.gz → 3.23.17__tar.gz

{pdfmd_cli-3.21.5/pdfmd_cli.egg-info → pdfmd_cli-3.23.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pdfmd-cli
-Version: 3.21.5
+Version: 3.23.17
 Summary: One command from Markdown to a good-looking PDF: a Pandoc wrapper with smart defaults, project-file discovery and a multi-engine fallback chain
 Author: Ali Perdekhan
 License-Expression: MIT
@@ -11,7 +11,7 @@ Classifier: Environment :: Console
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Text Processing :: Markup :: Markdown
 Classifier: Topic :: Text Processing :: Markup :: LaTeX
-Requires-Python: >=3.10
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 License-File: pdfmd_inkmd/LICENSE
@@ -23,6 +23,9 @@ Requires-Dist: pymd2pdf<0.7,>=0.6; python_version >= "3.11" and extra == "math"
 Requires-Dist: matplotlib; extra == "math"
 Provides-Extra: emoji
 Requires-Dist: inkmd<0.6,>=0.5; extra == "emoji"
+Provides-Extra: translit
+Requires-Dist: pypinyin; extra == "translit"
+Requires-Dist: anyascii; extra == "translit"
 Provides-Extra: pandoc
 Requires-Dist: pypandoc_binary; extra == "pandoc"
 Dynamic: license-file
@@ -81,7 +84,20 @@ pipx install pdfmd-cli
 ```
 
 That puts a `pdfmd` command on your PATH, with its Python dependencies, in
-its own isolated environment. `uv tool install pdfmd-cli` does the same.
+its own isolated environment. The same with uv:
+
+```sh
+uv tool install pdfmd-cli        # install; update with: uv tool upgrade pdfmd-cli
+uvx --from pdfmd-cli pdfmd file.md   # or run once without installing
+```
+
+On macOS, Homebrew installs it together with Pandoc and Typst, so nothing else
+is needed for a first PDF:
+
+```sh
+brew install aliperdehan/tap/pdfmd
+```
+
 To update later, run `pipx upgrade pdfmd-cli`. (The package is named
 `pdfmd-cli` because `pdfmd` on PyPI is an unrelated PDF-to-Markdown tool.
 The command is still `pdfmd`. For the latest unreleased code, use `pipx
@@ -124,7 +140,7 @@ Office files.
 <details>
 <summary>Without pipx</summary>
 
-`pdfmd.py` is a single file that needs Python 3.10+ (with `pdfmd_inkmd/`
+`pdfmd.py` is a single file that needs Python 3.9+ (with `pdfmd_inkmd/`
 beside it for the no-Pandoc fallback). It also runs
 directly, and `pyyaml`/`pypdf` are optional (features that need them are
 skipped with a warning):
@@ -173,8 +189,9 @@ WARN  native (inkmd): notes.md: 2 math expression(s) set as plain text (Unicode,
 OK    notes.md
 ```
 
-- **inkmd** is built in (vendored in `pdfmd_inkmd/`, about 2 MB, standard
-  library only, works offline, same input gives the same bytes). It reads
+- **[inkmd](https://github.com/eagredev/inkmd)**, written by Dylan Moir (MIT
+  licence), is built in (a copy is vendored in `pdfmd_inkmd/`, about 2 MB,
+  standard library only, works offline, same input gives the same bytes). It reads
   GitHub-flavoured Markdown: headings, emphasis, lists, task lists, tables,
   code blocks, quotes, links, images, and PNG/JPEG. It cannot typeset math,
   so formulas are set as readable text instead: Greek letters and operators
@@ -187,8 +204,11 @@ OK    notes.md
   150 MB, Python 3.11+, installs matplotlib so formulas render offline, centred,
   as real math; the few matplotlib cannot read are set as text like inkmd's).
   Footnotes, bookmarks and syntax-highlighted code come with it.
-- `pdfmd --install emoji` (`pdfmd-cli[emoji]`, about 11 MB) adds the colour
-  emoji font; without it emoji print as `[rocket]`-style labels.
+- `pdfmd --install emoji` (about 11 MB; `pdfmd-cli[emoji]` is the pip way for the
+  built-in renderer only) puts the colour emoji font in pdfmd's own fonts folder,
+  where the built-in renderer and every engine that can draw colour emoji
+  (Typst) find it; without it emoji print as `[rocket]`-style labels in the
+  built-in renderer.
 - `pdfmd --install full` leaves the fallback behind: it installs **Pandoc** (the
   real binary from PyPI's `pypandoc_binary`, about 35 MB, also `--install pandoc`
   / `pdfmd-cli[pandoc]`) and **Typst** (Typst's own release from GitHub, about
@@ -241,6 +261,23 @@ saying what it matched (`pdfmd animp`, `pdfmd glucose`). A name that fits two
 documents is an error, never a guess; `--no-auto lookup` switches the guessing
 off. An exact file name always wins.
 
+**Names in other scripts.** Russian, Ukrainian, Belarusian and Kazakh Cyrillic are
+always understood. Other scripts are off until you ask, one pack per script, so
+nobody carries tables they don't use: `--translit greek,hangul` (or
+`PDFMD_TRANSLIT=greek,hangul`) makes `pdfmd tyche` find `Τύχη.md` and
+`pdfmd unmyeong` find `운명.md`. Built in: `greek`, `armenian`, `georgian`,
+`hebrew`, `arabic` (consonants only, as the scripts write them), `hangul`
+(Revised Romanization) and `kana` (Hepburn). `han` gives pinyin without tones
+(`pdfmd mingyun` finds `命運.md`) and `other` covers every remaining script;
+they use [pypinyin](https://pypi.org/project/pypinyin/) (MIT) and
+[anyascii](https://pypi.org/project/anyascii/) (ISC), installed with
+`pdfmd --install translit` (`pip install "pdfmd-cli[translit]"`; about 3 MB).
+`--translit all` turns every pack on, `--translit none` even Cyrillic off,
+`--translit list` shows what is ready. These are for *finding* a name: plain
+letters, no tones, vowels left out where the script leaves them out. Names that
+romanize alike (homophones) are listed as ambiguous, never picked between. The
+same spelling rules find a section by its heading (`doc#tyche`).
+
 ### One section
 
 `pdfmd doc#onlyapart` builds just the section whose heading is "Only a Part"
@@ -337,6 +374,143 @@ pdfmd-options:
   html: {self-contained: true, css: style.css}
 ```
 
+### The PDF carries its own source
+
+A PDF can hold the document it was built from, so the file you send is also the
+file you can edit again. Ask for it, on the command line or in the document:
+
+```sh
+pdfmd report --attach-source     # the source goes into report.pdf as an attachment
+pdfmd report --bundle            # ... and the data and files its LaTeX reads
+pdfmd --restore report.pdf       # write the folder back into report.restored/
+```
+
+```yaml
+pdfmd-options:
+  attach-source: true            # alias: embed-source
+  bundle: true                   # or `all`: the whole folder, not just what the text points at
+  bundle-packages: true          # also your own LaTeX packages (see below)
+  strip-comments: true           # the default for the attached copy; see "Comments"
+  attach-bibliography: used      # or `all`
+```
+
+Put these in the shared metadata file and every document that finds it carries
+its source. The attachment is the assembled Markdown with the metadata,
+preamble, Lua filters, bibliography and CSL style folded in (the file
+`--assemble-only --embed-metadata` writes), plus a manifest of the original
+layout, so `--restore` gives back `report.md`, `metadata/`, `parts/` and the rest
+where they were, each as a standalone file (a symlinked metadata file or
+bibliography comes back as a copy). Any PDF viewer lists the attachments, and
+`pdfmd --restore report.pdf --list` shows them and what a rebuild needs. A
+restored folder builds the same document: checked on lab reports with a house
+style, a compound library, a shared bibliography and parts.
+
+- **Images come out of the PDF itself.** The PDF holds every picture it drew, so
+  a plain attach needs no bundle for them: a JPEG comes back byte for byte, a PNG
+  as the same pixels (alpha kept), and an SVG or PDF figure as a vector PDF
+  (`fig.svg` becomes `fig.pdf`, and the restored Markdown follows).
+- **`--bundle` is for what the PDF does not draw:** CSV data, LaTeX included
+  with `\input`, listings. They are stored as separate attachments (`files/...`),
+  found from the text, the preamble and, in turn, from the files found; also
+  those only your own TeX tree resolves (`\input{library}` from `texmf-home`).
+  `--bundle all` takes the whole folder except output and housekeeping. Over
+  100 MB (`bundle-max-mb`) only the source is attached. Data whose file name a
+  macro builds cannot be found: use `all`.
+- **Your own LaTeX packages** (a house style in `texmf-home`) are recorded with
+  their version, and `--restore` tells you what is missing; `--bundle-packages`
+  stores them too (with what they `\input` and the graphics they name).
+  The TeX distribution's own packages, fonts and Pandoc are not carried.
+- **Comments.** Notes to yourself do not travel: the attached source loses its
+  `<!-- -->` comments, the LaTeX preamble's `%` comments, the text between a
+  `.bib`'s entries and a CSL file's XML comments. Choose per kind:
+  `strip-comments: [preamble, bibliography]` or `{markdown: false}`
+  (`--strip-comments-in`, `--keep-comments-in`). Code blocks, `\verb`, `\url` and
+  verbatim environments are never touched, and a `%` that closes a line keeps a
+  bare `%`, because it swallows the line break. YAML metadata is merged and
+  written again, so its `#` comments are always gone.
+- **Only the bibliography entries you cite** are attached (and what they
+  cross-reference), not a shared file of hundreds; `attach-bibliography: all`
+  keeps it whole, and `nocite: '@*'` keeps every entry.
+- **It is opt-in, and it is everything you wrote.** An attached source includes
+  what the author would not send in an email; check `--list` before sharing.
+- `--restore` never overwrites a file and never writes outside its folder. A
+  restored Lua filter is code that pdfmd runs beside the document, so restore
+  names it: read it first if the PDF is not from you.
+- `-r/--report` builds attach their chapters too. A section build (`doc#section`)
+  attaches nothing.
+
+### PDF to Markdown
+
+The direction follows the files you give it:
+
+```sh
+pdfmd paper.pdf                  # -> paper.md, beside it
+pdfmd paper.pdf -o notes/p.md    # an exact file; -o notes/ puts notes/paper.md
+pdfmd paper.pdf --to txt         # plain text
+pdfmd scan.pdf --lang eng+rus --ocr tesseract --export-images
+```
+
+A PDF that carries its own pdfmd source (above) is restored instead; `--extract`
+(or naming an output file) reads its pages. Everything else is read by
+[batchocr](https://github.com/aliperdehan/batchocr): the text layer in reading
+order, OCR for scanned pages, headings, lists, tables, running headers removed,
+`<!-- Page N -->` markers (`--no-page-markers`, `--page-breaks`, `--keep-headers`).
+pdfmd only routes: any flag it does not know goes to batchocr unchanged, so the
+options of [M1ck4's pdfmd](https://github.com/M1ck4/pdfmd) (`--ocr`, `--lang`,
+`--export-images`, `--page-breaks`, `--preview-only`, `--stats`, `-q`, `--output`)
+work as they did. Install the reader once:
+
+```sh
+pdfmd --install batchocr         # pip install batchocr[md] from its GitHub release
+brew install tesseract poppler   # for scanned pages (apt: tesseract-ocr poppler-utils)
+```
+
+Credits and licences: batchocr's Markdown structure stages are M1ck4's
+(MIT; that project is archived) and are vendored inside batchocr with their
+licence. PyMuPDF, which they use, is AGPL-3.0: it is installed as its own
+package by batchocr's `md` extra and is never part of pdfmd. The command name
+`pdfmd` is also used by that project and by an unrelated PyPI package of the same
+name; this one installs as `pdfmd-cli` (and `pdfmd`), so if two are installed use
+`pdfmd-cli` to be sure which one runs.
+
+### Bookmarks, header and footer, linked files
+
+Ideas from [mdpdf](https://github.com/normanlorrain/mdpdf) (MIT), done on the
+finished PDF with pypdf, so they work with every engine. All off unless you ask:
+
+```sh
+pdfmd report.md --bookmarks                    # PDF bookmarks from the headings, if the engine made none
+pdfmd report.md --header "{title},,{date}" --footer ",Page {page} of {pages},{header}"
+pdfmd report.md --attach-links                 # the local files the text links to travel inside the PDF
+pdfmd report.md --pdf-author "A. Writer" --pdf-keywords "kinetics, rate"  --paper letter
+```
+
+A template is `left,middle,right`; `{page} {pages} {header} {date} {title}` are
+filled in per page (`{header}` is the current top-level heading; write `\,` for a
+comma inside a field). The text is Helvetica, so letters outside Latin-1 become
+their base letter or `?`. `--attach-links` stores each linked file as
+`linked/PATH` and puts a paperclip in the margin beside the link.
+
+### Other command names: `mdpdf` and `inkmd`
+
+pdfmd also installs `mdpdf` and `inkmd` commands that take those tools' own keys
+and do what they did, with pdfmd behind them:
+
+```sh
+mdpdf -o out.pdf -f ",,{page}" -a "Name" -p A4 chapter1.md chapter2.md
+inkmd notes.md -o notes.pdf --page-size a4 --family times
+cat notes.md | inkmd > notes.pdf            # standard input and output, like inkmd
+inkmd notes.md -e lualatex                   # naming another engine uses it instead
+```
+
+`mdpdf` turns bookmarks and attached links on and combines several inputs into one
+PDF. `inkmd` is `pdfmd -e inkmd`; like inkmd it writes the PDF to standard output
+when you give `-o -`, or no `-o` while output is piped (at a terminal it writes
+`notes.pdf` beside the file instead). The default page is pdfmd's, not Letter: pass
+`-p letter` / `--page-size letter`. If the original `mdpdf` or `inkmd` package is
+installed too, whichever was installed last owns the command name; both accept the
+original's keys.
+
 ### Slides
 
 ```sh
@@ -436,6 +610,26 @@ PDF and reuses it until the package, the preamble, the engine or the plot's
 data file changes. Off by default; `--no-cache` for one build; `pdfmd --clear-cache`
 (or deleting the folder) is always safe.
 
+**Where the cache lives.** By default in `~/.cache/pdfmd`, one folder per document.
+Folders get moved (a lab report from Downloads into OneDrive), and a cache keyed by
+the old path would be lost: so each cache folder records which document it is, and a
+document with no cache under its new path takes over the one of the same document
+that has lost its original: the same content (SHA-256), or, if you edited it before
+moving it, the same name in a folder of the same name, when only one fits. A copy
+(the original still there) never takes the original's cache, and plot entries find
+their data files again under the new folders. Or keep the cache with the document:
+
+```yaml
+pdfmd-options:
+  cache: {aux: true, location: document}   # or: --cache-location document
+```
+
+puts it in a `.cache/pdfmd` folder beside the document (with a `CACHEDIR.TAG`, so
+backup tools skip it, and a `.gitignore`), which travels with the folder. It is
+yours to choose per document, in the config file for everything, or with
+`--cache-location`. `--clear-cache` clears both places. (The colour-emoji pictures
+LaTeX uses are in the same cache.)
+
 ### Tables straight from a CSV file
 
 ```markdown
@@ -478,7 +672,8 @@ Most of these print an `AUTO` line, and each can be switched off individually.
 | `READER` | no YAML front matter | reads the file as GitHub-flavoured Markdown (content-sized table columns, relaxed blank-line rules) |
 | `TITLE` | no front matter, first line is `# Title` | that heading becomes the document title, and the remaining headings move up one level |
 | `MARGIN` | no margin or geometry set anywhere | 1-inch margins instead of LaTeX's wide defaults |
-| `MAINFONT` | no `mainfont:` and no `-f` | STIX Two Text, retried with DejaVu Serif if any glyph is missing (Times New Roman if DejaVu isn't installed) |
+| `MAINFONT` | no `mainfont:` and no `-f` | STIX Two Text (Times New Roman if it isn't installed), or a serif that has the letters when the document is mostly in a script STIX Two Text lacks (Kazakh Cyrillic, say); retried with DejaVu Serif if a glyph is still missing. A font you name is never changed except by `fallback: document` |
+| `UNICODE` | text the main font cannot draw (Arabic, Han, Greek with accents, rare symbols...) | sets each run in an installed font for its script (see *Other scripts*), instead of printing boxes |
 | `MONOFONT` | the document contains code | JetBrains Mono for code (Menlo or another installed monospace font if it isn't installed) |
 | `tablewidth` | a wide pipe table | balances column widths so the table fits the page, and leaves narrow tables at their natural width |
 | `YAML` / `TEX` / `LUA` | project files found | attaches `metadata.yaml`, `preamble.tex`, `<name>.lua` (see below) |
@@ -506,11 +701,89 @@ file paths are shortened here.)
 **When a PDF doesn't look the way the Markdown suggests, run `-v` first.**
 The cause is usually one of these automatic decisions, and `-v` names it.
 
+### Other scripts
+
+A document that mixes scripts needs more than one font. With lualatex or
+xelatex, `pdfmd` reads the character tables of your installed fonts, finds
+the characters the main font cannot draw and sets each run of them in a font
+that can: Arabic and Hebrew right to left, Chinese/Japanese/Korean in the
+flavour the text (kana, hangul) or `lang:` points to, punctuation and marks in
+the font of the text beside them. Characters that no installed font has are
+named in a warning, never dropped or swapped for something else. `-v` shows
+which font each script got.
+
+**How much changes is up to you** (`pdfmd-options: {fallback: MODE}`, `--fallback MODE`,
+or the config file's `options:`):
+
+| `fallback:` | What happens to a character the main font lacks |
+|---|---|
+| `word` (default) | the whole word goes in one fallback font that draws all of it, so no word mixes two fonts; your main font stays for every other word, even if the document names `DejaVu Serif` |
+| `char` | just that character is set in another installed font |
+| `document` | the installed font that draws most of the document replaces the main font (the old retry-with-DejaVu idea, chosen by coverage) |
+| `off` (or `false`) | nothing: what Pandoc and TeX do without pdfmd (a missing glyph vanishes or prints a box, and they warn) |
+| `box` | no fallback; each missing character is drawn as a black box, so it cannot be overlooked |
+| `error` | no fallback; the build stops, naming the missing characters |
+
+`pdfmd --check-fonts doc.md` shows what would happen without building: the main font (and
+why it was changed, if it was), which scripts get which fallback font, the characters no
+installed font draws and the `--install fonts:...` that would fix it; it exits 1 when
+something can't be drawn.
+
+Code is held to the same standard: a character the monofont lacks (an arrow, a CJK
+comment) in inline code or a code block is set in a fallback font under LaTeX (a code
+block that needs one is typeset as plain verbatim, without syntax highlighting), and
+through Typst's `codefont` list or a CSS font stack on Typst and WeasyPrint. Math is
+left to its own font.
+
+`missing: warn | box | error` (`--missing`) says what happens to a character that *no* installed
+font draws, after the fallback (`box` and `error` as a fallback mode imply it). `char`,
+`word` and `document` also turn emoji into pictures under LaTeX; `off`, `box` and
+`error` leave them alone.
+
+Emoji are set as pictures with lualatex and xelatex (which cannot draw colour
+fonts): each emoji, flag, skin tone, family or keycap is looked up in the colour
+emoji font (`pdfmd --install emoji`, a system Noto Color Emoji, or else Apple Color Emoji on
+a Mac), its picture taken out of the font file and included at the height of the text, in
+inline code and code blocks too. Apple's font is read less completely than Noto's: single
+emoji, skin tones and most joined emoji work, but flags and keycaps, and joined emoji it
+does not list, come out as their first emoji or are reported as undrawable (install Noto
+for those). The pictures
+are kept in pdfmd's cache folder; a `.tex` written with `--to latex` points
+there.
+
+The same fallback works with Typst (`-e typst`: `#text(font: ...)` runs, and
+the fonts folder handed to Typst) and WeasyPrint (styled spans). WeasyPrint draws
+Noto Color Emoji's bitmaps badly, so emoji are left to the system's own emoji font
+there; Typst draws them well.
+
+**Getting fonts.** Anything installed on the machine is used. For the rest,
+`pdfmd --install fonts` lists what pdfmd can fetch and
+`pdfmd --install fonts:arabic,cjk-sc` (or `fonts:core`, `fonts:scripts`,
+`fonts:all`; also `fonts:ja`, `fonts:kazakh`...) fetches it into pdfmd's own
+folder (`~/.local/share/pdfmd/fonts`, `%LOCALAPPDATA%\pdfmd\fonts`; no admin
+rights, nothing system-wide). Every file is checked against a checksum pinned in
+pdfmd before it is kept, and each package keeps its licence beside it (all SIL OFL,
+except DejaVu's own free licence). The list: STIX Two (pdfmd's default main font),
+JetBrains Mono (its default code font), Noto Serif, Amiri and Noto Naskh Arabic,
+Hebrew, Armenian, Georgian, the Indic and Southeast Asian scripts, Syriac,
+Thaana, Cherokee, Oriya, Mongolian, Coptic, Gothic, Runic, Tifinagh, N'Ko,
+Canadian syllabics, Urdu Nastaliq (used for `lang: ur`), sans-serif Indic faces,
+symbols, Noto Serif for Chinese (simplified/traditional), Japanese and Korean (one
+weight each, bold is synthesised) and the Noto Sans CJK ones (`cjk-sans`), DejaVu,
+and colour emoji. When a document needs a script no installed font has, the warning names
+the package to install. `pdfmd --uninstall fonts:NAME` removes some.
+
+It stays out of the way of a document that has set up its own fonts per
+script (`ucharclasses`, `\newfontfamily`, `xeCJK`, `\babelfont`,
+`CJKmainfont`, `mainfontfallback`): `pdfmd-options: {unicode: true}` adds
+pdfmd's anyway, and `--no-auto unicode` turns it off.
+
 ### Switching defaults off
 
 ```sh
 pdfmd lecture --no-auto                  # everything off: close to plain pandoc
-pdfmd lecture --no-auto margin mainfont  # only these
+pdfmd lecture --no-auto margin mainfont  # only these (kinds: reader title margin mainfont monofont tablewidth
+                                           #  unicode lookup yaml tex lua ...; `pdfmd --help` lists them)
 ```
 
 ### Passing options to Pandoc
@@ -541,6 +814,28 @@ that kind of engine. For example, a document full of chemical structures
 can require TeX, and a plain memo can skip the slow LaTeX run. `-e` on the
 command line always wins, and a bare `-e` means "try every engine".
 
+### Your own defaults: the config file
+
+`pdfmd --init-config` writes a commented `~/.config/pdfmd/config.yaml`
+(`$XDG_CONFIG_HOME`; `%APPDATA%\pdfmd` on Windows), `pdfmd --show-config` says
+where it is and what it sets. Its `options:` block takes the same keys as a
+document's `pdfmd-options:` and applies to every document, below anything the
+document or its metadata files say (the command line beats all of them);
+`translit:` is the romanization packs of the name lookup. `PDFMD_CONFIG=other.yaml`
+uses another file, `PDFMD_CONFIG=` (empty) none.
+
+```yaml
+translit: [greek, hangul]
+options:
+  fallback: word
+  pdf-engine: lualatex
+```
+
+The decisions pdfmd remembers for you (the Lua filters it trusts, an offer you
+dismissed) live in the same folder, not in `~/.cache/pdfmd`, so cleaning the cache
+(`--clear-cache`) cannot lose them; files from an older version are moved there
+the first time they are needed.
+
 ## Project files
 
 `pdfmd` looks next to the document, and in a `metadata/` folder beside it,