pdfmd-cli 3.23.17__tar.gz → 3.24.19__tar.gz

{pdfmd_cli-3.23.17/pdfmd_cli.egg-info → pdfmd_cli-3.24.19}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pdfmd-cli
-Version: 3.23.17
+Version: 3.24.19
 Summary: One command from Markdown to a good-looking PDF: a Pandoc wrapper with smart defaults, project-file discovery and a multi-engine fallback chain
 Author: Ali Perdekhan
 License-Expression: MIT
@@ -28,6 +28,8 @@ Requires-Dist: pypinyin; extra == "translit"
 Requires-Dist: anyascii; extra == "translit"
 Provides-Extra: pandoc
 Requires-Dist: pypandoc_binary; extra == "pandoc"
+Provides-Extra: tui
+Requires-Dist: prompt_toolkit>=3; extra == "tui"
 Dynamic: license-file
 
 # pdfmd
@@ -74,6 +76,16 @@ turns that knowledge into defaults:
 - **It doesn't give up on the first engine.** If `lualatex` fails or isn't
   installed, it tries the next engine, then the next, and tells you why each
   one failed.
+- **It handles every script and emoji.** Text the main font cannot draw (Arabic,
+  Hebrew, Chinese, Japanese, Korean, Greek, Cyrillic with Kazakh letters, Indic and
+  more) is set run by run in an installed font for its script, emoji become colour
+  pictures, and `pdfmd --install fonts` fetches missing fonts without admin rights
+  (see [Other scripts](#other-scripts)).
+- **It finds documents by name, in any script.** `pdfmd glyukoza` opens
+  `Глюкоза.md`; with `--translit` packs, `pdfmd tyche` finds `Τύχη.md` and
+  `pdfmd mingyun` finds `命運.md`.
+- **It reads PDFs too.** `pdfmd paper.pdf` restores a PDF's own embedded source or
+  OCRs the pages to Markdown (see [PDF to Markdown](#pdf-to-markdown)).
 - **It shows what it did.** Every automatic decision prints an `AUTO` line,
   `-v` shows the exact Pandoc command, and every default can be switched off.
 
@@ -121,8 +133,12 @@ brew install --cask mactex-no-gui           # optional: full LaTeX (large)
 
 ```sh
 sudo apt install pipx pandoc texlive-xetex  # Debian/Ubuntu
+sudo dnf install pipx pandoc texlive-xetex  # Fedora (add Typst with `pdfmd --install typst`)
+sudo pacman -S python-pipx pandoc-cli typst texlive   # Arch
 ```
 
+On any Linux, `pdfmd --install full` (Pandoc and Typst) needs no package manager and no root.
+
 ```powershell
 py -m pip install --user pipx; py -m pipx ensurepath          # Windows
 winget install --id JohnMacFarlane.Pandoc; winget install --id Typst.Typst
@@ -175,6 +191,34 @@ OK    quarto (only needed for .qmd files)  (/usr/local/bin/quarto)
 The numbers are the fallback order, and also shortcuts: `-e 6` means
 `-e typst`.
 
+For everything at once, `pdfmd --doctor` is one page on the Markdown-to-PDF tools, fonts
+and emoji, PDF reading (batchocr, Tesseract, Poppler, OCR languages), your config file and
+the cache, ending with the command that fixes each missing piece:
+
+```console
+$ pdfmd --doctor
+...
+Reading PDFs (PDF to Markdown):
+OK    batchocr 1.2.4
+MISS  tesseract: not found
+MISS  poppler: not found
+
+1 thing(s) to fix:
+  1. tesseract (scanned PDFs) and poppler (scanned PDFs)
+       brew install tesseract poppler
+```
+
+### Tab completion
+
+`pdfmd --completion bash|zsh|fish` prints a completion script, generated from the command
+line itself so it never goes out of date:
+
+```sh
+pdfmd --completion zsh > ~/.zfunc/_pdfmd                              # ~/.zfunc in $fpath, then compinit
+pdfmd --completion bash > ~/.local/share/bash-completion/completions/pdfmd
+pdfmd --completion fish > ~/.config/fish/completions/pdfmd.fish
+```
+
 ### No Pandoc? The built-in fallback
 
 On a machine with no Pandoc, or Pandoc but no PDF engine, `pdfmd` does not
@@ -305,6 +349,110 @@ pdfmd lecture --to typst -o lecture.typ
 pdfmd lecture -o lecture.tex      # a complete, compilable .tex, not a fragment
 ```
 
+### Word and OpenDocument output
+
+Plain Pandoc ignores `papersize`, `geometry`, `mainfont` and `fontsize` when it writes `.docx`
+or `.odt`: you get a Letter page in the default fonts. pdfmd writes them into the *reference
+document* Pandoc builds the file from, so the page and the text follow the same front matter as the PDF:
+
+| Front matter | In the Word file |
+|---|---|
+| `papersize: a4` (also `letter`, `legal`, `a5`, `b5`..., `classoption: [a4paper, landscape]`) | page size and orientation |
+| `geometry:` (a list, `margin=2cm`, `top=1in,left=...`, `paperwidth=...`) or `margin:` | page margins |
+| `mainfont`, `sansfont`, `monofont`, `CJKmainfont`, `fontsize`, `linestretch`, `lang`, `indent` | body, heading and code fonts, sizes, line spacing, language, first-line indent |
+
+A `.docx` carries font *names*, not fonts, and the person opening it has Times New Roman, not
+STIX Two Text. So well-known fonts Word does not ship are mapped to the closest one it does (STIX
+Two Text, Libertinus, Latin Modern to Times New Roman; Helvetica Neue, Inter to Arial; JetBrains
+Mono, Menlo, Fira Code to Consolas), with an `AUTO OFFICE` note; `office: {fonts: exact}` keeps
+your names. Without any of these keys pdfmd's own look applies: black bold headings sized like a LaTeX
+article, a centred title block, booktabs-style table rules, Times New Roman and Consolas.
+
+**Your own template.** A reference document decides the page and fonts itself, and pdfmd then
+changes only what the document's `office:` block names. It is found, in order, as
+`office: {reference-doc: FILE}`, `<name>-reference.docx` or `reference.docx` (`.dotx` works too,
+`.odt`/`.ott` for ODT, `.pptx`/`.potx` for slides) beside the document, in its `metadata/` folder or
+beside its metadata files, and finally in `~/.config/pdfmd/`. Pandoc's own `--reference-doc` is left
+alone. Only builds whose output is `.docx`/`.odt`/`.pptx` look for one. `pdfmd --init-reference`
+writes `reference.docx` (also `odt`, `pptx`): Pandoc's styles with pdfmd's look, to restyle in Word.
+
+A template keeps its own style names. `office: {styles: {Heading1: LRH1, BodyText: LRNormal, Table: TableGrid}}`
+makes the Heading 1, Body Text and table Pandoc writes look like the template's own `LR H1`, `LR Normal` and
+`Table Grid` (a missing Pandoc style is created from the source); `office: {replace: {NULAB-KICKER: "Course
+wording"}}` rewrites text in the template's headers and footers; `office: {title-page: true}` gives the first
+page a page of its own, without header or footer.
+
+**A LaTeX package can tell pdfmd about itself.** Beside a package's `.sty` (or in an `office/` or `pdfmd/` folder there)
+pdfmd looks for `<package>-pdfmd.yaml`, and for its Word support: `<package>-reference.docx`/`.dotx`,
+`<package>-office.lua` (the profile above) and `<package>-office.yaml` (the `office:` defaults, with `by-option:`
+entries for what an option of `\usepackage[opt]{package}` adds). A document that loads the package gets all of it with
+no setting of its own, and a package without these files behaves exactly as before.
+
+`<package>-pdfmd.yaml` holds `latex-keys:`, the front-matter keys the package's macros are filled from
+(`experiment: LabExperiment` turns `experiment: Titration` into `\renewcommand{\LabExperiment}{Titration}` after the
+preamble), and an optional `office:` section, the same as `<package>-office.yaml`, for a package that wants one file:
+
+```yaml
+latex-keys:
+  experiment: LabExperiment
+  group: LabGroup
+office:
+  reference-doc: house.dotx
+```
+
+```yaml
+pdfmd-options:
+  office:
+    reference-doc: ../templates/house.dotx   # or leave it to be found
+    papersize: a4                            # change just this on top of the template
+    fonts: exact                             # keep STIX Two Text and friends as named
+    style: plain                             # no pdfmd look (headings, title block, table rules)
+```
+
+`--no-auto officeref` turns the search off, `--no-auto officestyle` leaves Pandoc's default alone.
+
+**LaTeX in the document.** Pandoc turns `$x^2$` into a Word equation and drops everything else of
+LaTeX. For `.docx` and `.odt` pdfmd adds a filter that makes it native where it can:
+
+- `\ce{Ag+(aq) + Cl-(aq) -> AgCl(s)}` (mhchem) and `\si{\gram\per\mole}`, `\SI{5}{\milli\liter}`,
+  `\num{1.5e-3}` (siunitx) become formatted text with real subscripts, charges and units, and, inside
+  math, Word equation pieces;
+- `\begin{equation}`, `align`, `reaction`... become Word equations with their number at the right;
+- raw `figure`, `tabular`, lists, `\emph`, `\newpage`... are read by Pandoc's own LaTeX reader;
+- `\ref`, `\eqref`, `\cref`, `\Cref`, `\pageref` and the "Figure 3." in front of a caption use the numbers
+  LaTeX itself gave (pdfmd builds the PDF once, in its cache, and reads the `.aux`; reused while the sources
+  are older), with the caption separator (`labelsep=`) and `\crefname`s of your preamble;
+- **what is left is drawn by LaTeX**: tikz, chemfig, a house style's own macros, math Pandoc cannot convert are compiled in
+  *your document's own preamble* (the same fonts, packages and macros as the PDF, one fragment to a page), and the
+  pages go into the Word file as vector pictures: an SVG with a 300 dpi PNG fallback (Word 2016 and later draw the SVG,
+  LibreOffice and older readers the PNG). An inline fragment sits on the text's baseline, a wide one is scaled to the
+  text width, a figure's caption stays text with its number. `\includegraphics{plot.pdf}` and `![](plot.pdf)` (Word cannot
+  show a PDF) are converted the same way. Pictures are cached by content, so a rebuild draws only what changed
+  (`pdfmd --clear-cache` empties it). A fragment that will not compile is named in a warning and left out; the rest of
+  the document is built. Needs `pdftocairo` (Poppler) and a LaTeX engine; without them the fragments are reported only.
+- **a house style's own macros** can be told to pdfmd: `office.lua` (or `<name>-office.lua`) beside the document, in its
+  `metadata/` folder or in `~/.config/pdfmd/`, or `office: {profile: FILE}`, is a small Lua file:
+  `return {ignore = {"\\LabCaptionBegin"}, commands = {prelab = function(args) return {pandoc.Strong{pandoc.Str("[pre-lab " .. args[1] .. "]")}} end}}`
+  says which macros to drop and what the others mean in a Word file (a command may return text, or blocks such as a
+  native table); an optional `pandoc = function(doc, helpers)` hook may restructure the whole document. The helpers
+  include `inlines(tex)`/`blocks(tex)` (LaTeX read with the filter's own math, `\ce` and `\si` applied),
+  `read_file(name)` (a file the document's `\input` would find), `number_caption(table, "table")`, `labels()`,
+  `preamble()`, `meta()`, `warn(text)`; a profile can thus build a house table from the package's own data, as nulabreport
+  does for `\chemicals`.
+- what cannot be done at all is named in a warning (`-v` lists it).
+
+`pdfmd report.md --check-docx` says what a Word build would make native and what it would draw as pictures, naming the
+environment or macro behind each (`the tikzpicture environment`, `the macro \irspectrum`), without drawing or writing anything: a
+quick way to see what is left to give a recipe (see the profile below).
+
+`pdfmd-options: {office: {latex: off}}` or `--no-auto officelatex` skips all of it, `office: {labels: off}`
+skips the PDF build (numbers are then counted).
+
+The same filter serves the other outputs when LaTeX is the problem: **the soffice fallback** (the Word file above, then LibreOffice: `pdfmd -e soffice report.md` is the `.docx` as LibreOffice draws it) now
+keeps the page size, margins and fonts, shows equations and draws what LaTeX can; a **Typst or WeasyPrint run after LaTeX
+failed** gets what the document says in LaTeX as native text or pictures instead of nothing; and HTML, Typst and
+WeasyPrint builds do it on request (`pdfmd-options: {office: {latex: auto}}`).
+
 ### Stopping part-way
 
 `--stop-at` ends the build after a stage; everything before it runs as normal:
@@ -461,10 +609,32 @@ options of [M1ck4's pdfmd](https://github.com/M1ck4/pdfmd) (`--ocr`, `--lang`,
 work as they did. Install the reader once:
 
 ```sh
-pdfmd --install batchocr         # pip install batchocr[md] from its GitHub release
-brew install tesseract poppler   # for scanned pages (apt: tesseract-ocr poppler-utils)
+pdfmd --install batchocr         # pip install batchocr[md] from its GitHub release,
+                                 # then offers the system programs below
+```
+
+Scanned pages also need **Tesseract** (OCR) and **Poppler** (`pdftotext`, `pdftoppm`).
+Those are separate programs with their own builds for every platform, so pdfmd does not
+bundle them; `--install batchocr` finds your package manager (Homebrew, apt, dnf, pacman,
+zypper, apk, scoop, Chocolatey or winget), shows the exact command and, on an interactive
+terminal, runs it if you say yes (`PDFMD_NO_PROMPT=1` only prints it). By hand:
+
+```sh
+brew install tesseract poppler                       # macOS (and Linuxbrew)
+sudo apt install tesseract-ocr poppler-utils         # Debian/Ubuntu; dnf: tesseract poppler-utils
+scoop install tesseract poppler                      # Windows (or choco install ...)
 ```
 
+OCR languages are packs of their own. `pdfmd --install ocr` lists the Tesseract languages
+pdfmd can fetch, and `pdfmd --install ocr:rus,kaz` (or `ru`, `kazakh`, `zh`, `ja`...) downloads
+them into pdfmd's own folder, with no admin rights and each file checked against a checksum
+pinned in pdfmd (the compact `tessdata_fast` models, Apache-2.0; English and the orientation
+model come along). `pdfmd scan.pdf --lang rus` then uses that folder whenever it has every
+language you asked for, and Tesseract's own languages otherwise (a `TESSDATA_PREFIX` you set
+always wins). `pdfmd --uninstall ocr:rus` removes one. The package managers have them too:
+`brew install tesseract-lang`, `apt install tesseract-ocr-rus`, `dnf install
+tesseract-langpack-rus`. `pdfmd --check-dependencies` reports batchocr, Tesseract and Poppler.
+
 Credits and licences: batchocr's Markdown structure stages are M1ck4's
 (MIT; that project is archived) and are vendored inside batchocr with their
 licence. PyMuPDF, which they use, is AGPL-3.0: it is installed as its own
@@ -491,6 +661,25 @@ comma inside a field). The text is Helvetica, so letters outside Latin-1 become
 their base letter or `?`. `--attach-links` stores each linked file as
 `linked/PATH` and puts a paperclip in the margin beside the link.
 
+### Use it from Python
+
+```python
+import pdfmd
+
+pdfmd.convert_file("report.md", "docx")                      # -> Path("report.docx")
+pdfmd.convert_file("report.md", "pdf", "out/report.pdf", variables={"geometry": "margin=2cm"},
+                   extra_args=["--engine", "typst"])
+html = pdfmd.convert_text("# Hi\n\nSome *text*.\n", "html")   # str for text formats, bytes for pdf/docx/odt/pptx/epub
+rst = pdfmd.convert_text(open("notes.rst").read(), "docx", format="rst")
+```
+
+Each call runs the command-line tool (as `pypandoc` runs Pandoc), so a document builds exactly as `pdfmd` would build it,
+with all the auto-discovery of the document's folder, and nothing in the caller's process is touched. The source is not
+stamped or backed up unless you ask (`stamp=True`, `backup=True`). A failure raises `pdfmd.PdfmdError` (with
+`returncode`, `stdout` and `stderr`); `check=False` returns the path anyway. `metadata=` takes metadata files,
+`variables=` a dict of Pandoc variables, `extra_args=` anything else the command line takes, `cwd=` the folder it runs
+in and `timeout=` seconds.
+
 ### Other command names: `mdpdf` and `inkmd`
 
 pdfmd also installs `mdpdf` and `inkmd` commands that take those tools' own keys
@@ -643,6 +832,10 @@ Measured values:
   <img src="https://raw.githubusercontent.com/aliperdehan/pdfmd/main/docs/results.png" width="480" alt="A CSV file rendered as a table">
 </p>
 
+- Cells are read as Pandoc Markdown (`H~2~O`, `$x^2$`, `\ce{...}`, `[@key]`); `reader="gfm"` makes them plain GFM.
+- A caption is the line after the block (`: Caption {#tbl:id}`, or `Table: Caption`), or `caption="Caption"` inside the
+  braces, with the identifier as `#tbl:id` there too. A caption in the attributes wins over a line after the block.
+- A relative `file=` is found beside the document (and along the resource path), wherever Pandoc runs.
 - The delimiter is detected from the extension (`.tsv` means tab), or set
   with `delimiter=";"`.
 - The first row is the header unless you add `header="false"`.
@@ -663,6 +856,15 @@ pdfmd analysis.qmd       # handed to Quarto, so code chunks actually run
 pdfmd page.html          # anything else Pandoc can read (give the extension)
 ```
 
+**A plain-text file as Markdown (opt-in).** `pdfmd notes.txt --text-to-markdown` reads a `.txt` file the way a person typed it
+and guesses its structure: headings (capitals, `1.2 Title`, `====` underlines), bullet and numbered lists, tables from
+columns separated by runs of spaces, indented blocks as code, hard-wrapped lines joined into paragraphs. The words never
+change (the letters and digits of the result are the input's), and a text that measures as garbled (symbol soup, OCR
+noise, replacement characters) gets paragraphs only, with the rule that fired printed. `MODE` is `auto` (default),
+`force` (guess even in a garbled text) or `paragraphs`. It needs batchocr 1.2.5 (`pdfmd --install batchocr`), writes `notes.md`
+beside the text (never over a file that is there) and builds from it; with `-o x.md` the conversion is all it does.
+Off unless asked: `--text-to-markdown`, or `options: {text-to-markdown: auto}` in the config file (or `pdfmd --setup`).
+
 ## What it does automatically
 
 Most of these print an `AUTO` line, and each can be switched off individually.
@@ -703,6 +905,12 @@ The cause is usually one of these automatic decisions, and `-v` names it.
 
 ### Other scripts
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/aliperdehan/pdfmd/main/docs/scripts.png" width="560" alt="examples/scripts.md: Kazakh, Greek, Arabic, Hebrew, Chinese, Japanese, Korean and Hindi text, and emoji, in one PDF">
+</p>
+
+(That is [`examples/scripts.md`](https://github.com/aliperdehan/pdfmd/blob/main/examples/scripts.md), built with no configuration.)
+
 A document that mixes scripts needs more than one font. With lualatex or
 xelatex, `pdfmd` reads the character tables of your installed fonts, finds
 the characters the main font cannot draw and sets each run of them in a font
@@ -744,10 +952,10 @@ Emoji are set as pictures with lualatex and xelatex (which cannot draw colour
 fonts): each emoji, flag, skin tone, family or keycap is looked up in the colour
 emoji font (`pdfmd --install emoji`, a system Noto Color Emoji, or else Apple Color Emoji on
 a Mac), its picture taken out of the font file and included at the height of the text, in
-inline code and code blocks too. Apple's font is read less completely than Noto's: single
-emoji, skin tones and most joined emoji work, but flags and keycaps, and joined emoji it
-does not list, come out as their first emoji or are reported as undrawable (install Noto
-for those). The pictures
+inline code and code blocks too. Apple's font is read through its glyph names (the same
+way for single emoji, skin tones, flags, keycaps and the joined emoji it has a glyph for,
+such as a woman scientist); the few it composes in tables pdfmd does not read (a family of
+four, say) are reported as undrawable (install Noto for those). The pictures
 are kept in pdfmd's cache folder; a `.tex` written with `--to latex` points
 there.
 
@@ -831,6 +1039,27 @@ options:
   pdf-engine: lualatex
 ```
 
+**`pdfmd --setup`** changes the same defaults from a menu, without editing the files: a numbered list (nothing to
+install, works over ssh), each setting with its choices and what they mean. It covers what a document would otherwise
+say in `pdfmd-options:` or its front matter, for every document:
+
+- *global metadata* (a `metadata.yaml` beside the config file, below anything a document's own folder says): author, language,
+  paper size, font size, page geometry, main and code font, line spacing, coloured links, table of contents, and for
+  citations the bibliography file, the CSL style and linked citations;
+- characters a font lacks or none draws, fonts for other scripts, romanization packs;
+- the citation engine, the PDF engine, the default output format, HTML (one file, math method), LaTeX and reference
+  numbers in Word files;
+- PDF finishing (header, footer, bookmarks, attached links) and the source file (the "Compiled with..." stamp, backups,
+  comment stripping, attaching the source and its files to the PDF);
+- the cache, reading `.txt` as Markdown, and which of pdfmd's automatic behaviours (`--no-auto` kinds) are off.
+
+Choosing 0 (for a text, `-`) gives a setting its built-in default back; `s` saves, `q` quits. The files are rewritten without your
+comments (the old one is kept as `.bak`). The same keys can be written by hand under `options:` of `config.yaml`. A
+full-screen version is optional: `pdfmd --install tui` (prompt_toolkit, about 1 MB), then `pdfmd --setup fancy`, or set
+"This setup screen" to fancy to make it the one `pdfmd --setup` opens. The first time pdfmd finishes a build in a terminal
+with no config file yet, it prints one `TIP` line about `--setup` (never in a pipe, in CI, or from the Python API;
+`PDFMD_NO_PROMPT=1` silences it).
+
 The decisions pdfmd remembers for you (the Lua filters it trusts, an offer you
 dismissed) live in the same folder, not in `~/.cache/pdfmd`, so cleaning the cache
 (`--clear-cache`) cannot lose them; files from an older version are moved there
@@ -890,6 +1119,7 @@ read with `pdfinfo -meta`. Turn this off with `--no-stamp-pdf-metadata`.
 
 ## Reference
 
+- [`docs/recipes.md`](https://github.com/aliperdehan/pdfmd/blob/main/docs/recipes.md) has short "how do I..." answers: a paper in several scripts, a scanned PDF to Markdown, a folder with a house style, a book from many files.
 - `pdfmd --help` lists every flag.
 - The docstring at the top of [`pdfmd.py`](https://github.com/aliperdehan/pdfmd/blob/main/pdfmd.py) is the full reference
   for each behaviour and its edge cases.