@effortlessmotion/pdf-to-md 0.1.7 → 0.1.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +69 -76
  2. package/package.json +2 -8
package/README.md CHANGED
@@ -1,116 +1,109 @@
1
- # pdf-to-md
1
+ # @effortlessmotion/pdf-to-md
2
2
 
3
- > Convert PDFs — born-digital or scanned — into markdown you can trust, under an explicit
4
- > contract: recognized text is never silently lost, unrecoverable structure is marked rather
5
- > than guessed, and every claim the output makes is mechanically checkable. When the defaults
6
- > misread your document, you get the measured knobs, the detectors that point at exactly what to
7
- > distrust, and the raw analysis to converge on a lossless conversion yourself.
3
+ Convert born-digital and scanned PDFs to Markdown. `pdf-to-md` reads an existing text layer first and falls back to local OCR for image-only pages. It returns a QA report and marks text it cannot place reliably instead of silently dropping it.
8
4
 
9
- Pure TypeScript, runs anywhere `npm install` works. No system dependencies, no cloud calls: the OCR engine ([tesseract.js](https://www.npmjs.com/package/tesseract.js)) and its recognition model install with the package and run fully offline.
5
+ Use it when you need editable Markdown plus evidence about conversion quality. OCR runs locally and makes no cloud calls.
6
+
7
+ ## Requirements
8
+
9
+ - Node.js 22.13 or newer
10
+ - English text for the bundled OCR model. Text-layer extraction can preserve other languages.
11
+
12
+ ## Install
10
13
 
11
14
  ```bash
12
- npm install pdf-to-md
15
+ npm install @effortlessmotion/pdf-to-md
13
16
  ```
14
17
 
15
- Requires Node >=22.13.0 — below that, the OCR path silently finds no words instead of erroring (see Development for the measured reason).
18
+ ## Convert a PDF
16
19
 
17
- ## CLI
20
+ The package installs the `pdf-to-md` executable:
18
21
 
19
22
  ```bash
20
- npx pdf-to-md book.pdf # text layer -> book.md (auto-OCR if pages are images)
21
- npx pdf-to-md scan.pdf out.md --ocr # force the OCR path
23
+ npx pdf-to-md book.pdf
22
24
  ```
23
25
 
24
- ```
25
- pdf-to-md <file.pdf> [out.md] [flags...] convert (implicit `extract`)
26
- pdf-to-md render <file.pdf> <page>[-<page>] render a page to PNG for eyeballing
27
- pdf-to-md audit <words.jsonl> <file.md> severity-1 check: every recognized word present?
28
- pdf-to-md --help | --version
26
+ This writes `book.md`. If the PDF has too little embedded text, the command automatically uses OCR and reports that choice on stderr.
27
+
28
+ Specify an output path or force OCR when needed:
29
+
30
+ ```bash
31
+ npx pdf-to-md scan.pdf notes.md --ocr
29
32
  ```
30
33
 
31
- Everyday flags: `--ocr`, `--no-ocr`, `--pages N[-M]`, `--stdout`, `--json` (QA report), `--format md|txt|raw` (default `md`; `raw` = the full `Analysis` as JSON, report included; `txt` = reading-order plain text — line-granular on the text path with no paragraph reflow, block-granular on the OCR path), `--debug-words=FILE` (severity-1 audit input, see below), `--dpi=N` (OCR render resolution). Usage errors, runtime errors, and audit loss all exit non-zero.
34
+ A conversion can produce ordinary Markdown plus visible review markers:
32
35
 
33
- ## Library
36
+ ```markdown
37
+ ## DAILY TALLIES
34
38
 
35
- ```ts
36
- import { pdfToMarkdown } from 'pdf-to-md';
39
+ Every skiff was counted at the north pier.
37
40
 
38
- const { markdown, report, warnings } = await pdfToMarkdown('book.pdf');
41
+ > [review: unstructured labels] 12 47 380 5 91
39
42
  ```
40
43
 
41
- `pdfToMarkdown` is the headline API: text layer first, falling back to OCR on its own when the pages are images of text. For explicit control over which path runs (hybrid pages — a real text layer plus a large embedded image — are a class the tool itself flags, and reading both requires naming the path), the engine-agnostic word-input seam, and the `analyze`/`toMarkdown`/`toText` raw-analysis API described below, every exported function and option is documented at its declaration — `extractText`, `extractOcr`, `analyze`, `toMarkdown`, `toText`, `auditWords`, `DEFAULT_TUNING`, and the rest — visible in editor hover and in the [API Docs](https://kmalakoff.github.io/pdf-to-md/). TSDoc is the source of truth for per-option prose; this file stays a map, not a mirror.
44
+ The excerpt is illustrative. Review markers preserve text whose structure could not be recovered reliably.
42
45
 
43
- All library functions throw a typed `PdfToMdError` (`code: 'PDF_OPEN' | 'PAGE_RANGE' | 'PAGE_RENDER' | 'OCR_PAGE_FAILED' | 'WORDS_INPUT' | 'ANALYSIS_INPUT'`) and never write to stdout/stderr or exit the process; warnings are returned data (plus an `onWarning` callback for streaming them).
46
+ ## Library use
44
47
 
45
- ## Design contract
48
+ ```js
49
+ import { pdfToMarkdown } from '@effortlessmotion/pdf-to-md';
46
50
 
47
- What this tool promises about errors, in severity order — each with the instrument that enforces it:
51
+ const { markdown, report, warnings } = await pdfToMarkdown('book.pdf');
48
52
 
49
- 1. **Silent text loss is forbidden.** Every word the engine recognizes lands in the output — body,
50
- heading, or a visible `> [floats]` aside. **Enforced by:** `--debug-words=FILE` dumps every
51
- recognized word as clean JSON lines (`{page, text, x, y, w, h, confidence?}`); the shipped
52
- `auditWords()` (or `pdf-to-md audit words.jsonl out.md`) checks the dump against the markdown
53
- and exits non-zero on any loss (`MISSING > 0`) — a first-class part of the product, usable as a CI
54
- gate, not a path into the source tree.
55
- 2. **Invisible corruption is minimized and made detectable.** Right-words-wrong-order failures
56
- (column braiding, mid-sentence splits) are what the geometry works hardest to prevent; a second
57
- class — right order, WRONG VALUE (an OCR misread spliced into otherwise word-like prose) — is
58
- what confidence scoring catches. **Enforced by:** `report.danglingLong` (the order-failure
59
- detector no output-only check can prove), `report.reviewBlocks` (blocks the geometry couldn't
60
- order, labeled rather than guessed at), and `report.lowConfidenceWords` /
61
- `pageStats[].lowConfidenceSample` (words the engine itself scored below confidence 60 — see
62
- `LOW_CONFIDENCE_THRESHOLD`'s test in `test/unit/review-blocks.test.ts` for the measured derivation).
63
- 3. **Visible misplacement is acceptable.** A fragment in the floats aside, a `> [review: ...]`
64
- block, or an odd paragraph break is something a caller can see and edit; the two classes above
65
- are not. **Enforced by:** the `> [floats]` / `> [review: reason]` markers themselves — regular
66
- enough (`^> \[(floats|review: [^\]]+)\] (.*)$`) for a downstream agent to find and handle
67
- without guessing — surfaced in `--json` as `report.floats` / `report.reviewBlocks`.
53
+ console.log(markdown);
54
+ console.log(report.path); // "text" or "ocr"
55
+ console.warn(warnings);
56
+ ```
68
57
 
69
- The QA report (`--json`) exists to make 2 and 3 reviewable: per-page stats flag suspect pages, and `pdf-to-md render` shows you the page as printed — the only real ground truth.
58
+ `pdfToMarkdown()` uses the same text-first, automatic-OCR behavior as the CLI. It returns data and does not write files or exit the process. Conversion failures use `PdfToMdError` where the library assigns a package error code; native filesystem errors can also surface when inputs cannot be read.
70
59
 
71
- **Why there's no `--floats=drop`:** an earlier version offered one, filtering `> [floats]` lines out of the output. Measured against a real 100-page document, floats held real sentence tails on pages 29, 48, and 94 — "drop" invited exactly the silent loss rule 1 forbids, so the knob was cut. Filtering them back out downstream is one line (`^> \[floats\]`) a caller can write when floats truly are noise for their document; the tool won't make that call for you.
60
+ Use `extractText()` or `extractOcr()` when you need to choose the path yourself. The [API documentation](https://kmalakoff.github.io/pdf-to-md/) lists the public functions, options and return types.
72
61
 
73
- **`closedHyphens` as a cross-check, not a target:** the text path's `report.closedHyphens` counts line-final hyphens closed against a dictionary (e.g. "creat-" + "ive" → "creative"). It exists to explain the auditor's `strict_deficits` count, not to be optimized directly: a strict deficit is usually a hyphen-rejoin firing (a word split across two lines, correctly closed into one), not lost text — `MISSING` is the loss signal, and `closedHyphens` is what explains the difference.
62
+ ## Know which path ran
74
63
 
75
- ## Tuning
64
+ The QA report identifies the extraction path and highlights output that deserves review. Important cases:
76
65
 
77
- OCR-path heuristics run on constants measured against real documents — the derivation for each lives beside its value in `src/geometry.ts`'s `DEFAULT_TUNING` (the canonical source; typedoc renders it). Override per run instead of editing source:
66
+ - The bundled OCR model recognizes English. Use text-layer extraction for PDFs whose embedded text is in another language.
67
+ - A page can contain both embedded text and a large image with additional text. The text path warns about these hybrid pages so you can compare a targeted OCR pass.
68
+ - `> [floats]` and `> [review: ...]` blocks contain recognized text that the converter could not place confidently. They are intentionally visible.
69
+ - OCR quality depends on the scan. Check low-confidence words and rendered pages before relying on the result.
78
70
 
79
- | flag | controls | default |
80
- |---|---|---|
81
- | `--float-max-words` | fragments with more words than this are body text, not decoration | `3` |
82
- | `--float-max-width` | fraction of page width above which a fragment is too wide to be decoration | `0.15` |
83
- | `--float-margin` | how far a fragment's left edge must sit from its column edge to be decoration | `0.008` |
84
- | `--para-gap` | a vertical gap this many times the line pitch is a paragraph break | `1.6` |
85
- | `--heading-scale` | lines taller than this times body height are headings | `2.0` |
86
- | `--col-max-width` | boxes wider than this are excluded from gutter detection | `0.55` |
87
- | `--col-split` | fractional x of the two-column midpoint | `0.5` |
88
- | `--line-y-tol` | vertical distance within which two boxes are the same line | `0.006` |
89
- | `--line-height-ratio` | same-line boxes with a larger height ratio don't merge | `2.4` |
90
- | `--dpi` | OCR page-render resolution; raise for tiny print or chart/label pages, `>~400` measured worse on ordinary prose pages (`test/unit/sparse-retry.test.ts`) | `288` |
71
+ Use `--json` to print the report:
91
72
 
92
- **Hybrid pages:** raising `--dpi` fixes chart/label corruption but can make scattered numerals on the same page WORSE, not better (measured: a chart page's numerals recognized at `288` dpi, then none at `600`, while the page's own text layer already had them all) — `--dpi` is a chart-LABEL fix, not a scattered-numeral fix. For hybrid pages, compare (or merge, via `analyze()`) the text-layer and OCR readings rather than trusting one dpi setting to fix everything on the page.
73
+ ```bash
74
+ npx pdf-to-md book.pdf --json
75
+ ```
93
76
 
94
- Typical loop: run with `--json`, find suspect pages in `pageStats`, `pdf-to-md render` them, then re-extract just those pages with `--pages` and an adjusted flag — or use `--format=raw` to get the full `Analysis` (words + blocks + report) and converge programmatically.
77
+ See [quality checks and tuning](https://github.com/kmalakoff/pdf-to-md/blob/master/QUALITY.md) for the audit workflow, report fields and OCR controls.
95
78
 
96
- ## For AI agents
79
+ ## Common CLI options
97
80
 
98
- ```bash
99
- npx skills add kmalakoff/pdf-to-md # -g for global, -a claude-code to target
81
+ ```text
82
+ pdf-to-md <file.pdf> [out.md] [flags]
83
+ pdf-to-md render <file.pdf> <page>[-<page>]
84
+ pdf-to-md audit <words.jsonl> <file.md>
100
85
  ```
101
86
 
102
- One skill: `pdf-to-md` for driving the QA loop above (reading `pageStats`, rendering pages as ground truth, targeted re-extraction, the severity-1 audit, and the `> [floats]` / `> [review: ...]` markers) instead of trusting the first conversion.
87
+ Common flags:
88
+
89
+ - `--ocr` forces OCR; `--no-ocr` disables automatic OCR fallback.
90
+ - `--pages N[-M]` converts a page range.
91
+ - `--stdout` writes the converted content to stdout.
92
+ - `--json` prints the QA report.
93
+ - `--format md|txt|raw` selects Markdown, plain text or analysis JSON.
94
+ - `--debug-words=FILE` writes recognized OCR words for `pdf-to-md audit`.
95
+ - `--dpi=N` changes OCR render resolution. The default is 288; higher values are not always more accurate.
96
+
97
+ Run `npx pdf-to-md --help` for the complete option list.
98
+
99
+ ## Agent skill
103
100
 
104
- ## Development
101
+ An optional skill teaches compatible coding agents how to inspect the report, render suspect pages and run targeted conversions:
105
102
 
106
103
  ```bash
107
- npm run build # tsds build -> dist/cjs + dist/esm (pretest step; plain `npm test` builds first)
108
- npm test # synthetic fixtures (ground truth by construction), a fetched+sha256-pinned
109
- # public corpus (skips offline), real engine runs, plus self-skipping
110
- # private-document extras when available locally
111
- npm run prepublishOnly # tsds validate: build + biome + depcheck + docs
104
+ npx skills add kmalakoff/pdf-to-md
112
105
  ```
113
106
 
114
- Source is TypeScript; the published package ships a built `dist/` (dual CJS + ESM via `tsds build`) because Node's native type-stripping refuses to run `.ts` files once they're under `node_modules`. Tests still run directly against source; the CLI test helper spawns the built `bin/cli.js`, which is why a build has to happen first.
107
+ ## Support and license
115
108
 
116
- `engines.node` is `>=22.13.0`, measured (not assumed): `pdfjs-dist`'s CJS build needs Node's stable synchronous `require(esm)`, and its rendering path uses a V8 `ArrayBuffer` API missing before Node 22. Don't lower this floor without re-running that measurement.
109
+ Report problems through the [GitHub issue tracker](https://github.com/kmalakoff/pdf-to-md/issues). See [CONTRIBUTING.md](https://github.com/kmalakoff/pdf-to-md/blob/master/CONTRIBUTING.md) to work on the package. The package is licensed under the [MIT License](https://github.com/kmalakoff/pdf-to-md/blob/master/LICENSE).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@effortlessmotion/pdf-to-md",
3
- "version": "0.1.7",
3
+ "version": "0.1.8",
4
4
  "description": "Extract PDFs to structured markdown: text layer via pdf.js, OCR fallback for image-only pages, pure npm and cross-platform",
5
5
  "keywords": [
6
6
  "pdf",
@@ -67,12 +67,6 @@
67
67
  "access": "public"
68
68
  },
69
69
  "allowScripts": {
70
- "tesseract.js": true,
71
- "core-js-pure": true,
72
- "@swc/core": true,
73
- "node-filename-to-dist-paths": true,
74
- "node-semvers": true,
75
- "node-version-use": true,
76
- "thread-sleep-compat": true
70
+ "tesseract.js": true
77
71
  }
78
72
  }