@effortlessmotion/pdf-to-md 0.1.7 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +69 -76
- package/dist/cjs/analyze.d.cts +3 -1
- package/dist/cjs/analyze.d.ts +3 -1
- package/dist/cjs/analyze.js +38 -100
- package/dist/cjs/analyze.js.map +1 -1
- package/dist/cjs/audit.d.cts +6 -5
- package/dist/cjs/audit.d.ts +6 -5
- package/dist/cjs/audit.js +368 -60
- package/dist/cjs/audit.js.map +1 -1
- package/dist/cjs/cli/extract.js +13 -48
- package/dist/cjs/cli/extract.js.map +1 -1
- package/dist/cjs/collect.js +91 -35
- package/dist/cjs/collect.js.map +1 -1
- package/dist/cjs/errors.d.cts +2 -0
- package/dist/cjs/errors.d.ts +2 -0
- package/dist/cjs/errors.js.map +1 -1
- package/dist/cjs/extract.d.cts +20 -9
- package/dist/cjs/extract.d.ts +20 -9
- package/dist/cjs/extract.js +333 -106
- package/dist/cjs/extract.js.map +1 -1
- package/dist/cjs/lines.js +21 -10
- package/dist/cjs/lines.js.map +1 -1
- package/dist/cjs/markdown.js +140 -54
- package/dist/cjs/markdown.js.map +1 -1
- package/dist/cjs/page-marker.d.cts +1 -1
- package/dist/cjs/page-marker.d.ts +1 -1
- package/dist/cjs/page-marker.js +39 -8
- package/dist/cjs/page-marker.js.map +1 -1
- package/dist/cjs/render-analysis.d.cts +4 -2
- package/dist/cjs/render-analysis.d.ts +4 -2
- package/dist/cjs/render-analysis.js +5 -3
- package/dist/cjs/render-analysis.js.map +1 -1
- package/dist/cjs/report.d.cts +2 -2
- package/dist/cjs/report.d.ts +2 -2
- package/dist/cjs/report.js +27 -8
- package/dist/cjs/report.js.map +1 -1
- package/dist/cjs/text-analysis.d.cts +12 -0
- package/dist/cjs/text-analysis.d.ts +12 -0
- package/dist/cjs/text-analysis.js +175 -0
- package/dist/cjs/text-analysis.js.map +1 -0
- package/dist/cjs/types.d.cts +40 -3
- package/dist/cjs/types.d.ts +40 -3
- package/dist/cjs/types.js +2 -3
- package/dist/cjs/types.js.map +1 -1
- package/dist/esm/analyze.d.ts +3 -1
- package/dist/esm/analyze.js +18 -89
- package/dist/esm/analyze.js.map +1 -1
- package/dist/esm/audit.d.ts +6 -5
- package/dist/esm/audit.js +236 -29
- package/dist/esm/audit.js.map +1 -1
- package/dist/esm/cli/extract.js +10 -23
- package/dist/esm/cli/extract.js.map +1 -1
- package/dist/esm/collect.js +64 -15
- package/dist/esm/collect.js.map +1 -1
- package/dist/esm/errors.d.ts +2 -0
- package/dist/esm/errors.js.map +1 -1
- package/dist/esm/extract.d.ts +20 -9
- package/dist/esm/extract.js +167 -76
- package/dist/esm/extract.js.map +1 -1
- package/dist/esm/lines.js +14 -10
- package/dist/esm/lines.js.map +1 -1
- package/dist/esm/markdown.js +110 -27
- package/dist/esm/markdown.js.map +1 -1
- package/dist/esm/page-marker.d.ts +1 -1
- package/dist/esm/page-marker.js +15 -9
- package/dist/esm/page-marker.js.map +1 -1
- package/dist/esm/render-analysis.d.ts +4 -2
- package/dist/esm/render-analysis.js +9 -5
- package/dist/esm/render-analysis.js.map +1 -1
- package/dist/esm/report.d.ts +2 -2
- package/dist/esm/report.js +12 -7
- package/dist/esm/report.js.map +1 -1
- package/dist/esm/text-analysis.d.ts +12 -0
- package/dist/esm/text-analysis.js +81 -0
- package/dist/esm/text-analysis.js.map +1 -0
- package/dist/esm/types.d.ts +40 -3
- package/dist/esm/types.js +4 -4
- package/dist/esm/types.js.map +1 -1
- package/package.json +2 -8
- package/skills/pdf-to-md/SKILL.md +39 -66
|
@@ -1,102 +1,75 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: pdf-to-md
|
|
3
|
-
description: "Convert
|
|
3
|
+
description: "Convert PDFs to Markdown with pdf-to-md, investigate missing or misordered text, and verify conversions against source pages and extraction evidence. Use for conversion, targeted re-extraction, or quality checks of pdf-to-md output."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# pdf-to-md
|
|
7
7
|
|
|
8
|
-
pdf-to-md
|
|
9
|
-
image-only or sparse pages. Its contract: recognized text is never silently dropped,
|
|
10
|
-
unrecoverable structure is marked (`> [review: ...]`) rather than guessed, and every claim the
|
|
11
|
-
output makes is mechanically checkable. This skill is the QA loop that contract is built for —
|
|
12
|
-
convert, read the report, verify against the real page, re-extract, audit.
|
|
8
|
+
pdf-to-md reads embedded text and uses local OCR where needed. A preservation audit checks extracted text against Markdown. Recognition accuracy, reading order, tables and diagrams require comparison with rendered source pages.
|
|
13
9
|
|
|
14
|
-
##
|
|
10
|
+
## Convert with evidence
|
|
15
11
|
|
|
16
|
-
```
|
|
17
|
-
pdf-to-md file.pdf --json
|
|
12
|
+
```sh
|
|
13
|
+
pdf-to-md file.pdf out.md --json --page-markers --debug-words=words.jsonl
|
|
18
14
|
```
|
|
19
15
|
|
|
20
|
-
|
|
21
|
-
alongside the markdown. `--ocr` forces the OCR path; `--no-ocr` suppresses the fallback and keeps
|
|
22
|
-
the sparse text-layer output. `--pages N[-M]` limits to a page range; `--stdout` writes markdown
|
|
23
|
-
to stdout instead of a file.
|
|
16
|
+
Page markers connect the output to its source pages. `--debug-words` captures text-layer items or recognized OCR words before layout reconstruction. Text items can contain multiple words. Parse the dump as JSON Lines; key order is not fixed.
|
|
24
17
|
|
|
25
|
-
|
|
18
|
+
`--ocr` forces recognition and `--no-ocr` retains the text path. `--pages N[-M]` selects source pages. A mixed document can use different extraction paths per page. Keep the input identity, tool version, effective options, report and dump with the run's working artifacts so later corrections use matching evidence.
|
|
26
19
|
|
|
27
|
-
|
|
20
|
+
## Find pages to inspect
|
|
28
21
|
|
|
29
|
-
|
|
30
|
-
a likely invisible sentence split the geometry got wrong.
|
|
31
|
-
- `reviewBlocks` — count of `> [review: reason]` blocks: the geometry declined to guess an order
|
|
32
|
-
and labeled the block instead of guessing. Always worth reading.
|
|
33
|
-
- `lowConfidenceWords` / `lowConfidenceSample` — words the OCR engine itself scored below its
|
|
34
|
-
confidence threshold; the sample is the actual suspect tokens, worst first. Probable misreads.
|
|
35
|
-
- `junkHeadings` — lines promoted to headings by a font-size heuristic that are probably not
|
|
36
|
-
headings.
|
|
37
|
-
- `largeImages` (text path only) — a real text layer plus a large embedded image: text baked into
|
|
38
|
-
the image may duplicate or replace what the text layer extracted. Don't assume the text layer
|
|
39
|
-
is complete — compare it against `--pages N --ocr` on that page.
|
|
22
|
+
Read `report.pageStats` and warnings:
|
|
40
23
|
|
|
41
|
-
|
|
42
|
-
|
|
24
|
+
- `danglingLong` identifies prose that may have broken at the wrong place.
|
|
25
|
+
- `reviewBlocks` counts OCR blocks whose structure could not be recovered. The text path does not provide this detector; zero is not a layout guarantee.
|
|
26
|
+
- `lowConfidenceWords` and `lowConfidenceSample` identify uncertain OCR tokens. Missing confidence data is not a high-confidence result.
|
|
27
|
+
- `junkHeadings` identifies possible font-size classification errors.
|
|
28
|
+
- `largeImages` on text pages identifies images that may contain additional text. Compare a targeted OCR pass before replacing usable embedded text.
|
|
43
29
|
|
|
44
|
-
|
|
30
|
+
Also inspect sparse or empty pages and any layout the detectors do not cover. An unflagged page can still have incorrect columns, character mappings, tables or missing visual material.
|
|
45
31
|
|
|
46
|
-
|
|
32
|
+
## Compare against the page
|
|
33
|
+
|
|
34
|
+
```sh
|
|
47
35
|
pdf-to-md render file.pdf N
|
|
48
36
|
```
|
|
49
37
|
|
|
50
|
-
|
|
51
|
-
`--dpi`) — into the current directory (or `--out DIR`). View the PNG next to the flagged markdown;
|
|
52
|
-
this is the only real ground truth, everything in the report is inference on top of it.
|
|
38
|
+
The default render is 288 DPI; use `--out DIR` for its destination. Compare the rendered page with the output, including item numbers, captions, symbols and reading order. Word presence alone does not validate these relationships.
|
|
53
39
|
|
|
54
|
-
|
|
40
|
+
For an OCR comparison or tuning experiment:
|
|
55
41
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
```
|
|
59
|
-
pdf-to-md file.pdf --pages 27-31 --stdout --dpi 400
|
|
42
|
+
```sh
|
|
43
|
+
pdf-to-md file.pdf page-N.md --pages N --ocr --dpi 400 --debug-words=page-N.words.jsonl
|
|
60
44
|
```
|
|
61
45
|
|
|
62
|
-
|
|
63
|
-
defaults — read README's tuning table for what each one controls and its default rather than
|
|
64
|
-
guessing a value. Raising `--dpi` fixes chart/label corruption but can make scattered numerals on
|
|
65
|
-
the same page worse, not better — try it on the flagged page only and compare, don't assume higher
|
|
66
|
-
is strictly better.
|
|
46
|
+
OCR can recover image text and also damage correct embedded text. Higher DPI can improve small labels and worsen ordinary prose. Compare the result before accepting it. Read the package's `QUALITY.md` for tuning controls; their effects apply to OCR geometry.
|
|
67
47
|
|
|
68
|
-
|
|
48
|
+
If a page needs manual repair, retain its source dump and page number. An image can preserve a diagram visually but does not recover editable text. Record unresolved character mappings or flattened tables alongside the affected content.
|
|
69
49
|
|
|
70
|
-
|
|
71
|
-
recognized word was silently dropped:
|
|
50
|
+
## Audit the final Markdown
|
|
72
51
|
|
|
73
|
-
```
|
|
74
|
-
pdf-to-md file.pdf out.md --debug-words=words.jsonl
|
|
52
|
+
```sh
|
|
75
53
|
pdf-to-md audit words.jsonl out.md
|
|
76
54
|
```
|
|
77
55
|
|
|
78
|
-
|
|
79
|
-
line is the pass signal. `--debug-words` output is JSON Lines with no guaranteed key order —
|
|
80
|
-
parse each line as JSON, never grep the file for text.
|
|
56
|
+
A successful audit with `MISSING=0` means the supplied extracted text was accounted for under the auditor's matching rules. Empty or malformed evidence is an error. It does not prove that the dump captured every visible word or that the result is an accurate transcription.
|
|
81
57
|
|
|
82
|
-
|
|
58
|
+
The auditor accepts `### pN` headings and `<!-- pN -->` comments. Audit the final file after edits and marker changes. Use the matching extraction dump for any pages replaced with a different extraction. Keep editorial additions separate while checking preservation, because added text can satisfy a missing occurrence.
|
|
83
59
|
|
|
84
|
-
|
|
60
|
+
Report preservation results, visual/structural checks, and unresolved issues separately. Avoid a blanket "lossless" claim based on a word audit.
|
|
85
61
|
|
|
86
|
-
|
|
87
|
-
^> \[(floats|review: [^\]]+)\] (.*)$
|
|
88
|
-
```
|
|
62
|
+
## Programmatic repair
|
|
89
63
|
|
|
90
|
-
|
|
91
|
-
reason]` is a block the geometry couldn't order, with the reason inline. Both are visible in the
|
|
92
|
-
markdown and safe to post-process (move, drop, re-attach) once matched.
|
|
64
|
+
`--format raw` returns an `Analysis` with per-page source items, blocks and a report. Text pages contain PDF text runs; OCR pages contain recognized words. Boxes use page-normalized coordinates, origin bottom-left, y increasing upward. Block `wordIndexes` link to the page's source list. Preserve that evidence when reordering blocks; rebuilding the source list from corrected output would make the audit circular.
|
|
93
65
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
66
|
+
OCR uncertainty markers remain visible in Markdown:
|
|
67
|
+
|
|
68
|
+
```text
|
|
69
|
+
> [floats] ...
|
|
70
|
+
> [review: reason] ...
|
|
71
|
+
```
|
|
98
72
|
|
|
99
|
-
|
|
73
|
+
Moving or deleting these blocks can remove real content. Check their source pages before changing them.
|
|
100
74
|
|
|
101
|
-
|
|
102
|
-
exported function and option is documented at its declaration, visible in editor hover.
|
|
75
|
+
PDF compression is a separate operation. If requested, retain an identifiable original and verify text and visual material independently before replacing it. Lossy image compression can affect subsequent OCR and diagram checks even when extracted text stays identical.
|