@effortlessmotion/pdf-to-md 0.1.8 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. package/README.md +5 -5
  2. package/dist/cjs/analyze.d.cts +3 -1
  3. package/dist/cjs/analyze.d.ts +3 -1
  4. package/dist/cjs/analyze.js +38 -100
  5. package/dist/cjs/analyze.js.map +1 -1
  6. package/dist/cjs/audit.d.cts +6 -5
  7. package/dist/cjs/audit.d.ts +6 -5
  8. package/dist/cjs/audit.js +368 -60
  9. package/dist/cjs/audit.js.map +1 -1
  10. package/dist/cjs/cli/extract.js +13 -48
  11. package/dist/cjs/cli/extract.js.map +1 -1
  12. package/dist/cjs/collect.js +91 -35
  13. package/dist/cjs/collect.js.map +1 -1
  14. package/dist/cjs/errors.d.cts +2 -0
  15. package/dist/cjs/errors.d.ts +2 -0
  16. package/dist/cjs/errors.js.map +1 -1
  17. package/dist/cjs/extract.d.cts +20 -9
  18. package/dist/cjs/extract.d.ts +20 -9
  19. package/dist/cjs/extract.js +333 -106
  20. package/dist/cjs/extract.js.map +1 -1
  21. package/dist/cjs/lines.js +21 -10
  22. package/dist/cjs/lines.js.map +1 -1
  23. package/dist/cjs/markdown.js +140 -54
  24. package/dist/cjs/markdown.js.map +1 -1
  25. package/dist/cjs/page-marker.d.cts +1 -1
  26. package/dist/cjs/page-marker.d.ts +1 -1
  27. package/dist/cjs/page-marker.js +39 -8
  28. package/dist/cjs/page-marker.js.map +1 -1
  29. package/dist/cjs/render-analysis.d.cts +4 -2
  30. package/dist/cjs/render-analysis.d.ts +4 -2
  31. package/dist/cjs/render-analysis.js +5 -3
  32. package/dist/cjs/render-analysis.js.map +1 -1
  33. package/dist/cjs/report.d.cts +2 -2
  34. package/dist/cjs/report.d.ts +2 -2
  35. package/dist/cjs/report.js +27 -8
  36. package/dist/cjs/report.js.map +1 -1
  37. package/dist/cjs/text-analysis.d.cts +12 -0
  38. package/dist/cjs/text-analysis.d.ts +12 -0
  39. package/dist/cjs/text-analysis.js +175 -0
  40. package/dist/cjs/text-analysis.js.map +1 -0
  41. package/dist/cjs/types.d.cts +40 -3
  42. package/dist/cjs/types.d.ts +40 -3
  43. package/dist/cjs/types.js +2 -3
  44. package/dist/cjs/types.js.map +1 -1
  45. package/dist/esm/analyze.d.ts +3 -1
  46. package/dist/esm/analyze.js +18 -89
  47. package/dist/esm/analyze.js.map +1 -1
  48. package/dist/esm/audit.d.ts +6 -5
  49. package/dist/esm/audit.js +236 -29
  50. package/dist/esm/audit.js.map +1 -1
  51. package/dist/esm/cli/extract.js +10 -23
  52. package/dist/esm/cli/extract.js.map +1 -1
  53. package/dist/esm/collect.js +64 -15
  54. package/dist/esm/collect.js.map +1 -1
  55. package/dist/esm/errors.d.ts +2 -0
  56. package/dist/esm/errors.js.map +1 -1
  57. package/dist/esm/extract.d.ts +20 -9
  58. package/dist/esm/extract.js +167 -76
  59. package/dist/esm/extract.js.map +1 -1
  60. package/dist/esm/lines.js +14 -10
  61. package/dist/esm/lines.js.map +1 -1
  62. package/dist/esm/markdown.js +110 -27
  63. package/dist/esm/markdown.js.map +1 -1
  64. package/dist/esm/page-marker.d.ts +1 -1
  65. package/dist/esm/page-marker.js +15 -9
  66. package/dist/esm/page-marker.js.map +1 -1
  67. package/dist/esm/render-analysis.d.ts +4 -2
  68. package/dist/esm/render-analysis.js +9 -5
  69. package/dist/esm/render-analysis.js.map +1 -1
  70. package/dist/esm/report.d.ts +2 -2
  71. package/dist/esm/report.js +12 -7
  72. package/dist/esm/report.js.map +1 -1
  73. package/dist/esm/text-analysis.d.ts +12 -0
  74. package/dist/esm/text-analysis.js +81 -0
  75. package/dist/esm/text-analysis.js.map +1 -0
  76. package/dist/esm/types.d.ts +40 -3
  77. package/dist/esm/types.js +4 -4
  78. package/dist/esm/types.js.map +1 -1
  79. package/package.json +1 -1
  80. package/skills/pdf-to-md/SKILL.md +39 -66
@@ -1,102 +1,75 @@
1
1
  ---
2
2
  name: pdf-to-md
3
- description: "Convert a PDF to markdown with pdf-to-md and drive its QA loop to a lossless result: read the --json report's pageStats to find suspect pages, render pages to PNG as ground truth, re-extract targeted page ranges with tuning flags, and run the severity-1 word-loss audit before trusting the output. Use when converting a PDF to markdown with pdf-to-md, when a pdf-to-md conversion looks wrong (garbled reading order, misplaced text, misread words, a suspicious page), or when asked to verify a pdf-to-md conversion is lossless."
3
+ description: "Convert PDFs to Markdown with pdf-to-md, investigate missing or misordered text, and verify conversions against source pages and extraction evidence. Use for conversion, targeted re-extraction, or quality checks of pdf-to-md output."
4
4
  ---
5
5
 
6
6
  # pdf-to-md
7
7
 
8
- pdf-to-md converts PDFs to markdown: text layer via pdf.js, OCR (tesseract.js) fallback for
9
- image-only or sparse pages. Its contract: recognized text is never silently dropped,
10
- unrecoverable structure is marked (`> [review: ...]`) rather than guessed, and every claim the
11
- output makes is mechanically checkable. This skill is the QA loop that contract is built for —
12
- convert, read the report, verify against the real page, re-extract, audit.
8
+ pdf-to-md reads embedded text and uses local OCR where needed. A preservation audit checks extracted text against Markdown. Recognition accuracy, reading order, tables and diagrams require comparison with rendered source pages.
13
9
 
14
- ## 1. Convert
10
+ ## Convert with evidence
15
11
 
16
- ```
17
- pdf-to-md file.pdf --json
12
+ ```sh
13
+ pdf-to-md file.pdf out.md --json --page-markers --debug-words=words.jsonl
18
14
  ```
19
15
 
20
- Auto-detects the OCR need (falls back when the text layer is too sparse) and prints the QA report
21
- alongside the markdown. `--ocr` forces the OCR path; `--no-ocr` suppresses the fallback and keeps
22
- the sparse text-layer output. `--pages N[-M]` limits to a page range; `--stdout` writes markdown
23
- to stdout instead of a file.
16
+ Page markers connect the output to its source pages. `--debug-words` captures text-layer items or recognized OCR words before layout reconstruction. Text items can contain multiple words. Parse the dump as JSON Lines; key order is not fixed.
24
17
 
25
- ## 2. Read the QA report
18
+ `--ocr` forces recognition and `--no-ocr` retains the text path. `--pages N[-M]` selects source pages. A mixed document can use different extraction paths per page. Keep the input identity, tool version, effective options, report and dump with the run's working artifacts so later corrections use matching evidence.
26
19
 
27
- `report.pageStats` (one row per page) flags pages worth checking by hand:
20
+ ## Find pages to inspect
28
21
 
29
- - `danglingLong` — a paragraph broke off without terminal punctuation before the next one starts:
30
- a likely invisible sentence split the geometry got wrong.
31
- - `reviewBlocks` — count of `> [review: reason]` blocks: the geometry declined to guess an order
32
- and labeled the block instead of guessing. Always worth reading.
33
- - `lowConfidenceWords` / `lowConfidenceSample` — words the OCR engine itself scored below its
34
- confidence threshold; the sample is the actual suspect tokens, worst first. Probable misreads.
35
- - `junkHeadings` — lines promoted to headings by a font-size heuristic that are probably not
36
- headings.
37
- - `largeImages` (text path only) — a real text layer plus a large embedded image: text baked into
38
- the image may duplicate or replace what the text layer extracted. Don't assume the text layer
39
- is complete — compare it against `--pages N --ocr` on that page.
22
+ Read `report.pageStats` and warnings:
40
23
 
41
- A page with none of these flags isn't proof it's correct, only that these particular detectors
42
- found nothing — pair with step 3 for anything the report doesn't cover.
24
+ - `danglingLong` identifies prose that may have broken at the wrong place.
25
+ - `reviewBlocks` counts OCR blocks whose structure could not be recovered. The text path does not provide this detector; zero is not a layout guarantee.
26
+ - `lowConfidenceWords` and `lowConfidenceSample` identify uncertain OCR tokens. Missing confidence data is not a high-confidence result.
27
+ - `junkHeadings` identifies possible font-size classification errors.
28
+ - `largeImages` on text pages identifies images that may contain additional text. Compare a targeted OCR pass before replacing usable embedded text.
43
29
 
44
- ## 3. Ground truth: render the page
30
+ Also inspect sparse or empty pages and any layout the detectors do not cover. An unflagged page can still have incorrect columns, character mappings, tables or missing visual material.
45
31
 
46
- ```
32
+ ## Compare against the page
33
+
34
+ ```sh
47
35
  pdf-to-md render file.pdf N
48
36
  ```
49
37
 
50
- Renders page N to PNG at 288dpi — the same resolution the OCR path reads at (override with
51
- `--dpi`) — into the current directory (or `--out DIR`). View the PNG next to the flagged markdown;
52
- this is the only real ground truth, everything in the report is inference on top of it.
38
+ The default render is 288 DPI; use `--out DIR` for its destination. Compare the rendered page with the output, including item numbers, captions, symbols and reading order. Word presence alone does not validate these relationships.
53
39
 
54
- ## 4. Targeted re-extraction
40
+ For an OCR comparison or tuning experiment:
55
41
 
56
- Re-run just the flagged pages with an adjusted knob:
57
-
58
- ```
59
- pdf-to-md file.pdf --pages 27-31 --stdout --dpi 400
42
+ ```sh
43
+ pdf-to-md file.pdf page-N.md --pages N --ocr --dpi 400 --debug-words=page-N.words.jsonl
60
44
  ```
61
45
 
62
- Tuning flags (`--float-max-words`, `--para-gap`, `--heading-scale`, and the rest) are measured
63
- defaults — read README's tuning table for what each one controls and its default rather than
64
- guessing a value. Raising `--dpi` fixes chart/label corruption but can make scattered numerals on
65
- the same page worse, not better — try it on the flagged page only and compare, don't assume higher
66
- is strictly better.
46
+ OCR can recover image text and also damage correct embedded text. Higher DPI can improve small labels and worsen ordinary prose. Compare the result before accepting it. Read the package's `QUALITY.md` for tuning controls; their effects apply to OCR geometry.
67
47
 
68
- ## 5. Severity-1 audit
48
+ If a page needs manual repair, retain its source dump and page number. An image can preserve a diagram visually but does not recover editable text. Record unresolved character mappings or flattened tables alongside the affected content.
69
49
 
70
- Before trusting a conversion, run the audit — the only mechanically-checkable proof that no
71
- recognized word was silently dropped:
50
+ ## Audit the final Markdown
72
51
 
73
- ```
74
- pdf-to-md file.pdf out.md --debug-words=words.jsonl
52
+ ```sh
75
53
  pdf-to-md audit words.jsonl out.md
76
54
  ```
77
55
 
78
- Exits 1 if any word from the dump is missing from the markdown; `MISSING=0` in the printed summary
79
- line is the pass signal. `--debug-words` output is JSON Lines with no guaranteed key order —
80
- parse each line as JSON, never grep the file for text.
56
+ A successful audit with `MISSING=0` means the supplied extracted text was accounted for under the auditor's matching rules. Empty or malformed evidence is an error. It does not prove that the dump captured every visible word or that the result is an accurate transcription.
81
57
 
82
- ## 6. Machine-parseable markers
58
+ The auditor accepts `### pN` headings and `<!-- pN -->` comments. Audit the final file after edits and marker changes. Use the matching extraction dump for any pages replaced with a different extraction. Keep editorial additions separate while checking preservation, because added text can satisfy a missing occurrence.
83
59
 
84
- Two regex-matchable line prefixes carry content the geometry parked rather than lost:
60
+ Report preservation results, visual/structural checks, and unresolved issues separately. Avoid a blanket "lossless" claim based on a word audit.
85
61
 
86
- ```
87
- ^> \[(floats|review: [^\]]+)\] (.*)$
88
- ```
62
+ ## Programmatic repair
89
63
 
90
- `> [floats]` is a fragment (short line, margin text, etc.) set aside as decoration; `> [review:
91
- reason]` is a block the geometry couldn't order, with the reason inline. Both are visible in the
92
- markdown and safe to post-process (move, drop, re-attach) once matched.
64
+ `--format raw` returns an `Analysis` with per-page source items, blocks and a report. Text pages contain PDF text runs; OCR pages contain recognized words. Boxes use page-normalized coordinates, origin bottom-left, y increasing upward. Block `wordIndexes` link to the page's source list. Preserve that evidence when reordering blocks; rebuilding the source list from corrected output would make the audit circular.
93
65
 
94
- For programmatic convergence instead of eyeballing, `--format raw` returns the full `Analysis` as
95
- JSON: every word and block with its coordinates (normalized `[0,1]`, origin bottom-left, y up)
96
- plus the same `report` `--json` prints. Feed that into your own re-ordering or merge logic instead
97
- of re-parsing the markdown.
66
+ OCR uncertainty markers remain visible in Markdown:
67
+
68
+ ```text
69
+ > [floats] ...
70
+ > [review: reason] ...
71
+ ```
98
72
 
99
- ## Reference
73
+ Moving or deleting these blocks can remove real content. Check their source pages before changing them.
100
74
 
101
- Full flag list: `pdf-to-md --help`. Tuning table and design contract: the package README. Every
102
- exported function and option is documented at its declaration, visible in editor hover.
75
+ PDF compression is a separate operation. If requested, retain an identifiable original and verify text and visual material independently before replacing it. Lossy image compression can affect subsequent OCR and diagram checks even when extracted text stays identical.