extract-pdf 0.1.423 → 0.1.425
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +289 -289
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,289 +1,289 @@
|
|
|
1
|
-
<!-- template-git-repo:badges:start -->
|
|
2
|
-
<p align="center">
|
|
3
|
-
<img src="https://i.imgur.com/2ks47qa.png" /><br />
|
|
4
|
-
<a href="https://qwksearch.com/api"><img src="https://img.shields.io/badge/Docs-blue?logo=ReadTheDocs&logoColor=white" alt="Documentation" /></a>
|
|
5
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/stargazers"><img src="https://img.shields.io/github/stars/OpenSourceAGI/qwksearch-research-agent" alt="GitHub Stars" /></a>
|
|
6
|
-
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/dm/extract-pdf.svg" alt="NPM Monthly Downloads" /></a>
|
|
7
|
-
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/v/extract-pdf.svg" alt="npm version" /></a>
|
|
8
|
-
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/dt/extract-pdf.svg" alt="NPM Total Downloads" /></a>
|
|
9
|
-
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/types/extract-pdf" alt="TypeScript types" /></a>
|
|
10
|
-
<a href="https://packagephobia.com/result?p=extract-pdf"><img src="https://packagephobia.com/badge?p=extract-pdf" alt="Install size" /></a>
|
|
11
|
-
<br />
|
|
12
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/issues"><img src="https://img.shields.io/github/issues/OpenSourceAGI/qwksearch-research-agent?logo=github" alt="GitHub Issues" /></a>
|
|
13
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/pulls"><img src="https://img.shields.io/github/issues-pr/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs" alt="Open Pull Requests" /></a>
|
|
14
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/pulls?q=is%3Apr+is%3Aclosed"><img src="https://img.shields.io/github/issues-pr-closed/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs%20merged&color=8957e5" alt="Merged Pull Requests" /></a>
|
|
15
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/discussions"><img src="https://img.shields.io/github/discussions/OpenSourceAGI/qwksearch-research-agent" alt="GitHub Discussions" /></a>
|
|
16
|
-
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/commits/master/"><img src="https://img.shields.io/github/last-commit/OpenSourceAGI/qwksearch-research-agent.svg" alt="GitHub last commit" /></a>
|
|
17
|
-
<br />
|
|
18
|
-
<a href="https://stackblitz.com/github/OpenSourceAGI/qwksearch-research-agent/tree/master/packages/extract-pdf"><img height="20px" src="https://developer.stackblitz.com/img/open_in_stackblitz.svg" alt="Open in StackBlitz" /></a>
|
|
19
|
-
<img src="https://img.shields.io/badge/Bun-14151A?logo=bun&logoColor=white" alt="Bun" /> <img src="https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white" alt="TypeScript" /> <img src="https://img.shields.io/badge/Vite-646CFF?logo=vite&logoColor=white" alt="Vite" />
|
|
20
|
-
</p>
|
|
21
|
-
<!-- template-git-repo:badges:end -->
|
|
22
|
-
|
|
23
|
-
# extract-pdf
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
> **When users upload a PDF, they expect an instant chat response, not to wait for 5 min on OCR model.**
|
|
27
|
-
|
|
28
|
-
Instant no-backend-needed javascript to convert a PDF (URL or `ArrayBuffer`) into clean HTML with structural tagging — headings, lists, footnotes, code blocks, bold/italic, and Table of Contents entries. Works in Node.js, Cloudflare Workers, and browser environments via [pdfjs-serverless](https://github.com/johannschopplich/pdfjs-serverless).
|
|
29
|
-
|
|
30
|
-
**Slim by default:** the package bundles no PDF engine. PDF.js is loaded lazily at runtime from jsDelivr's ESM build of `pdfjs-serverless` (`https://cdn.jsdelivr.net/npm/pdfjs-serverless@1/+esm`, pinned to the major version) — a zero-dependency, single ~1.6 MB minified redistribution of Mozilla PDF.js built for serverless/edge runtimes. Node.js and Bun can't import remote URLs, so there the loader falls back to the locally-installed `pdfjs-serverless` optional dependency. Everything OCR-related (Granite Docling via `@huggingface/transformers`, `@napi-rs/canvas`) is likewise optional and imported only when actually used.
|
|
31
|
-
|
|
32
|
-
For OCR-grade fidelity on pages with infographics, charts, and tables, the package also ships the Granite Docling path (merged from the former `extract-pdf-docling` package): run all pages through the frontend JS parser, all pages through Docling OCR, a **hybrid** mode that regex-scans pages and OCRs only the ones that need it, or point at the URL of a remote docling-compatible processor (see `server/`).
|
|
33
|
-
|
|
34
|
-
## Install
|
|
35
|
-
|
|
36
|
-
```sh
|
|
37
|
-
bun add extract-pdf
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
##
|
|
41
|
-
|
|
42
|
-
[`
|
|
43
|
-
|
|
44
|
-
## Usage
|
|
45
|
-
|
|
46
|
-
```ts
|
|
47
|
-
import { convertPDFToHTML } from "extract-pdf";
|
|
48
|
-
|
|
49
|
-
const { html, title, author } = await convertPDFToHTML(
|
|
50
|
-
"https://example.com/paper.pdf",
|
|
51
|
-
);
|
|
52
|
-
// or pass an ArrayBuffer from fs.readFile / fetch
|
|
53
|
-
const { html } = await convertPDFToHTML(buffer, { addPageNumbers: true });
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
### Options
|
|
57
|
-
|
|
58
|
-
| Option | Default | Description |
|
|
59
|
-
| ----------------- | ---------------------- | ------------------------------------------------------------------------------------------ |
|
|
60
|
-
| `addPageNumbers` | `false` | Inserts `[n]` markers at each page boundary |
|
|
61
|
-
| `addCitation` | `true` | Reads PDF metadata and first-page heading to populate `title`/`author` in the return value |
|
|
62
|
-
| `method` | `"ts-block-algorithm"` | Parsing engine — `"ts-block-algorithm"`, `"liteparse"`, or `"liteparse-wasm"` (see below) |
|
|
63
|
-
| `liteParseOptions`| `{}` | Passed through to LiteParse's constructor when `method` is `"liteparse"` or `"liteparse-wasm"` |
|
|
64
|
-
| `processor` | `"frontend"` | Where OCR happens — `"frontend"`, `"hybrid"`, `"docling"`, or a processor URL (see below) |
|
|
65
|
-
| `processorUrl` | — | Remote docling-compatible API used by `"hybrid"`/`"docling"` instead of the in-process model |
|
|
66
|
-
| `ocrScanOptions` | `{}` | Threshold tuning for the hybrid page scan (`scanPagesForOCR`) |
|
|
67
|
-
| `doclingOptions` | `{}` | `prompt`, `maxTokens`, `scale` for the Docling OCR model, and `processorHeaders` sent to `processorUrl` |
|
|
68
|
-
|
|
69
|
-
### Return value
|
|
70
|
-
|
|
71
|
-
```ts
|
|
72
|
-
{ html: string, title?: string, author?: string, format: "pdf",
|
|
73
|
-
processor: string, ocrScan: OcrScanResult }
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
## Parse methods
|
|
77
|
-
|
|
78
|
-
`convertPDFToHTML` supports three interchangeable parsing engines via `options.method`:
|
|
79
|
-
|
|
80
|
-
| Method | Engine | Environments | OCR |
|
|
81
|
-
| ------------------------------------ | -------------------------------------------------------------------- | ----------------------------------- | --- |
|
|
82
|
-
| `"ts-block-algorithm"` (default) | The pure-TS pipeline documented below (this package) | Node.js, Cloudflare Workers, browser | No |
|
|
83
|
-
| `"liteparse"` | [LiteParse](https://github.com/run-llama/liteparse) (native, `@llamaindex/liteparse`) | Node.js only | Optional |
|
|
84
|
-
| `"liteparse-wasm"` | [LiteParse](https://github.com/run-llama/liteparse) (WASM, `@llamaindex/liteparse-wasm`) | Node.js, Cloudflare Workers, browser | Optional (via callback) |
|
|
85
|
-
|
|
86
|
-
```ts
|
|
87
|
-
import { convertPDFToHTML } from "extract-pdf";
|
|
88
|
-
|
|
89
|
-
const { html } = await convertPDFToHTML(buffer, { method: "liteparse" });
|
|
90
|
-
|
|
91
|
-
// Or the WASM build, which also runs in browsers and Cloudflare Workers:
|
|
92
|
-
const { html } = await convertPDFToHTML(buffer, { method: "liteparse-wasm" });
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
LiteParse ships a native (napi) addon, so `method: "liteparse"` only runs in
|
|
96
|
-
Node.js — it is not bundled into browser or Cloudflare Workers builds. Install
|
|
97
|
-
it explicitly (`bun add @llamaindex/liteparse`) since it's an optional
|
|
98
|
-
dependency; if it isn't installed, `convertPDFToHTML` returns `{ error }`
|
|
99
|
-
instead of throwing.
|
|
100
|
-
|
|
101
|
-
`method: "liteparse-wasm"` delegates to LiteParse's WebAssembly build instead,
|
|
102
|
-
which runs anywhere WASM does — browsers, Cloudflare Workers, and Node.js.
|
|
103
|
-
Install it explicitly (`bun add @llamaindex/liteparse-wasm`) since it's also
|
|
104
|
-
an optional dependency; if it isn't installed, `convertPDFToHTML` returns
|
|
105
|
-
`{ error }` instead of throwing. The WASM build has no OCR engine built in —
|
|
106
|
-
pass a `liteParseOptions.ocrEngine` callback (e.g. backed by `tesseract-js`)
|
|
107
|
-
to enable OCR.
|
|
108
|
-
|
|
109
|
-
By default both LiteParse paths run with OCR disabled (`ocrEnabled: false`) —
|
|
110
|
-
matching this package's "instant, no backend" philosophy. Use
|
|
111
|
-
`detectPdfNeedsOcr` (below) to decide when a document is worth re-parsing with
|
|
112
|
-
`liteParseOptions: { ocrEnabled: true }`.
|
|
113
|
-
|
|
114
|
-
## Processor modes: frontend, hybrid, Docling OCR, or a remote URL
|
|
115
|
-
|
|
116
|
-
Separately from `method`, the `processor` option decides whether pages go
|
|
117
|
-
through the [Granite Docling](https://huggingface.co/ibm-granite/granite-docling-258M)
|
|
118
|
-
OCR model — which preserves layout, tables, charts, code, and formulas at the
|
|
119
|
-
cost of model inference per page:
|
|
120
|
-
|
|
121
|
-
| `processor` | What happens |
|
|
122
|
-
| ----------------------- | --------------------------------------------------------------------------------------------------------- |
|
|
123
|
-
| `"frontend"` (default) | All pages parsed by the pure-JS text-layer pipeline. No OCR, no model, instant. |
|
|
124
|
-
| `"hybrid"` | JS pipeline everywhere + a regex scan (`scanPagesForOCR`) flags pages with infographics/figures/tables or no usable text layer; only those pages are rasterized and OCR'd with Docling. |
|
|
125
|
-
| `"docling"` | Every page rasterized and OCR'd with Docling. |
|
|
126
|
-
| `"https://..."` (URL) | Like `"docling"`, but every page is POSTed to that docling-compatible processor API instead of running the model in-process. |
|
|
127
|
-
|
|
128
|
-
```ts
|
|
129
|
-
// Hybrid: fast JS parse, OCR only the pages that need it
|
|
130
|
-
const { html, ocrScan } = await convertPDFToHTML(buffer, { processor: "hybrid" });
|
|
131
|
-
console.log(ocrScan.pagesNeedingOcr); // e.g. [3, 7] — pages with tables/figures
|
|
132
|
-
// each ocrScan.pages[i] lists reasons: "table-caption", "figure-caption",
|
|
133
|
-
// "table-rows", "numeric-grid", "no-text", "sparse-text", "garbled-text"
|
|
134
|
-
|
|
135
|
-
// All pages through a remote Docling processor (this package's server/):
|
|
136
|
-
const { html } = await convertPDFToHTML(buffer, {
|
|
137
|
-
processor: "http://localhost:3000",
|
|
138
|
-
});
|
|
139
|
-
|
|
140
|
-
// Hybrid with the OCR offloaded to a remote processor:
|
|
141
|
-
const { html } = await convertPDFToHTML(buffer, {
|
|
142
|
-
processor: "hybrid",
|
|
143
|
-
processorUrl: "http://localhost:3000",
|
|
144
|
-
});
|
|
145
|
-
```
|
|
146
|
-
|
|
147
|
-
OCR'd pages are emitted as `<section class="ocr-page" id="page-N">…</section>`
|
|
148
|
-
(tables become real `<table>` markup, captions `<figcaption>`, formulas
|
|
149
|
-
`<code class="formula">`, …); pages whose OCR fails keep their JS-parsed HTML.
|
|
150
|
-
|
|
151
|
-
Running Docling **in-process** requires the optional `@huggingface/transformers`
|
|
152
|
-
dependency (the ONNX model `onnx-community/granite-docling-258M-ONNX`, ~1 GB,
|
|
153
|
-
downloads on first use) and — in Node.js — `@napi-rs/canvas` for page
|
|
154
|
-
rasterization (browsers/Workers use `OffscreenCanvas`).
|
|
155
|
-
|
|
156
|
-
The regex scanner is also exported standalone, and so are the OCR helpers:
|
|
157
|
-
|
|
158
|
-
```ts
|
|
159
|
-
import { scanPagesForOCR, doctagsToHtml, ocrImageWithDocling } from "extract-pdf";
|
|
160
|
-
|
|
161
|
-
const scan = scanPagesForOCR(pageTexts); // string[] of per-page text
|
|
162
|
-
// { needsOcr, pagesNeedingOcr: number[], pages: [{ page, reasons, captions }] }
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
### Running the Docling processor server
|
|
166
|
-
|
|
167
|
-
The `server/` folder contains the Hono HTTP API (formerly the
|
|
168
|
-
`extract-pdf-docling` package) exposing the Granite Docling model:
|
|
169
|
-
|
|
170
|
-
```sh
|
|
171
|
-
bun run serve:docling # or dev:docling for --watch; port 3000
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
- Swagger UI: `http://localhost:3000/docs`, spec at `/openapi.json`
|
|
175
|
-
- `POST /api/v1/convert` — `{ imageUrl, prompt?, maxTokens? }`
|
|
176
|
-
- `POST /api/v1/convert-base64` — `{ imageBase64, mimeType?, prompt?, maxTokens? }`
|
|
177
|
-
- `POST /api/v1/convert-stream` — SSE token stream
|
|
178
|
-
- `GET /health`
|
|
179
|
-
|
|
180
|
-
A `server/wrangler.jsonc` is included for deploying it as a Cloudflare Worker.
|
|
181
|
-
Any deployment of it (or any API with the same contract) can be passed as the
|
|
182
|
-
`processor`/`processorUrl` option above.
|
|
183
|
-
|
|
184
|
-
### Hosting the processor on Hugging Face Spaces
|
|
185
|
-
|
|
186
|
-
[`docling-space/`](./docling-space) packages the same model as a Hugging Face
|
|
187
|
-
Docker Space: token auth, one generation at a time with a bounded queue, the
|
|
188
|
-
model loaded lazily, and optional sanitized HTML output. Its README has the
|
|
189
|
-
deploy steps. Send the Space's token (and, for a private Space, a Hugging Face
|
|
190
|
-
read token) with `doclingOptions.processorHeaders`:
|
|
191
|
-
|
|
192
|
-
```ts
|
|
193
|
-
const { html } = await convertPDFToHTML(buffer, {
|
|
194
|
-
processor: "hybrid",
|
|
195
|
-
processorUrl: "https://YOUR_HF_USERNAME-extract-pdf-docling.hf.space",
|
|
196
|
-
doclingOptions: {
|
|
197
|
-
processorHeaders: {
|
|
198
|
-
"X-Docling-Token": process.env.DOCLING_API_TOKEN,
|
|
199
|
-
Authorization: `Bearer ${process.env.HF_SPACE_TOKEN}`, // private Space only
|
|
200
|
-
},
|
|
201
|
-
maxTokens: 1500,
|
|
202
|
-
},
|
|
203
|
-
});
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
Hybrid mode still waits for every flagged page before it returns. To answer at
|
|
207
|
-
once and enhance afterwards, run `processor: "frontend"`, return its HTML, and
|
|
208
|
-
OCR `ocrScan.pagesNeedingOcr` as a follow-up; the [
|
|
209
|
-
exactly that.
|
|
210
|
-
|
|
211
|
-
## Detecting whether a PDF needs OCR
|
|
212
|
-
|
|
213
|
-
Before committing to a full (and potentially slow) OCR parse, `detectPdfNeedsOcr`
|
|
214
|
-
runs a cheap, text-layer-only pass and reports whether each page needs OCR or
|
|
215
|
-
other heavy parsing — useful for routing documents to different pipelines
|
|
216
|
-
(fast path vs. OCR vs. screenshots vs. a heavier parser like LlamaParse or
|
|
217
|
-
Docling):
|
|
218
|
-
|
|
219
|
-
```ts
|
|
220
|
-
import { detectPdfNeedsOcr, convertPDFToHTML } from "extract-pdf";
|
|
221
|
-
|
|
222
|
-
const assessment = await detectPdfNeedsOcr(buffer);
|
|
223
|
-
// { needsOcr: boolean, pages: PageComplexityStats[], reasons: string[] }
|
|
224
|
-
|
|
225
|
-
if (!assessment.needsOcr) {
|
|
226
|
-
const { html } = await convertPDFToHTML(buffer, { method: "liteparse" });
|
|
227
|
-
} else {
|
|
228
|
-
console.log("Needs OCR:", assessment.reasons); // e.g. ["scanned", "sparse-text"]
|
|
229
|
-
// Route to an OCR-enabled pipeline, e.g.:
|
|
230
|
-
const { html } = await convertPDFToHTML(buffer, {
|
|
231
|
-
method: "liteparse",
|
|
232
|
-
liteParseOptions: { ocrEnabled: true },
|
|
233
|
-
});
|
|
234
|
-
}
|
|
235
|
-
```
|
|
236
|
-
|
|
237
|
-
`reasons` collects every distinct signal found across pages: `"scanned"`,
|
|
238
|
-
`"no-text"`, `"sparse-text"`, `"embedded-images"`, `"garbled"`, or
|
|
239
|
-
`"vector-text"`. Like `method: "liteparse"`, this is Node.js only and requires
|
|
240
|
-
`@llamaindex/liteparse`.
|
|
241
|
-
|
|
242
|
-
## Pipeline
|
|
243
|
-
|
|
244
|
-
The conversion runs a sequential chain of transformations on a `ParseResult` (pages → items):
|
|
245
|
-
|
|
246
|
-
```
|
|
247
|
-
Raw pdfjs text spans
|
|
248
|
-
→ CalculateGlobalStats — font heights, distances, format map
|
|
249
|
-
→ CompactLines — merge spans on the same y-line into LineItems
|
|
250
|
-
→ RemoveRepetitiveElements — strip recurring page headers/footers
|
|
251
|
-
→ VerticalToHorizontal — rotate vertical character runs
|
|
252
|
-
→ DetectTOC — identify Table of Contents pages, link headings
|
|
253
|
-
→ DetectHeaders — classify items as H1–H6 by font height
|
|
254
|
-
→ DetectListItems — detect bullet/numbered list items
|
|
255
|
-
→ GatherBlocks — group adjacent same-type lines into blocks
|
|
256
|
-
→ DetectCodeQuoteBlocks — mark indented blocks as CODE
|
|
257
|
-
→ DetectListLevels — add indentation for nested list levels
|
|
258
|
-
→ ToTextBlocks — flatten blocks to { category, text } pairs
|
|
259
|
-
→ ToHTML — render pairs as <p>, <h1>–<h6>, <ul>, <code>
|
|
260
|
-
```
|
|
261
|
-
|
|
262
|
-
## Folder structure
|
|
263
|
-
|
|
264
|
-
```
|
|
265
|
-
src/
|
|
266
|
-
pdf-to-html.ts — main entry point (convertPDFToHTML)
|
|
267
|
-
liteparse-to-html.ts — "liteparse" method (native napi addon)
|
|
268
|
-
liteparse-wasm-to-html.ts — "liteparse-wasm" method (WASM, browser/edge)
|
|
269
|
-
ocr-page-scan.ts — regex scan flagging infographic/table pages for OCR
|
|
270
|
-
docling-ocr.ts — Granite Docling OCR (local ONNX or remote URL),
|
|
271
|
-
│ page rasterization, doctags → HTML
|
|
272
|
-
detect-needs-ocr.ts — LiteParse-based OCR routing (Node.js only)
|
|
273
|
-
models/ — data classes: Page, ParseResult, TextItem,
|
|
274
|
-
│ LineItem, LineItemBlock, Word, BlockType, …
|
|
275
|
-
transforms/
|
|
276
|
-
│ base/ — abstract Transformation, ToLineItem*, ToLineItemBlock*
|
|
277
|
-
│ line-item/ — per-line-item transformations
|
|
278
|
-
│ block/ — per-block transformations
|
|
279
|
-
│ calculate-global-stats.ts
|
|
280
|
-
│ to-text-blocks.ts
|
|
281
|
-
│ to-html.ts
|
|
282
|
-
utils/
|
|
283
|
-
load-pdfjs.ts — lazy CDN loader for pdfjs-serverless
|
|
284
|
-
string-functions.ts
|
|
285
|
-
page-item-functions.ts
|
|
286
|
-
page-number-functions.ts
|
|
287
|
-
server/ — Granite Docling HTTP processor (Hono):
|
|
288
|
-
server.js, routes.js, schemas.js, model.js, wrangler.jsonc
|
|
289
|
-
```
|
|
1
|
+
<!-- template-git-repo:badges:start -->
|
|
2
|
+
<p align="center">
|
|
3
|
+
<img src="https://i.imgur.com/2ks47qa.png" /><br />
|
|
4
|
+
<a href="https://qwksearch.com/api"><img src="https://img.shields.io/badge/Docs-blue?logo=ReadTheDocs&logoColor=white" alt="Documentation" /></a>
|
|
5
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/stargazers"><img src="https://img.shields.io/github/stars/OpenSourceAGI/qwksearch-research-agent" alt="GitHub Stars" /></a>
|
|
6
|
+
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/dm/extract-pdf.svg" alt="NPM Monthly Downloads" /></a>
|
|
7
|
+
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/v/extract-pdf.svg" alt="npm version" /></a>
|
|
8
|
+
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/dt/extract-pdf.svg" alt="NPM Total Downloads" /></a>
|
|
9
|
+
<a href="https://www.npmjs.com/package/extract-pdf"><img src="https://img.shields.io/npm/types/extract-pdf" alt="TypeScript types" /></a>
|
|
10
|
+
<a href="https://packagephobia.com/result?p=extract-pdf"><img src="https://packagephobia.com/badge?p=extract-pdf" alt="Install size" /></a>
|
|
11
|
+
<br />
|
|
12
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/issues"><img src="https://img.shields.io/github/issues/OpenSourceAGI/qwksearch-research-agent?logo=github" alt="GitHub Issues" /></a>
|
|
13
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/pulls"><img src="https://img.shields.io/github/issues-pr/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs" alt="Open Pull Requests" /></a>
|
|
14
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/pulls?q=is%3Apr+is%3Aclosed"><img src="https://img.shields.io/github/issues-pr-closed/OpenSourceAGI/qwksearch-research-agent?logo=github&label=PRs%20merged&color=8957e5" alt="Merged Pull Requests" /></a>
|
|
15
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/discussions"><img src="https://img.shields.io/github/discussions/OpenSourceAGI/qwksearch-research-agent" alt="GitHub Discussions" /></a>
|
|
16
|
+
<a href="https://github.com/OpenSourceAGI/qwksearch-research-agent/commits/master/"><img src="https://img.shields.io/github/last-commit/OpenSourceAGI/qwksearch-research-agent.svg" alt="GitHub last commit" /></a>
|
|
17
|
+
<br />
|
|
18
|
+
<a href="https://stackblitz.com/github/OpenSourceAGI/qwksearch-research-agent/tree/master/packages/extract-pdf"><img height="20px" src="https://developer.stackblitz.com/img/open_in_stackblitz.svg" alt="Open in StackBlitz" /></a>
|
|
19
|
+
<img src="https://img.shields.io/badge/Bun-14151A?logo=bun&logoColor=white" alt="Bun" /> <img src="https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white" alt="TypeScript" /> <img src="https://img.shields.io/badge/Vite-646CFF?logo=vite&logoColor=white" alt="Vite" />
|
|
20
|
+
</p>
|
|
21
|
+
<!-- template-git-repo:badges:end -->
|
|
22
|
+
|
|
23
|
+
# extract-pdf
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
> **When users upload a PDF, they expect an instant chat response, not to wait for 5 min on OCR model.**
|
|
27
|
+
|
|
28
|
+
Instant no-backend-needed javascript to convert a PDF (URL or `ArrayBuffer`) into clean HTML with structural tagging — headings, lists, footnotes, code blocks, bold/italic, and Table of Contents entries. Works in Node.js, Cloudflare Workers, and browser environments via [pdfjs-serverless](https://github.com/johannschopplich/pdfjs-serverless).
|
|
29
|
+
|
|
30
|
+
**Slim by default:** the package bundles no PDF engine. PDF.js is loaded lazily at runtime from jsDelivr's ESM build of `pdfjs-serverless` (`https://cdn.jsdelivr.net/npm/pdfjs-serverless@1/+esm`, pinned to the major version) — a zero-dependency, single ~1.6 MB minified redistribution of Mozilla PDF.js built for serverless/edge runtimes. Node.js and Bun can't import remote URLs, so there the loader falls back to the locally-installed `pdfjs-serverless` optional dependency. Everything OCR-related (Granite Docling via `@huggingface/transformers`, `@napi-rs/canvas`) is likewise optional and imported only when actually used.
|
|
31
|
+
|
|
32
|
+
For OCR-grade fidelity on pages with infographics, charts, and tables, the package also ships the Granite Docling path (merged from the former `extract-pdf-docling` package): run all pages through the frontend JS parser, all pages through Docling OCR, a **hybrid** mode that regex-scans pages and OCRs only the ones that need it, or point at the URL of a remote docling-compatible processor (see `server/`).
|
|
33
|
+
|
|
34
|
+
## Install
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
bun add extract-pdf
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Docs and live demo
|
|
41
|
+
|
|
42
|
+
[`site/`](./site) is one Cloudflare Worker serving the docs for this package and [extract-webpage](../extract-webpage), plus a live demo at `/demo`: upload a PDF or paste a URL and get HTML back from `/api/convert`, with the Docling OCR follow-up, or extract a webpage through `/api/extract`. Run it with `bun install` at the repository root, then `cd site && bun run dev`. CI deploys it on every push that touches it. See [site/README.md](./site/README.md) for secrets, custom domains, CPU limits and hardening.
|
|
43
|
+
|
|
44
|
+
## Usage
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
import { convertPDFToHTML } from "extract-pdf";
|
|
48
|
+
|
|
49
|
+
const { html, title, author } = await convertPDFToHTML(
|
|
50
|
+
"https://example.com/paper.pdf",
|
|
51
|
+
);
|
|
52
|
+
// or pass an ArrayBuffer from fs.readFile / fetch
|
|
53
|
+
const { html } = await convertPDFToHTML(buffer, { addPageNumbers: true });
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Options
|
|
57
|
+
|
|
58
|
+
| Option | Default | Description |
|
|
59
|
+
| ----------------- | ---------------------- | ------------------------------------------------------------------------------------------ |
|
|
60
|
+
| `addPageNumbers` | `false` | Inserts `[n]` markers at each page boundary |
|
|
61
|
+
| `addCitation` | `true` | Reads PDF metadata and first-page heading to populate `title`/`author` in the return value |
|
|
62
|
+
| `method` | `"ts-block-algorithm"` | Parsing engine — `"ts-block-algorithm"`, `"liteparse"`, or `"liteparse-wasm"` (see below) |
|
|
63
|
+
| `liteParseOptions`| `{}` | Passed through to LiteParse's constructor when `method` is `"liteparse"` or `"liteparse-wasm"` |
|
|
64
|
+
| `processor` | `"frontend"` | Where OCR happens — `"frontend"`, `"hybrid"`, `"docling"`, or a processor URL (see below) |
|
|
65
|
+
| `processorUrl` | — | Remote docling-compatible API used by `"hybrid"`/`"docling"` instead of the in-process model |
|
|
66
|
+
| `ocrScanOptions` | `{}` | Threshold tuning for the hybrid page scan (`scanPagesForOCR`) |
|
|
67
|
+
| `doclingOptions` | `{}` | `prompt`, `maxTokens`, `scale` for the Docling OCR model, and `processorHeaders` sent to `processorUrl` |
|
|
68
|
+
|
|
69
|
+
### Return value
|
|
70
|
+
|
|
71
|
+
```ts
|
|
72
|
+
{ html: string, title?: string, author?: string, format: "pdf",
|
|
73
|
+
processor: string, ocrScan: OcrScanResult }
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Parse methods
|
|
77
|
+
|
|
78
|
+
`convertPDFToHTML` supports three interchangeable parsing engines via `options.method`:
|
|
79
|
+
|
|
80
|
+
| Method | Engine | Environments | OCR |
|
|
81
|
+
| ------------------------------------ | -------------------------------------------------------------------- | ----------------------------------- | --- |
|
|
82
|
+
| `"ts-block-algorithm"` (default) | The pure-TS pipeline documented below (this package) | Node.js, Cloudflare Workers, browser | No |
|
|
83
|
+
| `"liteparse"` | [LiteParse](https://github.com/run-llama/liteparse) (native, `@llamaindex/liteparse`) | Node.js only | Optional |
|
|
84
|
+
| `"liteparse-wasm"` | [LiteParse](https://github.com/run-llama/liteparse) (WASM, `@llamaindex/liteparse-wasm`) | Node.js, Cloudflare Workers, browser | Optional (via callback) |
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
import { convertPDFToHTML } from "extract-pdf";
|
|
88
|
+
|
|
89
|
+
const { html } = await convertPDFToHTML(buffer, { method: "liteparse" });
|
|
90
|
+
|
|
91
|
+
// Or the WASM build, which also runs in browsers and Cloudflare Workers:
|
|
92
|
+
const { html } = await convertPDFToHTML(buffer, { method: "liteparse-wasm" });
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
LiteParse ships a native (napi) addon, so `method: "liteparse"` only runs in
|
|
96
|
+
Node.js — it is not bundled into browser or Cloudflare Workers builds. Install
|
|
97
|
+
it explicitly (`bun add @llamaindex/liteparse`) since it's an optional
|
|
98
|
+
dependency; if it isn't installed, `convertPDFToHTML` returns `{ error }`
|
|
99
|
+
instead of throwing.
|
|
100
|
+
|
|
101
|
+
`method: "liteparse-wasm"` delegates to LiteParse's WebAssembly build instead,
|
|
102
|
+
which runs anywhere WASM does — browsers, Cloudflare Workers, and Node.js.
|
|
103
|
+
Install it explicitly (`bun add @llamaindex/liteparse-wasm`) since it's also
|
|
104
|
+
an optional dependency; if it isn't installed, `convertPDFToHTML` returns
|
|
105
|
+
`{ error }` instead of throwing. The WASM build has no OCR engine built in —
|
|
106
|
+
pass a `liteParseOptions.ocrEngine` callback (e.g. backed by `tesseract-js`)
|
|
107
|
+
to enable OCR.
|
|
108
|
+
|
|
109
|
+
By default both LiteParse paths run with OCR disabled (`ocrEnabled: false`) —
|
|
110
|
+
matching this package's "instant, no backend" philosophy. Use
|
|
111
|
+
`detectPdfNeedsOcr` (below) to decide when a document is worth re-parsing with
|
|
112
|
+
`liteParseOptions: { ocrEnabled: true }`.
|
|
113
|
+
|
|
114
|
+
## Processor modes: frontend, hybrid, Docling OCR, or a remote URL
|
|
115
|
+
|
|
116
|
+
Separately from `method`, the `processor` option decides whether pages go
|
|
117
|
+
through the [Granite Docling](https://huggingface.co/ibm-granite/granite-docling-258M)
|
|
118
|
+
OCR model — which preserves layout, tables, charts, code, and formulas at the
|
|
119
|
+
cost of model inference per page:
|
|
120
|
+
|
|
121
|
+
| `processor` | What happens |
|
|
122
|
+
| ----------------------- | --------------------------------------------------------------------------------------------------------- |
|
|
123
|
+
| `"frontend"` (default) | All pages parsed by the pure-JS text-layer pipeline. No OCR, no model, instant. |
|
|
124
|
+
| `"hybrid"` | JS pipeline everywhere + a regex scan (`scanPagesForOCR`) flags pages with infographics/figures/tables or no usable text layer; only those pages are rasterized and OCR'd with Docling. |
|
|
125
|
+
| `"docling"` | Every page rasterized and OCR'd with Docling. |
|
|
126
|
+
| `"https://..."` (URL) | Like `"docling"`, but every page is POSTed to that docling-compatible processor API instead of running the model in-process. |
|
|
127
|
+
|
|
128
|
+
```ts
|
|
129
|
+
// Hybrid: fast JS parse, OCR only the pages that need it
|
|
130
|
+
const { html, ocrScan } = await convertPDFToHTML(buffer, { processor: "hybrid" });
|
|
131
|
+
console.log(ocrScan.pagesNeedingOcr); // e.g. [3, 7] — pages with tables/figures
|
|
132
|
+
// each ocrScan.pages[i] lists reasons: "table-caption", "figure-caption",
|
|
133
|
+
// "table-rows", "numeric-grid", "no-text", "sparse-text", "garbled-text"
|
|
134
|
+
|
|
135
|
+
// All pages through a remote Docling processor (this package's server/):
|
|
136
|
+
const { html } = await convertPDFToHTML(buffer, {
|
|
137
|
+
processor: "http://localhost:3000",
|
|
138
|
+
});
|
|
139
|
+
|
|
140
|
+
// Hybrid with the OCR offloaded to a remote processor:
|
|
141
|
+
const { html } = await convertPDFToHTML(buffer, {
|
|
142
|
+
processor: "hybrid",
|
|
143
|
+
processorUrl: "http://localhost:3000",
|
|
144
|
+
});
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
OCR'd pages are emitted as `<section class="ocr-page" id="page-N">…</section>`
|
|
148
|
+
(tables become real `<table>` markup, captions `<figcaption>`, formulas
|
|
149
|
+
`<code class="formula">`, …); pages whose OCR fails keep their JS-parsed HTML.
|
|
150
|
+
|
|
151
|
+
Running Docling **in-process** requires the optional `@huggingface/transformers`
|
|
152
|
+
dependency (the ONNX model `onnx-community/granite-docling-258M-ONNX`, ~1 GB,
|
|
153
|
+
downloads on first use) and — in Node.js — `@napi-rs/canvas` for page
|
|
154
|
+
rasterization (browsers/Workers use `OffscreenCanvas`).
|
|
155
|
+
|
|
156
|
+
The regex scanner is also exported standalone, and so are the OCR helpers:
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
import { scanPagesForOCR, doctagsToHtml, ocrImageWithDocling } from "extract-pdf";
|
|
160
|
+
|
|
161
|
+
const scan = scanPagesForOCR(pageTexts); // string[] of per-page text
|
|
162
|
+
// { needsOcr, pagesNeedingOcr: number[], pages: [{ page, reasons, captions }] }
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
### Running the Docling processor server
|
|
166
|
+
|
|
167
|
+
The `server/` folder contains the Hono HTTP API (formerly the
|
|
168
|
+
`extract-pdf-docling` package) exposing the Granite Docling model:
|
|
169
|
+
|
|
170
|
+
```sh
|
|
171
|
+
bun run serve:docling # or dev:docling for --watch; port 3000
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
- Swagger UI: `http://localhost:3000/docs`, spec at `/openapi.json`
|
|
175
|
+
- `POST /api/v1/convert` — `{ imageUrl, prompt?, maxTokens? }`
|
|
176
|
+
- `POST /api/v1/convert-base64` — `{ imageBase64, mimeType?, prompt?, maxTokens? }`
|
|
177
|
+
- `POST /api/v1/convert-stream` — SSE token stream
|
|
178
|
+
- `GET /health`
|
|
179
|
+
|
|
180
|
+
A `server/wrangler.jsonc` is included for deploying it as a Cloudflare Worker.
|
|
181
|
+
Any deployment of it (or any API with the same contract) can be passed as the
|
|
182
|
+
`processor`/`processorUrl` option above.
|
|
183
|
+
|
|
184
|
+
### Hosting the processor on Hugging Face Spaces
|
|
185
|
+
|
|
186
|
+
[`docling-space/`](./docling-space) packages the same model as a Hugging Face
|
|
187
|
+
Docker Space: token auth, one generation at a time with a bounded queue, the
|
|
188
|
+
model loaded lazily, and optional sanitized HTML output. Its README has the
|
|
189
|
+
deploy steps. Send the Space's token (and, for a private Space, a Hugging Face
|
|
190
|
+
read token) with `doclingOptions.processorHeaders`:
|
|
191
|
+
|
|
192
|
+
```ts
|
|
193
|
+
const { html } = await convertPDFToHTML(buffer, {
|
|
194
|
+
processor: "hybrid",
|
|
195
|
+
processorUrl: "https://YOUR_HF_USERNAME-extract-pdf-docling.hf.space",
|
|
196
|
+
doclingOptions: {
|
|
197
|
+
processorHeaders: {
|
|
198
|
+
"X-Docling-Token": process.env.DOCLING_API_TOKEN,
|
|
199
|
+
Authorization: `Bearer ${process.env.HF_SPACE_TOKEN}`, // private Space only
|
|
200
|
+
},
|
|
201
|
+
maxTokens: 1500,
|
|
202
|
+
},
|
|
203
|
+
});
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Hybrid mode still waits for every flagged page before it returns. To answer at
|
|
207
|
+
once and enhance afterwards, run `processor: "frontend"`, return its HTML, and
|
|
208
|
+
OCR `ocrScan.pagesNeedingOcr` as a follow-up; the [live demo](./site) does
|
|
209
|
+
exactly that.
|
|
210
|
+
|
|
211
|
+
## Detecting whether a PDF needs OCR
|
|
212
|
+
|
|
213
|
+
Before committing to a full (and potentially slow) OCR parse, `detectPdfNeedsOcr`
|
|
214
|
+
runs a cheap, text-layer-only pass and reports whether each page needs OCR or
|
|
215
|
+
other heavy parsing — useful for routing documents to different pipelines
|
|
216
|
+
(fast path vs. OCR vs. screenshots vs. a heavier parser like LlamaParse or
|
|
217
|
+
Docling):
|
|
218
|
+
|
|
219
|
+
```ts
|
|
220
|
+
import { detectPdfNeedsOcr, convertPDFToHTML } from "extract-pdf";
|
|
221
|
+
|
|
222
|
+
const assessment = await detectPdfNeedsOcr(buffer);
|
|
223
|
+
// { needsOcr: boolean, pages: PageComplexityStats[], reasons: string[] }
|
|
224
|
+
|
|
225
|
+
if (!assessment.needsOcr) {
|
|
226
|
+
const { html } = await convertPDFToHTML(buffer, { method: "liteparse" });
|
|
227
|
+
} else {
|
|
228
|
+
console.log("Needs OCR:", assessment.reasons); // e.g. ["scanned", "sparse-text"]
|
|
229
|
+
// Route to an OCR-enabled pipeline, e.g.:
|
|
230
|
+
const { html } = await convertPDFToHTML(buffer, {
|
|
231
|
+
method: "liteparse",
|
|
232
|
+
liteParseOptions: { ocrEnabled: true },
|
|
233
|
+
});
|
|
234
|
+
}
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
`reasons` collects every distinct signal found across pages: `"scanned"`,
|
|
238
|
+
`"no-text"`, `"sparse-text"`, `"embedded-images"`, `"garbled"`, or
|
|
239
|
+
`"vector-text"`. Like `method: "liteparse"`, this is Node.js only and requires
|
|
240
|
+
`@llamaindex/liteparse`.
|
|
241
|
+
|
|
242
|
+
## Pipeline
|
|
243
|
+
|
|
244
|
+
The conversion runs a sequential chain of transformations on a `ParseResult` (pages → items):
|
|
245
|
+
|
|
246
|
+
```
|
|
247
|
+
Raw pdfjs text spans
|
|
248
|
+
→ CalculateGlobalStats — font heights, distances, format map
|
|
249
|
+
→ CompactLines — merge spans on the same y-line into LineItems
|
|
250
|
+
→ RemoveRepetitiveElements — strip recurring page headers/footers
|
|
251
|
+
→ VerticalToHorizontal — rotate vertical character runs
|
|
252
|
+
→ DetectTOC — identify Table of Contents pages, link headings
|
|
253
|
+
→ DetectHeaders — classify items as H1–H6 by font height
|
|
254
|
+
→ DetectListItems — detect bullet/numbered list items
|
|
255
|
+
→ GatherBlocks — group adjacent same-type lines into blocks
|
|
256
|
+
→ DetectCodeQuoteBlocks — mark indented blocks as CODE
|
|
257
|
+
→ DetectListLevels — add indentation for nested list levels
|
|
258
|
+
→ ToTextBlocks — flatten blocks to { category, text } pairs
|
|
259
|
+
→ ToHTML — render pairs as <p>, <h1>–<h6>, <ul>, <code>
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
## Folder structure
|
|
263
|
+
|
|
264
|
+
```
|
|
265
|
+
src/
|
|
266
|
+
pdf-to-html.ts — main entry point (convertPDFToHTML)
|
|
267
|
+
liteparse-to-html.ts — "liteparse" method (native napi addon)
|
|
268
|
+
liteparse-wasm-to-html.ts — "liteparse-wasm" method (WASM, browser/edge)
|
|
269
|
+
ocr-page-scan.ts — regex scan flagging infographic/table pages for OCR
|
|
270
|
+
docling-ocr.ts — Granite Docling OCR (local ONNX or remote URL),
|
|
271
|
+
│ page rasterization, doctags → HTML
|
|
272
|
+
detect-needs-ocr.ts — LiteParse-based OCR routing (Node.js only)
|
|
273
|
+
models/ — data classes: Page, ParseResult, TextItem,
|
|
274
|
+
│ LineItem, LineItemBlock, Word, BlockType, …
|
|
275
|
+
transforms/
|
|
276
|
+
│ base/ — abstract Transformation, ToLineItem*, ToLineItemBlock*
|
|
277
|
+
│ line-item/ — per-line-item transformations
|
|
278
|
+
│ block/ — per-block transformations
|
|
279
|
+
│ calculate-global-stats.ts
|
|
280
|
+
│ to-text-blocks.ts
|
|
281
|
+
│ to-html.ts
|
|
282
|
+
utils/
|
|
283
|
+
load-pdfjs.ts — lazy CDN loader for pdfjs-serverless
|
|
284
|
+
string-functions.ts
|
|
285
|
+
page-item-functions.ts
|
|
286
|
+
page-number-functions.ts
|
|
287
|
+
server/ — Granite Docling HTTP processor (Hono):
|
|
288
|
+
server.js, routes.js, schemas.js, model.js, wrangler.jsonc
|
|
289
|
+
```
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "extract-pdf",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.425",
|
|
4
4
|
"description": "Convert a PDF (URL or ArrayBuffer) into clean HTML with structural tagging — headings, lists, footnotes, code blocks, bold/italic. Slim by default (PDF.js loads at runtime from the pdfjs-serverless CDN build), with optional Granite Docling OCR for pages with infographics/tables — in-process, hybrid, or via a remote processor URL. Works in Node.js, Cloudflare Workers, and browser environments.",
|
|
5
5
|
"author": "vtempest <grokthiscontact@gmail.com>",
|
|
6
6
|
"license": "rights.institute/PROSPER",
|