@pagesift/node 0.0.0-stage → 0.50.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/README.md +176 -2
  2. package/index.d.ts +258 -0
  3. package/index.js +788 -0
  4. package/package.json +59 -4
package/README.md CHANGED
@@ -1,3 +1,177 @@
1
- # Temporary Holding Version
1
+ # pagesift (Node.js)
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ 0.20.0: see the [release and migration guide](https://github.com/YOLOW92/pagesift/blob/main/docs/releases/0.20.0.md). New collection/archive cores are available through Rust and CLI; this package keeps its documented scraping interface.
4
+
5
+ The npm package `pagesift`: pagesift's Rust engine for Node.js, built with [napi-rs](https://napi.rs).
6
+ Prebuilt binaries, so you don't need a Rust toolchain or a headless-browser download.
7
+ The API is designed for throughput first: batches are scheduled in Rust, results stay in Rust until you read them,
8
+ and CPU work never runs on the Node main thread unless you ask for the synchronous `convert`.
9
+
10
+ ```js
11
+ import { readFile } from 'node:fs/promises'
12
+ import { convert, convertMany, Scraper, PagesiftError } from 'pagesift'
13
+
14
+ // HTML you already have -> Markdown (no network)
15
+ const page = convert(html, { url: 'https://example.com/post' })
16
+ console.log(page.markdown, page.metadata.title)
17
+
18
+ // Many documents: one call, bounded Rust workers, bytes in (no string transcoding)
19
+ const pages = await convertMany(await Promise.all(files.map((f) => readFile(f))))
20
+
21
+ // Fetching: create one Scraper and keep it (connection pool, robots.txt cache, rate-limit state, browsers)
22
+ const scraper = new Scraper()
23
+ try {
24
+ const p = await scraper.scrape('https://example.com', { formats: ['markdown', 'links'] })
25
+ } catch (e) {
26
+ if (e instanceof PagesiftError) console.log(e.code, e.details) // e.g. ROBOTS_DISALLOWED
27
+ }
28
+
29
+ // Results as they finish; `break` cancels the rest
30
+ for await (const { url, page, error } of scraper.scrapeStream(urls, { concurrency: 32 })) {
31
+ if (error) console.log(url, error.code)
32
+ else console.log(url, page.markdown.length)
33
+ }
34
+ await scraper.close()
35
+ ```
36
+
37
+ ## API
38
+
39
+ | | Returns | Notes |
40
+ |---|---|---|
41
+ | `convert(html, options?)` | `Page` | Synchronous, on the calling thread |
42
+ | `convertAsync(html, options?)` | `Promise<Page>` | On a Rust thread |
43
+ | `convertMany(htmls, options?)` | `Promise<Page[]>` | Whole batch in parallel in Rust; same order as input |
44
+ | `new Scraper(config?)` | | Keep one instance for the life of the process |
45
+ | `scraper.scrape(url, options?)` | `Promise<Page>` | Rejects with `PagesiftError` |
46
+ | `scraper.scrapeMany(urls, options?)` | `Promise<BatchItem[]>` | Same order as input; one failure only affects its own item |
47
+ | `scraper.scrapeStream(urls, options?)` | async iterable of `BatchItem` | Completion order; leaving the loop cancels pending work |
48
+ | `scraper.close()` | `Promise<void>` | Stops browser processes |
49
+
50
+ `html` is a `string` or UTF-8 bytes (`Uint8Array` / `Buffer`). Don't modify a buffer while a call that uses it is running.
51
+
52
+ **Options.** `formats` (`'markdown' | 'html' | 'rawHtml' | 'links' | 'images'`, default `['markdown']`; only the listed outputs are computed),
53
+ `onlyMainContent` (default `true`), `url` (convert only). Scraping adds `engine` (`auto` / `http` / `browser`), `timeout`, `waitFor`,
54
+ `headers`, `ignoreRobotsTxt`, `mobile`, `blockAds`, `ifNoneMatch`, `ifModifiedSince`; batches add `concurrency` (default 16).
55
+
56
+ **`Page`.** `markdown`, `html`, `rawHtml`, `links`, `images` (`null` when not requested) and `metadata`
57
+ (title, description, language, `statusCode`, `og*` keys, and `metadata.pagesift`: content hashes, redirect chain, ETag, engine used, timings).
58
+ Each property read copies that one field into JS; fields you never read are never copied. `toJSON()` returns everything as a plain object.
59
+
60
+ **`BatchItem`.** `{ index, url, page?, error? }`, with exactly one of `page` or `error`.
61
+ `error` is `{ code, message, details?, retryAfter? }`, a plain object rather than an `Error`, so a big batch doesn't collect stack traces.
62
+
63
+ **Errors.** Single calls throw or reject with `PagesiftError` (`code`, `details`, `retryAfter`). The codes match the pagesift HTTP API:
64
+ `ROBOTS_DISALLOWED`, `ACCESS_BARRIER`, `RATE_LIMITED`, `HOST_COOLDOWN`, `TIMEOUT`, `SSRF_BLOCKED`, `TOO_LARGE`,
65
+ `UPSTREAM_ERROR`, `NETWORK_ERROR`, `DNS_FAILURE`, `BUSY`, `UNSUPPORTED_OPTION`, `INVALID_REQUEST`, `INTERNAL_ERROR`.
66
+ pagesift never tries to get around a block: an access barrier is reported with `details` (vendor, kind, confidence), not bypassed.
67
+
68
+ **`Scraper` config.** `userAgent`, `allowPrivateNetwork` (off by default), `requestIntervalMs` (per host, default 1000),
69
+ `perHostConcurrency` (default 2), `maxBrowsers`.
70
+
71
+ ## Why it's shaped this way: benchmarks
72
+
73
+ Measured on 2026-10-21 on linux-x64 with 16 threads, Node v24.21.0, release build. Other jobs were running on the machine
74
+ (load average 2–6), so take the absolute numbers as rough; the ratios are what matter.
75
+ Reproduce with `bench/bench.mjs` (see Development).
76
+
77
+ ### Against Readability + Turndown
78
+
79
+ Same 597 pages (the frozen pagesift evaluation set, 179 MB of HTML). Readability needs a DOM, so JSDOM parsing is included in its time.
80
+
81
+ | Metric | pagesift | Readability + Turndown | |
82
+ |---|---|---|---|
83
+ | Per-page time p50, ms (lower is better) | 2.34 | 121.80 | 52× faster |
84
+ | Per-page time p95, ms (lower is better) | 11.86 | 716.18 | 60× faster |
85
+ | Per-page time mean, ms (lower is better) | 4.20 | 252.15 | 60× faster |
86
+ | Throughput, pages/s (higher is better; pagesift `convertMany` with Buffer input, Readability is single-threaded) | 1284 | 4.0 | 324× |
87
+ | Peak process RSS, MB (lower is better) | 1248 | 4455 | 72% lower |
88
+
89
+ ### Batching and zero-copy input (same 597 pages)
90
+
91
+ | Approach | Throughput, pages/s (higher is better) | vs sync | Main-thread utilisation (lower is better) |
92
+ |---|---|---|---|
93
+ | `convert` in a loop | 238 | 1× | 100% |
94
+ | `convertAsync` × 597 + `Promise.all` | 857 | 3.6× | 55% |
95
+ | `convertMany` with strings | 874 | 3.7× | 65% |
96
+ | `convertMany` with Buffers | **1284** | **5.4×** | **17%** |
97
+ | `convertMany` with Buffers, all four formats (for comparison) | 800 | 3.4× | — |
98
+
99
+ Turning a JS string into Rust UTF-8 has to happen on the main thread, and for 179 MB that's most of the remaining main-thread work.
100
+ With bytes there's nothing to transcode. `convertMany` with strings runs that whole transcoding inside one call, so for very large
101
+ string batches either pass Buffers or split the batch.
102
+ Computing only the formats you ask for adds another 1.6× (1284 vs 800 pages/s).
103
+
104
+ ### Lazy fields (one batch of 597 `Page`s with markdown, html, links and images computed)
105
+
106
+ | Read | Main-thread time, ms (lower is better) |
107
+ |---|---|
108
+ | `page.markdown` only | 13.81 |
109
+ | `page.toJSON()` (everything) | 203.17 |
110
+
111
+ Fields stay in Rust until read, so reading only what you need is 15× cheaper on the main thread.
112
+
113
+ ### Scraping (local HTTP server in a child process, 16 hosts, ~3 KB pages, so fixed per-request costs dominate)
114
+
115
+ | Design | Measurement | Result |
116
+ |---|---|---|
117
+ | Reuse one `Scraper` | 100 sequential pages, one Scraper, ms (lower is better) | 34 |
118
+ | | 100 sequential pages, new Scraper per page, ms (lower is better) | 289 (8.6× slower) |
119
+ | Batches scheduled in Rust | `scrapeMany`, 2000 pages, pages/s (higher is better) | 13864 |
120
+ | | `scrape` × 2000 + `Promise.all`, pages/s (higher is better) | 14329 |
121
+ | | Main-thread utilisation during `scrapeMany` (lower is better) | **2%** |
122
+ | | Main-thread utilisation during `scrape` + `Promise.all` (lower is better) | 51% |
123
+ | Completion order | `scrapeMany`: first result, one 1 s page among 99 fast ones, ms (lower is better) | 1002 |
124
+ | | `scrapeStream`: first result, ms (lower is better) | **21** |
125
+ | | `scrapeStream`: all 99 fast pages, ms (lower is better) | 157 |
126
+
127
+ A new `Scraper` per page pays for a fresh connection and a robots.txt fetch every time.
128
+ `scrapeMany` and individual `scrape` calls reach the same throughput, but the batch leaves the main thread almost idle (2% vs 51%),
129
+ which matters when the same process also serves requests. `scrapeStream` delivers results as soon as each page finishes instead of waiting for the slowest one.
130
+
131
+ Known limit: requests to a single host currently top out at about 1000 per second even with `requestIntervalMs: 0`
132
+ (the rate limiter waits for a 1 ms timer tick per request), which is why this benchmark spreads its pages over 16 hosts.
133
+
134
+ ## CPU budget
135
+
136
+ Asynchronous conversion shares at most `max(1, logical CPUs - 1)` Rust worker slots across calls,
137
+ leaving capacity for Node's event loop. `convertMany` validates and copies all inputs before starting
138
+ CPU workers, so parser work does not compete with the main thread while it copies strings.
139
+ Large string batches still copy synchronously; use UTF-8 buffers or smaller batches when needed.
140
+ The timing figures above predate this 0.20.0 scheduling change; rerun them on your deployment hardware.
141
+
142
+ ## Platforms
143
+
144
+ | Target | Status |
145
+ |---|---|
146
+ | linux-x64-gnu, linux-arm64-gnu | native build and full binding tests in Linux CI |
147
+ | darwin-x64, darwin-arm64 | native build and full binding tests in macOS CI |
148
+ | linux-x64-musl | native musl build and full binding tests on Alpine |
149
+ | win32-x64-msvc | native MSVC build and full binding tests on Windows |
150
+
151
+ Node 18 or newer is required. `npm install pagesift` selects the matching optional native package;
152
+ Linux musl and glibc packages are separate. Windows supports conversion and HTTP scraping;
153
+ local browser process management is unavailable. The Rust CDP client can connect to an existing remote endpoint.
154
+
155
+ ## Development
156
+
157
+ ```bash
158
+ npm install
159
+ npm run build # release build + index.js / index.d.ts
160
+ npm test # node --test, local HTTP server only
161
+ ```
162
+
163
+ Benchmarks, from the repository root:
164
+
165
+ ```bash
166
+ node crates/pagesift-node/bench/bench.mjs extract bench/data/inputs/manifest.jsonl <dir with jsdom, @mozilla/readability, turndown in node_modules>
167
+ node crates/pagesift-node/bench/bench.mjs scrape
168
+ ```
169
+
170
+ `index.js` and `index.d.ts` are generated by `napi build` (types from `src/`, plus `header.d.ts`) and committed.
171
+ `scripts/postbuild.mjs` adds the two exports the generated loader can't see (`PagesiftError`, `version`).
172
+
173
+ This crate doesn't use `#![forbid(unsafe_code)]`, because the napi-rs macros generate unsafe code. Our own code here contains no `unsafe`.
174
+
175
+ ## License
176
+
177
+ AGPL-3.0-only, with a commercial license available for use without AGPL obligations. See the repository's [LICENSE](../../LICENSE) and [COMMERCIAL.md](../../COMMERCIAL.md).
package/index.d.ts ADDED
@@ -0,0 +1,258 @@
1
+ /* eslint-disable */
2
+ /* 由 napi-rs 根据 src/ 生成;本段手写的类型来自 header.d.ts。 */
3
+
4
+ /** 包版本。 */
5
+ export declare const version: string
6
+
7
+ /** 和 pagesift HTTP 接口一致的错误码。 */
8
+ export type ErrorCode =
9
+ | 'ROBOTS_DISALLOWED'
10
+ | 'ACCESS_BARRIER'
11
+ | 'RATE_LIMITED'
12
+ | 'HOST_COOLDOWN'
13
+ | 'TIMEOUT'
14
+ | 'SSRF_BLOCKED'
15
+ | 'TOO_LARGE'
16
+ | 'UPSTREAM_ERROR'
17
+ | 'NETWORK_ERROR'
18
+ | 'DNS_FAILURE'
19
+ | 'BUSY'
20
+ | 'UNSUPPORTED_OPTION'
21
+ | 'INVALID_REQUEST'
22
+ | 'INTERNAL_ERROR'
23
+
24
+ /** 抓取或转换失败时抛出(或 Promise 以它拒绝)。 */
25
+ export declare class PagesiftError extends Error {
26
+ readonly name: 'PagesiftError'
27
+ readonly code: ErrorCode
28
+ /** 补充信息,比如拦截页的 `{ vendor, kind, confidence, ruleId }`。 */
29
+ readonly details?: unknown
30
+ /** 限流时建议多少秒后再试。 */
31
+ readonly retryAfter?: number
32
+ }
33
+
34
+ export type Format = 'markdown' | 'html' | 'rawHtml' | 'links' | 'images'
35
+
36
+ /** `Page.toJSON()` 的结果:一次取出全部字段的普通对象,没请求的格式不出现。 */
37
+ export interface PageJSON {
38
+ markdown?: string
39
+ html?: string
40
+ rawHtml?: string
41
+ links?: string[]
42
+ images?: string[]
43
+ metadata: PageMetadata
44
+ }
45
+
46
+ export interface PageMetadata {
47
+ title?: string
48
+ description?: string
49
+ language?: string
50
+ keywords?: string
51
+ robots?: string
52
+ favicon?: string
53
+ publishedTime?: string
54
+ modifiedTime?: string
55
+ /** 最终地址(跟随重定向之后)。 */
56
+ url?: string
57
+ /** 请求的地址。 */
58
+ sourceURL?: string
59
+ /** 只有抓取结果才有。 */
60
+ statusCode?: number
61
+ contentType?: string
62
+ proxyUsed?: string
63
+ /** PDF 的页数。 */
64
+ numPages?: number
65
+ /** 抓取结果的 pagesift 扩展字段。 */
66
+ pagesift?: PagesiftMetadata
67
+ /** `ogImage`、`twitterCard`、`articlePublishedTime` 这类 meta,键名是 camelCase。 */
68
+ [key: string]: unknown
69
+ }
70
+
71
+ export interface PagesiftMetadata {
72
+ /** Markdown、原始 HTML、正文纯文本的 BLAKE3 哈希(十六进制)。 */
73
+ contentHash?: string
74
+ /** Computed only when change tracking is enabled in a supporting API surface. */
75
+ rawHash?: string
76
+ textHash?: string
77
+ redirectChain?: string[]
78
+ etag?: string
79
+ lastModified?: string
80
+ /** 条件请求命中,页面没变(此时没有 markdown)。 */
81
+ notModified?: boolean
82
+ /** 这一页实际用的抓取方式。 */
83
+ engine?: 'http' | 'browser'
84
+ warnings?: string[]
85
+ /** 各阶段耗时,毫秒。 */
86
+ timings?: Record<string, number>
87
+ }
88
+
89
+
90
+ /**
91
+ * Which binding artifact the generated loader actually loaded: `'native'` for
92
+ * a native addon, otherwise the `platformArchABI` of the WASI flavor. Every
93
+ * flavor napi-rs can build is listed, because `NAPI_RS_NATIVE_LIBRARY_PATH`
94
+ * can point the loader at a WASI artifact this package does not build itself.
95
+ */
96
+ export declare const __napiBindingTarget: 'native' | 'wasm32-wasi' | 'wasm32-wasip1'
97
+
98
+ declare global {
99
+ interface __NapiRsAsyncGenerator<TOwner, T, TReturn, TNext> {
100
+ next(...[value]: [] | [TNext]): globalThis.Promise<globalThis.IteratorResult<T, TReturn | undefined>>
101
+ return(...[value]: [] | [TReturn]): globalThis.Promise<globalThis.IteratorResult<T, TReturn | undefined>>
102
+ throw(exception?: unknown): globalThis.Promise<globalThis.IteratorResult<T, TReturn | undefined>>
103
+ [globalThis.Symbol.asyncIterator](): this
104
+ }
105
+ }
106
+
107
+ /** 一页的转换或抓取结果。每次读属性都会新建一个 JS 值,需要多次使用时先存到变量里。 */
108
+ export declare class Page {
109
+ /** 正文的 Markdown;`formats` 里没有 `markdown`、或条件请求命中(`notModified`)时为 null。 */
110
+ get markdown(): string | null
111
+ /** 清洗后的 HTML(`formats` 含 `html` 时才有)。 */
112
+ get html(): string | null
113
+ /** 原始 HTML(`formats` 含 `rawHtml` 时才有)。 */
114
+ get rawHtml(): string | null
115
+ /** 页面里的链接,绝对地址(`formats` 含 `links` 时才有)。 */
116
+ get links(): Array<string> | null
117
+ /** 页面里的图片地址(`formats` 含 `images` 时才有)。 */
118
+ get images(): Array<string> | null
119
+ /** 标题、描述、状态码、`og:*` 等元数据,以及 `pagesift` 扩展字段。 */
120
+ get metadata(): PageMetadata
121
+ /** 一次性取出全部字段,得到普通对象(`JSON.stringify(page)` 也会调用它)。 */
122
+ toJSON(): PageJSON
123
+ }
124
+
125
+ /** 抓取器:连接池、robots.txt 缓存、限速状态和浏览器池都在它里面,长期复用一个实例最快。 */
126
+ export declare class Scraper {
127
+ constructor(config?: ScraperConfig | undefined | null)
128
+ /** 抓取一个 URL。失败时 Promise 以 `PagesiftError` 拒绝。 */
129
+ scrape(url: string, options?: ScrapeOptions | undefined | null): Promise<Page>
130
+ /** 抓取一批 URL,全部完成后一次返回,顺序和输入一致;单个失败只影响它自己的那一项。 */
131
+ scrapeMany(urls: Array<string>, options?: BatchOptions | undefined | null): Promise<BatchItem[]>
132
+ /** 抓取一批 URL,按完成顺序逐个产出(`for await`)。提前 `break` 会取消还没完成的抓取。 */
133
+ scrapeStream(urls: Array<string>, options?: BatchOptions | undefined | null): ScrapeStream
134
+ /** 关闭浏览器进程。之后再调用抓取方法会报错。 */
135
+ close(): Promise<void>
136
+ }
137
+
138
+ /**
139
+ * 按完成顺序产出 `BatchItem` 的异步迭代器。
140
+ *
141
+ * This type implements JavaScript's async iterable protocol.
142
+ * It can be used with `for await...of` loops.
143
+ *
144
+ * @see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols
145
+ */
146
+ export declare class ScrapeStream {
147
+
148
+ [globalThis.Symbol.asyncIterator](): globalThis.__NapiRsAsyncGenerator<ScrapeStream, BatchItem, void, undefined>
149
+ }
150
+
151
+ /** 批量结果里单个 URL 的错误:字段和 `PagesiftError` 一致,但不是 Error 实例(不采集调用栈)。 */
152
+ export interface BatchError {
153
+ code: ErrorCode
154
+ message: string
155
+ details?: unknown
156
+ /** 限流时建议多少秒后再试。 */
157
+ retryAfter?: number
158
+ }
159
+
160
+ /** 批量抓取里一个 URL 的结果:`page` 和 `error` 恰好有一个。 */
161
+ export interface BatchItem {
162
+ /** 在输入数组里的下标。 */
163
+ index: number
164
+ url: string
165
+ page?: Page
166
+ error?: BatchError
167
+ }
168
+
169
+ /** 批量抓取选项:`ScrapeOptions` 的全部字段,加上并发数。 */
170
+ export interface BatchOptions {
171
+ /** 同时进行的抓取数,默认 16;同一主机仍受限速约束。 */
172
+ concurrency?: number
173
+ /** 计算哪些输出,默认 `['markdown']`;只算需要的,越少越快。 */
174
+ formats?: Format[]
175
+ /** 只保留正文,去掉导航、页眉页脚、侧边栏,默认 true。 */
176
+ onlyMainContent?: boolean
177
+ /** `auto`(默认,正文太少时改用浏览器渲染)、`http` 或 `browser`。 */
178
+ engine?: 'auto' | 'http' | 'browser'
179
+ /** 总超时,毫秒,至少 1000。 */
180
+ timeout?: number
181
+ /** 浏览器渲染时页面加载后再等待多久,毫秒。 */
182
+ waitFor?: number
183
+ /** 额外的请求头。 */
184
+ headers?: Record<string, string>
185
+ /** 这一页不检查 robots.txt(只对单页抓取有效)。 */
186
+ ignoreRobotsTxt?: boolean
187
+ /** 浏览器渲染时模拟手机。 */
188
+ mobile?: boolean
189
+ /** 浏览器渲染时屏蔽广告,默认 true。 */
190
+ blockAds?: boolean
191
+ /** 条件请求:`If-None-Match`。 */
192
+ ifNoneMatch?: string
193
+ /** 条件请求:`If-Modified-Since`(HTTP 日期)。 */
194
+ ifModifiedSince?: string
195
+ }
196
+
197
+ /** 把一段 HTML 转成 Markdown,在当前线程上同步执行(会阻塞事件循环,适合脚本和小页面)。 */
198
+ export declare function convert(html: string | Uint8Array, options?: ConvertOptions): Page
199
+
200
+ /** 和 `convert` 相同,但在后台线程上执行,不阻塞事件循环。 */
201
+ export declare function convertAsync(html: string | Uint8Array, options?: ConvertOptions): Promise<Page>
202
+
203
+ /**
204
+ * 批量转换:使用共享的 Rust CPU 名额,结果顺序和输入一致。
205
+ * 先完成主线程上的输入校验和字符串复制,再启动 CPU 工作,避免输入复制与解析争用 CPU。
206
+ * `Uint8Array` 省去 UTF-8 字符串复制;调用完成前不得修改它。
207
+ */
208
+ export declare function convertMany(htmls: Array<string | Uint8Array>, options?: ConvertOptions): Promise<Page[]>
209
+
210
+ /** `convert`、`convertAsync`、`convertMany` 的选项。 */
211
+ export interface ConvertOptions {
212
+ /** 页面地址:把相对链接转成绝对地址,也写进 `metadata.url`。 */
213
+ url?: string
214
+ /** 只保留正文(去掉导航、页眉页脚、侧边栏等),默认 true。 */
215
+ onlyMainContent?: boolean
216
+ /** 计算哪些输出,默认 `['markdown']`;只算需要的,越少越快。 */
217
+ formats?: Format[]
218
+ }
219
+
220
+ /** 单页抓取选项。 */
221
+ export interface ScrapeOptions {
222
+ /** 计算哪些输出,默认 `['markdown']`;只算需要的,越少越快。 */
223
+ formats?: Format[]
224
+ /** 只保留正文,去掉导航、页眉页脚、侧边栏,默认 true。 */
225
+ onlyMainContent?: boolean
226
+ /** `auto`(默认,正文太少时改用浏览器渲染)、`http` 或 `browser`。 */
227
+ engine?: 'auto' | 'http' | 'browser'
228
+ /** 总超时,毫秒,至少 1000。 */
229
+ timeout?: number
230
+ /** 浏览器渲染时页面加载后再等待多久,毫秒。 */
231
+ waitFor?: number
232
+ /** 额外的请求头。 */
233
+ headers?: Record<string, string>
234
+ /** 这一页不检查 robots.txt(只对单页抓取有效)。 */
235
+ ignoreRobotsTxt?: boolean
236
+ /** 浏览器渲染时模拟手机。 */
237
+ mobile?: boolean
238
+ /** 浏览器渲染时屏蔽广告,默认 true。 */
239
+ blockAds?: boolean
240
+ /** 条件请求:`If-None-Match`。 */
241
+ ifNoneMatch?: string
242
+ /** 条件请求:`If-Modified-Since`(HTTP 日期)。 */
243
+ ifModifiedSince?: string
244
+ }
245
+
246
+ /** `Scraper` 的构造参数。 */
247
+ export interface ScraperConfig {
248
+ /** 请求头里的 User-Agent。 */
249
+ userAgent?: string
250
+ /** 允许抓取内网、回环地址(默认拒绝,防 SSRF;只作用于 HTTP 抓取)。 */
251
+ allowPrivateNetwork?: boolean
252
+ /** 同一主机两次请求之间的最小间隔,毫秒,默认 1000;robots.txt 的 Crawl-delay 优先。 */
253
+ requestIntervalMs?: number
254
+ /** 同一主机同时进行的请求数,默认 2。 */
255
+ perHostConcurrency?: number
256
+ /** 浏览器渲染时最多启动的浏览器进程数。 */
257
+ maxBrowsers?: number
258
+ }
package/index.js ADDED
@@ -0,0 +1,788 @@
1
+ /* eslint-disable */
2
+ // @ts-nocheck
3
+ /* auto-generated by NAPI-RS */
4
+
5
+ const { readFileSync } = require('fs')
6
+ let nativeBinding = null
7
+ // Which artifact actually loaded. The WASI fallback chain overwrites it with
8
+ // the flavor it resolved; the late native retry below leaves it alone because
9
+ // it only runs while no WASI candidate has been loaded.
10
+ let __napiLoadedBindingTarget = 'native'
11
+ const loadErrors = []
12
+
13
+ const isMusl = () => {
14
+ let musl = false
15
+ if (process.platform === 'linux') {
16
+ musl = isMuslFromFilesystem()
17
+ if (musl === null) {
18
+ musl = isMuslFromReport()
19
+ }
20
+ if (musl === null) {
21
+ musl = isMuslFromChildProcess()
22
+ }
23
+ }
24
+ return musl
25
+ }
26
+
27
+ const isFileMusl = (f) => f.includes('libc.musl-') || f.includes('ld-musl-')
28
+
29
+ const isMuslFromFilesystem = () => {
30
+ try {
31
+ return readFileSync('/usr/bin/ldd', 'utf-8').includes('musl')
32
+ } catch {
33
+ return null
34
+ }
35
+ }
36
+
37
+ const isMuslFromReport = () => {
38
+ let report = null
39
+ if (process.report && typeof process.report.getReport === 'function') {
40
+ process.report.excludeNetwork = true
41
+ report = process.report.getReport()
42
+ }
43
+ if (!report) {
44
+ return null
45
+ }
46
+ if (report.header && report.header.glibcVersionRuntime) {
47
+ return false
48
+ }
49
+ if (Array.isArray(report.sharedObjects)) {
50
+ if (report.sharedObjects.some(isFileMusl)) {
51
+ return true
52
+ }
53
+ }
54
+ return false
55
+ }
56
+
57
+ const isMuslFromChildProcess = () => {
58
+ try {
59
+ return require('child_process').execSync('ldd --version', { encoding: 'utf8' }).includes('musl')
60
+ } catch (e) {
61
+ // If we reach this case, we don't know if the system is musl or not, so is better to just fallback to false
62
+ return false
63
+ }
64
+ }
65
+
66
+ function requireNative() {
67
+ if (process.env.NAPI_RS_NATIVE_LIBRARY_PATH) {
68
+ try {
69
+ const overrideBinding = require(process.env.NAPI_RS_NATIVE_LIBRARY_PATH)
70
+ // The override may be a generated WASI loader, which already reports its
71
+ // own flavor. Adopt it: `module.exports` aliases this object, so claiming
72
+ // 'native' would both misreport the artifact and overwrite the loader's
73
+ // marker through the alias.
74
+ __napiLoadedBindingTarget =
75
+ overrideBinding && typeof overrideBinding.__napiBindingTarget === 'string'
76
+ ? overrideBinding.__napiBindingTarget
77
+ : 'native'
78
+ return overrideBinding
79
+ } catch (err) {
80
+ loadErrors.push(err)
81
+ }
82
+ } else if (process.platform === 'android') {
83
+ if (process.arch === 'arm64') {
84
+ try {
85
+ return require('./pagesift.android-arm64.node')
86
+ } catch (e) {
87
+ loadErrors.push(e)
88
+ }
89
+ try {
90
+ const binding = require('@pagesift/node-android-arm64')
91
+ const bindingPackageVersion = require('@pagesift/node-android-arm64/package.json').version
92
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
93
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
94
+ }
95
+ return binding
96
+ } catch (e) {
97
+ loadErrors.push(e)
98
+ }
99
+ } else if (process.arch === 'arm') {
100
+ try {
101
+ return require('./pagesift.android-arm-eabi.node')
102
+ } catch (e) {
103
+ loadErrors.push(e)
104
+ }
105
+ try {
106
+ const binding = require('@pagesift/node-android-arm-eabi')
107
+ const bindingPackageVersion = require('@pagesift/node-android-arm-eabi/package.json').version
108
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
109
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
110
+ }
111
+ return binding
112
+ } catch (e) {
113
+ loadErrors.push(e)
114
+ }
115
+ } else {
116
+ loadErrors.push(new Error(`Unsupported architecture on Android ${process.arch}`))
117
+ }
118
+ } else if (process.platform === 'win32') {
119
+ if (process.arch === 'x64') {
120
+ if ((process.config && process.config.variables && process.config.variables.shlib_suffix === 'dll.a') || (process.config && process.config.variables && process.config.variables.node_target_type === 'shared_library')) {
121
+ try {
122
+ return require('./pagesift.win32-x64-gnu.node')
123
+ } catch (e) {
124
+ loadErrors.push(e)
125
+ }
126
+ try {
127
+ const binding = require('@pagesift/node-win32-x64-gnu')
128
+ const bindingPackageVersion = require('@pagesift/node-win32-x64-gnu/package.json').version
129
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
130
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
131
+ }
132
+ return binding
133
+ } catch (e) {
134
+ loadErrors.push(e)
135
+ }
136
+ } else {
137
+ try {
138
+ return require('./pagesift.win32-x64-msvc.node')
139
+ } catch (e) {
140
+ loadErrors.push(e)
141
+ }
142
+ try {
143
+ const binding = require('@pagesift/node-win32-x64-msvc')
144
+ const bindingPackageVersion = require('@pagesift/node-win32-x64-msvc/package.json').version
145
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
146
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
147
+ }
148
+ return binding
149
+ } catch (e) {
150
+ loadErrors.push(e)
151
+ }
152
+ }
153
+ } else if (process.arch === 'ia32') {
154
+ try {
155
+ return require('./pagesift.win32-ia32-msvc.node')
156
+ } catch (e) {
157
+ loadErrors.push(e)
158
+ }
159
+ try {
160
+ const binding = require('@pagesift/node-win32-ia32-msvc')
161
+ const bindingPackageVersion = require('@pagesift/node-win32-ia32-msvc/package.json').version
162
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
163
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
164
+ }
165
+ return binding
166
+ } catch (e) {
167
+ loadErrors.push(e)
168
+ }
169
+ } else if (process.arch === 'arm64') {
170
+ try {
171
+ return require('./pagesift.win32-arm64-msvc.node')
172
+ } catch (e) {
173
+ loadErrors.push(e)
174
+ }
175
+ try {
176
+ const binding = require('@pagesift/node-win32-arm64-msvc')
177
+ const bindingPackageVersion = require('@pagesift/node-win32-arm64-msvc/package.json').version
178
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
179
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
180
+ }
181
+ return binding
182
+ } catch (e) {
183
+ loadErrors.push(e)
184
+ }
185
+ } else {
186
+ loadErrors.push(new Error(`Unsupported architecture on Windows: ${process.arch}`))
187
+ }
188
+ } else if (process.platform === 'darwin') {
189
+ try {
190
+ return require('./pagesift.darwin-universal.node')
191
+ } catch (e) {
192
+ loadErrors.push(e)
193
+ }
194
+ try {
195
+ const binding = require('@pagesift/node-darwin-universal')
196
+ const bindingPackageVersion = require('@pagesift/node-darwin-universal/package.json').version
197
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
198
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
199
+ }
200
+ return binding
201
+ } catch (e) {
202
+ loadErrors.push(e)
203
+ }
204
+ if (process.arch === 'x64') {
205
+ try {
206
+ return require('./pagesift.darwin-x64.node')
207
+ } catch (e) {
208
+ loadErrors.push(e)
209
+ }
210
+ try {
211
+ const binding = require('@pagesift/node-darwin-x64')
212
+ const bindingPackageVersion = require('@pagesift/node-darwin-x64/package.json').version
213
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
214
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
215
+ }
216
+ return binding
217
+ } catch (e) {
218
+ loadErrors.push(e)
219
+ }
220
+ } else if (process.arch === 'arm64') {
221
+ try {
222
+ return require('./pagesift.darwin-arm64.node')
223
+ } catch (e) {
224
+ loadErrors.push(e)
225
+ }
226
+ try {
227
+ const binding = require('@pagesift/node-darwin-arm64')
228
+ const bindingPackageVersion = require('@pagesift/node-darwin-arm64/package.json').version
229
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
230
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
231
+ }
232
+ return binding
233
+ } catch (e) {
234
+ loadErrors.push(e)
235
+ }
236
+ } else {
237
+ loadErrors.push(new Error(`Unsupported architecture on macOS: ${process.arch}`))
238
+ }
239
+ } else if (process.platform === 'freebsd') {
240
+ if (process.arch === 'x64') {
241
+ try {
242
+ return require('./pagesift.freebsd-x64.node')
243
+ } catch (e) {
244
+ loadErrors.push(e)
245
+ }
246
+ try {
247
+ const binding = require('@pagesift/node-freebsd-x64')
248
+ const bindingPackageVersion = require('@pagesift/node-freebsd-x64/package.json').version
249
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
250
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
251
+ }
252
+ return binding
253
+ } catch (e) {
254
+ loadErrors.push(e)
255
+ }
256
+ } else if (process.arch === 'arm64') {
257
+ try {
258
+ return require('./pagesift.freebsd-arm64.node')
259
+ } catch (e) {
260
+ loadErrors.push(e)
261
+ }
262
+ try {
263
+ const binding = require('@pagesift/node-freebsd-arm64')
264
+ const bindingPackageVersion = require('@pagesift/node-freebsd-arm64/package.json').version
265
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
266
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
267
+ }
268
+ return binding
269
+ } catch (e) {
270
+ loadErrors.push(e)
271
+ }
272
+ } else {
273
+ loadErrors.push(new Error(`Unsupported architecture on FreeBSD: ${process.arch}`))
274
+ }
275
+ } else if (process.platform === 'linux') {
276
+ if (process.arch === 'x64') {
277
+ if (isMusl()) {
278
+ try {
279
+ return require('./pagesift.linux-x64-musl.node')
280
+ } catch (e) {
281
+ loadErrors.push(e)
282
+ }
283
+ try {
284
+ const binding = require('@pagesift/node-linux-x64-musl')
285
+ const bindingPackageVersion = require('@pagesift/node-linux-x64-musl/package.json').version
286
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
287
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
288
+ }
289
+ return binding
290
+ } catch (e) {
291
+ loadErrors.push(e)
292
+ }
293
+ } else {
294
+ try {
295
+ return require('./pagesift.linux-x64-gnu.node')
296
+ } catch (e) {
297
+ loadErrors.push(e)
298
+ }
299
+ try {
300
+ const binding = require('@pagesift/node-linux-x64-gnu')
301
+ const bindingPackageVersion = require('@pagesift/node-linux-x64-gnu/package.json').version
302
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
303
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
304
+ }
305
+ return binding
306
+ } catch (e) {
307
+ loadErrors.push(e)
308
+ }
309
+ }
310
+ } else if (process.arch === 'arm64') {
311
+ if (isMusl()) {
312
+ try {
313
+ return require('./pagesift.linux-arm64-musl.node')
314
+ } catch (e) {
315
+ loadErrors.push(e)
316
+ }
317
+ try {
318
+ const binding = require('@pagesift/node-linux-arm64-musl')
319
+ const bindingPackageVersion = require('@pagesift/node-linux-arm64-musl/package.json').version
320
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
321
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
322
+ }
323
+ return binding
324
+ } catch (e) {
325
+ loadErrors.push(e)
326
+ }
327
+ } else {
328
+ try {
329
+ return require('./pagesift.linux-arm64-gnu.node')
330
+ } catch (e) {
331
+ loadErrors.push(e)
332
+ }
333
+ try {
334
+ const binding = require('@pagesift/node-linux-arm64-gnu')
335
+ const bindingPackageVersion = require('@pagesift/node-linux-arm64-gnu/package.json').version
336
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
337
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
338
+ }
339
+ return binding
340
+ } catch (e) {
341
+ loadErrors.push(e)
342
+ }
343
+ }
344
+ } else if (process.arch === 'arm') {
345
+ if (isMusl()) {
346
+ try {
347
+ return require('./pagesift.linux-arm-musleabihf.node')
348
+ } catch (e) {
349
+ loadErrors.push(e)
350
+ }
351
+ try {
352
+ const binding = require('@pagesift/node-linux-arm-musleabihf')
353
+ const bindingPackageVersion = require('@pagesift/node-linux-arm-musleabihf/package.json').version
354
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
355
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
356
+ }
357
+ return binding
358
+ } catch (e) {
359
+ loadErrors.push(e)
360
+ }
361
+ } else {
362
+ try {
363
+ return require('./pagesift.linux-arm-gnueabihf.node')
364
+ } catch (e) {
365
+ loadErrors.push(e)
366
+ }
367
+ try {
368
+ const binding = require('@pagesift/node-linux-arm-gnueabihf')
369
+ const bindingPackageVersion = require('@pagesift/node-linux-arm-gnueabihf/package.json').version
370
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
371
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
372
+ }
373
+ return binding
374
+ } catch (e) {
375
+ loadErrors.push(e)
376
+ }
377
+ }
378
+ } else if (process.arch === 'loong64') {
379
+ if (isMusl()) {
380
+ try {
381
+ return require('./pagesift.linux-loong64-musl.node')
382
+ } catch (e) {
383
+ loadErrors.push(e)
384
+ }
385
+ try {
386
+ const binding = require('@pagesift/node-linux-loong64-musl')
387
+ const bindingPackageVersion = require('@pagesift/node-linux-loong64-musl/package.json').version
388
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
389
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
390
+ }
391
+ return binding
392
+ } catch (e) {
393
+ loadErrors.push(e)
394
+ }
395
+ } else {
396
+ try {
397
+ return require('./pagesift.linux-loong64-gnu.node')
398
+ } catch (e) {
399
+ loadErrors.push(e)
400
+ }
401
+ try {
402
+ const binding = require('@pagesift/node-linux-loong64-gnu')
403
+ const bindingPackageVersion = require('@pagesift/node-linux-loong64-gnu/package.json').version
404
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
405
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
406
+ }
407
+ return binding
408
+ } catch (e) {
409
+ loadErrors.push(e)
410
+ }
411
+ }
412
+ } else if (process.arch === 'riscv64') {
413
+ if (isMusl()) {
414
+ try {
415
+ return require('./pagesift.linux-riscv64-musl.node')
416
+ } catch (e) {
417
+ loadErrors.push(e)
418
+ }
419
+ try {
420
+ const binding = require('@pagesift/node-linux-riscv64-musl')
421
+ const bindingPackageVersion = require('@pagesift/node-linux-riscv64-musl/package.json').version
422
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
423
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
424
+ }
425
+ return binding
426
+ } catch (e) {
427
+ loadErrors.push(e)
428
+ }
429
+ } else {
430
+ try {
431
+ return require('./pagesift.linux-riscv64-gnu.node')
432
+ } catch (e) {
433
+ loadErrors.push(e)
434
+ }
435
+ try {
436
+ const binding = require('@pagesift/node-linux-riscv64-gnu')
437
+ const bindingPackageVersion = require('@pagesift/node-linux-riscv64-gnu/package.json').version
438
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
439
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
440
+ }
441
+ return binding
442
+ } catch (e) {
443
+ loadErrors.push(e)
444
+ }
445
+ }
446
+ } else if (process.arch === 'ppc64') {
447
+ try {
448
+ return require('./pagesift.linux-ppc64-gnu.node')
449
+ } catch (e) {
450
+ loadErrors.push(e)
451
+ }
452
+ try {
453
+ const binding = require('@pagesift/node-linux-ppc64-gnu')
454
+ const bindingPackageVersion = require('@pagesift/node-linux-ppc64-gnu/package.json').version
455
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
456
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
457
+ }
458
+ return binding
459
+ } catch (e) {
460
+ loadErrors.push(e)
461
+ }
462
+ } else if (process.arch === 's390x') {
463
+ try {
464
+ return require('./pagesift.linux-s390x-gnu.node')
465
+ } catch (e) {
466
+ loadErrors.push(e)
467
+ }
468
+ try {
469
+ const binding = require('@pagesift/node-linux-s390x-gnu')
470
+ const bindingPackageVersion = require('@pagesift/node-linux-s390x-gnu/package.json').version
471
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
472
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
473
+ }
474
+ return binding
475
+ } catch (e) {
476
+ loadErrors.push(e)
477
+ }
478
+ } else {
479
+ loadErrors.push(new Error(`Unsupported architecture on Linux: ${process.arch}`))
480
+ }
481
+ } else if (process.platform === 'openharmony') {
482
+ if (process.arch === 'arm64') {
483
+ try {
484
+ return require('./pagesift.openharmony-arm64.node')
485
+ } catch (e) {
486
+ loadErrors.push(e)
487
+ }
488
+ try {
489
+ const binding = require('@pagesift/node-openharmony-arm64')
490
+ const bindingPackageVersion = require('@pagesift/node-openharmony-arm64/package.json').version
491
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
492
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
493
+ }
494
+ return binding
495
+ } catch (e) {
496
+ loadErrors.push(e)
497
+ }
498
+ } else if (process.arch === 'x64') {
499
+ try {
500
+ return require('./pagesift.openharmony-x64.node')
501
+ } catch (e) {
502
+ loadErrors.push(e)
503
+ }
504
+ try {
505
+ const binding = require('@pagesift/node-openharmony-x64')
506
+ const bindingPackageVersion = require('@pagesift/node-openharmony-x64/package.json').version
507
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
508
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
509
+ }
510
+ return binding
511
+ } catch (e) {
512
+ loadErrors.push(e)
513
+ }
514
+ } else if (process.arch === 'arm') {
515
+ try {
516
+ return require('./pagesift.openharmony-arm.node')
517
+ } catch (e) {
518
+ loadErrors.push(e)
519
+ }
520
+ try {
521
+ const binding = require('@pagesift/node-openharmony-arm')
522
+ const bindingPackageVersion = require('@pagesift/node-openharmony-arm/package.json').version
523
+ if (bindingPackageVersion !== '0.40.0' && process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
524
+ throw new Error(`Native binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
525
+ }
526
+ return binding
527
+ } catch (e) {
528
+ loadErrors.push(e)
529
+ }
530
+ } else {
531
+ loadErrors.push(new Error(`Unsupported architecture on OpenHarmony: ${process.arch}`))
532
+ }
533
+ } else {
534
+ loadErrors.push(new Error(`Unsupported OS: ${process.platform}, architecture: ${process.arch}`))
535
+ }
536
+ }
537
+
538
+ function createLoadErrorChain(errors) {
539
+ return errors.reduce((previous, current) => {
540
+ let message
541
+ try {
542
+ message =
543
+ current && typeof current.message === 'string'
544
+ ? current.message
545
+ : String(current)
546
+ } catch {
547
+ message = 'Unknown error'
548
+ }
549
+ const error = new Error(message)
550
+ error.cause = previous
551
+ return error
552
+ }, null)
553
+ }
554
+
555
+ // NAPI_RS_FORCE_WASI is a tri-state flag:
556
+ // unset / any other value → native binding preferred, WASI is only a fallback
557
+ // 'true' → prefer WASI, but retain native as a lazy fallback
558
+ // 'error' → require WASI without initializing a native fallback
559
+ // Treating any non-empty string as truthy (the historical behavior) meant
560
+ // NAPI_RS_FORCE_WASI=false, NAPI_RS_FORCE_WASI=0, etc. inadvertently triggered
561
+ // the WASI path, causing ENOENT for packages shipped without a .wasi.cjs file.
562
+ //
563
+ // NAPI_RS_WASI_FLAVOR selects one exact generated flavor and implies strict
564
+ // WASI loading. It never crosses into another flavor or falls back to native.
565
+ const __napiWasiFlavors = ['wasm32-wasi']
566
+ const __napiWasiFlavor = process.env.NAPI_RS_WASI_FLAVOR
567
+ const __napiWasiFlavorRequested =
568
+ typeof __napiWasiFlavor === 'string' && __napiWasiFlavor.length > 0
569
+ if (
570
+ __napiWasiFlavorRequested &&
571
+ __napiWasiFlavors.indexOf(__napiWasiFlavor) === -1
572
+ ) {
573
+ throw new Error(
574
+ 'Unsupported WASI flavor "' +
575
+ __napiWasiFlavor +
576
+ '". Available flavors: ' +
577
+ __napiWasiFlavors.join(', '),
578
+ )
579
+ }
580
+ const forceWasiError = process.env.NAPI_RS_FORCE_WASI === 'error'
581
+ const forceWasi =
582
+ process.env.NAPI_RS_FORCE_WASI === 'true' ||
583
+ forceWasiError ||
584
+ __napiWasiFlavorRequested
585
+
586
+ if (!forceWasi) {
587
+ nativeBinding = requireNative()
588
+ }
589
+
590
+ if (!nativeBinding || forceWasi) {
591
+ let wasiBinding = null
592
+ let wasiBindingLoaded = false
593
+ const wasiBindingErrors = []
594
+ const __napiWasiResolveCandidate = (specifier, isPackage, localArtifacts) => {
595
+ try {
596
+ require.resolve(specifier)
597
+ } catch (resolveError) {
598
+ if (!resolveError || resolveError.code !== 'MODULE_NOT_FOUND') {
599
+ throw resolveError
600
+ }
601
+ if (isPackage) {
602
+ try {
603
+ require.resolve(specifier + '/package.json')
604
+ } catch (packageError) {
605
+ if (packageError && packageError.code === 'MODULE_NOT_FOUND') {
606
+ return resolveError
607
+ }
608
+ // An exports restriction proves the package exists even when its
609
+ // package.json is not public. Preserve the root resolution failure.
610
+ throw resolveError
611
+ }
612
+ // The package exists but its main/export target is broken.
613
+ throw resolveError
614
+ }
615
+ return resolveError
616
+ }
617
+ if (localArtifacts) {
618
+ let artifactError = null
619
+ for (let i = 0; i < localArtifacts.length; i++) {
620
+ try {
621
+ require.resolve(localArtifacts[i])
622
+ return null
623
+ } catch (resolveError) {
624
+ if (!resolveError || resolveError.code !== 'MODULE_NOT_FOUND') {
625
+ throw resolveError
626
+ }
627
+ artifactError = resolveError
628
+ }
629
+ }
630
+ return artifactError
631
+ }
632
+ return null
633
+ }
634
+ if (!wasiBindingLoaded && (!__napiWasiFlavorRequested || __napiWasiFlavor === 'wasm32-wasi')) {
635
+ let candidateError = null
636
+ let candidateFailed = false
637
+ try {
638
+ candidateError = __napiWasiResolveCandidate('./pagesift.wasi.cjs', false, ['./pagesift.wasm32-wasi.debug.wasm', './pagesift.wasm32-wasi.wasm'])
639
+ candidateFailed = candidateError !== null
640
+ if (!candidateFailed) {
641
+ wasiBinding = require('./pagesift.wasi.cjs')
642
+ nativeBinding = wasiBinding
643
+ __napiLoadedBindingTarget = 'wasm32-wasi'
644
+ wasiBindingLoaded = true
645
+ }
646
+ } catch (err) {
647
+ candidateError = err
648
+ candidateFailed = true
649
+ }
650
+ if (candidateFailed) {
651
+ wasiBindingErrors.push(candidateError)
652
+ loadErrors.push(candidateError)
653
+ }
654
+ }
655
+ if (!wasiBindingLoaded && (!__napiWasiFlavorRequested || __napiWasiFlavor === 'wasm32-wasi')) {
656
+ let candidateError = null
657
+ let candidateFailed = false
658
+ try {
659
+ candidateError = __napiWasiResolveCandidate('pagesift-wasm32-wasi', true, undefined)
660
+ candidateFailed = candidateError !== null
661
+ if (!candidateFailed) {
662
+ if (process.env.NAPI_RS_ENFORCE_VERSION_CHECK && process.env.NAPI_RS_ENFORCE_VERSION_CHECK !== '0') {
663
+ const bindingPackageVersion = require('@pagesift/node-wasm32-wasi/package.json').version
664
+ if (bindingPackageVersion !== '0.40.0') {
665
+ throw new Error(`WASI binding package version mismatch, expected 0.40.0 but got ${bindingPackageVersion}. You can reinstall dependencies to fix this issue.`)
666
+ }
667
+ }
668
+ wasiBinding = require('@pagesift/node-wasm32-wasi')
669
+ nativeBinding = wasiBinding
670
+ __napiLoadedBindingTarget = 'wasm32-wasi'
671
+ wasiBindingLoaded = true
672
+ }
673
+ } catch (err) {
674
+ candidateError = err
675
+ candidateFailed = true
676
+ }
677
+ if (candidateFailed) {
678
+ wasiBindingErrors.push(candidateError)
679
+ loadErrors.push(candidateError)
680
+ }
681
+ }
682
+ if (
683
+ !wasiBindingLoaded &&
684
+ forceWasi &&
685
+ !forceWasiError &&
686
+ !__napiWasiFlavorRequested
687
+ ) {
688
+ nativeBinding = requireNative()
689
+ }
690
+ if ((forceWasiError || __napiWasiFlavorRequested) && !wasiBindingLoaded) {
691
+ const error = new Error(
692
+ __napiWasiFlavorRequested
693
+ ? 'WASI binding for flavor "' + __napiWasiFlavor + '" not found'
694
+ : 'WASI binding not found and NAPI_RS_FORCE_WASI is set to error',
695
+ )
696
+ error.cause = createLoadErrorChain(wasiBindingErrors)
697
+ throw error
698
+ }
699
+ }
700
+
701
+ if (!nativeBinding) {
702
+ if (loadErrors.length > 0) {
703
+ const error = new Error(
704
+ `Cannot find native binding. ` +
705
+ `npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). ` +
706
+ 'Please try `npm i` again after removing both package-lock.json and node_modules directory.',
707
+ )
708
+ // assign instead of the `new Error(message, { cause })` options form,
709
+ // which Node < 16.9 silently ignores
710
+ error.cause = createLoadErrorChain(loadErrors)
711
+ throw error
712
+ }
713
+ throw new Error(`Failed to load native binding`)
714
+ }
715
+
716
+ function __napiStampBindingTarget(exportsObject, target) {
717
+ if (
718
+ Object.prototype.hasOwnProperty.call(exportsObject, '__napiBindingTarget')
719
+ ) {
720
+ if (exportsObject.__napiBindingTarget === target) {
721
+ // Already ours: the root entry aliases the object it loaded, so a WASI
722
+ // fallback candidate — or a `NAPI_RS_NATIVE_LIBRARY_PATH` override that
723
+ // is a generated loader — arrives already stamped with this same value.
724
+ return target
725
+ }
726
+ const error = new Error(
727
+ '`__napiBindingTarget` is reserved by the generated binding loader, but the loaded binding already exports it. Rename the export, e.g. #[napi(js_name = "...")].',
728
+ )
729
+ error.code = 'ERR_NAPI_BINDING_TARGET_CONFLICT'
730
+ throw error
731
+ }
732
+ if (!Object.isExtensible(exportsObject)) {
733
+ // A `#[napi(module_exports)]` hook may seal or freeze this object
734
+ // (`Object::seal` / `Object::freeze`). Reporting the artifact is metadata,
735
+ // never a reason to fail an otherwise successful load, so the stamp is
736
+ // skipped. What a consumer still sees then follows the entry point: the
737
+ // browser and deferred loaders declare `__napiBindingTarget` at module
738
+ // level and go on reporting it, while the CommonJS entries hand back this
739
+ // very object as `module.exports`, so there the value is absent.
740
+ return target
741
+ }
742
+ try {
743
+ // [[Define]], not [[Set]]: an ordinary assignment walks the prototype
744
+ // chain, so an inherited accessor could swallow the value or throw and
745
+ // fail an otherwise successful load. The descriptor is what a successful
746
+ // assignment would have produced.
747
+ Object.defineProperty(exportsObject, '__napiBindingTarget', {
748
+ configurable: true,
749
+ enumerable: true,
750
+ value: target,
751
+ writable: true,
752
+ })
753
+ } catch {
754
+ // Same rule as the non-extensible skip above: reporting the artifact is
755
+ // metadata, never a reason to fail an otherwise successful load. An exotic
756
+ // object (a Proxy whose defineProperty trap refuses) is skipped, not
757
+ // thrown over.
758
+ }
759
+ // The CommonJS loaders assign this return value so `cjs-module-lexer` — and
760
+ // therefore Node's CJS -> ESM named export detection — can see
761
+ // `__napiBindingTarget` statically.
762
+ return target
763
+ }
764
+ // Stamp before the alias, not after. The guard only reads `nativeBinding`
765
+ // (`hasOwnProperty` plus a comparison), which is safe against any addon
766
+ // accessor; an assignment is not, because a `#[napi(module_exports)]` hook can
767
+ // expose a getter reporting this very value and a setter that throws. So the
768
+ // assignment lands on the loader's own `module.exports`, still the original
769
+ // object here, and the alias below replaces it.
770
+ //
771
+ // The assignment is what keeps the marker a statically visible CommonJS export:
772
+ // `cjs-module-lexer` is Node's CJS -> ESM named export detection, it cannot see
773
+ // a bare call, and the later `module.exports = nativeBinding` does not undo the
774
+ // detection. The assignment itself always succeeds — its target is this
775
+ // loader's own, still extensible `module.exports` — and the alias below then
776
+ // discards the value it wrote. What a consumer reads is whatever the guard put
777
+ // on `nativeBinding`, so on a frozen binding, where the guard skips, the
778
+ // linked import resolves to `undefined`.
779
+ module.exports.__napiBindingTarget = __napiStampBindingTarget(nativeBinding, __napiLoadedBindingTarget)
780
+ module.exports = nativeBinding
781
+ module.exports.Page = nativeBinding.Page
782
+ module.exports.Scraper = nativeBinding.Scraper
783
+ module.exports.ScrapeStream = nativeBinding.ScrapeStream
784
+ module.exports.convert = nativeBinding.convert
785
+ module.exports.convertAsync = nativeBinding.convertAsync
786
+ module.exports.convertMany = nativeBinding.convertMany
787
+ module.exports.PagesiftError = nativeBinding.PagesiftError
788
+ module.exports.version = nativeBinding.version
package/package.json CHANGED
@@ -1,6 +1,61 @@
1
1
  {
2
2
  "name": "@pagesift/node",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.50.0",
4
+ "description": "Turn web pages into clean Markdown for LLMs: a polite, Firecrawl-compatible scraping engine written in Rust.",
5
+ "license": "AGPL-3.0-only",
6
+ "repository": {
7
+ "type": "git",
8
+ "url": "git+https://github.com/YOLOW92/pagesift.git",
9
+ "directory": "crates/pagesift-node"
10
+ },
11
+ "keywords": [
12
+ "markdown",
13
+ "scraper",
14
+ "crawler",
15
+ "firecrawl",
16
+ "llm",
17
+ "html-to-markdown",
18
+ "napi-rs"
19
+ ],
20
+ "main": "index.js",
21
+ "types": "index.d.ts",
22
+ "files": [
23
+ "index.js",
24
+ "index.d.ts"
25
+ ],
26
+ "engines": {
27
+ "node": ">= 18"
28
+ },
29
+ "napi": {
30
+ "binaryName": "pagesift",
31
+ "targets": [
32
+ "x86_64-unknown-linux-gnu",
33
+ "aarch64-unknown-linux-gnu",
34
+ "x86_64-apple-darwin",
35
+ "aarch64-apple-darwin",
36
+ "x86_64-unknown-linux-musl",
37
+ "x86_64-pc-windows-msvc"
38
+ ],
39
+ "dtsHeaderFile": "header.d.ts"
40
+ },
41
+ "scripts": {
42
+ "build": "napi build --platform --release && node scripts/postbuild.mjs",
43
+ "build:debug": "napi build --platform && node scripts/postbuild.mjs",
44
+ "test": "node --test --test-concurrency=1",
45
+ "bench": "node bench/bench.mjs",
46
+ "artifacts": "napi artifacts",
47
+ "prepublishOnly": "napi prepublish -t npm --no-gh-release",
48
+ "version": "napi version"
49
+ },
50
+ "devDependencies": {
51
+ "@napi-rs/cli": "3.10.8"
52
+ },
53
+ "optionalDependencies": {
54
+ "@pagesift/node-linux-x64-gnu": "0.50.0",
55
+ "@pagesift/node-linux-arm64-gnu": "0.50.0",
56
+ "@pagesift/node-darwin-x64": "0.50.0",
57
+ "@pagesift/node-darwin-arm64": "0.50.0",
58
+ "@pagesift/node-linux-x64-musl": "0.50.0",
59
+ "@pagesift/node-win32-x64-msvc": "0.50.0"
60
+ }
61
+ }