@hunt-labs/bytekit-sdk 0.7.4 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +138 -2
- package/dist/generated/api.d.ts +219 -131
- package/dist/index.d.ts +1 -1
- package/dist/resources/fetch.d.ts +8 -36
- package/dist/resources/fetch.js +3 -5
- package/dist/resources/scrape.d.ts +1 -13
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -10,6 +10,130 @@ today, see [README.md](./README.md), which remains the authoritative usage guide
|
|
|
10
10
|
|
|
11
11
|
## [Unreleased]
|
|
12
12
|
|
|
13
|
+
## [0.9.0] - 2026-08-20
|
|
14
|
+
|
|
15
|
+
### Removed
|
|
16
|
+
|
|
17
|
+
- **BREAKING: the `cookies` and `headers` request-body fields are gone from every capture
|
|
18
|
+
endpoint** (#3242) — `/v1/scrape`, `/v1/scrape/bulk`, `/v1/screenshots`, `/v1/recordings` and
|
|
19
|
+
`/v1/schema`. A request that carries either field is rejected with `400 validation_error`; it
|
|
20
|
+
is not ignored, and there is no deprecation window, feature flag, or compatibility header.
|
|
21
|
+
`ScrapeOpts.cookies`, `ScrapeOpts.headers` and the exported `ScrapeCookie` type are removed
|
|
22
|
+
from the SDK surface, so a call that still passes them no longer compiles.
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
// before — compiled, and the values were forwarded to the target site
|
|
26
|
+
await client.scrape.create({ url, cookies: [{ name: 'session', value: 'abc' }] });
|
|
27
|
+
// after — does not compile; the same body over raw HTTP is a 400 validation_error
|
|
28
|
+
await client.scrape.create({ url });
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
ByteKit continues to manage its own cookie jar and request headers internally; what was removed
|
|
32
|
+
is the caller's ability to inject or override them.
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
|
|
36
|
+
- **The generated types now declare the `503` response of `GET`/`POST /v1/fetch`** (#3372).
|
|
37
|
+
`/v1/fetch` answers `503` on two conditions and both carry the ordinary `Error` envelope, so
|
|
38
|
+
a typed consumer reading `operations['getFetch']['responses']` / `['postFetch']` can handle it
|
|
39
|
+
instead of meeting an undeclared status at runtime. Read `error.code` to tell the two apart,
|
|
40
|
+
because they disagree about `error.http_status`:
|
|
41
|
+
- `upstream_error` — the target returned HTTP `502`, relayed at `503` because Cloudflare
|
|
42
|
+
replaces a clean origin 502 with its own error page. `error.http_status` stays `502`, the
|
|
43
|
+
**true upstream status**, and is the only surviving record of what the target answered.
|
|
44
|
+
- `internal_error` (`Fetch service unavailable.`) — the fetch service was unreachable. There
|
|
45
|
+
is no upstream, so `error.http_status` reports `503`, matching what was served.
|
|
46
|
+
|
|
47
|
+
No runtime behaviour changes, and every other documented `/v1/fetch` status is unaffected.
|
|
48
|
+
|
|
49
|
+
## [0.8.0] - 2026-08-19
|
|
50
|
+
|
|
51
|
+
### Removed
|
|
52
|
+
|
|
53
|
+
- **BREAKING: `billing_multiplier` and `billed_bytes` are gone from the scrape success
|
|
54
|
+
envelope** (#3143). Read `billing` instead — a `BillingNode` that names every factor
|
|
55
|
+
applied to the charge, the multiplier they compose to, and the resulting billed bytes:
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
// before
|
|
59
|
+
const spend = res.billed_bytes; // flat, base-endpoint only
|
|
60
|
+
// after
|
|
61
|
+
const spend = res.billing?.billed_bytes;
|
|
62
|
+
const why = res.billing?.factors; // [{ name: 'endpoint', value: 1.5, reason: 'scrape_md' }, …]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
This is a correction, not a rename: the two surfaces reported **different numbers for the
|
|
66
|
+
same charge**. The flat pair carried the BASE-endpoint composition, so a markdown cache hit
|
|
67
|
+
read `0.5` / `400` while the real charge — and the `X-Billing-Multiplier` header — was
|
|
68
|
+
`0.75` / `600`; an html-only miss read the per-format credits estimate `1.5` / `150` against
|
|
69
|
+
an endpoint factor of `1.0` / `100`. The flat pair also never carried the `clean_markdown`
|
|
70
|
+
×3 surcharge, so on a successful clean it under-reported the amount billed by two thirds.
|
|
71
|
+
`billing.multiplier` composes every applied factor behind exactly one rounding, so it is the
|
|
72
|
+
number that predicts your invoice.
|
|
73
|
+
|
|
74
|
+
`billing` is optional and is **absent** on resources that completed before the billing node
|
|
75
|
+
existed. A pre-node row omits the field rather than reporting a zero node, because a zero
|
|
76
|
+
node would assert that a real past charge was zero. Test the node for presence
|
|
77
|
+
(`res.billing !== undefined`, or equivalently a plain truthiness check — `billing` is an
|
|
78
|
+
object or `undefined`, and an object is always truthy, so the two are the same test here).
|
|
79
|
+
The hazard is a condition that reaches _into_ the node: `if (res.billing?.billed_bytes)`
|
|
80
|
+
silently skips a genuine zero-byte charge, because `0` is falsy. Check that the node is
|
|
81
|
+
present, then read `billed_bytes` from it.
|
|
82
|
+
|
|
83
|
+
```ts
|
|
84
|
+
// wrong — a real 0-byte charge is falsy and disappears
|
|
85
|
+
if (res.billing?.billed_bytes) record(res.billing.billed_bytes);
|
|
86
|
+
// right — presence first, value second
|
|
87
|
+
if (res.billing) record(res.billing.billed_bytes); // 0 is a charge, not an absence
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Two endpoints deliberately carry no node: `/v1/search` is billed per credit and reports
|
|
91
|
+
`credits_used`, and a single `/v1/fetch` reports its charge through response headers only.
|
|
92
|
+
The `/v1/logs` row field named `billing_multiplier` is a **different field** and is not the
|
|
93
|
+
envelope field this release removed. It carries the BASE settled `credits_charged` factor,
|
|
94
|
+
not the composed real-endpoint value the `X-Billing-Multiplier` header reports — a markdown
|
|
95
|
+
cache hit reads `0.5` on the logs row and `0.75` on the header.
|
|
96
|
+
|
|
97
|
+
### Changed
|
|
98
|
+
|
|
99
|
+
- Generated `ScrapeErrorEnvelope` types now describe terminal asynchronous `scrape.failed`
|
|
100
|
+
webhook delivery (#1856). No runtime SDK behavior or API shape changed.
|
|
101
|
+
|
|
102
|
+
### Fixed
|
|
103
|
+
|
|
104
|
+
- **`GET /v1/logs` entries now carry `billing_multiplier`, as a JSON number** (#3022). The
|
|
105
|
+
gateway had been serializing that value under an undeclared key `credits`, and as the raw
|
|
106
|
+
Postgres `NUMERIC(10,2)` string (`"1.50"`), while the OpenAPI spec — and therefore the
|
|
107
|
+
generated types in `src/generated/api.d.ts` — declared `billing_multiplier?: number | null`.
|
|
108
|
+
Anything typed against this package read `undefined` for the field.
|
|
109
|
+
|
|
110
|
+
**Breaking at the wire level:** the undeclared `credits` key is gone from every `/v1/logs`
|
|
111
|
+
entry. Code that reached past the generated types to read `credits` must read
|
|
112
|
+
`billing_multiplier`. No SDK method exposes this endpoint, so the only change inside this
|
|
113
|
+
package is the regenerated type description.
|
|
114
|
+
|
|
115
|
+
- **`change_threshold` (on `monitors.create`/`monitors.list`/`monitors.get`/
|
|
116
|
+
`monitors.update` responses) and `change_pct` (on `monitors.captures.list` entries)
|
|
117
|
+
now serialize as JSON numbers, not strings** (#3021). Both are Postgres `NUMERIC(5,2)`
|
|
118
|
+
columns; the gateway passed the raw driver string through unconverted (`"5.00"` /
|
|
119
|
+
`"7.25"`) while the OpenAPI spec's _request_ schemas already declared `number` — only
|
|
120
|
+
the _response_ schemas had been (incorrectly) edited to match the buggy string output.
|
|
121
|
+
This release fixes the gateway serializer and corrects the two response schemas to
|
|
122
|
+
`number` / `number | null`, so spec and runtime now agree.
|
|
123
|
+
|
|
124
|
+
**Breaking at the wire level:** `client.monitors.create()`, `client.monitors.list()`,
|
|
125
|
+
`client.monitors.get()`, and `client.monitors.update()` now return a JS number for
|
|
126
|
+
`change_threshold` instead of a string (screenshot-type monitors only); the create
|
|
127
|
+
response's `MonitorCreateResponse` and every other monitor route serialize through the
|
|
128
|
+
same `MonitorResponse` schema, so all four are affected identically.
|
|
129
|
+
`client.monitors.captures.list()` now returns a JS number for `change_pct` instead of a
|
|
130
|
+
string. Code doing `parseFloat(monitor.change_threshold)` or template-literal
|
|
131
|
+
interpolation keeps working; strict `typeof x === 'string'` checks or string methods
|
|
132
|
+
(`.trim()`, `.padStart()`) will now throw. `change_pct` is `null` (not `0`) when there is
|
|
133
|
+
no prior capture to diff against, unchanged from before. No SDK method-level shape
|
|
134
|
+
changed otherwise — the only change inside this package is the regenerated type
|
|
135
|
+
description.
|
|
136
|
+
|
|
13
137
|
## [0.7.4] - 2026-08-19
|
|
14
138
|
|
|
15
139
|
### Changed
|
|
@@ -157,6 +281,16 @@ behave exactly as before.
|
|
|
157
281
|
Bumping moves the anchor past those comments. The published `0.7.0` artifact is **not** stale in
|
|
158
282
|
any way a consumer can observe — the divergence is comment text only.
|
|
159
283
|
|
|
284
|
+
- **Corrected a false legacy-name claim carried since the rename (#2950).** The `0.7.0`-era note
|
|
285
|
+
below said the prior `rapidcrawl`/`@rapidcrawl/sdk` name "remains installable at its last
|
|
286
|
+
version but is deprecated." Verified 2026-08-13: both `rapidcrawl` and `@rapidcrawl/sdk` return
|
|
287
|
+
npm `404` ("is not in this registry"), and `npm deprecate` against either fails with the same
|
|
288
|
+
`404` — there is no published version left for a deprecation notice to attach to, so the claim
|
|
289
|
+
never held. No registry stub or deprecation was possible to publish; root `CLAUDE.md` carried
|
|
290
|
+
the same false claim and is corrected alongside this entry. Users of the old names should
|
|
291
|
+
install `@hunt-labs/bytekit-sdk` directly. No exported function, type, or runtime behavior
|
|
292
|
+
changes.
|
|
293
|
+
|
|
160
294
|
## [0.7.0] - 2026-08-11
|
|
161
295
|
|
|
162
296
|
### Added
|
|
@@ -538,8 +672,10 @@ against 0.1.0 — read each one before upgrading.
|
|
|
538
672
|
### Changed
|
|
539
673
|
|
|
540
674
|
- Package renamed to `@hunt-labs/bytekit-sdk` (the `@bytekit` npm scope was
|
|
541
|
-
unobtainable). The prior `rapidcrawl`/`@rapidcrawl/sdk` name remains installable at
|
|
542
|
-
its last version but is deprecated.
|
|
675
|
+
unobtainable). ~~The prior `rapidcrawl`/`@rapidcrawl/sdk` name remains installable at
|
|
676
|
+
its last version but is deprecated.~~ **Corrected in 0.7.1 (see above): that claim
|
|
677
|
+
does not hold — both names 404 on npm and there is no published version left to
|
|
678
|
+
deprecate.**
|
|
543
679
|
- Generated API types (`dist/generated/api.d.ts`) are now shipped in the tarball so
|
|
544
680
|
consumers get the typed request/response surface without regenerating from the spec.
|
|
545
681
|
|
package/dist/generated/api.d.ts
CHANGED
|
@@ -272,8 +272,9 @@ export interface paths {
|
|
|
272
272
|
};
|
|
273
273
|
/**
|
|
274
274
|
* Fetch a URL (GET)
|
|
275
|
-
* @description Direct HTTP fetch.
|
|
276
|
-
*
|
|
275
|
+
* @description Direct HTTP fetch. Successful responses return the upstream response bytes verbatim;
|
|
276
|
+
* metadata is returned in X-Fetch-* headers. Non-2xx upstream responses use the
|
|
277
|
+
* `upstream_error` JSON envelope with the upstream HTTP status.
|
|
277
278
|
*/
|
|
278
279
|
get: operations["getFetch"];
|
|
279
280
|
put?: never;
|
|
@@ -303,7 +304,7 @@ export interface paths {
|
|
|
303
304
|
/**
|
|
304
305
|
* Bulk fetch URLs
|
|
305
306
|
* @description Enqueues multiple URLs for HTTP fetching. Results delivered via webhook.
|
|
306
|
-
* Does not support options that require page rendering (
|
|
307
|
+
* Does not support options that require page rendering (wait_for_selector, etc.).
|
|
307
308
|
*
|
|
308
309
|
* **Limits:** request bodies are guarded in a two-tier shape. A streamed body
|
|
309
310
|
* with no `Content-Length` header is capped at 2 MB before your API key
|
|
@@ -588,8 +589,8 @@ export interface paths {
|
|
|
588
589
|
* Schema-driven structured extraction
|
|
589
590
|
* @description Fetches `url` and extracts structured JSON conforming to the supplied
|
|
590
591
|
* JSON Schema (draft 2020-12) using an LLM. Fast-path only — slow-path
|
|
591
|
-
* rendering options (`wait_until=networkidle`, `
|
|
592
|
-
*
|
|
592
|
+
* rendering options (`wait_until=networkidle`, `delay_ms`,
|
|
593
|
+
* `wait_for_selector`) cause a `400 unsupported_url`.
|
|
593
594
|
*
|
|
594
595
|
* Dual-axis billing: a fixed per-request credit charge **plus** the
|
|
595
596
|
* underlying page's bandwidth, billed exactly like `/v1/scrape`. The
|
|
@@ -1041,7 +1042,7 @@ export interface paths {
|
|
|
1041
1042
|
/**
|
|
1042
1043
|
* Query scrape by URL (deprecated)
|
|
1043
1044
|
* @deprecated
|
|
1044
|
-
* @description Use GET /v1/fetch?url
|
|
1045
|
+
* @description Use GET /v1/fetch?url=... instead.
|
|
1045
1046
|
*/
|
|
1046
1047
|
get: operations["queryScrapeLegacy"];
|
|
1047
1048
|
put?: never;
|
|
@@ -1404,14 +1405,6 @@ export interface components {
|
|
|
1404
1405
|
details?: unknown;
|
|
1405
1406
|
};
|
|
1406
1407
|
};
|
|
1407
|
-
Cookie: {
|
|
1408
|
-
name: string;
|
|
1409
|
-
value: string;
|
|
1410
|
-
domain?: string;
|
|
1411
|
-
path?: string;
|
|
1412
|
-
secure?: boolean;
|
|
1413
|
-
httpOnly?: boolean;
|
|
1414
|
-
};
|
|
1415
1408
|
Viewport: {
|
|
1416
1409
|
width: number;
|
|
1417
1410
|
height: number;
|
|
@@ -1436,6 +1429,93 @@ export interface components {
|
|
|
1436
1429
|
next_cursor: string | null;
|
|
1437
1430
|
has_more: boolean;
|
|
1438
1431
|
};
|
|
1432
|
+
/**
|
|
1433
|
+
* @description One named multiplicative term in a charge. A factor that did not apply is
|
|
1434
|
+
* ABSENT from the list rather than present with `value: 1` — an entry reading
|
|
1435
|
+
* `cache_hit ×1` would claim a cache-hit discount had been applied. The
|
|
1436
|
+
* `endpoint` factor is the exception: every billed request has an endpoint, so
|
|
1437
|
+
* it is always listed even when its value is 1.0.
|
|
1438
|
+
*/
|
|
1439
|
+
BillingFactor: {
|
|
1440
|
+
/**
|
|
1441
|
+
* @description Closed vocabulary. `endpoint` — the per-endpoint format factor.
|
|
1442
|
+
* `cache_hit` — the 0.5× replay discount. `clean_markdown` — the 3×
|
|
1443
|
+
* surcharge, present only when the LLM clean actually succeeded.
|
|
1444
|
+
* `phase1_render` — the screenshot/recording render-overhead factor,
|
|
1445
|
+
* which is never composed with `cache_hit` or `clean_markdown`.
|
|
1446
|
+
* Treat an unrecognised name as a term you do not model rather than as an
|
|
1447
|
+
* error: the product of all `value`s always equals `multiplier`.
|
|
1448
|
+
* @example endpoint
|
|
1449
|
+
* @enum {string}
|
|
1450
|
+
*/
|
|
1451
|
+
name: "endpoint" | "cache_hit" | "clean_markdown" | "phase1_render";
|
|
1452
|
+
/**
|
|
1453
|
+
* @description The multiplicative value of this term.
|
|
1454
|
+
* @example 1.5
|
|
1455
|
+
*/
|
|
1456
|
+
value: number;
|
|
1457
|
+
/**
|
|
1458
|
+
* @description Human-readable explanation of why this factor applied — for example the
|
|
1459
|
+
* endpoint bucket name, or `hit, age 412s`. NOT CONTRACTUAL: this string is
|
|
1460
|
+
* free text for display and debugging only. Its wording, its format, and
|
|
1461
|
+
* whether it carries any particular detail may change at any time without a
|
|
1462
|
+
* version bump. Never parse it, branch on it, or assert on it — read
|
|
1463
|
+
* `name` and `value` instead.
|
|
1464
|
+
* @example scrape_md
|
|
1465
|
+
*/
|
|
1466
|
+
reason: string;
|
|
1467
|
+
};
|
|
1468
|
+
/**
|
|
1469
|
+
* @description The authoritative, self-explaining account of what this call was billed and
|
|
1470
|
+
* why. `multiplier` is the product of every entry in `factors`, composed in
|
|
1471
|
+
* list order (`endpoint` → `cache_hit` → `clean_markdown`), and
|
|
1472
|
+
* `billed_bytes = round(raw_bytes × multiplier)` behind exactly one rounding.
|
|
1473
|
+
* The factor list is the DEFINITION of the charge; the `X-Raw-Bytes` /
|
|
1474
|
+
* `X-Billing-Multiplier` / `X-Billable-Bytes` response headers are a rendering
|
|
1475
|
+
* of this same node, so where a header and this node disagree, the header is
|
|
1476
|
+
* the bug.
|
|
1477
|
+
*
|
|
1478
|
+
* Two billable surfaces carry no node, by design rather than by omission:
|
|
1479
|
+
*
|
|
1480
|
+
* * `POST /v1/search` is billed per credit, not per byte. Its charge is
|
|
1481
|
+
* reported by `credits_used`; no byte multiplier applies to it and the
|
|
1482
|
+
* factor vocabulary has no `search` term.
|
|
1483
|
+
* * Single (non-bulk) `POST /v1/fetch` returns the fetched bytes as the
|
|
1484
|
+
* response body, so it has no JSON envelope in which to carry a node. Its
|
|
1485
|
+
* charge is reported by the `X-Raw-Bytes` / `X-Billing-Multiplier` /
|
|
1486
|
+
* `X-Billable-Bytes` header trio alone. Items under
|
|
1487
|
+
* `GET /v1/fetch/bulk/{id}` DO carry the node.
|
|
1488
|
+
*
|
|
1489
|
+
* Historical rows: for a resource that completed before its billing node was
|
|
1490
|
+
* persisted, this field is OMITTED rather than emitted as a zero node — a zero
|
|
1491
|
+
* node would assert that a real past charge was zero. Treat absence as "not
|
|
1492
|
+
* available for this row" and read the charge from `GET /v1/usage`.
|
|
1493
|
+
*/
|
|
1494
|
+
BillingNode: {
|
|
1495
|
+
/**
|
|
1496
|
+
* @description The unmultiplied basis of the charge. For byte-billed endpoints this is
|
|
1497
|
+
* the upstream wire bytes actually transferred. For screenshots and
|
|
1498
|
+
* recordings it is the Phase 1 HTML fetch's wire bytes — the only proxied
|
|
1499
|
+
* traffic in the happy path — never the size of the returned PNG/JPEG/MP4
|
|
1500
|
+
* artifact.
|
|
1501
|
+
* @example 1024
|
|
1502
|
+
*/
|
|
1503
|
+
raw_bytes: number;
|
|
1504
|
+
/** @description The named terms whose product is `multiplier`, in composition order. */
|
|
1505
|
+
factors: components["schemas"]["BillingFactor"][];
|
|
1506
|
+
/**
|
|
1507
|
+
* @description The product of every `factors[].value` — the REAL composed multiplier for
|
|
1508
|
+
* this call, including the per-endpoint format factor. A markdown cache hit
|
|
1509
|
+
* reads 0.75 here (1.5 × 0.5).
|
|
1510
|
+
* @example 0.75
|
|
1511
|
+
*/
|
|
1512
|
+
multiplier: number;
|
|
1513
|
+
/**
|
|
1514
|
+
* @description `round(raw_bytes × multiplier)` — the bandwidth actually settled for this call.
|
|
1515
|
+
* @example 768
|
|
1516
|
+
*/
|
|
1517
|
+
billed_bytes: number;
|
|
1518
|
+
};
|
|
1439
1519
|
ScrapeRequest: {
|
|
1440
1520
|
/** Format: uri */
|
|
1441
1521
|
url: string;
|
|
@@ -1447,12 +1527,6 @@ export interface components {
|
|
|
1447
1527
|
formats: ("raw_html" | "html" | "markdown" | "links" | "images")[];
|
|
1448
1528
|
/** @default US */
|
|
1449
1529
|
country: string;
|
|
1450
|
-
/** @default [] */
|
|
1451
|
-
cookies: components["schemas"]["Cookie"][];
|
|
1452
|
-
/** @default {} */
|
|
1453
|
-
headers: {
|
|
1454
|
-
[key: string]: string;
|
|
1455
|
-
};
|
|
1456
1530
|
/** @default 0 */
|
|
1457
1531
|
delay_ms: number;
|
|
1458
1532
|
/** @default 30000 */
|
|
@@ -1730,10 +1804,13 @@ export interface components {
|
|
|
1730
1804
|
stats?: components["schemas"]["ScrapeStats"];
|
|
1731
1805
|
/** @description Token-budget accounting. Present only when `token_budget` was supplied in the request. Contains `kept` (tokens in returned markdown), `total` (tokens in full pre-truncation markdown), and `truncated` (whether output was cut). Absent on uncapped requests to avoid noise. */
|
|
1732
1806
|
markdown_tokens?: components["schemas"]["ScrapeMarkdownTokens"];
|
|
1733
|
-
/**
|
|
1734
|
-
|
|
1735
|
-
|
|
1736
|
-
|
|
1807
|
+
/**
|
|
1808
|
+
* @description Itemised account of this call's charge — the basis, the named factors
|
|
1809
|
+
* that composed the multiplier, and the resulting billed bytes. This node
|
|
1810
|
+
* is the authoritative billing surface for the response body. Omitted for
|
|
1811
|
+
* rows persisted before the billing node existed.
|
|
1812
|
+
*/
|
|
1813
|
+
billing?: components["schemas"]["BillingNode"];
|
|
1737
1814
|
/** @description Custom metadata provided by the caller in the request `custom` field. Stored verbatim and round-tripped on the synchronous response and webhook delivery. Not included in `GET /v1/scrape/:id`, bulk results, or cached hits. Omitted when no custom metadata was provided. */
|
|
1738
1815
|
custom?: {
|
|
1739
1816
|
[key: string]: unknown;
|
|
@@ -1753,6 +1830,7 @@ export interface components {
|
|
|
1753
1830
|
cache_age_s?: number;
|
|
1754
1831
|
};
|
|
1755
1832
|
/**
|
|
1833
|
+
* @description Terminal asynchronous scrape failures are delivered to subscribers of `failed` as this payload with `X-ByteKit-Event: scrape.failed`.
|
|
1756
1834
|
* @example {
|
|
1757
1835
|
* "schema_version": 1,
|
|
1758
1836
|
* "status": "failed",
|
|
@@ -1823,8 +1901,6 @@ export interface components {
|
|
|
1823
1901
|
FetchRequest: {
|
|
1824
1902
|
/** Format: uri */
|
|
1825
1903
|
url: string;
|
|
1826
|
-
/** @enum {string} */
|
|
1827
|
-
format?: "markdown" | "html";
|
|
1828
1904
|
/** @default US */
|
|
1829
1905
|
country: string;
|
|
1830
1906
|
/** @default 60000 */
|
|
@@ -1841,37 +1917,6 @@ export interface components {
|
|
|
1841
1917
|
custom: {
|
|
1842
1918
|
[key: string]: unknown;
|
|
1843
1919
|
};
|
|
1844
|
-
/**
|
|
1845
|
-
* @description Markdown processing mode. Only used when format=markdown.
|
|
1846
|
-
* @enum {string}
|
|
1847
|
-
*/
|
|
1848
|
-
markdown_mode?: "article" | "raw" | "llm";
|
|
1849
|
-
/** @description BM25 query string for relevance-ranked filtering. Only used when format=markdown. */
|
|
1850
|
-
markdown_query?: string;
|
|
1851
|
-
/**
|
|
1852
|
-
* @description Link rendering style. Only used when format=markdown.
|
|
1853
|
-
* @enum {string}
|
|
1854
|
-
*/
|
|
1855
|
-
markdown_links?: "inline" | "references" | "none" | "text";
|
|
1856
|
-
/**
|
|
1857
|
-
* @description Image retention mode. Only used when format=markdown.
|
|
1858
|
-
* @enum {string}
|
|
1859
|
-
*/
|
|
1860
|
-
markdown_images?: "inline" | "references" | "none" | "text";
|
|
1861
|
-
/** @description Append a Links footer to the markdown output. Only used when format=markdown. */
|
|
1862
|
-
with_links_summary?: boolean;
|
|
1863
|
-
/** @description Append an Images footer to the markdown output. Only used when format=markdown. */
|
|
1864
|
-
with_images_summary?: boolean;
|
|
1865
|
-
/** @description Compact whitespace output. Only used when format=markdown. */
|
|
1866
|
-
markdown_compact?: boolean;
|
|
1867
|
-
/** @description Filter low-signal images. Only used when format=markdown. */
|
|
1868
|
-
markdown_filter_images?: boolean;
|
|
1869
|
-
/** @description When true, `formats.links` and `formats.images` return `ScrapeScoredLink[]` / `ScrapeScoredImage[]` (rich objects) instead of `string[]`, and a top-level `tables` array is included. Only used when format=markdown. */
|
|
1870
|
-
markdown_include_media?: boolean;
|
|
1871
|
-
/** @description When true, enables markdown-pipeline and tag-filter warnings for affected html and markdown requests. Artifact availability warnings such as `artifact_unavailable` may appear independently of this flag. */
|
|
1872
|
-
markdown_include_warnings?: boolean;
|
|
1873
|
-
/** @description When true, includes a top-level `stats` object with `ScrapeStats` (chars, tokens, blocks). Only used when format=markdown. */
|
|
1874
|
-
markdown_include_stats?: boolean;
|
|
1875
1920
|
};
|
|
1876
1921
|
ScreenshotRequest: {
|
|
1877
1922
|
/** Format: uri */
|
|
@@ -1909,12 +1954,6 @@ export interface components {
|
|
|
1909
1954
|
scroll: boolean;
|
|
1910
1955
|
/** @default null */
|
|
1911
1956
|
clip: components["schemas"]["Clip"] | null;
|
|
1912
|
-
/** @default {} */
|
|
1913
|
-
headers: {
|
|
1914
|
-
[key: string]: string;
|
|
1915
|
-
};
|
|
1916
|
-
/** @default [] */
|
|
1917
|
-
cookies: components["schemas"]["Cookie"][];
|
|
1918
1957
|
/** @default false */
|
|
1919
1958
|
dark_mode: boolean;
|
|
1920
1959
|
/**
|
|
@@ -1947,6 +1986,13 @@ export interface components {
|
|
|
1947
1986
|
*/
|
|
1948
1987
|
file_size_bytes: number | null;
|
|
1949
1988
|
credits_charged: number | null;
|
|
1989
|
+
/**
|
|
1990
|
+
* @description Itemised account of this capture's charge — the Phase 1 wire bytes it
|
|
1991
|
+
* was billed on, the `phase1_render` factor, and the resulting billed
|
|
1992
|
+
* bytes. Omitted for rows that completed before the billing node was
|
|
1993
|
+
* persisted, and for rows that have not completed.
|
|
1994
|
+
*/
|
|
1995
|
+
billing?: components["schemas"]["BillingNode"];
|
|
1950
1996
|
error_code: string | null;
|
|
1951
1997
|
error_message: string | null;
|
|
1952
1998
|
/** Format: date-time */
|
|
@@ -2027,12 +2073,6 @@ export interface components {
|
|
|
2027
2073
|
/** @default 0 */
|
|
2028
2074
|
delay_ms: number;
|
|
2029
2075
|
scroll?: components["schemas"]["ScrollOptions"];
|
|
2030
|
-
/** @default {} */
|
|
2031
|
-
headers: {
|
|
2032
|
-
[key: string]: string;
|
|
2033
|
-
};
|
|
2034
|
-
/** @default [] */
|
|
2035
|
-
cookies: components["schemas"]["Cookie"][];
|
|
2036
2076
|
/** @default false */
|
|
2037
2077
|
dark_mode: boolean;
|
|
2038
2078
|
/** @default US */
|
|
@@ -2066,6 +2106,13 @@ export interface components {
|
|
|
2066
2106
|
*/
|
|
2067
2107
|
file_size_bytes: number | null;
|
|
2068
2108
|
credits_charged: number | null;
|
|
2109
|
+
/**
|
|
2110
|
+
* @description Itemised account of this capture's charge — the Phase 1 wire bytes it
|
|
2111
|
+
* was billed on, the `phase1_render` factor, and the resulting billed
|
|
2112
|
+
* bytes. Omitted for rows that completed before the billing node was
|
|
2113
|
+
* persisted, and for rows that have not completed.
|
|
2114
|
+
*/
|
|
2115
|
+
billing?: components["schemas"]["BillingNode"];
|
|
2069
2116
|
error_code: string | null;
|
|
2070
2117
|
error_message: string | null;
|
|
2071
2118
|
/** Format: date-time */
|
|
@@ -2161,7 +2208,7 @@ export interface components {
|
|
|
2161
2208
|
webhook_url: string;
|
|
2162
2209
|
webhook_headers: Record<string, never> | null;
|
|
2163
2210
|
/** @description Present on screenshot monitors only. */
|
|
2164
|
-
change_threshold?:
|
|
2211
|
+
change_threshold?: number;
|
|
2165
2212
|
/** @description Present on scrape monitors only. */
|
|
2166
2213
|
scrape_options?: unknown;
|
|
2167
2214
|
/** @description Present on scrape monitors only. */
|
|
@@ -2203,7 +2250,7 @@ export interface components {
|
|
|
2203
2250
|
content_hash: string | null;
|
|
2204
2251
|
previous_hash: string | null;
|
|
2205
2252
|
hash_distance: number | null;
|
|
2206
|
-
change_pct:
|
|
2253
|
+
change_pct: number | null;
|
|
2207
2254
|
has_change: boolean;
|
|
2208
2255
|
title_changed: boolean | null;
|
|
2209
2256
|
/**
|
|
@@ -2284,6 +2331,12 @@ export interface components {
|
|
|
2284
2331
|
cache?: "miss" | "hit" | "bypass";
|
|
2285
2332
|
/** @description Age in whole seconds of the cached source job (its `completed_at`) replayed on a `cache: hit`. Clamped to ≥ 0 to absorb clock skew. Present only on `cache: hit`; null on `miss`/`bypass`. */
|
|
2286
2333
|
cache_age_s?: number | null;
|
|
2334
|
+
/**
|
|
2335
|
+
* @description Itemised account of this job's charge — the basis, the named factors
|
|
2336
|
+
* that composed the multiplier, and the resulting billed bytes. Omitted
|
|
2337
|
+
* for jobs that completed before the billing node was persisted.
|
|
2338
|
+
*/
|
|
2339
|
+
billing?: components["schemas"]["BillingNode"];
|
|
2287
2340
|
warnings: string[] | null;
|
|
2288
2341
|
error_code: string | null;
|
|
2289
2342
|
error_message: string | null;
|
|
@@ -2588,7 +2641,6 @@ export interface components {
|
|
|
2588
2641
|
*/
|
|
2589
2642
|
ClientSurfaceMarker: string;
|
|
2590
2643
|
FetchUrl: string;
|
|
2591
|
-
FetchFormat: "markdown" | "html";
|
|
2592
2644
|
FetchCountry: string;
|
|
2593
2645
|
FetchTimeoutMs: number;
|
|
2594
2646
|
/** @description How long a freshly fetched URL may be served from cache. `0` skips the cache read; cache-eligible fresh results are still written with the default 7-day TTL so later non-zero-TTL callers can HIT. Safety-classified WAF husks remain live-only and are not written. Accepts `Nh` (hours, up to 168h), `Nd` (days, up to 7d), or the integer `0`. Examples: `0`, `48h`, `2d`. Invalid values receive a 422 validation_error with an actionable message. */
|
|
@@ -2628,11 +2680,9 @@ export interface components {
|
|
|
2628
2680
|
XFetchURL: string;
|
|
2629
2681
|
/**
|
|
2630
2682
|
* @description Byte length of the response body served on this request (integer as string) — NOT the
|
|
2631
|
-
* billing basis.
|
|
2632
|
-
*
|
|
2633
|
-
* `
|
|
2634
|
-
* count and is typically far smaller than what is billed. Billing always reads the upstream
|
|
2635
|
-
* compressed wire bytes from `X-Raw-Bytes` / `X-Billable-Bytes`, on every path.
|
|
2683
|
+
* billing basis. The body is always the upstream response bytes, so this equals the upstream wire byte
|
|
2684
|
+
* count. Billing always reads the upstream compressed wire bytes from `X-Raw-Bytes` /
|
|
2685
|
+
* `X-Billable-Bytes`, on every path.
|
|
2636
2686
|
*/
|
|
2637
2687
|
XFetchContentLength: string;
|
|
2638
2688
|
XFetchCreditsCharged: string;
|
|
@@ -3110,7 +3160,7 @@ export interface operations {
|
|
|
3110
3160
|
"X-Raw-Bytes"?: number;
|
|
3111
3161
|
/** @description Chargeable bytes — round(X-Raw-Bytes × X-Billing-Multiplier). */
|
|
3112
3162
|
"X-Billable-Bytes"?: number;
|
|
3113
|
-
/** @description Billing multiplier applied to raw bytes to compute billable bytes. */
|
|
3163
|
+
/** @description Billing multiplier applied to raw bytes to compute billable bytes. A rendering of the response body's `billing` node, which is the authoritative surface: it names each factor that composed this value. Where the two disagree, the node is correct. */
|
|
3114
3164
|
"X-Billing-Multiplier"?: number;
|
|
3115
3165
|
/** @description HTTP status code returned by the target site during the screenshot capture. */
|
|
3116
3166
|
"X-Screenshot-Status-Code"?: number;
|
|
@@ -3145,7 +3195,7 @@ export interface operations {
|
|
|
3145
3195
|
"X-Raw-Bytes"?: number;
|
|
3146
3196
|
/** @description Chargeable bytes — round(X-Raw-Bytes × X-Billing-Multiplier). */
|
|
3147
3197
|
"X-Billable-Bytes"?: number;
|
|
3148
|
-
/** @description Billing multiplier applied to raw bytes to compute billable bytes. */
|
|
3198
|
+
/** @description Billing multiplier applied to raw bytes to compute billable bytes. A rendering of the response body's `billing` node, which is the authoritative surface: it names each factor that composed this value. Where the two disagree, the node is correct. */
|
|
3149
3199
|
"X-Billing-Multiplier"?: number;
|
|
3150
3200
|
[name: string]: unknown;
|
|
3151
3201
|
};
|
|
@@ -3336,7 +3386,7 @@ export interface operations {
|
|
|
3336
3386
|
"X-Raw-Bytes"?: number;
|
|
3337
3387
|
/** @description Chargeable bytes after applying the billing multiplier. Equals round(X-Raw-Bytes × X-Billing-Multiplier). */
|
|
3338
3388
|
"X-Billable-Bytes"?: number;
|
|
3339
|
-
/** @description Bandwidth cost multiplier (e.g. 1.0 for raw_html-only, 1.5 for +markdown/html). Cache hits compose the endpoint format factor: raw 0.5, markdown 0.75. Excludes the `clean_markdown` surcharge
|
|
3389
|
+
/** @description Bandwidth cost multiplier (e.g. 1.0 for raw_html-only, 1.5 for +markdown/html). Cache hits compose the endpoint format factor: raw 0.5, markdown 0.75. Excludes the `clean_markdown` surcharge: when `clean_markdown: true` and the clean succeeded, the settled usage is 3× this header's value, so this header does not predict the amount you are billed. The response body's `billing` node is the authoritative surface — it DOES carry the `clean_markdown` factor, so its `multiplier` and `billed_bytes` are what you are charged. Where the two disagree, the node is correct. */
|
|
3340
3390
|
"X-Billing-Multiplier"?: number;
|
|
3341
3391
|
/** @description Credits charged for this scrape (multiplier applied to base 1.0 per format). */
|
|
3342
3392
|
"X-Scrape-Credits-Charged"?: number;
|
|
@@ -3514,10 +3564,6 @@ export interface operations {
|
|
|
3514
3564
|
url: string;
|
|
3515
3565
|
formats?: ("raw_html" | "html" | "markdown" | "links" | "images")[];
|
|
3516
3566
|
country?: string;
|
|
3517
|
-
cookies?: components["schemas"]["Cookie"][];
|
|
3518
|
-
headers?: {
|
|
3519
|
-
[key: string]: string;
|
|
3520
|
-
};
|
|
3521
3567
|
delay_ms?: number;
|
|
3522
3568
|
timeout_ms?: number;
|
|
3523
3569
|
metadata?: components["schemas"]["Metadata"];
|
|
@@ -3552,10 +3598,6 @@ export interface operations {
|
|
|
3552
3598
|
formats?: ("raw_html" | "html" | "markdown" | "links" | "images")[];
|
|
3553
3599
|
/** @default US */
|
|
3554
3600
|
country?: string;
|
|
3555
|
-
cookies?: components["schemas"]["Cookie"][];
|
|
3556
|
-
headers?: {
|
|
3557
|
-
[key: string]: string;
|
|
3558
|
-
};
|
|
3559
3601
|
/** @default 0 */
|
|
3560
3602
|
delay_ms?: number;
|
|
3561
3603
|
/** @default 30000 */
|
|
@@ -3615,7 +3657,7 @@ export interface operations {
|
|
|
3615
3657
|
completed_at: string | null;
|
|
3616
3658
|
/** @description Always null on the create response — the job has not been billed yet (routes/scrape-bulk.ts builds the 202 body from a partial job row). Fractional when populated (credits are NUMERIC(10,2)); the numeric branch is unreachable on this path and exists only so the 202 and 200 bodies share one shape. */
|
|
3617
3659
|
total_credits_charged: number | null;
|
|
3618
|
-
/** @description Estimated credits at submission time. Null in GET responses.
|
|
3660
|
+
/** @description Estimated credits at submission time. Null in GET responses. Fetch-bulk items use the raw fetch bandwidth billing factor. */
|
|
3619
3661
|
estimated_credits: number | null;
|
|
3620
3662
|
/** @description Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported) */
|
|
3621
3663
|
webhook_url: string;
|
|
@@ -3715,7 +3757,6 @@ export interface operations {
|
|
|
3715
3757
|
parameters: {
|
|
3716
3758
|
query: {
|
|
3717
3759
|
url: components["parameters"]["FetchUrl"];
|
|
3718
|
-
format?: components["parameters"]["FetchFormat"];
|
|
3719
3760
|
country?: components["parameters"]["FetchCountry"];
|
|
3720
3761
|
timeout_ms?: components["parameters"]["FetchTimeoutMs"];
|
|
3721
3762
|
/** @description How long a freshly fetched URL may be served from cache. `0` skips the cache read; cache-eligible fresh results are still written with the default 7-day TTL so later non-zero-TTL callers can HIT. Safety-classified WAF husks remain live-only and are not written. Accepts `Nh` (hours, up to 168h), `Nd` (days, up to 7d), or the integer `0`. Examples: `0`, `48h`, `2d`. Invalid values receive a 422 validation_error with an actionable message. */
|
|
@@ -3749,16 +3790,14 @@ export interface operations {
|
|
|
3749
3790
|
"X-Raw-Bytes"?: number;
|
|
3750
3791
|
/** @description Chargeable bytes after applying the billing multiplier. Equals round(X-Raw-Bytes × X-Billing-Multiplier). */
|
|
3751
3792
|
"X-Billable-Bytes"?: number;
|
|
3752
|
-
/** @description Bandwidth cost multiplier applied to X-Raw-Bytes.
|
|
3793
|
+
/** @description Bandwidth cost multiplier applied to X-Raw-Bytes. Single `/v1/fetch` returns the fetched bytes as the body, so it carries no `billing` node — this header trio is the ONLY billing surface here. Items under `GET /v1/fetch/bulk/{id}` do carry the node. */
|
|
3753
3794
|
"X-Billing-Multiplier"?: number;
|
|
3754
3795
|
/** @description Back-compat duplicate of `X-Billing-Multiplier` — identical value on every path. Prefer `X-Billing-Multiplier`; this alias is retained for clients that read the `X-Fetch-*` header family. */
|
|
3755
3796
|
"X-Fetch-Billing-Multiplier"?: number;
|
|
3756
3797
|
[name: string]: unknown;
|
|
3757
3798
|
};
|
|
3758
3799
|
content: {
|
|
3759
|
-
"
|
|
3760
|
-
"text/markdown": string;
|
|
3761
|
-
"text/html": string;
|
|
3800
|
+
"text/plain": string;
|
|
3762
3801
|
};
|
|
3763
3802
|
};
|
|
3764
3803
|
/** @description Invalid URL (`invalid_url`) or oversized request. */
|
|
@@ -3770,7 +3809,7 @@ export interface operations {
|
|
|
3770
3809
|
"application/json": components["schemas"]["Error"];
|
|
3771
3810
|
};
|
|
3772
3811
|
};
|
|
3773
|
-
/** @description Validation error (e.g.
|
|
3812
|
+
/** @description Validation error (e.g. slow-path-only field). */
|
|
3774
3813
|
422: {
|
|
3775
3814
|
headers: {
|
|
3776
3815
|
[name: string]: unknown;
|
|
@@ -3789,10 +3828,14 @@ export interface operations {
|
|
|
3789
3828
|
};
|
|
3790
3829
|
};
|
|
3791
3830
|
/**
|
|
3792
|
-
* @description
|
|
3793
|
-
*
|
|
3794
|
-
*
|
|
3795
|
-
*
|
|
3831
|
+
* @description Content transform failed, or a non-2xx upstream response relayed as the unified
|
|
3832
|
+
* Error envelope. On an upstream relay the HTTP status equals the upstream status
|
|
3833
|
+
* (code `upstream_error`), so any 4xx/5xx the target returns surfaces here as an
|
|
3834
|
+
* Error envelope rather than a raw body (#1462).
|
|
3835
|
+
*
|
|
3836
|
+
* **One divergence.** An upstream `502` is relayed at HTTP `503`, not `502` — see
|
|
3837
|
+
* the `503` response below. A fetch service that cannot be reached also answers
|
|
3838
|
+
* `503`, not `502` (#3373); this status no longer covers that condition.
|
|
3796
3839
|
*/
|
|
3797
3840
|
502: {
|
|
3798
3841
|
headers: {
|
|
@@ -3802,6 +3845,30 @@ export interface operations {
|
|
|
3802
3845
|
"application/json": components["schemas"]["Error"];
|
|
3803
3846
|
};
|
|
3804
3847
|
};
|
|
3848
|
+
/**
|
|
3849
|
+
* @description Two distinct conditions answer `503`, and they disagree about what the body's
|
|
3850
|
+
* `error.http_status` reports. Read `error.code` to tell them apart.
|
|
3851
|
+
*
|
|
3852
|
+
* - **`upstream_error`** — the target returned HTTP `502`, relayed here as `503`.
|
|
3853
|
+
* Cloudflare fronts this API and replaces a clean origin 502 with its own generic
|
|
3854
|
+
* `text/plain` error page, discarding the envelope before it reaches you, while a
|
|
3855
|
+
* 503 is passed through unchanged. The body's `error.http_status` still reports
|
|
3856
|
+
* the **true upstream status**, `502` — not `503` — and is the only surviving
|
|
3857
|
+
* record of what the target answered, so the body rather than the HTTP status is
|
|
3858
|
+
* authoritative here (#3355).
|
|
3859
|
+
* - **`internal_error`** (message `Fetch service unavailable.`) — the fetch service
|
|
3860
|
+
* could not be reached at all. There is no upstream status to preserve: the `503`
|
|
3861
|
+
* is this API's own choice of wire status, so the body's `error.http_status`
|
|
3862
|
+
* reports `503`, mirroring what was served (#3373).
|
|
3863
|
+
*/
|
|
3864
|
+
503: {
|
|
3865
|
+
headers: {
|
|
3866
|
+
[name: string]: unknown;
|
|
3867
|
+
};
|
|
3868
|
+
content: {
|
|
3869
|
+
"application/json": components["schemas"]["Error"];
|
|
3870
|
+
};
|
|
3871
|
+
};
|
|
3805
3872
|
};
|
|
3806
3873
|
};
|
|
3807
3874
|
postFetch: {
|
|
@@ -3815,10 +3882,7 @@ export interface operations {
|
|
|
3815
3882
|
content: {
|
|
3816
3883
|
/**
|
|
3817
3884
|
* @example {
|
|
3818
|
-
* "url": "https://example.com"
|
|
3819
|
-
* "format": "markdown",
|
|
3820
|
-
* "markdown_mode": "article",
|
|
3821
|
-
* "markdown_links": "inline"
|
|
3885
|
+
* "url": "https://example.com"
|
|
3822
3886
|
* }
|
|
3823
3887
|
*/
|
|
3824
3888
|
"application/json": components["schemas"]["FetchRequest"];
|
|
@@ -3847,16 +3911,14 @@ export interface operations {
|
|
|
3847
3911
|
"X-Raw-Bytes"?: number;
|
|
3848
3912
|
/** @description Chargeable bytes after applying the billing multiplier. Equals round(X-Raw-Bytes × X-Billing-Multiplier). */
|
|
3849
3913
|
"X-Billable-Bytes"?: number;
|
|
3850
|
-
/** @description Bandwidth cost multiplier applied to X-Raw-Bytes.
|
|
3914
|
+
/** @description Bandwidth cost multiplier applied to X-Raw-Bytes. Single `/v1/fetch` returns the fetched bytes as the body, so it carries no `billing` node — this header trio is the ONLY billing surface here. Items under `GET /v1/fetch/bulk/{id}` do carry the node. */
|
|
3851
3915
|
"X-Billing-Multiplier"?: number;
|
|
3852
3916
|
/** @description Back-compat duplicate of `X-Billing-Multiplier` — identical value on every path. Prefer `X-Billing-Multiplier`; this alias is retained for clients that read the `X-Fetch-*` header family. */
|
|
3853
3917
|
"X-Fetch-Billing-Multiplier"?: number;
|
|
3854
3918
|
[name: string]: unknown;
|
|
3855
3919
|
};
|
|
3856
3920
|
content: {
|
|
3857
|
-
"
|
|
3858
|
-
"text/markdown": string;
|
|
3859
|
-
"text/html": string;
|
|
3921
|
+
"text/plain": string;
|
|
3860
3922
|
};
|
|
3861
3923
|
};
|
|
3862
3924
|
/** @description Invalid URL (`invalid_url`) or oversized request body. */
|
|
@@ -3868,7 +3930,7 @@ export interface operations {
|
|
|
3868
3930
|
"application/json": components["schemas"]["Error"];
|
|
3869
3931
|
};
|
|
3870
3932
|
};
|
|
3871
|
-
/** @description Validation error (e.g. malformed JSON, unsupported
|
|
3933
|
+
/** @description Validation error (e.g. malformed JSON, unsupported fetch format, or slow-path-only field). */
|
|
3872
3934
|
422: {
|
|
3873
3935
|
headers: {
|
|
3874
3936
|
[name: string]: unknown;
|
|
@@ -3887,10 +3949,14 @@ export interface operations {
|
|
|
3887
3949
|
};
|
|
3888
3950
|
};
|
|
3889
3951
|
/**
|
|
3890
|
-
* @description
|
|
3891
|
-
*
|
|
3892
|
-
*
|
|
3893
|
-
*
|
|
3952
|
+
* @description Content transform failed, or a non-2xx upstream response relayed as the unified
|
|
3953
|
+
* Error envelope. On an upstream relay the HTTP status equals the upstream status
|
|
3954
|
+
* (code `upstream_error`), so any 4xx/5xx the target returns surfaces here as an
|
|
3955
|
+
* Error envelope rather than a raw body (#1462).
|
|
3956
|
+
*
|
|
3957
|
+
* **One divergence.** An upstream `502` is relayed at HTTP `503`, not `502` — see
|
|
3958
|
+
* the `503` response below. A fetch service that cannot be reached also answers
|
|
3959
|
+
* `503`, not `502` (#3373); this status no longer covers that condition.
|
|
3894
3960
|
*/
|
|
3895
3961
|
502: {
|
|
3896
3962
|
headers: {
|
|
@@ -3900,6 +3966,30 @@ export interface operations {
|
|
|
3900
3966
|
"application/json": components["schemas"]["Error"];
|
|
3901
3967
|
};
|
|
3902
3968
|
};
|
|
3969
|
+
/**
|
|
3970
|
+
* @description Two distinct conditions answer `503`, and they disagree about what the body's
|
|
3971
|
+
* `error.http_status` reports. Read `error.code` to tell them apart.
|
|
3972
|
+
*
|
|
3973
|
+
* - **`upstream_error`** — the target returned HTTP `502`, relayed here as `503`.
|
|
3974
|
+
* Cloudflare fronts this API and replaces a clean origin 502 with its own generic
|
|
3975
|
+
* `text/plain` error page, discarding the envelope before it reaches you, while a
|
|
3976
|
+
* 503 is passed through unchanged. The body's `error.http_status` still reports
|
|
3977
|
+
* the **true upstream status**, `502` — not `503` — and is the only surviving
|
|
3978
|
+
* record of what the target answered, so the body rather than the HTTP status is
|
|
3979
|
+
* authoritative here (#3355).
|
|
3980
|
+
* - **`internal_error`** (message `Fetch service unavailable.`) — the fetch service
|
|
3981
|
+
* could not be reached at all. There is no upstream status to preserve: the `503`
|
|
3982
|
+
* is this API's own choice of wire status, so the body's `error.http_status`
|
|
3983
|
+
* reports `503`, mirroring what was served (#3373).
|
|
3984
|
+
*/
|
|
3985
|
+
503: {
|
|
3986
|
+
headers: {
|
|
3987
|
+
[name: string]: unknown;
|
|
3988
|
+
};
|
|
3989
|
+
content: {
|
|
3990
|
+
"application/json": components["schemas"]["Error"];
|
|
3991
|
+
};
|
|
3992
|
+
};
|
|
3903
3993
|
};
|
|
3904
3994
|
};
|
|
3905
3995
|
createFetchBulk: {
|
|
@@ -3915,14 +4005,10 @@ export interface operations {
|
|
|
3915
4005
|
urls: (string | {
|
|
3916
4006
|
/** Format: uri */
|
|
3917
4007
|
url: string;
|
|
3918
|
-
/** @enum {string} */
|
|
3919
|
-
format?: "markdown" | "html";
|
|
3920
4008
|
country?: string;
|
|
3921
4009
|
cache_ttl?: string;
|
|
3922
4010
|
})[];
|
|
3923
4011
|
defaults?: {
|
|
3924
|
-
/** @enum {string} */
|
|
3925
|
-
format?: "markdown" | "html";
|
|
3926
4012
|
/** @default US */
|
|
3927
4013
|
country?: string;
|
|
3928
4014
|
/** @default 48h */
|
|
@@ -3959,7 +4045,7 @@ export interface operations {
|
|
|
3959
4045
|
completed_at: string | null;
|
|
3960
4046
|
/** @description Always null on the create response — the job has not been billed yet (routes/fetch-bulk.ts builds the 202 body from a partial job row). Fractional when populated (credits are NUMERIC(10,2)); the numeric branch is unreachable on this path and exists only so the 202 and 200 bodies share one shape. */
|
|
3961
4047
|
total_credits_charged: number | null;
|
|
3962
|
-
/** @description Estimated credits at submission time. Null in GET responses.
|
|
4048
|
+
/** @description Estimated credits at submission time. Null in GET responses. Fetch items are raw passthrough requests and use the raw fetch bandwidth billing factor. */
|
|
3963
4049
|
estimated_credits: number | null;
|
|
3964
4050
|
/** @description Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported) */
|
|
3965
4051
|
webhook_url: string;
|
|
@@ -4052,6 +4138,12 @@ export interface operations {
|
|
|
4052
4138
|
content_url: string | null;
|
|
4053
4139
|
waf_vendor: string | null;
|
|
4054
4140
|
cache_hit: boolean;
|
|
4141
|
+
/**
|
|
4142
|
+
* @description Itemised account of this item's charge — the basis, the named factors
|
|
4143
|
+
* that composed the multiplier, and the resulting billed bytes. Omitted
|
|
4144
|
+
* for rows that completed before the billing node was persisted.
|
|
4145
|
+
*/
|
|
4146
|
+
billing?: components["schemas"]["BillingNode"];
|
|
4055
4147
|
error_code: string | null;
|
|
4056
4148
|
error_message: string | null;
|
|
4057
4149
|
/** Format: date-time */
|
|
@@ -4310,7 +4402,7 @@ export interface operations {
|
|
|
4310
4402
|
completed_at: string | null;
|
|
4311
4403
|
/** @description Always null on the create response — the job has not been billed yet (routes/bulk.ts builds the 202 body from a partial job row). Fractional when populated (credits are NUMERIC(10,2)); the numeric branch is unreachable on this path and exists only so the 202 and 200 bodies share one shape. */
|
|
4312
4404
|
total_credits_charged: number | null;
|
|
4313
|
-
/** @description Estimated credits at submission time. Null in GET responses.
|
|
4405
|
+
/** @description Estimated credits at submission time. Null in GET responses. Fetch-bulk items use the raw fetch bandwidth billing factor. */
|
|
4314
4406
|
estimated_credits: number | null;
|
|
4315
4407
|
/** @description Webhook endpoint URL (HTTPS only; http:// rejected with 400 invalid_url, reason scheme_unsupported) */
|
|
4316
4408
|
webhook_url: string;
|
|
@@ -4492,6 +4584,12 @@ export interface operations {
|
|
|
4492
4584
|
/** Format: uri */
|
|
4493
4585
|
url: string;
|
|
4494
4586
|
credits_charged: number | null;
|
|
4587
|
+
/**
|
|
4588
|
+
* @description Itemised account of this item's charge — the basis, the named factors
|
|
4589
|
+
* that composed the multiplier, and the resulting billed bytes. Omitted
|
|
4590
|
+
* for rows that completed before the billing node was persisted.
|
|
4591
|
+
*/
|
|
4592
|
+
billing?: components["schemas"]["BillingNode"];
|
|
4495
4593
|
error_code: string | null;
|
|
4496
4594
|
error_message: string | null;
|
|
4497
4595
|
/** Format: date-time */
|
|
@@ -4855,7 +4953,7 @@ export interface operations {
|
|
|
4855
4953
|
"X-Sitemap-ID"?: string;
|
|
4856
4954
|
/** @description Gzipped results CSV wire bytes — the bandwidth-billing basis. Emitted only when `status: completed`. */
|
|
4857
4955
|
"X-Raw-Bytes"?: number;
|
|
4858
|
-
/** @description Billing multiplier applied to raw bytes (1 for sitemap). Emitted only when `status: completed`. */
|
|
4956
|
+
/** @description Billing multiplier applied to raw bytes (1 for sitemap). Emitted only when `status: completed`. A rendering of the response body's `billing` node, which is the authoritative surface; where the two disagree, the node is correct. */
|
|
4859
4957
|
"X-Billing-Multiplier"?: number;
|
|
4860
4958
|
/** @description Billable bytes = round(X-Raw-Bytes × X-Billing-Multiplier). Emitted only when `status: completed`. */
|
|
4861
4959
|
"X-Billable-Bytes"?: number;
|
|
@@ -5125,18 +5223,8 @@ export interface operations {
|
|
|
5125
5223
|
* @enum {string}
|
|
5126
5224
|
*/
|
|
5127
5225
|
wait_until?: "load" | "domcontentloaded" | "networkidle";
|
|
5128
|
-
/** @default [] */
|
|
5129
|
-
cookies?: {
|
|
5130
|
-
name: string;
|
|
5131
|
-
value: string;
|
|
5132
|
-
domain: string;
|
|
5133
|
-
}[];
|
|
5134
5226
|
/** @default 0 */
|
|
5135
5227
|
delay_ms?: number;
|
|
5136
|
-
/** @default {} */
|
|
5137
|
-
headers?: {
|
|
5138
|
-
[key: string]: string;
|
|
5139
|
-
};
|
|
5140
5228
|
/**
|
|
5141
5229
|
* @description Two-letter country code, upper-cased. `null` is coerced to `US`.
|
|
5142
5230
|
* @default US
|
|
@@ -5501,9 +5589,9 @@ export interface operations {
|
|
|
5501
5589
|
status_code?: number | null;
|
|
5502
5590
|
/** @description Raw upstream bytes the request carried (the billing base before the multiplier). null while pending/failed. */
|
|
5503
5591
|
raw_bytes?: number | null;
|
|
5504
|
-
/** @description
|
|
5592
|
+
/** @description The bytes this request was BILLED — the same amount its own response reported in `X-Billable-Bytes`. For a scrape or fetch request this is the settled decomposition replayed as it was recorded at completion, so it already carries every factor that applied: the endpoint factor (`scrape` 1.0, `scrape_md` 1.5, `fetch` 1.0, `fetch_md` 1.5, a monitor run 1.0), the cache-hit ×0.5, and the clean-markdown ×3. Screenshots and recordings are `raw_bytes` × the fixed 1.5 Phase-1 render factor whatever capture options they carried — capture billing composes no discount factor, so that product is exact. It is NOT `raw_bytes` × `billing_multiplier`: that field reports a CREDIT count, a different quantity from the byte factors above, and on a recordings row their product is not bytes at all. One residual: a scrape or fetch row completed before the decomposition was recorded is reconstructed from its endpoint and cache-hit factors, which leaves a markdown `/v1/fetch` request of that vintage reported at 1.0 rather than 1.5 — nothing on such a row distinguishes it. That population ages out of this feed's retention window and is not backfilled. null while pending/failed (the row has no byte basis yet). */
|
|
5505
5593
|
billable_bytes?: number | null;
|
|
5506
|
-
/** @description
|
|
5594
|
+
/** @description BASE billing multiplier persisted for this request — the settled `credits_charged` factor. It is NOT what `billable_bytes` above is derived from: that field reports settled BYTES, and multiplying `raw_bytes` by this credit count does not give the amount billed. Its scale is not uniform and is not a rate you can carry from one row to another: each endpoint persists its own credit unit, and the single and bulk paths of the same endpoint may persist different factors for identical options. Read this as the factor this one row settled at, and see https://bytekit.com/docs/billing/credits for what each endpoint charges. This is a LOGS ROW field and is NOT the scrape response envelope field — the envelope carries no flat `billing_multiplier` at all; its billing surface is the `billing` node (see BillingNode). This row reports only the base settled factor, so it can differ from `billing.multiplier` and from the X-Billing-Multiplier response header, both of which report the composed real-endpoint value. null while pending/failed. */
|
|
5507
5595
|
billing_multiplier?: number | null;
|
|
5508
5596
|
duration_ms?: number | null;
|
|
5509
5597
|
/** @description How the request originated (single/bulk/monitor). */
|
package/dist/index.d.ts
CHANGED
|
@@ -3,7 +3,7 @@ export type { ByteKitOptions, RequestOptions, ApiResponse, ConnectionErrorCode,
|
|
|
3
3
|
export { V01_METHODS } from './allowlist.js';
|
|
4
4
|
export type { V01Method } from './allowlist.js';
|
|
5
5
|
export type { ScrapeId, ScreenshotId, BulkId, MonitorId, SitemapId, ApiKeyId, WebhookDeliveryId, SearchType, SearchDateRange, SearchOpts, SearchResult, SearchResultImage, SearchResponse, } from './types.js';
|
|
6
|
-
export type {
|
|
6
|
+
export type { ScrapeOpts, MarkdownMode, MarkdownLinks, MarkdownImages, TokenEncoding, ScrapeCreateResponse, ScrapeGetResponse, ScrapeBulkCreateRequest, ScrapeBulkCreateResponse, ScrapeBulkGetResponse, } from './resources/scrape.js';
|
|
7
7
|
export type { ScreenshotCreateRequest, ScreenshotCreateResponse, ScreenshotGetResponse, } from './resources/screenshots.js';
|
|
8
8
|
export type { FetchOpts, FetchGetOpts, FetchResult, FetchCacheStatus, FetchBulkCreateRequest, FetchBulkCreateResponse, FetchBulkGetResponse, } from './resources/fetch.js';
|
|
9
9
|
export type { MonitorCreateRequest, MonitorCreateResponse, ListMonitorsParams, MonitorStatus, ListMonitorsResponse, MonitorResponse, MonitorUpdateRequest, UpdateMonitorResponse, ListMonitorCapturesResponse, } from './resources/monitors.js';
|
|
@@ -15,32 +15,8 @@ export type FetchBulkGetResponse = operations['getFetchBulk']['responses'][200][
|
|
|
15
15
|
*/
|
|
16
16
|
export interface FetchOpts {
|
|
17
17
|
url: string;
|
|
18
|
-
/** Output format. `'markdown'` runs the HTML→markdown pipeline; `'html'` returns raw HTML. Narrowed to the spec union — off-spec values (e.g. `'pdf'`) are rejected at compile time. */
|
|
19
|
-
format?: 'markdown' | 'html';
|
|
20
18
|
country?: string;
|
|
21
19
|
timeout_ms?: number;
|
|
22
|
-
/** Markdown processing mode. Only used when `format=markdown`. article/raw/llm. */
|
|
23
|
-
markdown_mode?: 'article' | 'raw' | 'llm';
|
|
24
|
-
/** BM25 query string for relevance-ranked filtering. Only used when `format=markdown`. */
|
|
25
|
-
markdown_query?: string;
|
|
26
|
-
/** Link rendering style. Only used when `format=markdown`. */
|
|
27
|
-
markdown_links?: 'inline' | 'references' | 'none' | 'text';
|
|
28
|
-
/** Image retention mode. Only used when `format=markdown`. */
|
|
29
|
-
markdown_images?: 'inline' | 'references' | 'none' | 'text';
|
|
30
|
-
/** Append a `Links:` footer to the markdown output. Only used when `format=markdown`. */
|
|
31
|
-
with_links_summary?: boolean;
|
|
32
|
-
/** Append an `Images:` footer to the markdown output. Only used when `format=markdown`. */
|
|
33
|
-
with_images_summary?: boolean;
|
|
34
|
-
/** Compact whitespace output. Only used when `format=markdown`. */
|
|
35
|
-
markdown_compact?: boolean;
|
|
36
|
-
/** Filter low-signal images. Only used when `format=markdown`. */
|
|
37
|
-
markdown_filter_images?: boolean;
|
|
38
|
-
/** When true, `formats.links`/`formats.images` return rich objects and a top-level `tables` array is included. Only used when `format=markdown`. */
|
|
39
|
-
markdown_include_media?: boolean;
|
|
40
|
-
/** When true, enables markdown-pipeline and tag-filter warnings for affected html/markdown requests. */
|
|
41
|
-
markdown_include_warnings?: boolean;
|
|
42
|
-
/** When true, includes a top-level `stats` object (chars, tokens, blocks). Only used when `format=markdown`. */
|
|
43
|
-
markdown_include_stats?: boolean;
|
|
44
20
|
/**
|
|
45
21
|
* How long a freshly fetched URL may be served from cache. Matches the
|
|
46
22
|
* OpenAPI `FetchRequest.cache_ttl` schema: a duration string (`"48h"`,
|
|
@@ -66,17 +42,15 @@ export interface FetchOpts {
|
|
|
66
42
|
* the generated `getFetch` query type by `_AssertFetchGetQueryKeysCoverSpec`
|
|
67
43
|
* below, so it cannot drift away from the spec silently.
|
|
68
44
|
*/
|
|
69
|
-
declare const FETCH_GET_QUERY_KEYS: readonly ["url", "
|
|
45
|
+
declare const FETCH_GET_QUERY_KEYS: readonly ["url", "country", "timeout_ms", "cache_ttl"];
|
|
70
46
|
type FetchGetQueryKey = (typeof FETCH_GET_QUERY_KEYS)[number];
|
|
71
47
|
/**
|
|
72
48
|
* Options accepted by `fetch.get` (GET /v1/fetch).
|
|
73
49
|
*
|
|
74
|
-
* A strict subset of {@link FetchOpts}: only the
|
|
50
|
+
* A strict subset of {@link FetchOpts}: only the four params the spec's
|
|
75
51
|
* `GET /v1/fetch` actually declares. Everything else on the POST body —
|
|
76
|
-
* `custom`
|
|
77
|
-
*
|
|
78
|
-
* excluded here and passing it is a compile error. Use `create()` (POST) for
|
|
79
|
-
* those.
|
|
52
|
+
* `custom` has no GET query param and is excluded here; passing it is a compile
|
|
53
|
+
* error. Use `create()` (POST) for custom metadata.
|
|
80
54
|
*/
|
|
81
55
|
export type FetchGetOpts = Pick<FetchOpts, FetchGetQueryKey>;
|
|
82
56
|
/** Cache-lookup outcome reported in the `X-Fetch-Cache` response header. */
|
|
@@ -84,8 +58,8 @@ export type FetchCacheStatus = 'hit' | 'miss' | 'bypass';
|
|
|
84
58
|
/**
|
|
85
59
|
* The result of `fetch.create` / `fetch.get`.
|
|
86
60
|
*
|
|
87
|
-
* `/v1/fetch` returns the fetched content AS the response body
|
|
88
|
-
*
|
|
61
|
+
* `/v1/fetch` returns the fetched content AS the response body, including its
|
|
62
|
+
* original media type and raw bytes, with metadata carried in `X-Fetch-*`
|
|
89
63
|
* headers on the response — it is NOT a JSON envelope. `body` is therefore the raw
|
|
90
64
|
* content string, never a parsed object, and the named fields decode the
|
|
91
65
|
* headers the docs call out. Anything not surfaced here is available on
|
|
@@ -144,10 +118,8 @@ export declare class FetchResource {
|
|
|
144
118
|
*
|
|
145
119
|
* Resolves a `FetchResult` — the raw content body plus `X-Fetch-*` metadata.
|
|
146
120
|
*
|
|
147
|
-
* Accepts only the
|
|
148
|
-
*
|
|
149
|
-
* options with no GET query param — they were silently dropped by this builder,
|
|
150
|
-
* so they are now compile errors instead. Use `create()` (POST) for those.
|
|
121
|
+
* Accepts only the four params the spec declares for this endpoint. `custom`
|
|
122
|
+
* is a POST-body option with no GET query param; use `create()` for custom metadata.
|
|
151
123
|
*
|
|
152
124
|
* The query string is built by iterating `FETCH_GET_QUERY_KEYS`, the same list
|
|
153
125
|
* `FetchGetOpts` is derived from, so every accepted option is necessarily
|
package/dist/resources/fetch.js
CHANGED
|
@@ -9,7 +9,7 @@ import { buildPath } from '../client.js';
|
|
|
9
9
|
* the generated `getFetch` query type by `_AssertFetchGetQueryKeysCoverSpec`
|
|
10
10
|
* below, so it cannot drift away from the spec silently.
|
|
11
11
|
*/
|
|
12
|
-
const FETCH_GET_QUERY_KEYS = ['url', '
|
|
12
|
+
const FETCH_GET_QUERY_KEYS = ['url', 'country', 'timeout_ms', 'cache_ttl'];
|
|
13
13
|
const CACHE_STATUSES = ['hit', 'miss', 'bypass'];
|
|
14
14
|
/** Build a typed {@link FetchResult} from a raw body + its header set. */
|
|
15
15
|
function toFetchResult(raw) {
|
|
@@ -60,10 +60,8 @@ export class FetchResource {
|
|
|
60
60
|
*
|
|
61
61
|
* Resolves a `FetchResult` — the raw content body plus `X-Fetch-*` metadata.
|
|
62
62
|
*
|
|
63
|
-
* Accepts only the
|
|
64
|
-
*
|
|
65
|
-
* options with no GET query param — they were silently dropped by this builder,
|
|
66
|
-
* so they are now compile errors instead. Use `create()` (POST) for those.
|
|
63
|
+
* Accepts only the four params the spec declares for this endpoint. `custom`
|
|
64
|
+
* is a POST-body option with no GET query param; use `create()` for custom metadata.
|
|
67
65
|
*
|
|
68
66
|
* The query string is built by iterating `FETCH_GET_QUERY_KEYS`, the same list
|
|
69
67
|
* `FetchGetOpts` is derived from, so every accepted option is necessarily
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { ByteKit, RequestOptions } from '../client.js';
|
|
2
|
-
import type {
|
|
2
|
+
import type { operations } from '../generated/api.js';
|
|
3
3
|
import type { ScrapeId, BulkId } from '../types.js';
|
|
4
4
|
/**
|
|
5
5
|
* Response of POST /v1/scrape — a 200 sync success envelope or a 202 queued
|
|
@@ -26,23 +26,11 @@ export type MarkdownLinks = 'inline' | 'references' | 'none' | 'text';
|
|
|
26
26
|
export type MarkdownImages = 'inline' | 'references' | 'none' | 'text';
|
|
27
27
|
/** Tokenizer encoding hint for `token_budget`. */
|
|
28
28
|
export type TokenEncoding = 'cl100k_base' | 'o200k_base';
|
|
29
|
-
/**
|
|
30
|
-
* A cookie to set before loading the page — the spec's `Cookie` schema verbatim
|
|
31
|
-
* (`name` and `value` required; `domain`, `path`, `secure`, `httpOnly` optional).
|
|
32
|
-
*
|
|
33
|
-
* DERIVED from the generated schema (#2547), never hand-authored: `cookies` was
|
|
34
|
-
* previously an open bag of untyped records, so a nameless cookie compiled and only
|
|
35
|
-
* failed as a 422 on the wire.
|
|
36
|
-
*/
|
|
37
|
-
export type ScrapeCookie = components['schemas']['Cookie'];
|
|
38
29
|
/** Options for POST /v1/scrape. */
|
|
39
30
|
export interface ScrapeOpts {
|
|
40
31
|
url: string;
|
|
41
32
|
formats?: Array<'raw_html' | 'html' | 'markdown' | 'links' | 'images'>;
|
|
42
33
|
country?: string;
|
|
43
|
-
/** Cookies to set before loading the page. `name` and `value` are required (spec `Cookie` schema). */
|
|
44
|
-
cookies?: ScrapeCookie[];
|
|
45
|
-
headers?: Record<string, string>;
|
|
46
34
|
delay_ms?: number;
|
|
47
35
|
/** SERVER-side scrape budget in ms — how long the gateway may spend on the scrape itself. For the CLIENT-side abort budget (when this SDK gives up on the wire), see `RequestOptions.requestTimeoutMs`. */
|
|
48
36
|
timeout_ms?: number;
|
package/dist/version.d.ts
CHANGED
|
@@ -5,4 +5,4 @@
|
|
|
5
5
|
* outbound request carries — see the emit site in `client.ts`. The header name is spelled
|
|
6
6
|
* only there, so issue #2905's pre-merge gate 1 counts exactly one occurrence in `src/`.
|
|
7
7
|
*/
|
|
8
|
-
export declare const SDK_VERSION = "0.
|
|
8
|
+
export declare const SDK_VERSION = "0.9.0";
|
package/dist/version.js
CHANGED