@hunt-labs/bytekit-cli 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +203 -3
- package/README.md +74 -6
- package/dist/client.d.ts +126 -6
- package/dist/client.js +248 -30
- package/dist/commands/account.js +2 -1
- package/dist/commands/bulk.js +30 -8
- package/dist/commands/fetch.js +83 -20
- package/dist/commands/monitors.js +46 -8
- package/dist/commands/scrape.js +38 -9
- package/dist/commands/screenshots.js +184 -7
- package/dist/commands/search.js +2 -1
- package/dist/commands/sitemap.js +28 -3
- package/dist/commands/usage.js +6 -3
- package/dist/commands/webhook-deliveries.js +4 -2
- package/dist/lib/download-artifact.d.ts +30 -1
- package/dist/lib/download-artifact.js +90 -5
- package/dist/lib/parse-flags.js +3 -1
- package/dist/lib/url-lines.d.ts +46 -8
- package/dist/lib/url-lines.js +58 -19
- package/dist/output.d.ts +33 -2
- package/dist/output.js +113 -5
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -8,8 +8,186 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
This file records only the delta between versions. For how to use the package
|
|
9
9
|
today, see [README.md](./README.md), which remains the authoritative usage guide.
|
|
10
10
|
|
|
11
|
+
## [Unreleased]
|
|
12
|
+
|
|
13
|
+
## [0.7.0] - 2026-08-07
|
|
14
|
+
|
|
15
|
+
### Breaking
|
|
16
|
+
|
|
17
|
+
- **A not-ready `screenshots get -o` / `screenshots create -o` now exits `2` and prints the bare
|
|
18
|
+
`ss_` id, instead of exiting `1` with `not ready yet: screenshot is pending`.** The two poll
|
|
19
|
+
commands disagreed: `scrape get --raw` on a still-queued scrape already wrote the bare `sc_` id
|
|
20
|
+
to stdout and exited `2` — a distinct, scriptable "accepted but not ready" signal — while the
|
|
21
|
+
screenshots side reported the same state as a failure and threw the handle away, so a polling
|
|
22
|
+
script could branch on "queued" for one command and not the other. The screenshots branch now
|
|
23
|
+
mirrors scrape exactly: bare `ss_` id on stdout, a poll note on stderr, exit `2`, and (as before)
|
|
24
|
+
no file written. A script that treated any non-zero exit as "this screenshot failed" now sees `2`
|
|
25
|
+
for a job that is merely still running — branch on the code (`2` = poll again, `1` = failed)
|
|
26
|
+
rather than on non-zero. Three states deliberately do **not** move: a terminally `failed`
|
|
27
|
+
screenshot stays exit `1` with its error surfaced (it will never complete, so a "poll again"
|
|
28
|
+
signal would loop a script forever), an envelope carrying no `ss_` id stays exit `1` (there is no
|
|
29
|
+
handle to hand back), and `--json` keeps the envelope at exit `0`.
|
|
30
|
+
- **`--json` on a `fetch` command now emits `X-Fetch-ID` and `X-Fetch-URL`, not `X-Fetch-Id` and
|
|
31
|
+
`X-Fetch-Url`.** The metadata keys were derived by a generic segment-wise Title-Caser, which
|
|
32
|
+
lowercases every non-initial character of a segment — so the two acronym-bearing headers came out
|
|
33
|
+
in a spelling the API reference never uses, and a `jq '."X-Fetch-ID"'` filter written from the
|
|
34
|
+
docs returned `null`. All 16 `X-Fetch-*` keys now match the spec names byte-for-byte. The
|
|
35
|
+
**values** are unchanged, the non-`X-Fetch-*` headers stay filtered out, and every other key
|
|
36
|
+
(`X-Fetch-Waf-Vendor`, `X-Fetch-Duration-Ms`, `X-Fetch-Fast-Path`, …) is spelled exactly as
|
|
37
|
+
before. A consumer reading `.["X-Fetch-Id"]` must switch to `.["X-Fetch-ID"]`.
|
|
38
|
+
|
|
39
|
+
### Fixed
|
|
40
|
+
|
|
41
|
+
- **A `screenshots get -o` / `screenshots create -o` poll only says "poll again" for a `pending` or
|
|
42
|
+
`processing` screenshot.** The not-ready predicate decided from the presence of an `image_url`,
|
|
43
|
+
an `id` and a nested error alone — `status` was read only to phrase the message — so a
|
|
44
|
+
`completed` screenshot whose `image_url` never materialized, a status outside the closed enum,
|
|
45
|
+
and the flat `error_code`/`error_message` failed shape all received exit `2` and "no image
|
|
46
|
+
artifact is available yet. Poll it with: …" for a job that would never complete. Those states now
|
|
47
|
+
exit `1`: a terminal failure prints `screenshot failed: …`, and any other artifact-less state
|
|
48
|
+
prints `no image artifact available: screenshot is <status>`. Genuine `pending` / `processing`
|
|
49
|
+
polls keep the bare `ss_` id, the poll note and exit `2` exactly as before — **including** a
|
|
50
|
+
poll whose envelope carries an `error_code` from an attempt that failed and is being retried,
|
|
51
|
+
since the status, not the error, decides whether polling can still help. `--json` still prints
|
|
52
|
+
the envelope at exit `0`.
|
|
53
|
+
- **A terminally failed screenshot reported through the spec's flat `error_code`/`error_message`
|
|
54
|
+
fields now reaches stderr.** Only the nested `{error:{code,message}}` envelope was read, so the
|
|
55
|
+
shape `ScreenshotResponse` actually declares surfaced neither code nor message. It now prints
|
|
56
|
+
`screenshot failed: <message> (<code>)` — the formatting `scrape get --raw` already uses for a
|
|
57
|
+
failed scrape. The nested envelope's own message is unchanged.
|
|
58
|
+
- **`scrape get --raw` on a terminally failed scrape now reports why it failed.** It printed
|
|
59
|
+
`--raw: response contains no formats` — technically true of a `ScrapeErrorEnvelope`, and the one
|
|
60
|
+
diagnostic that cannot explain anything — while the envelope's own `error.code` and
|
|
61
|
+
`error.message` never reached the caller. Both now go to stderr (exit `1`, unchanged). The
|
|
62
|
+
failure is reported ahead of the format diagnostics, so an envelope carrying both a failure and
|
|
63
|
+
an empty `formats: {}` still says why it failed rather than pointing at a `--format` choice that
|
|
64
|
+
cannot help.
|
|
65
|
+
- **An empty-formats success now names the warning that explains it.** A completed scrape whose
|
|
66
|
+
persisted artifact could not be rehydrated comes back with `formats: {}` and an
|
|
67
|
+
`artifact_unavailable` warning; `--raw` rendered that as `--raw requires --format when the
|
|
68
|
+
response contains multiple formats`, which is false on both counts. It now names the warning's
|
|
69
|
+
code and message, or says plainly that the formats map is empty when no warning explains it.
|
|
70
|
+
- **Error paths no longer risk losing their own diagnostic.** `handleError` and the missing-API-key
|
|
71
|
+
path each wrote to stderr and then immediately hard-`process.exit(1)`. Node's stderr is
|
|
72
|
+
asynchronous when it points at a pipe, and `process.exit` discards whatever is still queued — so
|
|
73
|
+
the one line a script greps was the line most at risk of being thrown away, which is exactly the
|
|
74
|
+
defect this package's flush policy exists to prevent. Both now set `process.exitCode` and let the
|
|
75
|
+
process end naturally once stderr has drained. The exit codes and the rendered messages are
|
|
76
|
+
byte-identical.
|
|
77
|
+
- `--headers`, `--custom` and `--metadata` said `must be valid JSON (a object)` on a malformed
|
|
78
|
+
value; the article now agrees — `an object` / `an array`.
|
|
79
|
+
- The `[0.6.0]` link reference, missing from the bottom of this file, is restored.
|
|
80
|
+
|
|
81
|
+
## [0.6.0] - 2026-07-29
|
|
82
|
+
|
|
83
|
+
### Breaking
|
|
84
|
+
|
|
85
|
+
- **`screenshots create --async` now exits `2` in human mode instead of `0`.** The queued 202 is
|
|
86
|
+
not a completed capture — there is no artifact yet — and every other queued path in the CLI
|
|
87
|
+
already signalled that with exit code `2` (`scrape create --async`). A script that relied on
|
|
88
|
+
exit `0` to mean "the screenshot is ready" was reading a false success; under `set -e` such a
|
|
89
|
+
script now stops at the queue step, which is the intended correction. The envelope itself is
|
|
90
|
+
unchanged and still printed. Two carve-outs: **`--json` keeps exit `0`** (the envelope is a
|
|
91
|
+
complete machine-readable payload — identical to `scrape create --async --json`), and
|
|
92
|
+
`--async -o <file>` also exits `2` while still writing no file (it prints the queued envelope
|
|
93
|
+
and skips the download, as before). Poll with `bytekit screenshots get <id>`.
|
|
94
|
+
- **A `--file` per-item key outside a bulk endpoint's own strict schema now fails the whole
|
|
95
|
+
create with a `422`** instead of silently succeeding with that key ignored. In 0.5.x, `--file`
|
|
96
|
+
never read anything but `url` from a JSON line — every other key was dropped client-side
|
|
97
|
+
before the request was ever built. As of 0.6.0 those keys ride to the server (`items` on
|
|
98
|
+
`bulk create` / `scrape bulk create`; inside `urls` on `fetch bulk create`), and `/v1/bulk` /
|
|
99
|
+
`/v1/scrape/bulk` reject any key their `.strict()` per-item schema doesn't recognize. A file
|
|
100
|
+
that worked in 0.5.x purely because its extra keys were quietly discarded may now `422` — see
|
|
101
|
+
each command's `--file` help for the keys actually honored on that endpoint.
|
|
102
|
+
- **`fetch bulk create --file` per-item keys outside `url`/`format`/`country`/`cache_ttl` now
|
|
103
|
+
fail the whole create with a `422`** instead of silently succeeding with that key ignored.
|
|
104
|
+
`/v1/fetch/bulk`'s per-item override schema (inside `urls[]`) is now `.strict()` on the
|
|
105
|
+
gateway (issue #2648), matching the `/v1/bulk` / `/v1/scrape/bulk` posture above. The
|
|
106
|
+
`--file` help text for `fetch bulk create` is now uniform with `bulk create` /
|
|
107
|
+
`scrape bulk create` and no longer enumerates keys — this entry is where the honored set
|
|
108
|
+
(`url`, `format`, `country`, `cache_ttl`) is documented instead.
|
|
109
|
+
|
|
110
|
+
### Added
|
|
111
|
+
|
|
112
|
+
- **Per-item bulk configuration on all three bulk creates** (`bulk create`, `scrape bulk create`,
|
|
113
|
+
`fetch bulk create`). A `--file` line may now be a JSON object carrying `url` plus per-item
|
|
114
|
+
fields instead of only a bare URL; such a file is sent as `items` on `/v1/bulk` and
|
|
115
|
+
`/v1/scrape/bulk`, and as a heterogeneous `urls` array on `/v1/fetch/bulk` (which has no
|
|
116
|
+
`items` property). A line whose only key is `url` still normalizes to a bare URL string, so a
|
|
117
|
+
file of url-only lines produces a byte-identical request to before. A JSON line with a missing
|
|
118
|
+
or non-string `url` is now rejected locally with a usage error (exit `1`) instead of being sent
|
|
119
|
+
to the server as a URL. (See Breaking, above, for the per-item strict-schema behavior change
|
|
120
|
+
this enables.)
|
|
121
|
+
- `--defaults`, `--webhook-secret` and `--metadata` on all three bulk creates, plus `--custom` on
|
|
122
|
+
`scrape bulk create` only (`/v1/bulk` and `/v1/fetch/bulk` have no top-level `custom`, so the
|
|
123
|
+
flag would be rejected or stripped there). `--defaults` is forwarded verbatim; the server owns
|
|
124
|
+
precedence and unknown-key handling.
|
|
125
|
+
- The full markdown-tuning surface on `fetch create`: `--markdown-query`, `--markdown-links`,
|
|
126
|
+
`--markdown-images`, `--with-links-summary`, `--with-images-summary`, `--markdown-compact`,
|
|
127
|
+
`--markdown-filter-images`, `--markdown-include-media`, `--markdown-include-warnings`,
|
|
128
|
+
`--markdown-include-stats`, and `--custom`.
|
|
129
|
+
- Webhook delivery and processing flags on `sitemap create`: `--webhook-url`, `--webhook-secret`,
|
|
130
|
+
`--cache-ttl`, `--compact`, `--process`, `--metadata`. Webhook-driven sitemap crawls were
|
|
131
|
+
previously unreachable from the shell.
|
|
132
|
+
- `--headers`, `--cookies` and `--metadata` on `screenshots create`.
|
|
133
|
+
- `--metadata` on `monitors create`, and `--metadata` + `--url` on `monitors update`. `--type`
|
|
134
|
+
remains deliberately absent from `update` — it is immutable server-side.
|
|
135
|
+
|
|
136
|
+
## [0.5.1] - 2026-07-28 [NEVER PUBLISHED]
|
|
137
|
+
|
|
138
|
+
> **Never published to npm.** This version was cut in-tree but no `@hunt-labs/bytekit-cli@0.5.1`
|
|
139
|
+
> tarball exists — `npm i @hunt-labs/bytekit-cli@0.5.1` fails. Everything below shipped to users in
|
|
140
|
+
> **0.6.0**, which is the first published release containing it. Reconciled by issue #2680; the
|
|
141
|
+
> `release-drift` CI gate now blocks a new phantom heading from being introduced.
|
|
142
|
+
|
|
143
|
+
### Fixed
|
|
144
|
+
|
|
145
|
+
- **Corrects the scope of the 0.5.0 "stalled body" fix claim.** `screenshots create`/`get -o`
|
|
146
|
+
downloads the `image_url` artifact bytes with a bare `globalThis.fetch` call
|
|
147
|
+
(`lib/download-artifact.ts`) that sat entirely outside the transport seam `sendWithRetry` owns —
|
|
148
|
+
no abort signal, no connection-error taxonomy. The 0.5.0 entry below ("The per-attempt request
|
|
149
|
+
timeout now covers the response body read... a server that answered with headers and then
|
|
150
|
+
stalled its body... hung the CLI forever", #2546) reads as if it covered every response body; it
|
|
151
|
+
only ever covered API-call bodies read through the JSON/raw transports. A stalled artifact host
|
|
152
|
+
(headers, then a body that never completes) still hung the CLI indefinitely until now. The
|
|
153
|
+
download path now shares the same per-attempt abort budget and `ByteKitConnectionError` timeout
|
|
154
|
+
taxonomy as the API transport (`createAttemptBudget` / `toConnectionError`, exported from
|
|
155
|
+
`client.ts` — no second timeout implementation), with the same 120 s per-attempt default and no
|
|
156
|
+
retry (artifact GETs are not the API's 429 domain). Partial-file policy: the response is
|
|
157
|
+
buffered fully in memory before any write, so an aborted download never leaves a
|
|
158
|
+
partial/truncated file at the `-o` destination. **User-visible consequence:** a legitimately
|
|
159
|
+
slow (but not stalled) large artifact download now hard-fails after 120 s instead of waiting
|
|
160
|
+
indefinitely, and there is currently no CLI flag to raise that budget for a single command.
|
|
161
|
+
(#2587)
|
|
162
|
+
- While in the file: the 429-retry path left its (unread) response body undrained before sleeping
|
|
163
|
+
and retrying — undici cannot reclaim the underlying socket for connection-pool reuse until a body
|
|
164
|
+
is fully read. The body is now drained with `response.text()` on that path, bounded by its own
|
|
165
|
+
short (2 s) cap independent of the 120 s per-attempt budget, so a 429 whose body stalls can no
|
|
166
|
+
longer delay the retry by anywhere near the full attempt budget (or, with the budget disabled,
|
|
167
|
+
forever). **Correction to an intermediate `Unreleased` draft of this same fix:** that draft's cap
|
|
168
|
+
only bounded how long the _retry_ waited — it resolved on schedule but never touched the
|
|
169
|
+
abandoned `response.text()` call itself, which kept reading against a live socket indefinitely.
|
|
170
|
+
Because every CLI success path exits via a soft `process.exitCode` (#2544) rather than
|
|
171
|
+
`process.exit()`, that orphaned read kept the whole process alive forever — strictly worse than
|
|
172
|
+
having no drain at all, and on the exact class of hang this ticket exists to eliminate. The cap
|
|
173
|
+
now also **aborts** the request when it fires, so the abandoned read is actually torn down, not
|
|
174
|
+
merely ignored; this holds even when the per-attempt timeout itself is disabled
|
|
175
|
+
(`timeoutMs <= 0`). **User-visible consequence:** a 429 with a stalled body still costs the retry
|
|
176
|
+
at most ~2 s, exactly as before, but the CLI process now reliably exits afterward instead of
|
|
177
|
+
hanging forever regardless of that visible latency. The same discard-drain on the
|
|
178
|
+
artifact-download's non-2xx path is removed outright rather than capped: that path throws
|
|
179
|
+
straight into `process.exit(1)`, so there is no subsequent request in the process for a reclaimed
|
|
180
|
+
socket to serve, and the drain there bought nothing. (#2587)
|
|
181
|
+
|
|
11
182
|
## [0.5.0] - 2026-07-24
|
|
12
183
|
|
|
184
|
+
### Note
|
|
185
|
+
|
|
186
|
+
- The `0.3.1` and `0.4.0` versions below document real, tagged changes but were
|
|
187
|
+
never published to npm — the registry's version history jumps straight from
|
|
188
|
+
`0.3.0` to `0.5.0`. Don't go looking for those tarballs; the changes they
|
|
189
|
+
describe ship for the first time in this release.
|
|
190
|
+
|
|
13
191
|
### Added
|
|
14
192
|
|
|
15
193
|
- Surface parity round 2 (#2545): every `x-sdk-scope: v0.1` endpoint the SDK
|
|
@@ -83,7 +261,12 @@ today, see [README.md](./README.md), which remains the authoritative usage guide
|
|
|
83
261
|
`sc_` id** on stdout with a note on stderr (it does not print the queued job
|
|
84
262
|
envelope). (#2544)
|
|
85
263
|
|
|
86
|
-
## [0.4.0] - 2026-07-23
|
|
264
|
+
## [0.4.0] - 2026-07-23 [NEVER PUBLISHED]
|
|
265
|
+
|
|
266
|
+
> **Never published to npm.** This version was cut in-tree but no `@hunt-labs/bytekit-cli@0.4.0`
|
|
267
|
+
> tarball exists — `npm i @hunt-labs/bytekit-cli@0.4.0` fails. Everything below shipped to users in
|
|
268
|
+
> **0.5.0**, which is the first published release containing it. Reconciled by issue #2680; the
|
|
269
|
+
> `release-drift` CI gate now blocks a new phantom heading from being introduced.
|
|
87
270
|
|
|
88
271
|
### Removed (**BREAKING**)
|
|
89
272
|
|
|
@@ -104,7 +287,13 @@ today, see [README.md](./README.md), which remains the authoritative usage guide
|
|
|
104
287
|
**Replacement:** manage API keys from the dashboard at
|
|
105
288
|
[app.bytekit.com](https://app.bytekit.com). (#2536)
|
|
106
289
|
|
|
107
|
-
## [0.3.1] - 2026-07-22
|
|
290
|
+
## [0.3.1] - 2026-07-22 [NEVER PUBLISHED]
|
|
291
|
+
|
|
292
|
+
> **Never published to npm.** This version was cut in-tree but no `@hunt-labs/bytekit-cli@0.3.1`
|
|
293
|
+
> tarball exists — `npm i @hunt-labs/bytekit-cli@0.3.1` fails. Everything below shipped to users in
|
|
294
|
+
> **0.5.0**, which is the first published release containing it (0.4.0 was never published either —
|
|
295
|
+
> see its heading above). Reconciled by issue #2680; the `release-drift` CI gate now blocks a new
|
|
296
|
+
> phantom heading from being introduced.
|
|
108
297
|
|
|
109
298
|
### Fixed
|
|
110
299
|
|
|
@@ -187,7 +376,12 @@ today, see [README.md](./README.md), which remains the authoritative usage guide
|
|
|
187
376
|
rebuilding the shared `dist`), which is now fixed. No consumer-facing changes
|
|
188
377
|
beyond 0.2.1.
|
|
189
378
|
|
|
190
|
-
## [0.2.1] - 2026-07-16
|
|
379
|
+
## [0.2.1] - 2026-07-16 [NEVER PUBLISHED]
|
|
380
|
+
|
|
381
|
+
> **Never published to npm.** This version was cut in-tree but no `@hunt-labs/bytekit-cli@0.2.1`
|
|
382
|
+
> tarball exists — `npm i @hunt-labs/bytekit-cli@0.2.1` fails. Everything below shipped to users in
|
|
383
|
+
> **0.2.2**, which is the first published release containing it. Reconciled by issue #2680; the
|
|
384
|
+
> `release-drift` CI gate now blocks a new phantom heading from being introduced.
|
|
191
385
|
|
|
192
386
|
### Changed
|
|
193
387
|
|
|
@@ -236,7 +430,13 @@ Additive, lower-risk release — no breaking changes to existing commands.
|
|
|
236
430
|
- Initial release of the ByteKit CLI, wrapping the ByteKit SDK as shell
|
|
237
431
|
subcommands, published under the original `rapidcrawl-cli` name.
|
|
238
432
|
|
|
433
|
+
[Unreleased]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
434
|
+
[0.7.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
435
|
+
[0.6.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
436
|
+
[0.5.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
437
|
+
[0.5.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
239
438
|
[0.4.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
439
|
+
[0.3.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
240
440
|
[0.3.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
241
441
|
[0.2.2]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
242
442
|
[0.2.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
package/README.md
CHANGED
|
@@ -172,6 +172,17 @@ response metadata instead of the body. `-o <file>` writes the body bytes to a
|
|
|
172
172
|
file byte-for-byte (with `--json`, the metadata object goes to the file
|
|
173
173
|
instead). `fetch bulk` returns an ordinary JSON job envelope.
|
|
174
174
|
|
|
175
|
+
The `--json` metadata keys are the **spec header names exactly** as the API
|
|
176
|
+
reference declares them, so a filter written from the docs works unchanged:
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
bytekit fetch get --url https://example.com --json | jq -r '."X-Fetch-ID"'
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Note the acronym casing — `X-Fetch-ID` and `X-Fetch-URL`, not `X-Fetch-Id` /
|
|
183
|
+
`X-Fetch-Url`. (Versions up to 0.6.0 emitted the Title-Cased spellings; see the
|
|
184
|
+
CHANGELOG.)
|
|
185
|
+
|
|
175
186
|
Both verbs accept `--country <cc>`, `--timeout-ms <n>`, and `--cache-ttl <ttl>`
|
|
176
187
|
(`0`, `Nh`, or `Nd`), each omitted when absent. `--markdown-mode`
|
|
177
188
|
(`article`, `raw`, `llm`) is `fetch create` only.
|
|
@@ -182,6 +193,11 @@ Both verbs accept `--country <cc>`, `--timeout-ms <n>`, and `--cache-ttl <ttl>`
|
|
|
182
193
|
bytekit search "ai web scraping" --limit 10 --country us
|
|
183
194
|
```
|
|
184
195
|
|
|
196
|
+
`search` also accepts `--type <type>` (`web` (default), `news`, or `images`),
|
|
197
|
+
`--language <code>` (two-letter language code, e.g. `en`, `fr`), and
|
|
198
|
+
`--date-range <window>` (recency filter: `any` (default), `hour`, `day`,
|
|
199
|
+
`week`, `month`, `year`). Each flag is omitted from the request when absent.
|
|
200
|
+
|
|
185
201
|
### `screenshots` — capture a page
|
|
186
202
|
|
|
187
203
|
```bash
|
|
@@ -232,7 +248,7 @@ bytekit monitors create --url https://example.com/pricing --interval-type daily
|
|
|
232
248
|
bytekit monitors list --limit 25 --status active
|
|
233
249
|
bytekit monitors update mon_123abc --status paused # pause
|
|
234
250
|
bytekit monitors update mon_123abc --status active # resume
|
|
235
|
-
bytekit monitors captures list mon_123abc
|
|
251
|
+
bytekit monitors captures list mon_123abc --limit 25
|
|
236
252
|
bytekit monitors delete mon_123abc
|
|
237
253
|
```
|
|
238
254
|
|
|
@@ -240,6 +256,10 @@ bytekit monitors delete mon_123abc
|
|
|
240
256
|
(`active`, `processing`, `paused`, `cancelled`, `suspended`) as query filters,
|
|
241
257
|
each omitted when absent.
|
|
242
258
|
|
|
259
|
+
`monitors captures list <id>` accepts `--limit <n>` and `--cursor <cursor>` to
|
|
260
|
+
page through a monitor's capture history — pass the response's `next_cursor`
|
|
261
|
+
back as `--cursor` to fetch the next page. Both flags are omitted when absent.
|
|
262
|
+
|
|
243
263
|
`monitors update` gains `--status` (`active` | `paused` — pause and resume from
|
|
244
264
|
the shell), `--change-threshold <pct>` (accepts a float), `--notify-on`
|
|
245
265
|
(`change`, `every`), `--webhook-secret <secret>`, and three JSON-object flags:
|
|
@@ -294,11 +314,47 @@ omitted when absent. `retry <id>` requeues a failed or exhausted delivery.
|
|
|
294
314
|
|
|
295
315
|
## Exit codes
|
|
296
316
|
|
|
297
|
-
| Code | Meaning
|
|
298
|
-
| ---- |
|
|
299
|
-
| `0` | Success.
|
|
300
|
-
| `1` | Usage error, or an API/connection error.
|
|
301
|
-
| `2` | The
|
|
317
|
+
| Code | Meaning |
|
|
318
|
+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
319
|
+
| `0` | Success. |
|
|
320
|
+
| `1` | Usage error, or an API/connection error. |
|
|
321
|
+
| `2` | The job was accepted but is **not ready**, so there is no content body or artifact to emit. Applies to `scrape create --async` with `--raw` or a bare `-o` (which writes the bare `sc_` id to stdout) and in human mode; to `scrape get --raw`/`-o` on a still-queued job (bare `sc_` id to stdout); to `screenshots create --async` in human mode (including with `-o`, which deliberately writes no file); and to `screenshots create -o` / `screenshots get -o` on a pending or processing screenshot (bare `ss_` id to stdout, no file written). Not a failure: the job was accepted and billed. Poll it with `bytekit scrape get <id>` / `bytekit screenshots get <id>`. **`--json` keeps exit `0` in every case** — the envelope is a complete machine-readable payload. |
|
|
322
|
+
|
|
323
|
+
A **terminally failed** job is NOT code `2` — it will never complete, so it is
|
|
324
|
+
exit `1` with the reason on stderr. `scrape get --raw` on a failed scrape prints
|
|
325
|
+
that scrape's own `error.code` and message, and `screenshots get -o` on a failed
|
|
326
|
+
screenshot prints `screenshot failed: …`.
|
|
327
|
+
|
|
328
|
+
Only `pending` and `processing` earn code `2`. Any other screenshot state that
|
|
329
|
+
still has no artifact — a `completed` screenshot whose `image_url` never
|
|
330
|
+
materialized, or a status this CLI build does not know — is exit `1` with
|
|
331
|
+
`no image artifact available: screenshot is <status>` on stderr, because
|
|
332
|
+
polling it again cannot change the outcome.
|
|
333
|
+
|
|
334
|
+
The status is what decides, not the presence of an error. A `pending` or
|
|
335
|
+
`processing` screenshot that reports an `error_code` from an attempt that failed
|
|
336
|
+
and is being **retried** is still code `2` with its `ss_` id — the job has not
|
|
337
|
+
finished, and abandoning it there would throw away a capture the server is still
|
|
338
|
+
working on.
|
|
339
|
+
|
|
340
|
+
A **human-mode poll without `-o`/`--raw`** stays exit `0` whatever the job's
|
|
341
|
+
state: `bytekit scrape get <id>` and `bytekit screenshots get <id>` were asked
|
|
342
|
+
for the envelope, and the envelope is exactly what they printed — the status
|
|
343
|
+
field in it is the answer. Code `2` is reserved for the case where you asked for
|
|
344
|
+
a **body or artifact** that does not exist yet.
|
|
345
|
+
|
|
346
|
+
The stderr message for exit code `1` has two shapes, depending on where the
|
|
347
|
+
error originates:
|
|
348
|
+
|
|
349
|
+
- **App-level errors** (a rejected API call, a connection failure, a missing
|
|
350
|
+
key) are always prefixed `Error: ` (uppercase) — e.g.
|
|
351
|
+
`Error: API key required. Pass --key or set BYTEKIT_API_KEY.`
|
|
352
|
+
- **Commander-level usage errors** (an unknown command, a missing required
|
|
353
|
+
argument, an unrecognized flag) come from the `commander` library itself and
|
|
354
|
+
use its own lowercase `error: ` prefix — e.g. `error: unknown command 'bogus'`.
|
|
355
|
+
|
|
356
|
+
Scripts that grep stderr for a failure reason should match case-insensitively,
|
|
357
|
+
or handle both prefixes.
|
|
302
358
|
|
|
303
359
|
Code `2` is deliberately distinct from both `0` and `1` so a script can branch
|
|
304
360
|
on "queued" without parsing stderr:
|
|
@@ -312,6 +368,18 @@ case $? in
|
|
|
312
368
|
esac
|
|
313
369
|
```
|
|
314
370
|
|
|
371
|
+
The same branch works for a screenshot poll, because both commands now signal
|
|
372
|
+
"not ready" identically:
|
|
373
|
+
|
|
374
|
+
```bash
|
|
375
|
+
ss_id=$(bytekit screenshots get "$ss_id" -o shot.png)
|
|
376
|
+
case $? in
|
|
377
|
+
0) echo "downloaded shot.png" ;;
|
|
378
|
+
2) echo "$ss_id not ready — poll again" ;;
|
|
379
|
+
*) exit 1 ;; # terminal failure, reason on stderr
|
|
380
|
+
esac
|
|
381
|
+
```
|
|
382
|
+
|
|
315
383
|
## Documentation
|
|
316
384
|
|
|
317
385
|
Full API reference: [bytekit.com/docs](https://bytekit.com/docs).
|
package/dist/client.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { ByteKit } from '@hunt-labs/bytekit-sdk';
|
|
1
|
+
import { ByteKit, ByteKitConnectionError } from '@hunt-labs/bytekit-sdk';
|
|
2
2
|
export interface GlobalOptions {
|
|
3
3
|
key?: string | undefined;
|
|
4
4
|
baseUrl?: string | undefined;
|
|
@@ -22,6 +22,24 @@ export declare const MIN_RETRY_AFTER_MS = 1000;
|
|
|
22
22
|
export declare const MAX_RETRY_AFTER_MS = 60000;
|
|
23
23
|
/** Exactly one retry. A second 429 is terminal — no loop. */
|
|
24
24
|
export declare const MAX_RETRIES = 1;
|
|
25
|
+
/**
|
|
26
|
+
* Cap on a best-effort, discard-only body drain (#2587 cr2-1). Independent of the per-attempt
|
|
27
|
+
* abort budget: the 429 drain below (`response.text().catch(() => {})`) exists ONLY so undici can
|
|
28
|
+
* return the socket to the connection pool — nothing downstream reads the drained bytes — but
|
|
29
|
+
* binding that read to the FULL per-attempt budget (120 s default, or, under `timeoutMs <= 0`,
|
|
30
|
+
* to no bound at all) let a stalled 429 body burn the whole budget, or hang forever, before a
|
|
31
|
+
* retry that needs nothing from that body. Short enough that a legitimately stalled 429 body costs
|
|
32
|
+
* the retry very little wall-clock time.
|
|
33
|
+
*
|
|
34
|
+
* #2587 cr3-2: past this cap, the read is not merely abandoned — {@link drainBestEffort} actively
|
|
35
|
+
* ABORTS its underlying request. Resolving the race without aborting anything (the earlier
|
|
36
|
+
* cr2-1 shape) let the retry proceed on schedule while the abandoned `response.text()` kept
|
|
37
|
+
* reading against a live socket forever, which — because the CLI exits via a soft
|
|
38
|
+
* `process.exitCode` (#2544) rather than `process.exit()` — kept the whole process alive
|
|
39
|
+
* indefinitely. `.catch`-guarding the read only made it safe to abandon the RESULT; it never made
|
|
40
|
+
* abandoning the underlying READ safe.
|
|
41
|
+
*/
|
|
42
|
+
export declare const DRAIN_TIMEOUT_MS = 2000;
|
|
25
43
|
/**
|
|
26
44
|
* Default CLIENT-SIDE abort budget for a single fetch attempt, in ms.
|
|
27
45
|
*
|
|
@@ -33,6 +51,49 @@ export declare const MAX_RETRIES = 1;
|
|
|
33
51
|
* exists.
|
|
34
52
|
*/
|
|
35
53
|
export declare const DEFAULT_TIMEOUT_MS = 120000;
|
|
54
|
+
/**
|
|
55
|
+
* A per-attempt abort budget: an `AbortController`, optionally fired by a `setTimeout`.
|
|
56
|
+
*
|
|
57
|
+
* The budget spans the WHOLE attempt — time-to-headers **and** the response body read (#2546).
|
|
58
|
+
* Arming it only until headers arrive left `response.text()` / `response.arrayBuffer()` unbounded,
|
|
59
|
+
* so a server that answered with headers and then stalled its body hung the CLI forever.
|
|
60
|
+
*
|
|
61
|
+
* The controller ALWAYS exists, even when `ms <= 0` — only its own timeout timer is conditional
|
|
62
|
+
* on `ms > 0`. `sendWithRetry`'s 429-drain cap ({@link DRAIN_TIMEOUT_MS}, via the returned
|
|
63
|
+
* `abort()`) needs a live controller to tear down an abandoned body read regardless of whether the
|
|
64
|
+
* OVERALL per-attempt budget is enabled — `ms <= 0` must still mean "no full-attempt timeout", not
|
|
65
|
+
* "no controller to abort a stalled drain with" (#2587 cr3-2: the earlier drain fix abandoned the
|
|
66
|
+
* read entirely rather than aborting it, so the socket — and the event loop — stayed alive
|
|
67
|
+
* indefinitely, worse than the pre-fix baseline it replaced).
|
|
68
|
+
*
|
|
69
|
+
* Deliberately NOT `AbortSignal.timeout` (which the SDK prefers): that schedules on a timer the
|
|
70
|
+
* test runner's fake timers cannot advance, so the transport's timeout would be untestable under
|
|
71
|
+
* `vi.useFakeTimers()`. CLI requests are short-lived and single-shot, so the GC advantage
|
|
72
|
+
* `AbortSignal.timeout` buys the long-running SDK does not apply here. The timer is `unref`'d so a
|
|
73
|
+
* pending budget never keeps the process alive, and `cleanup()` clears it on every exit path.
|
|
74
|
+
*
|
|
75
|
+
* Exported so `lib/download-artifact.ts` can reuse the same budget for the artifact-download path
|
|
76
|
+
* (#2587) rather than growing a second timeout implementation — that path sits entirely outside
|
|
77
|
+
* `sendWithRetry` (it is not a JSON/raw API call), so it would otherwise never inherit this bound.
|
|
78
|
+
*/
|
|
79
|
+
export declare function createAttemptBudget(ms: number): {
|
|
80
|
+
signal: AbortSignal;
|
|
81
|
+
cleanup: () => void;
|
|
82
|
+
abort: (reason: unknown) => void;
|
|
83
|
+
};
|
|
84
|
+
/**
|
|
85
|
+
* Map a `fetch` or body-read rejection (no usable HTTP payload was produced) onto the SDK's
|
|
86
|
+
* `ByteKitConnectionError` taxonomy, so a CLI failure carries the same `{name, code, cause}` a
|
|
87
|
+
* direct SDK call would.
|
|
88
|
+
*
|
|
89
|
+
* `timedOut` is true when this attempt's own budget fired the abort, which is the reliable
|
|
90
|
+
* discriminator (undici rejects with the signal's reason — or a wrapper carrying it, see
|
|
91
|
+
* {@link abortNameOf} — whose `name` we also inspect as a fallback for a caller-style AbortError).
|
|
92
|
+
*
|
|
93
|
+
* Exported so `lib/download-artifact.ts` reuses the same taxonomy for the artifact-download path
|
|
94
|
+
* (#2587) instead of hand-rolling a second error mapping.
|
|
95
|
+
*/
|
|
96
|
+
export declare function toConnectionError(err: unknown, method: string, path: string, timedOut: boolean): ByteKitConnectionError;
|
|
36
97
|
/**
|
|
37
98
|
* Parse a `Retry-After` header into a clamped millisecond delay.
|
|
38
99
|
*
|
|
@@ -48,6 +109,18 @@ export declare function parseRetryAfterMs(raw: string | null): number;
|
|
|
48
109
|
* property of the client (see {@link buildClient}) routes all JSON commands through it from one
|
|
49
110
|
* seam. `/v1/fetch` does NOT ride this transport — it rides `requestRaw` (see
|
|
50
111
|
* {@link createRetryingRequestRaw}); `/v1/fetch/bulk` is a JSON job envelope and does.
|
|
112
|
+
*
|
|
113
|
+
* DECISION (#2586 dod-4): the 4th `RequestOptions` parameter of the SDK's contract is accepted
|
|
114
|
+
* here — so the signature matches `ByteKit['request']` exactly, closing the gap where a caller-typo
|
|
115
|
+
* of a 5th positional argument would silently no-op instead of a compile error — but deliberately
|
|
116
|
+
* NOT forwarded into `sendWithRetry`. Two reasons: (1) no CLI command constructs a `RequestOptions`
|
|
117
|
+
* today (no `--timeout` flag exists yet, and `signal` has no CLI-level source), so the parameter is
|
|
118
|
+
* always `undefined` in practice; (2) the CLI already owns a per-attempt abort budget with its own
|
|
119
|
+
* semantics (`createAttemptBudget`, `timeoutMs` above) that would need explicit composition with a
|
|
120
|
+
* caller `signal` to honor `RequestOptions` correctly (the SDK's `composeSignals`) — inventing that
|
|
121
|
+
* unexercised composition here, with no caller able to reach it, would be undertested surface for no
|
|
122
|
+
* present behavior change. If a future CLI flag needs per-call overrides, wire it through
|
|
123
|
+
* `sendWithRetry`'s `timeoutMs` (already per-attempt) at that point, with its own tests.
|
|
51
124
|
*/
|
|
52
125
|
export declare function createRetryingRequest(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['request'];
|
|
53
126
|
/**
|
|
@@ -62,19 +135,66 @@ export declare function createRetryingRequest(apiKey: string, baseUrl: string, t
|
|
|
62
135
|
* so `FetchResource` builds an ordinary `FetchResult` from it and the CLI never forks fetch
|
|
63
136
|
* handling away from the SDK. `bytes` is byte-exact (binary-safe); `body` is the best-effort UTF-8
|
|
64
137
|
* decode, matching the SDK's own semantics.
|
|
138
|
+
*
|
|
139
|
+
* The 4th `RequestOptions` parameter is accepted but not forwarded — see the DECISION note on
|
|
140
|
+
* {@link createRetryingRequest}, which applies here verbatim. `FetchResource.create`/`get` DO thread
|
|
141
|
+
* a `requestOptions` argument through to this seam (unlike every other resource), but no CLI `fetch`
|
|
142
|
+
* command constructs one today (no `--timeout` flag), so it is still always `undefined` in practice.
|
|
65
143
|
*/
|
|
66
144
|
export declare function createRetryingRequestRaw(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['requestRaw'];
|
|
145
|
+
/**
|
|
146
|
+
* Build the CLI's header-carrying JSON transport: the SDK's `requestWithResponse` contract
|
|
147
|
+
* (`{data, headers, status}`, #2547), plus the same bounded 429 retry every other CLI transport
|
|
148
|
+
* gets.
|
|
149
|
+
*
|
|
150
|
+
* Unused by any CLI command today — `screenshots.createWithResponse` / `getWithResponse` and
|
|
151
|
+
* `sitemap.getWithResponse` all exist on the SDK surface but no command builds on them yet. It is
|
|
152
|
+
* still installed (see {@link buildClient}) rather than left un-overridden: the transport-coverage
|
|
153
|
+
* contract test (#2586 dod-3) derives its method list from the real SDK prototype, and leaving a
|
|
154
|
+
* transport method un-overridden is exactly how #2535 shipped broken — the first command that DOES
|
|
155
|
+
* call a `*WithResponse` resource method must not silently ride the bare SDK transport and skip the
|
|
156
|
+
* CLI's retry/timeout/error taxonomy.
|
|
157
|
+
*
|
|
158
|
+
* Same body-read shape as {@link createRetryingRequest} (204/empty → `undefined`, `JSON.parse`
|
|
159
|
+
* otherwise), with `headers` and `status` threaded through from the successful response alongside
|
|
160
|
+
* the parsed `data`. The 4th `RequestOptions` parameter is accepted but not forwarded — see the
|
|
161
|
+
* DECISION note on {@link createRetryingRequest}.
|
|
162
|
+
*/
|
|
163
|
+
export declare function createRetryingRequestWithResponse(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['requestWithResponse'];
|
|
67
164
|
/**
|
|
68
165
|
* Build an authenticated SDK client from global CLI options.
|
|
69
166
|
*
|
|
70
167
|
* Resolution order (explicit input beats environment):
|
|
71
168
|
* apiKey: --key flag → BYTEKIT_API_KEY.
|
|
72
169
|
* baseUrl: --base-url flag → BYTEKIT_BASE_URL → DEFAULT_BASE_URL.
|
|
73
|
-
* Exits with a descriptive error when no key is available.
|
|
74
170
|
*
|
|
75
|
-
*
|
|
76
|
-
*
|
|
77
|
-
*
|
|
78
|
-
*
|
|
171
|
+
* THROWS a plain `Error` when no key is available (#2686 item 4). It used to `console.error` and
|
|
172
|
+
* then hard-exit — an async stderr write followed immediately by the one call that discards
|
|
173
|
+
* the userspace stdio queue, which is exactly the defect the package's #2544 flush policy exists to
|
|
174
|
+
* prevent. Throwing stops just as hard (there is no client to return, E4) while routing the message
|
|
175
|
+
* through `handleError`, whose soft `process.exitCode` lets stderr drain first. The rendered stderr
|
|
176
|
+
* line is byte-identical: `handleError` supplies the `Error: ` prefix, and a plain Error carries
|
|
177
|
+
* neither a status nor a ByteKit code, so no parenthesised suffix is added.
|
|
178
|
+
*
|
|
179
|
+
* Because it throws, EVERY caller invokes it from inside its action's `try` block. `index.ts` drives
|
|
180
|
+
* commander with `.parse()`, not `.parseAsync()`, so nothing awaits an action's promise: a throw
|
|
181
|
+
* raised above the try would escape as an unhandled rejection and print a raw stack trace instead of
|
|
182
|
+
* the clean one-liner — the same constraint `commands/scrape.ts` already documents for
|
|
183
|
+
* `resolveOutputMode`. `error-exit-contract.test.ts` pins that placement structurally, across every
|
|
184
|
+
* call site at once.
|
|
185
|
+
*
|
|
186
|
+
* The returned client's `request`, `requestRaw`, AND `requestWithResponse` are all replaced by the
|
|
187
|
+
* retrying transports, so every command inherits the 429 retry — JSON commands via `request`, the
|
|
188
|
+
* `/v1/fetch` content commands via `requestRaw`, and the header-carrying `*WithResponse` surface via
|
|
189
|
+
* `requestWithResponse`. Overriding only `request` is the #2535 defect: the SDK's fetch resource
|
|
190
|
+
* rides `requestRaw`, so fetch bypassed the retry and the CLI's error taxonomy entirely; leaving
|
|
191
|
+
* `requestWithResponse` un-overridden is the same class of defect (#2586) — it is unused by any CLI
|
|
192
|
+
* command TODAY, but a future command built on `screenshots.createWithResponse` /
|
|
193
|
+
* `sitemap.getWithResponse` would otherwise silently ride the bare SDK transport.
|
|
194
|
+
*
|
|
195
|
+
* `baseUrl` is normalized ONCE, here, before it reaches either the SDK constructor or the CLI's own
|
|
196
|
+
* transport builders below — the CLI's transports build their request URL from this local variable
|
|
197
|
+
* directly (they do not call the SDK's own normalized `send()`), so a trailing-slash flag would
|
|
198
|
+
* otherwise double-slash every request the CLI makes (#2586).
|
|
79
199
|
*/
|
|
80
200
|
export declare function buildClient(opts: GlobalOptions): ByteKit;
|