@hunt-labs/bytekit-cli 0.3.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +230 -0
- package/README.md +187 -29
- package/dist/client.d.ts +128 -11
- package/dist/client.js +369 -95
- package/dist/commands/account.d.ts +11 -0
- package/dist/commands/account.js +10 -127
- package/dist/commands/bulk.js +36 -4
- package/dist/commands/fetch.js +91 -10
- package/dist/commands/monitors.js +92 -11
- package/dist/commands/scrape.js +131 -20
- package/dist/commands/screenshots.js +86 -1
- package/dist/commands/sitemap.js +38 -6
- package/dist/commands/usage.d.ts +13 -0
- package/dist/commands/usage.js +81 -0
- package/dist/commands/webhook-deliveries.d.ts +11 -0
- package/dist/commands/webhook-deliveries.js +59 -0
- package/dist/index.js +4 -0
- package/dist/lib/download-artifact.d.ts +30 -1
- package/dist/lib/download-artifact.js +89 -5
- package/dist/lib/parse-flags.d.ts +30 -0
- package/dist/lib/parse-flags.js +58 -0
- package/dist/lib/url-lines.d.ts +46 -8
- package/dist/lib/url-lines.js +58 -19
- package/dist/output.d.ts +47 -9
- package/dist/output.js +103 -16
- package/package.json +3 -3
- package/dist/lib/fetch-body.d.ts +0 -18
- package/dist/lib/fetch-body.js +0 -6
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,231 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
This file records only the delta between versions. For how to use the package
|
|
9
9
|
today, see [README.md](./README.md), which remains the authoritative usage guide.
|
|
10
10
|
|
|
11
|
+
## [Unreleased]
|
|
12
|
+
|
|
13
|
+
## [0.6.0] - 2026-07-29
|
|
14
|
+
|
|
15
|
+
### Breaking
|
|
16
|
+
|
|
17
|
+
- **`screenshots create --async` now exits `2` in human mode instead of `0`.** The queued 202 is
|
|
18
|
+
not a completed capture — there is no artifact yet — and every other queued path in the CLI
|
|
19
|
+
already signalled that with exit code `2` (`scrape create --async`). A script that relied on
|
|
20
|
+
exit `0` to mean "the screenshot is ready" was reading a false success; under `set -e` such a
|
|
21
|
+
script now stops at the queue step, which is the intended correction. The envelope itself is
|
|
22
|
+
unchanged and still printed. Two carve-outs: **`--json` keeps exit `0`** (the envelope is a
|
|
23
|
+
complete machine-readable payload — identical to `scrape create --async --json`), and
|
|
24
|
+
`--async -o <file>` also exits `2` while still writing no file (it prints the queued envelope
|
|
25
|
+
and skips the download, as before). Poll with `bytekit screenshots get <id>`.
|
|
26
|
+
- **A `--file` per-item key outside a bulk endpoint's own strict schema now fails the whole
|
|
27
|
+
create with a `422`** instead of silently succeeding with that key ignored. In 0.5.x, `--file`
|
|
28
|
+
never read anything but `url` from a JSON line — every other key was dropped client-side
|
|
29
|
+
before the request was ever built. As of 0.6.0 those keys ride to the server (`items` on
|
|
30
|
+
`bulk create` / `scrape bulk create`; inside `urls` on `fetch bulk create`), and `/v1/bulk` /
|
|
31
|
+
`/v1/scrape/bulk` reject any key their `.strict()` per-item schema doesn't recognize. A file
|
|
32
|
+
that worked in 0.5.x purely because its extra keys were quietly discarded may now `422` — see
|
|
33
|
+
each command's `--file` help for the keys actually honored on that endpoint.
|
|
34
|
+
- **`fetch bulk create --file` per-item keys outside `url`/`format`/`country`/`cache_ttl` now
|
|
35
|
+
fail the whole create with a `422`** instead of silently succeeding with that key ignored.
|
|
36
|
+
`/v1/fetch/bulk`'s per-item override schema (inside `urls[]`) is now `.strict()` on the
|
|
37
|
+
gateway (issue #2648), matching the `/v1/bulk` / `/v1/scrape/bulk` posture above. The
|
|
38
|
+
`--file` help text for `fetch bulk create` is now uniform with `bulk create` /
|
|
39
|
+
`scrape bulk create` and no longer enumerates keys — this entry is where the honored set
|
|
40
|
+
(`url`, `format`, `country`, `cache_ttl`) is documented instead.
|
|
41
|
+
|
|
42
|
+
### Added
|
|
43
|
+
|
|
44
|
+
- **Per-item bulk configuration on all three bulk creates** (`bulk create`, `scrape bulk create`,
|
|
45
|
+
`fetch bulk create`). A `--file` line may now be a JSON object carrying `url` plus per-item
|
|
46
|
+
fields instead of only a bare URL; such a file is sent as `items` on `/v1/bulk` and
|
|
47
|
+
`/v1/scrape/bulk`, and as a heterogeneous `urls` array on `/v1/fetch/bulk` (which has no
|
|
48
|
+
`items` property). A line whose only key is `url` still normalizes to a bare URL string, so a
|
|
49
|
+
file of url-only lines produces a byte-identical request to before. A JSON line with a missing
|
|
50
|
+
or non-string `url` is now rejected locally with a usage error (exit `1`) instead of being sent
|
|
51
|
+
to the server as a URL. (See Breaking, above, for the per-item strict-schema behavior change
|
|
52
|
+
this enables.)
|
|
53
|
+
- `--defaults`, `--webhook-secret` and `--metadata` on all three bulk creates, plus `--custom` on
|
|
54
|
+
`scrape bulk create` only (`/v1/bulk` and `/v1/fetch/bulk` have no top-level `custom`, so the
|
|
55
|
+
flag would be rejected or stripped there). `--defaults` is forwarded verbatim; the server owns
|
|
56
|
+
precedence and unknown-key handling.
|
|
57
|
+
- The full markdown-tuning surface on `fetch create`: `--markdown-query`, `--markdown-links`,
|
|
58
|
+
`--markdown-images`, `--with-links-summary`, `--with-images-summary`, `--markdown-compact`,
|
|
59
|
+
`--markdown-filter-images`, `--markdown-include-media`, `--markdown-include-warnings`,
|
|
60
|
+
`--markdown-include-stats`, and `--custom`.
|
|
61
|
+
- Webhook delivery and processing flags on `sitemap create`: `--webhook-url`, `--webhook-secret`,
|
|
62
|
+
`--cache-ttl`, `--compact`, `--process`, `--metadata`. Webhook-driven sitemap crawls were
|
|
63
|
+
previously unreachable from the shell.
|
|
64
|
+
- `--headers`, `--cookies` and `--metadata` on `screenshots create`.
|
|
65
|
+
- `--metadata` on `monitors create`, and `--metadata` + `--url` on `monitors update`. `--type`
|
|
66
|
+
remains deliberately absent from `update` — it is immutable server-side.
|
|
67
|
+
|
|
68
|
+
## [0.5.1] - 2026-07-28
|
|
69
|
+
|
|
70
|
+
### Fixed
|
|
71
|
+
|
|
72
|
+
- **Corrects the scope of the 0.5.0 "stalled body" fix claim.** `screenshots create`/`get -o`
|
|
73
|
+
downloads the `image_url` artifact bytes with a bare `globalThis.fetch` call
|
|
74
|
+
(`lib/download-artifact.ts`) that sat entirely outside the transport seam `sendWithRetry` owns —
|
|
75
|
+
no abort signal, no connection-error taxonomy. The 0.5.0 entry below ("The per-attempt request
|
|
76
|
+
timeout now covers the response body read... a server that answered with headers and then
|
|
77
|
+
stalled its body... hung the CLI forever", #2546) reads as if it covered every response body; it
|
|
78
|
+
only ever covered API-call bodies read through the JSON/raw transports. A stalled artifact host
|
|
79
|
+
(headers, then a body that never completes) still hung the CLI indefinitely until now. The
|
|
80
|
+
download path now shares the same per-attempt abort budget and `ByteKitConnectionError` timeout
|
|
81
|
+
taxonomy as the API transport (`createAttemptBudget` / `toConnectionError`, exported from
|
|
82
|
+
`client.ts` — no second timeout implementation), with the same 120 s per-attempt default and no
|
|
83
|
+
retry (artifact GETs are not the API's 429 domain). Partial-file policy: the response is
|
|
84
|
+
buffered fully in memory before any write, so an aborted download never leaves a
|
|
85
|
+
partial/truncated file at the `-o` destination. **User-visible consequence:** a legitimately
|
|
86
|
+
slow (but not stalled) large artifact download now hard-fails after 120 s instead of waiting
|
|
87
|
+
indefinitely, and there is currently no CLI flag to raise that budget for a single command.
|
|
88
|
+
(#2587)
|
|
89
|
+
- While in the file: the 429-retry path left its (unread) response body undrained before sleeping
|
|
90
|
+
and retrying — undici cannot reclaim the underlying socket for connection-pool reuse until a body
|
|
91
|
+
is fully read. The body is now drained with `response.text()` on that path, bounded by its own
|
|
92
|
+
short (2 s) cap independent of the 120 s per-attempt budget, so a 429 whose body stalls can no
|
|
93
|
+
longer delay the retry by anywhere near the full attempt budget (or, with the budget disabled,
|
|
94
|
+
forever). **Correction to an intermediate `Unreleased` draft of this same fix:** that draft's cap
|
|
95
|
+
only bounded how long the _retry_ waited — it resolved on schedule but never touched the
|
|
96
|
+
abandoned `response.text()` call itself, which kept reading against a live socket indefinitely.
|
|
97
|
+
Because every CLI success path exits via a soft `process.exitCode` (#2544) rather than
|
|
98
|
+
`process.exit()`, that orphaned read kept the whole process alive forever — strictly worse than
|
|
99
|
+
having no drain at all, and on the exact class of hang this ticket exists to eliminate. The cap
|
|
100
|
+
now also **aborts** the request when it fires, so the abandoned read is actually torn down, not
|
|
101
|
+
merely ignored; this holds even when the per-attempt timeout itself is disabled
|
|
102
|
+
(`timeoutMs <= 0`). **User-visible consequence:** a 429 with a stalled body still costs the retry
|
|
103
|
+
at most ~2 s, exactly as before, but the CLI process now reliably exits afterward instead of
|
|
104
|
+
hanging forever regardless of that visible latency. The same discard-drain on the
|
|
105
|
+
artifact-download's non-2xx path is removed outright rather than capped: that path throws
|
|
106
|
+
straight into `process.exit(1)`, so there is no subsequent request in the process for a reclaimed
|
|
107
|
+
socket to serve, and the drain there bought nothing. (#2587)
|
|
108
|
+
|
|
109
|
+
## [0.5.0] - 2026-07-24
|
|
110
|
+
|
|
111
|
+
### Note
|
|
112
|
+
|
|
113
|
+
- The `0.3.1` and `0.4.0` versions below document real, tagged changes but were
|
|
114
|
+
never published to npm — the registry's version history jumps straight from
|
|
115
|
+
`0.3.0` to `0.5.0`. Don't go looking for those tarballs; the changes they
|
|
116
|
+
describe ship for the first time in this release.
|
|
117
|
+
|
|
118
|
+
### Added
|
|
119
|
+
|
|
120
|
+
- Surface parity round 2 (#2545): every `x-sdk-scope: v0.1` endpoint the SDK
|
|
121
|
+
exposes now has a CLI command, and the high-value request fields are reachable
|
|
122
|
+
by flags. Each new flag follows the omit-when-absent rule — absent → the key is
|
|
123
|
+
left out of the request, so a server default is never materialized.
|
|
124
|
+
- **New command groups:** `usage get | daily | by-endpoint` (billing-period
|
|
125
|
+
usage reporting), `webhook-deliveries list | retry <id>` (inspect and requeue
|
|
126
|
+
deliveries), and `bulk cancel <id>` (stop an in-flight bulk job).
|
|
127
|
+
- **Monitor pause/resume:** `monitors update --status paused` / `--status active`.
|
|
128
|
+
`monitors create`/`update` also gain `--change-threshold` (float), `--notify-on`,
|
|
129
|
+
`--webhook-secret`, and the JSON-object flags `--webhook-headers`, `--options`,
|
|
130
|
+
`--scrape-options`. `--type` is **create-only** (a monitor's type is immutable).
|
|
131
|
+
- **`scrape create` multi-format `--format`:** repeatable AND comma-separated —
|
|
132
|
+
`--format raw_html --format markdown` and `--format raw_html,markdown` both send
|
|
133
|
+
`formats: ["raw_html","markdown"]`. Raw output (`--raw`/`-o`) requires exactly one
|
|
134
|
+
format. New scrape flags: `--cache-ttl`, `--markdown-query`, `--cookies`,
|
|
135
|
+
`--headers`, `--delay-ms`, `--events`, `--token-encoding`, `--custom`,
|
|
136
|
+
`--remove-base64-images` / `--no-remove-base64-images` (tri-state), and the
|
|
137
|
+
`--markdown-*` / `--with-*-summary` family.
|
|
138
|
+
- **`scrape get --raw` / `-o` / `--format`:** retrieve a completed queued scrape's
|
|
139
|
+
body from the shell (a still-queued job prints the id and exits `2`).
|
|
140
|
+
- **`screenshots create`:** `--block-ads`/`--no-block-ads`,
|
|
141
|
+
`--block-cookie-banners`/`--no-block-cookie-banners`, `--scroll`/`--no-scroll`
|
|
142
|
+
(tri-state pairs), `--wait-for-selector`, `--delay-ms`, `--dark-mode`,
|
|
143
|
+
`--device-scale-factor`, `--include-html`, `--country`, `--language`.
|
|
144
|
+
- **`fetch create` / `fetch get`:** `--country`, `--timeout-ms`, `--cache-ttl`
|
|
145
|
+
(both verbs); `--markdown-mode` (create only).
|
|
146
|
+
- **`sitemap create`:** `--strategy`, `--max-depth`, `--max-urls`.
|
|
147
|
+
|
|
148
|
+
### Fixed
|
|
149
|
+
|
|
150
|
+
- `--json` no longer emits the literal text `undefined` — which is not valid JSON —
|
|
151
|
+
for a response with no body. `JSON.stringify(undefined)` returns the _value_
|
|
152
|
+
`undefined`, so `bytekit monitors delete <id> --json | jq .` failed with a parse
|
|
153
|
+
error on every delete, and `--json -o <file>` on the same response crashed with
|
|
154
|
+
`ERR_INVALID_ARG_TYPE` instead of writing anything. Every `--json` code path now
|
|
155
|
+
serializes through one helper. (#2544)
|
|
156
|
+
- `scrape create --async --raw` (and the bare `-o` form) no longer risks losing the
|
|
157
|
+
`sc_` id it writes. The id went to stdout immediately followed by a hard
|
|
158
|
+
`process.exit(2)`; Node documents `process.stdout` as **asynchronous** when it
|
|
159
|
+
points at a pipe, so anything still queued at that moment is discarded — exactly
|
|
160
|
+
the `sc_id=$(bytekit scrape create … --async --raw)` capture the flag exists for.
|
|
161
|
+
The queued signal is now a soft `process.exitCode`, so the process ends naturally
|
|
162
|
+
once stdout has drained. The exit code is unchanged. (#2544)
|
|
163
|
+
- The per-attempt request timeout now covers the **response body read**, not just the
|
|
164
|
+
time to first byte of headers. The abort budget was armed for `fetch` and cleared the
|
|
165
|
+
moment headers arrived, after which reading the body was unbounded — so a server that
|
|
166
|
+
answered with headers and then stalled its body (including a non-2xx whose error body
|
|
167
|
+
never finished) hung the CLI forever. The budget now spans headers **and** body on both
|
|
168
|
+
transports (`request` and the `/v1/fetch` raw path), and a mid-body abort surfaces as
|
|
169
|
+
the usual `ByteKitConnectionError` with `code: 'timeout'`.
|
|
170
|
+
|
|
171
|
+
The default is **unchanged at 120 s**, and the budget is still **per attempt** — the
|
|
172
|
+
single 429 retry and its backoff sleep still get a fresh full budget. One consequence
|
|
173
|
+
worth planning for: a legitimately large body (a multi-MB `/v1/fetch` document) now
|
|
174
|
+
counts against that same 120 s, where previously only the wait for headers did. (#2546)
|
|
175
|
+
|
|
176
|
+
### Documented
|
|
177
|
+
|
|
178
|
+
- **Output contract:** with `--json`, a `204 No Content` response (e.g.
|
|
179
|
+
`monitors delete`) renders as JSON `null`. Human mode still prints nothing for
|
|
180
|
+
the same response. (#2544)
|
|
181
|
+
- **Exit codes** now have their own README section. In particular exit code `2` —
|
|
182
|
+
"accepted but still queued", emitted by `scrape create --async` in raw and human
|
|
183
|
+
modes — was previously documented nowhere despite being observable since 0.3.0.
|
|
184
|
+
It is deliberately neither `0` (a content-body success) nor `1` (an error). (#2544)
|
|
185
|
+
- Corrected two false README claims about `scrape` output: a bare `-o` implies
|
|
186
|
+
`--raw` and writes the requested format's **body** (it does not write "the same
|
|
187
|
+
text rendering the terminal would show"), and `--async --raw` prints the **bare
|
|
188
|
+
`sc_` id** on stdout with a note on stderr (it does not print the queued job
|
|
189
|
+
envelope). (#2544)
|
|
190
|
+
|
|
191
|
+
## [0.4.0] - 2026-07-23
|
|
192
|
+
|
|
193
|
+
### Removed (**BREAKING**)
|
|
194
|
+
|
|
195
|
+
- The entire `bytekit account api-keys` command group — `create`, `list`, `get`,
|
|
196
|
+
`update`, `reveal`, and `revoke`. Every one of these subcommands was
|
|
197
|
+
structurally unreachable: they target `/v1/account/api-keys*`, which the API
|
|
198
|
+
mounts behind a **Clerk dashboard session** and marks `x-internal` in the
|
|
199
|
+
OpenAPI spec. The CLI's only credential is a bearer `sk_` API key, which can
|
|
200
|
+
never satisfy a session verifier, so all six returned `401 unauthorized` in
|
|
201
|
+
every published version. Running one now exits `1` with
|
|
202
|
+
`error: unknown command 'api-keys'`.
|
|
203
|
+
|
|
204
|
+
Key management was deliberately NOT moved onto bearer auth. The reveal
|
|
205
|
+
operation returns the **live plaintext** of any key on the account, and there
|
|
206
|
+
is no per-key scope or permission model, so accepting API-key auth there would
|
|
207
|
+
turn any leaked key into a self-perpetuating credential factory.
|
|
208
|
+
|
|
209
|
+
**Replacement:** manage API keys from the dashboard at
|
|
210
|
+
[app.bytekit.com](https://app.bytekit.com). (#2536)
|
|
211
|
+
|
|
212
|
+
## [0.3.1] - 2026-07-22
|
|
213
|
+
|
|
214
|
+
### Fixed
|
|
215
|
+
|
|
216
|
+
- `bytekit fetch get` / `fetch create` print the fetched content again. **0.3.0's
|
|
217
|
+
claim that fetch "prints the raw body by default, or the `X-Fetch-*` metadata
|
|
218
|
+
as JSON with `--json`" was not true of the shipped artifact**: the CLI replaced
|
|
219
|
+
only `client.request`, but SDK 0.3.0 moved `/v1/fetch` onto `client.requestRaw`,
|
|
220
|
+
so the CLI's raw-body branch was unreachable and every fetch fell through to the
|
|
221
|
+
generic key/value printer — emitting `body: …`, a JSON map of every byte under
|
|
222
|
+
`bytes:`, and `headers: {}`. The CLI now overrides BOTH transport seams and
|
|
223
|
+
renders the SDK's real `FetchResult`. Binary content is byte-exact in every
|
|
224
|
+
mode. (#2535)
|
|
225
|
+
- Fetch commands honor the CLI's single 429 retry again. Riding the
|
|
226
|
+
un-overridden SDK transport also bypassed the retry, the per-attempt 120 s
|
|
227
|
+
timeout, and the connection-error taxonomy; all three now apply to fetch
|
|
228
|
+
exactly as they do to the JSON commands. (#2535)
|
|
229
|
+
|
|
230
|
+
### Added
|
|
231
|
+
|
|
232
|
+
- `-o, --output <file>` on `fetch get` / `fetch create`: writes the fetched body
|
|
233
|
+
bytes to a file, byte-for-byte (with `--json`, writes the `X-Fetch-*` metadata
|
|
234
|
+
object instead). (#2535)
|
|
235
|
+
|
|
11
236
|
## [0.3.0] - 2026-07-21
|
|
12
237
|
|
|
13
238
|
### Removed (**BREAKING**)
|
|
@@ -116,6 +341,11 @@ Additive, lower-risk release — no breaking changes to existing commands.
|
|
|
116
341
|
- Initial release of the ByteKit CLI, wrapping the ByteKit SDK as shell
|
|
117
342
|
subcommands, published under the original `rapidcrawl-cli` name.
|
|
118
343
|
|
|
344
|
+
[Unreleased]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
345
|
+
[0.5.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
346
|
+
[0.5.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
347
|
+
[0.4.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
348
|
+
[0.3.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
119
349
|
[0.3.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
120
350
|
[0.2.2]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
121
351
|
[0.2.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
package/README.md
CHANGED
|
@@ -66,6 +66,16 @@ export BYTEKIT_BASE_URL=http://localhost:3001
|
|
|
66
66
|
| `--base-url <url>` | `BYTEKIT_BASE_URL` | API base URL (default `https://api.bytekit.com`). |
|
|
67
67
|
| `--json` | — | Print the raw JSON response instead of the human-readable rendering (for `fetch`, prints the `X-Fetch-*` metadata instead of the body). |
|
|
68
68
|
|
|
69
|
+
`--json` always emits a document `JSON.parse` (or `jq`) accepts. A command whose
|
|
70
|
+
API response carries no body — a `204 No Content`, as `monitors delete` returns
|
|
71
|
+
— prints JSON `null`:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
bytekit monitors delete mon_123abc --json | jq . # → null
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Human mode (no `--json`) prints nothing at all for the same response.
|
|
78
|
+
|
|
69
79
|
## Commands
|
|
70
80
|
|
|
71
81
|
### `scrape` — scrape URL content
|
|
@@ -77,39 +87,106 @@ bytekit scrape bulk create --file urls.txt --webhook-url https://your.app/hooks/
|
|
|
77
87
|
```
|
|
78
88
|
|
|
79
89
|
`scrape create` also accepts the full `ScrapeRequest` surface, each flag omitted
|
|
80
|
-
from the request when absent (a boolean flag never sends `false
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
|
84
|
-
|
|
|
85
|
-
| `--
|
|
86
|
-
| `--
|
|
87
|
-
| `--
|
|
88
|
-
| `--
|
|
89
|
-
| `--
|
|
90
|
-
| `--
|
|
91
|
-
| `--
|
|
92
|
-
| `--
|
|
93
|
-
| `--
|
|
94
|
-
|
|
95
|
-
`--
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
90
|
+
from the request when absent (a boolean flag never sends `false`, so a server
|
|
91
|
+
default is never materialized):
|
|
92
|
+
|
|
93
|
+
| Flag | Effect |
|
|
94
|
+
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
95
|
+
| `--format <fmt>` | Output format. **Repeatable and comma-separated** — `--format raw_html --format markdown` and `--format raw_html,markdown` both send `formats: ["raw_html","markdown"]`. Omitted → server default. |
|
|
96
|
+
| `--country <cc>` | Route through a country (e.g. `us`, `de`). |
|
|
97
|
+
| `--timeout-ms <n>` | Per-request timeout in milliseconds. |
|
|
98
|
+
| `--cache-ttl <ttl>` | Cache freshness: `0`, `Nh` (hours), or `Nd` (days) — e.g. `48h`, `2d`, `0`. Omitted → server default. |
|
|
99
|
+
| `--markdown-mode <mode>` | Markdown extraction mode: `article`, `raw`, `llm`. |
|
|
100
|
+
| `--markdown-query <q>` | BM25 query for relevance-ranked markdown filtering. |
|
|
101
|
+
| `--cookies <json>` | Cookies as a JSON array, e.g. `'[{"name":"a","value":"b"}]'`. |
|
|
102
|
+
| `--headers <json>` | Extra request headers as a JSON object, e.g. `'{"X-A":"b"}'`. |
|
|
103
|
+
| `--delay-ms <n>` | Post-load delay before capture, in milliseconds. |
|
|
104
|
+
| `--events <csv>` | Webhook events (comma-separated): `queued`, `completed`, `failed`. |
|
|
105
|
+
| `--token-budget <n>` | Cap the markdown returned to `n` tokens. |
|
|
106
|
+
| `--token-encoding <enc>` | Tokenizer for `--token-budget`: `cl100k_base`, `o200k_base`. |
|
|
107
|
+
| `--custom <json>` | User payload echoed on the envelope, as a JSON object. |
|
|
108
|
+
| `--remove-base64-images` / `--no-remove-base64-images` | Tri-state: strip / preserve base64 `data:` images. Omitted → server default (strip). |
|
|
109
|
+
| `--markdown-links <mode>` | Link rendering: `inline`, `references`, `none`, `text`. |
|
|
110
|
+
| `--markdown-images <mode>` | Image retention: `inline`, `references`, `none`, `text`. |
|
|
111
|
+
| `--with-links-summary` | Append a Links footer to the markdown. |
|
|
112
|
+
| `--with-images-summary` | Append an Images footer to the markdown. |
|
|
113
|
+
| `--markdown-compact` | Collapse excessive whitespace. |
|
|
114
|
+
| `--markdown-filter-images` | Filter low-signal images. |
|
|
115
|
+
| `--markdown-include-media` | Return rich link/image objects and a tables array. |
|
|
116
|
+
| `--markdown-include-warnings` | Include markdown-pipeline and tag-filter warnings. |
|
|
117
|
+
| `--markdown-include-stats` | Include a top-level stats object (chars, tokens, blocks). |
|
|
118
|
+
| `--clean-markdown` | LLM-clean the markdown (billed 3x). |
|
|
119
|
+
| `--include-tags <csv>` | Comma-separated HTML tags to keep. |
|
|
120
|
+
| `--exclude-tags <csv>` | Comma-separated HTML tags to drop. |
|
|
121
|
+
| `--mobile` | Render with a mobile viewport/user-agent. |
|
|
122
|
+
| `--async` | Queue the scrape and return a job envelope immediately. |
|
|
123
|
+
| `--webhook-url <url>` | Webhook notified when an async scrape completes. |
|
|
124
|
+
|
|
125
|
+
`scrape get` accepts the same output flags as `scrape create` —
|
|
126
|
+
`--raw`, `-o <file>`, and `--format <fmt>` — so a job queued with `--async --raw`
|
|
127
|
+
can be finished raw once it completes:
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw) || true
|
|
131
|
+
bytekit scrape get "$sc_id" --raw # prints the body once completed
|
|
132
|
+
bytekit scrape get "$sc_id" --format markdown -o page.md
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
`--raw`/`-o` require **exactly one** format: with several `--format` values,
|
|
136
|
+
`scrape create --raw` is a usage error (raw mode emits a single body). A
|
|
137
|
+
still-queued `scrape get --raw` prints the bare id and exits [code `2`](#exit-codes),
|
|
138
|
+
same as `scrape create --async --raw`.
|
|
139
|
+
|
|
140
|
+
On `scrape`, a bare `-o <file>` implies `--raw`: the file receives the requested
|
|
141
|
+
format's body (so `-o page.md` holds markdown), not a JSON envelope. Add
|
|
142
|
+
`--json` to write the full envelope to the file instead. `--json` and `--raw`
|
|
143
|
+
together are a usage error.
|
|
144
|
+
|
|
145
|
+
`--async` combined with `--raw` (or a bare `-o`) has no format body to print
|
|
146
|
+
yet, so it writes the bare `sc_` id — and nothing else — to **stdout**, prints a
|
|
147
|
+
human "queued, poll it with…" note to **stderr**, and exits with
|
|
148
|
+
[code `2`](#exit-codes). The `-o` file is deliberately never created: a file
|
|
149
|
+
holding a bare id would masquerade as scraped content. This makes the handle
|
|
150
|
+
directly capturable:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw) || true
|
|
154
|
+
bytekit scrape get "$sc_id"
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
`--async --json` is outside that rule — it prints the full queued envelope and
|
|
158
|
+
exits `0`.
|
|
99
159
|
|
|
100
160
|
### `fetch` — raw, unprocessed response bodies
|
|
101
161
|
|
|
102
162
|
```bash
|
|
103
163
|
bytekit fetch create --url https://example.com
|
|
164
|
+
bytekit fetch get --url https://example.com --format markdown
|
|
165
|
+
bytekit fetch get --url https://example.com/logo.png -o logo.png
|
|
104
166
|
bytekit fetch bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
105
167
|
```
|
|
106
168
|
|
|
169
|
+
`fetch create` / `fetch get` print the fetched body to stdout verbatim — no JSON
|
|
170
|
+
wrapper, no appended newline, binary-safe. `--json` prints the `X-Fetch-*`
|
|
171
|
+
response metadata instead of the body. `-o <file>` writes the body bytes to a
|
|
172
|
+
file byte-for-byte (with `--json`, the metadata object goes to the file
|
|
173
|
+
instead). `fetch bulk` returns an ordinary JSON job envelope.
|
|
174
|
+
|
|
175
|
+
Both verbs accept `--country <cc>`, `--timeout-ms <n>`, and `--cache-ttl <ttl>`
|
|
176
|
+
(`0`, `Nh`, or `Nd`), each omitted when absent. `--markdown-mode`
|
|
177
|
+
(`article`, `raw`, `llm`) is `fetch create` only.
|
|
178
|
+
|
|
107
179
|
### `search` — ranked web search
|
|
108
180
|
|
|
109
181
|
```bash
|
|
110
182
|
bytekit search "ai web scraping" --limit 10 --country us
|
|
111
183
|
```
|
|
112
184
|
|
|
185
|
+
`search` also accepts `--type <type>` (`web` (default), `news`, or `images`),
|
|
186
|
+
`--language <code>` (two-letter language code, e.g. `en`, `fr`), and
|
|
187
|
+
`--date-range <window>` (recency filter: `any` (default), `hour`, `day`,
|
|
188
|
+
`week`, `month`, `year`). Each flag is omitted from the request when absent.
|
|
189
|
+
|
|
113
190
|
### `screenshots` — capture a page
|
|
114
191
|
|
|
115
192
|
```bash
|
|
@@ -124,6 +201,15 @@ bytekit screenshots get ss_123abc -o shot.png
|
|
|
124
201
|
(queue and return a job envelope). Each flag is omitted from the request when
|
|
125
202
|
absent.
|
|
126
203
|
|
|
204
|
+
It also accepts `--wait-for-selector <sel>`, `--delay-ms <n>`,
|
|
205
|
+
`--device-scale-factor` (`1` or `2`), `--dark-mode`, `--include-html`,
|
|
206
|
+
`--country <cc>`, `--language <lang>`, and three **tri-state** blocker pairs:
|
|
207
|
+
`--block-ads` / `--no-block-ads`, `--block-cookie-banners` /
|
|
208
|
+
`--no-block-cookie-banners`, and `--scroll` / `--no-scroll`. Each of these three
|
|
209
|
+
defaults to `true` server-side, so passing neither half omits the key entirely
|
|
210
|
+
(the server default stands) — the `--no-` half sends `false`, the bare flag sends
|
|
211
|
+
`true`.
|
|
212
|
+
|
|
127
213
|
`-o <file>` downloads the captured image bytes to that file (use `-o -` for
|
|
128
214
|
stdout). Because `-o` writes BINARY on `screenshots`, it cannot be combined with
|
|
129
215
|
`--json` (that would emit binary into a JSON pipeline) — pass one or the other.
|
|
@@ -136,10 +222,12 @@ image is not ready yet).
|
|
|
136
222
|
bytekit bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
137
223
|
bulk_id=bulk_123abc
|
|
138
224
|
bytekit bulk get "$bulk_id"
|
|
225
|
+
bytekit bulk cancel "$bulk_id"
|
|
139
226
|
bytekit bulk screenshots list "$bulk_id"
|
|
140
227
|
```
|
|
141
228
|
|
|
142
229
|
`urls.txt` is newline-delimited: one plain URL (or `{"url":"https://..."}`) per line.
|
|
230
|
+
`bulk cancel <id>` stops an in-flight job.
|
|
143
231
|
|
|
144
232
|
### `monitors` — watch a page for changes
|
|
145
233
|
|
|
@@ -147,7 +235,9 @@ bytekit bulk screenshots list "$bulk_id"
|
|
|
147
235
|
bytekit monitors create --url https://example.com/pricing --interval-type daily \
|
|
148
236
|
--webhook-url https://your.app/hooks/bytekit
|
|
149
237
|
bytekit monitors list --limit 25 --status active
|
|
150
|
-
bytekit monitors
|
|
238
|
+
bytekit monitors update mon_123abc --status paused # pause
|
|
239
|
+
bytekit monitors update mon_123abc --status active # resume
|
|
240
|
+
bytekit monitors captures list mon_123abc --limit 25
|
|
151
241
|
bytekit monitors delete mon_123abc
|
|
152
242
|
```
|
|
153
243
|
|
|
@@ -155,26 +245,94 @@ bytekit monitors delete mon_123abc
|
|
|
155
245
|
(`active`, `processing`, `paused`, `cancelled`, `suspended`) as query filters,
|
|
156
246
|
each omitted when absent.
|
|
157
247
|
|
|
248
|
+
`monitors captures list <id>` accepts `--limit <n>` and `--cursor <cursor>` to
|
|
249
|
+
page through a monitor's capture history — pass the response's `next_cursor`
|
|
250
|
+
back as `--cursor` to fetch the next page. Both flags are omitted when absent.
|
|
251
|
+
|
|
252
|
+
`monitors update` gains `--status` (`active` | `paused` — pause and resume from
|
|
253
|
+
the shell), `--change-threshold <pct>` (accepts a float), `--notify-on`
|
|
254
|
+
(`change`, `every`), `--webhook-secret <secret>`, and three JSON-object flags:
|
|
255
|
+
`--webhook-headers`, `--options`, and `--scrape-options` (e.g.
|
|
256
|
+
`--options '{"device":"mobile"}'`). `monitors create` accepts the same set plus
|
|
257
|
+
`--type` (`screenshot` | `scrape`). `--type` is **create-only** — a monitor's
|
|
258
|
+
type is immutable, so `monitors update` deliberately has no `--type`. Every flag
|
|
259
|
+
is omitted from the request when absent.
|
|
260
|
+
|
|
158
261
|
### `sitemap` — crawl a site's URL graph
|
|
159
262
|
|
|
160
263
|
```bash
|
|
161
|
-
bytekit sitemap create --url https://example.com
|
|
264
|
+
bytekit sitemap create --url https://example.com --strategy deep --max-depth 3 --max-urls 500
|
|
162
265
|
bytekit sitemap get sm_123abc
|
|
163
266
|
```
|
|
164
267
|
|
|
165
|
-
|
|
268
|
+
`sitemap create` accepts `--strategy` (`quick`, `standard`, `deep`),
|
|
269
|
+
`--max-depth <n>`, and `--max-urls <n>`, each omitted when absent.
|
|
270
|
+
|
|
271
|
+
### `account` — account details
|
|
166
272
|
|
|
167
273
|
```bash
|
|
168
274
|
bytekit account get
|
|
169
|
-
bytekit account api-keys list
|
|
170
|
-
bytekit account api-keys create --name "ci-key"
|
|
171
|
-
bytekit account api-keys reveal ak_123abc # masked by default
|
|
172
|
-
bytekit account api-keys reveal ak_123abc --show # full plaintext
|
|
173
275
|
```
|
|
174
276
|
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
277
|
+
API key management is not available from the CLI: it requires a signed-in
|
|
278
|
+
dashboard session, not a bearer API key. Create, rename, reveal, and revoke keys
|
|
279
|
+
at [app.bytekit.com](https://app.bytekit.com).
|
|
280
|
+
|
|
281
|
+
### `usage` — account usage and billing-period reporting
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
bytekit usage get
|
|
285
|
+
bytekit usage daily --from 2026-01-01 --to 2026-01-31
|
|
286
|
+
bytekit usage by-endpoint --api-key ak_123abc
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
`usage get` returns the current billing-period summary. `usage daily` and
|
|
290
|
+
`usage by-endpoint` accept `--from <date>`, `--to <date>` (YYYY-MM-DD), and
|
|
291
|
+
`--api-key <ak_id>`, each omitted when absent (default range: last 30 days).
|
|
292
|
+
|
|
293
|
+
### `webhook-deliveries` — inspect and retry webhook deliveries
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
bytekit webhook-deliveries list --status failed --limit 10
|
|
297
|
+
bytekit webhook-deliveries retry wh_123abc
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
`list` accepts `--limit <n>`, `--cursor <cursor>`, `--status`
|
|
301
|
+
(`pending`, `delivered`, `failed`, `exhausted`), and `--since <iso>`, each
|
|
302
|
+
omitted when absent. `retry <id>` requeues a failed or exhausted delivery.
|
|
303
|
+
|
|
304
|
+
## Exit codes
|
|
305
|
+
|
|
306
|
+
| Code | Meaning |
|
|
307
|
+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
308
|
+
| `0` | Success. |
|
|
309
|
+
| `1` | Usage error, or an API/connection error. |
|
|
310
|
+
| `2` | The job was accepted but is still **queued**, so there is no content body or artifact to emit. Applies to `scrape create --async` with `--raw` or a bare `-o` (which writes the bare `sc_` id to stdout) and in human mode, and to `screenshots create --async` in human mode (including with `-o`, which deliberately writes no file). Not a failure: the job was accepted and billed. Poll it with `bytekit scrape get <id>` / `bytekit screenshots get <id>`. **`--json` keeps exit `0` in both cases** — the queued envelope is a complete machine-readable payload. |
|
|
311
|
+
|
|
312
|
+
The stderr message for exit code `1` has two shapes, depending on where the
|
|
313
|
+
error originates:
|
|
314
|
+
|
|
315
|
+
- **App-level errors** (a rejected API call, a connection failure, a missing
|
|
316
|
+
key) are always prefixed `Error: ` (uppercase) — e.g.
|
|
317
|
+
`Error: API key required. Pass --key or set BYTEKIT_API_KEY.`
|
|
318
|
+
- **Commander-level usage errors** (an unknown command, a missing required
|
|
319
|
+
argument, an unrecognized flag) come from the `commander` library itself and
|
|
320
|
+
use its own lowercase `error: ` prefix — e.g. `error: unknown command 'bogus'`.
|
|
321
|
+
|
|
322
|
+
Scripts that grep stderr for a failure reason should match case-insensitively,
|
|
323
|
+
or handle both prefixes.
|
|
324
|
+
|
|
325
|
+
Code `2` is deliberately distinct from both `0` and `1` so a script can branch
|
|
326
|
+
on "queued" without parsing stderr:
|
|
327
|
+
|
|
328
|
+
```bash
|
|
329
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw)
|
|
330
|
+
case $? in
|
|
331
|
+
0) echo "content: $sc_id" ;;
|
|
332
|
+
2) echo "queued as $sc_id — poll it" ;;
|
|
333
|
+
*) exit 1 ;;
|
|
334
|
+
esac
|
|
335
|
+
```
|
|
178
336
|
|
|
179
337
|
## Documentation
|
|
180
338
|
|