@hunt-labs/bytekit-cli 0.2.2 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +172 -0
- package/README.md +193 -15
- package/dist/client.d.ts +39 -18
- package/dist/client.js +230 -39
- package/dist/commands/account.d.ts +11 -0
- package/dist/commands/account.js +12 -80
- package/dist/commands/bulk.js +14 -0
- package/dist/commands/fetch.d.ts +14 -0
- package/dist/commands/fetch.js +45 -6
- package/dist/commands/monitors.js +83 -3
- package/dist/commands/scrape.js +190 -10
- package/dist/commands/screenshots.d.ts +12 -0
- package/dist/commands/screenshots.js +146 -6
- package/dist/commands/search.js +5 -1
- package/dist/commands/sitemap.js +17 -1
- package/dist/commands/usage.d.ts +13 -0
- package/dist/commands/usage.js +81 -0
- package/dist/commands/webhook-deliveries.d.ts +11 -0
- package/dist/commands/webhook-deliveries.js +59 -0
- package/dist/index.js +4 -0
- package/dist/lib/download-artifact.d.ts +13 -0
- package/dist/lib/download-artifact.js +25 -0
- package/dist/lib/parse-flags.d.ts +46 -0
- package/dist/lib/parse-flags.js +90 -0
- package/dist/output.d.ts +88 -0
- package/dist/output.js +188 -7
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,176 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
This file records only the delta between versions. For how to use the package
|
|
9
9
|
today, see [README.md](./README.md), which remains the authoritative usage guide.
|
|
10
10
|
|
|
11
|
+
## [0.5.0] - 2026-07-24
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- Surface parity round 2 (#2545): every `x-sdk-scope: v0.1` endpoint the SDK
|
|
16
|
+
exposes now has a CLI command, and the high-value request fields are reachable
|
|
17
|
+
by flags. Each new flag follows the omit-when-absent rule — absent → the key is
|
|
18
|
+
left out of the request, so a server default is never materialized.
|
|
19
|
+
- **New command groups:** `usage get | daily | by-endpoint` (billing-period
|
|
20
|
+
usage reporting), `webhook-deliveries list | retry <id>` (inspect and requeue
|
|
21
|
+
deliveries), and `bulk cancel <id>` (stop an in-flight bulk job).
|
|
22
|
+
- **Monitor pause/resume:** `monitors update --status paused` / `--status active`.
|
|
23
|
+
`monitors create`/`update` also gain `--change-threshold` (float), `--notify-on`,
|
|
24
|
+
`--webhook-secret`, and the JSON-object flags `--webhook-headers`, `--options`,
|
|
25
|
+
`--scrape-options`. `--type` is **create-only** (a monitor's type is immutable).
|
|
26
|
+
- **`scrape create` multi-format `--format`:** repeatable AND comma-separated —
|
|
27
|
+
`--format raw_html --format markdown` and `--format raw_html,markdown` both send
|
|
28
|
+
`formats: ["raw_html","markdown"]`. Raw output (`--raw`/`-o`) requires exactly one
|
|
29
|
+
format. New scrape flags: `--cache-ttl`, `--markdown-query`, `--cookies`,
|
|
30
|
+
`--headers`, `--delay-ms`, `--events`, `--token-encoding`, `--custom`,
|
|
31
|
+
`--remove-base64-images` / `--no-remove-base64-images` (tri-state), and the
|
|
32
|
+
`--markdown-*` / `--with-*-summary` family.
|
|
33
|
+
- **`scrape get --raw` / `-o` / `--format`:** retrieve a completed queued scrape's
|
|
34
|
+
body from the shell (a still-queued job prints the id and exits `2`).
|
|
35
|
+
- **`screenshots create`:** `--block-ads`/`--no-block-ads`,
|
|
36
|
+
`--block-cookie-banners`/`--no-block-cookie-banners`, `--scroll`/`--no-scroll`
|
|
37
|
+
(tri-state pairs), `--wait-for-selector`, `--delay-ms`, `--dark-mode`,
|
|
38
|
+
`--device-scale-factor`, `--include-html`, `--country`, `--language`.
|
|
39
|
+
- **`fetch create` / `fetch get`:** `--country`, `--timeout-ms`, `--cache-ttl`
|
|
40
|
+
(both verbs); `--markdown-mode` (create only).
|
|
41
|
+
- **`sitemap create`:** `--strategy`, `--max-depth`, `--max-urls`.
|
|
42
|
+
|
|
43
|
+
### Fixed
|
|
44
|
+
|
|
45
|
+
- `--json` no longer emits the literal text `undefined` — which is not valid JSON —
|
|
46
|
+
for a response with no body. `JSON.stringify(undefined)` returns the _value_
|
|
47
|
+
`undefined`, so `bytekit monitors delete <id> --json | jq .` failed with a parse
|
|
48
|
+
error on every delete, and `--json -o <file>` on the same response crashed with
|
|
49
|
+
`ERR_INVALID_ARG_TYPE` instead of writing anything. Every `--json` code path now
|
|
50
|
+
serializes through one helper. (#2544)
|
|
51
|
+
- `scrape create --async --raw` (and the bare `-o` form) no longer risks losing the
|
|
52
|
+
`sc_` id it writes. The id went to stdout immediately followed by a hard
|
|
53
|
+
`process.exit(2)`; Node documents `process.stdout` as **asynchronous** when it
|
|
54
|
+
points at a pipe, so anything still queued at that moment is discarded — exactly
|
|
55
|
+
the `sc_id=$(bytekit scrape create … --async --raw)` capture the flag exists for.
|
|
56
|
+
The queued signal is now a soft `process.exitCode`, so the process ends naturally
|
|
57
|
+
once stdout has drained. The exit code is unchanged. (#2544)
|
|
58
|
+
- The per-attempt request timeout now covers the **response body read**, not just the
|
|
59
|
+
time to first byte of headers. The abort budget was armed for `fetch` and cleared the
|
|
60
|
+
moment headers arrived, after which reading the body was unbounded — so a server that
|
|
61
|
+
answered with headers and then stalled its body (including a non-2xx whose error body
|
|
62
|
+
never finished) hung the CLI forever. The budget now spans headers **and** body on both
|
|
63
|
+
transports (`request` and the `/v1/fetch` raw path), and a mid-body abort surfaces as
|
|
64
|
+
the usual `ByteKitConnectionError` with `code: 'timeout'`.
|
|
65
|
+
|
|
66
|
+
The default is **unchanged at 120 s**, and the budget is still **per attempt** — the
|
|
67
|
+
single 429 retry and its backoff sleep still get a fresh full budget. One consequence
|
|
68
|
+
worth planning for: a legitimately large body (a multi-MB `/v1/fetch` document) now
|
|
69
|
+
counts against that same 120 s, where previously only the wait for headers did. (#2546)
|
|
70
|
+
|
|
71
|
+
### Documented
|
|
72
|
+
|
|
73
|
+
- **Output contract:** with `--json`, a `204 No Content` response (e.g.
|
|
74
|
+
`monitors delete`) renders as JSON `null`. Human mode still prints nothing for
|
|
75
|
+
the same response. (#2544)
|
|
76
|
+
- **Exit codes** now have their own README section. In particular exit code `2` —
|
|
77
|
+
"accepted but still queued", emitted by `scrape create --async` in raw and human
|
|
78
|
+
modes — was previously documented nowhere despite being observable since 0.3.0.
|
|
79
|
+
It is deliberately neither `0` (a content-body success) nor `1` (an error). (#2544)
|
|
80
|
+
- Corrected two false README claims about `scrape` output: a bare `-o` implies
|
|
81
|
+
`--raw` and writes the requested format's **body** (it does not write "the same
|
|
82
|
+
text rendering the terminal would show"), and `--async --raw` prints the **bare
|
|
83
|
+
`sc_` id** on stdout with a note on stderr (it does not print the queued job
|
|
84
|
+
envelope). (#2544)
|
|
85
|
+
|
|
86
|
+
## [0.4.0] - 2026-07-23
|
|
87
|
+
|
|
88
|
+
### Removed (**BREAKING**)
|
|
89
|
+
|
|
90
|
+
- The entire `bytekit account api-keys` command group — `create`, `list`, `get`,
|
|
91
|
+
`update`, `reveal`, and `revoke`. Every one of these subcommands was
|
|
92
|
+
structurally unreachable: they target `/v1/account/api-keys*`, which the API
|
|
93
|
+
mounts behind a **Clerk dashboard session** and marks `x-internal` in the
|
|
94
|
+
OpenAPI spec. The CLI's only credential is a bearer `sk_` API key, which can
|
|
95
|
+
never satisfy a session verifier, so all six returned `401 unauthorized` in
|
|
96
|
+
every published version. Running one now exits `1` with
|
|
97
|
+
`error: unknown command 'api-keys'`.
|
|
98
|
+
|
|
99
|
+
Key management was deliberately NOT moved onto bearer auth. The reveal
|
|
100
|
+
operation returns the **live plaintext** of any key on the account, and there
|
|
101
|
+
is no per-key scope or permission model, so accepting API-key auth there would
|
|
102
|
+
turn any leaked key into a self-perpetuating credential factory.
|
|
103
|
+
|
|
104
|
+
**Replacement:** manage API keys from the dashboard at
|
|
105
|
+
[app.bytekit.com](https://app.bytekit.com). (#2536)
|
|
106
|
+
|
|
107
|
+
## [0.3.1] - 2026-07-22
|
|
108
|
+
|
|
109
|
+
### Fixed
|
|
110
|
+
|
|
111
|
+
- `bytekit fetch get` / `fetch create` print the fetched content again. **0.3.0's
|
|
112
|
+
claim that fetch "prints the raw body by default, or the `X-Fetch-*` metadata
|
|
113
|
+
as JSON with `--json`" was not true of the shipped artifact**: the CLI replaced
|
|
114
|
+
only `client.request`, but SDK 0.3.0 moved `/v1/fetch` onto `client.requestRaw`,
|
|
115
|
+
so the CLI's raw-body branch was unreachable and every fetch fell through to the
|
|
116
|
+
generic key/value printer — emitting `body: …`, a JSON map of every byte under
|
|
117
|
+
`bytes:`, and `headers: {}`. The CLI now overrides BOTH transport seams and
|
|
118
|
+
renders the SDK's real `FetchResult`. Binary content is byte-exact in every
|
|
119
|
+
mode. (#2535)
|
|
120
|
+
- Fetch commands honor the CLI's single 429 retry again. Riding the
|
|
121
|
+
un-overridden SDK transport also bypassed the retry, the per-attempt 120 s
|
|
122
|
+
timeout, and the connection-error taxonomy; all three now apply to fetch
|
|
123
|
+
exactly as they do to the JSON commands. (#2535)
|
|
124
|
+
|
|
125
|
+
### Added
|
|
126
|
+
|
|
127
|
+
- `-o, --output <file>` on `fetch get` / `fetch create`: writes the fetched body
|
|
128
|
+
bytes to a file, byte-for-byte (with `--json`, writes the `X-Fetch-*` metadata
|
|
129
|
+
object instead). (#2535)
|
|
130
|
+
|
|
131
|
+
## [0.3.0] - 2026-07-21
|
|
132
|
+
|
|
133
|
+
### Removed (**BREAKING**)
|
|
134
|
+
|
|
135
|
+
- The legacy `RAPIDCRAWL_API_KEY` and `RAPIDCRAWL_BASE_URL` environment-variable
|
|
136
|
+
fallbacks (kept as a one-minor-version bridge in #2421) are removed. Use
|
|
137
|
+
`BYTEKIT_API_KEY` and `BYTEKIT_BASE_URL` (or the `--key` / `--base-url` flags).
|
|
138
|
+
(#2473)
|
|
139
|
+
|
|
140
|
+
### Fixed
|
|
141
|
+
|
|
142
|
+
- `bytekit fetch get` / `fetch create` no longer crash on non-JSON bodies. The
|
|
143
|
+
transport JSON-parsed every 2xx response, but `/v1/fetch` returns the fetched
|
|
144
|
+
content AS the body. The fetch command now prints the raw body by default, or
|
|
145
|
+
the `X-Fetch-*` metadata as JSON with `--json`; `fetch bulk` and every other
|
|
146
|
+
command still parse JSON. (#2467)
|
|
147
|
+
- `204 No Content` / empty-body responses (e.g. `monitors delete`,
|
|
148
|
+
`account api-keys revoke`) no longer throw a JSON-parse error, including after
|
|
149
|
+
a 429 retry. (#2467, #2468)
|
|
150
|
+
- `search --type images` no longer prints an `undefined` snippet line for
|
|
151
|
+
results that omit the field. (#2479)
|
|
152
|
+
|
|
153
|
+
### Added
|
|
154
|
+
|
|
155
|
+
- `scrape create` accepts the full `ScrapeRequest` surface: `--country`,
|
|
156
|
+
`--timeout-ms`, `--markdown-mode`, `--async`, `--webhook-url`,
|
|
157
|
+
`--token-budget`, `--clean-markdown`, `--include-tags`, `--exclude-tags`,
|
|
158
|
+
`--mobile`. Each flag is omitted from the request when absent. (#2480)
|
|
159
|
+
- `screenshots create` accepts `--device`, `--viewport WxH`, `--format`,
|
|
160
|
+
`--quality`, `--wait-until`, `--clip x,y,w,h`, and `--async`; `-o <file>`
|
|
161
|
+
on `screenshots create`/`get` downloads the captured image bytes (`-o -` for
|
|
162
|
+
stdout). `-o` cannot be combined with `--json` on screenshots. (#2480)
|
|
163
|
+
- `monitors list` accepts `--limit`, `--cursor`, and `--status` query
|
|
164
|
+
filters. (#2480)
|
|
165
|
+
- `account api-keys reveal <ak_id>`: prints the key masked by default
|
|
166
|
+
(`sk_live_…2345`); pass `--show` for the full plaintext. (#2480)
|
|
167
|
+
- Per-attempt transport timeout (120s, mirroring the SDK default): a hung
|
|
168
|
+
request now fails with `code: 'timeout'` instead of hanging forever. Network
|
|
169
|
+
failures are mapped onto the SDK's connection-error taxonomy
|
|
170
|
+
(`timeout` / `aborted` / `connection`), and error output renders the API or
|
|
171
|
+
connection error code. (#2479)
|
|
172
|
+
- A queued async job (HTTP 202) under `--raw` prints the job envelope with its
|
|
173
|
+
`sc_` id instead of an empty body. (#2479)
|
|
174
|
+
|
|
175
|
+
### Changed (**BREAKING**)
|
|
176
|
+
|
|
177
|
+
- `fetch --format` is validated locally against the spec union
|
|
178
|
+
`'markdown' | 'html'`; off-spec values are rejected with a clear error before
|
|
179
|
+
a request is sent (previously forwarded as-is). (#2475)
|
|
180
|
+
|
|
11
181
|
## [0.2.2] - 2026-07-16
|
|
12
182
|
|
|
13
183
|
### Changed
|
|
@@ -66,6 +236,8 @@ Additive, lower-risk release — no breaking changes to existing commands.
|
|
|
66
236
|
- Initial release of the ByteKit CLI, wrapping the ByteKit SDK as shell
|
|
67
237
|
subcommands, published under the original `rapidcrawl-cli` name.
|
|
68
238
|
|
|
239
|
+
[0.4.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
240
|
+
[0.3.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
69
241
|
[0.2.2]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
70
242
|
[0.2.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
|
71
243
|
[0.2.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
|
package/README.md
CHANGED
|
@@ -43,8 +43,7 @@ over the environment variable):
|
|
|
43
43
|
bytekit --key sk_live_xxx scrape create --url https://example.com
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
Precedence is `--key` flag → `BYTEKIT_API_KEY
|
|
47
|
-
fallback, deprecated — kept for existing scripts, will eventually be removed).
|
|
46
|
+
Precedence is `--key` flag → `BYTEKIT_API_KEY`.
|
|
48
47
|
|
|
49
48
|
Without a key, the CLI exits with `Error: API key required. Pass --key or set BYTEKIT_API_KEY.`
|
|
50
49
|
|
|
@@ -52,8 +51,7 @@ Without a key, the CLI exits with `Error: API key required. Pass --key or set BY
|
|
|
52
51
|
|
|
53
52
|
Requests go to `https://api.bytekit.com` by default. Point the CLI at staging, a
|
|
54
53
|
local gateway, or a proxy with the global `--base-url` flag or the
|
|
55
|
-
`BYTEKIT_BASE_URL` environment variable (the flag wins
|
|
56
|
-
a deprecated legacy fallback):
|
|
54
|
+
`BYTEKIT_BASE_URL` environment variable (the flag wins):
|
|
57
55
|
|
|
58
56
|
```bash
|
|
59
57
|
bytekit --base-url https://api-stg.bytekit.com scrape create --url https://example.com
|
|
@@ -62,11 +60,21 @@ export BYTEKIT_BASE_URL=http://localhost:3001
|
|
|
62
60
|
|
|
63
61
|
## Global options
|
|
64
62
|
|
|
65
|
-
| Flag | Env var
|
|
66
|
-
| ------------------ |
|
|
67
|
-
| `--key <key>` | `BYTEKIT_API_KEY`
|
|
68
|
-
| `--base-url <url>` | `BYTEKIT_BASE_URL`
|
|
69
|
-
| `--json` | —
|
|
63
|
+
| Flag | Env var | Purpose |
|
|
64
|
+
| ------------------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
65
|
+
| `--key <key>` | `BYTEKIT_API_KEY` | API key used for every request. |
|
|
66
|
+
| `--base-url <url>` | `BYTEKIT_BASE_URL` | API base URL (default `https://api.bytekit.com`). |
|
|
67
|
+
| `--json` | — | Print the raw JSON response instead of the human-readable rendering (for `fetch`, prints the `X-Fetch-*` metadata instead of the body). |
|
|
68
|
+
|
|
69
|
+
`--json` always emits a document `JSON.parse` (or `jq`) accepts. A command whose
|
|
70
|
+
API response carries no body — a `204 No Content`, as `monitors delete` returns
|
|
71
|
+
— prints JSON `null`:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
bytekit monitors delete mon_123abc --json | jq . # → null
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Human mode (no `--json`) prints nothing at all for the same response.
|
|
70
78
|
|
|
71
79
|
## Commands
|
|
72
80
|
|
|
@@ -78,13 +86,96 @@ bytekit scrape get sc_123abc
|
|
|
78
86
|
bytekit scrape bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
79
87
|
```
|
|
80
88
|
|
|
89
|
+
`scrape create` also accepts the full `ScrapeRequest` surface, each flag omitted
|
|
90
|
+
from the request when absent (a boolean flag never sends `false`, so a server
|
|
91
|
+
default is never materialized):
|
|
92
|
+
|
|
93
|
+
| Flag | Effect |
|
|
94
|
+
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
95
|
+
| `--format <fmt>` | Output format. **Repeatable and comma-separated** — `--format raw_html --format markdown` and `--format raw_html,markdown` both send `formats: ["raw_html","markdown"]`. Omitted → server default. |
|
|
96
|
+
| `--country <cc>` | Route through a country (e.g. `us`, `de`). |
|
|
97
|
+
| `--timeout-ms <n>` | Per-request timeout in milliseconds. |
|
|
98
|
+
| `--cache-ttl <ttl>` | Cache freshness: `0`, `Nh` (hours), or `Nd` (days) — e.g. `48h`, `2d`, `0`. Omitted → server default. |
|
|
99
|
+
| `--markdown-mode <mode>` | Markdown extraction mode: `article`, `raw`, `llm`. |
|
|
100
|
+
| `--markdown-query <q>` | BM25 query for relevance-ranked markdown filtering. |
|
|
101
|
+
| `--cookies <json>` | Cookies as a JSON array, e.g. `'[{"name":"a","value":"b"}]'`. |
|
|
102
|
+
| `--headers <json>` | Extra request headers as a JSON object, e.g. `'{"X-A":"b"}'`. |
|
|
103
|
+
| `--delay-ms <n>` | Post-load delay before capture, in milliseconds. |
|
|
104
|
+
| `--events <csv>` | Webhook events (comma-separated): `queued`, `completed`, `failed`. |
|
|
105
|
+
| `--token-budget <n>` | Cap the markdown returned to `n` tokens. |
|
|
106
|
+
| `--token-encoding <enc>` | Tokenizer for `--token-budget`: `cl100k_base`, `o200k_base`. |
|
|
107
|
+
| `--custom <json>` | User payload echoed on the envelope, as a JSON object. |
|
|
108
|
+
| `--remove-base64-images` / `--no-remove-base64-images` | Tri-state: strip / preserve base64 `data:` images. Omitted → server default (strip). |
|
|
109
|
+
| `--markdown-links <mode>` | Link rendering: `inline`, `references`, `none`, `text`. |
|
|
110
|
+
| `--markdown-images <mode>` | Image retention: `inline`, `references`, `none`, `text`. |
|
|
111
|
+
| `--with-links-summary` | Append a Links footer to the markdown. |
|
|
112
|
+
| `--with-images-summary` | Append an Images footer to the markdown. |
|
|
113
|
+
| `--markdown-compact` | Collapse excessive whitespace. |
|
|
114
|
+
| `--markdown-filter-images` | Filter low-signal images. |
|
|
115
|
+
| `--markdown-include-media` | Return rich link/image objects and a tables array. |
|
|
116
|
+
| `--markdown-include-warnings` | Include markdown-pipeline and tag-filter warnings. |
|
|
117
|
+
| `--markdown-include-stats` | Include a top-level stats object (chars, tokens, blocks). |
|
|
118
|
+
| `--clean-markdown` | LLM-clean the markdown (billed 3x). |
|
|
119
|
+
| `--include-tags <csv>` | Comma-separated HTML tags to keep. |
|
|
120
|
+
| `--exclude-tags <csv>` | Comma-separated HTML tags to drop. |
|
|
121
|
+
| `--mobile` | Render with a mobile viewport/user-agent. |
|
|
122
|
+
| `--async` | Queue the scrape and return a job envelope immediately. |
|
|
123
|
+
| `--webhook-url <url>` | Webhook notified when an async scrape completes. |
|
|
124
|
+
|
|
125
|
+
`scrape get` accepts the same output flags as `scrape create` —
|
|
126
|
+
`--raw`, `-o <file>`, and `--format <fmt>` — so a job queued with `--async --raw`
|
|
127
|
+
can be finished raw once it completes:
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw) || true
|
|
131
|
+
bytekit scrape get "$sc_id" --raw # prints the body once completed
|
|
132
|
+
bytekit scrape get "$sc_id" --format markdown -o page.md
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
`--raw`/`-o` require **exactly one** format: with several `--format` values,
|
|
136
|
+
`scrape create --raw` is a usage error (raw mode emits a single body). A
|
|
137
|
+
still-queued `scrape get --raw` prints the bare id and exits [code `2`](#exit-codes),
|
|
138
|
+
same as `scrape create --async --raw`.
|
|
139
|
+
|
|
140
|
+
On `scrape`, a bare `-o <file>` implies `--raw`: the file receives the requested
|
|
141
|
+
format's body (so `-o page.md` holds markdown), not a JSON envelope. Add
|
|
142
|
+
`--json` to write the full envelope to the file instead. `--json` and `--raw`
|
|
143
|
+
together are a usage error.
|
|
144
|
+
|
|
145
|
+
`--async` combined with `--raw` (or a bare `-o`) has no format body to print
|
|
146
|
+
yet, so it writes the bare `sc_` id — and nothing else — to **stdout**, prints a
|
|
147
|
+
human "queued, poll it with…" note to **stderr**, and exits with
|
|
148
|
+
[code `2`](#exit-codes). The `-o` file is deliberately never created: a file
|
|
149
|
+
holding a bare id would masquerade as scraped content. This makes the handle
|
|
150
|
+
directly capturable:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw) || true
|
|
154
|
+
bytekit scrape get "$sc_id"
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
`--async --json` is outside that rule — it prints the full queued envelope and
|
|
158
|
+
exits `0`.
|
|
159
|
+
|
|
81
160
|
### `fetch` — raw, unprocessed response bodies
|
|
82
161
|
|
|
83
162
|
```bash
|
|
84
163
|
bytekit fetch create --url https://example.com
|
|
164
|
+
bytekit fetch get --url https://example.com --format markdown
|
|
165
|
+
bytekit fetch get --url https://example.com/logo.png -o logo.png
|
|
85
166
|
bytekit fetch bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
86
167
|
```
|
|
87
168
|
|
|
169
|
+
`fetch create` / `fetch get` print the fetched body to stdout verbatim — no JSON
|
|
170
|
+
wrapper, no appended newline, binary-safe. `--json` prints the `X-Fetch-*`
|
|
171
|
+
response metadata instead of the body. `-o <file>` writes the body bytes to a
|
|
172
|
+
file byte-for-byte (with `--json`, the metadata object goes to the file
|
|
173
|
+
instead). `fetch bulk` returns an ordinary JSON job envelope.
|
|
174
|
+
|
|
175
|
+
Both verbs accept `--country <cc>`, `--timeout-ms <n>`, and `--cache-ttl <ttl>`
|
|
176
|
+
(`0`, `Nh`, or `Nd`), each omitted when absent. `--markdown-mode`
|
|
177
|
+
(`article`, `raw`, `llm`) is `fetch create` only.
|
|
178
|
+
|
|
88
179
|
### `search` — ranked web search
|
|
89
180
|
|
|
90
181
|
```bash
|
|
@@ -95,43 +186,130 @@ bytekit search "ai web scraping" --limit 10 --country us
|
|
|
95
186
|
|
|
96
187
|
```bash
|
|
97
188
|
bytekit screenshots create --url https://example.com --full-page
|
|
98
|
-
bytekit screenshots
|
|
189
|
+
bytekit screenshots create --url https://example.com --device mobile --format png -o shot.png
|
|
190
|
+
bytekit screenshots get ss_123abc -o shot.png
|
|
99
191
|
```
|
|
100
192
|
|
|
193
|
+
`screenshots create` accepts `--device` (`desktop`, `mobile`), `--viewport WxH`
|
|
194
|
+
(e.g. `1280x720`), `--format` (`jpeg`, `png`), `--quality <n>`, `--wait-until`
|
|
195
|
+
(`load`, `domcontentloaded`, `networkidle`), `--clip x,y,w,h`, and `--async`
|
|
196
|
+
(queue and return a job envelope). Each flag is omitted from the request when
|
|
197
|
+
absent.
|
|
198
|
+
|
|
199
|
+
It also accepts `--wait-for-selector <sel>`, `--delay-ms <n>`,
|
|
200
|
+
`--device-scale-factor` (`1` or `2`), `--dark-mode`, `--include-html`,
|
|
201
|
+
`--country <cc>`, `--language <lang>`, and three **tri-state** blocker pairs:
|
|
202
|
+
`--block-ads` / `--no-block-ads`, `--block-cookie-banners` /
|
|
203
|
+
`--no-block-cookie-banners`, and `--scroll` / `--no-scroll`. Each of these three
|
|
204
|
+
defaults to `true` server-side, so passing neither half omits the key entirely
|
|
205
|
+
(the server default stands) — the `--no-` half sends `false`, the bare flag sends
|
|
206
|
+
`true`.
|
|
207
|
+
|
|
208
|
+
`-o <file>` downloads the captured image bytes to that file (use `-o -` for
|
|
209
|
+
stdout). Because `-o` writes BINARY on `screenshots`, it cannot be combined with
|
|
210
|
+
`--json` (that would emit binary into a JSON pipeline) — pass one or the other.
|
|
211
|
+
`--async` + `-o` prints the queued job envelope and skips the download (the
|
|
212
|
+
image is not ready yet).
|
|
213
|
+
|
|
101
214
|
### `bulk` — bulk screenshot jobs
|
|
102
215
|
|
|
103
216
|
```bash
|
|
104
217
|
bytekit bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
105
218
|
bulk_id=bulk_123abc
|
|
106
219
|
bytekit bulk get "$bulk_id"
|
|
220
|
+
bytekit bulk cancel "$bulk_id"
|
|
107
221
|
bytekit bulk screenshots list "$bulk_id"
|
|
108
222
|
```
|
|
109
223
|
|
|
110
224
|
`urls.txt` is newline-delimited: one plain URL (or `{"url":"https://..."}`) per line.
|
|
225
|
+
`bulk cancel <id>` stops an in-flight job.
|
|
111
226
|
|
|
112
227
|
### `monitors` — watch a page for changes
|
|
113
228
|
|
|
114
229
|
```bash
|
|
115
230
|
bytekit monitors create --url https://example.com/pricing --interval-type daily \
|
|
116
231
|
--webhook-url https://your.app/hooks/bytekit
|
|
117
|
-
bytekit monitors list
|
|
232
|
+
bytekit monitors list --limit 25 --status active
|
|
233
|
+
bytekit monitors update mon_123abc --status paused # pause
|
|
234
|
+
bytekit monitors update mon_123abc --status active # resume
|
|
118
235
|
bytekit monitors captures list mon_123abc
|
|
119
236
|
bytekit monitors delete mon_123abc
|
|
120
237
|
```
|
|
121
238
|
|
|
239
|
+
`monitors list` accepts `--limit <n>`, `--cursor <cursor>`, and `--status`
|
|
240
|
+
(`active`, `processing`, `paused`, `cancelled`, `suspended`) as query filters,
|
|
241
|
+
each omitted when absent.
|
|
242
|
+
|
|
243
|
+
`monitors update` gains `--status` (`active` | `paused` — pause and resume from
|
|
244
|
+
the shell), `--change-threshold <pct>` (accepts a float), `--notify-on`
|
|
245
|
+
(`change`, `every`), `--webhook-secret <secret>`, and three JSON-object flags:
|
|
246
|
+
`--webhook-headers`, `--options`, and `--scrape-options` (e.g.
|
|
247
|
+
`--options '{"device":"mobile"}'`). `monitors create` accepts the same set plus
|
|
248
|
+
`--type` (`screenshot` | `scrape`). `--type` is **create-only** — a monitor's
|
|
249
|
+
type is immutable, so `monitors update` deliberately has no `--type`. Every flag
|
|
250
|
+
is omitted from the request when absent.
|
|
251
|
+
|
|
122
252
|
### `sitemap` — crawl a site's URL graph
|
|
123
253
|
|
|
124
254
|
```bash
|
|
125
|
-
bytekit sitemap create --url https://example.com
|
|
255
|
+
bytekit sitemap create --url https://example.com --strategy deep --max-depth 3 --max-urls 500
|
|
126
256
|
bytekit sitemap get sm_123abc
|
|
127
257
|
```
|
|
128
258
|
|
|
129
|
-
|
|
259
|
+
`sitemap create` accepts `--strategy` (`quick`, `standard`, `deep`),
|
|
260
|
+
`--max-depth <n>`, and `--max-urls <n>`, each omitted when absent.
|
|
261
|
+
|
|
262
|
+
### `account` — account details
|
|
130
263
|
|
|
131
264
|
```bash
|
|
132
265
|
bytekit account get
|
|
133
|
-
|
|
134
|
-
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
API key management is not available from the CLI: it requires a signed-in
|
|
269
|
+
dashboard session, not a bearer API key. Create, rename, reveal, and revoke keys
|
|
270
|
+
at [app.bytekit.com](https://app.bytekit.com).
|
|
271
|
+
|
|
272
|
+
### `usage` — account usage and billing-period reporting
|
|
273
|
+
|
|
274
|
+
```bash
|
|
275
|
+
bytekit usage get
|
|
276
|
+
bytekit usage daily --from 2026-01-01 --to 2026-01-31
|
|
277
|
+
bytekit usage by-endpoint --api-key ak_123abc
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
`usage get` returns the current billing-period summary. `usage daily` and
|
|
281
|
+
`usage by-endpoint` accept `--from <date>`, `--to <date>` (YYYY-MM-DD), and
|
|
282
|
+
`--api-key <ak_id>`, each omitted when absent (default range: last 30 days).
|
|
283
|
+
|
|
284
|
+
### `webhook-deliveries` — inspect and retry webhook deliveries
|
|
285
|
+
|
|
286
|
+
```bash
|
|
287
|
+
bytekit webhook-deliveries list --status failed --limit 10
|
|
288
|
+
bytekit webhook-deliveries retry wh_123abc
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
`list` accepts `--limit <n>`, `--cursor <cursor>`, `--status`
|
|
292
|
+
(`pending`, `delivered`, `failed`, `exhausted`), and `--since <iso>`, each
|
|
293
|
+
omitted when absent. `retry <id>` requeues a failed or exhausted delivery.
|
|
294
|
+
|
|
295
|
+
## Exit codes
|
|
296
|
+
|
|
297
|
+
| Code | Meaning |
|
|
298
|
+
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
299
|
+
| `0` | Success. |
|
|
300
|
+
| `1` | Usage error, or an API/connection error. The reason is on stderr, always prefixed `Error: `. |
|
|
301
|
+
| `2` | The scrape was accepted but is still **queued**, so there is no content body to emit — `scrape create --async` with `--raw` or a bare `-o` (which writes the bare `sc_` id to stdout), and the same command in human mode. Not a failure: the job was accepted and billed. Poll it with `bytekit scrape get <id>`. |
|
|
302
|
+
|
|
303
|
+
Code `2` is deliberately distinct from both `0` and `1` so a script can branch
|
|
304
|
+
on "queued" without parsing stderr:
|
|
305
|
+
|
|
306
|
+
```bash
|
|
307
|
+
sc_id=$(bytekit scrape create --url https://example.com --async --raw)
|
|
308
|
+
case $? in
|
|
309
|
+
0) echo "content: $sc_id" ;;
|
|
310
|
+
2) echo "queued as $sc_id — poll it" ;;
|
|
311
|
+
*) exit 1 ;;
|
|
312
|
+
esac
|
|
135
313
|
```
|
|
136
314
|
|
|
137
315
|
## Documentation
|
package/dist/client.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { ByteKit } from '@hunt-labs/bytekit-sdk';
|
|
2
2
|
export interface GlobalOptions {
|
|
3
3
|
key?: string | undefined;
|
|
4
4
|
baseUrl?: string | undefined;
|
|
@@ -22,6 +22,17 @@ export declare const MIN_RETRY_AFTER_MS = 1000;
|
|
|
22
22
|
export declare const MAX_RETRY_AFTER_MS = 60000;
|
|
23
23
|
/** Exactly one retry. A second 429 is terminal — no loop. */
|
|
24
24
|
export declare const MAX_RETRIES = 1;
|
|
25
|
+
/**
|
|
26
|
+
* Default CLIENT-SIDE abort budget for a single fetch attempt, in ms.
|
|
27
|
+
*
|
|
28
|
+
* Mirrors the SDK's own `DEFAULT_TIMEOUT_MS` (120s). Overriding `client.request` with this
|
|
29
|
+
* transport had discarded the SDK's default, so a hung server hung the CLI forever; this restores
|
|
30
|
+
* the same bound. The budget is **per attempt**, not per command: the single 429 retry (and its
|
|
31
|
+
* backoff sleep) gets a fresh budget, so the retry can never be cut short by time already spent on
|
|
32
|
+
* the first attempt. A future `--timeout` flag would override this default; today only the default
|
|
33
|
+
* exists.
|
|
34
|
+
*/
|
|
35
|
+
export declare const DEFAULT_TIMEOUT_MS = 120000;
|
|
25
36
|
/**
|
|
26
37
|
* Parse a `Retry-After` header into a clamped millisecond delay.
|
|
27
38
|
*
|
|
@@ -31,29 +42,39 @@ export declare const MAX_RETRIES = 1;
|
|
|
31
42
|
*/
|
|
32
43
|
export declare function parseRetryAfterMs(raw: string | null): number;
|
|
33
44
|
/**
|
|
34
|
-
* Build the CLI's
|
|
35
|
-
*
|
|
36
|
-
* Why the CLI owns the `fetch` instead of wrapping the SDK's `request()`: the SDK throws a
|
|
37
|
-
* header-less `RapidCrawlError` and discards the `Response`, so a delegating wrapper can see
|
|
38
|
-
* `status === 429` but can never read `Retry-After`. Owning the fetch is the only way to honor the
|
|
39
|
-
* server's advertised backoff without editing the SDK.
|
|
45
|
+
* Build the CLI's JSON transport: the SDK's `request` contract, plus the single bounded 429 retry.
|
|
40
46
|
*
|
|
41
47
|
* Every SDK resource dereferences `this.client.request` at call time, so installing this as an own
|
|
42
|
-
* property of the client (see {@link buildClient}) routes
|
|
48
|
+
* property of the client (see {@link buildClient}) routes all JSON commands through it from one
|
|
49
|
+
* seam. `/v1/fetch` does NOT ride this transport — it rides `requestRaw` (see
|
|
50
|
+
* {@link createRetryingRequestRaw}); `/v1/fetch/bulk` is a JSON job envelope and does.
|
|
51
|
+
*/
|
|
52
|
+
export declare function createRetryingRequest(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['request'];
|
|
53
|
+
/**
|
|
54
|
+
* Build the CLI's RAW transport: the SDK's `requestRaw` contract, plus the same bounded 429 retry.
|
|
55
|
+
*
|
|
56
|
+
* `/v1/fetch` returns the fetched content AS the body (HTML, markdown, or true binary) with its
|
|
57
|
+
* metadata in `X-Fetch-*` headers, so the SDK routes it through `requestRaw` rather than the JSON
|
|
58
|
+
* `request`. The CLI must override BOTH seams or fetch commands silently bypass the retry and the
|
|
59
|
+
* CLI's error taxonomy — the #2535 defect, where only `request` was replaced.
|
|
43
60
|
*
|
|
44
|
-
*
|
|
61
|
+
* The returned shape is the SDK's `requestRaw` contract verbatim (`{body, bytes, headers, status}`)
|
|
62
|
+
* so `FetchResource` builds an ordinary `FetchResult` from it and the CLI never forks fetch
|
|
63
|
+
* handling away from the SDK. `bytes` is byte-exact (binary-safe); `body` is the best-effort UTF-8
|
|
64
|
+
* decode, matching the SDK's own semantics.
|
|
45
65
|
*/
|
|
46
|
-
export declare function
|
|
66
|
+
export declare function createRetryingRequestRaw(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['requestRaw'];
|
|
47
67
|
/**
|
|
48
68
|
* Build an authenticated SDK client from global CLI options.
|
|
49
69
|
*
|
|
50
|
-
* Resolution order (explicit input beats environment
|
|
51
|
-
* apiKey: --key flag → BYTEKIT_API_KEY
|
|
52
|
-
* baseUrl: --base-url flag → BYTEKIT_BASE_URL →
|
|
53
|
-
*
|
|
54
|
-
* descriptive error when no key is available.
|
|
70
|
+
* Resolution order (explicit input beats environment):
|
|
71
|
+
* apiKey: --key flag → BYTEKIT_API_KEY.
|
|
72
|
+
* baseUrl: --base-url flag → BYTEKIT_BASE_URL → DEFAULT_BASE_URL.
|
|
73
|
+
* Exits with a descriptive error when no key is available.
|
|
55
74
|
*
|
|
56
|
-
* The returned client's `request`
|
|
57
|
-
* the 429 retry
|
|
75
|
+
* The returned client's `request` AND `requestRaw` are both replaced by the retrying transports, so
|
|
76
|
+
* every command inherits the 429 retry — JSON commands via `request`, the `/v1/fetch` content
|
|
77
|
+
* commands via `requestRaw`. Overriding only `request` is the #2535 defect: the SDK's fetch
|
|
78
|
+
* resource rides `requestRaw`, so fetch bypassed the retry and the CLI's error taxonomy entirely.
|
|
58
79
|
*/
|
|
59
|
-
export declare function buildClient(opts: GlobalOptions):
|
|
80
|
+
export declare function buildClient(opts: GlobalOptions): ByteKit;
|