@hunt-labs/bytekit-cli 0.7.1 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +90 -1
- package/README.md +139 -20
- package/dist/client.js +21 -0
- package/dist/commands/screenshots.d.ts +26 -0
- package/dist/commands/screenshots.js +94 -1
- package/dist/commands/search.js +25 -0
- package/dist/index.js +4 -7
- package/dist/package-version.d.ts +6 -0
- package/dist/package-version.js +31 -0
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -10,6 +10,95 @@ today, see [README.md](./README.md), which remains the authoritative usage guide
|
|
|
10
10
|
|
|
11
11
|
## [Unreleased]
|
|
12
12
|
|
|
13
|
+
## [0.8.1] - 2026-08-13
|
|
14
|
+
|
|
15
|
+
### Fixed
|
|
16
|
+
|
|
17
|
+
- **`bytekit search` explains a malformed `200` instead of leaking a `TypeError`.** A server
|
|
18
|
+
answering `200 {"ok":true}` — a proxy, a stub, a future envelope change — drove the human-mode
|
|
19
|
+
renderer into `Error: response.results is not iterable`, which named an internal expression
|
|
20
|
+
rather than the field the body was missing. It now reports, on one line and with no stack trace,
|
|
21
|
+
`search: the response carries no "results" array — the server returned an unexpected 200 body. Re-run with --json to inspect it.`
|
|
22
|
+
and exits `1`. A `results` value that is present but not an array is caught by the same gate: a
|
|
23
|
+
**string** was previously iterated CHARACTER BY CHARACTER, printing one garbled row per character
|
|
24
|
+
and exiting `0`. Unchanged: an empty `results: []` is still a successful zero-hit search that
|
|
25
|
+
prints nothing and exits `0`; valid responses render byte-identically; and `--json` never reaches
|
|
26
|
+
the renderer, so it keeps dumping the raw body at exit `0` — which is how you inspect an
|
|
27
|
+
unexpected body.
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
|
|
31
|
+
- **`screenshots … -o <file>` now says so when the filename contradicts the captured format.**
|
|
32
|
+
`--format` is omitted far more often than not and the server default is **jpeg**, so
|
|
33
|
+
`bytekit screenshots create --url … -o shot.png` wrote JPEG bytes into a `.png`-named file
|
|
34
|
+
silently, at exit `0`; `screenshots get` has no `--format` flag at all, so it could not even be
|
|
35
|
+
told what it was downloading. Both now print one advisory line to **stderr** —
|
|
36
|
+
`Note: the screenshot was captured as jpeg, but shot.png names .png. The file holds jpeg bytes — pass --format png to capture it as png.`
|
|
37
|
+
The captured format is read from the artifact URL's own path, which is the only place the CLI can
|
|
38
|
+
observe it. **Advisory, not enforcement:** the bytes written and the exit code are unchanged, and
|
|
39
|
+
the note goes to stderr, so `-o -` pipelines and stdout redirects stay byte-identical. It stays
|
|
40
|
+
silent unless both the destination and the artifact name a recognized image format and the two
|
|
41
|
+
differ — `.jpg` and `.jpeg` are the same format, and an extension-less destination, a non-image
|
|
42
|
+
extension, `-o -`, or an artifact URL with no usable extension all make no claim to contradict. A
|
|
43
|
+
not-ready poll is untouched: it still prints its bare id and poll note at exit `2`.
|
|
44
|
+
|
|
45
|
+
### Changed
|
|
46
|
+
|
|
47
|
+
- **The README now documents the whole flag surface, and says which reference wins.** Every flag
|
|
48
|
+
worked and every flag appeared in `--help`; the README simply framed subsets as exhaustive, so
|
|
49
|
+
`bulk create`'s `--webhook-secret` / `--defaults` / `--metadata`, `scrape bulk create`'s
|
|
50
|
+
`--custom`, `sitemap create`'s `--webhook-url` / `--webhook-secret` / `--cache-ttl` /
|
|
51
|
+
`--compact` / `--process` / `--metadata`, `monitors create` and `monitors update`'s `--cron`
|
|
52
|
+
(required with `--interval-type cron`) and `--metadata` plus the rest of `update`'s surface,
|
|
53
|
+
`fetch create`'s ten markdown flags and `--custom`, and `screenshots create`'s `--headers` /
|
|
54
|
+
`--cookies` / `--metadata` were all undocumented. All of them are documented now, and a note at
|
|
55
|
+
the top of the README states that `--help` is the authoritative flag reference. **Required flags
|
|
56
|
+
are marked as such** — `--file` and `--webhook-url` on all three bulk creators, `--url` /
|
|
57
|
+
`--interval-type` / `--webhook-url` on `monitors create`, and `--url` on `scrape create`,
|
|
58
|
+
`screenshots create`, `sitemap create` and both `fetch` verbs — so no reader mistakes one for
|
|
59
|
+
optional. Note in particular that a **bulk job has no poll-only mode**: `--webhook-url` is
|
|
60
|
+
mandatory there, unlike on `sitemap create` where omitting it really does mean poll-only. A test
|
|
61
|
+
asserts the parity in the direction that matters — every flag the parser accepts must appear in
|
|
62
|
+
the README — so this cannot drift again silently. No command, flag or behavior changed.
|
|
63
|
+
|
|
64
|
+
- **Alignment-release note, carried forward from this version's original cut.** `0.8.0` published
|
|
65
|
+
to npm at `2026-08-11T21:18:01.304Z`. Doc comments in `src/client.ts` and
|
|
66
|
+
`src/package-version.ts` were reworded **after** the commit that introduced the `0.8.0` version
|
|
67
|
+
string — they had re-spelled the `X-ByteKit-Client` header name in prose, which issue #2906's
|
|
68
|
+
pre-merge gate 1 counts. No command, flag, exit code, or runtime behavior differs from `0.8.0`
|
|
69
|
+
on account of _those_ comments.
|
|
70
|
+
|
|
71
|
+
`0.8.1` was cut for that reason because `check-release-drift` (#2680) is version-gated and
|
|
72
|
+
anchors on the commit that introduced a version, not on the commit whose tree was actually
|
|
73
|
+
published. Once `0.8.0` reached npm, any shipped-file change after that anchor reads as
|
|
74
|
+
post-release drift. Bumping moves the anchor past those comments. `0.8.1` had not yet reached
|
|
75
|
+
npm when the entries above landed, so they ride this same pending release rather than stranding
|
|
76
|
+
it as a never-published version.
|
|
77
|
+
|
|
78
|
+
## [0.8.0] - 2026-08-11
|
|
79
|
+
|
|
80
|
+
### Added
|
|
81
|
+
|
|
82
|
+
- **Every outgoing request now carries `X-ByteKit-Client: cli/<version>`.** The CLI identifies
|
|
83
|
+
itself with the sanctioned `client_surface` marker (issue #2906, slice 3 of #2701), so `bytekit`
|
|
84
|
+
usage is distinguishable in ByteKit's own analytics from both raw API calls (`api_direct`) and
|
|
85
|
+
programmatic `@hunt-labs/bytekit-sdk` usage (`sdk_ts`) instead of being counted as one of them.
|
|
86
|
+
`cli` is a member of the closed `client_surface` vocabulary declared in issue #2904; the version
|
|
87
|
+
affix is this package's own version, resolved from `package.json` at runtime — the same value
|
|
88
|
+
`bytekit --version` prints.
|
|
89
|
+
|
|
90
|
+
Set at exactly ONE place — the header literal inside the private transport core all three CLI
|
|
91
|
+
transports funnel through — so it rides on every command, on the JSON path, on the raw
|
|
92
|
+
`/v1/fetch` path, and on the 429 retry attempt alike. **Nothing else about a request changed:**
|
|
93
|
+
same `Authorization` and `Content-Type`, same body bytes, same per-attempt abort budget, same
|
|
94
|
+
single-retry 429 behavior, same error taxonomy, same exit codes and output shapes. The marker
|
|
95
|
+
carries no user data and cannot be set, overridden or read from the command line — `--headers`
|
|
96
|
+
is a _scrape_ option that travels in the request body to the captured site, and never touches
|
|
97
|
+
the CLI's own request headers.
|
|
98
|
+
|
|
99
|
+
Minor rather than patch because the wire format of every request changes, even though no
|
|
100
|
+
documented command behavior does. A gateway that does not yet recognize the header ignores it.
|
|
101
|
+
|
|
13
102
|
## [0.7.1] - 2026-08-10
|
|
14
103
|
|
|
15
104
|
### Changed
|
|
@@ -429,7 +518,7 @@ Additive, lower-risk release — no breaking changes to existing commands.
|
|
|
429
518
|
`workspace:^`, added `engines`, `keywords`, `homepage`, `bugs`, `README.md`,
|
|
430
519
|
`LICENSE`, and an MIT `license` field.
|
|
431
520
|
|
|
432
|
-
## [0.1.0] - 2026
|
|
521
|
+
## [0.1.0] - 2026-07-13
|
|
433
522
|
|
|
434
523
|
### Added
|
|
435
524
|
|
package/README.md
CHANGED
|
@@ -4,6 +4,12 @@ Command-line interface for the [ByteKit](https://bytekit.com) API — scrape,
|
|
|
4
4
|
fetch, screenshot, search, crawl sitemaps, and manage page monitors from your
|
|
5
5
|
shell. Wraps [`@hunt-labs/bytekit-sdk`](https://www.npmjs.com/package/@hunt-labs/bytekit-sdk).
|
|
6
6
|
|
|
7
|
+
> **`--help` is the authoritative flag reference.** Every command and subcommand
|
|
8
|
+
> accepts it — `bytekit sitemap create --help`, `bytekit monitors update --help` —
|
|
9
|
+
> and the list it prints is generated from the code that parses your argv, so it
|
|
10
|
+
> is never out of date. This README documents the same surface in prose for
|
|
11
|
+
> reading; where the two ever disagree, `--help` is right.
|
|
12
|
+
|
|
7
13
|
## Install
|
|
8
14
|
|
|
9
15
|
```bash
|
|
@@ -86,9 +92,13 @@ bytekit scrape get sc_123abc
|
|
|
86
92
|
bytekit scrape bulk create --file urls.txt --webhook-url https://your.app/hooks/bytekit
|
|
87
93
|
```
|
|
88
94
|
|
|
89
|
-
`scrape create`
|
|
90
|
-
|
|
91
|
-
|
|
95
|
+
`scrape bulk create` takes the shared bulk-job flags plus `--custom` — see
|
|
96
|
+
[`bulk`](#bulk--bulk-screenshot-jobs).
|
|
97
|
+
|
|
98
|
+
`scrape create` **requires** `--url <url>`. It also accepts the full
|
|
99
|
+
`ScrapeRequest` surface below, each of those flags omitted from the request when
|
|
100
|
+
absent (a boolean flag never sends `false`, so a server default is never
|
|
101
|
+
materialized):
|
|
92
102
|
|
|
93
103
|
| Flag | Effect |
|
|
94
104
|
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -183,9 +193,33 @@ Note the acronym casing — `X-Fetch-ID` and `X-Fetch-URL`, not `X-Fetch-Id` /
|
|
|
183
193
|
`X-Fetch-Url`. (Versions up to 0.6.0 emitted the Title-Cased spellings; see the
|
|
184
194
|
CHANGELOG.)
|
|
185
195
|
|
|
186
|
-
Both verbs
|
|
187
|
-
(`
|
|
188
|
-
(`
|
|
196
|
+
Both verbs **require** `--url <url>`. Both also accept `--format <format>`
|
|
197
|
+
(`markdown` or `html`), `--country <cc>`, `--timeout-ms <n>`, `--cache-ttl <ttl>`
|
|
198
|
+
(`0`, `Nh`, or `Nd`), and `-o, --output <file>` — each of those omitted from the
|
|
199
|
+
request when absent.
|
|
200
|
+
|
|
201
|
+
`fetch create` accepts the full markdown-pipeline surface on top of that, which
|
|
202
|
+
`fetch get` does not:
|
|
203
|
+
|
|
204
|
+
| Flag | Effect |
|
|
205
|
+
| ----------------------------- | ---------------------------------------------------------------------------- |
|
|
206
|
+
| `--markdown-mode <mode>` | Markdown extraction mode: `article`, `raw`, `llm`. |
|
|
207
|
+
| `--markdown-query <q>` | BM25 query for relevance-ranked markdown filtering. Omitted → disabled. |
|
|
208
|
+
| `--markdown-links <mode>` | Link rendering: `inline`, `references`, `none`, `text`. |
|
|
209
|
+
| `--markdown-images <mode>` | Image retention: `inline`, `references`, `none`, `text`. |
|
|
210
|
+
| `--with-links-summary` | Append a Links footer to the markdown. |
|
|
211
|
+
| `--with-images-summary` | Append an Images footer to the markdown. |
|
|
212
|
+
| `--markdown-compact` | Collapse excessive whitespace. |
|
|
213
|
+
| `--markdown-filter-images` | Filter low-signal images. |
|
|
214
|
+
| `--markdown-include-media` | Return rich link/image objects and a tables array. |
|
|
215
|
+
| `--markdown-include-warnings` | Include markdown-pipeline and tag-filter warnings. |
|
|
216
|
+
| `--markdown-include-stats` | Include a top-level stats object (chars, tokens, blocks). |
|
|
217
|
+
| `--custom <json>` | User payload echoed on the envelope, as a JSON object, e.g. `'{"job":"1"}'`. |
|
|
218
|
+
|
|
219
|
+
Each is omitted when absent, so a server default is never materialized.
|
|
220
|
+
|
|
221
|
+
`fetch bulk create` takes the shared bulk-job flags — see
|
|
222
|
+
[`bulk`](#bulk--bulk-screenshot-jobs).
|
|
189
223
|
|
|
190
224
|
### `search` — ranked web search
|
|
191
225
|
|
|
@@ -206,7 +240,8 @@ bytekit screenshots create --url https://example.com --device mobile --format pn
|
|
|
206
240
|
bytekit screenshots get ss_123abc -o shot.png
|
|
207
241
|
```
|
|
208
242
|
|
|
209
|
-
`screenshots create`
|
|
243
|
+
`screenshots create` **requires** `--url <url>`. It also accepts `--device`
|
|
244
|
+
(`desktop`, `mobile`), `--viewport WxH`
|
|
210
245
|
(e.g. `1280x720`), `--format` (`jpeg`, `png`), `--quality <n>`, `--wait-until`
|
|
211
246
|
(`load`, `domcontentloaded`, `networkidle`), `--clip x,y,w,h`, and `--async`
|
|
212
247
|
(queue and return a job envelope). Each flag is omitted from the request when
|
|
@@ -214,7 +249,10 @@ absent.
|
|
|
214
249
|
|
|
215
250
|
It also accepts `--wait-for-selector <sel>`, `--delay-ms <n>`,
|
|
216
251
|
`--device-scale-factor` (`1` or `2`), `--dark-mode`, `--include-html`,
|
|
217
|
-
`--country <cc>`, `--language <lang>`,
|
|
252
|
+
`--country <cc>`, `--language <lang>`, three JSON flags — `--headers <json>`
|
|
253
|
+
(extra request headers as an object, e.g. `'{"X-A":"b"}'`), `--cookies <json>`
|
|
254
|
+
(an array of `{name,value,domain}`), and `--metadata <json>` (echoed back on the
|
|
255
|
+
screenshot envelope) — and three **tri-state** blocker pairs:
|
|
218
256
|
`--block-ads` / `--no-block-ads`, `--block-cookie-banners` /
|
|
219
257
|
`--no-block-cookie-banners`, and `--scroll` / `--no-scroll`. Each of these three
|
|
220
258
|
defaults to `true` server-side, so passing neither half omits the key entirely
|
|
@@ -227,6 +265,23 @@ stdout). Because `-o` writes BINARY on `screenshots`, it cannot be combined with
|
|
|
227
265
|
`--async` + `-o` prints the queued job envelope and skips the download (the
|
|
228
266
|
image is not ready yet).
|
|
229
267
|
|
|
268
|
+
**`-o` never converts — it names the file, `--format` names the bytes.** The
|
|
269
|
+
server default format is **jpeg**, so `-o shot.png` on a capture that did not
|
|
270
|
+
pass `--format png` writes JPEG bytes into a `.png`-named file. When the
|
|
271
|
+
destination's extension and the captured format disagree, the CLI prints one
|
|
272
|
+
advisory line to stderr and downloads anyway:
|
|
273
|
+
|
|
274
|
+
```text
|
|
275
|
+
Note: the screenshot was captured as jpeg, but shot.png names .png.
|
|
276
|
+
The file holds jpeg bytes — pass --format png to capture it as png.
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
The bytes written and the exit code are unchanged — it is a note, not a refusal,
|
|
280
|
+
and it goes to stderr so `-o -` pipelines and stdout redirects stay
|
|
281
|
+
byte-identical. It stays silent when the two agree (`.jpg` and `.jpeg` are the
|
|
282
|
+
same format), and when the destination makes no format claim at all (`-o -`, no
|
|
283
|
+
extension, or a non-image extension).
|
|
284
|
+
|
|
230
285
|
### `bulk` — bulk screenshot jobs
|
|
231
286
|
|
|
232
287
|
```bash
|
|
@@ -237,8 +292,36 @@ bytekit bulk cancel "$bulk_id"
|
|
|
237
292
|
bytekit bulk screenshots list "$bulk_id"
|
|
238
293
|
```
|
|
239
294
|
|
|
240
|
-
`urls.txt` is newline-delimited: one plain URL (or `{"url":"https://..."}`) per
|
|
241
|
-
`
|
|
295
|
+
`urls.txt` is newline-delimited: one plain URL (or `{"url":"https://..."}`) per
|
|
296
|
+
line. A line whose only key is `url` is sent as a bare URL string; any line with
|
|
297
|
+
additional keys switches that entry to the per-item shape. `bulk cancel <id>`
|
|
298
|
+
stops an in-flight job; `bulk get <id>` and `bulk screenshots list <id>` take no
|
|
299
|
+
flags.
|
|
300
|
+
|
|
301
|
+
The three bulk-job creators — `bulk create`, `scrape bulk create` and
|
|
302
|
+
`fetch bulk create` — share one flag set:
|
|
303
|
+
|
|
304
|
+
| Flag | Effect |
|
|
305
|
+
| --------------------------- | ------------------------------------------------------------------------------------------------------ |
|
|
306
|
+
| `--file <path>` | **Required.** The newline-delimited input file described above. |
|
|
307
|
+
| `--webhook-url <url>` | **Required.** HTTPS webhook notified when the job completes. |
|
|
308
|
+
| `--webhook-secret <secret>` | HMAC secret used to sign that completion webhook. Omitted → unsigned. |
|
|
309
|
+
| `--defaults <json>` | Defaults applied to every entry, as a JSON object. Omitted → none. |
|
|
310
|
+
| `--metadata <json>` | Arbitrary metadata echoed on the job envelope, as a JSON object, e.g. `'{"run":"7"}'`. Omitted → none. |
|
|
311
|
+
|
|
312
|
+
A bulk job has **no poll-only mode**. `--file` and `--webhook-url` are both
|
|
313
|
+
required on all three creators — omitting either is a usage error raised before
|
|
314
|
+
any request is sent (`error: required option '--webhook-url <url>' not
|
|
315
|
+
specified`), and the API declares `webhook_url` required as well. The three
|
|
316
|
+
optional flags are omitted from the request when absent.
|
|
317
|
+
|
|
318
|
+
`--defaults` is validated differently per endpoint: `bulk` rejects unknown keys
|
|
319
|
+
with a `422`, while `scrape bulk` and `fetch bulk` ignore them — and `fetch bulk`
|
|
320
|
+
additionally rejects the browser-only keys (`cookies`, `wait_until`,
|
|
321
|
+
`wait_for_selector`, `delay_ms`) with a `400`.
|
|
322
|
+
|
|
323
|
+
`scrape bulk create` adds one flag the other two do not have: `--custom <json>`,
|
|
324
|
+
a user payload echoed on every scrape entry (a per-entry value in `--file` wins).
|
|
242
325
|
|
|
243
326
|
### `monitors` — watch a page for changes
|
|
244
327
|
|
|
@@ -260,14 +343,34 @@ each omitted when absent.
|
|
|
260
343
|
page through a monitor's capture history — pass the response's `next_cursor`
|
|
261
344
|
back as `--cursor` to fetch the next page. Both flags are omitted when absent.
|
|
262
345
|
|
|
263
|
-
`monitors
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
346
|
+
`monitors create` and `monitors update` share almost all of their flags. On
|
|
347
|
+
`update` every one is optional and omitted from the request when absent, so
|
|
348
|
+
`monitors update` changes only what you name. Three are **required on `create`**
|
|
349
|
+
and marked below:
|
|
350
|
+
|
|
351
|
+
| Flag | Effect |
|
|
352
|
+
| --------------------------- | ------------------------------------------------------------------------------------------ |
|
|
353
|
+
| `--url <url>` | **Required on `create`.** URL to monitor. |
|
|
354
|
+
| `--interval-type <type>` | **Required on `create`.** Check interval: `hourly`, `daily`, `weekly`, or `cron`. |
|
|
355
|
+
| `--cron <expr>` | Cron expression — **required** when `--interval-type` is `cron`, e.g. `'0 * * * *'`. |
|
|
356
|
+
| `--webhook-url <url>` | **Required on `create`.** HTTPS webhook notified on change. |
|
|
357
|
+
| `--change-threshold <pct>` | Change percentage that triggers a notification (accepts a float). |
|
|
358
|
+
| `--notify-on <event>` | When to notify: `change`, `every`. |
|
|
359
|
+
| `--webhook-secret <secret>` | HMAC secret for webhook signature verification. |
|
|
360
|
+
| `--webhook-headers <json>` | Extra webhook headers as a JSON object, e.g. `'{"X-Sig":"abc"}'`. |
|
|
361
|
+
| `--options <json>` | Screenshot-monitor options as a JSON object, e.g. `--options '{"device":"mobile"}'`. |
|
|
362
|
+
| `--scrape-options <json>` | Scrape-monitor options as a JSON object (`--type scrape` only). |
|
|
363
|
+
| `--metadata <json>` | Arbitrary metadata echoed on the monitor envelope, as a JSON object, e.g. `'{"run":"7"}'`. |
|
|
364
|
+
|
|
365
|
+
Two flags are not shared. `--type` (`screenshot` | `scrape`) is **create-only** —
|
|
366
|
+
a monitor's type is immutable, so `monitors update` deliberately has none.
|
|
367
|
+
`--status` (`active` | `paused`, to pause and resume from the shell) is
|
|
368
|
+
**update-only**.
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
bytekit monitors create --url https://example.com/pricing --interval-type cron \
|
|
372
|
+
--cron '0 * * * *' --webhook-url https://your.app/hooks/bytekit
|
|
373
|
+
```
|
|
271
374
|
|
|
272
375
|
### `sitemap` — crawl a site's URL graph
|
|
273
376
|
|
|
@@ -276,8 +379,24 @@ bytekit sitemap create --url https://example.com --strategy deep --max-depth 3 -
|
|
|
276
379
|
bytekit sitemap get sm_123abc
|
|
277
380
|
```
|
|
278
381
|
|
|
279
|
-
`sitemap
|
|
280
|
-
|
|
382
|
+
`sitemap get <id>` takes no flags. `sitemap create` accepts:
|
|
383
|
+
|
|
384
|
+
| Flag | Effect |
|
|
385
|
+
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
386
|
+
| `--url <url>` | **Required.** Root URL to crawl. |
|
|
387
|
+
| `--strategy <strategy>` | Crawl strategy: `quick`, `standard`, `deep`. Omitted → server default `standard`. |
|
|
388
|
+
| `--max-depth <n>` | Maximum crawl depth, 1–5. Omitted → server default `2`. |
|
|
389
|
+
| `--max-urls <n>` | Maximum URLs to collect. Omitted → server default `5000`. |
|
|
390
|
+
| `--webhook-url <url>` | HTTPS webhook notified when the crawl completes. Omitted → poll-only, no delivery. |
|
|
391
|
+
| `--webhook-secret <secret>` | HMAC secret used to sign that completion webhook. Omitted → unsigned. |
|
|
392
|
+
| `--cache-ttl <ttl>` | Cache freshness: `0`, `Nh` (hours), or `Nd` (days), e.g. `48h`, `30d`. Omitted → server default `30d`; max `90d`, enforced server-side. |
|
|
393
|
+
| `--compact` | Return a compact result payload. Omitted → server default off. |
|
|
394
|
+
| `--process <json>` | Post-crawl processing directive as a JSON object, e.g. `'{"action":"summarize"}'`. Omitted → none. |
|
|
395
|
+
| `--metadata <json>` | Arbitrary metadata echoed on the job envelope, as a JSON object, e.g. `'{"run":"7"}'`. Omitted → none. |
|
|
396
|
+
|
|
397
|
+
Only `--url` is required; every other flag is omitted from the request when
|
|
398
|
+
absent. Unlike a bulk job, a sitemap crawl really is poll-only without
|
|
399
|
+
`--webhook-url`.
|
|
281
400
|
|
|
282
401
|
### `account` — account details
|
|
283
402
|
|
package/dist/client.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { ByteKit, ByteKitError, ByteKitConnectionError } from '@hunt-labs/bytekit-sdk';
|
|
2
|
+
import { PACKAGE_VERSION } from './package-version.js';
|
|
2
3
|
/**
|
|
3
4
|
* The API host the SDK itself defaults to. The SDK's copy is module-private, so the CLI declares
|
|
4
5
|
* its own and always passes an explicit `baseUrl` to the constructor — that way the SDK client and
|
|
@@ -289,6 +290,26 @@ async function sendWithRetry(apiKey, baseUrl, timeoutMs, method, path, body, rea
|
|
|
289
290
|
headers: {
|
|
290
291
|
Authorization: `Bearer ${apiKey}`,
|
|
291
292
|
'Content-Type': 'application/json',
|
|
293
|
+
// The `client_surface` marker (issue #2906, slice 3 of #2701). THE CLI'S ONLY EMIT SITE:
|
|
294
|
+
// all three CLI transports funnel through this function, and this literal is rebuilt
|
|
295
|
+
// inside the retry loop, so the 429 retry attempt carries it too (E4) without a second
|
|
296
|
+
// site to keep in sync.
|
|
297
|
+
//
|
|
298
|
+
// The header name below (`CLIENT_SURFACE_HEADER`) and the closed surface vocabulary
|
|
299
|
+
// (`CLIENT_SURFACES`) are declared in `@bytekit/analytics-contract` — deliberately spelled
|
|
300
|
+
// as literals here rather than imported, because that package must never become a runtime
|
|
301
|
+
// dependency of the published CLI (#2906 E6). `__tests__/client-surface-marker.test.ts`
|
|
302
|
+
// imports the contract as a TEST-ONLY devDependency and binds these literals to the
|
|
303
|
+
// registry, which is what keeps the two from drifting.
|
|
304
|
+
//
|
|
305
|
+
// Grep gate 1 (#2906) counts occurrences of the header name in `src/` outside tests and
|
|
306
|
+
// expects exactly ONE — so this comment names the CONSTANT rather than re-spelling the
|
|
307
|
+
// string, and the single occurrence below is the emit site itself.
|
|
308
|
+
//
|
|
309
|
+
// The version affix comes from the package manifest via the shared `PACKAGE_VERSION`
|
|
310
|
+
// helper — the same value commander reports for `--version` — never a second hardcoded
|
|
311
|
+
// version string.
|
|
312
|
+
'X-ByteKit-Client': `cli/${PACKAGE_VERSION}`,
|
|
292
313
|
},
|
|
293
314
|
};
|
|
294
315
|
if (body !== undefined) {
|
|
@@ -12,6 +12,32 @@ export declare function parseClip(value: string): {
|
|
|
12
12
|
width: number;
|
|
13
13
|
height: number;
|
|
14
14
|
};
|
|
15
|
+
/**
|
|
16
|
+
* The advisory note for a `-o` destination whose extension contradicts the captured format, or
|
|
17
|
+
* undefined when there is nothing to say (#2956 item 3).
|
|
18
|
+
*
|
|
19
|
+
* THE DEFECT: `--format` is omitted far more often than not, and the server default is **jpeg** —
|
|
20
|
+
* so `screenshots create --url … -o shot.png` wrote JPEG bytes into a `.png`-named file, silently,
|
|
21
|
+
* at exit 0. The mislabelled file then propagated to whatever consumed it next. `screenshots get`
|
|
22
|
+
* has no `--format` flag at all, so it could not even be told what it was downloading.
|
|
23
|
+
*
|
|
24
|
+
* ADVISORY, NOT ENFORCEMENT — this returns a message; the caller writes it to STDERR and proceeds.
|
|
25
|
+
* The bytes on disk and the exit code are untouched, because the caller may well have meant it
|
|
26
|
+
* (`-o` is also how you rename an artifact), and because refusing the write would throw away a
|
|
27
|
+
* capture that was already taken and billed. It stays on the `-o` write path rather than moving
|
|
28
|
+
* into `downloadArtifact`: that helper is format-agnostic and has no business knowing about image
|
|
29
|
+
* extensions.
|
|
30
|
+
*
|
|
31
|
+
* Silent unless BOTH sides name a recognized image format AND they differ. Nothing to compare —
|
|
32
|
+
* `-o -` (stdout, no filename to mislabel), an extension-less or non-image destination, an
|
|
33
|
+
* artifact URL with no usable extension — means no note, never a guessed one.
|
|
34
|
+
*
|
|
35
|
+
* `-o -` needs no special case of its own: the literal `-` carries no extension, so
|
|
36
|
+
* {@link formatFromPath} already returns undefined for it. An explicit early return was written
|
|
37
|
+
* first and then removed — no mutation of it could turn a test red, which is the definition of a
|
|
38
|
+
* line that guards nothing. The stdout case stays PINNED by a test either way.
|
|
39
|
+
*/
|
|
40
|
+
export declare function formatMismatchNote(imageUrl: string, dest: string): string | undefined;
|
|
15
41
|
export declare function makeScreenshotsCommand(getGlobal: () => GlobalOptions & {
|
|
16
42
|
json: boolean;
|
|
17
43
|
}): Command;
|
|
@@ -99,6 +99,91 @@ function notReadyScreenshot(result) {
|
|
|
99
99
|
return undefined;
|
|
100
100
|
return { id: obj.id, status };
|
|
101
101
|
}
|
|
102
|
+
/**
|
|
103
|
+
* Image extensions this CLI recognizes, mapped to their canonical format name.
|
|
104
|
+
*
|
|
105
|
+
* `jpg` and `jpeg` are the SAME format under two spellings — collapsing them here is what keeps
|
|
106
|
+
* the note off `-o shot.jpg`, which is the commonest spelling of a perfectly correct destination.
|
|
107
|
+
* The set is deliberately small: it exists to decide whether a name makes a FORMAT CLAIM at all,
|
|
108
|
+
* and an extension outside it (`.bin`, `.tmp`, none) makes none.
|
|
109
|
+
*/
|
|
110
|
+
const IMAGE_EXTENSIONS = {
|
|
111
|
+
jpg: 'jpeg',
|
|
112
|
+
jpeg: 'jpeg',
|
|
113
|
+
png: 'png',
|
|
114
|
+
};
|
|
115
|
+
/**
|
|
116
|
+
* The canonical image format named by a path's extension, or undefined when it names none.
|
|
117
|
+
*
|
|
118
|
+
* Reads the extension off the last path SEGMENT so a directory containing a dot cannot supply one,
|
|
119
|
+
* and requires a non-empty stem so a dotfile (`.png`) is a name, not an extension.
|
|
120
|
+
*/
|
|
121
|
+
function formatFromPath(path) {
|
|
122
|
+
const segment = path.slice(path.lastIndexOf('/') + 1);
|
|
123
|
+
const dot = segment.lastIndexOf('.');
|
|
124
|
+
if (dot <= 0)
|
|
125
|
+
return undefined;
|
|
126
|
+
return IMAGE_EXTENSIONS[segment.slice(dot + 1).toLowerCase()];
|
|
127
|
+
}
|
|
128
|
+
/**
|
|
129
|
+
* The format the artifact was actually captured in, read off its `image_url`, or undefined.
|
|
130
|
+
*
|
|
131
|
+
* `screenshotKey` (`packages/worker/src/storage/b2.ts:60`) builds the object key as
|
|
132
|
+
* `screenshots/<account>/<date>/<screenshot_id>.<format>`, so the URL's PATHNAME extension is the
|
|
133
|
+
* captured format — the only place the CLI can observe it. `ScreenshotResponse`
|
|
134
|
+
* (`docs/api/openapi.yaml`) carries no `format` field, and `screenshots get` has no `--format`
|
|
135
|
+
* flag, so neither the envelope nor the argv can answer this on the poll path.
|
|
136
|
+
*
|
|
137
|
+
* The pathname — never the raw string. A completed `image_url` is an AWS SigV4 PRESIGNED URL
|
|
138
|
+
* (`?X-Amz-Credential=…&X-Amz-Signature=…`, `gateway/src/lib/signed-url.ts`), and a
|
|
139
|
+
* `lastIndexOf('.')` over the whole string reads the signature instead of the key. An unparseable
|
|
140
|
+
* URL yields undefined rather than throwing: this is advisory, and it runs immediately before a
|
|
141
|
+
* download that must not be blocked by it.
|
|
142
|
+
*/
|
|
143
|
+
function capturedFormat(imageUrl) {
|
|
144
|
+
try {
|
|
145
|
+
return formatFromPath(new URL(imageUrl).pathname);
|
|
146
|
+
}
|
|
147
|
+
catch {
|
|
148
|
+
return undefined;
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
/**
|
|
152
|
+
* The advisory note for a `-o` destination whose extension contradicts the captured format, or
|
|
153
|
+
* undefined when there is nothing to say (#2956 item 3).
|
|
154
|
+
*
|
|
155
|
+
* THE DEFECT: `--format` is omitted far more often than not, and the server default is **jpeg** —
|
|
156
|
+
* so `screenshots create --url … -o shot.png` wrote JPEG bytes into a `.png`-named file, silently,
|
|
157
|
+
* at exit 0. The mislabelled file then propagated to whatever consumed it next. `screenshots get`
|
|
158
|
+
* has no `--format` flag at all, so it could not even be told what it was downloading.
|
|
159
|
+
*
|
|
160
|
+
* ADVISORY, NOT ENFORCEMENT — this returns a message; the caller writes it to STDERR and proceeds.
|
|
161
|
+
* The bytes on disk and the exit code are untouched, because the caller may well have meant it
|
|
162
|
+
* (`-o` is also how you rename an artifact), and because refusing the write would throw away a
|
|
163
|
+
* capture that was already taken and billed. It stays on the `-o` write path rather than moving
|
|
164
|
+
* into `downloadArtifact`: that helper is format-agnostic and has no business knowing about image
|
|
165
|
+
* extensions.
|
|
166
|
+
*
|
|
167
|
+
* Silent unless BOTH sides name a recognized image format AND they differ. Nothing to compare —
|
|
168
|
+
* `-o -` (stdout, no filename to mislabel), an extension-less or non-image destination, an
|
|
169
|
+
* artifact URL with no usable extension — means no note, never a guessed one.
|
|
170
|
+
*
|
|
171
|
+
* `-o -` needs no special case of its own: the literal `-` carries no extension, so
|
|
172
|
+
* {@link formatFromPath} already returns undefined for it. An explicit early return was written
|
|
173
|
+
* first and then removed — no mutation of it could turn a test red, which is the definition of a
|
|
174
|
+
* line that guards nothing. The stdout case stays PINNED by a test either way.
|
|
175
|
+
*/
|
|
176
|
+
export function formatMismatchNote(imageUrl, dest) {
|
|
177
|
+
const requested = formatFromPath(dest);
|
|
178
|
+
if (requested === undefined)
|
|
179
|
+
return undefined;
|
|
180
|
+
const actual = capturedFormat(imageUrl);
|
|
181
|
+
if (actual === undefined || actual === requested)
|
|
182
|
+
return undefined;
|
|
183
|
+
const ext = dest.slice(dest.lastIndexOf('.'));
|
|
184
|
+
return (`Note: the screenshot was captured as ${actual}, but ${dest} names ${ext}. ` +
|
|
185
|
+
`The file holds ${actual} bytes — pass --format ${requested} to capture it as ${requested}.`);
|
|
186
|
+
}
|
|
102
187
|
/**
|
|
103
188
|
* The failure carried by the FLAT `error_code`/`error_message` fields, or undefined (#2792).
|
|
104
189
|
*
|
|
@@ -220,7 +305,15 @@ async function emitScreenshot(result, opts) {
|
|
|
220
305
|
process.exitCode = QUEUED_EXIT_CODE;
|
|
221
306
|
return;
|
|
222
307
|
}
|
|
223
|
-
|
|
308
|
+
const imageUrl = extractImageUrl(result);
|
|
309
|
+
// Advisory only — see {@link formatMismatchNote}. Emitted BEFORE the download so the note is
|
|
310
|
+
// never lost to a failing fetch, and to STDERR so `-o -` pipelines and stdout redirects stay
|
|
311
|
+
// byte-identical. It is deliberately unreachable from the not-ready branch above (no artifact
|
|
312
|
+
// exists, so there is no format to compare) and from `extractImageUrl`'s throw paths.
|
|
313
|
+
const note = formatMismatchNote(imageUrl, opts.output);
|
|
314
|
+
if (note !== undefined)
|
|
315
|
+
console.error(note);
|
|
316
|
+
await downloadArtifact(imageUrl, opts.output);
|
|
224
317
|
return;
|
|
225
318
|
}
|
|
226
319
|
printResult(result, opts.json);
|
package/dist/commands/search.js
CHANGED
|
@@ -6,8 +6,33 @@ import { printResult, handleError } from '../output.js';
|
|
|
6
6
|
*
|
|
7
7
|
* The CLI never surfaces provider-specific fields — only the ranked result
|
|
8
8
|
* columns the API returns.
|
|
9
|
+
*
|
|
10
|
+
* GUARD (#2956 item 1): `response.results` is typed as required, but the type describes what the
|
|
11
|
+
* API PROMISES, not what a given 200 actually carried — a proxy, a stub server, or a future
|
|
12
|
+
* envelope change can all answer `200 {"ok":true}`. Two distinct defects came out of trusting the
|
|
13
|
+
* type here:
|
|
14
|
+
*
|
|
15
|
+
* - a MISSING (or null) `results` drove the `for…of` below into
|
|
16
|
+
* `TypeError: response.results is not iterable`, which reached stderr as the CLI's whole
|
|
17
|
+
* diagnostic. It named an internal expression instead of the field, and left the caller with no
|
|
18
|
+
* next move;
|
|
19
|
+
* - a `results` that is a STRING is iterable, so it did not throw at all — it printed one garbled
|
|
20
|
+
* row per CHARACTER and exited 0, which is the quieter of the two failures.
|
|
21
|
+
*
|
|
22
|
+
* `Array.isArray` is therefore the gate (not a truthiness or length test), and it is checked
|
|
23
|
+
* BEFORE any row is written so a malformed body can never produce a half-rendered table. An EMPTY
|
|
24
|
+
* array stays a successful zero-hit search: it prints nothing and exits 0.
|
|
25
|
+
*
|
|
26
|
+
* The thrown error is a plain `Error` raised inside the action's `try`, so it reaches
|
|
27
|
+
* `handleError` — one `Error: `-prefixed stderr line, exit 1, no stack trace — exactly like every
|
|
28
|
+
* other command's unexpected-body diagnostic. It points at `--json`, which bypasses this renderer
|
|
29
|
+
* entirely and dumps the raw body, because inspecting that body is the caller's actual next step.
|
|
9
30
|
*/
|
|
10
31
|
function renderTable(response) {
|
|
32
|
+
if (!Array.isArray(response.results)) {
|
|
33
|
+
throw new Error('search: the response carries no "results" array — the server returned an unexpected 200 body. ' +
|
|
34
|
+
'Re-run with --json to inspect it.');
|
|
35
|
+
}
|
|
11
36
|
for (const result of response.results) {
|
|
12
37
|
console.log(`${result.position}. ${result.title}`);
|
|
13
38
|
console.log(` ${result.url}`);
|
package/dist/index.js
CHANGED
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { Command } from 'commander';
|
|
3
3
|
import { realpathSync } from 'node:fs';
|
|
4
|
-
import { createRequire } from 'node:module';
|
|
5
4
|
import { fileURLToPath } from 'node:url';
|
|
6
5
|
import { makeScrapeCommand } from './commands/scrape.js';
|
|
7
6
|
import { makeScreenshotsCommand } from './commands/screenshots.js';
|
|
@@ -13,12 +12,10 @@ import { makeSearchCommand } from './commands/search.js';
|
|
|
13
12
|
import { makeAccountCommand } from './commands/account.js';
|
|
14
13
|
import { makeUsageCommand } from './commands/usage.js';
|
|
15
14
|
import { makeWebhookDeliveriesCommand } from './commands/webhook-deliveries.js';
|
|
16
|
-
//
|
|
17
|
-
|
|
18
|
-
//
|
|
19
|
-
|
|
20
|
-
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
|
|
21
|
-
const version = typeof pkg.version === 'string' ? pkg.version : '0.1.0';
|
|
15
|
+
// Resolved from package.json at runtime (supports compiled output). Shared with `client.ts`'s
|
|
16
|
+
// `client_surface` marker (#2906) so `--version` and the emitted `cli/<version>` affix can never
|
|
17
|
+
// disagree.
|
|
18
|
+
import { PACKAGE_VERSION as version } from './package-version.js';
|
|
22
19
|
/** Construct the root CLI program. Exported so tests can inspect identity/structure without parsing argv. */
|
|
23
20
|
export function buildProgram() {
|
|
24
21
|
const program = new Command();
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Fallback `'0.1.0'` is carried over verbatim from the pre-#2906 inline resolution in `index.ts`.
|
|
3
|
+
* It cannot be reached from a correctly packed tarball; it exists so a malformed manifest degrades
|
|
4
|
+
* to a parseable semver instead of `undefined` leaking into `--version` output or onto the wire.
|
|
5
|
+
*/
|
|
6
|
+
export declare const PACKAGE_VERSION: string;
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import { createRequire } from 'node:module';
|
|
2
|
+
/**
|
|
3
|
+
* This package's own version, resolved at runtime from `package.json`.
|
|
4
|
+
*
|
|
5
|
+
* Two consumers share it so the CLI can never report one version to `--version` and affix another
|
|
6
|
+
* to its outgoing `client_surface` marker: `index.ts` (commander's `.version()`) and `client.ts`
|
|
7
|
+
* (the `cli/<version>` marker header, issue #2906 — spelled once, at that emit site).
|
|
8
|
+
*
|
|
9
|
+
* THIS MODULE MUST STAY DIRECTLY UNDER `src/`. `tsconfig.json` sets `rootDir: ./src` and
|
|
10
|
+
* `outDir: ./dist`, so `src/package-version.ts` compiles to `dist/package-version.js` and the
|
|
11
|
+
* `../package.json` below resolves to the PACKAGE ROOT from both trees. A copy of this block under
|
|
12
|
+
* `src/lib/` would compile to `dist/lib/`, where `../package.json` resolves to `dist/package.json`
|
|
13
|
+
* — a path that does not exist in the published tarball (`files` ships `dist` + `CHANGELOG.md`,
|
|
14
|
+
* not the manifest at that depth). `__tests__/entrypoint.test.ts` spawns the BUILT `dist` entry
|
|
15
|
+
* with `--version` and asserts the output equals the manifest version, so that dist-depth
|
|
16
|
+
* resolution is proven by an existing live test rather than argued.
|
|
17
|
+
*
|
|
18
|
+
* `createRequire(import.meta.url)` rather than a JSON import: the package is `type: module` and
|
|
19
|
+
* import attributes are still not uniformly available across the `engines.node: >=18` floor.
|
|
20
|
+
*/
|
|
21
|
+
const require = createRequire(import.meta.url);
|
|
22
|
+
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
|
|
23
|
+
const pkg = require('../package.json');
|
|
24
|
+
/**
|
|
25
|
+
* Fallback `'0.1.0'` is carried over verbatim from the pre-#2906 inline resolution in `index.ts`.
|
|
26
|
+
* It cannot be reached from a correctly packed tarball; it exists so a malformed manifest degrades
|
|
27
|
+
* to a parseable semver instead of `undefined` leaking into `--version` output or onto the wire.
|
|
28
|
+
*/
|
|
29
|
+
export const PACKAGE_VERSION =
|
|
30
|
+
// eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
|
|
31
|
+
typeof pkg.version === 'string' ? pkg.version : '0.1.0';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@hunt-labs/bytekit-cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "ByteKit CLI — wraps @hunt-labs/bytekit-sdk as shell subcommands",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -41,7 +41,7 @@
|
|
|
41
41
|
],
|
|
42
42
|
"dependencies": {
|
|
43
43
|
"commander": "^12.0.0",
|
|
44
|
-
"@hunt-labs/bytekit-sdk": "^0.
|
|
44
|
+
"@hunt-labs/bytekit-sdk": "^0.7.1"
|
|
45
45
|
},
|
|
46
46
|
"scripts": {
|
|
47
47
|
"build": "rm -rf dist tsconfig.tsbuildinfo && tsc",
|