@hunt-labs/bytekit-cli 0.5.0 → 0.6.0

package/CHANGELOG.md

@@ -8,8 +8,113 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 This file records only the delta between versions. For how to use the package
 today, see [README.md](./README.md), which remains the authoritative usage guide.
 
+## [Unreleased]
+
+## [0.6.0] - 2026-07-29
+
+### Breaking
+
+- **`screenshots create --async` now exits `2` in human mode instead of `0`.** The queued 202 is
+  not a completed capture — there is no artifact yet — and every other queued path in the CLI
+  already signalled that with exit code `2` (`scrape create --async`). A script that relied on
+  exit `0` to mean "the screenshot is ready" was reading a false success; under `set -e` such a
+  script now stops at the queue step, which is the intended correction. The envelope itself is
+  unchanged and still printed. Two carve-outs: **`--json` keeps exit `0`** (the envelope is a
+  complete machine-readable payload — identical to `scrape create --async --json`), and
+  `--async -o <file>` also exits `2` while still writing no file (it prints the queued envelope
+  and skips the download, as before). Poll with `bytekit screenshots get <id>`.
+- **A `--file` per-item key outside a bulk endpoint's own strict schema now fails the whole
+  create with a `422`** instead of silently succeeding with that key ignored. In 0.5.x, `--file`
+  never read anything but `url` from a JSON line — every other key was dropped client-side
+  before the request was ever built. As of 0.6.0 those keys ride to the server (`items` on
+  `bulk create` / `scrape bulk create`; inside `urls` on `fetch bulk create`), and `/v1/bulk` /
+  `/v1/scrape/bulk` reject any key their `.strict()` per-item schema doesn't recognize. A file
+  that worked in 0.5.x purely because its extra keys were quietly discarded may now `422` — see
+  each command's `--file` help for the keys actually honored on that endpoint.
+- **`fetch bulk create --file` per-item keys outside `url`/`format`/`country`/`cache_ttl` now
+  fail the whole create with a `422`** instead of silently succeeding with that key ignored.
+  `/v1/fetch/bulk`'s per-item override schema (inside `urls[]`) is now `.strict()` on the
+  gateway (issue #2648), matching the `/v1/bulk` / `/v1/scrape/bulk` posture above. The
+  `--file` help text for `fetch bulk create` is now uniform with `bulk create` /
+  `scrape bulk create` and no longer enumerates keys — this entry is where the honored set
+  (`url`, `format`, `country`, `cache_ttl`) is documented instead.
+
+### Added
+
+- **Per-item bulk configuration on all three bulk creates** (`bulk create`, `scrape bulk create`,
+  `fetch bulk create`). A `--file` line may now be a JSON object carrying `url` plus per-item
+  fields instead of only a bare URL; such a file is sent as `items` on `/v1/bulk` and
+  `/v1/scrape/bulk`, and as a heterogeneous `urls` array on `/v1/fetch/bulk` (which has no
+  `items` property). A line whose only key is `url` still normalizes to a bare URL string, so a
+  file of url-only lines produces a byte-identical request to before. A JSON line with a missing
+  or non-string `url` is now rejected locally with a usage error (exit `1`) instead of being sent
+  to the server as a URL. (See Breaking, above, for the per-item strict-schema behavior change
+  this enables.)
+- `--defaults`, `--webhook-secret` and `--metadata` on all three bulk creates, plus `--custom` on
+  `scrape bulk create` only (`/v1/bulk` and `/v1/fetch/bulk` have no top-level `custom`, so the
+  flag would be rejected or stripped there). `--defaults` is forwarded verbatim; the server owns
+  precedence and unknown-key handling.
+- The full markdown-tuning surface on `fetch create`: `--markdown-query`, `--markdown-links`,
+  `--markdown-images`, `--with-links-summary`, `--with-images-summary`, `--markdown-compact`,
+  `--markdown-filter-images`, `--markdown-include-media`, `--markdown-include-warnings`,
+  `--markdown-include-stats`, and `--custom`.
+- Webhook delivery and processing flags on `sitemap create`: `--webhook-url`, `--webhook-secret`,
+  `--cache-ttl`, `--compact`, `--process`, `--metadata`. Webhook-driven sitemap crawls were
+  previously unreachable from the shell.
+- `--headers`, `--cookies` and `--metadata` on `screenshots create`.
+- `--metadata` on `monitors create`, and `--metadata` + `--url` on `monitors update`. `--type`
+  remains deliberately absent from `update` — it is immutable server-side.
+
+## [0.5.1] - 2026-07-28
+
+### Fixed
+
+- **Corrects the scope of the 0.5.0 "stalled body" fix claim.** `screenshots create`/`get -o`
+  downloads the `image_url` artifact bytes with a bare `globalThis.fetch` call
+  (`lib/download-artifact.ts`) that sat entirely outside the transport seam `sendWithRetry` owns —
+  no abort signal, no connection-error taxonomy. The 0.5.0 entry below ("The per-attempt request
+  timeout now covers the response body read... a server that answered with headers and then
+  stalled its body... hung the CLI forever", #2546) reads as if it covered every response body; it
+  only ever covered API-call bodies read through the JSON/raw transports. A stalled artifact host
+  (headers, then a body that never completes) still hung the CLI indefinitely until now. The
+  download path now shares the same per-attempt abort budget and `ByteKitConnectionError` timeout
+  taxonomy as the API transport (`createAttemptBudget` / `toConnectionError`, exported from
+  `client.ts` — no second timeout implementation), with the same 120 s per-attempt default and no
+  retry (artifact GETs are not the API's 429 domain). Partial-file policy: the response is
+  buffered fully in memory before any write, so an aborted download never leaves a
+  partial/truncated file at the `-o` destination. **User-visible consequence:** a legitimately
+  slow (but not stalled) large artifact download now hard-fails after 120 s instead of waiting
+  indefinitely, and there is currently no CLI flag to raise that budget for a single command.
+  (#2587)
+- While in the file: the 429-retry path left its (unread) response body undrained before sleeping
+  and retrying — undici cannot reclaim the underlying socket for connection-pool reuse until a body
+  is fully read. The body is now drained with `response.text()` on that path, bounded by its own
+  short (2 s) cap independent of the 120 s per-attempt budget, so a 429 whose body stalls can no
+  longer delay the retry by anywhere near the full attempt budget (or, with the budget disabled,
+  forever). **Correction to an intermediate `Unreleased` draft of this same fix:** that draft's cap
+  only bounded how long the _retry_ waited — it resolved on schedule but never touched the
+  abandoned `response.text()` call itself, which kept reading against a live socket indefinitely.
+  Because every CLI success path exits via a soft `process.exitCode` (#2544) rather than
+  `process.exit()`, that orphaned read kept the whole process alive forever — strictly worse than
+  having no drain at all, and on the exact class of hang this ticket exists to eliminate. The cap
+  now also **aborts** the request when it fires, so the abandoned read is actually torn down, not
+  merely ignored; this holds even when the per-attempt timeout itself is disabled
+  (`timeoutMs <= 0`). **User-visible consequence:** a 429 with a stalled body still costs the retry
+  at most ~2 s, exactly as before, but the CLI process now reliably exits afterward instead of
+  hanging forever regardless of that visible latency. The same discard-drain on the
+  artifact-download's non-2xx path is removed outright rather than capped: that path throws
+  straight into `process.exit(1)`, so there is no subsequent request in the process for a reclaimed
+  socket to serve, and the drain there bought nothing. (#2587)
+
 ## [0.5.0] - 2026-07-24
 
+### Note
+
+- The `0.3.1` and `0.4.0` versions below document real, tagged changes but were
+  never published to npm — the registry's version history jumps straight from
+  `0.3.0` to `0.5.0`. Don't go looking for those tarballs; the changes they
+  describe ship for the first time in this release.
+
 ### Added
 
 - Surface parity round 2 (#2545): every `x-sdk-scope: v0.1` endpoint the SDK
@@ -236,7 +341,11 @@ Additive, lower-risk release — no breaking changes to existing commands.
 - Initial release of the ByteKit CLI, wrapping the ByteKit SDK as shell
   subcommands, published under the original `rapidcrawl-cli` name.
 
+[Unreleased]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
+[0.5.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
+[0.5.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
 [0.4.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
+[0.3.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
 [0.3.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
 [0.2.2]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli
 [0.2.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/cli

package/README.md

@@ -182,6 +182,11 @@ Both verbs accept `--country <cc>`, `--timeout-ms <n>`, and `--cache-ttl <ttl>`
 bytekit search "ai web scraping" --limit 10 --country us
 ```
 
+`search` also accepts `--type <type>` (`web` (default), `news`, or `images`),
+`--language <code>` (two-letter language code, e.g. `en`, `fr`), and
+`--date-range <window>` (recency filter: `any` (default), `hour`, `day`,
+`week`, `month`, `year`). Each flag is omitted from the request when absent.
+
 ### `screenshots` — capture a page
 
 ```bash
@@ -232,7 +237,7 @@ bytekit monitors create --url https://example.com/pricing --interval-type daily
 bytekit monitors list --limit 25 --status active
 bytekit monitors update mon_123abc --status paused    # pause
 bytekit monitors update mon_123abc --status active    # resume
-bytekit monitors captures list mon_123abc
+bytekit monitors captures list mon_123abc --limit 25
 bytekit monitors delete mon_123abc
 ```
 
@@ -240,6 +245,10 @@ bytekit monitors delete mon_123abc
 (`active`, `processing`, `paused`, `cancelled`, `suspended`) as query filters,
 each omitted when absent.
 
+`monitors captures list <id>` accepts `--limit <n>` and `--cursor <cursor>` to
+page through a monitor's capture history — pass the response's `next_cursor`
+back as `--cursor` to fetch the next page. Both flags are omitted when absent.
+
 `monitors update` gains `--status` (`active` | `paused` — pause and resume from
 the shell), `--change-threshold <pct>` (accepts a float), `--notify-on`
 (`change`, `every`), `--webhook-secret <secret>`, and three JSON-object flags:
@@ -294,11 +303,24 @@ omitted when absent. `retry <id>` requeues a failed or exhausted delivery.
 
 ## Exit codes
 
-| Code | Meaning                                                                                                                                                                                                                                                                                                            |
-| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `0`  | Success.                                                                                                                                                                                                                                                                                                           |
-| `1`  | Usage error, or an API/connection error. The reason is on stderr, always prefixed `Error: `.                                                                                                                                                                                                                       |
-| `2`  | The scrape was accepted but is still **queued**, so there is no content body to emit — `scrape create --async` with `--raw` or a bare `-o` (which writes the bare `sc_` id to stdout), and the same command in human mode. Not a failure: the job was accepted and billed. Poll it with `bytekit scrape get <id>`. |
+| Code | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `0`  | Success.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `1`  | Usage error, or an API/connection error.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `2`  | The job was accepted but is still **queued**, so there is no content body or artifact to emit. Applies to `scrape create --async` with `--raw` or a bare `-o` (which writes the bare `sc_` id to stdout) and in human mode, and to `screenshots create --async` in human mode (including with `-o`, which deliberately writes no file). Not a failure: the job was accepted and billed. Poll it with `bytekit scrape get <id>` / `bytekit screenshots get <id>`. **`--json` keeps exit `0` in both cases** — the queued envelope is a complete machine-readable payload. |
+
+The stderr message for exit code `1` has two shapes, depending on where the
+error originates:
+
+- **App-level errors** (a rejected API call, a connection failure, a missing
+  key) are always prefixed `Error: ` (uppercase) — e.g.
+  `Error: API key required. Pass --key or set BYTEKIT_API_KEY.`
+- **Commander-level usage errors** (an unknown command, a missing required
+  argument, an unrecognized flag) come from the `commander` library itself and
+  use its own lowercase `error: ` prefix — e.g. `error: unknown command 'bogus'`.
+
+Scripts that grep stderr for a failure reason should match case-insensitively,
+or handle both prefixes.
 
 Code `2` is deliberately distinct from both `0` and `1` so a script can branch
 on "queued" without parsing stderr:

package/dist/client.d.ts

@@ -1,4 +1,4 @@
-import { ByteKit } from '@hunt-labs/bytekit-sdk';
+import { ByteKit, ByteKitConnectionError } from '@hunt-labs/bytekit-sdk';
 export interface GlobalOptions {
     key?: string | undefined;
     baseUrl?: string | undefined;
@@ -22,6 +22,24 @@ export declare const MIN_RETRY_AFTER_MS = 1000;
 export declare const MAX_RETRY_AFTER_MS = 60000;
 /** Exactly one retry. A second 429 is terminal — no loop. */
 export declare const MAX_RETRIES = 1;
+/**
+ * Cap on a best-effort, discard-only body drain (#2587 cr2-1). Independent of the per-attempt
+ * abort budget: the 429 drain below (`response.text().catch(() => {})`) exists ONLY so undici can
+ * return the socket to the connection pool — nothing downstream reads the drained bytes — but
+ * binding that read to the FULL per-attempt budget (120 s default, or, under `timeoutMs <= 0`,
+ * to no bound at all) let a stalled 429 body burn the whole budget, or hang forever, before a
+ * retry that needs nothing from that body. Short enough that a legitimately stalled 429 body costs
+ * the retry very little wall-clock time.
+ *
+ * #2587 cr3-2: past this cap, the read is not merely abandoned — {@link drainBestEffort} actively
+ * ABORTS its underlying request. Resolving the race without aborting anything (the earlier
+ * cr2-1 shape) let the retry proceed on schedule while the abandoned `response.text()` kept
+ * reading against a live socket forever, which — because the CLI exits via a soft
+ * `process.exitCode` (#2544) rather than `process.exit()` — kept the whole process alive
+ * indefinitely. `.catch`-guarding the read only made it safe to abandon the RESULT; it never made
+ * abandoning the underlying READ safe.
+ */
+export declare const DRAIN_TIMEOUT_MS = 2000;
 /**
  * Default CLIENT-SIDE abort budget for a single fetch attempt, in ms.
  *
@@ -33,6 +51,49 @@ export declare const MAX_RETRIES = 1;
  * exists.
  */
 export declare const DEFAULT_TIMEOUT_MS = 120000;
+/**
+ * A per-attempt abort budget: an `AbortController`, optionally fired by a `setTimeout`.
+ *
+ * The budget spans the WHOLE attempt — time-to-headers **and** the response body read (#2546).
+ * Arming it only until headers arrive left `response.text()` / `response.arrayBuffer()` unbounded,
+ * so a server that answered with headers and then stalled its body hung the CLI forever.
+ *
+ * The controller ALWAYS exists, even when `ms <= 0` — only its own timeout timer is conditional
+ * on `ms > 0`. `sendWithRetry`'s 429-drain cap ({@link DRAIN_TIMEOUT_MS}, via the returned
+ * `abort()`) needs a live controller to tear down an abandoned body read regardless of whether the
+ * OVERALL per-attempt budget is enabled — `ms <= 0` must still mean "no full-attempt timeout", not
+ * "no controller to abort a stalled drain with" (#2587 cr3-2: the earlier drain fix abandoned the
+ * read entirely rather than aborting it, so the socket — and the event loop — stayed alive
+ * indefinitely, worse than the pre-fix baseline it replaced).
+ *
+ * Deliberately NOT `AbortSignal.timeout` (which the SDK prefers): that schedules on a timer the
+ * test runner's fake timers cannot advance, so the transport's timeout would be untestable under
+ * `vi.useFakeTimers()`. CLI requests are short-lived and single-shot, so the GC advantage
+ * `AbortSignal.timeout` buys the long-running SDK does not apply here. The timer is `unref`'d so a
+ * pending budget never keeps the process alive, and `cleanup()` clears it on every exit path.
+ *
+ * Exported so `lib/download-artifact.ts` can reuse the same budget for the artifact-download path
+ * (#2587) rather than growing a second timeout implementation — that path sits entirely outside
+ * `sendWithRetry` (it is not a JSON/raw API call), so it would otherwise never inherit this bound.
+ */
+export declare function createAttemptBudget(ms: number): {
+    signal: AbortSignal;
+    cleanup: () => void;
+    abort: (reason: unknown) => void;
+};
+/**
+ * Map a `fetch` or body-read rejection (no usable HTTP payload was produced) onto the SDK's
+ * `ByteKitConnectionError` taxonomy, so a CLI failure carries the same `{name, code, cause}` a
+ * direct SDK call would.
+ *
+ * `timedOut` is true when this attempt's own budget fired the abort, which is the reliable
+ * discriminator (undici rejects with the signal's reason — or a wrapper carrying it, see
+ * {@link abortNameOf} — whose `name` we also inspect as a fallback for a caller-style AbortError).
+ *
+ * Exported so `lib/download-artifact.ts` reuses the same taxonomy for the artifact-download path
+ * (#2587) instead of hand-rolling a second error mapping.
+ */
+export declare function toConnectionError(err: unknown, method: string, path: string, timedOut: boolean): ByteKitConnectionError;
 /**
  * Parse a `Retry-After` header into a clamped millisecond delay.
  *
@@ -48,6 +109,18 @@ export declare function parseRetryAfterMs(raw: string | null): number;
  * property of the client (see {@link buildClient}) routes all JSON commands through it from one
  * seam. `/v1/fetch` does NOT ride this transport — it rides `requestRaw` (see
  * {@link createRetryingRequestRaw}); `/v1/fetch/bulk` is a JSON job envelope and does.
+ *
+ * DECISION (#2586 dod-4): the 4th `RequestOptions` parameter of the SDK's contract is accepted
+ * here — so the signature matches `ByteKit['request']` exactly, closing the gap where a caller-typo
+ * of a 5th positional argument would silently no-op instead of a compile error — but deliberately
+ * NOT forwarded into `sendWithRetry`. Two reasons: (1) no CLI command constructs a `RequestOptions`
+ * today (no `--timeout` flag exists yet, and `signal` has no CLI-level source), so the parameter is
+ * always `undefined` in practice; (2) the CLI already owns a per-attempt abort budget with its own
+ * semantics (`createAttemptBudget`, `timeoutMs` above) that would need explicit composition with a
+ * caller `signal` to honor `RequestOptions` correctly (the SDK's `composeSignals`) — inventing that
+ * unexercised composition here, with no caller able to reach it, would be undertested surface for no
+ * present behavior change. If a future CLI flag needs per-call overrides, wire it through
+ * `sendWithRetry`'s `timeoutMs` (already per-attempt) at that point, with its own tests.
  */
 export declare function createRetryingRequest(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['request'];
 /**
@@ -62,8 +135,32 @@ export declare function createRetryingRequest(apiKey: string, baseUrl: string, t
  * so `FetchResource` builds an ordinary `FetchResult` from it and the CLI never forks fetch
  * handling away from the SDK. `bytes` is byte-exact (binary-safe); `body` is the best-effort UTF-8
  * decode, matching the SDK's own semantics.
+ *
+ * The 4th `RequestOptions` parameter is accepted but not forwarded — see the DECISION note on
+ * {@link createRetryingRequest}, which applies here verbatim. `FetchResource.create`/`get` DO thread
+ * a `requestOptions` argument through to this seam (unlike every other resource), but no CLI `fetch`
+ * command constructs one today (no `--timeout` flag), so it is still always `undefined` in practice.
  */
 export declare function createRetryingRequestRaw(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['requestRaw'];
+/**
+ * Build the CLI's header-carrying JSON transport: the SDK's `requestWithResponse` contract
+ * (`{data, headers, status}`, #2547), plus the same bounded 429 retry every other CLI transport
+ * gets.
+ *
+ * Unused by any CLI command today — `screenshots.createWithResponse` / `getWithResponse` and
+ * `sitemap.getWithResponse` all exist on the SDK surface but no command builds on them yet. It is
+ * still installed (see {@link buildClient}) rather than left un-overridden: the transport-coverage
+ * contract test (#2586 dod-3) derives its method list from the real SDK prototype, and leaving a
+ * transport method un-overridden is exactly how #2535 shipped broken — the first command that DOES
+ * call a `*WithResponse` resource method must not silently ride the bare SDK transport and skip the
+ * CLI's retry/timeout/error taxonomy.
+ *
+ * Same body-read shape as {@link createRetryingRequest} (204/empty → `undefined`, `JSON.parse`
+ * otherwise), with `headers` and `status` threaded through from the successful response alongside
+ * the parsed `data`. The 4th `RequestOptions` parameter is accepted but not forwarded — see the
+ * DECISION note on {@link createRetryingRequest}.
+ */
+export declare function createRetryingRequestWithResponse(apiKey: string, baseUrl: string, timeoutMs?: number): ByteKit['requestWithResponse'];
 /**
  * Build an authenticated SDK client from global CLI options.
  *
@@ -72,9 +169,18 @@ export declare function createRetryingRequestRaw(apiKey: string, baseUrl: string
  *   baseUrl: --base-url flag → BYTEKIT_BASE_URL → DEFAULT_BASE_URL.
  * Exits with a descriptive error when no key is available.
  *
- * The returned client's `request` AND `requestRaw` are both replaced by the retrying transports, so
- * every command inherits the 429 retry — JSON commands via `request`, the `/v1/fetch` content
- * commands via `requestRaw`. Overriding only `request` is the #2535 defect: the SDK's fetch
- * resource rides `requestRaw`, so fetch bypassed the retry and the CLI's error taxonomy entirely.
+ * The returned client's `request`, `requestRaw`, AND `requestWithResponse` are all replaced by the
+ * retrying transports, so every command inherits the 429 retry — JSON commands via `request`, the
+ * `/v1/fetch` content commands via `requestRaw`, and the header-carrying `*WithResponse` surface via
+ * `requestWithResponse`. Overriding only `request` is the #2535 defect: the SDK's fetch resource
+ * rides `requestRaw`, so fetch bypassed the retry and the CLI's error taxonomy entirely; leaving
+ * `requestWithResponse` un-overridden is the same class of defect (#2586) — it is unused by any CLI
+ * command TODAY, but a future command built on `screenshots.createWithResponse` /
+ * `sitemap.getWithResponse` would otherwise silently ride the bare SDK transport.
+ *
+ * `baseUrl` is normalized ONCE, here, before it reaches either the SDK constructor or the CLI's own
+ * transport builders below — the CLI's transports build their request URL from this local variable
+ * directly (they do not call the SDK's own normalized `send()`), so a trailing-slash flag would
+ * otherwise double-slash every request the CLI makes (#2586).
  */
 export declare function buildClient(opts: GlobalOptions): ByteKit;