@hunt-labs/bytekit-sdk 0.2.2 → 0.4.0

package/CHANGELOG.md

@@ -8,6 +8,122 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 This file records only the delta between versions. For how to use the package
 today, see [README.md](./README.md), which remains the authoritative usage guide.
 
+## [0.4.0] - 2026-07-24
+
+### Added
+
+- `RequestOptions` (per-call `requestTimeoutMs` + caller `signal`) is now accepted by
+  **every** resource method, not just `scrape.create` / `scrape.get`. It is always a
+  trailing optional parameter — on `screenshots.create` it is the third positional, after
+  `params` — so every existing call site compiles and serializes byte-identically. Per-call
+  options continue to beat the constructor's `timeoutMs` default; omitting them leaves the
+  constructor default in force exactly as before. (#2547)
+- `ApiResponse<T>` (`{ data, headers, status }`), a generic `client.requestWithResponse<T>()`
+  transport, and three typed siblings — `sitemap.getWithResponse`,
+  `screenshots.createWithResponse`, `screenshots.getWithResponse` — make the documented
+  `X-Raw-Bytes`, `X-Billable-Bytes`, `X-Billing-Multiplier` and `X-Screenshot-*` response
+  headers reachable. This surface is purely **additive**: it issues the identical request
+  with the identical options as the base method, and no existing return type changes.
+  (#2547)
+- `MonitorStatus` and `ScrapeCookie` are exported from the package entrypoint. (#2547)
+
+### Changed (**BREAKING** at compile time)
+
+- `ScrapeOpts.cookies` is now `ScrapeCookie[]`, derived from the spec's `Cookie` schema
+  (`name` and `value` required; `domain`, `path`, `secure`, `httpOnly` optional), instead of
+  an open bag of untyped records. A cookie missing `name` used to compile and fail as a 422
+  on the wire; it is a compile error now. Serialization is unchanged. (#2547)
+- `ListMonitorsParams.status` is now the spec enum
+  (`'active' | 'processing' | 'paused' | 'cancelled' | 'suspended'`), derived from the
+  generated `listMonitors` query type, instead of `string`. A typo like `'activee'` used to
+  compile and 422; it is a compile error now. The serialized query string is unchanged.
+  (#2547)
+
+### Fixed
+
+- `timeoutMs` / `requestTimeoutMs` (and a caller-supplied `signal`) now cover the
+  **response body read**, not just the time to first byte of headers. The abort budget was
+  armed for `fetch` and cleared the moment headers arrived, after which `request()`'s
+  `response.text()`, `requestRaw()`'s `response.arrayBuffer()`, and the error-body
+  `response.json()` on a non-2xx all read the body unbounded — so a server that answered
+  with headers and then stalled its body hung the caller forever. A mid-body abort now
+  surfaces as `ByteKitConnectionError` with `code: 'timeout'` (budget) or `'aborted'`
+  (caller signal) instead of a bare `TypeError: terminated`.
+
+  The default is **unchanged at 120 s**, and the precedence table for
+  `signal` / `requestTimeoutMs` / `timeoutMs` is untouched. One consequence worth planning
+  for: a legitimately large body (a multi-MB `/v1/fetch` document) now counts against that
+  same budget, where previously only the wait for headers did. (#2546)
+
+- The README's webhook-retry sample used an invalid delivery id prefix (`whd_`) that its own
+  branded type rejects. The sample is corrected, and a new test compiles every TypeScript
+  sample in the README against the SDK's sources so a non-compiling sample can no longer
+  ship. (#2547)
+
+### Fixed (**BREAKING** at compile time)
+
+- `fetch.get` no longer accepts the eleven markdown-tuning options (`markdown_mode`,
+  `markdown_query`, `markdown_links`, `markdown_images`, `with_links_summary`,
+  `with_images_summary`, `markdown_compact`, `markdown_filter_images`,
+  `markdown_include_media`, `markdown_include_warnings`, `markdown_include_stats`).
+  The spec's `GET /v1/fetch` declares only `url`, `format`, `country`, `timeout_ms`
+  and `cache_ttl`, so those options were silently dropped by the query builder — a
+  caller asking for `markdown_mode: 'llm'` compiled and received default article-mode
+  markdown. They are now compile errors instead of silent no-ops; use `fetch.create`
+  (POST), which accepts and serializes all of them unchanged. `FetchGetOpts` is now
+  derived from the generated `getFetch` query type, so future spec drift fails
+  `tsc --noEmit` rather than reaching production. (#2539)
+
+## [0.3.0] - 2026-07-21
+
+### Removed (**BREAKING**)
+
+- The deprecated `RapidCrawl`, `RapidCrawlError`, and `RapidCrawlConnectionError`
+  exports and the `RapidCrawlOptions` type alias (kept as a one-minor-version
+  bridge in #2421) are removed. Import the `ByteKit*` names instead. (#2473)
+- The `RecordingId` type export is removed. Recordings are deactivated
+  (`/v1/recordings` returns 410), so the branded id had no reachable use.
+  Breaking in theory only — unused in practice. (#2481)
+
+### Fixed
+
+- `fetch.create` / `fetch.get` no longer throw a bare `SyntaxError` on every
+  non-JSON body. `/v1/fetch` returns the fetched content AS the response body
+  (metadata in `X-Fetch-*` headers), but both methods routed through the
+  JSON-parsing transport. They now use a raw-capable transport and resolve to a
+  typed `FetchResult` — raw `body`, byte-exact `bytes`, plus `id`,
+  `creditsCharged`, `cache`, `custom`, `contentType`, and `headers` parsed from
+  the `X-Fetch-*` metadata. (#2466)
+- A trailing slash on the constructor `baseUrl` (e.g. `"https://api.bytekit.com/"`)
+  no longer produces double-slash request URLs — it is normalized once, preserving
+  a path-carrying baseUrl. (#2481)
+- `SearchResult.date` is now optional (`string | null | undefined`): `images`
+  search results omit the field entirely. (#2481)
+
+### Added
+
+- New `usage` resource (`get`, `daily`, `byEndpoint` — `GET /v1/usage`,
+  `/v1/usage/daily`, `/v1/usage/by-endpoint`) and `webhooks` resource (`list`,
+  `retry` — `GET /v1/webhook-deliveries`, `POST /v1/webhook-deliveries/{id}/retry`),
+  with their request/response types exported. (#2471)
+- `ScrapeOpts` and `FetchOpts` now cover the full documented request field set
+  (10 previously missing `ScrapeRequest` fields; 11 markdown-tuning
+  `FetchRequest` fields), plus the `'text'` markdown-links mode. A compile-time
+  drift guard keeps them aligned with the generated OpenAPI schema. (#2475)
+- `monitors`, `sitemap`, `bulk`, and `fetch.bulk` methods are now fully typed from
+  the generated OpenAPI `operations` (previously `Record<string, unknown>` in /
+  `Promise<unknown>` out); their request/response types are re-exported. (#2481)
+- `monitors.list` accepts typed query params and returns a typed response
+  (`ListMonitorsParams` / `ListMonitorsResponse`, both exported). (#2480)
+
+### Changed (**BREAKING**)
+
+- `fetch.get` now rejects a `custom` option at compile time: `GET /v1/fetch` has
+  no `custom` query param, so the value was silently dropped before. Use
+  `fetch.create` (POST) to echo a `custom` payload. (#2481)
+- `FetchOpts.format` is narrowed from `string` to the spec union
+  `'markdown' | 'html'`; off-spec values now fail to compile. (#2475)
+
 ## [0.2.2] - 2026-07-16
 
 ### Changed
@@ -85,6 +201,7 @@ against 0.1.0 — read each one before upgrading.
   (scrape, screenshot, search, and related endpoints), published under the original
   `@rapidcrawl/sdk` name.
 
+[0.3.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/sdk-typescript
 [0.2.2]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/sdk-typescript
 [0.2.1]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/sdk-typescript
 [0.2.0]: https://github.com/Hunt-Labs-Inc/ByteKit/tree/main/packages/sdk-typescript

package/README.md

@@ -27,16 +27,15 @@ Create an API key at [app.bytekit.com](https://app.bytekit.com) (keys are
 prefixed `sk_live_`) and pass it to the client constructor:
 
 ```ts
-import { RapidCrawl } from '@hunt-labs/bytekit-sdk';
+import { ByteKit } from '@hunt-labs/bytekit-sdk';
 
-const client = new RapidCrawl({ apiKey: process.env.BYTEKIT_API_KEY! }); // sk_live_...
+const client = new ByteKit({ apiKey: process.env.BYTEKIT_API_KEY! }); // sk_live_...
 ```
 
 The key is sent as `Authorization: Bearer <apiKey>` on every request. Keep it
 server-side — it is not safe to ship in a browser bundle.
 
-> The client class is named `RapidCrawl` for historical reasons (ByteKit's
-> previous product name). It is the SDK's only entrypoint class.
+> `ByteKit` is the SDK's only entrypoint class.
 
 ## Base-URL override
 
@@ -44,7 +43,7 @@ Requests go to `https://api.bytekit.com` by default. Point the client at
 staging, a local gateway, or a proxy with the `baseUrl` option:
 
 ```ts
-const client = new RapidCrawl({
+const client = new ByteKit({
   apiKey: process.env.BYTEKIT_API_KEY!,
   baseUrl: 'https://api-stg.bytekit.com', // default: https://api.bytekit.com
 });
@@ -69,8 +68,27 @@ const scrape = await client.scrape.get('sc_123abc');
 
 ### Raw fetch (unprocessed response body)
 
+`/v1/fetch` returns the fetched content **as the body**, with metadata in
+`X-Fetch-*` response headers. `fetch.create` / `fetch.get` resolve a
+`FetchResult` — `body` is the raw content string (a JSON-serving upstream
+comes back as the raw JSON string, never a parsed object), and the named
+fields decode the metadata headers. `body` is UTF-8-decoded, so for true
+binary content (images, PDFs, etc. — `/v1/fetch` supports raw passthrough)
+use `bytes` instead, which is byte-for-byte identical to the wire content:
+
 ```ts
-const raw = await client.fetch.create({ url: 'https://example.com' });
+const res = await client.fetch.create({ url: 'https://example.com' });
+
+console.log(res.body); // raw content (e.g. "<!doctype html>…") — UTF-8 decoded
+console.log(res.bytes); // Uint8Array — byte-exact, safe for binary content
+console.log(res.id); // X-Fetch-ID, e.g. "ft_0123abcd…"
+console.log(res.creditsCharged); // X-Fetch-Credits-Charged, as a number
+console.log(res.cache); // "hit" | "miss" | "bypass"
+console.log(res.custom); // base64 X-Fetch-Custom echo, or null
+console.log(res.contentType); // response Content-Type, e.g. "text/html"
+
+// Anything not surfaced as a named field is on res.headers:
+console.log(res.headers.get('X-Fetch-Waf-Vendor'));
 ```
 
 ### Web search
@@ -124,7 +142,10 @@ await client.bulk.screenshots.list('bulk_123abc');
 ```ts
 const monitor = await client.monitors.create({
   url: 'https://example.com/pricing',
+  type: 'screenshot',
   interval_type: 'daily',
+  change_threshold: 5,
+  notify_on: 'change',
   webhook_url: 'https://your.app/hooks/bytekit',
 });
 await client.monitors.list();
@@ -134,7 +155,17 @@ await client.monitors.delete('mon_123abc');
 ### Sitemap crawl
 
 ```ts
-const crawl = await client.sitemap.create({ url: 'https://example.com' });
+// `url` is the only required field — every other option can be omitted.
+const minimal = await client.sitemap.create({ url: 'https://example.com' });
+
+const crawl = await client.sitemap.create({
+  url: 'https://example.com',
+  strategy: 'standard',
+  max_depth: 2,
+  max_urls: 5000,
+  cache_ttl: '30d',
+  compact: false,
+});
 const sitemap = await client.sitemap.get('sm_123abc');
 ```
 
@@ -143,20 +174,59 @@ const sitemap = await client.sitemap.get('sm_123abc');
 ```ts
 const account = await client.account.get();
 const keys = await client.account.apiKeys.list();
+
+// Usage (Bearer API key auth)
+const usage = await client.usage.get();
+const daily = await client.usage.daily({ from: '2026-01-01', to: '2026-01-31' });
+const byEndpoint = await client.usage.byEndpoint();
+
+// Webhook deliveries
+const deliveries = await client.webhooks.list();
+await client.webhooks.retry('wh_123abc');
 ```
 
+> **Note:** `client.account.*` (including `client.account.apiKeys.*`) targets the
+> `/v1/account/*` endpoints, which authenticate with a Clerk dashboard session —
+> **not** a Bearer API key. Calling them with an API-key client will fail auth.
+> They remain on the SDK surface for dashboard/session consumers and are marked
+> `x-sdk-scope: internal` in the spec, so they are intentionally excluded from the
+> v0.1 Bearer-key parity guarantee.
+
+### Response headers
+
+The API carries billing and capture metadata in `X-*` response headers alongside the
+JSON body — `X-Raw-Bytes`, `X-Billable-Bytes`, `X-Billing-Multiplier`, and the
+`X-Screenshot-*` family. Every resource method returns just the parsed body, so use a
+`*WithResponse` sibling when you also need the headers:
+
+```ts
+const { data, headers, status } = await client.sitemap.getWithResponse('sm_123abc');
+
+console.log(data.status); // exactly what sitemap.get('sm_123abc') resolves to
+console.log(headers.get('X-Raw-Bytes')); // e.g. "12345"
+console.log(status); // 200
+```
+
+`client.screenshots.createWithResponse()` and `client.screenshots.getWithResponse()`
+work the same way. Any other endpoint's headers are reachable through the generic
+transport, `client.requestWithResponse('GET', '/v1/scrape/sc_123abc')`.
+
+These siblings are purely additive: they issue the identical request and accept the
+identical options as their base methods, and `data` is exactly what the base method
+resolves to — no existing return type changes.
+
 ## Error handling
 
-Any 4xx/5xx response throws a `RapidCrawlError` carrying the HTTP status and
+Any 4xx/5xx response throws a `ByteKitError` carrying the HTTP status and
 the API's error code:
 
 ```ts
-import { RapidCrawl, RapidCrawlError } from '@hunt-labs/bytekit-sdk';
+import { ByteKit, ByteKitError } from '@hunt-labs/bytekit-sdk';
 
 try {
   await client.scrape.create({ url: 'https://example.com' });
 } catch (err) {
-  if (err instanceof RapidCrawlError) {
+  if (err instanceof ByteKitError) {
     console.error(err.status, err.code, err.message);
   }
 }

package/dist/allowlist.d.ts

@@ -8,5 +8,5 @@
  *
  * To add a method: annotate the operation in the spec first, then add it here.
  */
-export declare const V01_METHODS: readonly ["scrape.create", "scrape.get", "screenshots.create", "screenshots.get", "bulk.create", "bulk.get", "bulk.cancel", "bulk.screenshots.list", "scrape.bulk.create", "scrape.bulk.get", "fetch.create", "fetch.get", "fetch.bulk.create", "fetch.bulk.get", "monitors.create", "monitors.list", "monitors.get", "monitors.update", "monitors.delete", "monitors.captures.list", "sitemap.create", "sitemap.get", "search.query", "account.get"];
+export declare const V01_METHODS: readonly ["scrape.create", "scrape.get", "screenshots.create", "screenshots.get", "bulk.create", "bulk.get", "bulk.cancel", "bulk.screenshots.list", "scrape.bulk.create", "scrape.bulk.get", "fetch.create", "fetch.get", "fetch.bulk.create", "fetch.bulk.get", "monitors.create", "monitors.list", "monitors.get", "monitors.update", "monitors.delete", "monitors.captures.list", "sitemap.create", "sitemap.get", "search.query", "account.get", "usage.get", "usage.daily", "usage.byEndpoint", "webhooks.list", "webhooks.retry"];
 export type V01Method = (typeof V01_METHODS)[number];

package/dist/allowlist.js

@@ -34,4 +34,11 @@ export const V01_METHODS = [
     'sitemap.get',
     'search.query',
     'account.get',
+    // usage + webhook-delivery ops — resource surface shipped in #2471, brought
+    // under the parity guard in #2476.
+    'usage.get',
+    'usage.daily',
+    'usage.byEndpoint',
+    'webhooks.list',
+    'webhooks.retry',
 ];

package/dist/client.d.ts

@@ -6,12 +6,16 @@ import { MonitorsResource } from './resources/monitors.js';
 import { SitemapResource } from './resources/sitemap.js';
 import { SearchResource } from './resources/search.js';
 import { AccountResource } from './resources/account.js';
+import { UsageResource } from './resources/usage.js';
+import { WebhooksResource } from './resources/webhooks.js';
 export interface ByteKitOptions {
     apiKey: string;
     baseUrl?: string;
     /**
      * Default CLIENT-SIDE request timeout in ms — the abort budget for the whole HTTP
-     * round-trip made by this SDK. `0` disables the SDK-side timeout. Default 120000.
+     * round-trip made by this SDK, covering time-to-headers **and** the response body
+     * read (so a large legitimate body counts against it too). `0` disables the SDK-side
+     * timeout. Default 120000.
      *
      * NOT the same thing as `ScrapeOpts.timeout_ms`, which is the SERVER-side budget the
      * gateway applies to the scrape itself. Override per call with
@@ -24,12 +28,14 @@ export interface RequestOptions {
     /**
      * Caller-owned abort signal. When supplied WITHOUT a per-call `requestTimeoutMs`, it
      * fully replaces the constructor timeout — supply `requestTimeoutMs` alongside it to
-     * keep a bound (the request then aborts on whichever fires first).
+     * keep a bound (the request then aborts on whichever fires first). Firing it aborts an
+     * in-flight response body read as well, not only the wait for headers.
      */
     signal?: AbortSignal;
     /**
-     * Per-call CLIENT-SIDE abort budget in ms. Wins over the constructor's `timeoutMs`.
-     * `0` disables the SDK-side timeout for this call.
+     * Per-call CLIENT-SIDE abort budget in ms, covering headers **and** the response body
+     * read. Wins over the constructor's `timeoutMs`. `0` disables the SDK-side timeout for
+     * this call.
      *
      * Deliberately NOT named `timeoutMs`: it rides on the same call as `ScrapeOpts.timeout_ms`
      * (the SERVER-side scrape budget), and the two mean different things —
@@ -38,6 +44,29 @@ export interface RequestOptions {
      */
     requestTimeoutMs?: number;
 }
+/**
+ * A JSON response together with its transport metadata (#2547).
+ *
+ * Returned by {@link ByteKit.requestWithResponse} and by the `*WithResponse` sibling
+ * methods. `data` is exactly what the corresponding base method resolves to, so a caller
+ * can migrate from `x = await r.get(id)` to `{ data: x, headers } = await
+ * r.getWithResponse(id)` with no change to how `x` is used.
+ *
+ * This is an ADDITIVE surface: it never changes the return type, the request bytes, or the
+ * abort semantics of any existing method.
+ */
+export interface ApiResponse<T> {
+    /** The parsed JSON body — identical to what the base method returns. `undefined` on 204/empty. */
+    data: T;
+    /**
+     * The full RESPONSE headers, e.g. `X-Raw-Bytes`, `X-Billable-Bytes`,
+     * `X-Billing-Multiplier`. The `Authorization` header lives on the REQUEST and is never
+     * reachable here.
+     */
+    headers: Headers;
+    /** The HTTP status code of the successful response (2xx; non-2xx already threw). */
+    status: number;
+}
 /** Why a request never produced an HTTP response. */
 export type ConnectionErrorCode = 'timeout' | 'aborted' | 'connection';
 /** Thrown when the API responds with a 4xx or 5xx status code. */
@@ -71,6 +100,8 @@ export declare class ByteKit {
     readonly sitemap: SitemapResource;
     readonly search: SearchResource;
     readonly account: AccountResource;
+    readonly usage: UsageResource;
+    readonly webhooks: WebhooksResource;
     constructor(options: ByteKitOptions);
     /**
      * Resolve the abort budget for one request from the caller's options and the
@@ -86,6 +117,31 @@ export declare class ByteKit {
      * A timeout of `0` (either level) means "no SDK-side timeout".
      */
     private resolveSignal;
+    /**
+     * Perform the authenticated round-trip and return whatever the caller's `readBody`
+     * continuation makes of the successful response, having already thrown for the two
+     * non-body failure modes:
+     *
+     * - `ByteKitConnectionError` when no HTTP response was produced at all
+     *   (DNS/socket/TLS failure, client-side timeout, or caller abort) — including one
+     *   that fires while the body is streaming.
+     * - `ByteKitError` on any 4xx/5xx, carrying the API's `code`/`message`.
+     *
+     * The body is read through a continuation each caller supplies — JSON text for
+     * `request()`, raw bytes for `requestRaw()`. This is the single place that owns auth,
+     * the abort budget, and the error taxonomy; the two public transports differ only in
+     * how they read a successful body.
+     *
+     * The abort budget spans headers **and** body (#2546). Returning the `Response` used to
+     * end the guarded region at the headers boundary, leaving `response.text()` /
+     * `response.arrayBuffer()` — and the `!response.ok` branch's `response.json()` —
+     * unbounded: a server that sent headers and then stalled its body hung the caller
+     * forever. `readBody` must therefore ONLY consume the stream; `JSON.parse` and
+     * `TextDecoder` run in the callers, outside the guard, so a malformed body still
+     * surfaces its own parse error rather than being remapped to a connection error.
+     * `cleanup()` still runs exactly once per call, now after the body completes.
+     */
+    private send;
     /**
      * Make an authenticated JSON request to the ByteKit API.
      *
@@ -94,6 +150,44 @@ export declare class ByteKit {
      * timeout, or caller abort).
      */
     request<T>(method: string, path: string, body?: unknown, options?: RequestOptions): Promise<T>;
+    /**
+     * Make an authenticated request whose response body IS the content, not a JSON
+     * envelope — the `/v1/fetch` contract, where metadata rides in `X-Fetch-*`
+     * response headers.
+     *
+     * Returns BOTH a UTF-8-decoded string AND the raw, byte-for-byte body, plus the
+     * response `Headers` and status, WITHOUT ever calling `JSON.parse`. This is
+     * deliberately a separate transport from `request()`: the generic JSON parse
+     * there is shared by every other resource and must not change (#2466).
+     *
+     * `/v1/fetch` supports true binary passthrough (images, PDFs, etc.) as a
+     * first-class mode, and UTF-8 decoding a non-UTF-8 byte stream is lossy — any
+     * invalid byte sequence gets silently replaced with U+FFFD. Reading the body
+     * once via `arrayBuffer()` lets both forms be derived from the same bytes: `bytes`
+     * is always byte-exact, `body` is a best-effort string for callers who know the
+     * content is text.
+     */
+    requestRaw(method: string, path: string, body?: unknown, options?: RequestOptions): Promise<{
+        body: string;
+        bytes: Uint8Array;
+        headers: Headers;
+        status: number;
+    }>;
+    /**
+     * Make an authenticated JSON request and return the parsed body TOGETHER WITH the
+     * response headers and status (#2547).
+     *
+     * Identical to {@link ByteKit.request} in every observable respect — same method, path,
+     * body, `RequestOptions` precedence, error taxonomy, and empty-body handling (both drain
+     * the body through {@link readSuccessText} and parse it through {@link parseJsonBody}). It
+     * differs only in what it returns, so nothing about an existing call changes by switching
+     * to it.
+     *
+     * Use it to read the documented `X-*` metadata the API emits alongside a JSON envelope —
+     * `X-Raw-Bytes`, `X-Billable-Bytes`, `X-Billing-Multiplier`, the `X-Screenshot-*` family
+     * — for any endpoint. `sitemap.getWithResponse`, `screenshots.createWithResponse` and
+     * `screenshots.getWithResponse` are typed shorthands over this for the endpoints whose
+     * headers are documented in the API reference.
+     */
+    requestWithResponse<T>(method: string, path: string, body?: unknown, options?: RequestOptions): Promise<ApiResponse<T>>;
 }
-export { ByteKit as RapidCrawl, ByteKitError as RapidCrawlError, ByteKitConnectionError as RapidCrawlConnectionError, };
-export type { ByteKitOptions as RapidCrawlOptions };