@zenrows/cli 1.11.0 → 1.13.0

package/CHANGELOG.md

@@ -0,0 +1,102 @@
+# Changelog
+
+## [1.13.0](https://github.com/ZenRows/cli/compare/v1.12.0...v1.13.0) (2026-11-06)
+
+
+### Features
+
+* **fetch:** add --retries to fetch and extract, off by default, following the shared retry policy ([ea4d557](https://github.com/ZenRows/cli/commit/ea4d5578fef496d0a01123259c139dd386ee55a1))
+
+## [1.12.0](https://github.com/ZenRows/cli/compare/v1.11.0...v1.12.0) (2026-10-30)
+
+
+### Features
+
+* **crawl:** get notified by webhook when a crawl starts, completes, stops or fails ([#75](https://github.com/ZenRows/cli/issues/75)) ([d71df97](https://github.com/ZenRows/cli/commit/d71df972a6a9e29d619d5c8a829022c6156530ea))
+
+
+### Bug Fixes
+
+* **batch:** stop a wait before a back-off that ends on its --timeout ([#76](https://github.com/ZenRows/cli/issues/76)) ([090c75f](https://github.com/ZenRows/cli/commit/090c75f4fabbfc19f86c2c1c6c79354175677636))
+
+## [1.11.0](https://github.com/ZenRows/cli/compare/v1.10.0...v1.11.0) (2026-10-27)
+
+
+### Features
+
+* **extract:** steer prepare callers to send several example URLs of one page type ([#72](https://github.com/ZenRows/cli/issues/72)) ([c8dc644](https://github.com/ZenRows/cli/commit/c8dc644d7b04d6a59ae8a72900961438e3b76b6f))
+
+## [1.10.0](https://github.com/ZenRows/cli/compare/v1.9.0...v1.10.0) (2026-10-23)
+
+
+### Features
+
+* **signup:** agents could not get past the signup question, which only surfaced as a raw HTTP 428 body ([#73](https://github.com/ZenRows/cli/issues/73)) ([eac6f7a](https://github.com/ZenRows/cli/commit/eac6f7a625fb51ab081f282a428966f187493cb6))
+
+
+### Bug Fixes
+
+* **skills:** name the local storage tools the MCP server registers since 2.15.0 ([#70](https://github.com/ZenRows/cli/issues/70)) ([eb61828](https://github.com/ZenRows/cli/commit/eb61828df42df32bf1b6c497caf22185ed6ec545))
+
+## [1.9.0](https://github.com/ZenRows/cli/compare/v1.8.0...v1.9.0) (2026-10-20)
+
+
+### Features
+
+* **extract:** remove the autoparse fallback and add zenrows extract prepare ([#47](https://github.com/ZenRows/cli/issues/47)) ([88a33d6](https://github.com/ZenRows/cli/commit/88a33d67f0c6c94accde1061f9ea10e7911f047b))
+
+## [1.8.0](https://github.com/ZenRows/cli/compare/v1.7.0...v1.8.0) (2026-10-20)
+
+
+### Features
+
+* **batch:** agents could not recover, inspect or manage batch jobs from the CLI ([#67](https://github.com/ZenRows/cli/issues/67)) ([a850ea4](https://github.com/ZenRows/cli/commit/a850ea437b013cf10e584e6b87366b94037ae81b))
+
+## [1.7.0](https://github.com/ZenRows/cli/compare/v1.6.2...v1.7.0) (2026-10-18)
+
+
+### Features
+
+* **account:** claim links expire after 30 days and accounts get claimed in the browser, so refresh the stored link and stop offering it once claimed ([#65](https://github.com/ZenRows/cli/issues/65)) ([ded5602](https://github.com/ZenRows/cli/commit/ded5602caa300aa288f6532a27abd38de83362a1))
+
+## [1.6.2](https://github.com/ZenRows/cli/compare/v1.6.1...v1.6.2) (2026-10-16)
+
+
+### Bug Fixes
+
+* **crawl:** say a crawl with nothing new to follow completes, so users stop chasing their patterns ([#50](https://github.com/ZenRows/cli/issues/50)) ([0301d47](https://github.com/ZenRows/cli/commit/0301d47036a8a8dacd7549c1aacf0f3c29be5afe))
+
+## [1.6.1](https://github.com/ZenRows/cli/compare/v1.6.0...v1.6.1) (2026-10-16)
+
+
+### Bug Fixes
+
+* **crawl:** exit 1 when a crawl stops because credits ran out ([#49](https://github.com/ZenRows/cli/issues/49)) ([8661688](https://github.com/ZenRows/cli/commit/8661688d97029f0ab8ff18fe1a0ce5cb54f6b416))
+
+## [1.6.0](https://github.com/ZenRows/cli/compare/v1.5.5...v1.6.0) (2026-10-16)
+
+
+### Features
+
+* **crawl:** return each page as Markdown, and choose how a crawl finds URLs ([#44](https://github.com/ZenRows/cli/issues/44)) ([6cd34a2](https://github.com/ZenRows/cli/commit/6cd34a2de8ed2c7a5fcfedc4ecddc8f1b8480a81))
+
+
+### Bug Fixes
+
+* **batch:** point an inactive subscription to Billing, not to re-login ([#45](https://github.com/ZenRows/cli/issues/45)) ([58a1229](https://github.com/ZenRows/cli/commit/58a12297728690af3d45315fe74a4b6438e1935f))
+
+## [1.5.5](https://github.com/ZenRows/cli/compare/v1.5.4...v1.5.5) (2026-10-15)
+
+
+### Bug Fixes
+
+* **release:** npm rejected the 1.5.4 publish because the repository URL casing did not match the provenance ([#54](https://github.com/ZenRows/cli/issues/54)) ([f778864](https://github.com/ZenRows/cli/commit/f7788649d7f834ba01b88e1bfe45dfd0cb67518d))
+
+## [1.5.4](https://github.com/ZenRows/cli/compare/v1.5.3...v1.5.4) (2026-10-15)
+
+
+### Bug Fixes
+
+* **skills:** strict YAML parsers skipped the crawl skill over an unquoted colon ([#51](https://github.com/ZenRows/cli/issues/51)) ([fdd9eaa](https://github.com/ZenRows/cli/commit/fdd9eaa839f041c27b6f7f8d1ef97f183983efb9))
+
+Releases before 1.5.4 are on [GitHub Releases](https://github.com/ZenRows/cli/releases).

package/README.md

@@ -125,8 +125,22 @@ zenrows fetch <url> # Adaptive Stealth Mode (mode=auto)
 zenrows fetch <url> --output markdown
 zenrows fetch <url> --manual --js-render --premium-proxy
 zenrows fetch <url> --proxy-country us --wait-for ".price"
+zenrows fetch <url> --retries 2 # retry a busy answer up to 2 times; off by default
 ```
 
+`fetch` and `extract` send each request once. With `--retries <n>`, they retry
+up to `n` times on a 429, 502, 503 or 504 from the API, and on a network error
+before the request reached the API (refused connection, DNS failure, connect
+timeout, unreachable network). Each wait is the `Retry-After` header in whole
+seconds (0.25 s to 30 s) or a random delay that grows with each retry, and no
+retry starts more than 90 s after the first request. They never retry a client
+timeout (`--timeout`) or a connection that closed after the request was sent,
+because the API may still finish and charge it; a status the target site sent
+(`--original-status`); or any request of `zenrows extract prepare`. If a 502,
+503 or 504 is still the answer after one or more retries, the error says how
+many retries ran and to wait a minute and run the same command again, without
+suggesting costlier options. A 429 keeps the error it has without `--retries`.
+
 ## 8. Extract
 
 ```bash

package/dist/src/adapters/protected-fetch.js

@@ -100,6 +100,6 @@ projectRoot) {
     validateAutoManual(opts, config);
     const params = buildParams(opts, config);
     const mode = params.mode === "auto" ? "auto" : "manual";
-    const result = await scrape(config.apiBase, apiKey, params, { timeoutMs: opts.timeoutMs, projectRoot });
+    const result = await scrape(config.apiBase, apiKey, params, { timeoutMs: opts.timeoutMs, retries: opts.retries, projectRoot });
     return { result, params, mode };
 }

package/dist/src/cli/commands/crawl.js

@@ -13,7 +13,7 @@ import { ensureApiKey } from "../../core/ensure-key.js";
 import { assertUsable } from "../../core/capabilities.js";
 import { assertDomainAllowed, assertWithinLimits, loadPolicy } from "../../core/policy.js";
 import { newRunId, writeRun } from "../../core/artifacts.js";
-import { contentIdOf, createCrawl, downloadCrawl, getCrawl, getCrawlContent, isTerminal, listAllResults, listCrawls, stopCrawl, waitForCrawl, } from "../../core/crawl-api.js";
+import { contentIdOf, createCrawl, downloadCrawl, getCrawl, getCrawlContent, isTerminal, listAllResults, listCrawls, stopCrawl, waitForCrawl, CRAWL_WEBHOOK_EVENTS, WEBHOOK_METADATA_LIMITS, } from "../../core/crawl-api.js";
 import { asNumber, asString, parse } from "../command.js";
 import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "../../core/errors.js";
 import { readAccount } from "../../core/agent-account.js";
@@ -42,6 +42,13 @@ export const crawl = {
         "                                    fetch counted by --max-pages",
         "    --discovery links               how the crawl finds the URLs it follows; links (the",
         "                                    default) follows the links in each page's HTML",
+        "    --webhook <url>                 post the crawl's events to this https URL, signed with the",
+        "                                    account's webhook signing key (Settings → API Keys)",
+        "    --webhook-event <event>         with --webhook: send only this event (repeatable); one of",
+        "                                    crawl.started, crawl.completed, crawl.stopped, crawl.failed;",
+        "                                    leave it out for all four",
+        "    --webhook-metadata <key=value>  with --webhook: sent back with every event (repeatable,",
+        "                                    up to 20)",
         "    --follow                        wait until the crawl ends (Ctrl-C stops the wait, not the crawl)",
         "    --timeout <s>                   with --follow: stop waiting after s seconds (default 600)",
         "    --idempotency-key <key>         so a retried create starts no second crawl",
@@ -113,6 +120,9 @@ async function createCmd(rest, ctx) {
         "exclude-patterns": { type: "string", multiple: true },
         "output-format": { type: "string" },
         discovery: { type: "string", multiple: true },
+        webhook: { type: "string" },
+        "webhook-event": { type: "string", multiple: true },
+        "webhook-metadata": { type: "string", multiple: true },
         follow: { type: "boolean" },
         timeout: { type: "string" },
         "idempotency-key": { type: "string" },
@@ -171,6 +181,9 @@ async function createCmd(rest, ctx) {
         }
         params.discovery = ["links"];
     }
+    const webhook = webhookOf(values);
+    if (webhook)
+        params.webhook = webhook;
     const timeoutMs = timeoutFlag(values.timeout);
     const idempotencyKey = asString(values["idempotency-key"]);
     assertUsable("crawl");
@@ -531,6 +544,11 @@ function printCrawl(c, json, headline, extra = {}) {
     if (cov) {
         log.info(`  ${cov.items_found} item(s) found · ${cov.pages_fetched} page(s) fetched · ${cov.pages_failed} failed`);
     }
+    const hook = c.webhook;
+    if (hook?.url) {
+        const meta = hook.metadata && Object.keys(hook.metadata).length ? ` · metadata: ${JSON.stringify(hook.metadata)}` : "";
+        log.info(`  webhook: ${hook.url} · events: ${(hook.events ?? []).join(", ") || "all"}${meta}`);
+    }
     if (extra.note)
         log.dim(`  next: zenrows crawl wait ${c.crawl_id}`);
     else if (note)
@@ -587,6 +605,77 @@ function pageLimit(v, max) {
     }
     return n;
 }
+/**
+ * `--webhook`, `--webhook-event` and `--webhook-metadata key=value` as the create's
+ * `webhook`, or undefined without `--webhook`. Checks the shapes and the API's limits
+ * before any request; whether the URL's host resolves is the API's to judge.
+ */
+function webhookOf(values) {
+    const url = asString(values.webhook);
+    const events = patterns(values["webhook-event"]);
+    const pairs = patterns(values["webhook-metadata"]);
+    if (url === undefined) {
+        if (events.length || pairs.length) {
+            throw webhookUsage(`${events.length ? "--webhook-event" : "--webhook-metadata"} needs --webhook.`, "Pass --webhook <https-url> with it, or leave it out.");
+        }
+        return undefined;
+    }
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        parsed = undefined;
+    }
+    if (parsed?.protocol !== "https:") {
+        throw webhookUsage(`Invalid --webhook value '${url}': it must be an absolute https URL.`, "Pass e.g. --webhook https://example.com/hooks/zenrows.");
+    }
+    const webhook = { url };
+    if (events.length) {
+        const known = new Set(CRAWL_WEBHOOK_EVENTS);
+        const unknown = events.find((e) => !known.has(e));
+        if (unknown !== undefined) {
+            throw webhookUsage(`Invalid --webhook-event value '${unknown}'.`, `Pass one of ${CRAWL_WEBHOOK_EVENTS.join(", ")}, or leave it out for all four.`);
+        }
+        const repeated = events.find((e, i) => events.indexOf(e) !== i);
+        if (repeated !== undefined) {
+            throw webhookUsage(`--webhook-event '${repeated}' is given twice.`, "Name each event once.");
+        }
+        webhook.events = events;
+    }
+    if (pairs.length) {
+        const { keys, keyLength, valueBytes } = WEBHOOK_METADATA_LIMITS;
+        const metadata = {};
+        for (const p of pairs) {
+            const eq = p.indexOf("=");
+            if (eq <= 0)
+                throw webhookUsage(`Invalid --webhook-metadata '${p}': use key=value.`, "Pass e.g. --webhook-metadata site=books.");
+            const key = p.slice(0, eq);
+            const value = p.slice(eq + 1);
+            if ([...key].length > keyLength)
+                throw webhookUsage(`--webhook-metadata key '${key}' is longer than ${keyLength} characters.`, "Use a shorter key.");
+            if (key in metadata)
+                throw webhookUsage(`--webhook-metadata key '${key}' is given twice.`, "Name each key once.");
+            if (Buffer.byteLength(JSON.stringify(value)) > valueBytes) {
+                throw webhookUsage(`--webhook-metadata '${key}' is longer than ${valueBytes} bytes as JSON.`, "Use a shorter value.");
+            }
+            metadata[key] = value;
+        }
+        if (Object.keys(metadata).length > keys)
+            throw webhookUsage(`--webhook-metadata has more than ${keys} keys.`, `Pass at most ${keys}.`);
+        webhook.metadata = metadata;
+    }
+    return webhook;
+}
+function webhookUsage(message, next_action) {
+    return new ToolkitError({
+        code: "INVALID_USAGE",
+        message,
+        likely_cause: "The webhook flags do not match what Crawl accepts.",
+        next_action,
+        suggested_commands: ["zenrows crawl --help"],
+    });
+}
 function patterns(v) {
     const list = Array.isArray(v) ? v : typeof v === "string" ? [v] : [];
     return list.filter((p) => typeof p === "string" && p.length > 0);

package/dist/src/cli/commands/extract.js

@@ -21,6 +21,7 @@ import { formatRequestCost } from "../../core/http.js";
 import { parse, asString, asNumber } from "../command.js";
 import { printError, writeOut } from "../output.js";
 import { refreshClaimAdvice } from "../../core/claim-link.js";
+import { normalizeRetries } from "./fetch.js";
 import { PREPARE_HELP, PREPARE_USAGE, runPrepare } from "./extract-prepare.js";
 export const extract = {
     name: "extract",
@@ -40,6 +41,7 @@ export const extract = {
         "Shared:",
         "  --manual --js-render --premium-proxy   manual fetch controls",
         "  --validate             fail if the result is not valid JSON",
+        "  --retries <n>          retry 429/502/503/504 and failed connects up to n times (default 0)",
         "  --out <file>           write output to a file",
         "  --no-signup            do not auto-create a Free plan account if no key exists",
         "  --json                 structured result",
@@ -65,6 +67,7 @@ export const extract = {
             wait: { type: "string" },
             "wait-for": { type: "string" },
             validate: { type: "boolean" },
+            retries: { type: "string" },
             out: { type: "string" },
             "no-signup": { type: "boolean" },
             json: { type: "boolean" },
@@ -112,6 +115,7 @@ export const extract = {
             wait: asNumber(values.wait),
             waitFor: asString(values["wait-for"]),
             validate: values.validate === true,
+            retries: normalizeRetries(values.retries),
         };
         assertUsable("extract");
         const config = loadConfig();

package/dist/src/cli/commands/fetch.js

@@ -36,6 +36,7 @@ export const fetch_ = {
         "  --screenshot           capture an above-the-fold screenshot",
         "  --out <file>           write the response body to a file",
         "  --timeout <ms>         client-side timeout (default 120000; above the API's own 90s budget)",
+        "  --retries <n>          retry 429/502/503/504 and failed connects up to n times (default 0)",
         "  --no-signup            do not auto-create a Free plan account if no key exists",
         "  --json                 print a structured result",
         "",
@@ -58,6 +59,7 @@ export const fetch_ = {
             screenshot: { type: "boolean" },
             out: { type: "string" },
             timeout: { type: "string" },
+            retries: { type: "string" },
             "no-signup": { type: "boolean" },
             json: { type: "boolean" },
         });
@@ -94,6 +96,7 @@ export const fetch_ = {
             output: normalizeOutput(asString(values.output)),
             screenshot: values.screenshot === true,
             timeoutMs: normalizeTimeout(values.timeout),
+            retries: normalizeRetries(values.retries),
         };
         const runId = newRunId();
         const startedAt = new Date().toISOString();
@@ -194,6 +197,20 @@ export function normalizeTimeout(v) {
     }
     return ms;
 }
+/** `--retries <n>`: a whole number, 0 or more. Like `--timeout`, a bad value is an error. */
+export function normalizeRetries(v) {
+    if (v === undefined)
+        return undefined;
+    if (typeof v !== "string" || !/^\d+$/.test(v)) {
+        throw new ToolkitError({
+            code: "INVALID_USAGE",
+            message: `Invalid --retries value '${String(v)}'.`,
+            likely_cause: "--retries takes a whole number, 0 or more.",
+            next_action: "Pass a count, e.g. --retries 2. 0, the default, sends the request once.",
+        });
+    }
+    return Number(v);
+}
 export function normalizeOutput(v) {
     if (!v)
         return undefined;

package/dist/src/core/batch-api.js

@@ -581,7 +581,7 @@ export async function waitForJob(id, opts) {
                 throw err;
             lastErr = err;
         }
-        if (Date.now() + delay > deadline) {
+        if (Date.now() + delay >= deadline) {
             // Out of time: report the last state read, or the error if none was.
             if (!last)
                 throw lastErr;

package/dist/src/core/config.js

@@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0";
  * entrypoint) so core modules — telemetry, signup provenance — can read it
  * without an import cycle. `VERSION` in `cli/index.ts` re-exports this.
  */
-export const CLI_VERSION = "1.11.0"; // x-release-please-version
+export const CLI_VERSION = "1.13.0"; // x-release-please-version
 /** The User-Agent on every request to a Zenrows API. The gateway reads the version from it. */
 export const CLI_USER_AGENT = `zenrows-cli/${CLI_VERSION}`;
 /** Env var to override the Fetch and Extract API base (local/staging testing). */

package/dist/src/core/crawl-api.js

@@ -10,8 +10,9 @@
  * Crawl SDKs do (see `retryStatuses`). `fetchImpl` / `sleepImpl` are injectable
  * for tests.
  *
- * `output_format` takes `html` or `markdown` (absent = URLs only). Response fields we do
- * not model pass through untouched.
+ * `output_format` takes `html` or `markdown` (absent = URLs only). A `webhook` gets the
+ * crawl's events (`crawl.started` and how it ended) posted to a URL, signed with the
+ * account's webhook signing key. Response fields we do not model pass through untouched.
  */
 import { Readable } from "node:stream";
 import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
@@ -36,6 +37,10 @@ export function crawlBase() {
 }
 /** The API's "Crawl is not enabled for this account" code (403). */
 export const CRAWL_NOT_ENABLED_CODE = "REQS008";
+/** The events a crawl's webhook can send. */
+export const CRAWL_WEBHOOK_EVENTS = ["crawl.started", "crawl.completed", "crawl.stopped", "crawl.failed"];
+/** The API's limits on a webhook's `metadata`. */
+export const WEBHOOK_METADATA_LIMITS = { keys: 20, keyLength: 64, valueBytes: 1024 };
 /** A crawl stops progressing once its status is anything but `running`. */
 export function isTerminal(status) {
     return !!status && status !== "running";
@@ -273,7 +278,9 @@ function classifyProblem(status, problem, body, request, serverCode, retryAfter)
             likely_cause: cause,
             next_action: serverCode === "idempotency_key_reused"
                 ? "This Idempotency-Key was already used for a different request. Send a new key, or no key. Do not retry as is."
-                : "Fix the reported parameter and retry. See `zenrows crawl --help` for the accepted flags and ranges.",
+                : serverCode === "webhook_signing_key_required"
+                    ? "Nothing was created. Create a webhook signing key in the dashboard (Settings → API Keys), then retry."
+                    : "Fix the reported parameter and retry. See `zenrows crawl --help` for the accepted flags and ranges.",
         });
     }
     return new ToolkitError({
@@ -300,6 +307,14 @@ export function createCrawl(params, opts) {
         body.output_format = params.output_format;
     if (params.discovery?.length)
         body.discovery = params.discovery;
+    if (params.webhook !== undefined) {
+        const webhook = { url: params.webhook.url };
+        if (params.webhook.events?.length)
+            webhook.events = params.webhook.events;
+        if (params.webhook.metadata !== undefined)
+            webhook.metadata = params.webhook.metadata;
+        body.webhook = webhook;
+    }
     return crawlJson("POST", "/crawls", {
         ...opts,
         body,

package/dist/src/core/http.js

@@ -9,7 +9,7 @@ import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./
 import { storedClaimHint, usableClaimUrl } from "./claim-link.js";
 import { readAccount } from "./agent-account.js";
 import { ENV_KEY, resolveApiKey } from "./auth.js";
-import { registerSecret } from "./logger.js";
+import { log, registerSecret } from "./logger.js";
 import { CLI_USER_AGENT } from "./config.js";
 import { agentClientHeader } from "./agent-client.js";
 import { normalizeDomain, PAGE_NOT_RECOGNISED_STEPS, prepareCommand } from "./prepared-domains-api.js";
@@ -27,6 +27,44 @@ import { normalizeDomain, PAGE_NOT_RECOGNISED_STEPS, prepareCommand } from "./pr
 export const DEFAULT_TIMEOUT_MS = 120_000;
 /** The gateway's own request budget. Kept here only to justify the default above. */
 export const SERVER_BUDGET_MS = 90_000;
+/** No retry starts later than this after the first request. */
+const RETRY_BUDGET_MS = 90_000;
+/** Busy answers from the API, retried with `--retries`. */
+const RETRY_STATUSES = new Set([429, 502, 503, 504]);
+/** Network error codes that mean the request never reached the API. */
+const NEVER_SENT_CODES = new Set([
+    "ECONNREFUSED",
+    "ENOTFOUND",
+    "EAI_AGAIN",
+    "ENETUNREACH",
+    "EHOSTUNREACH",
+    "UND_ERR_CONNECT_TIMEOUT",
+]);
+/** `Retry-After` in whole seconds, clamped to 0.25-30 s; undefined for any other value. */
+export function retryAfterMs(value) {
+    if (!value || !/^\d+$/.test(value.trim()))
+        return undefined;
+    return Math.min(Math.max(Number(value) * 1000, 250), 30_000);
+}
+/** Full jitter before retry `n` (0-based): uniform(0.25 s, min(30 s, 2^n s)). */
+export function jitterMs(n, random = Math.random) {
+    return 250 + random() * (Math.min(30_000, 1000 * 2 ** n) - 250);
+}
+/**
+ * True when a fetch error's `cause` means the request never reached the API.
+ * ETIMEDOUT counts only on connect. A happy-eyeballs AggregateError counts
+ * only when every address failed to connect.
+ */
+export function neverSent(cause) {
+    if (cause instanceof AggregateError)
+        return cause.errors.length > 0 && cause.errors.every(neverSent);
+    const { code, syscall } = (cause ?? {});
+    return code === "ETIMEDOUT" ? syscall === "connect" : NEVER_SENT_CODES.has(String(code));
+}
+/** A busy answer from the API. A status the target site sent is never retried. */
+function isRetryable(status, params) {
+    return RETRY_STATUSES.has(status) && !params.original_status && params.allowed_status_codes === undefined;
+}
 function buildUrl(apiBase, apiKey, params) {
     const u = new URL(apiBase);
     u.searchParams.set("apikey", apiKey);
@@ -45,43 +83,66 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
     registerSecret(apiKey);
     const { full, redacted } = buildUrl(apiBase, apiKey, params);
     const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
-    const controller = new AbortController();
-    // Our own timer is the only thing that aborts this controller, so this flag —
-    // not `err.name === "AbortError"` — is what tells a client-side give-up apart
-    // from a genuine transport failure. The two must never share an error code.
-    let timedOut = false;
-    const timeout = setTimeout(() => {
-        timedOut = true;
-        controller.abort();
-    }, timeoutMs);
-    const startedAt = Date.now();
+    const retries = opts.retries ?? 0;
+    const now = opts.now ?? Date.now;
+    const firstSentAt = now();
+    /** Waits before retry `n` and returns true, unless no retry is left or it would start after the budget. */
+    const retry = async (n, delayMs, reason) => {
+        if (n >= retries || now() - firstSentAt + delayMs > RETRY_BUDGET_MS)
+            return false;
+        log.dim(`${reason}; retry ${n + 1}/${retries} in ${formatMs(delayMs)}`);
+        await (opts.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms))))(delayMs);
+        return true;
+    };
     let res;
-    try {
-        res = await fetch(full, {
-            method: "GET",
-            headers: { "User-Agent": CLI_USER_AGENT, "Accept-Encoding": "gzip, deflate", ...agentClientHeader({ projectRoot: opts.projectRoot }) },
-            signal: controller.signal,
-        });
-    }
-    catch (err) {
+    let raw;
+    let retriesUsed = 0;
+    for (;; retriesUsed++) {
+        const controller = new AbortController();
+        // Our own timer is the only thing that aborts this controller, so this flag —
+        // not `err.name === "AbortError"` — is what tells a client-side give-up apart
+        // from a genuine transport failure. The two must never share an error code.
+        let timedOut = false;
+        const timeout = setTimeout(() => {
+            timedOut = true;
+            controller.abort();
+        }, timeoutMs);
+        const startedAt = Date.now();
+        try {
+            res = await fetch(full, {
+                method: "GET",
+                headers: { "User-Agent": CLI_USER_AGENT, "Accept-Encoding": "gzip, deflate", ...agentClientHeader({ projectRoot: opts.projectRoot }) },
+                signal: controller.signal,
+            });
+        }
+        catch (err) {
+            clearTimeout(timeout);
+            const elapsedMs = Date.now() - startedAt;
+            // A timeout, or an error after the request was sent, is never retried:
+            // the API may still finish and charge that request.
+            if (timedOut)
+                throw requestTimeout(params.url, timeoutMs, elapsedMs);
+            const retryable = err instanceof Error && neverSent(err.cause);
+            if (retryable && (await retry(retriesUsed, jitterMs(retriesUsed), "Network error before the request reached the API")))
+                continue;
+            const cause = err instanceof Error ? err.message : String(err);
+            throw new ToolkitError({
+                code: "BACKEND_UNAVAILABLE",
+                message: `Could not reach the Zenrows API.`,
+                likely_cause: `Network error after ${formatMs(elapsedMs)}: ${cause}`,
+                next_action: "Check connectivity and retry. Verify api base in `zenrows config show`.",
+                suggested_commands: ["zenrows status", "zenrows config show"],
+            });
+        }
         clearTimeout(timeout);
-        const elapsedMs = Date.now() - startedAt;
-        if (timedOut)
-            throw requestTimeout(params.url, timeoutMs, elapsedMs);
-        const cause = err instanceof Error ? err.message : String(err);
-        throw new ToolkitError({
-            code: "BACKEND_UNAVAILABLE",
-            message: `Could not reach the Zenrows API.`,
-            likely_cause: `Network error after ${formatMs(elapsedMs)}: ${cause}`,
-            next_action: "Check connectivity and retry. Verify api base in `zenrows config show`.",
-            suggested_commands: ["zenrows status", "zenrows config show"],
-        });
+        // Read the raw bytes once. `body` (UTF-8) drives all error detection and text
+        // output; `raw` preserves the exact bytes so binary responses (screenshot /
+        // PDF) can be written without the UTF-8 round-trip that corrupts them.
+        raw = Buffer.from(await res.arrayBuffer());
+        const delayMs = retryAfterMs(res.headers.get("retry-after")) ?? jitterMs(retriesUsed);
+        if (!isRetryable(res.status, params) || !(await retry(retriesUsed, delayMs, `HTTP ${res.status}`)))
+            break;
     }
-    clearTimeout(timeout);
-    // Read the raw bytes once. `body` (UTF-8) drives all error detection and text
-    // output; `raw` preserves the exact bytes so binary responses (screenshot /
-    // PDF) can be written without the UTF-8 round-trip that corrupts them.
-    const raw = Buffer.from(await res.arrayBuffer());
     const body = raw.toString("utf8");
     // A binary body is only ever produced by a successful screenshot/PDF request.
     // Zenrows error responses are always a JSON envelope (text), so exclude those.
@@ -228,6 +289,15 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
                 : "Verify the URL. Rendering or premium proxies will not turn a 404 into content — do not retry; it will fail identically and still cost credits.",
         });
     }
+    if (retriesUsed > 0 && isRetryable(res.status, params)) {
+        // A busy answer that outlasted the retries: a costlier configuration does not fix it.
+        throw new ToolkitError({
+            code: "FETCH_FAILED",
+            message: `The Zenrows API answered HTTP ${res.status} after ${retriesUsed} ${retriesUsed === 1 ? "retry" : "retries"}.`,
+            likely_cause: snippet(body) || "The Zenrows API is busy.",
+            next_action: "Wait a minute, then run the same command again. Do not add --js-render or --premium-proxy: they cost more and do not help here.",
+        });
+    }
     if (res.status === 422 || res.status >= 500 || (res.status >= 400 && !looksLikeContent(result))) {
         throw new ToolkitError({
             code: "FETCH_FAILED",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zenrows/cli",
-  "version": "1.11.0",
+  "version": "1.13.0",
   "description": "Open-source CLI, MCP, skills, templates, workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.",
   "type": "module",
   "license": "MIT",
@@ -42,6 +42,7 @@
     "recipes",
     "evals",
     "README.md",
+    "CHANGELOG.md",
     "LICENSE"
   ],
   "scripts": {

package/skills/crawl/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: crawl
-description: "Collect the URLs behind one start page with Crawl (new): create a crawl, wait for it, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs."
+description: "Collect the URLs behind one start page with Crawl (new): create a crawl, wait for it or get notified by webhook, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs."
 version: 0.1.0
 requires_backend_capabilities: [crawl]
 ---
@@ -71,6 +71,36 @@ of URLs. A result's `content_status` is `pending`, `fetched` or `failed`; only
 `fetched` results have content (else `CRAWL_CONTENT_NOT_FOUND`). `download`
 reports the crawl's `status`, and `partial: true` while it runs.
 
+## Webhooks
+
+Instead of polling, have Crawl post its events to a URL you run:
+
+```
+zenrows crawl create <url> --depth 1 --webhook https://example.com/hooks/zenrows \
+  --webhook-event crawl.completed --webhook-event crawl.failed \
+  --webhook-metadata site=books --json
+```
+
+- `--webhook <url>` takes an absolute `https` URL. It is fixed at create.
+- `--webhook-event` repeats; each of `crawl.started`, `crawl.completed`,
+  `crawl.stopped`, `crawl.failed` at most once. Leave it out for all four.
+  Exactly one of the last three is sent per crawl.
+- `--webhook-metadata key=value` repeats (up to 20 keys of 1 to 64
+  characters, each value at most 1,024 bytes as JSON); every event carries it
+  back, to match the crawl to your own records.
+- The crawl shows `webhook` as `{url, events, metadata}`, `events` being the
+  effective list; a crawl without one has none.
+- Events are signed JSON (`schema: zenrows.webhook/v1`, `event_id`,
+  `event_type`, `occurred_at`, `resource {type: crawl, id, url}`, `metadata`).
+  Verify the `X-Signature: t=…,v1=…,kid=…` header: `v1` is the hex
+  HMAC-SHA256 of `"<t>." + body` with the account's webhook signing key
+  (Settings → API Keys). Events can repeat (drop one whose `event_id` you
+  handled) and arrive out of order (sort by `occurred_at`).
+- The account needs a webhook signing key: without one the create gets 422
+  `webhook_signing_key_required` (create the key, then retry). A bad URL,
+  event or metadata gets 422 `invalid_webhook`. Both are
+  `CRAWL_INVALID_REQUEST`, and nothing was created.
+
 ## While it runs, and after
 
 ```

package/skills/extract/SKILL.md

@@ -68,6 +68,7 @@ zenrows extract https://www.example.com/products/a  # 3. extract again
   `limit` / `remaining`.
 - `EXTRACT_PREPARE_OUT_OF_ALLOWANCE`: no slots left this month; use `--css`.
   `EXTRACT_PREPARE_UNAVAILABLE`: retry in a few minutes; do not loop.
+  `prepare` sends each request once: `--retries` does not apply to it.
   A `failed` status prints the next step for its failure code and says whether
   the attempt used an allowance slot.
 - A domain that is already prepared answers `ready` at once

package/skills/protected-fetch/SKILL.md

@@ -47,6 +47,11 @@ zenrows fetch <url> --manual --js-render --premium-proxy   # the most expensive
   needs `--premium-proxy` (else `PARAM_PROXY_COUNTRY_REQUIRES_PREMIUM`).
 - You are billed only for the configuration that succeeds.
 - On failure, read `zenrows trace explain <run-id>` before escalating.
+- The CLI sends each request once. `--retries <n>` retries a 429, 502, 503 or
+  504 and a network error before the request reached the API, up to `n` times,
+  within 90 s; prefer it to your own retry loop. It never retries a timeout, a
+  connection closed after the request was sent, or a target status under
+  `--original-status`.
 
 Every run is saved under `.zenrows/runs/<run-id>/` (no secrets). See [[extract]]
 for structured output and [[cost-control]] for the cost model.

package/skills/trace-debug/SKILL.md

@@ -26,6 +26,11 @@ zenrows trace export <run-id>     # JSON for sharing
   `--manual --js-render` run; auto mode may ignore it. Each step
   multiplies the cost of the request, and both together are the most expensive
   configuration available ([[cost-control]]).
+- `FETCH_FAILED` "The Zenrows API answered HTTP 502/503/504 after N retries" →
+  the CLI already retried a busy answer (`--retries`). Wait a minute and run the
+  same command again; do not escalate, a costlier configuration does not fix
+  it. A 429 ("Rate limited") keeps its advice with or without `--retries`: wait
+  a few seconds and retry; if it persists, lower concurrency.
 - `REQUEST_TIMEOUT` → the CLI stopped waiting; the API was reached. Raise
   `--timeout` (default 120000ms, above the API's own 90s budget), or drop
   `--wait-for` so the request finishes inside that budget and the API returns