@zenrows/cli 1.10.0 → 1.12.0

package/README.md

@@ -147,10 +147,18 @@ few minutes if it should work, or use `--css`; preparing again does not change
 it. `zenrows extract` never falls back to Autoparse. The `fallbackAutoparse`
 adapter option is deprecated and has no effect.
 
+Pass 1 to 5 example URLs. One works; we recommend 3 or more of the same page
+type (e.g. 3 product pages), because Extract compares pages of one type to check
+the fields it found and fill in fields one page is missing. Several page types
+can share the 5 (e.g. 3 product pages + 2 search-result pages).
+
 ```bash
-zenrows extract prepare example.com --url https://example.com/p/1 --url https://example.com/list --wait
+zenrows extract prepare example.com \
+  --url https://www.example.com/products/a \
+  --url https://www.example.com/products/b \
+  --url https://www.example.com/products/c --wait
 zenrows extract prepare status example.com   # progress + monthly allowance
-zenrows extract https://example.com/p/1      # then extract again
+zenrows extract https://www.example.com/products/a   # then extract again
 ```
 
 ## 9. Batch

package/agent-plugin/README.md

@@ -62,7 +62,7 @@ Use MCP tool names — not CLI `zenrows fetch`:
 | --- | --- |
 | `zenrows` | Decision tree across all primitives |
 | `protected-fetch` | `scrape` |
-| `extract` | `extract` (fallback: `scrape` + css_extractor); on REQS007 with reason `domain_not_prepared`, `prepare_domain` then `prepared_domain_status` |
+| `extract` | `extract` (fallback: `scrape` + css_extractor); on REQS007 with reason `domain_not_prepared`, `prepare_domain` (1 to 5 example URLs; we recommend 3 or more of the same page type) then `prepared_domain_status` |
 | `batch-jobs` | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` |
 | `interact-browser` | `browser_*` (escalation only) |
 | `cost-control` | Cost multipliers + escalate-with-evidence |

package/agent-plugin/skills/extract/SKILL.md

@@ -48,15 +48,20 @@ Preparing a domain:
 
 ```
 extract({ url })                                          # 1. REQS007, reason domain_not_prepared
-prepare_domain({ domain: "example.com",
-  example_urls: ["https://example.com/p/1", "https://example.com/list"] })
+prepare_domain({ domain: "example.com", example_urls: [
+  "https://www.example.com/products/a",
+  "https://www.example.com/products/b",
+  "https://www.example.com/products/c"] })
                                                           # 2. returns at once, status "preparing"
 prepared_domain_status({ domain: "example.com" })         # 3. about every 30 seconds until "ready"
 extract({ url })                                          # 4. extract again
 ```
 
-- `example_urls`: 1 to 5 real pages you want to extract, one per page kind
-  (for example a product page and a listing page).
+- `example_urls`: 1 to 5 real pages you want to extract. One works; we
+  recommend 3 or more of the same page type (e.g. 3 product pages), because
+  Extract compares pages of one type to check the fields it found and fill in
+  fields one page is missing. Several page types can share the 5 (e.g. 3
+  product pages + 2 search-result pages).
 - The domain is the host: `shop.example.com` and `example.com` are prepared
   separately, and a leading `www.` is dropped.
 - Preparing takes a few minutes and costs no credits. Each new domain uses one

package/agent-plugin/skills/zenrows/SKILL.md

@@ -71,7 +71,7 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
 | Extract | `extract` | Structured JSON (`extract=auto` on any prepared domain, css_extractor; autoparse deprecated). `extract=auto` is currently free; billing may apply later |
-| Prepare a domain | `prepare_domain`, `prepared_domain_status` | After an `extract` REQS007 with reason `domain_not_prepared`: prepare the domain with 1 to 5 example URLs, poll about every 30 seconds until ready, extract again. No credits; uses one monthly allowance slot. Not for `page_not_recognised`: that domain is already prepared |
+| Prepare a domain | `prepare_domain`, `prepared_domain_status` | After an `extract` REQS007 with reason `domain_not_prepared`: prepare the domain with 1 to 5 example URLs (one works; we recommend 3 or more of the same page type, because Extract compares pages of one type to check the fields it found and fill in fields one page is missing), poll about every 30 seconds until ready, extract again. No credits; uses one monthly allowance slot. Not for `page_not_recognised`: that domain is already prepared |
 | Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 

package/dist/src/cli/commands/crawl.js

@@ -13,7 +13,7 @@ import { ensureApiKey } from "../../core/ensure-key.js";
 import { assertUsable } from "../../core/capabilities.js";
 import { assertDomainAllowed, assertWithinLimits, loadPolicy } from "../../core/policy.js";
 import { newRunId, writeRun } from "../../core/artifacts.js";
-import { contentIdOf, createCrawl, downloadCrawl, getCrawl, getCrawlContent, isTerminal, listAllResults, listCrawls, stopCrawl, waitForCrawl, } from "../../core/crawl-api.js";
+import { contentIdOf, createCrawl, downloadCrawl, getCrawl, getCrawlContent, isTerminal, listAllResults, listCrawls, stopCrawl, waitForCrawl, CRAWL_WEBHOOK_EVENTS, WEBHOOK_METADATA_LIMITS, } from "../../core/crawl-api.js";
 import { asNumber, asString, parse } from "../command.js";
 import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "../../core/errors.js";
 import { readAccount } from "../../core/agent-account.js";
@@ -42,6 +42,13 @@ export const crawl = {
         "                                    fetch counted by --max-pages",
         "    --discovery links               how the crawl finds the URLs it follows; links (the",
         "                                    default) follows the links in each page's HTML",
+        "    --webhook <url>                 post the crawl's events to this https URL, signed with the",
+        "                                    account's webhook signing key (Settings → API Keys)",
+        "    --webhook-event <event>         with --webhook: send only this event (repeatable); one of",
+        "                                    crawl.started, crawl.completed, crawl.stopped, crawl.failed;",
+        "                                    leave it out for all four",
+        "    --webhook-metadata <key=value>  with --webhook: sent back with every event (repeatable,",
+        "                                    up to 20)",
         "    --follow                        wait until the crawl ends (Ctrl-C stops the wait, not the crawl)",
         "    --timeout <s>                   with --follow: stop waiting after s seconds (default 600)",
         "    --idempotency-key <key>         so a retried create starts no second crawl",
@@ -113,6 +120,9 @@ async function createCmd(rest, ctx) {
         "exclude-patterns": { type: "string", multiple: true },
         "output-format": { type: "string" },
         discovery: { type: "string", multiple: true },
+        webhook: { type: "string" },
+        "webhook-event": { type: "string", multiple: true },
+        "webhook-metadata": { type: "string", multiple: true },
         follow: { type: "boolean" },
         timeout: { type: "string" },
         "idempotency-key": { type: "string" },
@@ -171,6 +181,9 @@ async function createCmd(rest, ctx) {
         }
         params.discovery = ["links"];
     }
+    const webhook = webhookOf(values);
+    if (webhook)
+        params.webhook = webhook;
     const timeoutMs = timeoutFlag(values.timeout);
     const idempotencyKey = asString(values["idempotency-key"]);
     assertUsable("crawl");
@@ -531,6 +544,11 @@ function printCrawl(c, json, headline, extra = {}) {
     if (cov) {
         log.info(`  ${cov.items_found} item(s) found · ${cov.pages_fetched} page(s) fetched · ${cov.pages_failed} failed`);
     }
+    const hook = c.webhook;
+    if (hook?.url) {
+        const meta = hook.metadata && Object.keys(hook.metadata).length ? ` · metadata: ${JSON.stringify(hook.metadata)}` : "";
+        log.info(`  webhook: ${hook.url} · events: ${(hook.events ?? []).join(", ") || "all"}${meta}`);
+    }
     if (extra.note)
         log.dim(`  next: zenrows crawl wait ${c.crawl_id}`);
     else if (note)
@@ -587,6 +605,77 @@ function pageLimit(v, max) {
     }
     return n;
 }
+/**
+ * `--webhook`, `--webhook-event` and `--webhook-metadata key=value` as the create's
+ * `webhook`, or undefined without `--webhook`. Checks the shapes and the API's limits
+ * before any request; whether the URL's host resolves is the API's to judge.
+ */
+function webhookOf(values) {
+    const url = asString(values.webhook);
+    const events = patterns(values["webhook-event"]);
+    const pairs = patterns(values["webhook-metadata"]);
+    if (url === undefined) {
+        if (events.length || pairs.length) {
+            throw webhookUsage(`${events.length ? "--webhook-event" : "--webhook-metadata"} needs --webhook.`, "Pass --webhook <https-url> with it, or leave it out.");
+        }
+        return undefined;
+    }
+    let parsed;
+    try {
+        parsed = new URL(url);
+    }
+    catch {
+        parsed = undefined;
+    }
+    if (parsed?.protocol !== "https:") {
+        throw webhookUsage(`Invalid --webhook value '${url}': it must be an absolute https URL.`, "Pass e.g. --webhook https://example.com/hooks/zenrows.");
+    }
+    const webhook = { url };
+    if (events.length) {
+        const known = new Set(CRAWL_WEBHOOK_EVENTS);
+        const unknown = events.find((e) => !known.has(e));
+        if (unknown !== undefined) {
+            throw webhookUsage(`Invalid --webhook-event value '${unknown}'.`, `Pass one of ${CRAWL_WEBHOOK_EVENTS.join(", ")}, or leave it out for all four.`);
+        }
+        const repeated = events.find((e, i) => events.indexOf(e) !== i);
+        if (repeated !== undefined) {
+            throw webhookUsage(`--webhook-event '${repeated}' is given twice.`, "Name each event once.");
+        }
+        webhook.events = events;
+    }
+    if (pairs.length) {
+        const { keys, keyLength, valueBytes } = WEBHOOK_METADATA_LIMITS;
+        const metadata = {};
+        for (const p of pairs) {
+            const eq = p.indexOf("=");
+            if (eq <= 0)
+                throw webhookUsage(`Invalid --webhook-metadata '${p}': use key=value.`, "Pass e.g. --webhook-metadata site=books.");
+            const key = p.slice(0, eq);
+            const value = p.slice(eq + 1);
+            if ([...key].length > keyLength)
+                throw webhookUsage(`--webhook-metadata key '${key}' is longer than ${keyLength} characters.`, "Use a shorter key.");
+            if (key in metadata)
+                throw webhookUsage(`--webhook-metadata key '${key}' is given twice.`, "Name each key once.");
+            if (Buffer.byteLength(JSON.stringify(value)) > valueBytes) {
+                throw webhookUsage(`--webhook-metadata '${key}' is longer than ${valueBytes} bytes as JSON.`, "Use a shorter value.");
+            }
+            metadata[key] = value;
+        }
+        if (Object.keys(metadata).length > keys)
+            throw webhookUsage(`--webhook-metadata has more than ${keys} keys.`, `Pass at most ${keys}.`);
+        webhook.metadata = metadata;
+    }
+    return webhook;
+}
+function webhookUsage(message, next_action) {
+    return new ToolkitError({
+        code: "INVALID_USAGE",
+        message,
+        likely_cause: "The webhook flags do not match what Crawl accepts.",
+        next_action,
+        suggested_commands: ["zenrows crawl --help"],
+    });
+}
 function patterns(v) {
     const list = Array.isArray(v) ? v : typeof v === "string" ? [v] : [];
     return list.filter((p) => typeof p === "string" && p.length > 0);

package/dist/src/cli/commands/extract-prepare.js

@@ -5,14 +5,15 @@
  * The way out of REQS007: prepare the domain so `extract=auto` works on it.
  * Submitting returns at once; `--wait` polls until the preparation is ready or
  * failed (a few minutes). Costs no credits; each new domain uses one slot of the
- * account's monthly allowance.
+ * account's monthly allowance. One --url works; with fewer than 3 the command prints a
+ * tip to stderr (never an error): we recommend 3 or more of the same page type, up to 5.
  */
 import { ensureApiKey } from "../../core/ensure-key.js";
 import { loadConfig } from "../../core/config.js";
 import { loadPolicy } from "../../core/policy.js";
 import { log } from "../../core/logger.js";
 import { ToolkitError } from "../../core/errors.js";
-import { DEFAULT_WAIT_TIMEOUT_MS, MAX_EXAMPLE_URLS, failureNextStep, getAllowance, getPreparedDomain, normalizeDomain, prepareDomain, statusNextStep, waitForPreparedDomain, } from "../../core/prepared-domains-api.js";
+import { DEFAULT_WAIT_TIMEOUT_MS, MAX_EXAMPLE_URLS, RECOMMENDED_EXAMPLE_URLS, exampleUrlsHint, failureNextStep, getAllowance, getPreparedDomain, normalizeDomain, prepareDomain, statusNextStep, waitForPreparedDomain, } from "../../core/prepared-domains-api.js";
 import { parse, asString } from "../command.js";
 import { printError } from "../output.js";
 export const PREPARE_USAGE = "zenrows extract prepare <domain> --url <page-url> [--url <page-url>...] [--wait [--timeout <seconds>]]\n" +
@@ -23,7 +24,8 @@ export const PREPARE_HELP = [
     "Not for EXTRACT_PAGE_NOT_RECOGNISED: that domain is already prepared, and preparing it",
     "again does not change that answer.",
     "",
-    `  --url <page-url>       1 to ${MAX_EXAMPLE_URLS} real pages you want to extract, one per page kind (repeat --url)`,
+    `  --url <page-url>       a real page you want to extract (repeat --url, 1 to ${MAX_EXAMPLE_URLS}). One works;`,
+    `                         we recommend ${RECOMMENDED_EXAMPLE_URLS} or more of the same page type (e.g. 3 product pages)`,
     "  --wait                 poll every 30 seconds until the domain is ready or failed; off by default",
     `  --timeout <seconds>    give up waiting after this long (default ${DEFAULT_WAIT_TIMEOUT_MS / 1000})`,
     "  status <domain>        show progress and the account's monthly allowance",
@@ -33,6 +35,14 @@ export const PREPARE_HELP = [
     "account's monthly allowance; a domain that is already prepared answers ready at once and",
     "uses none. The domain is the host (shop.example.com and example.com are separate); a",
     "leading www. is dropped.",
+    "",
+    `1 to ${MAX_EXAMPLE_URLS} example URLs. One works; we recommend ${RECOMMENDED_EXAMPLE_URLS} or more of the same page type`,
+    "(e.g. 3 product pages), because Extract compares pages of one type to check the fields",
+    `it found and fill in fields one page is missing. Several page types can share the ${MAX_EXAMPLE_URLS}`,
+    "(e.g. 3 product pages + 2 search-result pages). Three product pages:",
+    "",
+    "  zenrows extract prepare example.com --url https://www.example.com/products/a \\",
+    "    --url https://www.example.com/products/b --url https://www.example.com/products/c --wait",
 ].join("\n");
 export async function runPrepare(argv, ctx, deps = {}) {
     const isStatus = argv[0] === "status";
@@ -69,6 +79,10 @@ export async function runPrepare(argv, ctx, deps = {}) {
             return 0;
         }
         log.step(`Prepare ${domain} for Extract (${urls.length} example URL${urls.length === 1 ? "" : "s"})…`);
+        // Advice on stderr only: the --json payload on stdout and the exit code are unchanged.
+        const hint = exampleUrlsHint(urls.length);
+        if (hint)
+            log.dim(hint);
         let record = await prepareDomain(domain, urls, opts);
         if (record.already_in_progress)
             log.info(`${domain} was already being prepared; following that preparation.`);
@@ -108,7 +122,7 @@ function validateUrls(raw, domain) {
             code: "INVALID_USAGE",
             message: `Pass 1 to ${MAX_EXAMPLE_URLS} example page URLs with --url (got ${urls.length}).`,
             likely_cause: "Preparation learns the domain from real pages you want to extract.",
-            next_action: "Add one --url per page kind, e.g. a product page and a listing page.",
+            next_action: `Add 1 to ${MAX_EXAMPLE_URLS} --url pages. One works; ${RECOMMENDED_EXAMPLE_URLS} or more of the same page type work better.`,
             suggested_commands: [`zenrows extract prepare ${domain} --url https://${domain}/<page>`],
         });
     }

package/dist/src/core/batch-api.js

@@ -581,7 +581,7 @@ export async function waitForJob(id, opts) {
                 throw err;
             lastErr = err;
         }
-        if (Date.now() + delay > deadline) {
+        if (Date.now() + delay >= deadline) {
             // Out of time: report the last state read, or the error if none was.
             if (!last)
                 throw lastErr;

package/dist/src/core/config.js

@@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0";
  * entrypoint) so core modules — telemetry, signup provenance — can read it
  * without an import cycle. `VERSION` in `cli/index.ts` re-exports this.
  */
-export const CLI_VERSION = "1.10.0"; // x-release-please-version
+export const CLI_VERSION = "1.12.0"; // x-release-please-version
 /** The User-Agent on every request to a Zenrows API. The gateway reads the version from it. */
 export const CLI_USER_AGENT = `zenrows-cli/${CLI_VERSION}`;
 /** Env var to override the Fetch and Extract API base (local/staging testing). */

package/dist/src/core/crawl-api.js

@@ -10,8 +10,9 @@
  * Crawl SDKs do (see `retryStatuses`). `fetchImpl` / `sleepImpl` are injectable
  * for tests.
  *
- * `output_format` takes `html` or `markdown` (absent = URLs only). Response fields we do
- * not model pass through untouched.
+ * `output_format` takes `html` or `markdown` (absent = URLs only). A `webhook` gets the
+ * crawl's events (`crawl.started` and how it ended) posted to a URL, signed with the
+ * account's webhook signing key. Response fields we do not model pass through untouched.
  */
 import { Readable } from "node:stream";
 import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
@@ -36,6 +37,10 @@ export function crawlBase() {
 }
 /** The API's "Crawl is not enabled for this account" code (403). */
 export const CRAWL_NOT_ENABLED_CODE = "REQS008";
+/** The events a crawl's webhook can send. */
+export const CRAWL_WEBHOOK_EVENTS = ["crawl.started", "crawl.completed", "crawl.stopped", "crawl.failed"];
+/** The API's limits on a webhook's `metadata`. */
+export const WEBHOOK_METADATA_LIMITS = { keys: 20, keyLength: 64, valueBytes: 1024 };
 /** A crawl stops progressing once its status is anything but `running`. */
 export function isTerminal(status) {
     return !!status && status !== "running";
@@ -273,7 +278,9 @@ function classifyProblem(status, problem, body, request, serverCode, retryAfter)
             likely_cause: cause,
             next_action: serverCode === "idempotency_key_reused"
                 ? "This Idempotency-Key was already used for a different request. Send a new key, or no key. Do not retry as is."
-                : "Fix the reported parameter and retry. See `zenrows crawl --help` for the accepted flags and ranges.",
+                : serverCode === "webhook_signing_key_required"
+                    ? "Nothing was created. Create a webhook signing key in the dashboard (Settings → API Keys), then retry."
+                    : "Fix the reported parameter and retry. See `zenrows crawl --help` for the accepted flags and ranges.",
         });
     }
     return new ToolkitError({
@@ -300,6 +307,14 @@ export function createCrawl(params, opts) {
         body.output_format = params.output_format;
     if (params.discovery?.length)
         body.discovery = params.discovery;
+    if (params.webhook !== undefined) {
+        const webhook = { url: params.webhook.url };
+        if (params.webhook.events?.length)
+            webhook.events = params.webhook.events;
+        if (params.webhook.metadata !== undefined)
+            webhook.metadata = params.webhook.metadata;
+        body.webhook = webhook;
+    }
     return crawlJson("POST", "/crawls", {
         ...opts,
         body,

package/dist/src/core/http.js

@@ -314,7 +314,7 @@ export function extractDomainNotPrepared(url, body) {
         code: "EXTRACT_DOMAIN_NOT_PREPARED",
         message: `Extract hasn't prepared ${domain} yet. Prepare it with \`${prepare}\` (a few example URLs, a few minutes, no credits), then retry; meanwhile use --css.`,
         likely_cause: detail ?? `Extract has not prepared ${domain} yet.`,
-        next_action: `Run \`${prepare} --wait\` (up to 5 --url pages, one per page kind you want), then retry \`zenrows extract ${url}\`. Without --wait, check progress about every 30 seconds with \`zenrows extract prepare status ${domain}\`. Meanwhile extract with --css and explicit selectors.`,
+        next_action: `Run \`${prepare} --wait\` (1 to 5 --url pages: one works; we recommend 3 or more of the same page type), then retry \`zenrows extract ${url}\`. Without --wait, check progress about every 30 seconds with \`zenrows extract prepare status ${domain}\`. Meanwhile extract with --css and explicit selectors.`,
         suggested_commands: [`${prepare} --wait`, `zenrows extract prepare status ${domain}`, css],
         server_code: "REQS007",
     });

package/dist/src/core/prepared-domains-api.js

@@ -4,7 +4,8 @@
  * `extract=auto` answers REQS007 on a domain Extract has not prepared yet. Preparing
  * it is a call any key can make: `POST {apiBase}/extract/prepared-domains` with the
  * domain and 1 to 5 example URLs answers 202 at once; the preparation itself takes a
- * few minutes and costs no credits. `GET .../{domain}` reports progress and
+ * few minutes and costs no credits. One example URL works; we recommend 3 or more of
+ * the same page type (see `RECOMMENDED_EXAMPLE_URLS`). `GET .../{domain}` reports progress and
  * `GET .../allowance` the account's monthly allowance of new domains. Auth is the
  * `X-API-Key` header; errors are problem+json and we branch on `status` + `code`.
  * `fetchImpl` / `sleepImpl` / `now` are injectable for tests.
@@ -14,6 +15,23 @@ import { registerSecret } from "./logger.js";
 import { CLI_USER_AGENT, loadConfig } from "./config.js";
 import { agentClientHeader } from "./agent-client.js";
 export const MAX_EXAMPLE_URLS = 5;
+/**
+ * How many example URLs to recommend: 3 or more of the same page type, because
+ * Extract compares pages of one type to check the fields it found and fill in fields
+ * one page is missing. Advice only: 1 and 2 URLs are still valid.
+ */
+export const RECOMMENDED_EXAMPLE_URLS = 3;
+/**
+ * A short, non-blocking hint when fewer than `RECOMMENDED_EXAMPLE_URLS` example
+ * URLs were given; undefined otherwise. Never an error.
+ */
+export function exampleUrlsHint(count) {
+    if (count >= RECOMMENDED_EXAMPLE_URLS)
+        return undefined;
+    return (`Tip: ${count === 1 ? "one example URL works" : `${count} example URLs work`}; we recommend ` +
+        `${RECOMMENDED_EXAMPLE_URLS} or more of the same page type (e.g. 3 product pages), up to ${MAX_EXAMPLE_URLS} in total, ` +
+        "because Extract compares pages of one type to check the fields it found and fill in fields one page is missing.");
+}
 /** How often `--wait` polls, and for how long by default. */
 export const DEFAULT_POLL_INTERVAL_MS = 30_000;
 export const DEFAULT_WAIT_TIMEOUT_MS = 15 * 60_000;
@@ -247,7 +265,7 @@ export function failureNextStep(domain, failure) {
             step = `Run the same zenrows extract prepare ${domain} command again.`;
             break;
         case "discovery_failed":
-            step = `Run zenrows extract prepare ${domain} again with 1 to ${MAX_EXAMPLE_URLS} --url page URLs.`;
+            step = `Run zenrows extract prepare ${domain} again with 1 to ${MAX_EXAMPLE_URLS} --url page URLs; we recommend ${RECOMMENDED_EXAMPLE_URLS} or more of the same page type (e.g. 3 product pages).`;
             break;
         case "unpreparable":
             step = `Run zenrows extract prepare ${domain} again with different --url pages from the same site; the same URLs fail again.`;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zenrows/cli",
-  "version": "1.10.0",
+  "version": "1.12.0",
   "description": "Open-source CLI, MCP, skills, templates, workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.",
   "type": "module",
   "license": "MIT",

package/registry/capabilities.json

@@ -17,7 +17,7 @@
       "command": "zenrows extract",
       "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, css_extractor, outputs, response_type; autoparse deprecated)",
       "requiresAuth": true,
-      "notes": "Default is extract=auto, which works on any prepared domain with any key. A REQS007 (legacy AUTH010) is reported, never retried, with the next step for its reason: on a domain not prepared yet (EXTRACT_DOMAIN_NOT_PREPARED), prepare it with `zenrows extract prepare <domain> --url <page> [--wait]` (POST /v1/extract/prepared-domains: 1 to 5 example URLs, a few minutes, no credits) and check it with `zenrows extract prepare status <domain>`; on a page Extract did not recognise on a prepared domain (EXTRACT_PAGE_NOT_RECOGNISED), try a product, article or listing page, retry it once in a few minutes, or use --css, because preparing again does not change it. CSS / outputs / markdown work on any domain; autoparse is deprecated and runs only with --autoparse."
+      "notes": "Default is extract=auto, which works on any prepared domain with any key. A REQS007 (legacy AUTH010) is reported, never retried, with the next step for its reason: on a domain not prepared yet (EXTRACT_DOMAIN_NOT_PREPARED), prepare it with `zenrows extract prepare <domain> --url <page> [--wait]` (POST /v1/extract/prepared-domains: 1 to 5 example URLs; one works, we recommend 3 or more of the same page type because Extract compares pages of one type to check the fields it found and fill in fields one page is missing; a few minutes, no credits) and check it with `zenrows extract prepare status <domain>`; on a page Extract did not recognise on a prepared domain (EXTRACT_PAGE_NOT_RECOGNISED), try a product, article or listing page, retry it once in a few minutes, or use --css, because preparing again does not change it. CSS / outputs / markdown work on any domain; autoparse is deprecated and runs only with --autoparse."
     },
     "batch": {
       "key": "batch",

package/skills/crawl/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: crawl
-description: "Collect the URLs behind one start page with Crawl (new): create a crawl, wait for it, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs."
+description: "Collect the URLs behind one start page with Crawl (new): create a crawl, wait for it or get notified by webhook, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs."
 version: 0.1.0
 requires_backend_capabilities: [crawl]
 ---
@@ -71,6 +71,36 @@ of URLs. A result's `content_status` is `pending`, `fetched` or `failed`; only
 `fetched` results have content (else `CRAWL_CONTENT_NOT_FOUND`). `download`
 reports the crawl's `status`, and `partial: true` while it runs.
 
+## Webhooks
+
+Instead of polling, have Crawl post its events to a URL you run:
+
+```
+zenrows crawl create <url> --depth 1 --webhook https://example.com/hooks/zenrows \
+  --webhook-event crawl.completed --webhook-event crawl.failed \
+  --webhook-metadata site=books --json
+```
+
+- `--webhook <url>` takes an absolute `https` URL. It is fixed at create.
+- `--webhook-event` repeats; each of `crawl.started`, `crawl.completed`,
+  `crawl.stopped`, `crawl.failed` at most once. Leave it out for all four.
+  Exactly one of the last three is sent per crawl.
+- `--webhook-metadata key=value` repeats (up to 20 keys of 1 to 64
+  characters, each value at most 1,024 bytes as JSON); every event carries it
+  back, to match the crawl to your own records.
+- The crawl shows `webhook` as `{url, events, metadata}`, `events` being the
+  effective list; a crawl without one has none.
+- Events are signed JSON (`schema: zenrows.webhook/v1`, `event_id`,
+  `event_type`, `occurred_at`, `resource {type: crawl, id, url}`, `metadata`).
+  Verify the `X-Signature: t=…,v1=…,kid=…` header: `v1` is the hex
+  HMAC-SHA256 of `"<t>." + body` with the account's webhook signing key
+  (Settings → API Keys). Events can repeat (drop one whose `event_id` you
+  handled) and arrive out of order (sort by `occurred_at`).
+- The account needs a webhook signing key: without one the create gets 422
+  `webhook_signing_key_required` (create the key, then retry). A bad URL,
+  event or metadata gets 422 `invalid_webhook`. Both are
+  `CRAWL_INVALID_REQUEST`, and nothing was created.
+
 ## While it runs, and after
 
 ```

package/skills/extract/SKILL.md

@@ -43,15 +43,21 @@ of three cases it is and prints the next step. It never retries with Autoparse.
 Preparing a domain:
 
 ```
-zenrows extract https://example.com/p/1             # 1. fails: EXTRACT_DOMAIN_NOT_PREPARED
+zenrows extract https://www.example.com/products/a  # 1. fails: EXTRACT_DOMAIN_NOT_PREPARED
 zenrows extract prepare example.com \
-  --url https://example.com/p/1 --url https://example.com/list --wait
+  --url https://www.example.com/products/a \
+  --url https://www.example.com/products/b \
+  --url https://www.example.com/products/c --wait
                                                     # 2. prepare it and wait until ready (a few minutes)
-zenrows extract https://example.com/p/1             # 3. extract again
+zenrows extract https://www.example.com/products/a  # 3. extract again
 ```
 
-- `--url`: 1 to 5 real pages you want to extract, one per page kind (for
-  example a product page and a listing page).
+- `--url`: 1 to 5 real pages you want to extract. One works; we recommend 3 or
+  more of the same page type (e.g. 3 product pages), because Extract compares
+  pages of one type to check the fields it found and fill in fields one page is
+  missing. Several page types can share the 5 (e.g. 3 product pages + 2
+  search-result pages). With fewer than 3 the command prints a tip on stderr;
+  it is never an error.
 - Without `--wait` the command returns at once; then check about every 30
   seconds with `zenrows extract prepare status example.com` until it says
   `ready`. `--wait` polls at the same pace.

package/skills/zenrows/SKILL.md

@@ -90,7 +90,8 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 
 - Extract: `extract=auto` works on any prepared domain. On one not prepared yet
   (`EXTRACT_DOMAIN_NOT_PREPARED`), run `zenrows extract prepare <domain> --url
-  <page> --wait`, then extract again. `EXTRACT_PAGE_NOT_RECOGNISED` means the
+  <page> --wait` (repeat `--url`, 1 to 5: one works; we recommend 3 or more of
+  the same page type), then extract again. `EXTRACT_PAGE_NOT_RECOGNISED` means the
   domain is prepared but not this page: do not prepare again; see [[extract]].
 - Batch: the reliable path at scale, at Fetch's prices (create, list, status, wait, results, content, pause, resume, rerun, delete). Estimate specs locally with no key.
 - Crawl: collects the URLs behind one start page, optionally with each page's HTML.