@zenrows/cli 1.4.2 → 1.5.1

package/README.md

@@ -37,8 +37,9 @@ that endpoint with extraction parameters, not a separate product.
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
 | `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` |
-| `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always |
+| `zenrows extract` | Extract: `extract=auto` (any prepared domain; falls back to Autoparse on one not prepared yet), CSS, or Markdown | available, same `/v1/` |
+| `zenrows batch` | Batch: fan out over many URLs | **available**: `https://async.api.zenrows.com/v1`; local validate/estimate need no key |
+| `zenrows crawl` | Crawl (new) — collect the URLs behind one start page, optionally with each page's HTML | **available** — `https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |
 | `zenrows mcp` | MCP server config (remote + local) | **available** |
 | Zenrows CLI | this repo | available |
@@ -122,12 +123,11 @@ zenrows extract <url> --outputs emails,links                            # built-
 zenrows extract <url> --output markdown
 ```
 
-## 9. Batch (beta)
+## 9. Batch
 
 Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a
-protected fetch/extract out over many URLs. It is a real product in
-**beta**: the cloud subcommands work once your API key has
-beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec
+protected fetch/extract out over many URLs. The cloud subcommands work
+with any active API key and spend the same credits as Fetch. Local spec
 validation + credit estimation always work (no key needed).
 
 ```bash
@@ -140,8 +140,7 @@ zenrows batch cancel <job-id>                        # stop an in-flight run
 
 `jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
 per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
-`extract`). Request beta access from Zenrows to run in the cloud; until then,
-validate/estimate locally or fan out with `zenrows fetch` per URL.
+`extract`). Validate and estimate locally first, then submit with `create`.
 
 **Extract in Batch.** Set `"extract": "auto"` on a line to run that URL through
 Extract — structured data instead of raw HTML:
@@ -156,7 +155,79 @@ An Extract task costs the same as a regular one (1 credit at base tier), so
 keys: `html` (the raw page) and `parsed` (the structured data) — validate a
 sample with `zenrows extract <url>` before running the full batch.
 
-## 10. Browser Sessions
+## 10. Crawl (new)
+
+Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.
+
+Crawl (`https://api.zenrows.com/v1/crawls`)
+takes one start URL and returns the URLs it finds behind it, optionally with
+each page's HTML. Use it when you know the listing or section but not the item
+URLs. Crawl follows the links on each page, up to `--depth` hops when you set
+it, and stays on the start URL's registrable domain (subdomains count). Without
+`--output-format html` it returns URLs only. An account without Crawl access gets
+`CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account", HTTP 403
+`REQS008`).
+
+```bash
+zenrows crawl create https://example.com/products/ --depth 1 \
+  --max-items 20 --include-patterns /product/ --follow  # create and wait for the end
+zenrows crawl results <crawl-id> --out urls.jsonl       # every kept URL (paginated)
+zenrows crawl create <url> --depth 2 --output-format html  # also fetch each kept page's HTML
+zenrows crawl content <crawl-id> <content-id>           # one page's HTML
+zenrows crawl download <crawl-id>                       # NDJSON export (URLs + HTML) → <crawl-id>.jsonl
+zenrows crawl get <crawl-id>                            # status, coverage, stop reason, one page of results
+zenrows crawl wait <crawl-id>                           # wait for the end
+zenrows crawl list                                      # your crawls, newest first
+zenrows crawl stop <crawl-id>                           # stop a running crawl
+```
+
+`--depth` (link hops, 1 to 100000) is optional: leave it out and the crawl
+follows links until `--max-items` or `--max-pages` stops it, or nothing is left
+to open. `--max-items` and `--max-pages` cap the crawl (API default 100 each),
+and `--max-pages` bounds the cost, since each page is one fetch. The local policy `max_pages_per_run` caps `--max-pages` (default
+1000). `--output-format html` uses up the page budget: each kept page is one
+more fetch, and `--max-pages` counts it. `--include-patterns` /
+`--exclude-patterns` are repeatable substring filters on the URL. `get` and
+`list` read one page and take `--cursor` and `--limit`. `results` follows every
+page (`--limit` is the page size); on a running crawl it returns the URLs kept
+so far, with `partial: true`. `download` reports the crawl's `status` (`null`
+when the API sends none), and `partial: true` while it runs.
+
+With `--json`, `create`, `get`, `wait` and `stop` print
+`{ok, crawl_id, status, crawl}`. The crawl object (with `coverage`,
+`stop_reason` and `error`) is under `crawl`. `get` adds `results` and
+`next_cursor`. All keys are snake_case.
+
+`wait`, and `create --follow`, poll until the crawl ends or `--timeout` seconds
+run out (default 600). When the time runs out, the command exits 0 and prints
+the crawl with status `running` and the hint `zenrows crawl wait <crawl-id>`
+(under `--json`, in `note`); the crawl keeps running. Ctrl-C stops the wait, not the crawl, and exits 130;
+resume with `zenrows crawl wait <crawl-id>`. A crawl that ends `failed` exits 1
+with `CRAWL_FAILED`.
+
+Errors carry the API's code as `server_code`, `crawl_id` on any call about one
+crawl, and `retry_after` on a 429. When the account has reached its limit of
+active jobs (3 by default), shared with its Batch jobs, a create gets 429
+`too_many_crawls`, mapped to `CRAWL_TOO_MANY_CRAWLS`, and creates nothing; retry
+after `retry_after` seconds. Any other 429 is `CRAWL_RATE_LIMITED`: requests
+are being throttled; retry after `retry_after` seconds. Other codes: `CRAWL_INSUFFICIENT_CREDITS` and `CRAWL_KEY_CAP_REACHED` (402, with the
+same claim, top-up and key-cap advice as other commands),
+`CRAWL_INVALID_REQUEST` (400/422, do not retry as is), `CRAWL_REQUEST_IN_FLIGHT`
+(409, retry once the first request ends), `CRAWL_NOT_FOUND`,
+`CRAWL_CONTENT_NOT_FOUND`, `CRAWL_BUSY` (503 on a stop that was not saved yet;
+run it again, it is safe) and `CRAWL_FAILED`. Set `ZENROWS_CRAWL_API_BASE` to
+point the CLI at another deployment.
+
+The CLI retries transient failures, as the Crawl SDKs do: up to 3 retries,
+waiting the `Retry-After` seconds (at most 30) or 250 ms x 2^n (+/-20% jitter,
+at most 10 s).
+Reads retry on 429, 502, 503, 504 and network errors (a timeout counts).
+`create` retries only with `--idempotency-key`, and only on 502, 503, 504 and
+network errors, never on 429. `stop` retries on 429, 502, 503 (a `crawl_busy`
+means the stop was not saved and the crawl still runs) and 504, waiting the
+`Retry-After` seconds (at most 30) or 1 s; repeating a stop is safe.
+
+## 11. Browser Sessions
 
 Escalation only — **prefer `fetch`/`extract` for the vast majority of cases**;
 they cost less. Use the browser for logins, forms, and multi-step JS flows that
@@ -170,7 +241,7 @@ bring-your-own Playwright/Puppeteer.
 close interactive sessions with `zenrows browser close`. On by default; opt out
 with `zenrows policy set allow_browser false`.
 
-## 11. MCP
+## 12. MCP
 
 ```bash
 zenrows mcp status
@@ -188,7 +259,7 @@ both servers in `mcp.json` (no secrets) and ships MCP-native skills under
 `agent-plugin/skills/`. It is included in the published `@zenrows/cli` npm
 tarball (`node_modules/@zenrows/cli/agent-plugin`).
 
-## 12. Plugins
+## 13. Plugins
 
 ```bash
 zenrows plugin list
@@ -201,7 +272,7 @@ MCP-native skills). See [`agent-plugin/README.md`](agent-plugin/README.md).
 Legacy per-client snippets from `zenrows plugin install` remain available.
 CLI-oriented skills stay under repo-root `skills/`.
 
-## 13. Skills
+## 14. Skills
 
 Agent-readable playbooks that teach agents how to choose primitives.
 
@@ -213,28 +284,28 @@ zenrows skill validate zenrows
 
 The master skill `skills/zenrows/SKILL.md` teaches the full decision tree.
 
-## 14. Templates
+## 15. Templates
 
 ```bash
 zenrows template list
 zenrows template create protected-fetch-node --output ./my-project
 ```
 
-## 15. Workflows
+## 16. Workflows
 
 ```bash
 zenrows workflow list
 zenrows workflow explain competitor-intelligence
 ```
 
-## 16. Recipes
+## 17. Recipes
 
 ```bash
 zenrows recipe list
 zenrows recipe run fetch-protected-page
 ```
 
-## 17. Evals
+## 18. Evals
 
 Reproducible, transparent benchmarks. **No competitor keys are bundled** and
 nothing is hardcoded to make Zenrows win — comparison evals require you to supply
@@ -249,7 +320,7 @@ zenrows eval report protected-fetch-smoke
 Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 `cost.json`, and `traces/` under `.zenrows/evals/<run-id>/`.
 
-## 18. Security
+## 19. Security
 
 - API keys are never printed or written into run artifacts/assets.
 - Secrets live in `.zenrows/secrets.json` (0600, gitignored); logs are redacted.
@@ -259,12 +330,12 @@ Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 - Destructive `uninstall` requires `--yes`. Browser is on by default; opt out with
   `zenrows policy set allow_browser false`. Experimental features are off by default.
 
-## 19. Capability matrix
+## 20. Capability matrix
 
 See [`registry/capabilities.json`](registry/capabilities.json). `zenrows status
 --json` emits it.
 
-## 20. Contributing
+## 21. Contributing
 
 The CLI is TypeScript with **zero runtime dependencies** (native `fetch`,
 `node:util` `parseArgs`, `node:test`). Node 20+ runs the published build;

package/agent-plugin/skills/batch-jobs/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: batch-jobs
-description: Scale protected fetch/extract over many URLs via Zenrows MCP batch_* tools (beta).
+description: Scale protected fetch/extract over many URLs via Zenrows MCP batch_* tools.
 version: 0.1.0
 ---
 
@@ -9,8 +9,8 @@ version: 0.1.0
 Process large workloads asynchronously. Batch is where Zenrows' high-scale
 anti-bot advantage becomes obvious — thousands to millions of URLs.
 
-> Status: **beta**. Cloud Batch lives on `async.api.zenrows.com/v1`. Without
-> beta access the API returns `BATCH_ACCESS_DENIED`. Do not fake a successful
+> Cloud Batch lives on `async.api.zenrows.com/v1`. A 403 maps to
+> `BATCH_ACCESS_DENIED`. Do not fake a successful
 > run. This is **not** `browser_batch` (that batches browser actions in one
 > session).
 

package/agent-plugin/skills/cost-control/SKILL.md

@@ -18,7 +18,7 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Premium proxies (`premium_proxy`): **10×**
 - Both: **25×**
 
-`extract` `mode=auto` is **open beta** and currently free; billing may apply
+`extract` `mode=auto` is currently free; billing may apply
 later. Do not promise permanent free extract pricing. Autoparse / CSS still
 follow the multipliers above when stealth flags are set.
 
@@ -32,7 +32,6 @@ Always call `browser_close`. Prefer `scrape` / `extract` when they can do the jo
 
 Batch is usually better at **scale** (many URLs) than fan-out of per-URL tool
 calls. Validate one URL with `scrape`/`extract` first, then `batch_create`.
-Batch access may be beta-gated (`BATCH_ACCESS_DENIED`).
 
 ## Tactics
 

package/agent-plugin/skills/extract/SKILL.md

@@ -20,10 +20,10 @@ extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
 extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
-`mode=auto` (`extract=auto`) is **open beta**: richest on enabled domains,
-currently free; billing may apply later. If the domain is not enabled (AUTH010)
-or not prepared yet (REQS007), the tool retries once with Autoparse by default
-(`fallback_autoparse`).
+`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
+key, and is currently free; billing may apply later. If the
+domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
+once with Autoparse by default (`fallback_autoparse`).
 
 Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
 protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
@@ -45,7 +45,7 @@ scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 
 ## When to use which
 
-- **`mode=auto`**: site-tailored fields on beta-enabled domains.
+- **`mode=auto`**: site-tailored fields on any prepared domain.
 - **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
 - **`mode=css`**: known fields/selectors; deterministic.
 - **Markdown via `scrape`**: feed clean content to an LLM yourself.

package/agent-plugin/skills/zenrows/SKILL.md

@@ -42,7 +42,7 @@ If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with scrape/extract first,
                                      then batch_create → batch_status → batch_results)
                                      Prefer Batch over fan-out of scrape/extract
-                                     per URL for bulk work (may be beta-gated).
+                                     per URL for bulk work.
                                      Cancel with batch_cancel when needed.
 
 If the user needs login, clicks, forms, sessions, or persistent state:
@@ -66,8 +66,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` open beta (currently free; billing may apply later) |
-| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Beta; may return `BATCH_ACCESS_DENIED` without access |
+| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
+| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 
 ## Cost notes

package/dist/src/adapters/batch.js

@@ -1,9 +1,8 @@
 /**
  * Batch Jobs adapter.
  *
- * Status: `beta` — the Zenrows Batch API is in beta.
- * With beta access the cloud subcommands run for real (see `core/batch-api.ts`).
- * Without access the API returns 403 → BATCH_ACCESS_DENIED. This adapter owns
+ * Status: `available`. The cloud subcommands call the Batch API with an API key
+ * (see `core/batch-api.ts`); a 403 maps to BATCH_ACCESS_DENIED. This adapter owns
  * the local, no-network pieces: validating a JSONL job spec, estimating credit
  * cost, and mapping the validated spec to the Batch API request body.
  */

package/dist/src/adapters/extract.js

@@ -55,6 +55,18 @@ async function runExtractOnce(opts, config, policy, apiKey) {
         output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html",
     };
     const outcome = await runFetch(fetchOpts, config, policy, apiKey);
+    if (outcome.result.noDataExtracted) {
+        if (opts.validate) {
+            throw new ToolkitError({
+                code: "EXTRACT_VALIDATION_FAILED",
+                message: "Extract found no data on this page.",
+                likely_cause: "Extract knows this kind of page, but none of the fields came back filled (HTTP 422 no_data_extracted). The request was not charged.",
+                next_action: "Check that the page rendered fully: try --manual --js-render with --wait-for <selector>. Protected sites may also need --premium-proxy.",
+                suggested_commands: [`zenrows extract ${opts.url} --manual --js-render --premium-proxy`],
+            });
+        }
+        return { ...outcome, method, empty: true, noDataExtracted: true };
+    }
     const { data, html, empty } = parseExtractBody(method, outcome.result.body, opts);
     return { ...outcome, method, data, html, empty };
 }
@@ -80,7 +92,7 @@ function parseExtractBody(method, body, opts) {
     }
     let data = parsed;
     let html;
-    // extract=auto beta shape: { parsed, html }. Prefer `parsed` for callers.
+    // extract=auto shape: { parsed, html }. Prefer `parsed` for callers.
     if (method === "extract" && parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
         const envelope = parsed;
         if ("parsed" in envelope) {

package/dist/src/cli/commands/batch.js

@@ -1,10 +1,10 @@
 /**
- * `zenrows batch` — Batch API (status: beta).
+ * `zenrows batch`: Batch API (status: available).
  *
  * Local (no key, always works): `estimate`/`create --dry-run`-style spec
- * validation + credit estimate. Cloud (needs a key + Batch beta access):
- * `create`, `status`, `results`, `cancel`, `wait`, `retry-failed`. Without beta
- * access the API returns 403 → BATCH_ACCESS_DENIED.
+ * validation + credit estimate. Cloud (needs a key):
+ * `create`, `status`, `results`, `cancel`, `wait`, `retry-failed`. A 403 maps to
+ * BATCH_ACCESS_DENIED.
  */
 import { statSync } from "node:fs";
 import { join } from "node:path";
@@ -22,12 +22,12 @@ import { ToolkitError, isKeyCapReached, keyCapReached } from "../../core/errors.
 import { printError, writeOut } from "../output.js";
 export const batch = {
     name: "batch",
-    summary: "Run JSONL batch jobs on Zenrows Batch (beta).",
+    summary: "Run JSONL batch jobs on Zenrows Batch.",
     usage: "zenrows batch <estimate|create <file.jsonl>|status <id>|results <id>|cancel <id>|wait <id>|retry-failed <id>>",
     help: [
         "Local (no key):",
         "  estimate <file.jsonl> [--manual] validate the spec + estimate credits (upper bound)",
-        "Cloud (needs a key + Batch beta access):",
+        "Cloud (needs a key):",
         "  create <file.jsonl> [flags]      validate, then submit the job",
         "    (default)                      Adaptive Stealth Mode (mode=auto) on every task that",
         "                                   doesn't force js_render / premium_proxy itself",