@zenrows/cli 1.4.2 → 1.5.1

package/skills/batch-jobs/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: batch-jobs
-description: Scale protected fetch/extract over many URLs via Batch (beta). Cloud create/status/results/cancel/wait/retry-failed work with beta access; estimate/validate run locally with no key.
+description: Scale protected fetch/extract over many URLs via Batch. Cloud create/status/results/cancel/wait/retry-failed work with an API key; estimate/validate run locally with no key.
 version: 0.1.0
 requires_backend_capabilities: [batch]
 ---
@@ -11,11 +11,10 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows'
 high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow
 runs over thousands, millions, or recurring sets of URLs.
 
-> Status: **beta**. The Zenrows **Batch** is a real
-> product in beta and runs on a separate host
-> (`async.api.zenrows.com/v1`). The cloud subcommands work once your account has
-> beta access; without it the API returns 403 → `BATCH_ACCESS_DENIED`. The
-> toolkit never fakes a cloud run.
+> Zenrows **Batch** runs on a separate host (`async.api.zenrows.com/v1`) and
+> spends the same credits as Fetch. The cloud subcommands work with any active
+> API key; a 403 maps to `BATCH_ACCESS_DENIED`. The toolkit never fakes a
+> cloud run.
 
 ## Local (no key, always works)
 ```
@@ -40,7 +39,7 @@ correctly. The result carries two keys: `html` (the
 raw page) and `parsed` (the structured data). Validate on a single URL with
 `zenrows extract <url>` before running the full batch.
 
-## Cloud (needs a key + beta access)
+## Cloud (needs a key)
 ```
 zenrows batch create jobs.jsonl [--proxy-country cc] [--wait]
 zenrows batch create jobs.jsonl --manual [--js-render] [--premium-proxy]

package/skills/crawl/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: crawl
+description: Collect the URLs behind one start page with Crawl (new): create a crawl, wait for it, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs.
+version: 0.1.0
+requires_backend_capabilities: [crawl]
+---
+
+# Crawl (new)
+
+Give Crawl one start URL; it follows the links on that page (and on the
+pages behind it, up to the depth you set) and returns the URLs it keeps. Use it
+when you know where the items live (a category page, a blog index, a docs
+section) but not the item URLs themselves. Once you have the URLs, read pages
+with `zenrows fetch` / `zenrows extract`, or ask Crawl for each page's HTML
+with `--output-format html`.
+
+Crawl stays on the start URL's registrable domain; subdomains count. An account without Crawl access gets
+403 `REQS008` → `CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account");
+do not retry it.
+
+## Create, wait, read
+
+```
+zenrows crawl create https://example.com/products/ --depth 1 \
+  --max-items 20 --max-pages 25 --include-patterns /product/ --follow --json
+zenrows crawl results <id> --json            # every kept URL
+zenrows crawl results <id> --out urls.jsonl  # same, as JSONL
+```
+
+- `--depth N` (1 to 100000) is optional: link hops from the start URL. `1` =
+  the start page's links; `2` = also the links on each of those pages. Leave
+  it out for no depth limit: the crawl follows links until `--max-items` or
+  `--max-pages` stops it, or nothing is left to open. A crawl created without
+  a depth has no `depth` field in its output.
+- `--max-items` (kept URLs) and `--max-pages` (fetched pages) cap the crawl;
+  both default to 100. `--max-pages` bounds the cost: each page is one fetch.
+  The local policy `max_pages_per_run` caps `--max-pages` (default 1000);
+  above it the CLI refuses with `POLICY_LIMIT_EXCEEDED` before any request.
+- `--include-patterns` / `--exclude-patterns` are substring matches on the
+  normalized URL and repeat:
+  `--include-patterns /product/ --exclude-patterns ?add-to-cart`.
+- `create` returns at once. `--follow` (or `zenrows crawl wait <id>`) polls
+  until the crawl ends or `--timeout` seconds run out (default 600). When the
+  time runs out, the command exits 0 and prints the crawl with status
+  `running` and the hint `zenrows crawl wait <id>` (under `--json`, in
+  `note`). The crawl keeps running;
+  it is not stopped. Ctrl-C also stops the wait, not the crawl, and exits 130;
+  resume with `zenrows crawl wait <id>`.
+- `results` on a running crawl returns the URLs kept so far, with
+  `partial: true`. Read again once the crawl ends. `--limit` sets the page size
+  of each request.
+
+## Page HTML
+
+```
+zenrows crawl create <url> --depth 1 --output-format html --max-pages 20 --follow
+zenrows crawl results <id> --json        # fetched results carry content_url
+zenrows crawl content <id> <content_id>  # one page's HTML (or pass the content_url)
+zenrows crawl download <id> --out pages.jsonl  # every URL + HTML, NDJSON
+```
+
+`--output-format html` uses up the page budget: each kept page is one more
+fetch, and `--max-pages` counts it. Raise `--max-pages` to keep the same number
+of URLs. A result's `content_status` is `pending`, `fetched` or `failed`; only
+`fetched` results have content (else `CRAWL_CONTENT_NOT_FOUND`). `download`
+reports the crawl's `status`, and `partial: true` while it runs.
+
+## While it runs, and after
+
+```
+zenrows crawl get <id>                       # status, coverage, stop_reason / error, one page of results
+zenrows crawl get <id> --cursor <c> --json   # the next page; keep polling with next_cursor
+zenrows crawl list                           # your crawls, newest first (--cursor, --limit)
+zenrows crawl stop <id>                      # stop it; kept URLs stay readable
+```
+
+A crawl ends `completed` (nothing left, or a cap hit: `stop_reason`
+`max_items` / `max_pages`), `stopped` (you stopped it) or `failed` (`error.code`
+and `error.detail` say why). Only `failed` exits non-zero (`CRAWL_FAILED`).
+`stop` on a crawl that already ended prints the status it ended with.
+
+With `--json`, `create`, `get`, `wait` and `stop` print
+`{ok, crawl_id, status, crawl}`; read `coverage`, `stop_reason` and `error`
+under `crawl`. `get` adds `results` and `next_cursor`. `download` prints
+`status: null` when the API sends no status.
+
+## Limits
+
+When the account has reached its limit of active jobs (3 by default), shared
+with its Batch jobs, a create gets 429 `too_many_crawls` →
+`CRAWL_TOO_MANY_CRAWLS`; nothing was created. Retry after `retry_after`
+seconds (about 30), or stop one of your crawls or Batch jobs first. Any other
+429 is `CRAWL_RATE_LIMITED`: requests are being throttled; retry after
+`retry_after` seconds.
+
+## Errors
+
+Every Crawl error carries the API's code as `server_code` (when the API sent
+one), and `crawl_id` on any call about one crawl.
+
+The CLI retries transient failures, as the Crawl SDKs do: up to 3 retries,
+waiting the `Retry-After` seconds (at most 30) or 250 ms x 2^n (+/-20% jitter,
+at most 10 s). Reads retry on 429, 502, 503, 504 and network errors (a timeout
+counts). `create` retries only with `--idempotency-key`, and only on 502, 503,
+504 and network errors, never on 429. `stop` retries on 429, 502, 503 (including
+`crawl_busy`) and 504, waiting the `Retry-After` seconds (at most 30) or 1 s.
+The "Retry?" column is for errors that remain after these retries.
+
+| Code | Meaning | Retry? |
+| --- | --- | --- |
+| `CRAWL_NOT_ENABLED` | 403 `REQS008`: Crawl is not enabled for this account | no |
+| `AUTH_INVALID` | 401: the API key was rejected | no, log in again |
+| `CRAWL_INSUFFICIENT_CREDITS` | 402: the account is out of credits (claim the account or add credits) | after credits renew or are added |
+| `CRAWL_KEY_CAP_REACHED` | 402 `AUTH014` or `api_key_cap_reached`: this API key reached its credit cap | after the cap resets or is raised |
+| `CRAWL_INVALID_REQUEST` | 400 or 422: fix the request. For `idempotency_key_reused`, send a new key or no key | no, not as is |
+| `CRAWL_REQUEST_IN_FLIGHT` | 409: the same Idempotency-Key is still in flight | yes, once the first request ends |
+| `CRAWL_NOT_FOUND` / `CRAWL_CONTENT_NOT_FOUND` | 404: wrong id, or the page has no content | no |
+| `CRAWL_TOO_MANY_CRAWLS` | 429 `too_many_crawls` on `create`: the account's active-job limit | yes, after `retry_after` |
+| `CRAWL_RATE_LIMITED` | any other 429: requests are being throttled | yes, after `retry_after` |
+| `CRAWL_BUSY` | 503 `crawl_busy` on `stop`: the stop was not saved yet and the crawl still runs | yes, run the stop again after `retry_after` |
+| `CRAWL_FAILED` | the crawl ended `failed`, or another error | 5xx: yes, with backoff |
+| `BACKEND_UNAVAILABLE` | the API could not be reached | yes |
+
+Rules: start small (`--depth 1`, low caps) and check the results before a
+deep crawl. Mind the cost ([[cost-control]]). Read errors with
+[[trace-debug]].

package/skills/extract/SKILL.md

@@ -12,7 +12,7 @@ access + extraction**, not generic LLM parsing.
 
 ## Methods
 ```
-zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if domain not enabled
+zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if the domain is not prepared yet
 zenrows extract <url> --autoparse                  # general-purpose Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}'   # selector map
 zenrows extract <url> --outputs emails,links       # built-in output filters → JSON
@@ -21,8 +21,8 @@ zenrows extract <url> --validate                   # fail if not valid JSON
 ```
 
 ## When to use which
-- **Extract** (`extract=auto`, default): richest fields on domains enabled for the open beta.
-- **Autoparse**: any website; also the automatic fallback when Extract is not enabled for the domain.
+- **Extract** (`extract=auto`, default): richest fields; works on any domain Zenrows has prepared.
+- **Autoparse**: any website; also the automatic fallback when the domain is not prepared for Extract yet.
 - **CSS**: you know the exact fields/selectors and want determinism.
 - **Markdown/plaintext**: feed clean content to an LLM yourself.
 

package/skills/trace-debug/SKILL.md

@@ -45,6 +45,18 @@ zenrows trace export <run-id>     # JSON for sharing
   auto-created Free plan.
 - `BATCH_FAILED` → the batch run ended `failed`; `failure_reason` and
   `failure_detail` in `zenrows batch status <id> --json` say why.
+- `CRAWL_NOT_ENABLED` → the API answered 403 `REQS008`: Crawl is not
+  enabled for this account. Do not retry; ask Zenrows support for access.
+- `CRAWL_INSUFFICIENT_CREDITS` / `CRAWL_KEY_CAP_REACHED` → Crawl's 402: the
+  account is out of credits, or this key reached its credit cap (`AUTH014`).
+  Same advice as `POLICY_MAX_CREDITS_EXCEEDED` / `KEY_CREDIT_CAP_REACHED`.
+- `CRAWL_TOO_MANY_CRAWLS` → 429 `too_many_crawls`: the account has reached
+  its limit of active jobs (3 by default), shared with its Batch jobs.
+  Nothing was created; retry after `retry_after` seconds (see
+  `zenrows crawl list`).
+- `CRAWL_FAILED` → the crawl ended `failed`; `error.code` in
+  `zenrows crawl get <id> --json` says why, with `error.detail`. See
+  [[crawl]].
 - `PARAM_CONFLICT_AUTO_MANUAL` → drop the managed flags or add `--manual`.
 - `CAPABILITY_UNAVAILABLE` → the primitive is not available on this account (e.g. beta/invite-only); use the local-spec path where offered.
 

package/skills/zenrows/SKILL.md

@@ -34,6 +34,10 @@ If the user has a known URL and wants page content:
 If the user has a known URL and wants structured data:
   → Use Extract.                    (zenrows extract <url> | --autoparse | --css)
 
+If the user has a start page (listing, index, section) but not the item URLs:
+  → Use Crawl.                      (zenrows crawl create <url> --depth 1 --follow,
+                                     then `zenrows crawl results <id>`)
+
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with fetch/extract first,
                                      then `zenrows batch create <jobs.jsonl>`)
@@ -68,8 +72,9 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 | Primitive | Command | Status |
 | --- | --- | --- |
 | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) |
-| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | beta (same `/v1/`; extract=auto falls back to autoparse) |
-| Batch | `zenrows batch` | beta — runs cloud jobs (create/status/results); cheapest path at scale. Estimate specs locally with no key |
+| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`; extract=auto falls back to autoparse) |
+| Batch | `zenrows batch` | available: runs cloud jobs (create/status/results); cheapest path at scale. Estimate specs locally with no key |
+| Crawl | `zenrows crawl` | new — collect the URLs behind one start page, optionally with each page's HTML |
 | Browser | `zenrows browser` | available (Browser Sessions REST API / MCP) |
 | MCP | `zenrows mcp` | available (remote + local server) |
 

package/templates/batch-jsonl-pipeline/README.md

@@ -16,10 +16,10 @@ results carry `html` and `parsed` keys.
 zenrows batch estimate jobs.example.jsonl    # validate the spec + estimate credits
 ```
 
-## Cloud (needs a key + Batch beta access)
+## Cloud (needs a key)
 
-Batch is in **beta**. With beta access the
-cloud subcommands run for real; without it the API returns `BATCH_ACCESS_DENIED`.
+The cloud subcommands run with any active API key and spend the same credits
+as Fetch.
 
 ```bash
 zenrows batch create jobs.example.jsonl --wait          # submit + wait for the run to finish
@@ -33,5 +33,5 @@ Tasks use Adaptive Stealth Mode (`mode=auto`) by default. Job-level scrape
 options come from flags (`--js-render`, `--premium-proxy`, `--proxy-country`,
 `--output`, or `--manual` to turn Adaptive Stealth Mode off); per-line keys
 override them for that URL.
-Without beta access yet, validate/estimate locally or fan out with
-`zenrows fetch` per URL on a small sample first.
+Validate/estimate locally, or try `zenrows fetch` per URL on a small sample
+first.

package/templates/extract-pricing-pages/README.md

@@ -13,5 +13,5 @@ zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
 zenrows extract <product-url> --css '{"title":"h1","price":".price"}' --validate
 ```
 
-Requires `extract` (open beta). Start on one page, confirm the fields, then
+Requires `extract`. Start on one page, confirm the fields, then
 fan out `zenrows extract` across the rest of your URLs.

package/workflows/competitor-intelligence/WORKFLOW.md

@@ -20,7 +20,7 @@ simple per-URL loop).
    ```
    zenrows fetch https://competitor.example/products
    ```
-2. **Validate extraction** on one page (extract is beta; extract=auto falls back to autoparse).
+2. **Validate extraction** on one page (extract=auto falls back to autoparse on a domain not prepared yet).
    ```
    zenrows extract https://competitor.example/p/123 --validate
    ```