@zenrows/cli 1.7.0 → 1.9.0

package/README.md

@@ -37,7 +37,7 @@ that endpoint with extraction parameters, not a separate product.
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
 | `zenrows fetch` | Fetch: retrieve a protected page | **available**: `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (falls back to Autoparse on one not prepared yet), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
+| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (prepare one yourself with `zenrows extract prepare`), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
 | `zenrows batch` | Batch: run Fetch and Extract over many URLs in one job; specs validate and estimate locally with no key | **available**: `POST https://async.api.zenrows.com/v1/jobs` |
 | `zenrows crawl` | Crawl (new): collect the URLs behind one start page, optionally with each page's HTML | **available**: `POST https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions: a real cloud browser over REST (same backend as MCP `browser_*`); escalation-only, billed by bandwidth + time | **available**: `POST https://mcp.zenrows.com/browser/sessions` |
@@ -56,7 +56,7 @@ npm install -g @zenrows/cli # puts `zenrows` on PATH (466 ms, zero deps)
 zenrows init
 zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use
 zenrows extract https://www.owler.com/company/meltwater # extract=auto on a prepared domain
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --css '{"title":"h1"}' # CSS selectors (any domain)
 ```
 
 > Prefer no global install? Prefix each command with `npx -y @zenrows/cli`
@@ -118,30 +118,101 @@ zenrows fetch <url> --proxy-country us --wait-for ".price"
 
 ```bash
 zenrows extract https://www.owler.com/company/meltwater                # extract=auto (prepared domain)
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # deprecated Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}' --validate
 zenrows extract <url> --outputs emails,links                            # built-in output filters → JSON
 zenrows extract <url> --output markdown
 ```
 
+On a domain Extract hasn't prepared yet, `zenrows extract` fails with
+`EXTRACT_DOMAIN_NOT_PREPARED` (API code REQS007) and prints the command to
+prepare it. Preparing takes a few minutes and costs no credits; each new domain
+uses one slot of the monthly allowance. `EXTRACT_PAGE_NOT_RECOGNISED` (also
+REQS007) is different: the domain is prepared, but this page is not one Extract
+extracts. Try a product, article or listing page instead, retry it once in a
+few minutes if it should work, or use `--css`; preparing again does not change
+it. `zenrows extract` never falls back to Autoparse. The `fallbackAutoparse`
+adapter option is deprecated and has no effect.
+
+```bash
+zenrows extract prepare example.com --url https://example.com/p/1 --url https://example.com/list --wait
+zenrows extract prepare status example.com   # progress + monthly allowance
+zenrows extract https://example.com/p/1      # then extract again
+```
+
 ## 9. Batch
 
 Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a
 protected fetch/extract out over many URLs. The cloud subcommands work
-with any active API key and spend the same credits as Fetch. Local spec
-validation + credit estimation always work (no key needed).
+with any active API key and cost the same credits as Fetch: Batch is the
+reliable path at scale, at Fetch's prices. Local spec validation + credit
+estimation always work (no key needed).
 
 ```bash
-zenrows batch estimate jobs.jsonl                    # validate the spec + estimate credits (local)
-zenrows batch create jobs.jsonl --wait               # submit a job and wait for it to finish
-zenrows batch status <job-id>                        # run status + progress stats
-zenrows batch results <job-id> --out results.jsonl   # collect results (paginated)
-zenrows batch cancel <job-id>                        # stop an in-flight run
+zenrows batch estimate jobs.jsonl                     # validate the spec + estimate credits (local)
+zenrows batch create jobs.jsonl --follow              # submit a job and wait for it to finish
+zenrows batch list                                    # your jobs, newest first
+zenrows batch status <job-id>                         # run status, progress stats, pause state
+zenrows batch wait <job-id> --timeout 30m             # wait for the run to end
+zenrows batch results <job-id> --out results.jsonl    # collect results (paginated)
+zenrows batch content <job-id> <task-id>              # one task's result body
+zenrows batch pause <job-id>                          # pause the run (resume <job-id> continues it)
+zenrows batch cancel <job-id>                         # stop an in-flight run
+zenrows batch retry-failed <job-id> --include-pending # rerun every task that did not succeed
+zenrows batch rerun <job-id>                          # rerun failed + pending tasks (--all: every task)
+zenrows batch delete <job-id> --yes                   # delete the job, its runs and results
 ```
 
 `jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
 per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
-`extract`). Validate and estimate locally first, then submit with `create`.
+`extract`, nested `zenrows_params`, an `external_id`, string `metadata`, and
+`"method": "POST"` with a `body` of up to 16 KiB). `estimate` and `create`
+check every line against the API's limits before anything is sent: `url` up
+to 2048 bytes, `external_id` up to 128 letters, digits, `.`, `_` or `-`,
+and `metadata` up to 20 string entries (4 KiB). Validate and estimate locally
+first, then submit with `create`. `estimate` takes `create`'s
+`--manual` / `--js-render` / `--premium-proxy` / `--proxy-country` flags and
+prices the same request, as a range (Adaptive Stealth Mode costs 1 to 25).
+
+`create` also takes `--name`, `--external-id` and `--metadata key=value`
+(repeatable) for the job itself. It sends an `Idempotency-Key` (generated, or
+`--idempotency-key`) and retries network errors and 502/503/504 under it; an
+outcome that stays unknown prints the command that repeats the request with
+that key, which cannot create a second job.
+
+The local policy caps a run at 10,000 URLs and 250,000 estimated credits by
+default (an Adaptive Stealth Mode task counts 25), so a default run of up to
+10,000 URLs passes in any mode. Raise them for larger jobs (the API takes up to
+100,000 tasks per job): `zenrows policy set max_pages_per_run 100000` and
+`zenrows policy set max_credits_per_run 2500000`. A workspace created by an
+earlier version keeps its caps (1,000 and 10,000) in `policy.json`.
+
+`wait`, `create --follow` and `rerun --follow` poll until the run ends, the run
+is paused, or `--timeout` runs out (default 10 minutes; `30s`, `5m`, `1h`; a
+bare number is milliseconds, as before). When the time runs out, or the run is
+paused, the command exits 0 and prints the run with a `note` and the next
+command; the job keeps running. Ctrl-C stops the wait, not the job, and exits
+130. A run that ends `failed` exits 1: out of credits is
+`POLICY_MAX_CREDITS_EXCEEDED`, an inactive subscription `AUTH_INVALID`, a key
+cap `KEY_CREDIT_CAP_REACHED` (the same codes a refused request gets), anything
+else `BATCH_FAILED`, each with the run's `failure_reason` as `server_code` and
+the `job_id`. Then run the unfinished tasks with
+`zenrows batch rerun <job-id> --status failed,pending,processing`. With no
+flag, `rerun` reruns the `failed` and `pending` tasks, so successful tasks are
+never billed again by surprise; `--status <list>` picks the statuses and
+`--all` reruns every task (not with `--status`). A partial rerun works for 10
+days after the run started; `--all` always does.
+
+With `--json`, `create`, `status`, `wait`, `cancel`, `pause`, `resume` and
+`rerun` print `{ok, job_id, run_id, status, stats, job}` (`jobId` is kept for
+older scripts), plus `paused`, `ingest_status`, `accepted_tasks`, `rerun_of`,
+`retried_tasks` and `inherited_tasks` when the API sends them. Results are
+kept 14 days by default and then read `expired`; a `result_url` is valid 2
+hours. Lifecycle errors get their own codes (`BATCH_RUN_IN_PROGRESS`,
+`BATCH_NO_MATCHING_TASKS`, `BATCH_RERUN_WINDOW_EXPIRED`,
+`BATCH_REQUEST_IN_FLIGHT`, `BATCH_RESULT_EXPIRED`, ...) and carry the API's code
+as `server_code`. Schedules, webhooks, ZIP exports, CSV inputs and open jobs
+are available through the Batch API and the SDKs, not this CLI.
 
 **Extract in Batch.** Set `"extract": "auto"` on a line to run that URL through
 Extract — structured data instead of raw HTML:
@@ -186,7 +257,7 @@ zenrows crawl stop <crawl-id>                           # stop a running crawl
 follows links until `--max-items` or `--max-pages` stops it, or nothing is left
 to open. `--max-items` and `--max-pages` cap the crawl (API default 100 each),
 and `--max-pages` bounds the cost, since each page is one fetch. The local policy `max_pages_per_run` caps `--max-pages` (default
-1000). `--output-format html` uses up the page budget: each kept page is one
+10000). `--output-format html` uses up the page budget: each kept page is one
 more fetch, and `--max-pages` counts it. `--include-patterns` /
 `--exclude-patterns` are repeatable substring filters on the URL. A crawl whose
 pages held nothing new to follow ends `completed` with no results and exits 0.

package/agent-plugin/README.md

@@ -62,7 +62,7 @@ Use MCP tool names — not CLI `zenrows fetch`:
 | --- | --- |
 | `zenrows` | Decision tree across all primitives |
 | `protected-fetch` | `scrape` |
-| `extract` | `extract` (fallback: `scrape` + autoparse / css_extractor) |
+| `extract` | `extract` (fallback: `scrape` + css_extractor); on REQS007 with reason `domain_not_prepared`, `prepare_domain` then `prepared_domain_status` |
 | `batch-jobs` | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` |
 | `interact-browser` | `browser_*` (escalation only) |
 | `cost-control` | Cost multipliers + escalate-with-evidence |

package/agent-plugin/skills/batch-jobs/SKILL.md

@@ -7,7 +7,8 @@ version: 0.1.0
 # Batch Jobs (MCP)
 
 Process large workloads asynchronously. Batch is where Zenrows' high-scale
-anti-bot advantage becomes obvious — thousands to millions of URLs.
+anti-bot advantage becomes obvious: thousands to 100,000 URLs per job, at
+Fetch's prices.
 
 > Cloud Batch lives on `async.api.zenrows.com/v1`. A 403 maps to
 > `BATCH_ACCESS_DENIED`. Do not fake a successful
@@ -33,10 +34,22 @@ Optional if exposed by the server: `batch_wait` to block until terminal status.
    `autoparse`, `extract`, `external_id`, `metadata`). `extract: "auto"`
    returns structured data instead of raw HTML, at the same credit cost as a
    regular task; the result carries `html` and `parsed` keys.
-3. Poll `batch_status` until terminal (`completed` / `stopped` / …).
-4. Collect with `batch_results`.
+3. Poll `batch_status` until terminal (`completed`, `stopped`, `failed` or
+   `deleted`). A paused run (`pause_state: paused`) is not terminal and picks
+   up no new task until resumed: report it instead of polling on.
+4. Collect with `batch_results`. Results are kept 14 days by default (then a
+   row reads `expired`); a `result_url` is valid 2 hours.
 5. `batch_cancel` if the user aborts or the run should stop early.
 
+A run ends `failed` when an account problem stops it: `failure_reason`
+`insufficient_credits` (add credits), `subscription_inactive` (fix it in
+Billing) or `api_key_cap_reached` (the key's own cap; the account still has
+credits). The tasks not yet run stay `pending`: once fixed, rerun them with
+status `failed,pending,processing` rather than resubmitting the job.
+
+Schedules, webhooks, ZIP exports and CSV inputs are available through the
+Batch API and the SDKs.
+
 ## Rules
 
 - **Never batch before validating** on a single page ([[extract]] /

package/agent-plugin/skills/cost-control/SKILL.md

@@ -19,8 +19,8 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Both: **25×**
 
 `extract` `mode=auto` is currently free; billing may apply
-later. Do not promise permanent free extract pricing. Autoparse / CSS still
-follow the multipliers above when stealth flags are set.
+later. Do not promise permanent free extract pricing. CSS (and the deprecated
+Autoparse) still follow the multipliers above when stealth flags are set.
 
 ## Browser sessions (`browser_*`)
 

package/agent-plugin/skills/extract/SKILL.md

@@ -14,28 +14,67 @@ Convert protected pages into structured data. Value is **protected page access
 Use **`extract`** when you need structured JSON fields rather than a full page:
 
 ```
-extract({ url })                                    # mode=auto (default)
-extract({ url, mode: "autoparse" })                 # general Autoparse (any domain)
+extract({ url })                                    # mode=auto (default), returns `parsed` only
+extract({ url, include_html: true })                # also return the full page HTML
+extract({ url, mode: "autoparse" })                 # deprecated Autoparse (any domain)
 extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
 extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
-`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
-key, and is currently free; billing may apply later. If the
-domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
-once with Autoparse by default (`fallback_autoparse`).
+`mode=auto` (`extract=auto`) works on any prepared domain, with any key, and
+is currently free; billing may apply later. The result carries `parsed` only;
+the page HTML is most of the response size, so ask for it with
+`include_html: true` only when you need it.
 
 Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
 protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
 fixed configuration; that turns Adaptive Stealth Mode off. Check `empty` in the
 result: true means no field came back with a value.
 
+## REQS007: prepare the domain, then extract again
+
+`mode=auto` works on any prepared domain, and you can prepare one yourself.
+When Extract refuses a page, `extract` returns a tool error with code
+`REQS007` (legacy `AUTH010`) and a `reason`. It never retries with Autoparse.
+`fallback_autoparse` is deprecated and has no effect; don't send it.
+
+| `reason` | Meaning | Next step |
+| --- | --- | --- |
+| `domain_not_prepared` | The domain is not prepared yet. | `prepare_domain` (the error's `next_tool` holds its arguments), then poll, then `extract` again. |
+| `page_not_recognised` | The domain is prepared, but Extract did not recognise this page as one it can extract (a page with nothing to extract, such as a home page, or a transient failure). | Try a product, article or listing page on the same site instead. If this page should work, retry it once in a few minutes; do not retry in a loop. Otherwise use `mode=css`. Do not call `prepare_domain`: preparing again does not change this. |
+| `domain_not_enabled` | Extract is not enabled for the site. | Use `mode=css`, or contact Zenrows support to have the site enabled. Preparing does not change this. |
+
+Preparing a domain:
+
+```
+extract({ url })                                          # 1. REQS007, reason domain_not_prepared
+prepare_domain({ domain: "example.com",
+  example_urls: ["https://example.com/p/1", "https://example.com/list"] })
+                                                          # 2. returns at once, status "preparing"
+prepared_domain_status({ domain: "example.com" })         # 3. about every 30 seconds until "ready"
+extract({ url })                                          # 4. extract again
+```
+
+- `example_urls`: 1 to 5 real pages you want to extract, one per page kind
+  (for example a product page and a listing page).
+- The domain is the host: `shop.example.com` and `example.com` are prepared
+  separately, and a leading `www.` is dropped.
+- Preparing takes a few minutes and costs no credits. Each new domain uses one
+  slot of the account's monthly allowance; `prepared_domain_status` returns it
+  (`used` / `limit` / `remaining`) and a `next` field with what to do now.
+- `out_of_allowance`: no slots left this month; use `mode=css`.
+  `overloaded` / `disabled`: retry in a few minutes; do not loop.
+  A `failed` status carries `failure.code` and `failure.slot_consumed`; follow `next`.
+- A domain that is already prepared answers `ready` at once
+  (`already_prepared: true`) and uses no slot: call `extract`. If it then
+  returns `page_not_recognised`, follow that row of the table, not `prepare_domain`.
+
 ## Fallback: `scrape` extraction params
 
 If `extract` is unavailable, similar outcomes are available on **`scrape`**:
 
 ```
-scrape({ url, autoparse: true })
+scrape({ url, autoparse: true })                    # deprecated
 scrape({ url, css_extractor: '{"title":"h1","price":".price"}' })
 scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ```
@@ -46,7 +85,8 @@ scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ## When to use which
 
 - **`mode=auto`**: site-tailored fields on any prepared domain.
-- **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
+- **`mode=autoparse`** (deprecated): any website, only when asked for. Do not
+  build new work on it.
 - **`mode=css`**: known fields/selectors; deterministic.
 - **Markdown via `scrape`**: feed clean content to an LLM yourself.
 
@@ -57,7 +97,7 @@ Stealth flags (`js_render`, `premium_proxy`) still apply the usual multipliers
 ## Rules
 
 - Validate on a single page before `batch_create` ([[batch-jobs]]).
-- If Autoparse misses fields, switch to `css_extractor`, or add `js_render`
+- If a field is missing, switch to `css_extractor`, or add `js_render`
   for JS-heavy pages.
 - Prefer MCP `extract` / `scrape` over CLI `zenrows extract` when the MCP
   server is connected.

package/agent-plugin/skills/protected-fetch/SKILL.md

@@ -34,7 +34,7 @@ scrape({ url, js_render: true, wait_for: ".price" })
 - Prefer `response_type: "markdown"` for LLM consumption; use `html` only when
   you need raw markup.
 - For **structured** fields, prefer the `extract` tool (or `scrape` with
-  `autoparse` / `css_extractor` / `outputs`) — see [[extract]].
+  `css_extractor` / `outputs`) — see [[extract]].
 - On failure, escalate with evidence before `browser_*` ([[interact-browser]]).
 - Cost model: [[cost-control]].
 

package/agent-plugin/skills/zenrows/SKILL.md

@@ -37,7 +37,7 @@ If the user has a known URL and wants page content:
 
 If the user has a known URL and wants structured data:
   → Use Extract.                    (MCP tool: extract)
-                                     Fallback: scrape with autoparse / css_extractor / outputs
+                                     Fallback: scrape with css_extractor / outputs
 
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with scrape/extract first,
@@ -67,7 +67,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
+| Extract | `extract` | Structured JSON (`extract=auto` on any prepared domain, css_extractor; autoparse deprecated). `extract=auto` is currently free; billing may apply later |
+| Prepare a domain | `prepare_domain`, `prepared_domain_status` | After an `extract` REQS007 with reason `domain_not_prepared`: prepare the domain with 1 to 5 example URLs, poll about every 30 seconds until ready, extract again. No credits; uses one monthly allowance slot. Not for `page_not_recognised`: that domain is already prepared |
 | Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |