@zenrows/cli 1.8.0 → 1.9.0

package/README.md

@@ -37,7 +37,7 @@ that endpoint with extraction parameters, not a separate product.
 | Command | What it does | Status (this build) |
 | --- | --- | --- |
 | `zenrows fetch` | Fetch: retrieve a protected page | **available**: `GET https://api.zenrows.com/v1/` |
-| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (falls back to Autoparse on one not prepared yet), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
+| `zenrows extract` | Extract: turn a page into structured data with `extract=auto` on any prepared domain (prepare one yourself with `zenrows extract prepare`), CSS selectors, or Markdown | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
 | `zenrows batch` | Batch: run Fetch and Extract over many URLs in one job; specs validate and estimate locally with no key | **available**: `POST https://async.api.zenrows.com/v1/jobs` |
 | `zenrows crawl` | Crawl (new): collect the URLs behind one start page, optionally with each page's HTML | **available**: `POST https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions: a real cloud browser over REST (same backend as MCP `browser_*`); escalation-only, billed by bandwidth + time | **available**: `POST https://mcp.zenrows.com/browser/sessions` |
@@ -56,7 +56,7 @@ npm install -g @zenrows/cli # puts `zenrows` on PATH (466 ms, zero deps)
 zenrows init
 zenrows fetch https://httpbin.io/html # auto-provisions a Free plan account on first use
 zenrows extract https://www.owler.com/company/meltwater # extract=auto on a prepared domain
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --css '{"title":"h1"}' # CSS selectors (any domain)
 ```
 
 > Prefer no global install? Prefix each command with `npx -y @zenrows/cli`
@@ -118,12 +118,28 @@ zenrows fetch <url> --proxy-country us --wait-for ".price"
 
 ```bash
 zenrows extract https://www.owler.com/company/meltwater                # extract=auto (prepared domain)
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # Autoparse (any domain)
+zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse   # deprecated Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}' --validate
 zenrows extract <url> --outputs emails,links                            # built-in output filters → JSON
 zenrows extract <url> --output markdown
 ```
 
+On a domain Extract hasn't prepared yet, `zenrows extract` fails with
+`EXTRACT_DOMAIN_NOT_PREPARED` (API code REQS007) and prints the command to
+prepare it. Preparing takes a few minutes and costs no credits; each new domain
+uses one slot of the monthly allowance. `EXTRACT_PAGE_NOT_RECOGNISED` (also
+REQS007) is different: the domain is prepared, but this page is not one Extract
+extracts. Try a product, article or listing page instead, retry it once in a
+few minutes if it should work, or use `--css`; preparing again does not change
+it. `zenrows extract` never falls back to Autoparse. The `fallbackAutoparse`
+adapter option is deprecated and has no effect.
+
+```bash
+zenrows extract prepare example.com --url https://example.com/p/1 --url https://example.com/list --wait
+zenrows extract prepare status example.com   # progress + monthly allowance
+zenrows extract https://example.com/p/1      # then extract again
+```
+
 ## 9. Batch
 
 Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a

package/agent-plugin/README.md

@@ -62,7 +62,7 @@ Use MCP tool names — not CLI `zenrows fetch`:
 | --- | --- |
 | `zenrows` | Decision tree across all primitives |
 | `protected-fetch` | `scrape` |
-| `extract` | `extract` (fallback: `scrape` + autoparse / css_extractor) |
+| `extract` | `extract` (fallback: `scrape` + css_extractor); on REQS007 with reason `domain_not_prepared`, `prepare_domain` then `prepared_domain_status` |
 | `batch-jobs` | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` |
 | `interact-browser` | `browser_*` (escalation only) |
 | `cost-control` | Cost multipliers + escalate-with-evidence |

package/agent-plugin/skills/cost-control/SKILL.md

@@ -19,8 +19,8 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
 - Both: **25×**
 
 `extract` `mode=auto` is currently free; billing may apply
-later. Do not promise permanent free extract pricing. Autoparse / CSS still
-follow the multipliers above when stealth flags are set.
+later. Do not promise permanent free extract pricing. CSS (and the deprecated
+Autoparse) still follow the multipliers above when stealth flags are set.
 
 ## Browser sessions (`browser_*`)
 

package/agent-plugin/skills/extract/SKILL.md

@@ -14,28 +14,67 @@ Convert protected pages into structured data. Value is **protected page access
 Use **`extract`** when you need structured JSON fields rather than a full page:
 
 ```
-extract({ url })                                    # mode=auto (default)
-extract({ url, mode: "autoparse" })                 # general Autoparse (any domain)
+extract({ url })                                    # mode=auto (default), returns `parsed` only
+extract({ url, include_html: true })                # also return the full page HTML
+extract({ url, mode: "autoparse" })                 # deprecated Autoparse (any domain)
 extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
 extract({ url, proxy_country: "US" })               # geo-targeted
 ```
 
-`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
-key, and is currently free; billing may apply later. If the
-domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
-once with Autoparse by default (`fallback_autoparse`).
+`mode=auto` (`extract=auto`) works on any prepared domain, with any key, and
+is currently free; billing may apply later. The result carries `parsed` only;
+the page HTML is most of the response size, so ask for it with
+`include_html: true` only when you need it.
 
 Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
 protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
 fixed configuration; that turns Adaptive Stealth Mode off. Check `empty` in the
 result: true means no field came back with a value.
 
+## REQS007: prepare the domain, then extract again
+
+`mode=auto` works on any prepared domain, and you can prepare one yourself.
+When Extract refuses a page, `extract` returns a tool error with code
+`REQS007` (legacy `AUTH010`) and a `reason`. It never retries with Autoparse.
+`fallback_autoparse` is deprecated and has no effect; don't send it.
+
+| `reason` | Meaning | Next step |
+| --- | --- | --- |
+| `domain_not_prepared` | The domain is not prepared yet. | `prepare_domain` (the error's `next_tool` holds its arguments), then poll, then `extract` again. |
+| `page_not_recognised` | The domain is prepared, but Extract did not recognise this page as one it can extract (a page with nothing to extract, such as a home page, or a transient failure). | Try a product, article or listing page on the same site instead. If this page should work, retry it once in a few minutes; do not retry in a loop. Otherwise use `mode=css`. Do not call `prepare_domain`: preparing again does not change this. |
+| `domain_not_enabled` | Extract is not enabled for the site. | Use `mode=css`, or contact Zenrows support to have the site enabled. Preparing does not change this. |
+
+Preparing a domain:
+
+```
+extract({ url })                                          # 1. REQS007, reason domain_not_prepared
+prepare_domain({ domain: "example.com",
+  example_urls: ["https://example.com/p/1", "https://example.com/list"] })
+                                                          # 2. returns at once, status "preparing"
+prepared_domain_status({ domain: "example.com" })         # 3. about every 30 seconds until "ready"
+extract({ url })                                          # 4. extract again
+```
+
+- `example_urls`: 1 to 5 real pages you want to extract, one per page kind
+  (for example a product page and a listing page).
+- The domain is the host: `shop.example.com` and `example.com` are prepared
+  separately, and a leading `www.` is dropped.
+- Preparing takes a few minutes and costs no credits. Each new domain uses one
+  slot of the account's monthly allowance; `prepared_domain_status` returns it
+  (`used` / `limit` / `remaining`) and a `next` field with what to do now.
+- `out_of_allowance`: no slots left this month; use `mode=css`.
+  `overloaded` / `disabled`: retry in a few minutes; do not loop.
+  A `failed` status carries `failure.code` and `failure.slot_consumed`; follow `next`.
+- A domain that is already prepared answers `ready` at once
+  (`already_prepared: true`) and uses no slot: call `extract`. If it then
+  returns `page_not_recognised`, follow that row of the table, not `prepare_domain`.
+
 ## Fallback: `scrape` extraction params
 
 If `extract` is unavailable, similar outcomes are available on **`scrape`**:
 
 ```
-scrape({ url, autoparse: true })
+scrape({ url, autoparse: true })                    # deprecated
 scrape({ url, css_extractor: '{"title":"h1","price":".price"}' })
 scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ```
@@ -46,7 +85,8 @@ scrape({ url, outputs: "emails,links" })            # or "*" for all built-ins
 ## When to use which
 
 - **`mode=auto`**: site-tailored fields on any prepared domain.
-- **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
+- **`mode=autoparse`** (deprecated): any website, only when asked for. Do not
+  build new work on it.
 - **`mode=css`**: known fields/selectors; deterministic.
 - **Markdown via `scrape`**: feed clean content to an LLM yourself.
 
@@ -57,7 +97,7 @@ Stealth flags (`js_render`, `premium_proxy`) still apply the usual multipliers
 ## Rules
 
 - Validate on a single page before `batch_create` ([[batch-jobs]]).
-- If Autoparse misses fields, switch to `css_extractor`, or add `js_render`
+- If a field is missing, switch to `css_extractor`, or add `js_render`
   for JS-heavy pages.
 - Prefer MCP `extract` / `scrape` over CLI `zenrows extract` when the MCP
   server is connected.

package/agent-plugin/skills/protected-fetch/SKILL.md

@@ -34,7 +34,7 @@ scrape({ url, js_render: true, wait_for: ".price" })
 - Prefer `response_type: "markdown"` for LLM consumption; use `html` only when
   you need raw markup.
 - For **structured** fields, prefer the `extract` tool (or `scrape` with
-  `autoparse` / `css_extractor` / `outputs`) — see [[extract]].
+  `css_extractor` / `outputs`) — see [[extract]].
 - On failure, escalate with evidence before `browser_*` ([[interact-browser]]).
 - Cost model: [[cost-control]].
 

package/agent-plugin/skills/zenrows/SKILL.md

@@ -37,7 +37,7 @@ If the user has a known URL and wants page content:
 
 If the user has a known URL and wants structured data:
   → Use Extract.                    (MCP tool: extract)
-                                     Fallback: scrape with autoparse / css_extractor / outputs
+                                     Fallback: scrape with css_extractor / outputs
 
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with scrape/extract first,
@@ -67,7 +67,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
 | Primitive | MCP tools | Notes |
 | --- | --- | --- |
 | Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
-| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
+| Extract | `extract` | Structured JSON (`extract=auto` on any prepared domain, css_extractor; autoparse deprecated). `extract=auto` is currently free; billing may apply later |
+| Prepare a domain | `prepare_domain`, `prepared_domain_status` | After an `extract` REQS007 with reason `domain_not_prepared`: prepare the domain with 1 to 5 example URLs, poll about every 30 seconds until ready, extract again. No credits; uses one monthly allowance slot. Not for `page_not_recognised`: that domain is already prepared |
 | Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
 | Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
 

package/dist/src/adapters/extract.js

@@ -1,17 +1,26 @@
 import { ToolkitError } from "../core/errors.js";
+import { log } from "../core/logger.js";
 import { runFetch } from "./protected-fetch.js";
-/** True when a parsed extraction result carries no usable structured data. */
-function isEmptyData(data) {
+/**
+ * True when a parsed extraction result carries no usable structured data: null,
+ * a blank string, or an array / object whose every value is itself empty. So
+ * `{"title": null, "items": [], "author": {"name": ""}}` is empty. Numbers and
+ * booleans are values, so `0` and `false` count as data.
+ */
+export function isEmptyData(data) {
     if (data === null || data === undefined)
         return true;
-    if (Array.isArray(data))
-        return data.length === 0;
-    if (typeof data === "object")
-        return Object.keys(data).length === 0;
     if (typeof data === "string")
         return data.trim() === "";
+    if (Array.isArray(data))
+        return data.every(isEmptyData);
+    if (typeof data === "object")
+        return Object.values(data).every(isEmptyData);
     return false;
 }
+/** The warning a caller that still passes `fallbackAutoparse: true` gets. */
+export const FALLBACK_AUTOPARSE_DEPRECATION = "fallbackAutoparse is deprecated and has no effect: Extract no longer falls back to Autoparse. " +
+    "On REQS007, prepare the domain with `zenrows extract prepare <domain> --url <page>` instead.";
 export async function runExtract(opts, config, policy, apiKey) {
     const method = opts.method ?? (opts.outputs ? "outputs" : opts.cssExtractor ? "css" : "extract");
     if (method === "css" && !opts.cssExtractor) {
@@ -30,18 +39,8 @@ export async function runExtract(opts, config, policy, apiKey) {
             next_action: `Pass one or more filters, e.g. --outputs emails,links (or --outputs '*').`,
         });
     }
-    if (method === "extract" && opts.fallbackAutoparse !== false) {
-        try {
-            return await runExtractOnce({ ...opts, method: "extract" }, config, policy, apiKey);
-        }
-        catch (err) {
-            if (err instanceof ToolkitError && err.code === "EXTRACT_DOMAIN_NOT_ENABLED") {
-                const outcome = await runExtractOnce({ ...opts, method: "autoparse" }, config, policy, apiKey);
-                return { ...outcome, fellBackToAutoparse: true };
-            }
-            throw err;
-        }
-    }
+    if (opts.fallbackAutoparse === true)
+        log.warn(FALLBACK_AUTOPARSE_DEPRECATION);
     return runExtractOnce({ ...opts, method }, config, policy, apiKey);
 }
 async function runExtractOnce(opts, config, policy, apiKey) {
@@ -101,7 +100,7 @@ function parseExtractBody(method, body, opts) {
         }
     }
     const empty = isEmptyData(data);
-    // ACT-1514: an empty result must not read as a silent success. Under
+    // An empty result must not read as a silent success. Under
     // --validate it's an error; otherwise the command surfaces a warning.
     if (opts.validate && empty) {
         throw new ToolkitError({

package/dist/src/cli/commands/extract-prepare.js

@@ -0,0 +1,168 @@
+/**
+ * `zenrows extract prepare <domain> --url <u> [--url <u>...] [--wait]` and
+ * `zenrows extract prepare status <domain>`.
+ *
+ * The way out of REQS007: prepare the domain so `extract=auto` works on it.
+ * Submitting returns at once; `--wait` polls until the preparation is ready or
+ * failed (a few minutes). Costs no credits; each new domain uses one slot of the
+ * account's monthly allowance.
+ */
+import { ensureApiKey } from "../../core/ensure-key.js";
+import { loadConfig } from "../../core/config.js";
+import { loadPolicy } from "../../core/policy.js";
+import { log } from "../../core/logger.js";
+import { ToolkitError } from "../../core/errors.js";
+import { DEFAULT_WAIT_TIMEOUT_MS, MAX_EXAMPLE_URLS, failureNextStep, getAllowance, getPreparedDomain, normalizeDomain, prepareDomain, statusNextStep, waitForPreparedDomain, } from "../../core/prepared-domains-api.js";
+import { parse, asString } from "../command.js";
+import { printError } from "../output.js";
+export const PREPARE_USAGE = "zenrows extract prepare <domain> --url <page-url> [--url <page-url>...] [--wait [--timeout <seconds>]]\n" +
+    "       zenrows extract prepare status <domain>";
+export const PREPARE_HELP = [
+    "Prepare a domain so `zenrows extract <url>` (extract=auto) works on it. Use it when an",
+    "extract fails with EXTRACT_DOMAIN_NOT_PREPARED (REQS007: the domain is not prepared yet).",
+    "Not for EXTRACT_PAGE_NOT_RECOGNISED: that domain is already prepared, and preparing it",
+    "again does not change that answer.",
+    "",
+    `  --url <page-url>       1 to ${MAX_EXAMPLE_URLS} real pages you want to extract, one per page kind (repeat --url)`,
+    "  --wait                 poll every 30 seconds until the domain is ready or failed; off by default",
+    `  --timeout <seconds>    give up waiting after this long (default ${DEFAULT_WAIT_TIMEOUT_MS / 1000})`,
+    "  status <domain>        show progress and the account's monthly allowance",
+    "  --json                 structured result",
+    "",
+    "Preparing takes a few minutes and costs no credits. Each new domain uses one slot of the",
+    "account's monthly allowance; a domain that is already prepared answers ready at once and",
+    "uses none. The domain is the host (shop.example.com and example.com are separate); a",
+    "leading www. is dropped.",
+].join("\n");
+export async function runPrepare(argv, ctx, deps = {}) {
+    const isStatus = argv[0] === "status";
+    const { values, positionals } = parse(isStatus ? argv.slice(1) : argv, {
+        url: { type: "string", multiple: true },
+        wait: { type: "boolean" },
+        timeout: { type: "string" },
+        json: { type: "boolean" },
+        help: { type: "boolean" },
+    });
+    const json = ctx.json || values.json === true;
+    try {
+        const raw = positionals[0];
+        if (!raw) {
+            throw new ToolkitError({
+                code: "INVALID_USAGE",
+                message: "Missing domain.",
+                likely_cause: "No <domain> was given.",
+                next_action: `Usage: ${PREPARE_USAGE}`,
+                suggested_commands: ["zenrows extract prepare example.com --url https://example.com/products/1"],
+            });
+        }
+        const domain = normalizeDomain(raw);
+        const urls = isStatus ? [] : validateUrls(values.url, domain);
+        const timeoutMs = parseTimeout(asString(values.timeout));
+        const policy = loadPolicy();
+        const apiKey = deps.apiKey ??
+            // Preparing never creates a Free plan account: it needs a key the user already has.
+            (await ensureApiKey({ ...policy, auto_signup: false }));
+        const opts = { apiKey, apiBase: loadConfig().apiBase, fetchImpl: deps.fetchImpl };
+        if (isStatus) {
+            const [record, allowance] = await Promise.all([getPreparedDomain(domain, opts), getAllowance(opts).catch(() => undefined)]);
+            report(record, allowance, json);
+            return 0;
+        }
+        log.step(`Prepare ${domain} for Extract (${urls.length} example URL${urls.length === 1 ? "" : "s"})…`);
+        let record = await prepareDomain(domain, urls, opts);
+        if (record.already_in_progress)
+            log.info(`${domain} was already being prepared; following that preparation.`);
+        // A submit that answers ready at once found the domain already prepared.
+        const alreadyReady = record.status === "ready" && !record.already_in_progress;
+        if (values.wait === true && record.status !== "ready" && record.status !== "failed") {
+            record = await waitForPreparedDomain(domain, {
+                ...opts,
+                timeoutMs,
+                intervalMs: deps.intervalMs,
+                sleepImpl: deps.sleepImpl,
+                now: deps.now,
+                onPoll: (r) => log.dim(`  ${r.domain}: ${r.status} (${r.state ?? r.status})`),
+            });
+        }
+        if (record.status === "failed") {
+            throw new ToolkitError({
+                code: "EXTRACT_PREPARE_FAILED",
+                message: `Preparing ${domain} failed${record.failure?.code ? ` (${record.failure.code})` : ""}.`,
+                likely_cause: record.failure?.detail ?? "The preparation did not finish.",
+                next_action: failureNextStep(domain, record.failure),
+                server_code: record.failure?.code,
+            });
+        }
+        report(record, undefined, json, { exampleUrl: urls[0], alreadyReady });
+        return 0;
+    }
+    catch (err) {
+        printError(err, json);
+        return 1;
+    }
+}
+function validateUrls(raw, domain) {
+    const urls = (Array.isArray(raw) ? raw : raw === undefined ? [] : [raw]).map(String);
+    if (urls.length < 1 || urls.length > MAX_EXAMPLE_URLS) {
+        throw new ToolkitError({
+            code: "INVALID_USAGE",
+            message: `Pass 1 to ${MAX_EXAMPLE_URLS} example page URLs with --url (got ${urls.length}).`,
+            likely_cause: "Preparation learns the domain from real pages you want to extract.",
+            next_action: "Add one --url per page kind, e.g. a product page and a listing page.",
+            suggested_commands: [`zenrows extract prepare ${domain} --url https://${domain}/<page>`],
+        });
+    }
+    for (const u of urls) {
+        let parsed;
+        try {
+            parsed = new URL(u);
+        }
+        catch {
+            // reported below
+        }
+        if (!parsed || (parsed.protocol !== "http:" && parsed.protocol !== "https:")) {
+            throw new ToolkitError({
+                code: "INVALID_USAGE",
+                message: `--url ${u} is not a full http(s) URL.`,
+                likely_cause: "Each --url must be a page URL including its scheme.",
+                next_action: `Use full URLs, e.g. --url https://${domain}/<page>.`,
+            });
+        }
+    }
+    return urls;
+}
+function parseTimeout(v) {
+    if (v === undefined)
+        return DEFAULT_WAIT_TIMEOUT_MS;
+    const n = Number(v);
+    if (!Number.isFinite(n) || n <= 0) {
+        throw new ToolkitError({
+            code: "INVALID_USAGE",
+            message: `--timeout must be a positive number of seconds (got ${v}).`,
+            likely_cause: "The value is not a number.",
+            next_action: "Pass seconds, e.g. --timeout 600.",
+        });
+    }
+    return n * 1000;
+}
+function report(record, allowance, json, opts = {}) {
+    const next = statusNextStep(record, opts);
+    if (json) {
+        log.out(JSON.stringify({ ok: true, ...record, ...(opts.alreadyReady ? { already_prepared: true } : {}), ...(allowance ? { allowance } : {}), next }, null, 2));
+        return;
+    }
+    const line = `${record.domain}: ${record.status}${record.state && record.state !== record.status ? ` (${record.state})` : ""}`;
+    if (record.status === "ready")
+        log.success(line);
+    else if (record.status === "failed")
+        log.warn(line);
+    else
+        log.info(line);
+    if (record.failure)
+        log.dim(`  failure: ${record.failure.code}${record.failure.detail ? ` — ${record.failure.detail}` : ""}`);
+    for (const role of record.roles ?? [])
+        log.dim(`  ${role.role}: ${role.status}${role.url ? ` ${role.url}` : ""}`);
+    if (allowance)
+        log.dim(`  allowance: ${allowance.used} of ${allowance.limit} used, ${allowance.remaining} left${allowance.month ? ` (${allowance.month})` : ""}`);
+    log.out(next);
+}

package/dist/src/cli/commands/extract.js

@@ -1,9 +1,12 @@
 /**
  * `zenrows extract <url> [flags]` → Extract (structured extraction on `/v1/`).
  *
- * Default: `extract=auto` (site-tailored Extract). If the domain is
- * not enabled or prepared (AUTH010 / REQS007), falls back once to Autoparse. Opt into Autoparse
- * directly with `--autoparse`. Other methods: `--css`, `--outputs`, `--output`.
+ * Default: `extract=auto` (site-tailored Extract). On a REQS007 / AUTH010 the
+ * error names the next step for its reason (prepare the domain, or try another
+ * page); nothing is retried. `--autoparse` runs the deprecated Autoparse
+ * explicitly. Other
+ * methods: `--css`, `--outputs`, `--output`. `zenrows extract prepare` prepares a
+ * domain (see extract-prepare.ts).
  */
 import { ensureApiKey } from "../../core/ensure-key.js";
 import { maybeNudgeClaim } from "../../core/nudge.js";
@@ -18,14 +21,17 @@ import { formatRequestCost } from "../../core/http.js";
 import { parse, asString, asNumber } from "../command.js";
 import { printError, writeOut } from "../output.js";
 import { refreshClaimAdvice } from "../../core/claim-link.js";
+import { PREPARE_HELP, PREPARE_USAGE, runPrepare } from "./extract-prepare.js";
 export const extract = {
     name: "extract",
     summary: "Turn a protected page into structured data (Extract / Autoparse / CSS / Markdown).",
-    usage: "zenrows extract <url> [--autoparse | --css <json> | --outputs <filters> | --output md|text] [flags]",
+    usage: "zenrows extract <url> [--autoparse | --css <json> | --outputs <filters> | --output md|text] [flags]\n       " +
+        PREPARE_USAGE,
     help: [
         "Methods:",
-        "  (default)              extract=auto: site-tailored Extract; falls back to Autoparse if the domain is not enabled or prepared",
-        "  --autoparse            general-purpose Autoparse (any domain, no Extract fallback)",
+        "  (default)              extract=auto: site-tailored Extract on any prepared domain",
+        "                         (EXTRACT_DOMAIN_NOT_PREPARED = not prepared yet: run zenrows extract prepare, below)",
+        "  --autoparse            deprecated general-purpose Autoparse (any domain)",
         "  --css <json>           CSS selector map, e.g. '{\"title\":\"h1\",\"price\":\".price\"}'",
         "  --outputs <list>       built-in output filters → JSON: emails, phone_numbers, headings,",
         "                         images, audios, videos, links, menus, hashtags, metadata, tables,",
@@ -40,8 +46,13 @@ export const extract = {
         "",
         "Cost (credits per request): 1x normal · 5x --js-render · 10x --premium-proxy · 25x both.",
         "The exact charge is reported after every request (costCredits / X-Request-Credits).",
+        "",
+        "Prepare a domain:",
+        PREPARE_HELP.split("\n").map((l) => (l ? "  " + l : l)).join("\n"),
     ].join("\n"),
     async run(argv, ctx) {
+        if (argv[0] === "prepare")
+            return runPrepare(argv.slice(1), ctx);
         const { values, positionals } = parse(argv, {
             autoparse: { type: "boolean" },
             css: { type: "string" },
@@ -114,14 +125,14 @@ export const extract = {
         const runId = newRunId();
         const startedAt = new Date().toISOString();
         log.step(`Extract ${url} (method=${opts.method ?? "extract"})…`);
+        if (opts.method === "autoparse") {
+            log.warn("Autoparse is deprecated. Prefer the default extract=auto on a prepared domain, or --css.");
+        }
         try {
             const outcome = await runExtract(opts, config, policy, apiKey);
             const finishedAt = new Date().toISOString();
             const safeParams = { ...outcome.params };
             delete safeParams.apikey;
-            if (outcome.fellBackToAutoparse) {
-                log.info("Extract not available for this domain yet — fell back to Autoparse.");
-            }
             const runDir = writeRun({
                 runId,
                 command: "zenrows extract",
@@ -133,7 +144,6 @@ export const extract = {
                 request: {
                     ...safeParams,
                     method: outcome.method,
-                    fellBackToAutoparse: outcome.fellBackToAutoparse ?? false,
                 },
                 result: { httpStatus: outcome.result.status, bytes: outcome.result.body.length, parsed: outcome.data !== undefined },
                 costUsd: outcome.result.costUsd,
@@ -145,8 +155,7 @@ export const extract = {
                 writeOut(values.out, payload);
                 log.success(`Wrote output → ${values.out}`);
             }
-            const via = outcome.fellBackToAutoparse ? `${outcome.method} (fallback)` : outcome.method;
-            log.success(`Extracted via ${via} · ${outcome.result.body.length} bytes · ${formatRequestCost(outcome.result.costUsd, outcome.result.costCredits)} · run ${runId}`);
+            log.success(`Extracted via ${outcome.method} · ${outcome.result.body.length} bytes · ${formatRequestCost(outcome.result.costUsd, outcome.result.costCredits)} · run ${runId}`);
             if (runDir)
                 log.dim(`  artifact: ${runDir}`);
             if (outcome.noDataExtracted) {
@@ -160,7 +169,6 @@ export const extract = {
                     ok: true,
                     runId,
                     method: outcome.method,
-                    fellBackToAutoparse: outcome.fellBackToAutoparse ?? false,
                     empty: outcome.empty ?? false,
                     noDataExtracted: outcome.noDataExtracted ?? false,
                     httpStatus: outcome.result.status,

package/dist/src/core/config.js

@@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0";
  * entrypoint) so core modules — telemetry, signup provenance — can read it
  * without an import cycle. `VERSION` in `cli/index.ts` re-exports this.
  */
-export const CLI_VERSION = "1.8.0"; // x-release-please-version
+export const CLI_VERSION = "1.9.0"; // x-release-please-version
 /** The User-Agent on every request to a Zenrows API. The gateway reads the version from it. */
 export const CLI_USER_AGENT = `zenrows-cli/${CLI_VERSION}`;
 /** Env var to override the Fetch and Extract API base (local/staging testing). */

package/dist/src/core/http.js

@@ -12,6 +12,7 @@ import { ENV_KEY, resolveApiKey } from "./auth.js";
 import { registerSecret } from "./logger.js";
 import { CLI_USER_AGENT } from "./config.js";
 import { agentClientHeader } from "./agent-client.js";
+import { normalizeDomain, PAGE_NOT_RECOGNISED_STEPS, prepareCommand } from "./prepared-domains-api.js";
 /**
  * Client-side timeout for a Fetch and Extract request, deliberately ABOVE the
  * gateway's own 90s request budget.
@@ -99,17 +100,10 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
         concurrencyRemaining: numOrNull(res.headers.get("concurrency-remaining")),
         redactedUrl: redacted,
     };
-    // REQS007 on extract=auto: the domain hasn't been prepared for Extract yet.
-    // Same recovery as AUTH010 (the extract adapter falls back to autoparse), so it
-    // maps to the same code. Checked before the 403 auth branch below.
+    // REQS007 on extract=auto: Extract has not prepared the domain yet. Reported,
+    // never retried: the fix is to prepare the domain, which the error names.
     if (res.status === 403 && params.extract !== undefined && zrErrorCode(body) === "REQS007") {
-        throw new ToolkitError({
-            code: "EXTRACT_DOMAIN_NOT_ENABLED",
-            message: "Extract has not been prepared for this domain yet.",
-            likely_cause: zrErrorDetail(body) ?? "This domain is not prepared for Extract.",
-            next_action: "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
-            suggested_commands: [`zenrows extract ${params.url} --autoparse`],
-        });
+        throw extractDomainNotPrepared(params.url, body);
     }
     // 422 no_data_extracted on an Extract request: the page was processed and had
     // nothing to extract. It is an answer, not a failure to reach the target, so
@@ -163,17 +157,10 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
         }
     }
     if (res.status === 402 && isZenrowsErrorEnvelope(body)) {
-        // AUTH010 on extract=auto means the target domain is not enabled for
-        // Extract, distinct from credit exhaustion (AUTH004). Let the extract
-        // adapter fall back to autoparse.
+        // AUTH010 on extract=auto is the legacy code for REQS007: the domain is
+        // not prepared for Extract. Distinct from credit exhaustion (AUTH004).
         if (params.extract !== undefined && zrErrorCode(body) === "AUTH010") {
-            throw new ToolkitError({
-                code: "EXTRACT_DOMAIN_NOT_ENABLED",
-                message: "Extract is not enabled for this domain yet.",
-                likely_cause: zrErrorDetail(body) ?? "This domain is not enabled for Extract.",
-                next_action: "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
-                suggested_commands: [`zenrows extract ${params.url} --autoparse`],
-            });
+            throw extractDomainNotPrepared(params.url, body);
         }
         // AUTH014: this key hit one of its credit caps; the account still has credits.
         if (isKeyCapReached(zrErrorCode(body) ?? undefined)) {
@@ -261,6 +248,77 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
     }
     return result;
 }
+const REFUSAL_REASONS = ["domain_not_prepared", "page_not_recognised", "domain_not_enabled"];
+/**
+ * Which REQS007 this is. Only domain_not_prepared is fixed by preparing the
+ * domain: on page_not_recognised the domain is already prepared, and
+ * resubmitting it returns the existing ready record, so pointing there would
+ * loop.
+ *
+ * The gateway says it in the problem's `reason` field, read first. The sentence
+ * match below is only the fallback for a gateway deployed before that field
+ * existed, and reads the titles that gateway sends; remove it once every gateway
+ * sends `reason`.
+ */
+export function refusalReason(body) {
+    let j;
+    try {
+        j = JSON.parse(body);
+    }
+    catch {
+        return "domain_not_prepared";
+    }
+    if (typeof j.reason === "string" && REFUSAL_REASONS.includes(j.reason))
+        return j.reason;
+    const title = typeof j.title === "string" ? j.title : "";
+    const detail = typeof j.detail === "string" ? j.detail : "";
+    if (/did not recogni[sz]e this page/i.test(title) || /^Extract serves /.test(detail))
+        return "page_not_recognised";
+    if (/^Extract is not enabled/i.test(title))
+        return "domain_not_enabled";
+    return "domain_not_prepared";
+}
+/**
+ * REQS007 (or the legacy AUTH010) on extract=auto. Each reason names its own
+ * next step, because agents copy it: the exact prepare command for a domain not
+ * prepared yet, the gateway's advice in the gateway's order for a page it did not
+ * recognise. None offers Autoparse.
+ */
+export function extractDomainNotPrepared(url, body) {
+    const domain = normalizeDomain(url);
+    const detail = zrErrorDetail(body);
+    const css = `zenrows extract ${url} --css '{"title":"h1"}'`;
+    const reason = refusalReason(body);
+    if (reason === "page_not_recognised") {
+        return new ToolkitError({
+            code: "EXTRACT_PAGE_NOT_RECOGNISED",
+            message: `Extract serves ${domain} but did not recognise this page as one it can extract (it may be a page with nothing to extract, such as a home page). ${PAGE_NOT_RECOGNISED_STEPS} The request was not charged.`,
+            likely_cause: detail ?? `Extract serves ${domain} but did not recognise this page.`,
+            next_action: PAGE_NOT_RECOGNISED_STEPS,
+            suggested_commands: [css],
+            server_code: "REQS007",
+        });
+    }
+    if (reason === "domain_not_enabled") {
+        return new ToolkitError({
+            code: "EXTRACT_DOMAIN_NOT_ENABLED",
+            message: `Extract is not enabled for ${domain}, and preparing it does not change this.`,
+            likely_cause: detail ?? `Extract is not enabled for ${domain}.`,
+            next_action: "Extract this page with --css and explicit selectors, or contact Zenrows support to have the site enabled.",
+            suggested_commands: [css],
+            server_code: "REQS007",
+        });
+    }
+    const prepare = prepareCommand(url);
+    return new ToolkitError({
+        code: "EXTRACT_DOMAIN_NOT_PREPARED",
+        message: `Extract hasn't prepared ${domain} yet. Prepare it with \`${prepare}\` (a few example URLs, a few minutes, no credits), then retry; meanwhile use --css.`,
+        likely_cause: detail ?? `Extract has not prepared ${domain} yet.`,
+        next_action: `Run \`${prepare} --wait\` (up to 5 --url pages, one per page kind you want), then retry \`zenrows extract ${url}\`. Without --wait, check progress about every 30 seconds with \`zenrows extract prepare status ${domain}\`. Meanwhile extract with --css and explicit selectors.`,
+        suggested_commands: [`${prepare} --wait`, `zenrows extract prepare status ${domain}`, css],
+        server_code: "REQS007",
+    });
+}
 /**
  * The CLI gave up waiting before the API answered. This is NOT unreachability:
  * the connection was established and the gateway was still working, so telling

package/dist/src/core/prepared-domains-api.js

@@ -0,0 +1,289 @@
+/**
+ * Client for Extract's prepared-domains API.
+ *
+ * `extract=auto` answers REQS007 on a domain Extract has not prepared yet. Preparing
+ * it is a call any key can make: `POST {apiBase}/extract/prepared-domains` with the
+ * domain and 1 to 5 example URLs answers 202 at once; the preparation itself takes a
+ * few minutes and costs no credits. `GET .../{domain}` reports progress and
+ * `GET .../allowance` the account's monthly allowance of new domains. Auth is the
+ * `X-API-Key` header; errors are problem+json and we branch on `status` + `code`.
+ * `fetchImpl` / `sleepImpl` / `now` are injectable for tests.
+ */
+import { ToolkitError } from "./errors.js";
+import { registerSecret } from "./logger.js";
+import { CLI_USER_AGENT, loadConfig } from "./config.js";
+import { agentClientHeader } from "./agent-client.js";
+export const MAX_EXAMPLE_URLS = 5;
+/** How often `--wait` polls, and for how long by default. */
+export const DEFAULT_POLL_INTERVAL_MS = 30_000;
+export const DEFAULT_WAIT_TIMEOUT_MS = 15 * 60_000;
+/**
+ * The domain as the API keys it: host only, no scheme, path or leading `www.`.
+ * Accepts a URL.
+ *
+ * Deliberately not the registrable domain: Extract prepares and serves each host
+ * on its own. A prepared `es.wikipedia.org` does not cover `pt.wikipedia.org`
+ * (REQS007 names `pt.wikipedia.org`), and `wikipedia.org` has no record of its
+ * own. The API itself folds `www.` into the bare host, so dropping it here
+ * matches what it stores.
+ */
+export function normalizeDomain(input) {
+    const trimmed = input.trim();
+    let host = trimmed;
+    try {
+        host = new URL(/^[a-z][a-z0-9+.-]*:\/\//i.test(trimmed) ? trimmed : `https://${trimmed}`).hostname;
+    }
+    catch {
+        // Left as given: the API answers invalid_argument with the reason.
+    }
+    return host.toLowerCase().replace(/^www\./, "");
+}
+/** The exact command an agent runs to prepare the domain of `url`. */
+export function prepareCommand(url) {
+    return `zenrows extract prepare ${normalizeDomain(url)} --url ${url}`;
+}
+function base(opts) {
+    return (opts.apiBase ?? loadConfig().apiBase).replace(/\/+$/, "");
+}
+async function request(method, path, domain, opts) {
+    registerSecret(opts.apiKey);
+    const headers = {
+        "X-API-Key": opts.apiKey,
+        Accept: "application/json",
+        "User-Agent": CLI_USER_AGENT,
+        ...agentClientHeader(),
+    };
+    if (opts.body !== undefined)
+        headers["Content-Type"] = "application/json";
+    let res;
+    try {
+        res = await (opts.fetchImpl ?? fetch)(`${base(opts)}/extract/prepared-domains${path}`, {
+            method,
+            headers,
+            body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined,
+        });
+    }
+    catch (err) {
+        throw new ToolkitError({
+            code: "BACKEND_UNAVAILABLE",
+            message: "Could not reach the Zenrows API.",
+            likely_cause: err instanceof Error ? err.message : String(err),
+            next_action: "Check connectivity and retry. Verify the api base in `zenrows config show`.",
+            suggested_commands: ["zenrows status", "zenrows config show"],
+        });
+    }
+    const text = await res.text();
+    if (res.status < 200 || res.status >= 300)
+        throw prepareError(res.status, text, domain);
+    try {
+        return JSON.parse(text);
+    }
+    catch {
+        throw new ToolkitError({
+            code: "EXTRACT_PREPARE_FAILED",
+            message: "The prepared-domains API returned a non-JSON answer.",
+            likely_cause: text.slice(0, 200),
+            next_action: "Retry in a minute.",
+        });
+    }
+}
+/** Map a non-2xx prepared-domains answer to an error that names the next step. */
+export function prepareError(status, body, domain) {
+    let p = {};
+    try {
+        p = JSON.parse(body) ?? {};
+    }
+    catch {
+        // not problem+json
+    }
+    const code = typeof p.code === "string" ? p.code.toLowerCase() : undefined;
+    const detail = typeof p.detail === "string" ? p.detail : body.slice(0, 200) || `HTTP ${status}`;
+    const cause = code ? `(${code}) ${detail}` : detail;
+    if (code === "invalid_argument" || status === 400) {
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_INVALID",
+            message: `The request to prepare ${domain} was rejected.`,
+            likely_cause: p.reason ? `${cause} [reason: ${String(p.reason)}]` : cause,
+            next_action: `Pass the domain without scheme or path, and 1 to ${MAX_EXAMPLE_URLS} full page URLs from that site with --url.`,
+            suggested_commands: [`zenrows extract prepare ${domain} --url https://${domain}/<page>`],
+            server_code: code,
+        });
+    }
+    if (code === "not_found" || status === 404) {
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_NOT_FOUND",
+            message: `This account has not submitted ${domain} for preparation.`,
+            likely_cause: cause,
+            next_action: `Prepare it first: zenrows extract prepare ${domain} --url <page url> (1 to ${MAX_EXAMPLE_URLS} --url).`,
+            suggested_commands: [`zenrows extract prepare ${domain} --url https://${domain}/<page>`],
+            server_code: code,
+        });
+    }
+    if (code === "in_progress") {
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_IN_PROGRESS",
+            message: `${domain} is already being prepared.`,
+            likely_cause: cause,
+            next_action: "Do not resubmit. Retry the extract in a few minutes.",
+            suggested_commands: [`zenrows extract prepare status ${domain}`],
+            server_code: code,
+        });
+    }
+    if (code === "idempotency_conflict") {
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_INVALID",
+            message: "This idempotency key was already used with a different request.",
+            likely_cause: cause,
+            next_action: "Retry without the key, or with a new one.",
+            server_code: code,
+        });
+    }
+    if (code === "out_of_allowance" || status === 429) {
+        const used = p.used !== undefined && p.limit !== undefined ? ` (${String(p.used)} of ${String(p.limit)} used)` : "";
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_OUT_OF_ALLOWANCE",
+            message: `This account has used its monthly allowance of new domains to prepare${used}.`,
+            likely_cause: cause,
+            next_action: "Do not retry this month. Meanwhile extract this domain with --css and explicit selectors.",
+            suggested_commands: [`zenrows extract <url> --css '{"title":"h1"}'`],
+            server_code: code,
+        });
+    }
+    if (code === "overloaded" || code === "disabled" || status === 503) {
+        return new ToolkitError({
+            code: "EXTRACT_PREPARE_UNAVAILABLE",
+            message: "Domain preparation is unavailable right now.",
+            likely_cause: cause,
+            next_action: "Retry in a few minutes; do not loop. Meanwhile extract with --css and explicit selectors.",
+            server_code: code,
+        });
+    }
+    if (status === 401 || status === 403) {
+        return new ToolkitError({
+            code: "AUTH_INVALID",
+            message: "Zenrows rejected the API key.",
+            likely_cause: cause,
+            next_action: "Re-check your key and log in again.",
+            suggested_commands: ["zenrows login --api-key <your-key>"],
+            server_code: code,
+        });
+    }
+    return new ToolkitError({
+        code: "EXTRACT_PREPARE_FAILED",
+        message: `Preparing ${domain} failed with HTTP ${status}.`,
+        likely_cause: cause,
+        next_action: "Retry in a few minutes.",
+        server_code: code,
+    });
+}
+export function submitPreparedDomain(domain, exampleUrls, opts) {
+    return request("POST", "", domain, { ...opts, body: { domain, example_urls: exampleUrls } });
+}
+export function getPreparedDomain(domain, opts) {
+    return request("GET", `/${encodeURIComponent(domain)}`, domain, opts);
+}
+export function getAllowance(opts) {
+    return request("GET", "/allowance", "", opts);
+}
+/**
+ * Submit a domain. A 409 in_progress (already being prepared) answers with the
+ * current status when this account can read it, so the caller just follows it.
+ */
+export async function prepareDomain(domain, exampleUrls, opts) {
+    try {
+        return await submitPreparedDomain(domain, exampleUrls, opts);
+    }
+    catch (err) {
+        if (!(err instanceof ToolkitError) || err.code !== "EXTRACT_PREPARE_IN_PROGRESS")
+            throw err;
+        try {
+            return { ...(await getPreparedDomain(domain, opts)), already_in_progress: true };
+        }
+        catch {
+            throw err;
+        }
+    }
+}
+export function isTerminal(record) {
+    return record.status === "ready" || record.status === "failed";
+}
+/** Poll until ready or failed. Throws EXTRACT_PREPARE_TIMEOUT when the wait runs out. */
+export async function waitForPreparedDomain(domain, opts) {
+    const sleep = opts.sleepImpl ?? ((ms) => new Promise((r) => setTimeout(r, ms)));
+    const now = opts.now ?? Date.now;
+    const interval = opts.intervalMs ?? DEFAULT_POLL_INTERVAL_MS;
+    const deadline = now() + (opts.timeoutMs ?? DEFAULT_WAIT_TIMEOUT_MS);
+    for (;;) {
+        const record = await getPreparedDomain(domain, opts);
+        opts.onPoll?.(record);
+        if (isTerminal(record))
+            return record;
+        if (now() + interval > deadline) {
+            throw new ToolkitError({
+                code: "EXTRACT_PREPARE_TIMEOUT",
+                message: `${domain} is still being prepared (state: ${record.state ?? record.status}).`,
+                likely_cause: "Preparation usually takes a few minutes; this one is taking longer than the wait.",
+                next_action: "Check again later; it keeps running on the server.",
+                suggested_commands: [`zenrows extract prepare status ${domain}`],
+            });
+        }
+        await sleep(interval);
+    }
+}
+/** Whether a failed preparation used an allowance slot, from the record's failure.slot_consumed. */
+function slotSentence(slotConsumed) {
+    if (slotConsumed === true)
+        return " This attempt used an allowance slot.";
+    if (slotConsumed === false)
+        return " This attempt did not use an allowance slot.";
+    return "";
+}
+/** What to do after a failed preparation, by failure.code (docs.zenrows.com extract/endpoints). */
+export function failureNextStep(domain, failure) {
+    let step;
+    switch (failure?.code) {
+        case "interrupted":
+        case "preparation_exhausted":
+            step = `Run the same zenrows extract prepare ${domain} command again.`;
+            break;
+        case "discovery_failed":
+            step = `Run zenrows extract prepare ${domain} again with 1 to ${MAX_EXAMPLE_URLS} --url page URLs.`;
+            break;
+        case "unpreparable":
+            step = `Run zenrows extract prepare ${domain} again with different --url pages from the same site; the same URLs fail again.`;
+            break;
+        case "preparation_failed":
+            step = "Try other --url pages, or contact Zenrows support.";
+            break;
+        default:
+            step = "Contact Zenrows support before preparing this domain again. Meanwhile extract with --css.";
+    }
+    return step + slotSentence(failure?.slot_consumed);
+}
+/**
+ * What to do when a page on a prepared domain answers REQS007 with reason
+ * page_not_recognised, in the gateway's own order. On a prepared domain Extract
+ * learns a new kind of page on its first request, so that answer means this page
+ * is not one Extract extracts (a page with nothing to extract, such as a home page) or a
+ * transient failure kept it from trying, never that the domain needs preparing
+ * again: resubmitting returns the existing ready record, so pointing there would
+ * loop.
+ */
+export const PAGE_NOT_RECOGNISED_STEPS = "Try a product, article or listing page on the same site instead. If this page should work, retry it once in a few minutes; do not retry in a loop. Otherwise use --css with explicit selectors. Preparing the domain again does not change this.";
+/** Said once a domain is ready, so a user who then meets EXTRACT_PAGE_NOT_RECOGNISED does not resubmit. */
+const IF_PAGE_NOT_RECOGNISED = `If the extract then fails with EXTRACT_PAGE_NOT_RECOGNISED, Extract did not recognise that page. ${PAGE_NOT_RECOGNISED_STEPS}`;
+/**
+ * The next step for a status record. `alreadyReady` is set when a submit answered
+ * ready at once: the domain was prepared before this call.
+ */
+export function statusNextStep(record, opts = {}) {
+    if (record.status === "ready") {
+        const extract = opts.exampleUrl ? ` (\`zenrows extract ${opts.exampleUrl}\`)` : "";
+        if (opts.alreadyReady)
+            return `${record.domain} is already prepared, so nothing new was submitted: run the extract${extract}. ${IF_PAGE_NOT_RECOGNISED}`;
+        return `${record.domain} is ready: run the extract again${extract}. ${IF_PAGE_NOT_RECOGNISED}`;
+    }
+    if (record.status === "failed")
+        return failureNextStep(record.domain, record.failure);
+    return `Preparing; it takes a few minutes. Check about every 30 seconds with: zenrows extract prepare status ${record.domain}`;
+}

package/evals/extract-smoke/README.md

@@ -2,7 +2,7 @@
 
 Reproducible smoke test for Extract (`extract=auto`).
 
-- **Target:** `https://www.owler.com/company/meltwater` (enabled domain, real company page)
+- **Target:** `https://www.owler.com/company/meltwater` (prepared domain, real company page)
 - **Config:** `extract=auto`
 - **Success criteria:** HTTP 2xx and a non-empty body
 - **Cost:** ~1× basic request

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zenrows/cli",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "description": "Open-source CLI, MCP, skills, templates, workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.",
   "type": "module",
   "license": "MIT",

package/registry/capabilities.json

@@ -12,12 +12,12 @@
     },
     "extract": {
       "key": "extract",
-      "label": "Extract (extract=auto / Autoparse / CSS / Markdown)",
+      "label": "Extract (extract=auto / CSS / Markdown)",
       "status": "available",
       "command": "zenrows extract",
-      "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)",
+      "backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, css_extractor, outputs, response_type; autoparse deprecated)",
       "requiresAuth": true,
-      "notes": "Default is extract=auto, which works on any prepared domain with any key; CLI falls back to autoparse on REQS007 (domain not prepared yet) or AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
+      "notes": "Default is extract=auto, which works on any prepared domain with any key. A REQS007 (legacy AUTH010) is reported, never retried, with the next step for its reason: on a domain not prepared yet (EXTRACT_DOMAIN_NOT_PREPARED), prepare it with `zenrows extract prepare <domain> --url <page> [--wait]` (POST /v1/extract/prepared-domains: 1 to 5 example URLs, a few minutes, no credits) and check it with `zenrows extract prepare status <domain>`; on a page Extract did not recognise on a prepared domain (EXTRACT_PAGE_NOT_RECOGNISED), try a product, article or listing page, retry it once in a few minutes, or use --css, because preparing again does not change it. CSS / outputs / markdown work on any domain; autoparse is deprecated and runs only with --autoparse."
     },
     "batch": {
       "key": "batch",

package/registry/skills.json

@@ -25,13 +25,13 @@
     {
       "name": "extract",
       "type": "skill",
-      "description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.",
+      "description": "Turn protected pages into structured data with Extract / CSS / Markdown (Autoparse is deprecated).",
       "status": "available",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,
       "version": "0.1.0",
       "path": "skills/extract",
-      "tags": ["extract", "autoparse", "structured-data"]
+      "tags": ["extract", "structured-data"]
     },
     {
       "name": "batch-jobs",

package/registry/templates.json

@@ -14,7 +14,7 @@
     {
       "name": "extract-pricing-pages",
       "type": "template",
-      "description": "Extract structured pricing fields from product pages (Autoparse / CSS).",
+      "description": "Extract structured pricing fields from product pages (extract=auto / CSS).",
       "status": "available",
       "requires_backend_capabilities": ["extract"],
       "requires_auth": true,

package/skills/extract/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: extract
-description: Turn protected pages into structured data with Extract / Autoparse / CSS / Markdown.
+description: Turn protected pages into structured data with Extract / CSS / Markdown (Autoparse is deprecated).
 version: 0.1.0
 requires_backend_capabilities: [extract]
 ---
@@ -12,8 +12,8 @@ access + extraction**, not generic LLM parsing.
 
 ## Methods
 ```
-zenrows extract <url>                              # extract=auto (default); falls back to Autoparse if the domain is not prepared yet
-zenrows extract <url> --autoparse                  # general-purpose Autoparse (any domain)
+zenrows extract <url>                              # extract=auto (default), any prepared domain
+zenrows extract <url> --autoparse                  # deprecated general-purpose Autoparse (any domain)
 zenrows extract <url> --css '{"title":"h1","price":".price"}'   # selector map
 zenrows extract <url> --outputs emails,links       # built-in output filters → JSON
 zenrows extract <url> --output markdown            # Markdown conversion
@@ -21,14 +21,58 @@ zenrows extract <url> --validate                   # fail if not valid JSON
 ```
 
 ## When to use which
-- **Extract** (`extract=auto`, default): richest fields; works on any domain Zenrows has prepared.
-- **Autoparse**: any website; also the automatic fallback when the domain is not prepared for Extract yet.
+- **Extract** (`extract=auto`, default): richest fields; works on any prepared
+  domain (you can prepare one yourself, see below).
+- **Autoparse** (deprecated): only with `--autoparse`. Never automatic; do not
+  build new work on it.
 - **CSS**: you know the exact fields/selectors and want determinism.
 - **Markdown/plaintext**: feed clean content to an LLM yourself.
 
+## REQS007: prepare the domain, then extract again
+
+`extract=auto` works on any prepared domain, and you can prepare one yourself.
+When Extract refuses a page with `403 REQS007`, `zenrows extract` says which
+of three cases it is and prints the next step. It never retries with Autoparse.
+
+| Error code | Meaning | Next step |
+| --- | --- | --- |
+| `EXTRACT_DOMAIN_NOT_PREPARED` | The domain is not prepared yet (also the legacy `402 AUTH010`). | Prepare it (below), then extract again. |
+| `EXTRACT_PAGE_NOT_RECOGNISED` | The domain is prepared, but Extract did not recognise this page as one it can extract (a page with nothing to extract, such as a home page, or a transient failure). | Try a product, article or listing page on the same site instead. If this page should work, retry it once in a few minutes; do not retry in a loop. Otherwise use `--css`. Preparing the domain again does not change this. |
+| `EXTRACT_DOMAIN_NOT_ENABLED` | Extract is not enabled for the site. | Use `--css`, or contact Zenrows support to have the site enabled. Preparing does not change this. |
+
+Preparing a domain:
+
+```
+zenrows extract https://example.com/p/1             # 1. fails: EXTRACT_DOMAIN_NOT_PREPARED
+zenrows extract prepare example.com \
+  --url https://example.com/p/1 --url https://example.com/list --wait
+                                                    # 2. prepare it and wait until ready (a few minutes)
+zenrows extract https://example.com/p/1             # 3. extract again
+```
+
+- `--url`: 1 to 5 real pages you want to extract, one per page kind (for
+  example a product page and a listing page).
+- Without `--wait` the command returns at once; then check about every 30
+  seconds with `zenrows extract prepare status example.com` until it says
+  `ready`. `--wait` polls at the same pace.
+- The domain is the host: `shop.example.com` and `example.com` are prepared
+  separately, and a leading `www.` is dropped.
+- Preparing takes a few minutes and costs no credits. Each new domain uses one
+  slot of the account's monthly allowance; `prepare status` shows `used` /
+  `limit` / `remaining`.
+- `EXTRACT_PREPARE_OUT_OF_ALLOWANCE`: no slots left this month; use `--css`.
+  `EXTRACT_PREPARE_UNAVAILABLE`: retry in a few minutes; do not loop.
+  A `failed` status prints the next step for its failure code and says whether
+  the attempt used an allowance slot.
+- A domain that is already prepared answers `ready` at once
+  (`already_prepared`) and uses no slot: run the extract. If it then fails with
+  `EXTRACT_PAGE_NOT_RECOGNISED`, follow that row of the table, not `prepare`.
+
 ## Rules
 - Validate on a single page before scaling across many URLs.
-- If Autoparse misses fields, switch to `--css` with explicit selectors. Auto
+- Check `empty` in `--json` output: true means every field came back null or
+  empty, so the result is not data.
+- If a field is missing, switch to `--css` with explicit selectors. Auto
   mode already renders JS when the target needs it, so reach for
   `--manual --js-render` only after auto mode has returned an incomplete page:
   it costs several times a basic call ([[cost-control]]) and it makes the

package/skills/zenrows/SKILL.md

@@ -34,7 +34,7 @@ If the user has a known URL and wants page content:
   → Use Protected Fetch.            (zenrows fetch <url>)
 
 If the user has a known URL and wants structured data:
-  → Use Extract.                    (zenrows extract <url> | --autoparse | --css)
+  → Use Extract.                    (zenrows extract <url> | --css)
 
 If the user has a start page (listing, index, section) but not the item URLs:
   → Use Crawl.                      (zenrows crawl create <url> --depth 1 --follow,
@@ -75,13 +75,16 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 | Primitive | Command | Status |
 | --- | --- | --- |
 | Protected Fetch | `zenrows fetch` | **available**: `GET https://api.zenrows.com/v1/` |
-| Extract (`extract=auto`, Autoparse, CSS, Markdown) | `zenrows extract` | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
+| Extract (`extract=auto`, CSS, Markdown; Autoparse deprecated) | `zenrows extract` | **available**: `GET https://api.zenrows.com/v1/?extract=auto` |
 | Batch | `zenrows batch` | **available**: `POST https://async.api.zenrows.com/v1/jobs` |
 | Crawl (new) | `zenrows crawl` | **available**: `POST https://api.zenrows.com/v1/crawls` |
 | Browser | `zenrows browser` | **available**: `POST https://mcp.zenrows.com/browser/sessions` |
 | MCP | `zenrows mcp` | **available**: `https://mcp.zenrows.com/mcp` |
 
-- Extract: `extract=auto` falls back to Autoparse on a domain not prepared yet.
+- Extract: `extract=auto` works on any prepared domain. On one not prepared yet
+  (`EXTRACT_DOMAIN_NOT_PREPARED`), run `zenrows extract prepare <domain> --url
+  <page> --wait`, then extract again. `EXTRACT_PAGE_NOT_RECOGNISED` means the
+  domain is prepared but not this page: do not prepare again; see [[extract]].
 - Batch: the reliable path at scale, at Fetch's prices (create, list, status, wait, results, content, pause, resume, rerun, delete). Estimate specs locally with no key.
 - Crawl: collects the URLs behind one start page, optionally with each page's HTML.
 - Browser: escalation-only, for pages Fetch can't handle; same backend as the MCP `browser_*` tools.

package/templates/extract-pricing-pages/README.md

@@ -3,13 +3,11 @@
 Extract structured pricing fields from product pages with Zenrows Extract.
 
 ```bash
-# Default: extract=auto on an enabled domain (site-tailored JSON)
+# Default: extract=auto on a prepared domain (site-tailored JSON).
+# Not prepared yet (EXTRACT_DOMAIN_NOT_PREPARED)? zenrows extract prepare <domain> --url <product-url> --wait
 zenrows extract https://www.owler.com/company/meltwater
 
-# Autoparse — general-purpose JSON on any domain
-zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
-
-# Or a precise CSS selector map
+# A precise CSS selector map, on any domain
 zenrows extract <product-url> --css '{"title":"h1","price":".price"}' --validate
 ```
 

package/workflows/competitor-intelligence/WORKFLOW.md

@@ -9,7 +9,7 @@ status: available
 
 A higher-level process built on Zenrows **Fetch and Extract**: retrieve
 protected pages with `zenrows fetch`, then turn them into structured data with
-`zenrows extract` (the same API's Autoparse / CSS / Markdown — not a separate
+`zenrows extract` (the same API's `extract=auto` / CSS / Markdown — not a separate
 product). Scheduling and scaling use the tools you already have (cron/CI, a
 simple per-URL loop).
 
@@ -20,7 +20,7 @@ simple per-URL loop).
    ```
    zenrows fetch https://competitor.example/products
    ```
-2. **Validate extraction** on one page (extract=auto falls back to autoparse on a domain not prepared yet).
+2. **Validate extraction** on one page (on a domain not prepared yet, `EXTRACT_DOMAIN_NOT_PREPARED`, run `zenrows extract prepare <domain> --url <page> --wait` first).
    ```
    zenrows extract https://competitor.example/p/123 --validate
    ```