@zenrows/cli 1.5.0 → 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -8
- package/agent-plugin/skills/batch-jobs/SKILL.md +3 -3
- package/agent-plugin/skills/cost-control/SKILL.md +1 -2
- package/agent-plugin/skills/extract/SKILL.md +5 -5
- package/agent-plugin/skills/zenrows/SKILL.md +3 -3
- package/dist/src/adapters/batch.js +2 -3
- package/dist/src/adapters/extract.js +13 -1
- package/dist/src/cli/commands/batch.js +6 -6
- package/dist/src/cli/commands/extract.js +7 -3
- package/dist/src/core/batch-api.js +2 -2
- package/dist/src/core/capabilities.js +2 -2
- package/dist/src/core/config.js +1 -1
- package/dist/src/core/http.js +24 -5
- package/package.json +1 -1
- package/recipes/batch-estimate/RECIPE.md +2 -2
- package/registry/capabilities.json +5 -5
- package/registry/skills.json +3 -3
- package/registry/templates.json +2 -2
- package/skills/batch-jobs/SKILL.md +6 -7
- package/skills/extract/SKILL.md +3 -3
- package/skills/zenrows/SKILL.md +2 -2
- package/templates/batch-jsonl-pipeline/README.md +5 -5
- package/templates/extract-pricing-pages/README.md +1 -1
- package/workflows/competitor-intelligence/WORKFLOW.md +1 -1
package/README.md
CHANGED
|
@@ -37,8 +37,8 @@ that endpoint with extraction parameters, not a separate product.
|
|
|
37
37
|
| Command | What it does | Status (this build) |
|
|
38
38
|
| --- | --- | --- |
|
|
39
39
|
| `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
|
|
40
|
-
| `zenrows extract` | Extract
|
|
41
|
-
| `zenrows batch` | Batch
|
|
40
|
+
| `zenrows extract` | Extract: `extract=auto` (any prepared domain; falls back to Autoparse on one not prepared yet), CSS, or Markdown | available, same `/v1/` |
|
|
41
|
+
| `zenrows batch` | Batch: fan out over many URLs | **available**: `https://async.api.zenrows.com/v1`; local validate/estimate need no key |
|
|
42
42
|
| `zenrows crawl` | Crawl (new) — collect the URLs behind one start page, optionally with each page's HTML | **available** — `https://api.zenrows.com/v1/crawls` |
|
|
43
43
|
| `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |
|
|
44
44
|
| `zenrows mcp` | MCP server config (remote + local) | **available** |
|
|
@@ -123,12 +123,11 @@ zenrows extract <url> --outputs emails,links # built-
|
|
|
123
123
|
zenrows extract <url> --output markdown
|
|
124
124
|
```
|
|
125
125
|
|
|
126
|
-
## 9. Batch
|
|
126
|
+
## 9. Batch
|
|
127
127
|
|
|
128
128
|
Zenrows **Batch** (`https://async.api.zenrows.com/v1`) fans a
|
|
129
|
-
protected fetch/extract out over many URLs.
|
|
130
|
-
|
|
131
|
-
beta access; without it the API returns `BATCH_ACCESS_DENIED`. Local spec
|
|
129
|
+
protected fetch/extract out over many URLs. The cloud subcommands work
|
|
130
|
+
with any active API key and spend the same credits as Fetch. Local spec
|
|
132
131
|
validation + credit estimation always work (no key needed).
|
|
133
132
|
|
|
134
133
|
```bash
|
|
@@ -141,8 +140,7 @@ zenrows batch cancel <job-id> # stop an in-flight run
|
|
|
141
140
|
|
|
142
141
|
`jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
|
|
143
142
|
per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
|
|
144
|
-
`extract`).
|
|
145
|
-
validate/estimate locally or fan out with `zenrows fetch` per URL.
|
|
143
|
+
`extract`). Validate and estimate locally first, then submit with `create`.
|
|
146
144
|
|
|
147
145
|
**Extract in Batch.** Set `"extract": "auto"` on a line to run that URL through
|
|
148
146
|
Extract — structured data instead of raw HTML:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: batch-jobs
|
|
3
|
-
description: Scale protected fetch/extract over many URLs via Zenrows MCP batch_* tools
|
|
3
|
+
description: Scale protected fetch/extract over many URLs via Zenrows MCP batch_* tools.
|
|
4
4
|
version: 0.1.0
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -9,8 +9,8 @@ version: 0.1.0
|
|
|
9
9
|
Process large workloads asynchronously. Batch is where Zenrows' high-scale
|
|
10
10
|
anti-bot advantage becomes obvious — thousands to millions of URLs.
|
|
11
11
|
|
|
12
|
-
>
|
|
13
|
-
>
|
|
12
|
+
> Cloud Batch lives on `async.api.zenrows.com/v1`. A 403 maps to
|
|
13
|
+
> `BATCH_ACCESS_DENIED`. Do not fake a successful
|
|
14
14
|
> run. This is **not** `browser_batch` (that batches browser actions in one
|
|
15
15
|
> session).
|
|
16
16
|
|
|
@@ -18,7 +18,7 @@ These multipliers apply when you enable the flags (on `scrape` or `extract`):
|
|
|
18
18
|
- Premium proxies (`premium_proxy`): **10×**
|
|
19
19
|
- Both: **25×**
|
|
20
20
|
|
|
21
|
-
`extract` `mode=auto` is
|
|
21
|
+
`extract` `mode=auto` is currently free; billing may apply
|
|
22
22
|
later. Do not promise permanent free extract pricing. Autoparse / CSS still
|
|
23
23
|
follow the multipliers above when stealth flags are set.
|
|
24
24
|
|
|
@@ -32,7 +32,6 @@ Always call `browser_close`. Prefer `scrape` / `extract` when they can do the jo
|
|
|
32
32
|
|
|
33
33
|
Batch is usually better at **scale** (many URLs) than fan-out of per-URL tool
|
|
34
34
|
calls. Validate one URL with `scrape`/`extract` first, then `batch_create`.
|
|
35
|
-
Batch access may be beta-gated (`BATCH_ACCESS_DENIED`).
|
|
36
35
|
|
|
37
36
|
## Tactics
|
|
38
37
|
|
|
@@ -20,10 +20,10 @@ extract({ url, mode: "css", css_extractor: '{"title":"h1","price":".price"}' })
|
|
|
20
20
|
extract({ url, proxy_country: "US" }) # geo-targeted
|
|
21
21
|
```
|
|
22
22
|
|
|
23
|
-
`mode=auto` (`extract=auto`)
|
|
24
|
-
currently free; billing may apply later. If the
|
|
25
|
-
or not prepared yet (REQS007), the tool retries
|
|
26
|
-
(`fallback_autoparse`).
|
|
23
|
+
`mode=auto` (`extract=auto`) works on any domain Zenrows has prepared, with any
|
|
24
|
+
key, and is currently free; billing may apply later. If the
|
|
25
|
+
domain is not enabled (AUTH010) or not prepared yet (REQS007), the tool retries
|
|
26
|
+
once with Autoparse by default (`fallback_autoparse`).
|
|
27
27
|
|
|
28
28
|
Every mode uses Adaptive Stealth Mode by default: pass just the URL, even for
|
|
29
29
|
protected or dynamic sites. Set `js_render` or `premium_proxy` only to force a
|
|
@@ -45,7 +45,7 @@ scrape({ url, outputs: "emails,links" }) # or "*" for all built-ins
|
|
|
45
45
|
|
|
46
46
|
## When to use which
|
|
47
47
|
|
|
48
|
-
- **`mode=auto`**: site-tailored fields on
|
|
48
|
+
- **`mode=auto`**: site-tailored fields on any prepared domain.
|
|
49
49
|
- **`mode=autoparse`**: any website; also the automatic AUTH010 / REQS007 fallback.
|
|
50
50
|
- **`mode=css`**: known fields/selectors; deterministic.
|
|
51
51
|
- **Markdown via `scrape`**: feed clean content to an LLM yourself.
|
|
@@ -42,7 +42,7 @@ If the user has many URLs (bulk):
|
|
|
42
42
|
→ Use Batch. (validate one page with scrape/extract first,
|
|
43
43
|
then batch_create → batch_status → batch_results)
|
|
44
44
|
Prefer Batch over fan-out of scrape/extract
|
|
45
|
-
per URL for bulk work
|
|
45
|
+
per URL for bulk work.
|
|
46
46
|
Cancel with batch_cancel when needed.
|
|
47
47
|
|
|
48
48
|
If the user needs login, clicks, forms, sessions, or persistent state:
|
|
@@ -66,8 +66,8 @@ If the user needs login, clicks, forms, sessions, or persistent state:
|
|
|
66
66
|
| Primitive | MCP tools | Notes |
|
|
67
67
|
| --- | --- | --- |
|
|
68
68
|
| Protected Fetch | `scrape` | Default markdown; HTML / plaintext / PDF / screenshot options |
|
|
69
|
-
| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto`
|
|
70
|
-
| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` |
|
|
69
|
+
| Extract | `extract` | Structured JSON (`extract=auto`, autoparse, css_extractor). `extract=auto` is currently free; billing may apply later |
|
|
70
|
+
| Batch | `batch_create`, `batch_status`, `batch_results`, `batch_cancel` | Same credits as Fetch; a 403 maps to `BATCH_ACCESS_DENIED` |
|
|
71
71
|
| Browser | `browser_*` (start with `browser_navigate`) | Escalation-only; same backend as Browser Sessions |
|
|
72
72
|
|
|
73
73
|
## Cost notes
|
|
@@ -1,9 +1,8 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Batch Jobs adapter.
|
|
3
3
|
*
|
|
4
|
-
* Status: `
|
|
5
|
-
*
|
|
6
|
-
* Without access the API returns 403 → BATCH_ACCESS_DENIED. This adapter owns
|
|
4
|
+
* Status: `available`. The cloud subcommands call the Batch API with an API key
|
|
5
|
+
* (see `core/batch-api.ts`); a 403 maps to BATCH_ACCESS_DENIED. This adapter owns
|
|
7
6
|
* the local, no-network pieces: validating a JSONL job spec, estimating credit
|
|
8
7
|
* cost, and mapping the validated spec to the Batch API request body.
|
|
9
8
|
*/
|
|
@@ -55,6 +55,18 @@ async function runExtractOnce(opts, config, policy, apiKey) {
|
|
|
55
55
|
output: method === "markdown" ? "markdown" : method === "plaintext" ? "plaintext" : "html",
|
|
56
56
|
};
|
|
57
57
|
const outcome = await runFetch(fetchOpts, config, policy, apiKey);
|
|
58
|
+
if (outcome.result.noDataExtracted) {
|
|
59
|
+
if (opts.validate) {
|
|
60
|
+
throw new ToolkitError({
|
|
61
|
+
code: "EXTRACT_VALIDATION_FAILED",
|
|
62
|
+
message: "Extract found no data on this page.",
|
|
63
|
+
likely_cause: "Extract knows this kind of page, but none of the fields came back filled (HTTP 422 no_data_extracted). The request was not charged.",
|
|
64
|
+
next_action: "Check that the page rendered fully: try --manual --js-render with --wait-for <selector>. Protected sites may also need --premium-proxy.",
|
|
65
|
+
suggested_commands: [`zenrows extract ${opts.url} --manual --js-render --premium-proxy`],
|
|
66
|
+
});
|
|
67
|
+
}
|
|
68
|
+
return { ...outcome, method, empty: true, noDataExtracted: true };
|
|
69
|
+
}
|
|
58
70
|
const { data, html, empty } = parseExtractBody(method, outcome.result.body, opts);
|
|
59
71
|
return { ...outcome, method, data, html, empty };
|
|
60
72
|
}
|
|
@@ -80,7 +92,7 @@ function parseExtractBody(method, body, opts) {
|
|
|
80
92
|
}
|
|
81
93
|
let data = parsed;
|
|
82
94
|
let html;
|
|
83
|
-
// extract=auto
|
|
95
|
+
// extract=auto shape: { parsed, html }. Prefer `parsed` for callers.
|
|
84
96
|
if (method === "extract" && parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
|
85
97
|
const envelope = parsed;
|
|
86
98
|
if ("parsed" in envelope) {
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* `zenrows batch
|
|
2
|
+
* `zenrows batch`: Batch API (status: available).
|
|
3
3
|
*
|
|
4
4
|
* Local (no key, always works): `estimate`/`create --dry-run`-style spec
|
|
5
|
-
* validation + credit estimate. Cloud (needs a key
|
|
6
|
-
* `create`, `status`, `results`, `cancel`, `wait`, `retry-failed`.
|
|
7
|
-
*
|
|
5
|
+
* validation + credit estimate. Cloud (needs a key):
|
|
6
|
+
* `create`, `status`, `results`, `cancel`, `wait`, `retry-failed`. A 403 maps to
|
|
7
|
+
* BATCH_ACCESS_DENIED.
|
|
8
8
|
*/
|
|
9
9
|
import { statSync } from "node:fs";
|
|
10
10
|
import { join } from "node:path";
|
|
@@ -22,12 +22,12 @@ import { ToolkitError, isKeyCapReached, keyCapReached } from "../../core/errors.
|
|
|
22
22
|
import { printError, writeOut } from "../output.js";
|
|
23
23
|
export const batch = {
|
|
24
24
|
name: "batch",
|
|
25
|
-
summary: "Run JSONL batch jobs on Zenrows Batch
|
|
25
|
+
summary: "Run JSONL batch jobs on Zenrows Batch.",
|
|
26
26
|
usage: "zenrows batch <estimate|create <file.jsonl>|status <id>|results <id>|cancel <id>|wait <id>|retry-failed <id>>",
|
|
27
27
|
help: [
|
|
28
28
|
"Local (no key):",
|
|
29
29
|
" estimate <file.jsonl> [--manual] validate the spec + estimate credits (upper bound)",
|
|
30
|
-
"Cloud (needs a key
|
|
30
|
+
"Cloud (needs a key):",
|
|
31
31
|
" create <file.jsonl> [flags] validate, then submit the job",
|
|
32
32
|
" (default) Adaptive Stealth Mode (mode=auto) on every task that",
|
|
33
33
|
" doesn't force js_render / premium_proxy itself",
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* `zenrows extract <url> [flags]` → Extract (structured extraction on `/v1/`).
|
|
3
3
|
*
|
|
4
|
-
* Default: `extract=auto` (site-tailored Extract
|
|
4
|
+
* Default: `extract=auto` (site-tailored Extract). If the domain is
|
|
5
5
|
* not enabled or prepared (AUTH010 / REQS007), falls back once to Autoparse. Opt into Autoparse
|
|
6
6
|
* directly with `--autoparse`. Other methods: `--css`, `--outputs`, `--output`.
|
|
7
7
|
*/
|
|
@@ -23,7 +23,7 @@ export const extract = {
|
|
|
23
23
|
usage: "zenrows extract <url> [--autoparse | --css <json> | --outputs <filters> | --output md|text] [flags]",
|
|
24
24
|
help: [
|
|
25
25
|
"Methods:",
|
|
26
|
-
" (default) extract=auto
|
|
26
|
+
" (default) extract=auto: site-tailored Extract; falls back to Autoparse if the domain is not enabled or prepared",
|
|
27
27
|
" --autoparse general-purpose Autoparse (any domain, no Extract fallback)",
|
|
28
28
|
" --css <json> CSS selector map, e.g. '{\"title\":\"h1\",\"price\":\".price\"}'",
|
|
29
29
|
" --outputs <list> built-in output filters → JSON: emails, phone_numbers, headings,",
|
|
@@ -148,7 +148,10 @@ export const extract = {
|
|
|
148
148
|
log.success(`Extracted via ${via} · ${outcome.result.body.length} bytes · ${formatRequestCost(outcome.result.costUsd, outcome.result.costCredits)} · run ${runId}`);
|
|
149
149
|
if (runDir)
|
|
150
150
|
log.dim(` artifact: ${runDir}`);
|
|
151
|
-
if (outcome.
|
|
151
|
+
if (outcome.noDataExtracted) {
|
|
152
|
+
log.warn("Extract found no data on this page (HTTP 422 no_data_extracted, not charged). Check that the page rendered fully: try --manual --js-render with --wait-for <selector>. Protected sites may also need --premium-proxy.");
|
|
153
|
+
}
|
|
154
|
+
else if (outcome.empty) {
|
|
152
155
|
log.warn("No structured data extracted — the page may render via JavaScript, or this layout isn't auto-extractable. Try --manual --js-render, or --css with explicit selectors.");
|
|
153
156
|
}
|
|
154
157
|
if (ctx.json || values.json) {
|
|
@@ -158,6 +161,7 @@ export const extract = {
|
|
|
158
161
|
method: outcome.method,
|
|
159
162
|
fellBackToAutoparse: outcome.fellBackToAutoparse ?? false,
|
|
160
163
|
empty: outcome.empty ?? false,
|
|
164
|
+
noDataExtracted: outcome.noDataExtracted ?? false,
|
|
161
165
|
httpStatus: outcome.result.status,
|
|
162
166
|
data: outcome.data ?? null,
|
|
163
167
|
bytes: outcome.result.body.length,
|
|
@@ -111,8 +111,8 @@ function problemToError(status, body, method, path) {
|
|
|
111
111
|
return new ToolkitError({
|
|
112
112
|
code: "BATCH_ACCESS_DENIED",
|
|
113
113
|
message: "The Batch API rejected this request (access denied).",
|
|
114
|
-
likely_cause: `${cause}.
|
|
115
|
-
next_action: "
|
|
114
|
+
likely_cause: `${cause}. Batch is not enabled for this account or API key.`,
|
|
115
|
+
next_action: "Check that your Zenrows API key is active, and contact Zenrows support if the 403 persists. Meanwhile validate/estimate specs locally and fan out with `zenrows fetch` per URL.",
|
|
116
116
|
suggested_commands: ["zenrows batch estimate jobs.jsonl"],
|
|
117
117
|
});
|
|
118
118
|
}
|
|
@@ -24,8 +24,8 @@ export function getCapability(key) {
|
|
|
24
24
|
const USABLE = new Set([
|
|
25
25
|
"available",
|
|
26
26
|
"available-but-needs-confirmation",
|
|
27
|
-
//
|
|
28
|
-
//
|
|
27
|
+
// Beta: any key can call; product-specific limits are handled by the
|
|
28
|
+
// adapters / API errors, not here.
|
|
29
29
|
"beta",
|
|
30
30
|
]);
|
|
31
31
|
export function isUsable(key) {
|
package/dist/src/core/config.js
CHANGED
|
@@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0";
|
|
|
14
14
|
* entrypoint) so core modules — telemetry, signup provenance — can read it
|
|
15
15
|
* without an import cycle. `VERSION` in `cli/index.ts` re-exports this.
|
|
16
16
|
*/
|
|
17
|
-
export const CLI_VERSION = "1.5.
|
|
17
|
+
export const CLI_VERSION = "1.5.1";
|
|
18
18
|
/** The User-Agent on every request to a Zenrows API. The gateway reads the version from it. */
|
|
19
19
|
export const CLI_USER_AGENT = `zenrows-cli/${CLI_VERSION}`;
|
|
20
20
|
/** Env var to override the Fetch and Extract API base (local/staging testing). */
|
package/dist/src/core/http.js
CHANGED
|
@@ -106,10 +106,29 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
|
|
|
106
106
|
code: "EXTRACT_DOMAIN_NOT_ENABLED",
|
|
107
107
|
message: "Extract has not been prepared for this domain yet.",
|
|
108
108
|
likely_cause: zrErrorDetail(body) ?? "This domain is not prepared for Extract.",
|
|
109
|
-
next_action: "Retry with --autoparse for general-purpose extraction on any site.",
|
|
109
|
+
next_action: "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
|
|
110
110
|
suggested_commands: [`zenrows extract ${params.url} --autoparse`],
|
|
111
111
|
});
|
|
112
112
|
}
|
|
113
|
+
// 422 no_data_extracted on an Extract request: the page was processed and had
|
|
114
|
+
// nothing to extract. It is an answer, not a failure to reach the target, so
|
|
115
|
+
// return it for the extract adapter to report as an empty result instead of
|
|
116
|
+
// advising escalation (which would repeat a slow, deterministic request).
|
|
117
|
+
if (res.status === 422 && params.extract !== undefined && zrErrorCode(body) === "NO_DATA_EXTRACTED") {
|
|
118
|
+
return { ...result, noDataExtracted: true };
|
|
119
|
+
}
|
|
120
|
+
// 422 blocked_document on an Extract request: the fetched page was a bot wall,
|
|
121
|
+
// an error page or an empty shell, so Extract had nothing real to read. Not
|
|
122
|
+
// charged. The fix is a stronger fetch of the same URL, not another extractor.
|
|
123
|
+
if (res.status === 422 && params.extract !== undefined && zrErrorCode(body) === "BLOCKED_DOCUMENT") {
|
|
124
|
+
throw new ToolkitError({
|
|
125
|
+
code: "EXTRACT_FAILED",
|
|
126
|
+
message: "Extract could not read this page: the site answered with a block or error page (HTTP 422 blocked_document, not charged).",
|
|
127
|
+
likely_cause: zrErrorDetail(body) ?? "The fetched page was a bot challenge, an error page or a page that had not finished loading.",
|
|
128
|
+
next_action: "Retry with a stronger fetch: --manual --js-render --premium-proxy. Check the URL if the site may be returning an error page.",
|
|
129
|
+
suggested_commands: [`zenrows extract ${params.url} --manual --js-render --premium-proxy`],
|
|
130
|
+
});
|
|
131
|
+
}
|
|
113
132
|
if (res.status === 401 || res.status === 403) {
|
|
114
133
|
// Distinguish Zenrows auth errors from target-site 403s. Zenrows auth
|
|
115
134
|
// failures surface AUTH00x codes in the body.
|
|
@@ -143,15 +162,15 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
|
|
|
143
162
|
}
|
|
144
163
|
}
|
|
145
164
|
if (res.status === 402 && isZenrowsErrorEnvelope(body)) {
|
|
146
|
-
// AUTH010 on extract=auto means the target domain is not
|
|
147
|
-
//
|
|
165
|
+
// AUTH010 on extract=auto means the target domain is not enabled for
|
|
166
|
+
// Extract, distinct from credit exhaustion (AUTH004). Let the extract
|
|
148
167
|
// adapter fall back to autoparse.
|
|
149
168
|
if (params.extract !== undefined && zrErrorCode(body) === "AUTH010") {
|
|
150
169
|
throw new ToolkitError({
|
|
151
170
|
code: "EXTRACT_DOMAIN_NOT_ENABLED",
|
|
152
171
|
message: "Extract is not enabled for this domain yet.",
|
|
153
|
-
likely_cause: zrErrorDetail(body) ?? "This domain is not
|
|
154
|
-
next_action: "Retry with --autoparse for general-purpose extraction on any site, or
|
|
172
|
+
likely_cause: zrErrorDetail(body) ?? "This domain is not enabled for Extract.",
|
|
173
|
+
next_action: "Retry with --autoparse for general-purpose extraction on any site, or get the domain prepared for Extract (POST /v1/extract/prepared-domains).",
|
|
155
174
|
suggested_commands: [`zenrows extract ${params.url} --autoparse`],
|
|
156
175
|
});
|
|
157
176
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zenrows/cli",
|
|
3
|
-
"version": "1.5.
|
|
3
|
+
"version": "1.5.1",
|
|
4
4
|
"description": "Open-source CLI, MCP, skills, templates, workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -26,8 +26,8 @@ Swap `jobs.jsonl` for your own spec. Per-line keys (`js_render`,
|
|
|
26
26
|
`premium_proxy`, `proxy_country`, `mode`, `autoparse`, `external_id`) tune each
|
|
27
27
|
URL; cost multipliers are basic 1x / JS 5x / proxy 10x / both 25x.
|
|
28
28
|
|
|
29
|
-
Once the spec is clean, submit it
|
|
29
|
+
Once the spec is clean, submit it:
|
|
30
30
|
```
|
|
31
31
|
zenrows batch create jobs.jsonl --wait
|
|
32
32
|
```
|
|
33
|
-
See [[batch-jobs]] for the full cloud flow
|
|
33
|
+
See [[batch-jobs]] for the full cloud flow.
|
|
@@ -13,20 +13,20 @@
|
|
|
13
13
|
"extract": {
|
|
14
14
|
"key": "extract",
|
|
15
15
|
"label": "Extract (extract=auto / Autoparse / CSS / Markdown)",
|
|
16
|
-
"status": "
|
|
16
|
+
"status": "available",
|
|
17
17
|
"command": "zenrows extract",
|
|
18
18
|
"backend": "Extract — GET https://api.zenrows.com/v1/ (extract=auto, autoparse, css_extractor, outputs, response_type)",
|
|
19
19
|
"requiresAuth": true,
|
|
20
|
-
"notes": "
|
|
20
|
+
"notes": "Default is extract=auto, which works on any prepared domain with any key; CLI falls back to autoparse on REQS007 (domain not prepared yet) or AUTH010. Autoparse / CSS / outputs / markdown work on any domain."
|
|
21
21
|
},
|
|
22
22
|
"batch": {
|
|
23
23
|
"key": "batch",
|
|
24
|
-
"label": "Batch
|
|
25
|
-
"status": "
|
|
24
|
+
"label": "Batch",
|
|
25
|
+
"status": "available",
|
|
26
26
|
"command": "zenrows batch",
|
|
27
27
|
"backend": "Batch — https://async.api.zenrows.com/v1 (X-API-Key header; separate host from the Fetch/Extract /v1/)",
|
|
28
28
|
"requiresAuth": true,
|
|
29
|
-
"notes": "
|
|
29
|
+
"notes": "Cloud subcommands (create/status/results/cancel/wait/retry-failed) call the Batch API with your API key; `zenrows batch estimate` works locally with no API key."
|
|
30
30
|
},
|
|
31
31
|
"crawl": {
|
|
32
32
|
"key": "crawl",
|
package/registry/skills.json
CHANGED
|
@@ -26,7 +26,7 @@
|
|
|
26
26
|
"name": "extract",
|
|
27
27
|
"type": "skill",
|
|
28
28
|
"description": "Turn protected pages into structured data with Autoparse / CSS / Markdown.",
|
|
29
|
-
"status": "
|
|
29
|
+
"status": "available",
|
|
30
30
|
"requires_backend_capabilities": ["extract"],
|
|
31
31
|
"requires_auth": true,
|
|
32
32
|
"version": "0.1.0",
|
|
@@ -36,8 +36,8 @@
|
|
|
36
36
|
{
|
|
37
37
|
"name": "batch-jobs",
|
|
38
38
|
"type": "skill",
|
|
39
|
-
"description": "Scale protected fetch/extract over many URLs with Batch
|
|
40
|
-
"status": "
|
|
39
|
+
"description": "Scale protected fetch/extract over many URLs with Batch: submit/track/collect jobs with an API key; validate + estimate specs locally with no key.",
|
|
40
|
+
"status": "available",
|
|
41
41
|
"requires_backend_capabilities": ["batch"],
|
|
42
42
|
"requires_auth": true,
|
|
43
43
|
"version": "0.1.0",
|
package/registry/templates.json
CHANGED
|
@@ -25,8 +25,8 @@
|
|
|
25
25
|
{
|
|
26
26
|
"name": "batch-jsonl-pipeline",
|
|
27
27
|
"type": "template",
|
|
28
|
-
"description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with
|
|
29
|
-
"status": "
|
|
28
|
+
"description": "JSONL job-spec scaffold for high-scale workloads on Batch: submit/track/collect with an API key, validate + estimate locally with no key.",
|
|
29
|
+
"status": "available",
|
|
30
30
|
"requires_backend_capabilities": [],
|
|
31
31
|
"requires_auth": false,
|
|
32
32
|
"version": "0.1.0",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: batch-jobs
|
|
3
|
-
description: Scale protected fetch/extract over many URLs via Batch
|
|
3
|
+
description: Scale protected fetch/extract over many URLs via Batch. Cloud create/status/results/cancel/wait/retry-failed work with an API key; estimate/validate run locally with no key.
|
|
4
4
|
version: 0.1.0
|
|
5
5
|
requires_backend_capabilities: [batch]
|
|
6
6
|
---
|
|
@@ -11,11 +11,10 @@ Process large workloads reliably and asynchronously. Batch is where Zenrows'
|
|
|
11
11
|
high-scale anti-bot advantage becomes obvious — Zenrows wins when the workflow
|
|
12
12
|
runs over thousands, millions, or recurring sets of URLs.
|
|
13
13
|
|
|
14
|
-
>
|
|
15
|
-
>
|
|
16
|
-
>
|
|
17
|
-
>
|
|
18
|
-
> toolkit never fakes a cloud run.
|
|
14
|
+
> Zenrows **Batch** runs on a separate host (`async.api.zenrows.com/v1`) and
|
|
15
|
+
> spends the same credits as Fetch. The cloud subcommands work with any active
|
|
16
|
+
> API key; a 403 maps to `BATCH_ACCESS_DENIED`. The toolkit never fakes a
|
|
17
|
+
> cloud run.
|
|
19
18
|
|
|
20
19
|
## Local (no key, always works)
|
|
21
20
|
```
|
|
@@ -40,7 +39,7 @@ correctly. The result carries two keys: `html` (the
|
|
|
40
39
|
raw page) and `parsed` (the structured data). Validate on a single URL with
|
|
41
40
|
`zenrows extract <url>` before running the full batch.
|
|
42
41
|
|
|
43
|
-
## Cloud (needs a key
|
|
42
|
+
## Cloud (needs a key)
|
|
44
43
|
```
|
|
45
44
|
zenrows batch create jobs.jsonl [--proxy-country cc] [--wait]
|
|
46
45
|
zenrows batch create jobs.jsonl --manual [--js-render] [--premium-proxy]
|
package/skills/extract/SKILL.md
CHANGED
|
@@ -12,7 +12,7 @@ access + extraction**, not generic LLM parsing.
|
|
|
12
12
|
|
|
13
13
|
## Methods
|
|
14
14
|
```
|
|
15
|
-
zenrows extract <url> # extract=auto (default); falls back to Autoparse if domain not
|
|
15
|
+
zenrows extract <url> # extract=auto (default); falls back to Autoparse if the domain is not prepared yet
|
|
16
16
|
zenrows extract <url> --autoparse # general-purpose Autoparse (any domain)
|
|
17
17
|
zenrows extract <url> --css '{"title":"h1","price":".price"}' # selector map
|
|
18
18
|
zenrows extract <url> --outputs emails,links # built-in output filters → JSON
|
|
@@ -21,8 +21,8 @@ zenrows extract <url> --validate # fail if not valid JSON
|
|
|
21
21
|
```
|
|
22
22
|
|
|
23
23
|
## When to use which
|
|
24
|
-
- **Extract** (`extract=auto`, default): richest fields on
|
|
25
|
-
- **Autoparse**: any website; also the automatic fallback when
|
|
24
|
+
- **Extract** (`extract=auto`, default): richest fields; works on any domain Zenrows has prepared.
|
|
25
|
+
- **Autoparse**: any website; also the automatic fallback when the domain is not prepared for Extract yet.
|
|
26
26
|
- **CSS**: you know the exact fields/selectors and want determinism.
|
|
27
27
|
- **Markdown/plaintext**: feed clean content to an LLM yourself.
|
|
28
28
|
|
package/skills/zenrows/SKILL.md
CHANGED
|
@@ -72,8 +72,8 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
|
|
|
72
72
|
| Primitive | Command | Status |
|
|
73
73
|
| --- | --- | --- |
|
|
74
74
|
| Protected Fetch | `zenrows fetch` | available (`GET /v1/`) |
|
|
75
|
-
| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` |
|
|
76
|
-
| Batch | `zenrows batch` |
|
|
75
|
+
| Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | available (same `/v1/`; extract=auto falls back to autoparse) |
|
|
76
|
+
| Batch | `zenrows batch` | available: runs cloud jobs (create/status/results); cheapest path at scale. Estimate specs locally with no key |
|
|
77
77
|
| Crawl | `zenrows crawl` | new — collect the URLs behind one start page, optionally with each page's HTML |
|
|
78
78
|
| Browser | `zenrows browser` | available (Browser Sessions REST API / MCP) |
|
|
79
79
|
| MCP | `zenrows mcp` | available (remote + local server) |
|
|
@@ -16,10 +16,10 @@ results carry `html` and `parsed` keys.
|
|
|
16
16
|
zenrows batch estimate jobs.example.jsonl # validate the spec + estimate credits
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
## Cloud (needs a key
|
|
19
|
+
## Cloud (needs a key)
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
The cloud subcommands run with any active API key and spend the same credits
|
|
22
|
+
as Fetch.
|
|
23
23
|
|
|
24
24
|
```bash
|
|
25
25
|
zenrows batch create jobs.example.jsonl --wait # submit + wait for the run to finish
|
|
@@ -33,5 +33,5 @@ Tasks use Adaptive Stealth Mode (`mode=auto`) by default. Job-level scrape
|
|
|
33
33
|
options come from flags (`--js-render`, `--premium-proxy`, `--proxy-country`,
|
|
34
34
|
`--output`, or `--manual` to turn Adaptive Stealth Mode off); per-line keys
|
|
35
35
|
override them for that URL.
|
|
36
|
-
|
|
37
|
-
|
|
36
|
+
Validate/estimate locally, or try `zenrows fetch` per URL on a small sample
|
|
37
|
+
first.
|
|
@@ -13,5 +13,5 @@ zenrows extract https://www.scrapingcourse.com/ecommerce/ --autoparse
|
|
|
13
13
|
zenrows extract <product-url> --css '{"title":"h1","price":".price"}' --validate
|
|
14
14
|
```
|
|
15
15
|
|
|
16
|
-
Requires `extract
|
|
16
|
+
Requires `extract`. Start on one page, confirm the fields, then
|
|
17
17
|
fan out `zenrows extract` across the rest of your URLs.
|
|
@@ -20,7 +20,7 @@ simple per-URL loop).
|
|
|
20
20
|
```
|
|
21
21
|
zenrows fetch https://competitor.example/products
|
|
22
22
|
```
|
|
23
|
-
2. **Validate extraction** on one page (extract
|
|
23
|
+
2. **Validate extraction** on one page (extract=auto falls back to autoparse on a domain not prepared yet).
|
|
24
24
|
```
|
|
25
25
|
zenrows extract https://competitor.example/p/123 --validate
|
|
26
26
|
```
|