@zenrows/cli 1.6.1 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -84,11 +84,12 @@ unclaimed Zenrows Free plan account**, stores the key in `.zenrows/secrets.json`
84
84
  (no secret — just the accountId, Free-period info, and a claim link).
85
85
 
86
86
  When you (or your agent) want to keep the usage and history, **claim** the
87
- account — this sets an email + password; the same key keeps working:
87
+ account — this confirms an email (Google, GitHub or a 6-digit code); the same
88
+ key keeps working:
88
89
 
89
90
  ```bash
90
91
  zenrows account status # show account + Free period + claim link
91
- zenrows account claim # open the claim URL in the browser
92
+ zenrows account claim # open the claim URL (fresh one if the stored link is near its 30-day expiry)
92
93
  zenrows usage # plan usage, credits & concurrency (does not count against your limit)
93
94
  ```
94
95
 
@@ -187,7 +188,10 @@ to open. `--max-items` and `--max-pages` cap the crawl (API default 100 each),
187
188
  and `--max-pages` bounds the cost, since each page is one fetch. The local policy `max_pages_per_run` caps `--max-pages` (default
188
189
  1000). `--output-format html` uses up the page budget: each kept page is one
189
190
  more fetch, and `--max-pages` counts it. `--include-patterns` /
190
- `--exclude-patterns` are repeatable substring filters on the URL. `get` and
191
+ `--exclude-patterns` are repeatable substring filters on the URL. A crawl whose
192
+ pages held nothing new to follow ends `completed` with no results and exits 0.
193
+ When patterns were set, it prints a note (under `--json`, in `note`) to check
194
+ them against the URLs the start page links to. `get` and
191
195
  `list` read one page and take `--cursor` and `--limit`. `results` follows every
192
196
  page (`--limit` is the page size); on a running crawl it returns the URLs kept
193
197
  so far, with `partial: true`. `download` reports the crawl's `status` (`null`
@@ -21,7 +21,8 @@ You do **not** need an API key in the plugin package.
21
21
  - **stdio** (`npx -y @zenrows/mcp`): on first cloud call the server can
22
22
  auto-create a **free, unclaimed** Zenrows Free plan account and persist the
23
23
  key locally. When the Free plan nears its limit (or the user asks), surface
24
- the **claim URL** so a human can claim the account (email + password). The
24
+ the **claim URL** so a human can claim the account (confirming an email
25
+ with Google, GitHub or a 6-digit code). Claim links expire after 30 days. The
25
26
  same key keeps working after claim. Never handle payment or upgrades —
26
27
  only show the claim link.
27
28
  - **Remote** (`https://mcp.zenrows.com/mcp`): the client runs **OAuth**
@@ -1,11 +1,14 @@
1
1
  /**
2
2
  * `zenrows account <status|claim>` — inspect the auto-provisioned Zenrows
3
3
  * account and surface the human-facing claim link. Never handles payment or
4
- * upgrades; it only reports status and opens the claim URL.
4
+ * upgrades; it only reports status and opens the claim URL, asking the API for
5
+ * a fresh one when the stored link is missing or close to its 30-day expiry.
5
6
  */
6
7
  import { fetchAccountStatus, readAccount, writeAccount } from "../../core/agent-account.js";
7
8
  import { authState, ENV_KEY, resolveApiKey } from "../../core/auth.js";
9
+ import { refreshClaimState, usableClaimUrl } from "../../core/claim-link.js";
8
10
  import { loadConfig } from "../../core/config.js";
11
+ import { ToolkitError } from "../../core/errors.js";
9
12
  import { log } from "../../core/logger.js";
10
13
  import { openUrl } from "../../core/open-url.js";
11
14
  import { fetchUsage } from "../../core/usage.js";
@@ -17,11 +20,39 @@ export const account = {
17
20
  async run(argv, ctx) {
18
21
  const { values, positionals } = parse(argv, { "no-open": { type: "boolean" } });
19
22
  const sub = positionals[0] ?? "status";
20
- let acct = readAccount();
21
23
  const auth = authState();
22
- // Claiming happens in the browser, out-of-band from the CLI. If we hold an
23
- // unclaimed record and a key, ask the backend whether it has since been
24
- // claimed and refresh account.json. Best-effort — stay on local state offline.
24
+ if (sub === "claim") {
25
+ const state = await refreshClaimState({ force: true });
26
+ if (state.status === "none") {
27
+ log.info("No agent-created account to claim. Use `zenrows login --api-key` if you already have a key.");
28
+ return 0;
29
+ }
30
+ if (state.status === "claimed") {
31
+ log.success("This account is already claimed. Nothing to do: the same API key keeps working.");
32
+ if (ctx.json)
33
+ log.out(JSON.stringify({ ok: true, claimed: true }, null, 2));
34
+ return 0;
35
+ }
36
+ if (!state.claimUrl) {
37
+ throw new ToolkitError({
38
+ code: "FETCH_FAILED",
39
+ message: "No valid claim link is available.",
40
+ likely_cause: auth.hasKey
41
+ ? "The stored claim link expired and a fresh one could not be requested."
42
+ : "The stored claim link expired, and no API key is configured to request a fresh one.",
43
+ next_action: "Check connectivity and run `zenrows account claim` again.",
44
+ suggested_commands: ["zenrows account claim"],
45
+ });
46
+ }
47
+ log.info("Claim this Zenrows account: open the link and confirm your email (Google, GitHub or a 6-digit code).");
48
+ log.info(state.claimUrl);
49
+ if (!values["no-open"])
50
+ await openUrl(state.claimUrl);
51
+ if (ctx.json)
52
+ log.out(JSON.stringify({ ok: true, claimUrl: state.claimUrl, refreshed: state.refreshed }, null, 2));
53
+ return 0;
54
+ }
55
+ let acct = readAccount();
25
56
  if (acct && acct.unclaimed && auth.hasKey) {
26
57
  const key = resolveApiKey().key;
27
58
  if (key) {
@@ -33,30 +64,9 @@ export const account = {
33
64
  }
34
65
  }
35
66
  catch {
36
- // endpoint unavailable / offline — keep the local record
37
67
  }
38
68
  }
39
69
  }
40
- if (sub === "claim") {
41
- if (!acct) {
42
- log.info("No agent-created account to claim. Use `zenrows login --api-key` if you already have a key.");
43
- return 0;
44
- }
45
- if (!acct.unclaimed) {
46
- log.success("This account is already claimed.");
47
- if (ctx.json)
48
- log.out(JSON.stringify({ ok: true, claimed: true }, null, 2));
49
- return 0;
50
- }
51
- log.info("Claim this Zenrows account (set your email + password):");
52
- log.info(acct.claimUrl);
53
- if (!values["no-open"])
54
- await openUrl(acct.claimUrl);
55
- if (ctx.json)
56
- log.out(JSON.stringify({ ok: true, claimUrl: acct.claimUrl }, null, 2));
57
- return 0;
58
- }
59
- // status — cycle dates from subscriptions/self/details
60
70
  let periodEndsAt;
61
71
  if (auth.hasKey) {
62
72
  const key = resolveApiKey().key;
@@ -86,7 +96,7 @@ export const account = {
86
96
  if (periodEndsAt)
87
97
  log.dim(`Billing period ends: ${periodEndsAt}`);
88
98
  if (acct.unclaimed)
89
- log.dim(`Claim it: ${acct.claimUrl}`);
99
+ log.dim(`Claim it: ${usableClaimUrl(acct) ?? "run `zenrows account claim` for a fresh link"}`);
90
100
  }
91
101
  else {
92
102
  if (auth.hasKey) {
@@ -20,6 +20,7 @@ import { createJob, downloadResults, getJob, listResults, rerunJob, stopJob, wai
20
20
  import { asNumber, asString, parse } from "../command.js";
21
21
  import { ToolkitError, isKeyCapReached, keyCapReached } from "../../core/errors.js";
22
22
  import { printError, writeOut } from "../output.js";
23
+ import { refreshClaimAdvice } from "../../core/claim-link.js";
23
24
  export const batch = {
24
25
  name: "batch",
25
26
  summary: "Run JSONL batch jobs on Zenrows Batch.",
@@ -74,7 +75,9 @@ export const batch = {
74
75
  });
75
76
  }
76
77
  }
77
- catch (err) {
78
+ catch (caught) {
79
+ // A quota error's claim advice is built from local state; check it is still current.
80
+ const err = await refreshClaimAdvice(caught);
78
81
  printError(err, ctx.json);
79
82
  return 1;
80
83
  }
@@ -23,6 +23,7 @@ import { newRunId, writeRun } from "../../core/artifacts.js";
23
23
  import { ToolkitError } from "../../core/errors.js";
24
24
  import { parse, asString, asNumber } from "../command.js";
25
25
  import { printError, writeOut } from "../output.js";
26
+ import { refreshClaimAdvice } from "../../core/claim-link.js";
26
27
  import { createSession, sessionCall, closeSession, decodeBinary, connectUrl, normalizeSelectValue, } from "../../core/browser-api.js";
27
28
  const VERBS = {
28
29
  navigate: { verb: "navigate", opts: { url: { type: "string" } }, required: ["url"], body: (v) => ({ url: asString(v.url) }) },
@@ -155,7 +156,9 @@ export const browser = {
155
156
  });
156
157
  }
157
158
  }
158
- catch (err) {
159
+ catch (caught) {
160
+ // A quota error's claim advice is built from local state; check it is still current.
161
+ const err = await refreshClaimAdvice(caught);
159
162
  printError(err, ctx.json);
160
163
  return err instanceof ToolkitError && err.code === "POLICY_BROWSER_DISABLED" ? 2 : 1;
161
164
  }
@@ -18,6 +18,7 @@ import { asNumber, asString, parse } from "../command.js";
18
18
  import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "../../core/errors.js";
19
19
  import { readAccount } from "../../core/agent-account.js";
20
20
  import { printError, writeOut, writeOutStream } from "../output.js";
21
+ import { refreshClaimAdvice, storedClaimHint } from "../../core/claim-link.js";
21
22
  /** The API's default for `max_pages` (and `max_items`) when the flag is unset. */
22
23
  const DEFAULT_MAX_PAGES = 100;
23
24
  /** Largest `depth` / `max_items` / `max_pages` the API takes. */
@@ -62,7 +63,8 @@ export const crawl = {
62
63
  " a crawl still running then is printed, not stopped",
63
64
  " --json print structured output: create/get/wait/stop print",
64
65
  " {ok, crawl_id, status, crawl}; get adds results and next_cursor;",
65
- " a wait that runs out adds note",
66
+ " a wait that runs out, or a completed crawl with patterns",
67
+ " and 0 results, adds note",
66
68
  ].join("\n"),
67
69
  async run(argv, ctx) {
68
70
  const [sub, ...rest] = argv;
@@ -94,7 +96,9 @@ export const crawl = {
94
96
  });
95
97
  }
96
98
  }
97
- catch (err) {
99
+ catch (caught) {
100
+ // A quota error's claim advice is built from local state; check it is still current.
101
+ const err = await refreshClaimAdvice(caught);
98
102
  printError(err, ctx.json);
99
103
  return 1;
100
104
  }
@@ -461,13 +465,13 @@ export function crawlFailure(c) {
461
465
  }
462
466
  if (code === "insufficient_credits") {
463
467
  const acct = readAccount();
464
- const quota = quotaExhausted(where, acct?.unclaimed ? acct.claimUrl : undefined, { status: null, detail }).toJSON();
468
+ const quota = quotaExhausted(where, storedClaimHint(acct), { status: null, detail }).toJSON();
465
469
  return new ToolkitError({ ...quota, ...ids, code: "CRAWL_INSUFFICIENT_CREDITS", suggested_commands: [...(quota.suggested_commands ?? []), getCmd] });
466
470
  }
467
471
  const why = [code, c.error?.detail].filter(Boolean).join(": ");
468
472
  const next = {
469
- seed_unreachable: "The start URL could not be fetched (the site may be blocking it). Check it with `zenrows fetch <url>`.",
470
- no_items_found: "Nothing matched. Loosen --include-patterns / --exclude-patterns, raise or drop --depth, or check the start page with `zenrows fetch <url>`.",
473
+ seed_unreachable: "The start URL could not be fetched or returned an empty page (the site may be blocking it). Check it with `zenrows fetch <url>`.",
474
+ no_items_found: "The crawl read the start page but found no data it could use, so it had nothing to follow. Start from a listing or product page.",
471
475
  };
472
476
  return new ToolkitError({
473
477
  code: "CRAWL_FAILED",
@@ -478,6 +482,18 @@ export function crawlFailure(c) {
478
482
  ...ids,
479
483
  });
480
484
  }
485
+ /**
486
+ * Set only when a crawl with patterns completed and kept nothing. The API ends a
487
+ * crawl whose pages held nothing new to follow `completed`, and does not count
488
+ * the links the patterns rejected, so the patterns are the first thing to check.
489
+ */
490
+ function noMatchNote(c) {
491
+ const crawl = c;
492
+ const patterns = [...(crawl.include_patterns ?? []), ...(crawl.exclude_patterns ?? [])];
493
+ if (crawl.status !== "completed" || crawl.coverage?.items_found !== 0 || !patterns.length)
494
+ return undefined;
495
+ return "The crawl completed with no results. Check --include-patterns / --exclude-patterns against the URLs the start page links to.";
496
+ }
481
497
  /**
482
498
  * Print a crawl (or a stop answer), structured under --json, and return the exit
483
499
  * code: 1 when it ended `failed`, or `stopped` because credits ran out or the
@@ -487,6 +503,7 @@ export function crawlFailure(c) {
487
503
  */
488
504
  function printCrawl(c, json, headline, extra = {}) {
489
505
  const err = crawlFailure(c);
506
+ const note = extra.note ?? noMatchNote(c);
490
507
  const crawlFields = { ...c };
491
508
  // A wait polls with one result to stay cheap; that page is not "the results".
492
509
  delete crawlFields.results;
@@ -498,6 +515,7 @@ function printCrawl(c, json, headline, extra = {}) {
498
515
  status: c.status,
499
516
  crawl: crawlFields,
500
517
  ...extra,
518
+ note,
501
519
  ...(err ? { error: err.toJSON() } : {}),
502
520
  }, null, 2));
503
521
  return err ? 1 : 0;
@@ -515,6 +533,8 @@ function printCrawl(c, json, headline, extra = {}) {
515
533
  }
516
534
  if (extra.note)
517
535
  log.dim(` next: zenrows crawl wait ${c.crawl_id}`);
536
+ else if (note)
537
+ log.warn(` ${note}`);
518
538
  if (err)
519
539
  printError(err, false);
520
540
  return err ? 1 : 0;
@@ -17,6 +17,7 @@ import { runExtract } from "../../adapters/extract.js";
17
17
  import { formatRequestCost } from "../../core/http.js";
18
18
  import { parse, asString, asNumber } from "../command.js";
19
19
  import { printError, writeOut } from "../output.js";
20
+ import { refreshClaimAdvice } from "../../core/claim-link.js";
20
21
  export const extract = {
21
22
  name: "extract",
22
23
  summary: "Turn a protected page into structured data (Extract / Autoparse / CSS / Markdown).",
@@ -175,7 +176,9 @@ export const extract = {
175
176
  await maybeNudgeClaim(config, {});
176
177
  return 0;
177
178
  }
178
- catch (err) {
179
+ catch (caught) {
180
+ // A quota error's claim advice is built from local state; check it is still current.
181
+ const err = await refreshClaimAdvice(caught);
179
182
  writeRun({
180
183
  runId,
181
184
  command: "zenrows extract",
@@ -16,6 +16,7 @@ import { runFetch } from "../../adapters/protected-fetch.js";
16
16
  import { formatRequestCost } from "../../core/http.js";
17
17
  import { parse, asString, asNumber } from "../command.js";
18
18
  import { printError, writeOut } from "../output.js";
19
+ import { refreshClaimAdvice } from "../../core/claim-link.js";
19
20
  export const fetch_ = {
20
21
  name: "fetch",
21
22
  summary: "Retrieve a protected page (Fetch).",
@@ -152,7 +153,9 @@ export const fetch_ = {
152
153
  await maybeNudgeClaim(config, {});
153
154
  return 0;
154
155
  }
155
- catch (err) {
156
+ catch (caught) {
157
+ // A quota error's claim advice is built from local state; check it is still current.
158
+ const err = await refreshClaimAdvice(caught);
156
159
  writeRun({
157
160
  runId,
158
161
  command: "zenrows fetch",
@@ -21,12 +21,13 @@ export const signup = {
21
21
  if (values.agent) {
22
22
  const account = readAccount();
23
23
  if (account) {
24
- log.info("An agent account already exists. Claim it: " + account.claimUrl);
24
+ log.info("An agent account already exists. Claim it with: zenrows account claim");
25
25
  return 0;
26
26
  }
27
27
  const res = await signupAgent();
28
28
  saveApiKey(res.apiKey);
29
- writeAccount({ accountId: res.accountId, unclaimed: true, claimUrl: res.claimUrl, createdAt: new Date().toISOString() });
29
+ const now = new Date().toISOString();
30
+ writeAccount({ accountId: res.accountId, unclaimed: true, claimUrl: res.claimUrl, createdAt: now, claimUrlIssuedAt: now });
30
31
  log.success("Created a Zenrows Free plan account.");
31
32
  log.dim(`Claim it anytime: ${res.claimUrl}`);
32
33
  if (ctx.json)
@@ -8,6 +8,7 @@
8
8
  */
9
9
  import { CLI_VERSION } from "../core/config.js";
10
10
  import { ToolkitError } from "../core/errors.js";
11
+ import { refreshClaimAdvice } from "../core/claim-link.js";
11
12
  import { log, ANSI, c } from "../core/logger.js";
12
13
  import { makeAssetCommand } from "./asset-command.js";
13
14
  import { printError } from "./output.js";
@@ -82,7 +83,7 @@ export async function main(rawArgv) {
82
83
  return await cmd.run(argv, ctx);
83
84
  }
84
85
  catch (err) {
85
- printError(err, json);
86
+ printError(await refreshClaimAdvice(err), json);
86
87
  // Any error exits 1 — the exit code doesn't encode error kind. Machine
87
88
  // consumers read the precise `error.code` from the --json output instead.
88
89
  return 1;
@@ -5,13 +5,13 @@
5
5
  * It never holds the API key (that lives in secrets.json, 0600) — only the
6
6
  * accountId, the human-facing claim URL, and claim metadata.
7
7
  */
8
- import { chmodSync, existsSync, unlinkSync } from "node:fs";
8
+ import { existsSync, unlinkSync } from "node:fs";
9
9
  import { attributionEnabled, getOrCreateTelemetryId, loadConfig, CLI_USER_AGENT, CLI_VERSION } from "./config.js";
10
10
  import { agentClientHeader } from "./agent-client.js";
11
11
  import { ToolkitError } from "./errors.js";
12
12
  import { AGENT_SIGNUP_API_URL, WELL_KNOWN_PROTECTED_RESOURCE } from "./open-url.js";
13
13
  import { detectClient } from "./provenance.js";
14
- import { findWorkspace, readJson, workspacePaths, writeJson } from "./workspace.js";
14
+ import { findWorkspace, readJson, workspacePaths, writePrivateJson } from "./workspace.js";
15
15
  /** Env var to override the signup endpoint (highest priority; for local testing). */
16
16
  export const SIGNUP_URL_ENV = "ZENROWS_AGENT_SIGNUP_URL";
17
17
  /** Env var to override the discovery base origin (for local testing). */
@@ -49,7 +49,9 @@ export async function discoverSignupUrl(projectRoot, opts = {}) {
49
49
  return null;
50
50
  const json = (await res.json());
51
51
  const endpoint = json?.agent_auth?.signup_endpoint;
52
- if (typeof endpoint === "string" && endpoint.trim())
52
+ // The key is later sent to URLs derived from this one, so only a Zenrows
53
+ // https endpoint is accepted from a document fetched over the network.
54
+ if (typeof endpoint === "string" && isZenrowsHttpsUrl(endpoint.trim()))
53
55
  return endpoint.trim();
54
56
  return null;
55
57
  }
@@ -89,7 +91,7 @@ export async function signupCandidates(projectRoot, opts = {}) {
89
91
  export async function resolveSignupUrl(projectRoot, opts = {}) {
90
92
  return (await signupCandidates(projectRoot, opts))[0];
91
93
  }
92
- function accountPath(projectRoot) {
94
+ export function accountPath(projectRoot) {
93
95
  const ws = projectRoot ? workspacePaths(projectRoot) : (findWorkspace() ?? workspacePaths());
94
96
  return ws.account;
95
97
  }
@@ -98,14 +100,8 @@ export function readAccount(projectRoot) {
98
100
  }
99
101
  export function writeAccount(acct, projectRoot) {
100
102
  const file = accountPath(projectRoot);
101
- writeJson(file, acct);
102
- try {
103
- // account.json holds the single-use claim token — restrict to owner-only.
104
- chmodSync(file, 0o600);
105
- }
106
- catch {
107
- // best-effort on platforms without POSIX permissions
108
- }
103
+ // account.json holds the single-use claim token: owner-only from creation.
104
+ writePrivateJson(file, acct);
109
105
  }
110
106
  /** Remove local agent account metadata (used by `zenrows logout`). */
111
107
  export function clearAccount(projectRoot) {
@@ -201,7 +197,108 @@ function signupError(status, body) {
201
197
  * `signupUrl` override applies here too (local/staging testing).
202
198
  */
203
199
  export async function resolveAccountUrl(projectRoot, opts = {}) {
204
- return (await resolveSignupUrl(projectRoot, opts)).replace(/\/signup\/?$/, "/account");
200
+ return siblingOfSignup(await resolveSignupUrl(projectRoot, opts), "account");
201
+ }
202
+ /**
203
+ * `…/agent/signup` → `…/agent/<name>`. A signup URL that does not end in
204
+ * `/signup` gives no safe sibling, so the built-in endpoint's sibling is used
205
+ * rather than sending the key somewhere unexpected.
206
+ */
207
+ export function siblingOfSignup(signupUrl, name) {
208
+ const re = /\/signup\/?$/;
209
+ // These two endpoints receive the API key. A signup URL from the environment
210
+ // or a project's config (which a cloned repository could ship) is only
211
+ // trusted when it is Zenrows or this machine; anything else gets the
212
+ // built-in endpoint.
213
+ const base = re.test(signupUrl) && isAgentKeyDestination(signupUrl) ? signupUrl : AGENT_SIGNUP_API_URL;
214
+ return base.replace(re, `/${name}`);
215
+ }
216
+ /**
217
+ * Where the CLI may send the API key for the agent account endpoints: a
218
+ * Zenrows https URL, or http(s) on loopback (local testing). No userinfo.
219
+ */
220
+ export function isAgentKeyDestination(value) {
221
+ if (isZenrowsHttpsUrl(value))
222
+ return true;
223
+ try {
224
+ const u = new URL(value);
225
+ const host = u.hostname.toLowerCase();
226
+ return ((u.protocol === "https:" || u.protocol === "http:") &&
227
+ u.username === "" &&
228
+ u.password === "" &&
229
+ (host === "127.0.0.1" || host === "localhost" || host === "[::1]"));
230
+ }
231
+ catch {
232
+ return false;
233
+ }
234
+ }
235
+ /** True for an https URL on zenrows.com or a subdomain, with no userinfo. */
236
+ export function isZenrowsHttpsUrl(value) {
237
+ if (typeof value !== "string")
238
+ return false;
239
+ try {
240
+ const u = new URL(value);
241
+ const host = u.hostname.toLowerCase();
242
+ return (u.protocol === "https:" &&
243
+ u.username === "" &&
244
+ u.password === "" &&
245
+ (host === "zenrows.com" || host.endsWith(".zenrows.com")));
246
+ }
247
+ catch {
248
+ return false;
249
+ }
250
+ }
251
+ /** The fresh-claim-link endpoint, derived from the signup URL like the status one. */
252
+ export async function resolveClaimUrl(projectRoot, opts = {}) {
253
+ return siblingOfSignup(await resolveSignupUrl(projectRoot, opts), "claim");
254
+ }
255
+ /**
256
+ * Mint a fresh claim link for the account behind `apiKey`. Each call
257
+ * invalidates the previous link and the endpoint is rate-limited per key, so
258
+ * callers only ask when the link they hold is missing or near expiry.
259
+ * A 409 `already_claimed` means a person owns the account. Throws ToolkitError
260
+ * on any other failure.
261
+ */
262
+ export async function requestClaimLink(apiKey, opts = {}) {
263
+ const url = opts.url ?? (await resolveClaimUrl(opts.projectRoot, { fetchImpl: opts.fetchImpl }));
264
+ const doFetch = opts.fetchImpl ?? fetch;
265
+ const res = await doFetch(url, {
266
+ method: "POST",
267
+ headers: { "X-API-Key": apiKey, Accept: "application/json", "User-Agent": CLI_USER_AGENT, ...agentClientHeader() },
268
+ });
269
+ const body = await res.text();
270
+ if (res.status === 200) {
271
+ let json = null;
272
+ try {
273
+ json = JSON.parse(body);
274
+ }
275
+ catch {
276
+ // not JSON: reported below
277
+ }
278
+ if (!json || typeof json.claimUrl !== "string") {
279
+ throw new ToolkitError({
280
+ code: "FETCH_FAILED",
281
+ message: "The claim-link endpoint answered HTTP 200 without a claim link (unexpected response body).",
282
+ likely_cause: body.slice(0, 240),
283
+ next_action: "Retry later.",
284
+ });
285
+ }
286
+ return {
287
+ status: "unclaimed",
288
+ claimUrl: json.claimUrl,
289
+ ...(typeof json.expiresAt === "string" ? { expiresAt: json.expiresAt } : {}),
290
+ };
291
+ }
292
+ if (res.status === 409 && /already_claimed/.test(body))
293
+ return { status: "claimed" };
294
+ if (res.status === 409 && /not_agent_account/.test(body))
295
+ return { status: "not_agent" };
296
+ throw new ToolkitError({
297
+ code: res.status === 401 || res.status === 403 ? "AUTH_INVALID" : "FETCH_FAILED",
298
+ message: `Claim link request failed (HTTP ${res.status}).`,
299
+ likely_cause: body.slice(0, 240),
300
+ next_action: res.status === 429 ? "Too many claim links requested for this key; retry in an hour." : "Retry later.",
301
+ });
205
302
  }
206
303
  /**
207
304
  * Ask the backend whether the account behind `apiKey` has been claimed
@@ -209,7 +306,7 @@ export async function resolveAccountUrl(projectRoot, opts = {}) {
209
306
  * `X-API-Key` header. Throws ToolkitError on failure.
210
307
  */
211
308
  export async function fetchAccountStatus(apiKey, opts = {}) {
212
- const url = opts.url ?? (await resolveAccountUrl());
309
+ const url = opts.url ?? (await resolveAccountUrl(opts.projectRoot, { fetchImpl: opts.fetchImpl }));
213
310
  const doFetch = opts.fetchImpl ?? fetch;
214
311
  const res = await doFetch(url, {
215
312
  method: "GET",
@@ -4,11 +4,11 @@
4
4
  * Resolution order: explicit secrets file → ZENROWS_API_KEY env var.
5
5
  * The key is never printed in full and never written into run artifacts.
6
6
  */
7
- import { chmodSync, existsSync } from "node:fs";
7
+ import { existsSync } from "node:fs";
8
8
  import { ToolkitError } from "./errors.js";
9
9
  import { registerSecret } from "./logger.js";
10
10
  import { mask } from "./redact.js";
11
- import { findWorkspace, readJson, workspacePaths, writeJson } from "./workspace.js";
11
+ import { findWorkspace, readJson, workspacePaths, writePrivateJson } from "./workspace.js";
12
12
  export const ENV_KEY = "ZENROWS_API_KEY";
13
13
  function secretsPath(projectRoot) {
14
14
  const ws = projectRoot ? workspacePaths(projectRoot) : (findWorkspace() ?? workspacePaths());
@@ -49,20 +49,14 @@ export function saveApiKey(apiKey, projectRoot) {
49
49
  });
50
50
  }
51
51
  const file = secretsPath(projectRoot);
52
- writeJson(file, { apiKey: trimmed });
53
- try {
54
- chmodSync(file, 0o600);
55
- }
56
- catch {
57
- // best-effort on platforms without POSIX permissions
58
- }
52
+ writePrivateJson(file, { apiKey: trimmed });
59
53
  registerSecret(trimmed);
60
54
  }
61
55
  export function clearApiKey(projectRoot) {
62
56
  const file = secretsPath(projectRoot);
63
57
  if (!existsSync(file))
64
58
  return false;
65
- writeJson(file, {});
59
+ writePrivateJson(file, {});
66
60
  return true;
67
61
  }
68
62
  /** Throw AUTH_MISSING if no key is configured. */
@@ -17,6 +17,7 @@
17
17
  import { mkdirSync, writeFileSync } from "node:fs";
18
18
  import { join } from "node:path";
19
19
  import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
20
+ import { storedClaimHint } from "./claim-link.js";
20
21
  import { readAccount } from "./agent-account.js";
21
22
  import { registerSecret } from "./logger.js";
22
23
  import { CLI_USER_AGENT } from "./config.js";
@@ -163,7 +164,7 @@ function problemToError(status, body, method, path) {
163
164
  // a claim link (unclaimed Free plan) or dashboard/upgrade pointer, not a
164
165
  // generic "fix the problem and retry".
165
166
  const acct = readAccount();
166
- return quotaExhausted(`${method} ${path}`, acct?.unclaimed ? acct.claimUrl : undefined, {
167
+ return quotaExhausted(`${method} ${path}`, storedClaimHint(acct), {
167
168
  status: 402,
168
169
  detail: problem.detail || problem.title || undefined,
169
170
  });
@@ -12,6 +12,7 @@
12
12
  * them (`closeSession`) — the `browser run` command does so in a `finally`.
13
13
  */
14
14
  import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
15
+ import { storedClaimHint } from "./claim-link.js";
15
16
  import { readAccount } from "./agent-account.js";
16
17
  import { registerSecret } from "./logger.js";
17
18
  import { CLI_USER_AGENT } from "./config.js";
@@ -116,7 +117,7 @@ export function browserProblemToError(status, body, method, path) {
116
117
  if (status === 402) {
117
118
  // Out of credits — same exhausted state as the scraper/batch 402.
118
119
  const acct = readAccount();
119
- return quotaExhausted(`${method} ${path}`, acct?.unclaimed ? acct.claimUrl : undefined, {
120
+ return quotaExhausted(`${method} ${path}`, storedClaimHint(acct), {
120
121
  status: 402,
121
122
  detail: parsed.error || parsed.detail || parsed.title || undefined,
122
123
  });
@@ -0,0 +1,237 @@
1
+ import { randomBytes } from "node:crypto";
2
+ import { linkSync, readFileSync, renameSync, statSync, unlinkSync, writeFileSync } from "node:fs";
3
+ import { accountPath, fetchAccountStatus, isZenrowsHttpsUrl, readAccount, requestClaimLink, writeAccount } from "./agent-account.js";
4
+ import { resolveApiKey } from "./auth.js";
5
+ import { CLAIM_LINK_REFRESH, CLAIM_NEXT_ACTION_PREFIX, ToolkitError, quotaNextAction } from "./errors.js";
6
+ const DAY_MS = 24 * 60 * 60 * 1000;
7
+ /** The backend's claim-link lifetime. A link this old no longer works. */
8
+ export const CLAIM_LINK_TTL_MS = 30 * DAY_MS;
9
+ /** Ask for a fresh link once the held one is this old, before it lapses. */
10
+ export const CLAIM_LINK_REFRESH_AGE_MS = 25 * DAY_MS;
11
+ /** On error paths, check claim status at most this often. */
12
+ export const CLAIM_CHECK_THROTTLE_MS = 6 * 60 * 60 * 1000;
13
+ /** Each claim request gives up after this long, so an error is never held up for long. */
14
+ const CLAIM_REQUEST_TIMEOUT_MS = 5000;
15
+ /** Only an https link on zenrows.com (or a subdomain), with no userinfo, is shown or opened. */
16
+ export function isTrustedClaimUrl(value) {
17
+ return isZenrowsHttpsUrl(value);
18
+ }
19
+ /** Age of the held link in ms, or Infinity when its issue time is unknown. */
20
+ function linkAgeMs(acct, now) {
21
+ const issued = new Date(acct.claimUrlIssuedAt ?? acct.createdAt).getTime();
22
+ return Number.isNaN(issued) ? Infinity : now.getTime() - issued;
23
+ }
24
+ /** The held link if it is trusted and not yet expired, else undefined. No network. */
25
+ export function usableClaimUrl(acct, now = new Date()) {
26
+ if (!isTrustedClaimUrl(acct.claimUrl))
27
+ return undefined;
28
+ return linkAgeMs(acct, now) < CLAIM_LINK_TTL_MS ? acct.claimUrl : undefined;
29
+ }
30
+ /** The claim hint for an error from local state alone. No network. */
31
+ export function storedClaimHint(acct, now = new Date()) {
32
+ if (!acct?.unclaimed)
33
+ return undefined;
34
+ return usableClaimUrl(acct, now) ?? CLAIM_LINK_REFRESH;
35
+ }
36
+ /**
37
+ * Bring the stored claim state up to date, as far as the network allows, and
38
+ * report it. Steps: ask whether the account was claimed; if not, and the held
39
+ * link is missing, untrusted or 25+ days old, mint a fresh one and store it.
40
+ * Network failures are swallowed: the held link is still offered while it is
41
+ * under 30 days old. Unless `force` is set, the network is skipped entirely
42
+ * when the last check was under 6 hours ago. Throws only for a forced call
43
+ * whose fresh-link request failed with no usable link to fall back on.
44
+ */
45
+ export async function refreshClaimState(opts = {}) {
46
+ const { projectRoot } = opts;
47
+ const now = opts.now ?? new Date();
48
+ const first = readAccount(projectRoot);
49
+ if (!first)
50
+ return { status: "none" };
51
+ if (!first.unclaimed)
52
+ return { status: "claimed" };
53
+ if (!opts.force && throttled(first, now))
54
+ return offlineState(first, now);
55
+ const key = opts.apiKey ?? resolveApiKey(projectRoot).key;
56
+ if (!key)
57
+ return offlineState(first, now);
58
+ // Several `zenrows` processes can hit a 402 at once. Only the one holding the
59
+ // lock talks to the claim endpoints; the others use what is stored.
60
+ const release = acquireClaimLock(projectRoot);
61
+ if (!release)
62
+ return offlineState(readAccount(projectRoot) ?? first, now);
63
+ try {
64
+ return await refreshLocked(key, opts, now);
65
+ }
66
+ finally {
67
+ release();
68
+ }
69
+ }
70
+ async function refreshLocked(key, opts, now) {
71
+ const { projectRoot } = opts;
72
+ // Re-read under the lock: another process may have just finished a refresh.
73
+ let acct = readAccount(projectRoot);
74
+ if (!acct)
75
+ return { status: "none" };
76
+ if (!acct.unclaimed)
77
+ return { status: "claimed" };
78
+ if (!opts.force && throttled(acct, now))
79
+ return offlineState(acct, now);
80
+ // Take the 6-hour slot before any network call, so a failing backend is not
81
+ // retried by every command that follows.
82
+ acct = { ...acct, lastClaimCheckAt: now.toISOString() };
83
+ writeAccount(acct, projectRoot);
84
+ const base = opts.fetchImpl ?? fetch;
85
+ const fetchImpl = ((url, init) => base(url, { ...init, signal: AbortSignal.timeout(CLAIM_REQUEST_TIMEOUT_MS) }));
86
+ try {
87
+ const remote = await fetchAccountStatus(key, { url: opts.accountUrl, fetchImpl, projectRoot });
88
+ if (remote.claimed === true)
89
+ return markClaimed(acct, projectRoot);
90
+ }
91
+ catch {
92
+ // Status unknown: carry on and judge the held link on its own.
93
+ }
94
+ // Re-read before minting: minting replaces the link everyone else holds.
95
+ acct = readAccount(projectRoot) ?? acct;
96
+ if (!acct.unclaimed)
97
+ return { status: "claimed" };
98
+ if (!needsNewLink(acct, now))
99
+ return offlineState(acct, now);
100
+ let fresh;
101
+ try {
102
+ fresh = await requestClaimLink(key, { url: opts.claimUrl, fetchImpl, projectRoot });
103
+ }
104
+ catch (err) {
105
+ if (opts.force && !usableClaimUrl(acct, now)) {
106
+ if (err instanceof ToolkitError)
107
+ throw err;
108
+ throw new ToolkitError({
109
+ code: "BACKEND_UNAVAILABLE",
110
+ message: "Could not reach Zenrows to get a fresh claim link.",
111
+ likely_cause: err instanceof Error ? err.message : String(err),
112
+ next_action: "The stored claim link has expired. Check connectivity and run `zenrows account claim` again.",
113
+ suggested_commands: ["zenrows account claim"],
114
+ });
115
+ }
116
+ return offlineState(acct, now);
117
+ }
118
+ // Both 409s are final: a person owns the account, so there is nothing to claim.
119
+ if (fresh.status === "claimed" || fresh.status === "not_agent")
120
+ return markClaimed(acct, projectRoot);
121
+ if (!isTrustedClaimUrl(fresh.claimUrl)) {
122
+ if (opts.force && !usableClaimUrl(acct, now)) {
123
+ throw new ToolkitError({
124
+ code: "FETCH_FAILED",
125
+ message: "The claim-link endpoint returned a URL outside zenrows.com; it was not used.",
126
+ likely_cause: "The signup/claim endpoint override points at an unexpected server.",
127
+ next_action: "Check ZENROWS_AGENT_SIGNUP_URL / `signupUrl` in the zenrows config, then retry.",
128
+ });
129
+ }
130
+ return offlineState(acct, now);
131
+ }
132
+ const latest = readAccount(projectRoot) ?? acct;
133
+ writeAccount({ ...latest, claimUrl: fresh.claimUrl, claimUrlIssuedAt: now.toISOString() }, projectRoot);
134
+ return { status: "unclaimed", claimUrl: fresh.claimUrl, refreshed: true };
135
+ }
136
+ function throttled(acct, now) {
137
+ if (!acct.lastClaimCheckAt)
138
+ return false;
139
+ const last = new Date(acct.lastClaimCheckAt).getTime();
140
+ return !Number.isNaN(last) && now.getTime() - last < CLAIM_CHECK_THROTTLE_MS;
141
+ }
142
+ function needsNewLink(acct, now) {
143
+ return !isTrustedClaimUrl(acct.claimUrl) || linkAgeMs(acct, now) >= CLAIM_LINK_REFRESH_AGE_MS;
144
+ }
145
+ function offlineState(acct, now) {
146
+ if (!acct.unclaimed)
147
+ return { status: "claimed" };
148
+ return { status: "unclaimed", claimUrl: usableClaimUrl(acct, now), refreshed: false };
149
+ }
150
+ /** A lock older than this is from a process that died mid-refresh. */
151
+ const CLAIM_LOCK_STALE_MS = 60 * 1000;
152
+ /**
153
+ * Create `account.json.lock` exclusively, holding a random token. Returns a
154
+ * release function, or null when another live process holds the lock.
155
+ *
156
+ * A lock older than CLAIM_LOCK_STALE_MS is taken over by renaming it aside
157
+ * first (only one process can rename it away). If what was renamed turns out
158
+ * to be live, it is put back and this process gives up. Release deletes the
159
+ * lock only while it still holds this process's token.
160
+ */
161
+ function acquireClaimLock(projectRoot) {
162
+ const lock = `${accountPath(projectRoot)}.lock`;
163
+ const token = randomBytes(16).toString("hex");
164
+ const release = () => {
165
+ try {
166
+ if (readFileSync(lock, "utf8") === token)
167
+ unlinkSync(lock);
168
+ }
169
+ catch {
170
+ }
171
+ };
172
+ for (let attempt = 0; attempt < 2; attempt++) {
173
+ try {
174
+ writeFileSync(lock, token, { flag: "wx", mode: 0o600 });
175
+ return readFileSync(lock, "utf8") === token ? release : null;
176
+ }
177
+ catch (err) {
178
+ if (err.code !== "EEXIST")
179
+ return null;
180
+ try {
181
+ if (Date.now() - statSync(lock).mtimeMs < CLAIM_LOCK_STALE_MS)
182
+ return null;
183
+ const aside = `${lock}.${token}.stale`;
184
+ renameSync(lock, aside);
185
+ if (Date.now() - statSync(aside).mtimeMs < CLAIM_LOCK_STALE_MS) {
186
+ // Another process replaced the stale lock between our checks: restore it.
187
+ try {
188
+ linkSync(aside, lock);
189
+ }
190
+ catch {
191
+ }
192
+ unlinkSync(aside);
193
+ return null;
194
+ }
195
+ unlinkSync(aside);
196
+ }
197
+ catch {
198
+ return null;
199
+ }
200
+ }
201
+ }
202
+ return null;
203
+ }
204
+ /** Test-only: the lock path, so tests can simulate other processes. */
205
+ export function _claimLockPath(projectRoot) {
206
+ return `${accountPath(projectRoot)}.lock`;
207
+ }
208
+ function markClaimed(acct, projectRoot) {
209
+ writeAccount({ ...acct, unclaimed: false }, projectRoot);
210
+ return { status: "claimed" };
211
+ }
212
+ /** The error hint for a refreshed state. */
213
+ export function claimHintFor(state) {
214
+ if (state.status !== "unclaimed")
215
+ return undefined;
216
+ return state.claimUrl ?? CLAIM_LINK_REFRESH;
217
+ }
218
+ /**
219
+ * Bring a quota error's claim advice up to date before it is shown. Errors are
220
+ * built synchronously from `account.json`; this runs the throttled refresh and
221
+ * rewrites `next_action` if the account was claimed or the link changed. Never
222
+ * throws: on any failure the error is returned as it was.
223
+ */
224
+ export async function refreshClaimAdvice(err, opts = {}) {
225
+ try {
226
+ if (!(err instanceof ToolkitError) || !err.next_action.startsWith(CLAIM_NEXT_ACTION_PREFIX))
227
+ return err;
228
+ const state = await refreshClaimState({ ...opts, force: false });
229
+ const next = quotaNextAction(claimHintFor(state));
230
+ if (next === err.next_action)
231
+ return err;
232
+ return new ToolkitError({ ...err.toJSON(), next_action: next });
233
+ }
234
+ catch {
235
+ return err;
236
+ }
237
+ }
@@ -14,7 +14,7 @@ export const CONFIG_VERSION = "0.1.0";
14
14
  * entrypoint) so core modules — telemetry, signup provenance — can read it
15
15
  * without an import cycle. `VERSION` in `cli/index.ts` re-exports this.
16
16
  */
17
- export const CLI_VERSION = "1.6.1"; // x-release-please-version
17
+ export const CLI_VERSION = "1.7.0"; // x-release-please-version
18
18
  /** The User-Agent on every request to a Zenrows API. The gateway reads the version from it. */
19
19
  export const CLI_USER_AGENT = `zenrows-cli/${CLI_VERSION}`;
20
20
  /** Env var to override the Fetch and Extract API base (local/staging testing). */
@@ -15,6 +15,7 @@
15
15
  */
16
16
  import { Readable } from "node:stream";
17
17
  import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
18
+ import { storedClaimHint } from "./claim-link.js";
18
19
  import { readAccount } from "./agent-account.js";
19
20
  import { registerSecret } from "./logger.js";
20
21
  import { CLI_USER_AGENT, loadConfig } from "./config.js";
@@ -245,7 +246,7 @@ function classifyProblem(status, problem, body, request, serverCode, retryAfter)
245
246
  return new ToolkitError({ ...keyCapReached(request, { status, detail }).toJSON(), code: "CRAWL_KEY_CAP_REACHED" });
246
247
  }
247
248
  const acct = readAccount();
248
- const quota = quotaExhausted(request, acct?.unclaimed ? acct.claimUrl : undefined, { status, detail });
249
+ const quota = quotaExhausted(request, storedClaimHint(acct), { status, detail });
249
250
  return new ToolkitError({ ...quota.toJSON(), code: "CRAWL_INSUFFICIENT_CREDITS" });
250
251
  }
251
252
  if (status === 409) {
@@ -15,6 +15,7 @@ export async function ensureApiKey(policy, opts = {}) {
15
15
  claimUrl: res.claimUrl,
16
16
  createdAt: new Date().toISOString(),
17
17
  };
18
+ account.claimUrlIssuedAt = account.createdAt;
18
19
  writeAccount(account, opts.projectRoot);
19
20
  opts.onProvision?.(account);
20
21
  return res.apiKey;
@@ -49,26 +49,36 @@ export class ToolkitError extends Error {
49
49
  * credits stopped: its status call succeeded, so there is no HTTP error to cite,
50
50
  * only `url` (the crawl).
51
51
  */
52
- export function quotaExhausted(url, claimUrl, opts = {}) {
53
- // Say that the allowance comes back. Without it this reads as a permanent paywall,
54
- // which is how the API's own AUTH004 text reads ("Purchase a new subscription to
55
- // continue") and why exhausted clients retry-loop instead of waiting or upgrading —
56
- // one account spent seven days at ~3 req/s against this wall. `zenrows usage` prints
57
- // the exact `period_ends_at`, so the date is one command away rather than guessed here.
58
- const renewLine = "Credits renew at the end of the billing period — run `zenrows usage` for the date.";
59
- const claimLine = claimUrl
60
- ? `You are on the Zenrows Free plan. Claim your account to keep your usage and add credits: ${claimUrl}. ${renewLine}`
61
- : `You are out of Zenrows credits. ${renewLine} To carry on now, add a credit pack (${BILLING_TOPUP_URL} opens the purchase directly) or upgrade your plan: ${PLANS_URL}`;
52
+ export function quotaExhausted(url, claim, opts = {}) {
62
53
  const detail = opts.detail ? `${opts.detail.replace(/\.\s*$/, "")}. ` : "";
63
54
  const where = opts.status === null ? `Stopped ${url}` : `HTTP ${opts.status ?? 429} for ${url}`;
64
55
  return new ToolkitError({
65
56
  code: "POLICY_MAX_CREDITS_EXCEEDED",
66
57
  message: "Zenrows request quota exhausted.",
67
58
  likely_cause: `${detail}${where}`,
68
- next_action: claimLine,
59
+ next_action: quotaNextAction(claim),
69
60
  suggested_commands: ["zenrows usage"],
70
61
  });
71
62
  }
63
+ export const CLAIM_LINK_REFRESH = Symbol("claim-link-refresh");
64
+ /** Start of `next_action` on a quota error for an unclaimed account. */
65
+ export const CLAIM_NEXT_ACTION_PREFIX = "You are on the Zenrows Free plan.";
66
+ /** The `next_action` of a quota error for the given claim state. */
67
+ export function quotaNextAction(claim) {
68
+ // Say that the allowance comes back. Without it this reads as a permanent paywall,
69
+ // which is how the API's own AUTH004 text reads ("Purchase a new subscription to
70
+ // continue") and why exhausted clients retry-loop instead of waiting or upgrading.
71
+ // `zenrows usage` prints the exact `period_ends_at`, so the date is one command away
72
+ // rather than guessed here.
73
+ const renewLine = "Credits renew at the end of the billing period — run `zenrows usage` for the date.";
74
+ if (claim === CLAIM_LINK_REFRESH) {
75
+ return `${CLAIM_NEXT_ACTION_PREFIX} Claim your account to keep your usage and add credits: run \`zenrows account claim\` for a fresh claim link. ${renewLine}`;
76
+ }
77
+ if (claim) {
78
+ return `${CLAIM_NEXT_ACTION_PREFIX} Claim your account to keep your usage and add credits: ${claim}. ${renewLine}`;
79
+ }
80
+ return `You are out of Zenrows credits. ${renewLine} To carry on now, add a credit pack (${BILLING_TOPUP_URL} opens the purchase directly) or upgrade your plan: ${PLANS_URL}`;
81
+ }
72
82
  /** Where an account manages its API keys and their credit caps. */
73
83
  export const API_KEYS_SETTINGS_URL = "https://app.zenrows.com/settings/api-keys";
74
84
  /**
@@ -6,6 +6,7 @@
6
6
  * from any logged URL.
7
7
  */
8
8
  import { ToolkitError, isKeyCapReached, keyCapReached, quotaExhausted } from "./errors.js";
9
+ import { storedClaimHint, usableClaimUrl } from "./claim-link.js";
9
10
  import { readAccount } from "./agent-account.js";
10
11
  import { ENV_KEY, resolveApiKey } from "./auth.js";
11
12
  import { registerSecret } from "./logger.js";
@@ -143,7 +144,7 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
143
144
  message: "The auto-provisioned Free plan key was rejected by the Zenrows API.",
144
145
  likely_cause: `${cause}. The signup endpoint issued this key, but the scraping API did not accept it.`,
145
146
  next_action: "If you are testing against a local/staging backend, ensure `apiBase` (zenrows config) points at the same environment that issued the key. Otherwise re-provision with `zenrows signup --agent`, or claim the account: " +
146
- acct.claimUrl,
147
+ (usableClaimUrl(acct) ?? "run `zenrows account claim` for a link"),
147
148
  suggested_commands: ["zenrows status", "zenrows signup --agent"],
148
149
  });
149
150
  }
@@ -184,7 +185,7 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
184
185
  // out of credits. This is NOT scraped content — surface it as a credits
185
186
  // error (claim link for an unclaimed Free plan, dashboard/upgrade otherwise).
186
187
  const acct = readAccount();
187
- throw quotaExhausted(redacted, acct?.unclaimed ? acct.claimUrl : undefined, {
188
+ throw quotaExhausted(redacted, storedClaimHint(acct), {
188
189
  status: 402,
189
190
  detail: zrErrorDetail(body) ?? undefined,
190
191
  });
@@ -196,7 +197,7 @@ export async function scrape(apiBase, apiKey, params, opts = {}) {
196
197
  // limit; otherwise advise a brief retry.
197
198
  if (isQuotaError(body)) {
198
199
  const acct = readAccount();
199
- throw quotaExhausted(redacted, acct?.unclaimed ? acct.claimUrl : undefined);
200
+ throw quotaExhausted(redacted, storedClaimHint(acct));
200
201
  }
201
202
  const detail = zrErrorDetail(body) ?? snippet(body);
202
203
  throw new ToolkitError({
@@ -1,5 +1,6 @@
1
1
  import { readAccount, writeAccount } from "./agent-account.js";
2
2
  import { resolveApiKey } from "./auth.js";
3
+ import { usableClaimUrl } from "./claim-link.js";
3
4
  import { log } from "./logger.js";
4
5
  import { fetchUsage } from "./usage.js";
5
6
  /** How high usage must be (percent) before we nudge. */
@@ -15,7 +16,8 @@ export function nudgeMessage(account, usagePercent, _now) {
15
16
  if (!account.unclaimed)
16
17
  return null;
17
18
  if (typeof usagePercent === "number" && usagePercent >= USAGE_NUDGE_PERCENT) {
18
- return `You've used ${usagePercent}% of your Zenrows Free plan — claim your account to keep your usage & history: ${account.claimUrl}`;
19
+ const link = usableClaimUrl(account, _now) ?? "run `zenrows account claim` for a claim link";
20
+ return `You've used ${usagePercent}% of your Zenrows Free plan — claim your account to keep your usage & history: ${link}`;
19
21
  }
20
22
  return null;
21
23
  }
@@ -10,17 +10,25 @@
10
10
  function hasPrefix(env, prefix) {
11
11
  return Object.keys(env).some((k) => k.startsWith(prefix));
12
12
  }
13
+ /**
14
+ * Variables the agent itself sets in the shells it runs commands in, so they
15
+ * mean the agent is driving this command (the same signals as agent-client.ts).
16
+ * Variables that only show the agent is installed or configured (CODEX_HOME,
17
+ * ANTHROPIC_API_KEY, OPENAI_API_KEY, other CLAUDE_* settings) are not used.
18
+ */
19
+ const CLAUDE_VARS = ["CLAUDECODE", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_CODE_CHILD_SESSION"];
20
+ const CODEX_VARS = ["CODEX_SANDBOX", "CODEX_SANDBOX_NETWORK_DISABLED", "CODEX_THREAD_ID"];
13
21
  /** Detect the host agent brand + runtime, best-effort, from env vars. */
14
22
  export function detectClient(env = process.env) {
15
23
  const term = env.TERM_PROGRAM;
16
24
  let client = "unknown";
17
- if (env.CLAUDECODE || hasPrefix(env, "CLAUDE_CODE") || hasPrefix(env, "CLAUDE_")) {
25
+ if (CLAUDE_VARS.some((v) => env[v])) {
18
26
  client = "claude-code";
19
27
  }
20
28
  else if (env.CURSOR_TRACE_ID || hasPrefix(env, "CURSOR_") || term === "cursor") {
21
29
  client = "cursor";
22
30
  }
23
- else if (hasPrefix(env, "OPENAI_") || hasPrefix(env, "CODEX_")) {
31
+ else if (CODEX_VARS.some((v) => env[v])) {
24
32
  client = "openai-codex";
25
33
  }
26
34
  else if (hasPrefix(env, "WINDSURF")) {
@@ -7,6 +7,7 @@
7
7
  * safe to poll. The API key is registered as a secret so it is redacted in logs.
8
8
  */
9
9
  import { ToolkitError, quotaExhausted } from "./errors.js";
10
+ import { storedClaimHint } from "./claim-link.js";
10
11
  import { isQuotaError, zrErrorDetail } from "./http.js";
11
12
  import { readAccount } from "./agent-account.js";
12
13
  import { ENV_KEY, resolveApiKey } from "./auth.js";
@@ -63,7 +64,7 @@ export async function fetchUsage(apiBase, apiKey, opts = {}) {
63
64
  // scraper 402. Surface it as a credits error (claim link for an unclaimed
64
65
  // Free plan account, dashboard/upgrade otherwise) instead of a generic "retry".
65
66
  const acct = readAccount();
66
- throw quotaExhausted(url, acct?.unclaimed ? acct.claimUrl : undefined, {
67
+ throw quotaExhausted(url, storedClaimHint(acct), {
67
68
  status: res.status,
68
69
  detail: zrErrorDetail(body) ?? undefined,
69
70
  });
@@ -5,7 +5,7 @@
5
5
  * assets, traces, and logs. Secrets live in `.zenrows/secrets.json`, which is
6
6
  * gitignored and never written into run artifacts.
7
7
  */
8
- import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
8
+ import { chmodSync, existsSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs";
9
9
  import { dirname, join, resolve } from "node:path";
10
10
  export const WORKSPACE_DIRNAME = ".zenrows";
11
11
  /** Subdirectories created inside `.zenrows/`. */
@@ -96,3 +96,31 @@ export function writeJson(file, value) {
96
96
  mkdirSync(dirname(file), { recursive: true });
97
97
  writeFileSync(file, JSON.stringify(value, null, 2) + "\n");
98
98
  }
99
+ /**
100
+ * Write a file only its owner may read (secrets, account metadata). The data
101
+ * goes to a temp file created 0600 and is renamed into place, so the target is
102
+ * never readable by others, not even for a moment, and a reader never sees a
103
+ * half-written file. A directory this creates is 0700.
104
+ */
105
+ export function writePrivateJson(file, value) {
106
+ mkdirSync(dirname(file), { recursive: true, mode: 0o700 });
107
+ const tmp = `${file}.${process.pid}.${Date.now()}.tmp`;
108
+ try {
109
+ writeFileSync(tmp, JSON.stringify(value, null, 2) + "\n", { mode: 0o600, flag: "wx" });
110
+ try {
111
+ chmodSync(tmp, 0o600); // the umask can only remove bits, but be explicit
112
+ }
113
+ catch {
114
+ // best-effort on platforms without POSIX permissions
115
+ }
116
+ renameSync(tmp, file);
117
+ }
118
+ catch (err) {
119
+ try {
120
+ unlinkSync(tmp);
121
+ }
122
+ catch {
123
+ }
124
+ throw err;
125
+ }
126
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@zenrows/cli",
3
- "version": "1.6.1",
3
+ "version": "1.7.0",
4
4
  "description": "Open-source CLI, MCP, skills, templates, workflows, recipes, and evals layer for giving AI agents reliable access to protected web data through Zenrows cloud infrastructure.",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -39,6 +39,10 @@ zenrows crawl results <id> --out urls.jsonl # same, as JSONL
39
39
  - `--include-patterns` / `--exclude-patterns` are substring matches on the
40
40
  normalized URL and repeat:
41
41
  `--include-patterns /product/ --exclude-patterns ?add-to-cart`.
42
+ - A crawl whose pages held nothing new to follow (no link, only links to other
43
+ sites, or only links the patterns reject) ends `completed` with no results
44
+ and exits 0. When patterns were set, it prints a note (under `--json`, in
45
+ `note`) to check them against the URLs the start page links to.
42
46
  - `create` returns at once. `--follow` (or `zenrows crawl wait <id>`) polls
43
47
  until the crawl ends or `--timeout` seconds run out (default 600). When the
44
48
  time runs out, the command exits 0 and prints the crawl with status
@@ -85,7 +89,10 @@ credit cap; the pages fetched before that stay readable. `failed`, and a
85
89
  `stopped` crawl with an `error`, exit non-zero: `CRAWL_INSUFFICIENT_CREDITS` or
86
90
  `CRAWL_KEY_CAP_REACHED` when credits ended it (the same codes a refused
87
91
  `create` gets, also when an older deployment reports it `failed`), else
88
- `CRAWL_FAILED`. `server_code` carries the API's `error.code`.
92
+ `CRAWL_FAILED`. `server_code` carries the API's `error.code`:
93
+ `seed_unreachable` when the start URL could not be fetched or returned an empty
94
+ page, `no_items_found` when the crawl read the start page but found no data it
95
+ could use.
89
96
  `stop` on a crawl that already ended prints the status it ended with.
90
97
 
91
98
  With `--json`, `create`, `get`, `wait` and `stop` print
@@ -22,7 +22,9 @@ key locally — no human step required. To opt out, pass `--no-signup` or set
22
22
 
23
23
  When the Free plan nears its limit (or the user asks), tell the human they can **claim** the
24
24
  account to keep their usage and history: run `zenrows account claim` (or open the claim URL
25
- shown on quota errors). Claiming sets an email + password; the same key keeps working.
25
+ shown on quota errors). It prints a valid link, requesting a fresh one when the stored link is
26
+ near its 30-day expiry. Claiming confirms an email (Google, GitHub or a 6-digit code); the same
27
+ key keeps working.
26
28
  Never attempt to handle payment or upgrades — only surface the claim link.
27
29
 
28
30
  ## The decision tree (follow this exactly)