webscraping-ai-cli 1.0.2 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +14 -4
- package/CHANGELOG.md +21 -0
- package/README.md +39 -30
- package/dist/cli.js +266 -92
- package/dist/cli.js.map +1 -1
- package/package.json +2 -2
- package/plugins/webscraping-ai/.claude-plugin/plugin.json +13 -3
- package/plugins/webscraping-ai/skills/webscraping-ai/SKILL.md +26 -13
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/claude-code-plugin-marketplace.json",
|
|
3
3
|
"name": "webscraping-ai",
|
|
4
|
-
"description": "Official WebScraping.AI plugins for Claude Code — scrape and AI-extract data from any URL.",
|
|
4
|
+
"description": "Official WebScraping.AI plugins for Claude Code — scrape and AI-extract data from any URL, and get parsed Google search (SERP) results.",
|
|
5
5
|
"owner": {
|
|
6
6
|
"name": "WebScraping.AI",
|
|
7
7
|
"url": "https://webscraping.ai"
|
|
@@ -10,10 +10,20 @@
|
|
|
10
10
|
{
|
|
11
11
|
"name": "webscraping-ai",
|
|
12
12
|
"source": "./plugins/webscraping-ai",
|
|
13
|
-
"description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — via the WebScraping.AI CLI.",
|
|
14
|
-
"version": "1.
|
|
13
|
+
"description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — and parsed Google search results, via the WebScraping.AI CLI.",
|
|
14
|
+
"version": "1.1.0",
|
|
15
15
|
"category": "web-scraping",
|
|
16
|
-
"keywords": [
|
|
16
|
+
"keywords": [
|
|
17
|
+
"web-scraping",
|
|
18
|
+
"html",
|
|
19
|
+
"text-extraction",
|
|
20
|
+
"ai-extraction",
|
|
21
|
+
"scraping",
|
|
22
|
+
"search",
|
|
23
|
+
"serp",
|
|
24
|
+
"google-search",
|
|
25
|
+
"cli"
|
|
26
|
+
]
|
|
17
27
|
}
|
|
18
28
|
]
|
|
19
29
|
}
|
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,27 @@ All notable changes to `webscraping-ai-cli` are documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## 1.1.0 — 2026-09-25
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `serp <query>` subcommand (alias `search`) wrapping the new `/serp` endpoint: parsed Google search results as JSON. Flags: `--engine` (`google`), `--gl`, `--hl`, `--page <n>`. Multi-word queries don't need quoting; `-` reads one query per line from stdin. The scrape flags (`--js`, `--proxy`, `--country`, …) don't apply and aren't registered on it. Flat 15 credits per search.
|
|
13
|
+
- Bundled agent skill documents when to reach for `serp` (query-first research) instead of scraping a search URL.
|
|
14
|
+
|
|
15
|
+
### Changed
|
|
16
|
+
|
|
17
|
+
- Requires `webscraping-ai` SDK `^4.1.0` (adds `serp()`).
|
|
18
|
+
- `serp --page` is validated strictly (digits only, >= 1): `1.5`, `2abc` and `0` are rejected with exit code 2 before stdin is read or the key is resolved (the server also rejects them with a 400, not billed; checking client-side saves the round trip). Pages are 1–100: the server rejects a `page` above 100 with a 400.
|
|
19
|
+
- `serp` stdin batches skip only blank lines, so `#hashtag` queries are searched rather than dropped as comments. URL commands still treat `#` lines as comments.
|
|
20
|
+
- `serp` sends the query as typed (whitespace-only queries are still rejected). `from_cli` is not sent on `serp`: the SDK forwards only the SERP parameters.
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
|
|
24
|
+
- `--pretty` / `--no-pretty` no longer fail with "option '--no-pretty' cannot be used with option '--no-pretty'" (a self-conflict on every scrape command and `serp`).
|
|
25
|
+
- `-o/--output` in stdin batch mode kept only the last result; the file is now truncated once and each result appended.
|
|
26
|
+
- Error messages printed to stderr redact the API key and any `api_key=...` pattern (an HTML error body echoing the request URL used to print the key).
|
|
27
|
+
- Smoke script (`npm run smoke`) now checks results, not just exit codes: `serp` needs non-empty `organic_results` and a matching `search_parameters.q`, `selected-multiple` needs a non-empty inner array; signal-killed runs fail; page commands run with `--no-js --proxy datacenter` (~31 credits per sweep); FAIL lines redact the key.
|
|
28
|
+
|
|
8
29
|
## 1.0.2 — 2026-07-17
|
|
9
30
|
|
|
10
31
|
### Changed
|
package/README.md
CHANGED
|
@@ -45,29 +45,32 @@ webscraping-ai selected <url> --selector <css> # One CSS area
|
|
|
45
45
|
webscraping-ai selected-multiple <url> --selector a --selector b ...
|
|
46
46
|
webscraping-ai ask <url> --question "..." # AI question about the page
|
|
47
47
|
webscraping-ai extract <url> --fields '{"title":"...","price":"..."}'
|
|
48
|
+
webscraping-ai serp <query> [--gl us] [--hl en] [--page 1] # Parsed Google results (alias: search)
|
|
48
49
|
webscraping-ai account # Remaining credits / quota
|
|
49
50
|
```
|
|
50
51
|
|
|
52
|
+
`serp` (alias `search`) is query-shaped, not URL-shaped: it takes `--engine` (`google`, the default), `--gl` (country), `--hl` (language), `--page` (1-based, 10 results per page) plus `--api-key`/`--output`/`--pretty`, and none of the scrape flags below. It prints the JSON result (`search_parameters`, `search_information`, `organic_results`, `related_searches`, `pagination`). Flat 15 credits per search; failed searches are not charged. `--page` must be an integer >= 1 (pages are 1–100; the server rejects values above 100 with a 400); anything else is a usage error (exit 2) before any request. Multi-word queries don't need quoting; `-` reads one query per line from stdin, skipping only blank lines (a leading `#` is kept as part of the query, e.g. `#coffee`). The CLI's `from_cli` analytics flag is not sent on `serp` (the SDK forwards only the SERP parameters).
|
|
53
|
+
|
|
51
54
|
Common flags shared by every scrape command:
|
|
52
55
|
|
|
53
|
-
| Flag
|
|
54
|
-
|
|
|
55
|
-
| `--js` / `--no-js`
|
|
56
|
-
| `--proxy TYPE`
|
|
57
|
-
| `--country CC`
|
|
58
|
-
| `--device TYPE`
|
|
59
|
-
| `--timeout MS`
|
|
60
|
-
| `--js-timeout MS`
|
|
61
|
-
| `--wait-for CSS`
|
|
62
|
-
| `--headers JSON`
|
|
63
|
-
| `--custom-proxy URL`
|
|
64
|
-
| `--js-script CODE`
|
|
65
|
-
| `--error-on-404`
|
|
66
|
-
| `--error-on-redirect`
|
|
67
|
-
| `--output FILE`
|
|
68
|
-
| `--pretty / --no-pretty` | Force pretty / single-line JSON
|
|
69
|
-
|
|
70
|
-
`<url>` may be `-` to read URLs from stdin (one per line; lines starting with `#` are ignored).
|
|
56
|
+
| Flag | Purpose |
|
|
57
|
+
| ------------------------ | ---------------------------------------------------- |
|
|
58
|
+
| `--js` / `--no-js` | Toggle JavaScript rendering (default: on) |
|
|
59
|
+
| `--proxy TYPE` | `datacenter` (default), `residential`, `stealth` |
|
|
60
|
+
| `--country CC` | Two-letter proxy country code |
|
|
61
|
+
| `--device TYPE` | `desktop`, `mobile`, `tablet` |
|
|
62
|
+
| `--timeout MS` | Total page-retrieval timeout (max 30000) |
|
|
63
|
+
| `--js-timeout MS` | JS rendering time after page load (max 20000) |
|
|
64
|
+
| `--wait-for CSS` | CSS selector to wait for |
|
|
65
|
+
| `--headers JSON` | HTTP headers to forward; inline JSON or `@file.json` |
|
|
66
|
+
| `--custom-proxy URL` | `http://user:pass@host:port` |
|
|
67
|
+
| `--js-script CODE` | Custom JS to evaluate on the target page |
|
|
68
|
+
| `--error-on-404` | Treat 404 responses as errors |
|
|
69
|
+
| `--error-on-redirect` | Treat redirects as errors |
|
|
70
|
+
| `--output FILE` | Write to FILE instead of stdout |
|
|
71
|
+
| `--pretty / --no-pretty` | Force pretty / single-line JSON |
|
|
72
|
+
|
|
73
|
+
`<url>` may be `-` to read URLs from stdin (one per line; lines starting with `#` are ignored). With `--output FILE`, a stdin batch writes every result to FILE, one after another (the file is truncated once at the start).
|
|
71
74
|
|
|
72
75
|
## Install the AI agent skill
|
|
73
76
|
|
|
@@ -88,18 +91,18 @@ You can also install the skill via the Claude Code [Plugin Marketplace](https://
|
|
|
88
91
|
|
|
89
92
|
## Exit codes
|
|
90
93
|
|
|
91
|
-
| Code | Meaning
|
|
92
|
-
| ---- |
|
|
93
|
-
| 0 | Success
|
|
94
|
-
| 2 | Usage error (commander)
|
|
95
|
-
| 3 | 400 Bad Request from the API
|
|
96
|
-
| 4 | 403 Authentication error
|
|
97
|
-
| 5 | 402 Payment Required (out of credits)
|
|
98
|
-
| 6 | 429 Rate limit
|
|
99
|
-
| 7 | 500 Server error (or wrapped target-page error)
|
|
100
|
-
| 8 | Timeout (504 or local AbortController)
|
|
101
|
-
| 9 | Connection / DNS failure
|
|
102
|
-
| 1 | Anything else
|
|
94
|
+
| Code | Meaning |
|
|
95
|
+
| ---- | ----------------------------------------------- |
|
|
96
|
+
| 0 | Success |
|
|
97
|
+
| 2 | Usage error (commander) |
|
|
98
|
+
| 3 | 400 Bad Request from the API |
|
|
99
|
+
| 4 | 403 Authentication error |
|
|
100
|
+
| 5 | 402 Payment Required (out of credits) |
|
|
101
|
+
| 6 | 429 Rate limit |
|
|
102
|
+
| 7 | 500 Server error (or wrapped target-page error) |
|
|
103
|
+
| 8 | Timeout (504 or local AbortController) |
|
|
104
|
+
| 9 | Connection / DNS failure |
|
|
105
|
+
| 1 | Anything else |
|
|
103
106
|
|
|
104
107
|
Scripts can branch on these without parsing stderr.
|
|
105
108
|
|
|
@@ -118,6 +121,12 @@ cat urls.txt | webscraping-ai extract - \
|
|
|
118
121
|
--fields '{"title":"Product title","price":"Current price"}' \
|
|
119
122
|
--output products.ndjson
|
|
120
123
|
|
|
124
|
+
# Google results for a query, page 2, from Germany in German
|
|
125
|
+
webscraping-ai serp coffee machines --gl de --hl de --page 2
|
|
126
|
+
|
|
127
|
+
# Just the organic result links
|
|
128
|
+
webscraping-ai search "best espresso grinder" | jq -r '.organic_results[].link'
|
|
129
|
+
|
|
121
130
|
# Headers from a file
|
|
122
131
|
webscraping-ai html https://example.com --headers @./headers.json
|
|
123
132
|
```
|
package/dist/cli.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
3
|
// src/cli.ts
|
|
4
|
-
import { Command as
|
|
4
|
+
import { Command as Command11 } from "commander";
|
|
5
5
|
import { readFileSync } from "fs";
|
|
6
6
|
import { dirname as dirname3, resolve as resolve2 } from "path";
|
|
7
7
|
import { fileURLToPath as fileURLToPath2 } from "url";
|
|
@@ -67,15 +67,25 @@ function maskApiKey(key) {
|
|
|
67
67
|
|
|
68
68
|
// src/lib/output.ts
|
|
69
69
|
import { promises as fs2 } from "fs";
|
|
70
|
-
async function emit(value, opts = {}) {
|
|
70
|
+
async function emit(value, opts = {}, write = {}) {
|
|
71
71
|
const text = stringify(value, opts);
|
|
72
72
|
if (opts.output && opts.output.trim() !== "") {
|
|
73
|
-
|
|
73
|
+
const data = text + (text.endsWith("\n") ? "" : "\n");
|
|
74
|
+
if (write.append) await fs2.appendFile(opts.output, data);
|
|
75
|
+
else await fs2.writeFile(opts.output, data);
|
|
74
76
|
return;
|
|
75
77
|
}
|
|
76
78
|
process.stdout.write(text);
|
|
77
79
|
if (!text.endsWith("\n")) process.stdout.write("\n");
|
|
78
80
|
}
|
|
81
|
+
function createEmitter(opts = {}) {
|
|
82
|
+
let first = true;
|
|
83
|
+
return async (value) => {
|
|
84
|
+
const append = !first;
|
|
85
|
+
first = false;
|
|
86
|
+
await emit(value, opts, { append });
|
|
87
|
+
};
|
|
88
|
+
}
|
|
79
89
|
function stringify(value, opts = {}) {
|
|
80
90
|
if (typeof value === "string") return value;
|
|
81
91
|
const pretty = opts.pretty ?? process.stdout.isTTY ?? false;
|
|
@@ -84,7 +94,29 @@ function stringify(value, opts = {}) {
|
|
|
84
94
|
|
|
85
95
|
// src/lib/sdk.ts
|
|
86
96
|
import { WebScrapingAI } from "webscraping-ai";
|
|
97
|
+
|
|
98
|
+
// src/lib/redact.ts
|
|
99
|
+
var secrets = /* @__PURE__ */ new Set();
|
|
100
|
+
function registerSecret(value) {
|
|
101
|
+
if (value && value.trim() !== "") secrets.add(value);
|
|
102
|
+
}
|
|
103
|
+
var API_KEY_PARAM = /(api_key=)[^&\s"'<>]*/gi;
|
|
104
|
+
function redactSecrets(text) {
|
|
105
|
+
let out = text.replace(API_KEY_PARAM, "$1[REDACTED]");
|
|
106
|
+
const candidates = new Set(secrets);
|
|
107
|
+
const env = process.env.WEBSCRAPING_AI_API_KEY?.trim();
|
|
108
|
+
if (env) candidates.add(env);
|
|
109
|
+
for (const secret of candidates) {
|
|
110
|
+
for (const form of /* @__PURE__ */ new Set([secret, encodeURIComponent(secret)])) {
|
|
111
|
+
out = out.split(form).join("[REDACTED]");
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
return out;
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
// src/lib/sdk.ts
|
|
87
118
|
function createClient(options) {
|
|
119
|
+
registerSecret(options.apiKey);
|
|
88
120
|
const inner = new WebScrapingAI({
|
|
89
121
|
apiKey: options.apiKey,
|
|
90
122
|
timeoutMs: options.requestTimeoutMs
|
|
@@ -107,6 +139,8 @@ function createClient(options) {
|
|
|
107
139
|
return (o) => target.question({ ...o, ...tag });
|
|
108
140
|
case "fields":
|
|
109
141
|
return (o) => target.fields({ ...o, ...tag });
|
|
142
|
+
case "serp":
|
|
143
|
+
return (o) => target.serp({ ...o, ...tag });
|
|
110
144
|
case "account":
|
|
111
145
|
return value.bind(target);
|
|
112
146
|
default:
|
|
@@ -177,19 +211,29 @@ function buildQuestionOptions(url, question, flags, resolved) {
|
|
|
177
211
|
function buildFieldsOptions(url, fields, flags, resolved) {
|
|
178
212
|
return { url, fields, ...buildCommonOptions(flags, resolved) };
|
|
179
213
|
}
|
|
214
|
+
function buildSerpOptions(query, flags = {}) {
|
|
215
|
+
if (typeof query !== "string" || query.trim() === "") {
|
|
216
|
+
throw new Error("serp requires a non-empty query");
|
|
217
|
+
}
|
|
218
|
+
const out = { q: query };
|
|
219
|
+
if (flags.engine !== void 0) out.engine = flags.engine;
|
|
220
|
+
if (flags.gl) out.gl = flags.gl;
|
|
221
|
+
if (flags.hl) out.hl = flags.hl;
|
|
222
|
+
if (flags.page !== void 0) {
|
|
223
|
+
if (!Number.isSafeInteger(flags.page) || flags.page < 1) {
|
|
224
|
+
throw new Error(`--page must be an integer >= 1, got: ${flags.page}`);
|
|
225
|
+
}
|
|
226
|
+
out.page = flags.page;
|
|
227
|
+
}
|
|
228
|
+
return out;
|
|
229
|
+
}
|
|
180
230
|
|
|
181
231
|
// src/lib/options.ts
|
|
182
232
|
import { promises as fs3 } from "fs";
|
|
183
233
|
import { Option } from "commander";
|
|
184
234
|
function addCommonScrapeOptions(cmd) {
|
|
185
|
-
return cmd.option("-k, --api-key <key>", "API key (overrides $WEBSCRAPING_AI_API_KEY and config file)").option("-o, --output <file>", "write result to FILE instead of stdout").option("--pretty", "pretty-print JSON output (default on TTY)").addOption(
|
|
186
|
-
new Option("--
|
|
187
|
-
).option("--js", "execute JS via a headless browser (default: enabled)").option("--no-js", "disable JS execution (faster, cheaper)").addOption(
|
|
188
|
-
new Option("--proxy <type>", "proxy pool").choices([
|
|
189
|
-
"datacenter",
|
|
190
|
-
"residential",
|
|
191
|
-
"stealth"
|
|
192
|
-
])
|
|
235
|
+
return cmd.option("-k, --api-key <key>", "API key (overrides $WEBSCRAPING_AI_API_KEY and config file)").option("-o, --output <file>", "write result to FILE instead of stdout").option("--pretty", "pretty-print JSON output (default on TTY)").option("--no-pretty", "force single-line JSON output").option("--js", "execute JS via a headless browser (default: enabled)").option("--no-js", "disable JS execution (faster, cheaper)").addOption(
|
|
236
|
+
new Option("--proxy <type>", "proxy pool").choices(["datacenter", "residential", "stealth"])
|
|
193
237
|
).option("--country <code>", "two-letter proxy country code, e.g. us, gb, de").addOption(
|
|
194
238
|
new Option("--device <type>", "device emulation").choices(["desktop", "mobile", "tablet"])
|
|
195
239
|
).option("--timeout <ms>", "page-retrieval timeout in ms (max 30000)", integerParser).option("--js-timeout <ms>", "JS rendering time in ms after load (max 20000)", integerParser).option("--wait-for <selector>", "CSS selector to wait for before returning content").option(
|
|
@@ -197,6 +241,9 @@ function addCommonScrapeOptions(cmd) {
|
|
|
197
241
|
"HTTP headers to forward to the target page; JSON object or @file.json"
|
|
198
242
|
).option("--js-script <code>", "custom JS to evaluate on the target page").option("--custom-proxy <url>", "custom proxy URL: http://user:pass@host:port").option("--error-on-404", "return error on 404").option("--error-on-redirect", "return error on redirect");
|
|
199
243
|
}
|
|
244
|
+
function isPositiveIntegerString(value) {
|
|
245
|
+
return /^[1-9]\d*$/.test(value) && Number.isSafeInteger(Number(value));
|
|
246
|
+
}
|
|
200
247
|
function integerParser(value) {
|
|
201
248
|
const n = Number.parseInt(value, 10);
|
|
202
249
|
if (!Number.isFinite(n)) throw new Error(`expected integer, got: ${value}`);
|
|
@@ -246,6 +293,13 @@ async function* readUrlsFromStdin() {
|
|
|
246
293
|
yield line;
|
|
247
294
|
}
|
|
248
295
|
}
|
|
296
|
+
async function* readQueriesFromStdin(input = process.stdin) {
|
|
297
|
+
const rl = readline.createInterface({ input, crlfDelay: Infinity });
|
|
298
|
+
for await (const line of rl) {
|
|
299
|
+
if (line.trim() === "") continue;
|
|
300
|
+
yield line;
|
|
301
|
+
}
|
|
302
|
+
}
|
|
249
303
|
async function* urlIterator(arg) {
|
|
250
304
|
if (urlIsStdin(arg)) {
|
|
251
305
|
yield* readUrlsFromStdin();
|
|
@@ -253,6 +307,13 @@ async function* urlIterator(arg) {
|
|
|
253
307
|
}
|
|
254
308
|
yield arg;
|
|
255
309
|
}
|
|
310
|
+
async function* queryIterator(arg) {
|
|
311
|
+
if (urlIsStdin(arg)) {
|
|
312
|
+
yield* readQueriesFromStdin();
|
|
313
|
+
return;
|
|
314
|
+
}
|
|
315
|
+
yield arg;
|
|
316
|
+
}
|
|
256
317
|
|
|
257
318
|
// src/commands/ask.ts
|
|
258
319
|
function askCommand() {
|
|
@@ -263,10 +324,11 @@ function askCommand() {
|
|
|
263
324
|
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
264
325
|
const headers = await parseHeaders(opts.headers);
|
|
265
326
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
327
|
+
const write = createEmitter(opts);
|
|
266
328
|
for await (const target of urlIterator(url)) {
|
|
267
329
|
const sdkOptions = buildQuestionOptions(target, opts.question, opts, { headers });
|
|
268
330
|
const result = await client.question(sdkOptions);
|
|
269
|
-
await
|
|
331
|
+
await write(result);
|
|
270
332
|
}
|
|
271
333
|
});
|
|
272
334
|
return cmd;
|
|
@@ -286,8 +348,10 @@ function authCommand() {
|
|
|
286
348
|
const persisted = await readPersistedConfig();
|
|
287
349
|
const env = process.env.WEBSCRAPING_AI_API_KEY;
|
|
288
350
|
if (env && env.trim() !== "") {
|
|
289
|
-
process.stdout.write(
|
|
290
|
-
`)
|
|
351
|
+
process.stdout.write(
|
|
352
|
+
`Active key (env WEBSCRAPING_AI_API_KEY): ${maskApiKey(env.trim())}
|
|
353
|
+
`
|
|
354
|
+
);
|
|
291
355
|
}
|
|
292
356
|
if (persisted.apiKey && persisted.apiKey.trim() !== "") {
|
|
293
357
|
process.stdout.write(
|
|
@@ -301,8 +365,10 @@ function authCommand() {
|
|
|
301
365
|
});
|
|
302
366
|
cmd.command("clear").description("Remove the stored API key file").action(async () => {
|
|
303
367
|
const removed = await clearPersistedConfig();
|
|
304
|
-
process.stdout.write(
|
|
305
|
-
|
|
368
|
+
process.stdout.write(
|
|
369
|
+
removed ? `Removed ${configPath()}
|
|
370
|
+
` : "No stored API key to remove.\n"
|
|
371
|
+
);
|
|
306
372
|
});
|
|
307
373
|
return cmd;
|
|
308
374
|
}
|
|
@@ -321,10 +387,11 @@ function extractCommand() {
|
|
|
321
387
|
const headers = await parseHeaders(opts.headers);
|
|
322
388
|
const fields = await parseFields(opts.fields);
|
|
323
389
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
390
|
+
const write = createEmitter(opts);
|
|
324
391
|
for await (const target of urlIterator(url)) {
|
|
325
392
|
const sdkOptions = buildFieldsOptions(target, fields, opts, { headers });
|
|
326
393
|
const result = await client.fields(sdkOptions);
|
|
327
|
-
await
|
|
394
|
+
await write(result);
|
|
328
395
|
}
|
|
329
396
|
});
|
|
330
397
|
return cmd;
|
|
@@ -333,18 +400,26 @@ function extractCommand() {
|
|
|
333
400
|
// src/commands/html.ts
|
|
334
401
|
import { Command as Command5, Option as Option3 } from "commander";
|
|
335
402
|
function htmlCommand() {
|
|
336
|
-
const cmd = new Command5("html").description("Fetch the full HTML of a page").argument("<url>", "page URL (or `-` to read URLs from stdin)").addOption(
|
|
403
|
+
const cmd = new Command5("html").description("Fetch the full HTML of a page").argument("<url>", "page URL (or `-` to read URLs from stdin)").addOption(
|
|
404
|
+
new Option3("--return-script-result", "return the result of --js-script instead of HTML")
|
|
405
|
+
).addOption(new Option3("--format <fmt>", "response wrapping").choices(["text", "json"]));
|
|
337
406
|
addCommonScrapeOptions(cmd);
|
|
338
407
|
cmd.action(async (url, opts) => {
|
|
339
408
|
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
340
409
|
const headers = await parseHeaders(opts.headers);
|
|
341
410
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
411
|
+
const write = createEmitter(opts);
|
|
342
412
|
for await (const target of urlIterator(url)) {
|
|
343
|
-
const sdkOptions = buildHtmlOptions(
|
|
344
|
-
|
|
345
|
-
|
|
413
|
+
const sdkOptions = buildHtmlOptions(
|
|
414
|
+
target,
|
|
415
|
+
opts,
|
|
416
|
+
{ headers },
|
|
417
|
+
{
|
|
418
|
+
returnScriptResult: opts.returnScriptResult
|
|
419
|
+
}
|
|
420
|
+
);
|
|
346
421
|
const result = await client.html(sdkOptions);
|
|
347
|
-
await
|
|
422
|
+
await write(result);
|
|
348
423
|
}
|
|
349
424
|
});
|
|
350
425
|
return cmd;
|
|
@@ -362,10 +437,11 @@ function selectedCommand() {
|
|
|
362
437
|
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
363
438
|
const headers = await parseHeaders(opts.headers);
|
|
364
439
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
440
|
+
const write = createEmitter(opts);
|
|
365
441
|
for await (const target of urlIterator(url)) {
|
|
366
442
|
const sdkOptions = buildSelectedOptions(target, opts.selector, opts, { headers });
|
|
367
443
|
const result = await client.selected(sdkOptions);
|
|
368
|
-
await
|
|
444
|
+
await write(result);
|
|
369
445
|
}
|
|
370
446
|
});
|
|
371
447
|
return cmd;
|
|
@@ -386,10 +462,11 @@ function selectedMultipleCommand() {
|
|
|
386
462
|
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
387
463
|
const headers = await parseHeaders(opts.headers);
|
|
388
464
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
465
|
+
const write = createEmitter(opts);
|
|
389
466
|
for await (const target of urlIterator(url)) {
|
|
390
467
|
const sdkOptions = buildSelectedMultipleOptions(target, selectors, opts, { headers });
|
|
391
468
|
const result = await client.selectedMultiple(sdkOptions);
|
|
392
|
-
await
|
|
469
|
+
await write(result);
|
|
393
470
|
}
|
|
394
471
|
});
|
|
395
472
|
return cmd;
|
|
@@ -398,8 +475,98 @@ function collectMulti(value, previous) {
|
|
|
398
475
|
return [...previous, value];
|
|
399
476
|
}
|
|
400
477
|
|
|
478
|
+
// src/commands/serp.ts
|
|
479
|
+
import { Command as Command8, Option as Option5 } from "commander";
|
|
480
|
+
|
|
481
|
+
// src/lib/errors.ts
|
|
482
|
+
import {
|
|
483
|
+
APIConnectionError,
|
|
484
|
+
APIError,
|
|
485
|
+
APITimeoutError,
|
|
486
|
+
AuthenticationError,
|
|
487
|
+
BadRequestError,
|
|
488
|
+
GatewayTimeoutError,
|
|
489
|
+
PaymentRequiredError,
|
|
490
|
+
RateLimitError,
|
|
491
|
+
ServerError,
|
|
492
|
+
WebScrapingAIError
|
|
493
|
+
} from "webscraping-ai";
|
|
494
|
+
var EXIT_CODES = {
|
|
495
|
+
ok: 0,
|
|
496
|
+
generic: 1,
|
|
497
|
+
usage: 2,
|
|
498
|
+
badRequest: 3,
|
|
499
|
+
auth: 4,
|
|
500
|
+
paymentRequired: 5,
|
|
501
|
+
rateLimit: 6,
|
|
502
|
+
serverError: 7,
|
|
503
|
+
timeout: 8,
|
|
504
|
+
connection: 9
|
|
505
|
+
};
|
|
506
|
+
function exitCodeFor(err) {
|
|
507
|
+
if (err instanceof BadRequestError) return EXIT_CODES.badRequest;
|
|
508
|
+
if (err instanceof AuthenticationError) return EXIT_CODES.auth;
|
|
509
|
+
if (err instanceof PaymentRequiredError) return EXIT_CODES.paymentRequired;
|
|
510
|
+
if (err instanceof RateLimitError) return EXIT_CODES.rateLimit;
|
|
511
|
+
if (err instanceof ServerError) return EXIT_CODES.serverError;
|
|
512
|
+
if (err instanceof GatewayTimeoutError) return EXIT_CODES.timeout;
|
|
513
|
+
if (err instanceof APITimeoutError) return EXIT_CODES.timeout;
|
|
514
|
+
if (err instanceof APIConnectionError) return EXIT_CODES.connection;
|
|
515
|
+
if (err instanceof APIError) return EXIT_CODES.serverError;
|
|
516
|
+
if (err instanceof WebScrapingAIError) return EXIT_CODES.generic;
|
|
517
|
+
return EXIT_CODES.generic;
|
|
518
|
+
}
|
|
519
|
+
function formatError(err) {
|
|
520
|
+
return redactSecrets(rawMessage(err));
|
|
521
|
+
}
|
|
522
|
+
function rawMessage(err) {
|
|
523
|
+
if (err instanceof APIError) {
|
|
524
|
+
const parts = [err.message];
|
|
525
|
+
if (err.status) parts.push(`(HTTP ${err.status})`);
|
|
526
|
+
return parts.join(" ");
|
|
527
|
+
}
|
|
528
|
+
if (err instanceof Error) return err.message;
|
|
529
|
+
return String(err);
|
|
530
|
+
}
|
|
531
|
+
|
|
532
|
+
// src/commands/serp.ts
|
|
533
|
+
function serpCommand() {
|
|
534
|
+
const cmd = new Command8("serp").alias("search").description("Search Google and print parsed results as JSON (wraps /serp, 15 credits/search)").argument("<query...>", "search query (or `-` to read queries from stdin, one per line)").addOption(new Option5("--engine <name>", "search engine (default: google)").choices(["google"])).option("--gl <code>", "two-letter country code for search geolocation (default: us)").option("--hl <code>", "two-letter language code for the results (default: en)").option(
|
|
535
|
+
"--page <n>",
|
|
536
|
+
"results page number, integer >= 1, 10 results per page (default: 1; server rejects > 100 with a 400)"
|
|
537
|
+
).option("-k, --api-key <key>", "API key (overrides $WEBSCRAPING_AI_API_KEY and config file)").option("-o, --output <file>", "write result to FILE instead of stdout").option("--pretty", "pretty-print JSON output (default on TTY)").option("--no-pretty", "force single-line JSON output");
|
|
538
|
+
cmd.action(async (words, opts) => {
|
|
539
|
+
const query = words.join(" ");
|
|
540
|
+
let page;
|
|
541
|
+
if (opts.page !== void 0) {
|
|
542
|
+
if (!isPositiveIntegerString(opts.page)) {
|
|
543
|
+
cmd.error(`error: --page must be an integer >= 1, got: ${opts.page}`, {
|
|
544
|
+
exitCode: EXIT_CODES.usage,
|
|
545
|
+
code: "wsai.invalidPage"
|
|
546
|
+
});
|
|
547
|
+
}
|
|
548
|
+
page = Number(opts.page);
|
|
549
|
+
}
|
|
550
|
+
if (!urlIsStdin(query) && query.trim() === "") {
|
|
551
|
+
cmd.error("error: serp requires a non-empty query", {
|
|
552
|
+
exitCode: EXIT_CODES.usage,
|
|
553
|
+
code: "wsai.emptyQuery"
|
|
554
|
+
});
|
|
555
|
+
}
|
|
556
|
+
const flags = { engine: opts.engine, gl: opts.gl, hl: opts.hl, page };
|
|
557
|
+
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
558
|
+
const client = createClient({ apiKey });
|
|
559
|
+
const write = createEmitter(opts);
|
|
560
|
+
for await (const q of queryIterator(query)) {
|
|
561
|
+
const result = await client.serp(buildSerpOptions(q, flags));
|
|
562
|
+
await write(result);
|
|
563
|
+
}
|
|
564
|
+
});
|
|
565
|
+
return cmd;
|
|
566
|
+
}
|
|
567
|
+
|
|
401
568
|
// src/commands/setupSkill.ts
|
|
402
|
-
import { Command as
|
|
569
|
+
import { Command as Command9 } from "commander";
|
|
403
570
|
|
|
404
571
|
// src/lib/skill.ts
|
|
405
572
|
import { promises as fs4 } from "fs";
|
|
@@ -408,24 +575,72 @@ import { dirname as dirname2, join as join2, resolve } from "path";
|
|
|
408
575
|
import { fileURLToPath } from "url";
|
|
409
576
|
var EDITORS = [
|
|
410
577
|
// Claude Code: user-scope skills live under ~/.claude/skills/<name>/
|
|
411
|
-
{
|
|
578
|
+
{
|
|
579
|
+
id: "claude-code",
|
|
580
|
+
label: "Claude Code",
|
|
581
|
+
mode: "dir",
|
|
582
|
+
kind: "user",
|
|
583
|
+
path: ".claude/skills/webscraping-ai"
|
|
584
|
+
},
|
|
412
585
|
// Cursor: project-scope rule file. Cursor doesn't have a user-wide skill
|
|
413
586
|
// dir, so this is the editor's "install" surface.
|
|
414
|
-
{
|
|
587
|
+
{
|
|
588
|
+
id: "cursor",
|
|
589
|
+
label: "Cursor",
|
|
590
|
+
mode: "file",
|
|
591
|
+
kind: "project",
|
|
592
|
+
path: ".cursor/rules/webscraping-ai.mdc"
|
|
593
|
+
},
|
|
415
594
|
// Windsurf: project-scope rules. Same shape as Cursor.
|
|
416
|
-
{
|
|
595
|
+
{
|
|
596
|
+
id: "windsurf",
|
|
597
|
+
label: "Windsurf",
|
|
598
|
+
mode: "file",
|
|
599
|
+
kind: "project",
|
|
600
|
+
path: ".windsurf/rules/webscraping-ai.md"
|
|
601
|
+
},
|
|
417
602
|
// Kiro: user-scope steering directory.
|
|
418
603
|
{ id: "kiro", label: "Kiro", mode: "dir", kind: "user", path: ".kiro/steering/webscraping-ai" },
|
|
419
604
|
// OpenCode: project-scope skills (mirrors Claude shape).
|
|
420
|
-
{
|
|
605
|
+
{
|
|
606
|
+
id: "opencode",
|
|
607
|
+
label: "OpenCode",
|
|
608
|
+
mode: "dir",
|
|
609
|
+
kind: "project",
|
|
610
|
+
path: ".opencode/skills/webscraping-ai"
|
|
611
|
+
},
|
|
421
612
|
// Gemini CLI: user-scope extension dir with a GEMINI.md.
|
|
422
|
-
{
|
|
613
|
+
{
|
|
614
|
+
id: "gemini",
|
|
615
|
+
label: "Gemini CLI",
|
|
616
|
+
mode: "file",
|
|
617
|
+
kind: "user",
|
|
618
|
+
path: ".gemini/extensions/webscraping-ai/GEMINI.md"
|
|
619
|
+
},
|
|
423
620
|
// GitHub Copilot: project-scope custom instructions file. Single file.
|
|
424
|
-
{
|
|
621
|
+
{
|
|
622
|
+
id: "copilot",
|
|
623
|
+
label: "GitHub Copilot",
|
|
624
|
+
mode: "file",
|
|
625
|
+
kind: "project",
|
|
626
|
+
path: ".github/copilot-instructions.md"
|
|
627
|
+
},
|
|
425
628
|
// Augment: project-scope rules file.
|
|
426
|
-
{
|
|
629
|
+
{
|
|
630
|
+
id: "augment",
|
|
631
|
+
label: "Augment",
|
|
632
|
+
mode: "file",
|
|
633
|
+
kind: "project",
|
|
634
|
+
path: ".augment/rules/webscraping-ai.md"
|
|
635
|
+
},
|
|
427
636
|
// Factory: project-scope rules file.
|
|
428
|
-
{
|
|
637
|
+
{
|
|
638
|
+
id: "factory",
|
|
639
|
+
label: "Factory",
|
|
640
|
+
mode: "file",
|
|
641
|
+
kind: "project",
|
|
642
|
+
path: ".factory/rules/webscraping-ai.md"
|
|
643
|
+
}
|
|
429
644
|
];
|
|
430
645
|
function editorById(id) {
|
|
431
646
|
return EDITORS.find((e) => e.id === id);
|
|
@@ -468,7 +683,7 @@ async function copyDirectory(source, destination) {
|
|
|
468
683
|
|
|
469
684
|
// src/commands/setupSkill.ts
|
|
470
685
|
function setupCommand() {
|
|
471
|
-
const cmd = new
|
|
686
|
+
const cmd = new Command9("setup").description("Install the AI agent skill into your editor(s)");
|
|
472
687
|
cmd.command("skill").description("Install the WebScraping.AI skill into one or more AI coding editors").option(
|
|
473
688
|
"-e, --editor <id>",
|
|
474
689
|
`editor id (repeatable). Supported: ${EDITORS.map((e) => e.id).join(", ")}`,
|
|
@@ -511,76 +726,34 @@ function resolveSelection(opts) {
|
|
|
511
726
|
}
|
|
512
727
|
|
|
513
728
|
// src/commands/text.ts
|
|
514
|
-
import { Command as
|
|
729
|
+
import { Command as Command10, Option as Option6 } from "commander";
|
|
515
730
|
function textCommand() {
|
|
516
|
-
const cmd = new
|
|
517
|
-
new
|
|
731
|
+
const cmd = new Command10("text").description("Fetch the visible text of a page").argument("<url>", "page URL (or `-` to read URLs from stdin)").addOption(
|
|
732
|
+
new Option6("--text-format <fmt>", "text response format").choices(["plain", "json", "xml"])
|
|
518
733
|
).option("--return-links", "include page links (text-format=json only)");
|
|
519
734
|
addCommonScrapeOptions(cmd);
|
|
520
735
|
cmd.action(async (url, opts) => {
|
|
521
736
|
const apiKey = await resolveApiKey({ flag: opts.apiKey });
|
|
522
737
|
const headers = await parseHeaders(opts.headers);
|
|
523
738
|
const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });
|
|
739
|
+
const write = createEmitter(opts);
|
|
524
740
|
for await (const target of urlIterator(url)) {
|
|
525
|
-
const sdkOptions = buildTextOptions(
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
741
|
+
const sdkOptions = buildTextOptions(
|
|
742
|
+
target,
|
|
743
|
+
opts,
|
|
744
|
+
{ headers },
|
|
745
|
+
{
|
|
746
|
+
textFormat: opts.textFormat,
|
|
747
|
+
returnLinks: opts.returnLinks
|
|
748
|
+
}
|
|
749
|
+
);
|
|
529
750
|
const result = await client.text(sdkOptions);
|
|
530
|
-
await
|
|
751
|
+
await write(result);
|
|
531
752
|
}
|
|
532
753
|
});
|
|
533
754
|
return cmd;
|
|
534
755
|
}
|
|
535
756
|
|
|
536
|
-
// src/lib/errors.ts
|
|
537
|
-
import {
|
|
538
|
-
APIConnectionError,
|
|
539
|
-
APIError,
|
|
540
|
-
APITimeoutError,
|
|
541
|
-
AuthenticationError,
|
|
542
|
-
BadRequestError,
|
|
543
|
-
GatewayTimeoutError,
|
|
544
|
-
PaymentRequiredError,
|
|
545
|
-
RateLimitError,
|
|
546
|
-
ServerError,
|
|
547
|
-
WebScrapingAIError
|
|
548
|
-
} from "webscraping-ai";
|
|
549
|
-
var EXIT_CODES = {
|
|
550
|
-
ok: 0,
|
|
551
|
-
generic: 1,
|
|
552
|
-
usage: 2,
|
|
553
|
-
badRequest: 3,
|
|
554
|
-
auth: 4,
|
|
555
|
-
paymentRequired: 5,
|
|
556
|
-
rateLimit: 6,
|
|
557
|
-
serverError: 7,
|
|
558
|
-
timeout: 8,
|
|
559
|
-
connection: 9
|
|
560
|
-
};
|
|
561
|
-
function exitCodeFor(err) {
|
|
562
|
-
if (err instanceof BadRequestError) return EXIT_CODES.badRequest;
|
|
563
|
-
if (err instanceof AuthenticationError) return EXIT_CODES.auth;
|
|
564
|
-
if (err instanceof PaymentRequiredError) return EXIT_CODES.paymentRequired;
|
|
565
|
-
if (err instanceof RateLimitError) return EXIT_CODES.rateLimit;
|
|
566
|
-
if (err instanceof ServerError) return EXIT_CODES.serverError;
|
|
567
|
-
if (err instanceof GatewayTimeoutError) return EXIT_CODES.timeout;
|
|
568
|
-
if (err instanceof APITimeoutError) return EXIT_CODES.timeout;
|
|
569
|
-
if (err instanceof APIConnectionError) return EXIT_CODES.connection;
|
|
570
|
-
if (err instanceof APIError) return EXIT_CODES.serverError;
|
|
571
|
-
if (err instanceof WebScrapingAIError) return EXIT_CODES.generic;
|
|
572
|
-
return EXIT_CODES.generic;
|
|
573
|
-
}
|
|
574
|
-
function formatError(err) {
|
|
575
|
-
if (err instanceof APIError) {
|
|
576
|
-
const parts = [err.message];
|
|
577
|
-
if (err.status) parts.push(`(HTTP ${err.status})`);
|
|
578
|
-
return parts.join(" ");
|
|
579
|
-
}
|
|
580
|
-
if (err instanceof Error) return err.message;
|
|
581
|
-
return String(err);
|
|
582
|
-
}
|
|
583
|
-
|
|
584
757
|
// src/cli.ts
|
|
585
758
|
function packageVersion() {
|
|
586
759
|
const here = dirname3(fileURLToPath2(import.meta.url));
|
|
@@ -589,9 +762,9 @@ function packageVersion() {
|
|
|
589
762
|
return raw.version;
|
|
590
763
|
}
|
|
591
764
|
function buildProgram() {
|
|
592
|
-
const program = new
|
|
765
|
+
const program = new Command11();
|
|
593
766
|
program.name("webscraping-ai").description(
|
|
594
|
-
"Official CLI for the WebScraping.AI API. Fetch pages, ask AI, extract structured data.\nSet $WEBSCRAPING_AI_API_KEY or run `webscraping-ai auth set <key>` first."
|
|
767
|
+
"Official CLI for the WebScraping.AI API. Fetch pages, search Google, ask AI, extract structured data.\nSet $WEBSCRAPING_AI_API_KEY or run `webscraping-ai auth set <key>` first."
|
|
595
768
|
).version(packageVersion(), "-v, --version", "output the version number").showHelpAfterError();
|
|
596
769
|
program.addCommand(htmlCommand());
|
|
597
770
|
program.addCommand(textCommand());
|
|
@@ -599,6 +772,7 @@ function buildProgram() {
|
|
|
599
772
|
program.addCommand(selectedMultipleCommand());
|
|
600
773
|
program.addCommand(askCommand());
|
|
601
774
|
program.addCommand(extractCommand());
|
|
775
|
+
program.addCommand(serpCommand());
|
|
602
776
|
program.addCommand(accountCommand());
|
|
603
777
|
program.addCommand(authCommand());
|
|
604
778
|
program.addCommand(setupCommand());
|
package/dist/cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"sources":["../src/cli.ts","../src/commands/account.ts","../src/lib/config.ts","../src/lib/output.ts","../src/lib/sdk.ts","../src/commands/ask.ts","../src/lib/buildRequest.ts","../src/lib/options.ts","../src/lib/stdin.ts","../src/commands/auth.ts","../src/commands/extract.ts","../src/commands/html.ts","../src/commands/selected.ts","../src/commands/selectedMultiple.ts","../src/commands/setupSkill.ts","../src/lib/skill.ts","../src/commands/text.ts","../src/lib/errors.ts"],"sourcesContent":["/**\n * `webscraping-ai` / `wsai` CLI entry point.\n *\n * Wires every subcommand into a single commander root. Errors thrown from\n * subcommand actions land in the top-level `.catch`, which maps SDK\n * exception classes to stable exit codes (see `lib/errors.ts`).\n */\n\nimport { Command } from 'commander';\nimport { readFileSync } from 'node:fs';\nimport { dirname, resolve } from 'node:path';\nimport { fileURLToPath } from 'node:url';\n\nimport { accountCommand } from './commands/account.js';\nimport { askCommand } from './commands/ask.js';\nimport { authCommand } from './commands/auth.js';\nimport { extractCommand } from './commands/extract.js';\nimport { htmlCommand } from './commands/html.js';\nimport { selectedCommand } from './commands/selected.js';\nimport { selectedMultipleCommand } from './commands/selectedMultiple.js';\nimport { setupCommand } from './commands/setupSkill.js';\nimport { textCommand } from './commands/text.js';\nimport { exitCodeFor, formatError } from './lib/errors.js';\n\nfunction packageVersion(): string {\n // Read package.json at runtime so we don't need a code-gen step. The dist\n // bundle sits at `dist/cli.js`, so package.json is one directory up.\n const here = dirname(fileURLToPath(import.meta.url));\n const pkgPath = resolve(here, '..', 'package.json');\n const raw = JSON.parse(readFileSync(pkgPath, 'utf8')) as { version: string };\n return raw.version;\n}\n\nexport function buildProgram(): Command {\n const program = new Command();\n program\n .name('webscraping-ai')\n .description(\n 'Official CLI for the WebScraping.AI API. Fetch pages, ask AI, extract structured data.\\n' +\n 'Set $WEBSCRAPING_AI_API_KEY or run `webscraping-ai auth set <key>` first.',\n )\n .version(packageVersion(), '-v, --version', 'output the version number')\n .showHelpAfterError();\n\n program.addCommand(htmlCommand());\n program.addCommand(textCommand());\n program.addCommand(selectedCommand());\n program.addCommand(selectedMultipleCommand());\n program.addCommand(askCommand());\n program.addCommand(extractCommand());\n program.addCommand(accountCommand());\n program.addCommand(authCommand());\n program.addCommand(setupCommand());\n\n return program;\n}\n\nasync function main(argv: string[]): Promise<void> {\n const program = buildProgram();\n try {\n await program.parseAsync(argv);\n } catch (err) {\n process.stderr.write(`Error: ${formatError(err)}\\n`);\n process.exit(exitCodeFor(err));\n }\n}\n\n// Only execute when run as a script. When imported (e.g. from tests),\n// `buildProgram` is the public surface and `main` stays dormant.\nconst entryPath = process.argv[1] ? fileURLToPath(import.meta.url) : '';\nif (entryPath && process.argv[1] === entryPath) {\n main(process.argv).catch((err: unknown) => {\n process.stderr.write(`Error: ${formatError(err)}\\n`);\n process.exit(exitCodeFor(err));\n });\n}\n","import { Command } from 'commander';\n\nimport { resolveApiKey } from '../lib/config.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\n\ninterface AccountFlags {\n apiKey?: string;\n output?: string;\n pretty?: boolean;\n}\n\nexport function accountCommand(): Command {\n return new Command('account')\n .description('Show account quota and usage information')\n .option('-k, --api-key <key>', 'API key (overrides $WEBSCRAPING_AI_API_KEY and config file)')\n .option('-o, --output <file>', 'write result to FILE instead of stdout')\n .option('--pretty', 'pretty-print JSON output (default on TTY)')\n .option('--no-pretty', 'force single-line JSON output')\n .action(async (opts: AccountFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const client = createClient({ apiKey });\n const result = await client.account();\n await emit(result, opts);\n });\n}\n","/**\n * API key resolution and persisted config.\n *\n * Resolution order (first non-empty wins):\n * 1. `--api-key` flag (handled by the caller, passed in)\n * 2. `WEBSCRAPING_AI_API_KEY` env var\n * 3. `~/.config/webscraping-ai/config.json` (or `$XDG_CONFIG_HOME/...`)\n *\n * The config file is plain JSON with `{ \"apiKey\": \"...\" }`. We chmod it to\n * 0600 so it's not world-readable.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { homedir } from 'node:os';\nimport { dirname, join } from 'node:path';\n\nconst CONFIG_FILENAME = 'config.json';\n\nexport function configDir(): string {\n const xdg = process.env.XDG_CONFIG_HOME;\n if (xdg && xdg.trim() !== '') return join(xdg, 'webscraping-ai');\n return join(homedir(), '.config', 'webscraping-ai');\n}\n\nexport function configPath(): string {\n return join(configDir(), CONFIG_FILENAME);\n}\n\nexport interface PersistedConfig {\n apiKey?: string;\n}\n\nexport async function readPersistedConfig(): Promise<PersistedConfig> {\n try {\n const text = await fs.readFile(configPath(), 'utf8');\n const parsed = JSON.parse(text) as unknown;\n if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {\n return parsed as PersistedConfig;\n }\n return {};\n } catch (err) {\n if ((err as NodeJS.ErrnoException).code === 'ENOENT') return {};\n throw err;\n }\n}\n\nexport async function writePersistedConfig(config: PersistedConfig): Promise<void> {\n const path = configPath();\n await fs.mkdir(dirname(path), { recursive: true });\n await fs.writeFile(path, JSON.stringify(config, null, 2) + '\\n', { mode: 0o600 });\n // `mode` above only applies when the file is freshly created; an existing\n // file keeps its old (possibly world-readable) perms. Enforce 0600\n // unconditionally — chmod on a just-written file is cheap and idempotent.\n // No-op on Windows, but harmless.\n await fs.chmod(path, 0o600);\n}\n\nexport async function clearPersistedConfig(): Promise<boolean> {\n try {\n await fs.unlink(configPath());\n return true;\n } catch (err) {\n if ((err as NodeJS.ErrnoException).code === 'ENOENT') return false;\n throw err;\n }\n}\n\nexport interface ResolveApiKeyOptions {\n /** `--api-key` flag value, if the user passed one. */\n flag?: string;\n}\n\nexport async function resolveApiKey(opts: ResolveApiKeyOptions = {}): Promise<string> {\n if (opts.flag && opts.flag.trim() !== '') return opts.flag.trim();\n\n const env = process.env.WEBSCRAPING_AI_API_KEY;\n if (env && env.trim() !== '') return env.trim();\n\n const persisted = await readPersistedConfig();\n if (persisted.apiKey && persisted.apiKey.trim() !== '') return persisted.apiKey.trim();\n\n throw new Error(\n 'No API key found. Set $WEBSCRAPING_AI_API_KEY, pass --api-key, or run `webscraping-ai auth set <key>`.',\n );\n}\n\nexport function maskApiKey(key: string): string {\n if (key.length <= 8) return '*'.repeat(key.length);\n return `${key.slice(0, 4)}${'*'.repeat(key.length - 8)}${key.slice(-4)}`;\n}\n","/**\n * Output handling for command results.\n *\n * All command handlers funnel through `emit()`. The result type drives format:\n * - strings → written verbatim\n * - objects/arrays → JSON.stringified (pretty by default; `--pretty` is the\n * default for terminals, single-line for pipes/redirects)\n *\n * `--output FILE` writes to disk instead of stdout.\n */\n\nimport { promises as fs } from 'node:fs';\n\nexport interface EmitOptions {\n output?: string;\n pretty?: boolean;\n}\n\nexport async function emit(value: unknown, opts: EmitOptions = {}): Promise<void> {\n const text = stringify(value, opts);\n\n if (opts.output && opts.output.trim() !== '') {\n await fs.writeFile(opts.output, text + (text.endsWith('\\n') ? '' : '\\n'));\n return;\n }\n\n process.stdout.write(text);\n if (!text.endsWith('\\n')) process.stdout.write('\\n');\n}\n\nexport function stringify(value: unknown, opts: EmitOptions = {}): string {\n if (typeof value === 'string') return value;\n\n const pretty = opts.pretty ?? process.stdout.isTTY ?? false;\n return JSON.stringify(value, null, pretty ? 2 : 0);\n}\n","/**\n * SDK adapter.\n *\n * Wraps the published `webscraping-ai` Node SDK. The CLI sends a\n * `from_cli=true` analytics flag on every request (mirrors `from_n8n` and\n * `from_mcp_server` used by the n8n node and MCP server). The SDK's typed\n * options don't accept arbitrary keys, so we go through a single cast point\n * here rather than scattering casts across each command.\n */\n\nimport { WebScrapingAI } from 'webscraping-ai';\nimport type {\n FieldsOptions,\n HtmlOptions,\n QuestionOptions,\n SelectedMultipleOptions,\n SelectedOptions,\n TextOptions,\n} from 'webscraping-ai';\n\nexport interface CreateClientOptions {\n apiKey: string;\n /** Per-request timeout in ms. Maps to the SDK's `timeoutMs`. */\n requestTimeoutMs?: number;\n}\n\n/**\n * Returns a `WebScrapingAI` instance with the CLI's analytics tag baked into\n * every endpoint method. The methods preserve the SDK's public signatures.\n */\nexport function createClient(options: CreateClientOptions): WebScrapingAI {\n const inner = new WebScrapingAI({\n apiKey: options.apiKey,\n timeoutMs: options.requestTimeoutMs,\n });\n\n // Wrap each endpoint method so it injects the CLI analytics tag. We splice\n // the tag in at this seam rather than at every callsite. The cast is local;\n // the SDK passes unknown query params through to the wire.\n const tag = { from_cli: true } as unknown as Record<string, never>;\n\n return new Proxy(inner, {\n get(target, prop, receiver) {\n const value = Reflect.get(target, prop, receiver);\n if (typeof value !== 'function') return value;\n switch (prop) {\n case 'html':\n return (o: HtmlOptions) => target.html({ ...o, ...tag });\n case 'text':\n return (o: TextOptions) => target.text({ ...o, ...tag });\n case 'selected':\n return (o: SelectedOptions) => target.selected({ ...o, ...tag });\n case 'selectedMultiple':\n return (o: SelectedMultipleOptions) => target.selectedMultiple({ ...o, ...tag });\n case 'question':\n return (o: QuestionOptions) => target.question({ ...o, ...tag });\n case 'fields':\n return (o: FieldsOptions) => target.fields({ ...o, ...tag });\n case 'account':\n // `account` takes no options; the SDK builds the request itself, so\n // there's no seam here for the analytics tag — accepted limitation.\n return value.bind(target);\n default:\n return typeof value === 'function' ? value.bind(target) : value;\n }\n },\n });\n}\n","import { Command, Option } from 'commander';\n\nimport { buildQuestionOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface AskFlags extends CommonRawFlags {\n question?: string;\n}\n\nexport function askCommand(): Command {\n const cmd = new Command('ask')\n .description('Ask an AI question about a page (wraps /ai/question)')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .requiredOption('-q, --question <text>', 'the question / instructions for the model')\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: AskFlags) => {\n if (!opts.question) throw new Error('--question is required');\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildQuestionOptions(target, opts.question, opts, { headers });\n const result = await client.question(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n","/**\n * Pure builder that maps `--flag` values into SDK option objects.\n *\n * Extracted as a separate function so unit tests can pin the mapping without\n * spinning up commander or making HTTP calls. Each command handler reads its\n * own commander options, awaits `parseHeaders` / `parseFields`, then calls\n * the matching `build*` function here.\n */\n\nimport type {\n CommonRequestOptions,\n FieldsOptions,\n HtmlOptions,\n QuestionOptions,\n SelectedMultipleOptions,\n SelectedOptions,\n TextOptions,\n} from 'webscraping-ai';\n\nimport type { CommonRawFlags } from './options.js';\n\nexport interface ResolvedCommonOptions {\n headers?: Record<string, string>;\n}\n\nexport function buildCommonOptions(\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions = {},\n): CommonRequestOptions {\n const out: CommonRequestOptions = {};\n // `flags.js` is `false` when `--no-js`, `true` when `--js`, `undefined`\n // when neither passed. Pass through only when explicit, so the API default\n // applies otherwise.\n if (flags.js !== undefined) out.js = flags.js;\n if (flags.proxy !== undefined) out.proxy = flags.proxy;\n if (flags.country !== undefined) out.country = flags.country;\n if (flags.device !== undefined) out.device = flags.device;\n if (flags.timeout !== undefined) out.timeout = flags.timeout;\n if (flags.jsTimeout !== undefined) out.js_timeout = flags.jsTimeout;\n if (flags.waitFor !== undefined) out.wait_for = flags.waitFor;\n if (flags.jsScript !== undefined) out.js_script = flags.jsScript;\n if (flags.customProxy !== undefined) out.custom_proxy = flags.customProxy;\n if (flags.errorOn404) out.error_on_404 = true;\n if (flags.errorOnRedirect) out.error_on_redirect = true;\n if (resolved.headers) out.headers = resolved.headers;\n return out;\n}\n\nexport interface HtmlExtras {\n returnScriptResult?: boolean;\n}\n\nexport function buildHtmlOptions(\n url: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n extras: HtmlExtras = {},\n): HtmlOptions {\n const out: HtmlOptions = { url, ...buildCommonOptions(flags, resolved) };\n if (flags.format) out.format = flags.format;\n if (extras.returnScriptResult) out.return_script_result = true;\n return out;\n}\n\nexport interface TextExtras {\n textFormat?: 'plain' | 'json' | 'xml';\n returnLinks?: boolean;\n}\n\nexport function buildTextOptions(\n url: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n extras: TextExtras = {},\n): TextOptions {\n const out: TextOptions = { url, ...buildCommonOptions(flags, resolved) };\n if (extras.textFormat) out.text_format = extras.textFormat;\n if (extras.returnLinks) out.return_links = true;\n return out;\n}\n\nexport function buildSelectedOptions(\n url: string,\n selector: string | undefined,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): SelectedOptions {\n // `selector` is optional per the OpenAPI spec: omitting it returns the\n // whole-page HTML. Only attach it when provided. (The installed SDK types\n // still mark it required, so build the object as a partial and cast.)\n const out: Partial<SelectedOptions> = { url, ...buildCommonOptions(flags, resolved) };\n if (selector !== undefined) out.selector = selector;\n if (flags.format) out.format = flags.format;\n return out as SelectedOptions;\n}\n\nexport function buildSelectedMultipleOptions(\n url: string,\n selectors: readonly string[],\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): SelectedMultipleOptions {\n // `selectors` is optional per the OpenAPI spec: omitting it returns the\n // whole-page HTML. Only attach it when non-empty. (The installed SDK types\n // still mark it required, so build the object as a partial and cast.)\n const out: Partial<SelectedMultipleOptions> = { url, ...buildCommonOptions(flags, resolved) };\n if (selectors.length > 0) out.selectors = selectors;\n return out as SelectedMultipleOptions;\n}\n\nexport function buildQuestionOptions(\n url: string,\n question: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): QuestionOptions {\n const out: QuestionOptions = { url, question, ...buildCommonOptions(flags, resolved) };\n if (flags.format) out.format = flags.format;\n return out;\n}\n\nexport function buildFieldsOptions(\n url: string,\n fields: Record<string, string>,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): FieldsOptions {\n return { url, fields, ...buildCommonOptions(flags, resolved) };\n}\n","/**\n * Shared commander options and the parsers/expanders behind them.\n *\n * All scrape endpoints share the same nine \"common\" flags. We define them in\n * one place and apply via `addCommonScrapeOptions(cmd)` so the surface stays\n * consistent and any change lands everywhere at once.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { Option } from 'commander';\nimport type { Command } from 'commander';\n\nexport interface CommonRawFlags {\n apiKey?: string;\n output?: string;\n format?: 'text' | 'json';\n pretty?: boolean;\n js?: boolean;\n proxy?: 'datacenter' | 'residential' | 'stealth';\n country?: string;\n device?: 'desktop' | 'mobile' | 'tablet';\n timeout?: number;\n jsTimeout?: number;\n waitFor?: string;\n headers?: string;\n jsScript?: string;\n customProxy?: string;\n errorOn404?: boolean;\n errorOnRedirect?: boolean;\n}\n\n/**\n * Add the flags every scrape-endpoint command needs. Adding here keeps the\n * surface aligned across `html`, `text`, `selected`, etc.\n */\nexport function addCommonScrapeOptions(cmd: Command): Command {\n return cmd\n .option('-k, --api-key <key>', 'API key (overrides $WEBSCRAPING_AI_API_KEY and config file)')\n .option('-o, --output <file>', 'write result to FILE instead of stdout')\n .option('--pretty', 'pretty-print JSON output (default on TTY)')\n .addOption(\n new Option('--no-pretty', 'force single-line JSON output').conflicts('pretty'),\n )\n .option('--js', 'execute JS via a headless browser (default: enabled)')\n .option('--no-js', 'disable JS execution (faster, cheaper)')\n .addOption(\n new Option('--proxy <type>', 'proxy pool').choices([\n 'datacenter',\n 'residential',\n 'stealth',\n ]),\n )\n .option('--country <code>', 'two-letter proxy country code, e.g. us, gb, de')\n .addOption(\n new Option('--device <type>', 'device emulation').choices(['desktop', 'mobile', 'tablet']),\n )\n .option('--timeout <ms>', 'page-retrieval timeout in ms (max 30000)', integerParser)\n .option('--js-timeout <ms>', 'JS rendering time in ms after load (max 20000)', integerParser)\n .option('--wait-for <selector>', 'CSS selector to wait for before returning content')\n .option(\n '--headers <json>',\n 'HTTP headers to forward to the target page; JSON object or @file.json',\n )\n .option('--js-script <code>', 'custom JS to evaluate on the target page')\n .option('--custom-proxy <url>', 'custom proxy URL: http://user:pass@host:port')\n .option('--error-on-404', 'return error on 404')\n .option('--error-on-redirect', 'return error on redirect');\n}\n\nexport function integerParser(value: string): number {\n const n = Number.parseInt(value, 10);\n if (!Number.isFinite(n)) throw new Error(`expected integer, got: ${value}`);\n return n;\n}\n\n/**\n * Parse a `--headers` argument. Accepts a JSON object literal or `@path` to\n * load JSON from a file. Returns `undefined` if the input is empty.\n */\nexport async function parseHeaders(input: string | undefined): Promise<Record<string, string> | undefined> {\n if (!input || input.trim() === '') return undefined;\n const text = input.startsWith('@') ? await fs.readFile(input.slice(1), 'utf8') : input;\n return parseJsonObject(text, 'headers');\n}\n\n/**\n * Parse a `--fields` argument for `extract`. Accepts JSON object literal or\n * `@path` to a file.\n */\nexport async function parseFields(input: string): Promise<Record<string, string>> {\n const text = input.startsWith('@') ? await fs.readFile(input.slice(1), 'utf8') : input;\n const parsed = parseJsonObject(text, 'fields');\n if (!parsed || Object.keys(parsed).length === 0) {\n throw new Error('fields must be a non-empty JSON object of { fieldName: description }');\n }\n return parsed;\n}\n\nfunction parseJsonObject(text: string, label: string): Record<string, string> {\n let parsed: unknown;\n try {\n parsed = JSON.parse(text);\n } catch (err) {\n throw new Error(`${label}: invalid JSON — ${(err as Error).message}`);\n }\n if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {\n throw new Error(`${label} must be a JSON object`);\n }\n const out: Record<string, string> = {};\n for (const [k, v] of Object.entries(parsed as Record<string, unknown>)) {\n if (typeof v !== 'string') throw new Error(`${label}.${k}: value must be a string`);\n out[k] = v;\n }\n return out;\n}\n","/**\n * Stdin batching helpers.\n *\n * When a URL argument is `-`, the command reads one URL per line from stdin\n * and runs the operation for each. Empty lines and lines starting with `#`\n * are skipped so users can comment URL lists.\n */\n\nimport readline from 'node:readline';\n\nexport function urlIsStdin(url: string): boolean {\n return url === '-';\n}\n\nexport async function* readUrlsFromStdin(): AsyncIterable<string> {\n const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });\n for await (const raw of rl) {\n const line = raw.trim();\n if (line === '' || line.startsWith('#')) continue;\n yield line;\n }\n}\n\n/**\n * Iterate URLs from either a single CLI arg or from stdin (if the arg is `-`).\n */\nexport async function* urlIterator(arg: string): AsyncIterable<string> {\n if (urlIsStdin(arg)) {\n yield* readUrlsFromStdin();\n return;\n }\n yield arg;\n}\n","import { Command } from 'commander';\n\nimport {\n clearPersistedConfig,\n configPath,\n maskApiKey,\n readPersistedConfig,\n writePersistedConfig,\n} from '../lib/config.js';\n\nexport function authCommand(): Command {\n const cmd = new Command('auth').description('Manage your stored API key');\n\n cmd\n .command('set <key>')\n .description(`Save an API key to ${configPath()}`)\n .action(async (key: string) => {\n if (!key || key.trim() === '') throw new Error('API key is empty');\n await writePersistedConfig({ apiKey: key.trim() });\n process.stdout.write(`API key saved to ${configPath()}\\n`);\n });\n\n cmd\n .command('show')\n .description('Display the currently stored API key (masked)')\n .action(async () => {\n const persisted = await readPersistedConfig();\n const env = process.env.WEBSCRAPING_AI_API_KEY;\n if (env && env.trim() !== '') {\n process.stdout.write(`Active key (env WEBSCRAPING_AI_API_KEY): ${maskApiKey(env.trim())}\\n`);\n }\n if (persisted.apiKey && persisted.apiKey.trim() !== '') {\n process.stdout.write(\n `Stored key (${configPath()}): ${maskApiKey(persisted.apiKey.trim())}\\n`,\n );\n }\n if (!env && !persisted.apiKey) {\n process.stdout.write('No API key configured. Run `webscraping-ai auth set <key>`.\\n');\n }\n });\n\n cmd\n .command('clear')\n .description('Remove the stored API key file')\n .action(async () => {\n const removed = await clearPersistedConfig();\n process.stdout.write(removed ? `Removed ${configPath()}\\n` : 'No stored API key to remove.\\n');\n });\n\n return cmd;\n}\n","import { Command } from 'commander';\n\nimport { buildFieldsOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseFields, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface ExtractFlags extends CommonRawFlags {\n fields?: string;\n}\n\nexport function extractCommand(): Command {\n const cmd = new Command('extract')\n .description('Extract structured fields from a page using AI (wraps /ai/fields)')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .requiredOption(\n '-f, --fields <json>',\n 'fields to extract — JSON object {name: description} or @file.json',\n );\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: ExtractFlags) => {\n if (!opts.fields) throw new Error('--fields is required');\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const fields = await parseFields(opts.fields);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildFieldsOptions(target, fields, opts, { headers });\n const result = await client.fields(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n","import { Command, Option } from 'commander';\n\nimport { buildHtmlOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface HtmlFlags extends CommonRawFlags {\n returnScriptResult?: boolean;\n}\n\nexport function htmlCommand(): Command {\n const cmd = new Command('html')\n .description('Fetch the full HTML of a page')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .addOption(new Option('--return-script-result', 'return the result of --js-script instead of HTML'))\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: HtmlFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildHtmlOptions(target, opts, { headers }, {\n returnScriptResult: opts.returnScriptResult,\n });\n const result = await client.html(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n","import { Command, Option } from 'commander';\n\nimport { buildSelectedOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface SelectedFlags extends CommonRawFlags {\n selector?: string;\n}\n\nexport function selectedCommand(): Command {\n const cmd = new Command('selected')\n .description('Fetch HTML of one CSS-selected page area')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .option(\n '-s, --selector <css>',\n 'CSS selector for the target element(s); omit for whole-page HTML',\n )\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: SelectedFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildSelectedOptions(target, opts.selector, opts, { headers });\n const result = await client.selected(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n","import { Command } from 'commander';\n\nimport { buildSelectedMultipleOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface SelectedMultipleFlags extends CommonRawFlags {\n selector?: string[];\n}\n\nexport function selectedMultipleCommand(): Command {\n const cmd = new Command('selected-multiple')\n .description('Fetch HTML of multiple CSS-selected page areas')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .option(\n '-s, --selector <css>',\n 'CSS selector — pass multiple times for multiple selectors',\n collectMulti,\n [] as string[],\n );\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: SelectedMultipleFlags) => {\n // `--selector` is optional: omitting it returns whole-page HTML.\n const selectors = opts.selector ?? [];\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildSelectedMultipleOptions(target, selectors, opts, { headers });\n const result = await client.selectedMultiple(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n\nfunction collectMulti(value: string, previous: string[]): string[] {\n return [...previous, value];\n}\n","import { Command } from 'commander';\n\nimport { EDITORS, editorById, installSkill, resolveTarget } from '../lib/skill.js';\nimport type { EditorAdapter } from '../lib/skill.js';\n\ninterface SetupSkillFlags {\n editor?: string[];\n all?: boolean;\n force?: boolean;\n}\n\nexport function setupCommand(): Command {\n const cmd = new Command('setup').description('Install the AI agent skill into your editor(s)');\n\n cmd\n .command('skill')\n .description('Install the WebScraping.AI skill into one or more AI coding editors')\n .option(\n '-e, --editor <id>',\n `editor id (repeatable). Supported: ${EDITORS.map((e) => e.id).join(', ')}`,\n collect,\n [] as string[],\n )\n .option('-a, --all', 'install into every supported editor')\n .option('-f, --force', 'overwrite existing skill files')\n .action(async (opts: SetupSkillFlags) => {\n const selected = resolveSelection(opts);\n if (selected.length === 0) {\n throw new Error(\n 'no editor selected — pass --all or --editor <id> (e.g. --editor claude-code)',\n );\n }\n\n for (const editor of selected) {\n const target = resolveTarget(editor);\n const result = await installSkill(target, { force: opts.force });\n process.stdout.write(`${editor.label}: ${result.status} → ${target.destination}\\n`);\n }\n });\n\n cmd\n .command('list', { hidden: false })\n .description('List supported editors for `setup skill`')\n .action(() => {\n for (const e of EDITORS) {\n const target = resolveTarget(e);\n process.stdout.write(`${e.id.padEnd(14)} ${e.label.padEnd(18)} ${target.destination}\\n`);\n }\n });\n\n return cmd;\n}\n\nfunction collect(value: string, previous: string[]): string[] {\n return [...previous, value];\n}\n\nfunction resolveSelection(opts: SetupSkillFlags): EditorAdapter[] {\n if (opts.all) return [...EDITORS];\n const ids = opts.editor ?? [];\n return ids.map((id) => {\n const editor = editorById(id);\n if (!editor) throw new Error(`unknown editor: ${id}`);\n return editor;\n });\n}\n","/**\n * Multi-editor skill installer.\n *\n * The canonical skill lives in this package at\n * `plugins/webscraping-ai/skills/webscraping-ai/SKILL.md`. Each supported\n * editor reads skill content from a different location and may want either\n * a single Markdown file or a whole skill directory. The `editors` registry\n * captures those differences so `setup skill` is a small driver over it.\n *\n * `mode: 'dir'` copies the entire skill directory (Claude Code-shaped\n * editors that scan a folder per skill). `mode: 'file'` writes the SKILL.md\n * contents to a single file (editors that read flat instructions, like\n * GitHub Copilot's `.github/copilot-instructions.md`).\n *\n * `kind: 'user'` writes to `$HOME`; `kind: 'project'` writes to the current\n * working directory. We prefer user-scope where the editor supports it so\n * the skill becomes globally available without polluting individual repos.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { homedir } from 'node:os';\nimport { dirname, join, resolve } from 'node:path';\nimport { fileURLToPath } from 'node:url';\n\nexport type EditorMode = 'dir' | 'file';\nexport type EditorKind = 'user' | 'project';\n\nexport interface EditorAdapter {\n /** Stable identifier used by `--editor <name>`. */\n id: string;\n /** Human-readable label printed in summaries. */\n label: string;\n mode: EditorMode;\n kind: EditorKind;\n /**\n * Where the skill ends up. For `mode: 'dir'`, the skill directory copies\n * to `{root}/{path}`. For `mode: 'file'`, the SKILL.md contents write to\n * `{root}/{path}` (i.e. `path` ends in a filename).\n *\n * `root` is `$HOME` (user) or CWD (project).\n */\n path: string;\n}\n\n/**\n * Registry of supported editors. Order matters: it's also the order the\n * `setup skill` summary lists them in.\n */\nexport const EDITORS: readonly EditorAdapter[] = [\n // Claude Code: user-scope skills live under ~/.claude/skills/<name>/\n { id: 'claude-code', label: 'Claude Code', mode: 'dir', kind: 'user', path: '.claude/skills/webscraping-ai' },\n // Cursor: project-scope rule file. Cursor doesn't have a user-wide skill\n // dir, so this is the editor's \"install\" surface.\n { id: 'cursor', label: 'Cursor', mode: 'file', kind: 'project', path: '.cursor/rules/webscraping-ai.mdc' },\n // Windsurf: project-scope rules. Same shape as Cursor.\n { id: 'windsurf', label: 'Windsurf', mode: 'file', kind: 'project', path: '.windsurf/rules/webscraping-ai.md' },\n // Kiro: user-scope steering directory.\n { id: 'kiro', label: 'Kiro', mode: 'dir', kind: 'user', path: '.kiro/steering/webscraping-ai' },\n // OpenCode: project-scope skills (mirrors Claude shape).\n { id: 'opencode', label: 'OpenCode', mode: 'dir', kind: 'project', path: '.opencode/skills/webscraping-ai' },\n // Gemini CLI: user-scope extension dir with a GEMINI.md.\n { id: 'gemini', label: 'Gemini CLI', mode: 'file', kind: 'user', path: '.gemini/extensions/webscraping-ai/GEMINI.md' },\n // GitHub Copilot: project-scope custom instructions file. Single file.\n { id: 'copilot', label: 'GitHub Copilot', mode: 'file', kind: 'project', path: '.github/copilot-instructions.md' },\n // Augment: project-scope rules file.\n { id: 'augment', label: 'Augment', mode: 'file', kind: 'project', path: '.augment/rules/webscraping-ai.md' },\n // Factory: project-scope rules file.\n { id: 'factory', label: 'Factory', mode: 'file', kind: 'project', path: '.factory/rules/webscraping-ai.md' },\n] as const;\n\nexport function editorById(id: string): EditorAdapter | undefined {\n return EDITORS.find((e) => e.id === id);\n}\n\n/**\n * Where this package keeps the canonical skill. Resolved relative to the\n * compiled `dist/cli.js` so it works after `npm install -g`.\n */\nexport function bundledSkillDir(): string {\n const here = dirname(fileURLToPath(import.meta.url));\n // dist/ sits at the package root, so the plugins dir is two levels up\n // from any compiled file: dist/cli.js → package root → plugins/...\n return resolve(here, '..', 'plugins', 'webscraping-ai', 'skills', 'webscraping-ai');\n}\n\nexport interface InstallTarget {\n editor: EditorAdapter;\n /** Absolute destination path (file or directory depending on `mode`). */\n destination: string;\n}\n\nexport interface ResolveOptions {\n cwd?: string;\n home?: string;\n}\n\nexport function resolveTarget(editor: EditorAdapter, opts: ResolveOptions = {}): InstallTarget {\n const root = editor.kind === 'user' ? (opts.home ?? homedir()) : (opts.cwd ?? process.cwd());\n return { editor, destination: join(root, editor.path) };\n}\n\nexport interface InstallOptions {\n /** If true, overwrite existing files/dirs. */\n force?: boolean;\n /** Override skill source dir (for tests). */\n sourceDir?: string;\n}\n\nexport type InstallStatus = 'installed' | 'skipped-exists' | 'overwritten';\n\nexport interface InstallResult {\n target: InstallTarget;\n status: InstallStatus;\n}\n\nexport async function installSkill(\n target: InstallTarget,\n opts: InstallOptions = {},\n): Promise<InstallResult> {\n const source = opts.sourceDir ?? bundledSkillDir();\n const exists = await pathExists(target.destination);\n\n if (exists && !opts.force) {\n return { target, status: 'skipped-exists' };\n }\n\n if (target.editor.mode === 'dir') {\n if (exists) await fs.rm(target.destination, { recursive: true, force: true });\n await copyDirectory(source, target.destination);\n } else {\n const skillBody = await fs.readFile(join(source, 'SKILL.md'), 'utf8');\n await fs.mkdir(dirname(target.destination), { recursive: true });\n await fs.writeFile(target.destination, skillBody);\n }\n\n return { target, status: exists ? 'overwritten' : 'installed' };\n}\n\nasync function pathExists(path: string): Promise<boolean> {\n try {\n await fs.access(path);\n return true;\n } catch {\n return false;\n }\n}\n\nasync function copyDirectory(source: string, destination: string): Promise<void> {\n // `fs.cp` with `recursive` exists on Node 16.7+; we target Node 20.\n await fs.cp(source, destination, { recursive: true, force: true });\n}\n","import { Command, Option } from 'commander';\n\nimport { buildTextOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface TextFlags extends CommonRawFlags {\n textFormat?: 'plain' | 'json' | 'xml';\n returnLinks?: boolean;\n}\n\nexport function textCommand(): Command {\n const cmd = new Command('text')\n .description('Fetch the visible text of a page')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .addOption(\n new Option('--text-format <fmt>', 'text response format').choices(['plain', 'json', 'xml']),\n )\n .option('--return-links', 'include page links (text-format=json only)');\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: TextFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildTextOptions(target, opts, { headers }, {\n textFormat: opts.textFormat,\n returnLinks: opts.returnLinks,\n });\n const result = await client.text(sdkOptions);\n await emit(result, opts);\n }\n });\n\n return cmd;\n}\n","/**\n * Exit-code mapping for SDK errors.\n *\n * The SDK throws typed subclasses of `WebScrapingAIError`. Map each to a\n * stable exit code so shell scripts can branch on the failure mode without\n * parsing stderr.\n */\n\nimport {\n APIConnectionError,\n APIError,\n APITimeoutError,\n AuthenticationError,\n BadRequestError,\n GatewayTimeoutError,\n PaymentRequiredError,\n RateLimitError,\n ServerError,\n WebScrapingAIError,\n} from 'webscraping-ai';\n\nexport const EXIT_CODES = {\n ok: 0,\n generic: 1,\n usage: 2,\n badRequest: 3,\n auth: 4,\n paymentRequired: 5,\n rateLimit: 6,\n serverError: 7,\n timeout: 8,\n connection: 9,\n} as const;\n\nexport function exitCodeFor(err: unknown): number {\n if (err instanceof BadRequestError) return EXIT_CODES.badRequest;\n if (err instanceof AuthenticationError) return EXIT_CODES.auth;\n if (err instanceof PaymentRequiredError) return EXIT_CODES.paymentRequired;\n if (err instanceof RateLimitError) return EXIT_CODES.rateLimit;\n if (err instanceof ServerError) return EXIT_CODES.serverError;\n if (err instanceof GatewayTimeoutError) return EXIT_CODES.timeout;\n if (err instanceof APITimeoutError) return EXIT_CODES.timeout;\n if (err instanceof APIConnectionError) return EXIT_CODES.connection;\n if (err instanceof APIError) return EXIT_CODES.serverError;\n if (err instanceof WebScrapingAIError) return EXIT_CODES.generic;\n return EXIT_CODES.generic;\n}\n\nexport function formatError(err: unknown): string {\n if (err instanceof APIError) {\n const parts = [err.message];\n if (err.status) parts.push(`(HTTP ${err.status})`);\n return parts.join(' ');\n }\n if (err instanceof Error) return err.message;\n return String(err);\n}\n"],"mappings":";;;AAQA,SAAS,WAAAA,iBAAe;AACxB,SAAS,oBAAoB;AAC7B,SAAS,WAAAC,UAAS,WAAAC,gBAAe;AACjC,SAAS,iBAAAC,sBAAqB;;;ACX9B,SAAS,eAAe;;;ACYxB,SAAS,YAAY,UAAU;AAC/B,SAAS,eAAe;AACxB,SAAS,SAAS,YAAY;AAE9B,IAAM,kBAAkB;AAEjB,SAAS,YAAoB;AAClC,QAAM,MAAM,QAAQ,IAAI;AACxB,MAAI,OAAO,IAAI,KAAK,MAAM,GAAI,QAAO,KAAK,KAAK,gBAAgB;AAC/D,SAAO,KAAK,QAAQ,GAAG,WAAW,gBAAgB;AACpD;AAEO,SAAS,aAAqB;AACnC,SAAO,KAAK,UAAU,GAAG,eAAe;AAC1C;AAMA,eAAsB,sBAAgD;AACpE,MAAI;AACF,UAAM,OAAO,MAAM,GAAG,SAAS,WAAW,GAAG,MAAM;AACnD,UAAM,SAAS,KAAK,MAAM,IAAI;AAC9B,QAAI,UAAU,OAAO,WAAW,YAAY,CAAC,MAAM,QAAQ,MAAM,GAAG;AAClE,aAAO;AAAA,IACT;AACA,WAAO,CAAC;AAAA,EACV,SAAS,KAAK;AACZ,QAAK,IAA8B,SAAS,SAAU,QAAO,CAAC;AAC9D,UAAM;AAAA,EACR;AACF;AAEA,eAAsB,qBAAqB,QAAwC;AACjF,QAAM,OAAO,WAAW;AACxB,QAAM,GAAG,MAAM,QAAQ,IAAI,GAAG,EAAE,WAAW,KAAK,CAAC;AACjD,QAAM,GAAG,UAAU,MAAM,KAAK,UAAU,QAAQ,MAAM,CAAC,IAAI,MAAM,EAAE,MAAM,IAAM,CAAC;AAKhF,QAAM,GAAG,MAAM,MAAM,GAAK;AAC5B;AAEA,eAAsB,uBAAyC;AAC7D,MAAI;AACF,UAAM,GAAG,OAAO,WAAW,CAAC;AAC5B,WAAO;AAAA,EACT,SAAS,KAAK;AACZ,QAAK,IAA8B,SAAS,SAAU,QAAO;AAC7D,UAAM;AAAA,EACR;AACF;AAOA,eAAsB,cAAc,OAA6B,CAAC,GAAoB;AACpF,MAAI,KAAK,QAAQ,KAAK,KAAK,KAAK,MAAM,GAAI,QAAO,KAAK,KAAK,KAAK;AAEhE,QAAM,MAAM,QAAQ,IAAI;AACxB,MAAI,OAAO,IAAI,KAAK,MAAM,GAAI,QAAO,IAAI,KAAK;AAE9C,QAAM,YAAY,MAAM,oBAAoB;AAC5C,MAAI,UAAU,UAAU,UAAU,OAAO,KAAK,MAAM,GAAI,QAAO,UAAU,OAAO,KAAK;AAErF,QAAM,IAAI;AAAA,IACR;AAAA,EACF;AACF;AAEO,SAAS,WAAW,KAAqB;AAC9C,MAAI,IAAI,UAAU,EAAG,QAAO,IAAI,OAAO,IAAI,MAAM;AACjD,SAAO,GAAG,IAAI,MAAM,GAAG,CAAC,CAAC,GAAG,IAAI,OAAO,IAAI,SAAS,CAAC,CAAC,GAAG,IAAI,MAAM,EAAE,CAAC;AACxE;;;AC9EA,SAAS,YAAYC,WAAU;AAO/B,eAAsB,KAAK,OAAgB,OAAoB,CAAC,GAAkB;AAChF,QAAM,OAAO,UAAU,OAAO,IAAI;AAElC,MAAI,KAAK,UAAU,KAAK,OAAO,KAAK,MAAM,IAAI;AAC5C,UAAMA,IAAG,UAAU,KAAK,QAAQ,QAAQ,KAAK,SAAS,IAAI,IAAI,KAAK,KAAK;AACxE;AAAA,EACF;AAEA,UAAQ,OAAO,MAAM,IAAI;AACzB,MAAI,CAAC,KAAK,SAAS,IAAI,EAAG,SAAQ,OAAO,MAAM,IAAI;AACrD;AAEO,SAAS,UAAU,OAAgB,OAAoB,CAAC,GAAW;AACxE,MAAI,OAAO,UAAU,SAAU,QAAO;AAEtC,QAAM,SAAS,KAAK,UAAU,QAAQ,OAAO,SAAS;AACtD,SAAO,KAAK,UAAU,OAAO,MAAM,SAAS,IAAI,CAAC;AACnD;;;ACzBA,SAAS,qBAAqB;AAoBvB,SAAS,aAAa,SAA6C;AACxE,QAAM,QAAQ,IAAI,cAAc;AAAA,IAC9B,QAAQ,QAAQ;AAAA,IAChB,WAAW,QAAQ;AAAA,EACrB,CAAC;AAKD,QAAM,MAAM,EAAE,UAAU,KAAK;AAE7B,SAAO,IAAI,MAAM,OAAO;AAAA,IACtB,IAAI,QAAQ,MAAM,UAAU;AAC1B,YAAM,QAAQ,QAAQ,IAAI,QAAQ,MAAM,QAAQ;AAChD,UAAI,OAAO,UAAU,WAAY,QAAO;AACxC,cAAQ,MAAM;AAAA,QACZ,KAAK;AACH,iBAAO,CAAC,MAAmB,OAAO,KAAK,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACzD,KAAK;AACH,iBAAO,CAAC,MAAmB,OAAO,KAAK,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACzD,KAAK;AACH,iBAAO,CAAC,MAAuB,OAAO,SAAS,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjE,KAAK;AACH,iBAAO,CAAC,MAA+B,OAAO,iBAAiB,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjF,KAAK;AACH,iBAAO,CAAC,MAAuB,OAAO,SAAS,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjE,KAAK;AACH,iBAAO,CAAC,MAAqB,OAAO,OAAO,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QAC7D,KAAK;AAGH,iBAAO,MAAM,KAAK,MAAM;AAAA,QAC1B;AACE,iBAAO,OAAO,UAAU,aAAa,MAAM,KAAK,MAAM,IAAI;AAAA,MAC9D;AAAA,IACF;AAAA,EACF,CAAC;AACH;;;AHvDO,SAAS,iBAA0B;AACxC,SAAO,IAAI,QAAQ,SAAS,EACzB,YAAY,0CAA0C,EACtD,OAAO,uBAAuB,6DAA6D,EAC3F,OAAO,uBAAuB,wCAAwC,EACtE,OAAO,YAAY,2CAA2C,EAC9D,OAAO,eAAe,+BAA+B,EACrD,OAAO,OAAO,SAAuB;AACpC,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,SAAS,aAAa,EAAE,OAAO,CAAC;AACtC,UAAM,SAAS,MAAM,OAAO,QAAQ;AACpC,UAAM,KAAK,QAAQ,IAAI;AAAA,EACzB,CAAC;AACL;;;AIzBA,SAAS,WAAAC,UAAS,UAAAC,eAAc;;;ACyBzB,SAAS,mBACd,OACA,WAAkC,CAAC,GACb;AACtB,QAAM,MAA4B,CAAC;AAInC,MAAI,MAAM,OAAO,OAAW,KAAI,KAAK,MAAM;AAC3C,MAAI,MAAM,UAAU,OAAW,KAAI,QAAQ,MAAM;AACjD,MAAI,MAAM,YAAY,OAAW,KAAI,UAAU,MAAM;AACrD,MAAI,MAAM,WAAW,OAAW,KAAI,SAAS,MAAM;AACnD,MAAI,MAAM,YAAY,OAAW,KAAI,UAAU,MAAM;AACrD,MAAI,MAAM,cAAc,OAAW,KAAI,aAAa,MAAM;AAC1D,MAAI,MAAM,YAAY,OAAW,KAAI,WAAW,MAAM;AACtD,MAAI,MAAM,aAAa,OAAW,KAAI,YAAY,MAAM;AACxD,MAAI,MAAM,gBAAgB,OAAW,KAAI,eAAe,MAAM;AAC9D,MAAI,MAAM,WAAY,KAAI,eAAe;AACzC,MAAI,MAAM,gBAAiB,KAAI,oBAAoB;AACnD,MAAI,SAAS,QAAS,KAAI,UAAU,SAAS;AAC7C,SAAO;AACT;AAMO,SAAS,iBACd,KACA,OACA,UACA,SAAqB,CAAC,GACT;AACb,QAAM,MAAmB,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACvE,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,MAAI,OAAO,mBAAoB,KAAI,uBAAuB;AAC1D,SAAO;AACT;AAOO,SAAS,iBACd,KACA,OACA,UACA,SAAqB,CAAC,GACT;AACb,QAAM,MAAmB,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACvE,MAAI,OAAO,WAAY,KAAI,cAAc,OAAO;AAChD,MAAI,OAAO,YAAa,KAAI,eAAe;AAC3C,SAAO;AACT;AAEO,SAAS,qBACd,KACA,UACA,OACA,UACiB;AAIjB,QAAM,MAAgC,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACpF,MAAI,aAAa,OAAW,KAAI,WAAW;AAC3C,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,SAAO;AACT;AAEO,SAAS,6BACd,KACA,WACA,OACA,UACyB;AAIzB,QAAM,MAAwC,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AAC5F,MAAI,UAAU,SAAS,EAAG,KAAI,YAAY;AAC1C,SAAO;AACT;AAEO,SAAS,qBACd,KACA,UACA,OACA,UACiB;AACjB,QAAM,MAAuB,EAAE,KAAK,UAAU,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACrF,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,SAAO;AACT;AAEO,SAAS,mBACd,KACA,QACA,OACA,UACe;AACf,SAAO,EAAE,KAAK,QAAQ,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AAC/D;;;ACxHA,SAAS,YAAYC,WAAU;AAC/B,SAAS,cAAc;AA0BhB,SAAS,uBAAuB,KAAuB;AAC5D,SAAO,IACJ,OAAO,uBAAuB,6DAA6D,EAC3F,OAAO,uBAAuB,wCAAwC,EACtE,OAAO,YAAY,2CAA2C,EAC9D;AAAA,IACC,IAAI,OAAO,eAAe,+BAA+B,EAAE,UAAU,QAAQ;AAAA,EAC/E,EACC,OAAO,QAAQ,sDAAsD,EACrE,OAAO,WAAW,wCAAwC,EAC1D;AAAA,IACC,IAAI,OAAO,kBAAkB,YAAY,EAAE,QAAQ;AAAA,MACjD;AAAA,MACA;AAAA,MACA;AAAA,IACF,CAAC;AAAA,EACH,EACC,OAAO,oBAAoB,gDAAgD,EAC3E;AAAA,IACC,IAAI,OAAO,mBAAmB,kBAAkB,EAAE,QAAQ,CAAC,WAAW,UAAU,QAAQ,CAAC;AAAA,EAC3F,EACC,OAAO,kBAAkB,4CAA4C,aAAa,EAClF,OAAO,qBAAqB,kDAAkD,aAAa,EAC3F,OAAO,yBAAyB,mDAAmD,EACnF;AAAA,IACC;AAAA,IACA;AAAA,EACF,EACC,OAAO,sBAAsB,0CAA0C,EACvE,OAAO,wBAAwB,8CAA8C,EAC7E,OAAO,kBAAkB,qBAAqB,EAC9C,OAAO,uBAAuB,0BAA0B;AAC7D;AAEO,SAAS,cAAc,OAAuB;AACnD,QAAM,IAAI,OAAO,SAAS,OAAO,EAAE;AACnC,MAAI,CAAC,OAAO,SAAS,CAAC,EAAG,OAAM,IAAI,MAAM,0BAA0B,KAAK,EAAE;AAC1E,SAAO;AACT;AAMA,eAAsB,aAAa,OAAwE;AACzG,MAAI,CAAC,SAAS,MAAM,KAAK,MAAM,GAAI,QAAO;AAC1C,QAAM,OAAO,MAAM,WAAW,GAAG,IAAI,MAAMA,IAAG,SAAS,MAAM,MAAM,CAAC,GAAG,MAAM,IAAI;AACjF,SAAO,gBAAgB,MAAM,SAAS;AACxC;AAMA,eAAsB,YAAY,OAAgD;AAChF,QAAM,OAAO,MAAM,WAAW,GAAG,IAAI,MAAMA,IAAG,SAAS,MAAM,MAAM,CAAC,GAAG,MAAM,IAAI;AACjF,QAAM,SAAS,gBAAgB,MAAM,QAAQ;AAC7C,MAAI,CAAC,UAAU,OAAO,KAAK,MAAM,EAAE,WAAW,GAAG;AAC/C,UAAM,IAAI,MAAM,sEAAsE;AAAA,EACxF;AACA,SAAO;AACT;AAEA,SAAS,gBAAgB,MAAc,OAAuC;AAC5E,MAAI;AACJ,MAAI;AACF,aAAS,KAAK,MAAM,IAAI;AAAA,EAC1B,SAAS,KAAK;AACZ,UAAM,IAAI,MAAM,GAAG,KAAK,yBAAqB,IAAc,OAAO,EAAE;AAAA,EACtE;AACA,MAAI,CAAC,UAAU,OAAO,WAAW,YAAY,MAAM,QAAQ,MAAM,GAAG;AAClE,UAAM,IAAI,MAAM,GAAG,KAAK,wBAAwB;AAAA,EAClD;AACA,QAAM,MAA8B,CAAC;AACrC,aAAW,CAAC,GAAG,CAAC,KAAK,OAAO,QAAQ,MAAiC,GAAG;AACtE,QAAI,OAAO,MAAM,SAAU,OAAM,IAAI,MAAM,GAAG,KAAK,IAAI,CAAC,0BAA0B;AAClF,QAAI,CAAC,IAAI;AAAA,EACX;AACA,SAAO;AACT;;;AC1GA,OAAO,cAAc;AAEd,SAAS,WAAW,KAAsB;AAC/C,SAAO,QAAQ;AACjB;AAEA,gBAAuB,oBAA2C;AAChE,QAAM,KAAK,SAAS,gBAAgB,EAAE,OAAO,QAAQ,OAAO,WAAW,SAAS,CAAC;AACjF,mBAAiB,OAAO,IAAI;AAC1B,UAAM,OAAO,IAAI,KAAK;AACtB,QAAI,SAAS,MAAM,KAAK,WAAW,GAAG,EAAG;AACzC,UAAM;AAAA,EACR;AACF;AAKA,gBAAuB,YAAY,KAAoC;AACrE,MAAI,WAAW,GAAG,GAAG;AACnB,WAAO,kBAAkB;AACzB;AAAA,EACF;AACA,QAAM;AACR;;;AHlBO,SAAS,aAAsB;AACpC,QAAM,MAAM,IAAIC,SAAQ,KAAK,EAC1B,YAAY,sDAAsD,EAClE,SAAS,SAAS,2CAA2C,EAC7D,eAAe,yBAAyB,2CAA2C,EACnF,UAAU,IAAIC,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAmB;AAChD,QAAI,CAAC,KAAK,SAAU,OAAM,IAAI,MAAM,wBAAwB;AAC5D,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,qBAAqB,QAAQ,KAAK,UAAU,MAAM,EAAE,QAAQ,CAAC;AAChF,YAAM,SAAS,MAAM,OAAO,SAAS,UAAU;AAC/C,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AIrCA,SAAS,WAAAC,gBAAe;AAUjB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAAE,YAAY,4BAA4B;AAExE,MACG,QAAQ,WAAW,EACnB,YAAY,sBAAsB,WAAW,CAAC,EAAE,EAChD,OAAO,OAAO,QAAgB;AAC7B,QAAI,CAAC,OAAO,IAAI,KAAK,MAAM,GAAI,OAAM,IAAI,MAAM,kBAAkB;AACjE,UAAM,qBAAqB,EAAE,QAAQ,IAAI,KAAK,EAAE,CAAC;AACjD,YAAQ,OAAO,MAAM,oBAAoB,WAAW,CAAC;AAAA,CAAI;AAAA,EAC3D,CAAC;AAEH,MACG,QAAQ,MAAM,EACd,YAAY,+CAA+C,EAC3D,OAAO,YAAY;AAClB,UAAM,YAAY,MAAM,oBAAoB;AAC5C,UAAM,MAAM,QAAQ,IAAI;AACxB,QAAI,OAAO,IAAI,KAAK,MAAM,IAAI;AAC5B,cAAQ,OAAO,MAAM,4CAA4C,WAAW,IAAI,KAAK,CAAC,CAAC;AAAA,CAAI;AAAA,IAC7F;AACA,QAAI,UAAU,UAAU,UAAU,OAAO,KAAK,MAAM,IAAI;AACtD,cAAQ,OAAO;AAAA,QACb,eAAe,WAAW,CAAC,MAAM,WAAW,UAAU,OAAO,KAAK,CAAC,CAAC;AAAA;AAAA,MACtE;AAAA,IACF;AACA,QAAI,CAAC,OAAO,CAAC,UAAU,QAAQ;AAC7B,cAAQ,OAAO,MAAM,+DAA+D;AAAA,IACtF;AAAA,EACF,CAAC;AAEH,MACG,QAAQ,OAAO,EACf,YAAY,gCAAgC,EAC5C,OAAO,YAAY;AAClB,UAAM,UAAU,MAAM,qBAAqB;AAC3C,YAAQ,OAAO,MAAM,UAAU,WAAW,WAAW,CAAC;AAAA,IAAO,gCAAgC;AAAA,EAC/F,CAAC;AAEH,SAAO;AACT;;;AClDA,SAAS,WAAAC,gBAAe;AAcjB,SAAS,iBAA0B;AACxC,QAAM,MAAM,IAAIC,SAAQ,SAAS,EAC9B,YAAY,mEAAmE,EAC/E,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,EACF;AAEF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAuB;AACpD,QAAI,CAAC,KAAK,OAAQ,OAAM,IAAI,MAAM,sBAAsB;AACxD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,MAAM,YAAY,KAAK,MAAM;AAC5C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,mBAAmB,QAAQ,QAAQ,MAAM,EAAE,QAAQ,CAAC;AACvE,YAAM,SAAS,MAAM,OAAO,OAAO,UAAU;AAC7C,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;ACxCA,SAAS,WAAAC,UAAS,UAAAC,eAAc;AAczB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAC3B,YAAY,+BAA+B,EAC3C,SAAS,SAAS,2CAA2C,EAC7D,UAAU,IAAIC,QAAO,0BAA0B,kDAAkD,CAAC,EAClG,UAAU,IAAIA,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAoB;AACjD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,iBAAiB,QAAQ,MAAM,EAAE,QAAQ,GAAG;AAAA,QAC7D,oBAAoB,KAAK;AAAA,MAC3B,CAAC;AACD,YAAM,SAAS,MAAM,OAAO,KAAK,UAAU;AAC3C,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;ACtCA,SAAS,WAAAC,UAAS,UAAAC,eAAc;AAczB,SAAS,kBAA2B;AACzC,QAAM,MAAM,IAAIC,SAAQ,UAAU,EAC/B,YAAY,0CAA0C,EACtD,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,EACF,EACC,UAAU,IAAIC,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAwB;AACrD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,qBAAqB,QAAQ,KAAK,UAAU,MAAM,EAAE,QAAQ,CAAC;AAChF,YAAM,SAAS,MAAM,OAAO,SAAS,UAAU;AAC/C,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;ACvCA,SAAS,WAAAC,gBAAe;AAcjB,SAAS,0BAAmC;AACjD,QAAM,MAAM,IAAIC,SAAQ,mBAAmB,EACxC,YAAY,gDAAgD,EAC5D,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,IACA;AAAA,IACA,CAAC;AAAA,EACH;AAEF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAgC;AAE7D,UAAM,YAAY,KAAK,YAAY,CAAC;AACpC,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,6BAA6B,QAAQ,WAAW,MAAM,EAAE,QAAQ,CAAC;AACpF,YAAM,SAAS,MAAM,OAAO,iBAAiB,UAAU;AACvD,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;AAEA,SAAS,aAAa,OAAe,UAA8B;AACjE,SAAO,CAAC,GAAG,UAAU,KAAK;AAC5B;;;AC9CA,SAAS,WAAAC,gBAAe;;;ACmBxB,SAAS,YAAYC,WAAU;AAC/B,SAAS,WAAAC,gBAAe;AACxB,SAAS,WAAAC,UAAS,QAAAC,OAAM,eAAe;AACvC,SAAS,qBAAqB;AA0BvB,IAAM,UAAoC;AAAA;AAAA,EAE/C,EAAE,IAAI,eAAe,OAAO,eAAe,MAAM,OAAO,MAAM,QAAQ,MAAM,gCAAgC;AAAA;AAAA;AAAA,EAG5G,EAAE,IAAI,UAAU,OAAO,UAAU,MAAM,QAAQ,MAAM,WAAW,MAAM,mCAAmC;AAAA;AAAA,EAEzG,EAAE,IAAI,YAAY,OAAO,YAAY,MAAM,QAAQ,MAAM,WAAW,MAAM,oCAAoC;AAAA;AAAA,EAE9G,EAAE,IAAI,QAAQ,OAAO,QAAQ,MAAM,OAAO,MAAM,QAAQ,MAAM,gCAAgC;AAAA;AAAA,EAE9F,EAAE,IAAI,YAAY,OAAO,YAAY,MAAM,OAAO,MAAM,WAAW,MAAM,kCAAkC;AAAA;AAAA,EAE3G,EAAE,IAAI,UAAU,OAAO,cAAc,MAAM,QAAQ,MAAM,QAAQ,MAAM,8CAA8C;AAAA;AAAA,EAErH,EAAE,IAAI,WAAW,OAAO,kBAAkB,MAAM,QAAQ,MAAM,WAAW,MAAM,kCAAkC;AAAA;AAAA,EAEjH,EAAE,IAAI,WAAW,OAAO,WAAW,MAAM,QAAQ,MAAM,WAAW,MAAM,mCAAmC;AAAA;AAAA,EAE3G,EAAE,IAAI,WAAW,OAAO,WAAW,MAAM,QAAQ,MAAM,WAAW,MAAM,mCAAmC;AAC7G;AAEO,SAAS,WAAW,IAAuC;AAChE,SAAO,QAAQ,KAAK,CAAC,MAAM,EAAE,OAAO,EAAE;AACxC;AAMO,SAAS,kBAA0B;AACxC,QAAM,OAAOD,SAAQ,cAAc,YAAY,GAAG,CAAC;AAGnD,SAAO,QAAQ,MAAM,MAAM,WAAW,kBAAkB,UAAU,gBAAgB;AACpF;AAaO,SAAS,cAAc,QAAuB,OAAuB,CAAC,GAAkB;AAC7F,QAAM,OAAO,OAAO,SAAS,SAAU,KAAK,QAAQD,SAAQ,IAAM,KAAK,OAAO,QAAQ,IAAI;AAC1F,SAAO,EAAE,QAAQ,aAAaE,MAAK,MAAM,OAAO,IAAI,EAAE;AACxD;AAgBA,eAAsB,aACpB,QACA,OAAuB,CAAC,GACA;AACxB,QAAM,SAAS,KAAK,aAAa,gBAAgB;AACjD,QAAM,SAAS,MAAM,WAAW,OAAO,WAAW;AAElD,MAAI,UAAU,CAAC,KAAK,OAAO;AACzB,WAAO,EAAE,QAAQ,QAAQ,iBAAiB;AAAA,EAC5C;AAEA,MAAI,OAAO,OAAO,SAAS,OAAO;AAChC,QAAI,OAAQ,OAAMH,IAAG,GAAG,OAAO,aAAa,EAAE,WAAW,MAAM,OAAO,KAAK,CAAC;AAC5E,UAAM,cAAc,QAAQ,OAAO,WAAW;AAAA,EAChD,OAAO;AACL,UAAM,YAAY,MAAMA,IAAG,SAASG,MAAK,QAAQ,UAAU,GAAG,MAAM;AACpE,UAAMH,IAAG,MAAME,SAAQ,OAAO,WAAW,GAAG,EAAE,WAAW,KAAK,CAAC;AAC/D,UAAMF,IAAG,UAAU,OAAO,aAAa,SAAS;AAAA,EAClD;AAEA,SAAO,EAAE,QAAQ,QAAQ,SAAS,gBAAgB,YAAY;AAChE;AAEA,eAAe,WAAW,MAAgC;AACxD,MAAI;AACF,UAAMA,IAAG,OAAO,IAAI;AACpB,WAAO;AAAA,EACT,QAAQ;AACN,WAAO;AAAA,EACT;AACF;AAEA,eAAe,cAAc,QAAgB,aAAoC;AAE/E,QAAMA,IAAG,GAAG,QAAQ,aAAa,EAAE,WAAW,MAAM,OAAO,KAAK,CAAC;AACnE;;;AD3IO,SAAS,eAAwB;AACtC,QAAM,MAAM,IAAII,SAAQ,OAAO,EAAE,YAAY,gDAAgD;AAE7F,MACG,QAAQ,OAAO,EACf,YAAY,qEAAqE,EACjF;AAAA,IACC;AAAA,IACA,sCAAsC,QAAQ,IAAI,CAAC,MAAM,EAAE,EAAE,EAAE,KAAK,IAAI,CAAC;AAAA,IACzE;AAAA,IACA,CAAC;AAAA,EACH,EACC,OAAO,aAAa,qCAAqC,EACzD,OAAO,eAAe,gCAAgC,EACtD,OAAO,OAAO,SAA0B;AACvC,UAAM,WAAW,iBAAiB,IAAI;AACtC,QAAI,SAAS,WAAW,GAAG;AACzB,YAAM,IAAI;AAAA,QACR;AAAA,MACF;AAAA,IACF;AAEA,eAAW,UAAU,UAAU;AAC7B,YAAM,SAAS,cAAc,MAAM;AACnC,YAAM,SAAS,MAAM,aAAa,QAAQ,EAAE,OAAO,KAAK,MAAM,CAAC;AAC/D,cAAQ,OAAO,MAAM,GAAG,OAAO,KAAK,KAAK,OAAO,MAAM,WAAM,OAAO,WAAW;AAAA,CAAI;AAAA,IACpF;AAAA,EACF,CAAC;AAEH,MACG,QAAQ,QAAQ,EAAE,QAAQ,MAAM,CAAC,EACjC,YAAY,0CAA0C,EACtD,OAAO,MAAM;AACZ,eAAW,KAAK,SAAS;AACvB,YAAM,SAAS,cAAc,CAAC;AAC9B,cAAQ,OAAO,MAAM,GAAG,EAAE,GAAG,OAAO,EAAE,CAAC,IAAI,EAAE,MAAM,OAAO,EAAE,CAAC,IAAI,OAAO,WAAW;AAAA,CAAI;AAAA,IACzF;AAAA,EACF,CAAC;AAEH,SAAO;AACT;AAEA,SAAS,QAAQ,OAAe,UAA8B;AAC5D,SAAO,CAAC,GAAG,UAAU,KAAK;AAC5B;AAEA,SAAS,iBAAiB,MAAwC;AAChE,MAAI,KAAK,IAAK,QAAO,CAAC,GAAG,OAAO;AAChC,QAAM,MAAM,KAAK,UAAU,CAAC;AAC5B,SAAO,IAAI,IAAI,CAAC,OAAO;AACrB,UAAM,SAAS,WAAW,EAAE;AAC5B,QAAI,CAAC,OAAQ,OAAM,IAAI,MAAM,mBAAmB,EAAE,EAAE;AACpD,WAAO;AAAA,EACT,CAAC;AACH;;;AEjEA,SAAS,WAAAC,UAAS,UAAAC,eAAc;AAezB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAC3B,YAAY,kCAAkC,EAC9C,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC,IAAIC,QAAO,uBAAuB,sBAAsB,EAAE,QAAQ,CAAC,SAAS,QAAQ,KAAK,CAAC;AAAA,EAC5F,EACC,OAAO,kBAAkB,4CAA4C;AAExE,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAoB;AACjD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,iBAAiB,QAAQ,MAAM,EAAE,QAAQ,GAAG;AAAA,QAC7D,YAAY,KAAK;AAAA,QACjB,aAAa,KAAK;AAAA,MACpB,CAAC;AACD,YAAM,SAAS,MAAM,OAAO,KAAK,UAAU;AAC3C,YAAM,KAAK,QAAQ,IAAI;AAAA,IACzB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AClCA;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,OACK;AAEA,IAAM,aAAa;AAAA,EACxB,IAAI;AAAA,EACJ,SAAS;AAAA,EACT,OAAO;AAAA,EACP,YAAY;AAAA,EACZ,MAAM;AAAA,EACN,iBAAiB;AAAA,EACjB,WAAW;AAAA,EACX,aAAa;AAAA,EACb,SAAS;AAAA,EACT,YAAY;AACd;AAEO,SAAS,YAAY,KAAsB;AAChD,MAAI,eAAe,gBAAiB,QAAO,WAAW;AACtD,MAAI,eAAe,oBAAqB,QAAO,WAAW;AAC1D,MAAI,eAAe,qBAAsB,QAAO,WAAW;AAC3D,MAAI,eAAe,eAAgB,QAAO,WAAW;AACrD,MAAI,eAAe,YAAa,QAAO,WAAW;AAClD,MAAI,eAAe,oBAAqB,QAAO,WAAW;AAC1D,MAAI,eAAe,gBAAiB,QAAO,WAAW;AACtD,MAAI,eAAe,mBAAoB,QAAO,WAAW;AACzD,MAAI,eAAe,SAAU,QAAO,WAAW;AAC/C,MAAI,eAAe,mBAAoB,QAAO,WAAW;AACzD,SAAO,WAAW;AACpB;AAEO,SAAS,YAAY,KAAsB;AAChD,MAAI,eAAe,UAAU;AAC3B,UAAM,QAAQ,CAAC,IAAI,OAAO;AAC1B,QAAI,IAAI,OAAQ,OAAM,KAAK,SAAS,IAAI,MAAM,GAAG;AACjD,WAAO,MAAM,KAAK,GAAG;AAAA,EACvB;AACA,MAAI,eAAe,MAAO,QAAO,IAAI;AACrC,SAAO,OAAO,GAAG;AACnB;;;AjBhCA,SAAS,iBAAyB;AAGhC,QAAM,OAAOC,SAAQC,eAAc,YAAY,GAAG,CAAC;AACnD,QAAM,UAAUC,SAAQ,MAAM,MAAM,cAAc;AAClD,QAAM,MAAM,KAAK,MAAM,aAAa,SAAS,MAAM,CAAC;AACpD,SAAO,IAAI;AACb;AAEO,SAAS,eAAwB;AACtC,QAAM,UAAU,IAAIC,UAAQ;AAC5B,UACG,KAAK,gBAAgB,EACrB;AAAA,IACC;AAAA,EAEF,EACC,QAAQ,eAAe,GAAG,iBAAiB,2BAA2B,EACtE,mBAAmB;AAEtB,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,gBAAgB,CAAC;AACpC,UAAQ,WAAW,wBAAwB,CAAC;AAC5C,UAAQ,WAAW,WAAW,CAAC;AAC/B,UAAQ,WAAW,eAAe,CAAC;AACnC,UAAQ,WAAW,eAAe,CAAC;AACnC,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,aAAa,CAAC;AAEjC,SAAO;AACT;AAEA,eAAe,KAAK,MAA+B;AACjD,QAAM,UAAU,aAAa;AAC7B,MAAI;AACF,UAAM,QAAQ,WAAW,IAAI;AAAA,EAC/B,SAAS,KAAK;AACZ,YAAQ,OAAO,MAAM,UAAU,YAAY,GAAG,CAAC;AAAA,CAAI;AACnD,YAAQ,KAAK,YAAY,GAAG,CAAC;AAAA,EAC/B;AACF;AAIA,IAAM,YAAY,QAAQ,KAAK,CAAC,IAAIF,eAAc,YAAY,GAAG,IAAI;AACrE,IAAI,aAAa,QAAQ,KAAK,CAAC,MAAM,WAAW;AAC9C,OAAK,QAAQ,IAAI,EAAE,MAAM,CAAC,QAAiB;AACzC,YAAQ,OAAO,MAAM,UAAU,YAAY,GAAG,CAAC;AAAA,CAAI;AACnD,YAAQ,KAAK,YAAY,GAAG,CAAC;AAAA,EAC/B,CAAC;AACH;","names":["Command","dirname","resolve","fileURLToPath","fs","Command","Option","fs","Command","Option","Command","Command","Command","Command","Command","Option","Command","Option","Command","Option","Command","Option","Command","Command","Command","fs","homedir","dirname","join","Command","Command","Option","Command","Option","dirname","fileURLToPath","resolve","Command"]}
|
|
1
|
+
{"version":3,"sources":["../src/cli.ts","../src/commands/account.ts","../src/lib/config.ts","../src/lib/output.ts","../src/lib/sdk.ts","../src/lib/redact.ts","../src/commands/ask.ts","../src/lib/buildRequest.ts","../src/lib/options.ts","../src/lib/stdin.ts","../src/commands/auth.ts","../src/commands/extract.ts","../src/commands/html.ts","../src/commands/selected.ts","../src/commands/selectedMultiple.ts","../src/commands/serp.ts","../src/lib/errors.ts","../src/commands/setupSkill.ts","../src/lib/skill.ts","../src/commands/text.ts"],"sourcesContent":["/**\n * `webscraping-ai` / `wsai` CLI entry point.\n *\n * Wires every subcommand into a single commander root. Errors thrown from\n * subcommand actions land in the top-level `.catch`, which maps SDK\n * exception classes to stable exit codes (see `lib/errors.ts`).\n */\n\nimport { Command } from 'commander';\nimport { readFileSync } from 'node:fs';\nimport { dirname, resolve } from 'node:path';\nimport { fileURLToPath } from 'node:url';\n\nimport { accountCommand } from './commands/account.js';\nimport { askCommand } from './commands/ask.js';\nimport { authCommand } from './commands/auth.js';\nimport { extractCommand } from './commands/extract.js';\nimport { htmlCommand } from './commands/html.js';\nimport { selectedCommand } from './commands/selected.js';\nimport { selectedMultipleCommand } from './commands/selectedMultiple.js';\nimport { serpCommand } from './commands/serp.js';\nimport { setupCommand } from './commands/setupSkill.js';\nimport { textCommand } from './commands/text.js';\nimport { exitCodeFor, formatError } from './lib/errors.js';\n\nfunction packageVersion(): string {\n // Read package.json at runtime so we don't need a code-gen step. The dist\n // bundle sits at `dist/cli.js`, so package.json is one directory up.\n const here = dirname(fileURLToPath(import.meta.url));\n const pkgPath = resolve(here, '..', 'package.json');\n const raw = JSON.parse(readFileSync(pkgPath, 'utf8')) as { version: string };\n return raw.version;\n}\n\nexport function buildProgram(): Command {\n const program = new Command();\n program\n .name('webscraping-ai')\n .description(\n 'Official CLI for the WebScraping.AI API. Fetch pages, search Google, ask AI, extract structured data.\\n' +\n 'Set $WEBSCRAPING_AI_API_KEY or run `webscraping-ai auth set <key>` first.',\n )\n .version(packageVersion(), '-v, --version', 'output the version number')\n .showHelpAfterError();\n\n program.addCommand(htmlCommand());\n program.addCommand(textCommand());\n program.addCommand(selectedCommand());\n program.addCommand(selectedMultipleCommand());\n program.addCommand(askCommand());\n program.addCommand(extractCommand());\n program.addCommand(serpCommand());\n program.addCommand(accountCommand());\n program.addCommand(authCommand());\n program.addCommand(setupCommand());\n\n return program;\n}\n\nasync function main(argv: string[]): Promise<void> {\n const program = buildProgram();\n try {\n await program.parseAsync(argv);\n } catch (err) {\n process.stderr.write(`Error: ${formatError(err)}\\n`);\n process.exit(exitCodeFor(err));\n }\n}\n\n// Only execute when run as a script. When imported (e.g. from tests),\n// `buildProgram` is the public surface and `main` stays dormant.\nconst entryPath = process.argv[1] ? fileURLToPath(import.meta.url) : '';\nif (entryPath && process.argv[1] === entryPath) {\n main(process.argv).catch((err: unknown) => {\n process.stderr.write(`Error: ${formatError(err)}\\n`);\n process.exit(exitCodeFor(err));\n });\n}\n","import { Command } from 'commander';\n\nimport { resolveApiKey } from '../lib/config.js';\nimport { emit } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\n\ninterface AccountFlags {\n apiKey?: string;\n output?: string;\n pretty?: boolean;\n}\n\nexport function accountCommand(): Command {\n return new Command('account')\n .description('Show account quota and usage information')\n .option('-k, --api-key <key>', 'API key (overrides $WEBSCRAPING_AI_API_KEY and config file)')\n .option('-o, --output <file>', 'write result to FILE instead of stdout')\n .option('--pretty', 'pretty-print JSON output (default on TTY)')\n .option('--no-pretty', 'force single-line JSON output')\n .action(async (opts: AccountFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const client = createClient({ apiKey });\n const result = await client.account();\n await emit(result, opts);\n });\n}\n","/**\n * API key resolution and persisted config.\n *\n * Resolution order (first non-empty wins):\n * 1. `--api-key` flag (handled by the caller, passed in)\n * 2. `WEBSCRAPING_AI_API_KEY` env var\n * 3. `~/.config/webscraping-ai/config.json` (or `$XDG_CONFIG_HOME/...`)\n *\n * The config file is plain JSON with `{ \"apiKey\": \"...\" }`. We chmod it to\n * 0600 so it's not world-readable.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { homedir } from 'node:os';\nimport { dirname, join } from 'node:path';\n\nconst CONFIG_FILENAME = 'config.json';\n\nexport function configDir(): string {\n const xdg = process.env.XDG_CONFIG_HOME;\n if (xdg && xdg.trim() !== '') return join(xdg, 'webscraping-ai');\n return join(homedir(), '.config', 'webscraping-ai');\n}\n\nexport function configPath(): string {\n return join(configDir(), CONFIG_FILENAME);\n}\n\nexport interface PersistedConfig {\n apiKey?: string;\n}\n\nexport async function readPersistedConfig(): Promise<PersistedConfig> {\n try {\n const text = await fs.readFile(configPath(), 'utf8');\n const parsed = JSON.parse(text) as unknown;\n if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {\n return parsed as PersistedConfig;\n }\n return {};\n } catch (err) {\n if ((err as NodeJS.ErrnoException).code === 'ENOENT') return {};\n throw err;\n }\n}\n\nexport async function writePersistedConfig(config: PersistedConfig): Promise<void> {\n const path = configPath();\n await fs.mkdir(dirname(path), { recursive: true });\n await fs.writeFile(path, JSON.stringify(config, null, 2) + '\\n', { mode: 0o600 });\n // `mode` above only applies when the file is freshly created; an existing\n // file keeps its old (possibly world-readable) perms. Enforce 0600\n // unconditionally — chmod on a just-written file is cheap and idempotent.\n // No-op on Windows, but harmless.\n await fs.chmod(path, 0o600);\n}\n\nexport async function clearPersistedConfig(): Promise<boolean> {\n try {\n await fs.unlink(configPath());\n return true;\n } catch (err) {\n if ((err as NodeJS.ErrnoException).code === 'ENOENT') return false;\n throw err;\n }\n}\n\nexport interface ResolveApiKeyOptions {\n /** `--api-key` flag value, if the user passed one. */\n flag?: string;\n}\n\nexport async function resolveApiKey(opts: ResolveApiKeyOptions = {}): Promise<string> {\n if (opts.flag && opts.flag.trim() !== '') return opts.flag.trim();\n\n const env = process.env.WEBSCRAPING_AI_API_KEY;\n if (env && env.trim() !== '') return env.trim();\n\n const persisted = await readPersistedConfig();\n if (persisted.apiKey && persisted.apiKey.trim() !== '') return persisted.apiKey.trim();\n\n throw new Error(\n 'No API key found. Set $WEBSCRAPING_AI_API_KEY, pass --api-key, or run `webscraping-ai auth set <key>`.',\n );\n}\n\nexport function maskApiKey(key: string): string {\n if (key.length <= 8) return '*'.repeat(key.length);\n return `${key.slice(0, 4)}${'*'.repeat(key.length - 8)}${key.slice(-4)}`;\n}\n","/**\n * Output handling for command results.\n *\n * All command handlers funnel through `emit()`. The result type drives format:\n * - strings → written verbatim\n * - objects/arrays → JSON.stringified (pretty by default; `--pretty` is the\n * default for terminals, single-line for pipes/redirects)\n *\n * `--output FILE` writes to disk instead of stdout. In stdin batch mode use\n * `createEmitter()`: it truncates the file on the first result and appends\n * every later one, so each item survives instead of overwriting the last.\n */\n\nimport { promises as fs } from 'node:fs';\n\nexport interface EmitOptions {\n output?: string;\n pretty?: boolean;\n}\n\nexport interface EmitWriteOptions {\n /** Append to `--output` instead of truncating it (batch items after the first). */\n append?: boolean;\n}\n\nexport async function emit(\n value: unknown,\n opts: EmitOptions = {},\n write: EmitWriteOptions = {},\n): Promise<void> {\n const text = stringify(value, opts);\n\n if (opts.output && opts.output.trim() !== '') {\n const data = text + (text.endsWith('\\n') ? '' : '\\n');\n if (write.append) await fs.appendFile(opts.output, data);\n else await fs.writeFile(opts.output, data);\n return;\n }\n\n process.stdout.write(text);\n if (!text.endsWith('\\n')) process.stdout.write('\\n');\n}\n\n/**\n * Returns an `emit` bound to `opts` for a (possibly batched) loop: the first\n * call truncates `--output`, later calls append.\n */\nexport function createEmitter(opts: EmitOptions = {}): (value: unknown) => Promise<void> {\n let first = true;\n return async (value: unknown) => {\n const append = !first;\n first = false;\n await emit(value, opts, { append });\n };\n}\n\nexport function stringify(value: unknown, opts: EmitOptions = {}): string {\n if (typeof value === 'string') return value;\n\n const pretty = opts.pretty ?? process.stdout.isTTY ?? false;\n return JSON.stringify(value, null, pretty ? 2 : 0);\n}\n","/**\n * SDK adapter.\n *\n * Wraps the published `webscraping-ai` Node SDK. The CLI sends a\n * `from_cli=true` analytics flag on the page/AI endpoints (mirrors `from_n8n`\n * and `from_mcp_server` used by the n8n node and MCP server). It is NOT sent\n * on `serp` or `account`: the SDK builds those query strings from a fixed set\n * of keys and drops anything else. The SDK's typed\n * options don't accept arbitrary keys, so we go through a single cast point\n * here rather than scattering casts across each command.\n */\n\nimport { WebScrapingAI } from 'webscraping-ai';\n\nimport { registerSecret } from './redact.js';\nimport type {\n FieldsOptions,\n HtmlOptions,\n QuestionOptions,\n SelectedMultipleOptions,\n SelectedOptions,\n SerpOptions,\n TextOptions,\n} from 'webscraping-ai';\n\nexport interface CreateClientOptions {\n apiKey: string;\n /** Per-request timeout in ms. Maps to the SDK's `timeoutMs`. */\n requestTimeoutMs?: number;\n}\n\n/**\n * Returns a `WebScrapingAI` instance with the CLI's analytics tag baked into\n * every endpoint method. The methods preserve the SDK's public signatures.\n */\nexport function createClient(options: CreateClientOptions): WebScrapingAI {\n registerSecret(options.apiKey);\n const inner = new WebScrapingAI({\n apiKey: options.apiKey,\n timeoutMs: options.requestTimeoutMs,\n });\n\n // Wrap each endpoint method so it injects the CLI analytics tag. We splice\n // the tag in at this seam rather than at every callsite. The cast is local;\n // the SDK passes unknown query params through to the wire.\n const tag = { from_cli: true } as unknown as Record<string, never>;\n\n return new Proxy(inner, {\n get(target, prop, receiver) {\n const value = Reflect.get(target, prop, receiver);\n if (typeof value !== 'function') return value;\n switch (prop) {\n case 'html':\n return (o: HtmlOptions) => target.html({ ...o, ...tag });\n case 'text':\n return (o: TextOptions) => target.text({ ...o, ...tag });\n case 'selected':\n return (o: SelectedOptions) => target.selected({ ...o, ...tag });\n case 'selectedMultiple':\n return (o: SelectedMultipleOptions) => target.selectedMultiple({ ...o, ...tag });\n case 'question':\n return (o: QuestionOptions) => target.question({ ...o, ...tag });\n case 'fields':\n return (o: FieldsOptions) => target.fields({ ...o, ...tag });\n case 'serp':\n // Passed for parity only: SDK 4.1.0's `serp()` forwards just\n // q/engine/gl/hl/page and drops every other key, so `from_cli` is\n // not sent on /serp. Accepted limitation, like `account`.\n return (o: SerpOptions) => target.serp({ ...o, ...tag });\n case 'account':\n // `account` takes no options; the SDK builds the request itself, so\n // there's no seam here for the analytics tag — accepted limitation.\n return value.bind(target);\n default:\n return typeof value === 'function' ? value.bind(target) : value;\n }\n },\n });\n}\n","/**\n * Secret redaction for anything the CLI prints to stderr.\n *\n * API errors can echo the request URL (e.g. an HTML error page that includes\n * `?api_key=...`), so every error message goes through `redactSecrets()`\n * before it is printed. The key in use is registered by `createClient()`.\n */\n\nconst secrets = new Set<string>();\n\n/** Remember a secret so later error output never prints it. */\nexport function registerSecret(value: string | undefined): void {\n if (value && value.trim() !== '') secrets.add(value);\n}\n\n/** Forget registered secrets (tests only). */\nexport function clearSecrets(): void {\n secrets.clear();\n}\n\nconst API_KEY_PARAM = /(api_key=)[^&\\s\"'<>]*/gi;\n\nexport function redactSecrets(text: string): string {\n let out = text.replace(API_KEY_PARAM, '$1[REDACTED]');\n const candidates = new Set(secrets);\n const env = process.env.WEBSCRAPING_AI_API_KEY?.trim();\n if (env) candidates.add(env);\n for (const secret of candidates) {\n for (const form of new Set([secret, encodeURIComponent(secret)])) {\n out = out.split(form).join('[REDACTED]');\n }\n }\n return out;\n}\n","import { Command, Option } from 'commander';\n\nimport { buildQuestionOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface AskFlags extends CommonRawFlags {\n question?: string;\n}\n\nexport function askCommand(): Command {\n const cmd = new Command('ask')\n .description('Ask an AI question about a page (wraps /ai/question)')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .requiredOption('-q, --question <text>', 'the question / instructions for the model')\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: AskFlags) => {\n if (!opts.question) throw new Error('--question is required');\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildQuestionOptions(target, opts.question, opts, { headers });\n const result = await client.question(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n","/**\n * Pure builder that maps `--flag` values into SDK option objects.\n *\n * Extracted as a separate function so unit tests can pin the mapping without\n * spinning up commander or making HTTP calls. Each command handler reads its\n * own commander options, awaits `parseHeaders` / `parseFields`, then calls\n * the matching `build*` function here.\n */\n\nimport type {\n CommonRequestOptions,\n FieldsOptions,\n HtmlOptions,\n QuestionOptions,\n SelectedMultipleOptions,\n SelectedOptions,\n SerpOptions,\n TextOptions,\n} from 'webscraping-ai';\n\nimport type { CommonRawFlags } from './options.js';\n\nexport interface ResolvedCommonOptions {\n headers?: Record<string, string>;\n}\n\nexport function buildCommonOptions(\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions = {},\n): CommonRequestOptions {\n const out: CommonRequestOptions = {};\n // `flags.js` is `false` when `--no-js`, `true` when `--js`, `undefined`\n // when neither passed. Pass through only when explicit, so the API default\n // applies otherwise.\n if (flags.js !== undefined) out.js = flags.js;\n if (flags.proxy !== undefined) out.proxy = flags.proxy;\n if (flags.country !== undefined) out.country = flags.country;\n if (flags.device !== undefined) out.device = flags.device;\n if (flags.timeout !== undefined) out.timeout = flags.timeout;\n if (flags.jsTimeout !== undefined) out.js_timeout = flags.jsTimeout;\n if (flags.waitFor !== undefined) out.wait_for = flags.waitFor;\n if (flags.jsScript !== undefined) out.js_script = flags.jsScript;\n if (flags.customProxy !== undefined) out.custom_proxy = flags.customProxy;\n if (flags.errorOn404) out.error_on_404 = true;\n if (flags.errorOnRedirect) out.error_on_redirect = true;\n if (resolved.headers) out.headers = resolved.headers;\n return out;\n}\n\nexport interface HtmlExtras {\n returnScriptResult?: boolean;\n}\n\nexport function buildHtmlOptions(\n url: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n extras: HtmlExtras = {},\n): HtmlOptions {\n const out: HtmlOptions = { url, ...buildCommonOptions(flags, resolved) };\n if (flags.format) out.format = flags.format;\n if (extras.returnScriptResult) out.return_script_result = true;\n return out;\n}\n\nexport interface TextExtras {\n textFormat?: 'plain' | 'json' | 'xml';\n returnLinks?: boolean;\n}\n\nexport function buildTextOptions(\n url: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n extras: TextExtras = {},\n): TextOptions {\n const out: TextOptions = { url, ...buildCommonOptions(flags, resolved) };\n if (extras.textFormat) out.text_format = extras.textFormat;\n if (extras.returnLinks) out.return_links = true;\n return out;\n}\n\nexport function buildSelectedOptions(\n url: string,\n selector: string | undefined,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): SelectedOptions {\n // `selector` is optional per the OpenAPI spec: omitting it returns the\n // whole-page HTML. Only attach it when provided. (The installed SDK types\n // still mark it required, so build the object as a partial and cast.)\n const out: Partial<SelectedOptions> = { url, ...buildCommonOptions(flags, resolved) };\n if (selector !== undefined) out.selector = selector;\n if (flags.format) out.format = flags.format;\n return out as SelectedOptions;\n}\n\nexport function buildSelectedMultipleOptions(\n url: string,\n selectors: readonly string[],\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): SelectedMultipleOptions {\n // `selectors` is optional per the OpenAPI spec: omitting it returns the\n // whole-page HTML. Only attach it when non-empty. (The installed SDK types\n // still mark it required, so build the object as a partial and cast.)\n const out: Partial<SelectedMultipleOptions> = { url, ...buildCommonOptions(flags, resolved) };\n if (selectors.length > 0) out.selectors = selectors;\n return out as SelectedMultipleOptions;\n}\n\nexport function buildQuestionOptions(\n url: string,\n question: string,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): QuestionOptions {\n const out: QuestionOptions = { url, question, ...buildCommonOptions(flags, resolved) };\n if (flags.format) out.format = flags.format;\n return out;\n}\n\nexport function buildFieldsOptions(\n url: string,\n fields: Record<string, string>,\n flags: CommonRawFlags,\n resolved: ResolvedCommonOptions,\n): FieldsOptions {\n return { url, fields, ...buildCommonOptions(flags, resolved) };\n}\n\nexport interface SerpRawFlags {\n engine?: 'google';\n gl?: string;\n hl?: string;\n page?: number;\n}\n\n/**\n * `/serp` is query-shaped: it takes none of the common scrape options, so\n * this deliberately does not go through `buildCommonOptions`. `page` must be\n * an integer >= 1 (the server also rejects anything else with a 400, not\n * billed; checking client-side saves the round trip). Pages are 1-100: the\n * server rejects a page above 100 with a 400.\n */\nexport function buildSerpOptions(query: string, flags: SerpRawFlags = {}): SerpOptions {\n // Validate on the trimmed value but send the query as the user typed it.\n if (typeof query !== 'string' || query.trim() === '') {\n throw new Error('serp requires a non-empty query');\n }\n const out: SerpOptions = { q: query };\n if (flags.engine !== undefined) out.engine = flags.engine;\n if (flags.gl) out.gl = flags.gl;\n if (flags.hl) out.hl = flags.hl;\n if (flags.page !== undefined) {\n if (!Number.isSafeInteger(flags.page) || flags.page < 1) {\n throw new Error(`--page must be an integer >= 1, got: ${flags.page}`);\n }\n out.page = flags.page;\n }\n return out;\n}\n","/**\n * Shared commander options and the parsers/expanders behind them.\n *\n * All scrape endpoints share the same nine \"common\" flags. We define them in\n * one place and apply via `addCommonScrapeOptions(cmd)` so the surface stays\n * consistent and any change lands everywhere at once.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { Option } from 'commander';\nimport type { Command } from 'commander';\n\nexport interface CommonRawFlags {\n apiKey?: string;\n output?: string;\n format?: 'text' | 'json';\n pretty?: boolean;\n js?: boolean;\n proxy?: 'datacenter' | 'residential' | 'stealth';\n country?: string;\n device?: 'desktop' | 'mobile' | 'tablet';\n timeout?: number;\n jsTimeout?: number;\n waitFor?: string;\n headers?: string;\n jsScript?: string;\n customProxy?: string;\n errorOn404?: boolean;\n errorOnRedirect?: boolean;\n}\n\n/**\n * Add the flags every scrape-endpoint command needs. Adding here keeps the\n * surface aligned across `html`, `text`, `selected`, etc.\n */\nexport function addCommonScrapeOptions(cmd: Command): Command {\n return cmd\n .option('-k, --api-key <key>', 'API key (overrides $WEBSCRAPING_AI_API_KEY and config file)')\n .option('-o, --output <file>', 'write result to FILE instead of stdout')\n .option('--pretty', 'pretty-print JSON output (default on TTY)')\n .option('--no-pretty', 'force single-line JSON output')\n .option('--js', 'execute JS via a headless browser (default: enabled)')\n .option('--no-js', 'disable JS execution (faster, cheaper)')\n .addOption(\n new Option('--proxy <type>', 'proxy pool').choices(['datacenter', 'residential', 'stealth']),\n )\n .option('--country <code>', 'two-letter proxy country code, e.g. us, gb, de')\n .addOption(\n new Option('--device <type>', 'device emulation').choices(['desktop', 'mobile', 'tablet']),\n )\n .option('--timeout <ms>', 'page-retrieval timeout in ms (max 30000)', integerParser)\n .option('--js-timeout <ms>', 'JS rendering time in ms after load (max 20000)', integerParser)\n .option('--wait-for <selector>', 'CSS selector to wait for before returning content')\n .option(\n '--headers <json>',\n 'HTTP headers to forward to the target page; JSON object or @file.json',\n )\n .option('--js-script <code>', 'custom JS to evaluate on the target page')\n .option('--custom-proxy <url>', 'custom proxy URL: http://user:pass@host:port')\n .option('--error-on-404', 'return error on 404')\n .option('--error-on-redirect', 'return error on redirect');\n}\n\n/**\n * Strict positive-integer check for flags like `serp --page`: digits only,\n * no leading zero, so `1.5`, `2abc`, `0` and `-1` are all rejected (unlike\n * `parseInt`, which would read `2abc` as 2). Values beyond\n * `Number.MAX_SAFE_INTEGER` are rejected too.\n */\nexport function isPositiveIntegerString(value: string): boolean {\n // isSafeInteger: a huge digit string becomes e.g. 1e+21 on the wire, which\n // the server rejects with a 400 (not billed); checking here saves the round trip.\n return /^[1-9]\\d*$/.test(value) && Number.isSafeInteger(Number(value));\n}\n\nexport function integerParser(value: string): number {\n const n = Number.parseInt(value, 10);\n if (!Number.isFinite(n)) throw new Error(`expected integer, got: ${value}`);\n return n;\n}\n\n/**\n * Parse a `--headers` argument. Accepts a JSON object literal or `@path` to\n * load JSON from a file. Returns `undefined` if the input is empty.\n */\nexport async function parseHeaders(\n input: string | undefined,\n): Promise<Record<string, string> | undefined> {\n if (!input || input.trim() === '') return undefined;\n const text = input.startsWith('@') ? await fs.readFile(input.slice(1), 'utf8') : input;\n return parseJsonObject(text, 'headers');\n}\n\n/**\n * Parse a `--fields` argument for `extract`. Accepts JSON object literal or\n * `@path` to a file.\n */\nexport async function parseFields(input: string): Promise<Record<string, string>> {\n const text = input.startsWith('@') ? await fs.readFile(input.slice(1), 'utf8') : input;\n const parsed = parseJsonObject(text, 'fields');\n if (!parsed || Object.keys(parsed).length === 0) {\n throw new Error('fields must be a non-empty JSON object of { fieldName: description }');\n }\n return parsed;\n}\n\nfunction parseJsonObject(text: string, label: string): Record<string, string> {\n let parsed: unknown;\n try {\n parsed = JSON.parse(text);\n } catch (err) {\n throw new Error(`${label}: invalid JSON — ${(err as Error).message}`);\n }\n if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {\n throw new Error(`${label} must be a JSON object`);\n }\n const out: Record<string, string> = {};\n for (const [k, v] of Object.entries(parsed as Record<string, unknown>)) {\n if (typeof v !== 'string') throw new Error(`${label}.${k}: value must be a string`);\n out[k] = v;\n }\n return out;\n}\n","/**\n * Stdin batching helpers.\n *\n * When a URL argument is `-`, the command reads one URL per line from stdin\n * and runs the operation for each. Empty lines and lines starting with `#`\n * are skipped so users can comment URL lists.\n *\n * `serp` reads queries instead (`queryIterator`): only blank lines are\n * skipped there, because `#hashtag` is a legitimate search query.\n */\n\nimport readline from 'node:readline';\n\nexport function urlIsStdin(url: string): boolean {\n return url === '-';\n}\n\nexport async function* readUrlsFromStdin(): AsyncIterable<string> {\n const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });\n for await (const raw of rl) {\n const line = raw.trim();\n if (line === '' || line.startsWith('#')) continue;\n yield line;\n }\n}\n\n/**\n * Read one search query per line. Blank / whitespace-only lines are skipped;\n * every other line is yielded verbatim (no `#` comment handling, no trim).\n */\nexport async function* readQueriesFromStdin(\n input: NodeJS.ReadableStream = process.stdin,\n): AsyncIterable<string> {\n const rl = readline.createInterface({ input, crlfDelay: Infinity });\n for await (const line of rl) {\n if (line.trim() === '') continue;\n yield line;\n }\n}\n\n/**\n * Iterate URLs from either a single CLI arg or from stdin (if the arg is `-`).\n */\nexport async function* urlIterator(arg: string): AsyncIterable<string> {\n if (urlIsStdin(arg)) {\n yield* readUrlsFromStdin();\n return;\n }\n yield arg;\n}\n\n/**\n * Iterate `serp` queries from a single CLI arg or from stdin (if it is `-`).\n */\nexport async function* queryIterator(arg: string): AsyncIterable<string> {\n if (urlIsStdin(arg)) {\n yield* readQueriesFromStdin();\n return;\n }\n yield arg;\n}\n","import { Command } from 'commander';\n\nimport {\n clearPersistedConfig,\n configPath,\n maskApiKey,\n readPersistedConfig,\n writePersistedConfig,\n} from '../lib/config.js';\n\nexport function authCommand(): Command {\n const cmd = new Command('auth').description('Manage your stored API key');\n\n cmd\n .command('set <key>')\n .description(`Save an API key to ${configPath()}`)\n .action(async (key: string) => {\n if (!key || key.trim() === '') throw new Error('API key is empty');\n await writePersistedConfig({ apiKey: key.trim() });\n process.stdout.write(`API key saved to ${configPath()}\\n`);\n });\n\n cmd\n .command('show')\n .description('Display the currently stored API key (masked)')\n .action(async () => {\n const persisted = await readPersistedConfig();\n const env = process.env.WEBSCRAPING_AI_API_KEY;\n if (env && env.trim() !== '') {\n process.stdout.write(\n `Active key (env WEBSCRAPING_AI_API_KEY): ${maskApiKey(env.trim())}\\n`,\n );\n }\n if (persisted.apiKey && persisted.apiKey.trim() !== '') {\n process.stdout.write(\n `Stored key (${configPath()}): ${maskApiKey(persisted.apiKey.trim())}\\n`,\n );\n }\n if (!env && !persisted.apiKey) {\n process.stdout.write('No API key configured. Run `webscraping-ai auth set <key>`.\\n');\n }\n });\n\n cmd\n .command('clear')\n .description('Remove the stored API key file')\n .action(async () => {\n const removed = await clearPersistedConfig();\n process.stdout.write(\n removed ? `Removed ${configPath()}\\n` : 'No stored API key to remove.\\n',\n );\n });\n\n return cmd;\n}\n","import { Command } from 'commander';\n\nimport { buildFieldsOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseFields, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface ExtractFlags extends CommonRawFlags {\n fields?: string;\n}\n\nexport function extractCommand(): Command {\n const cmd = new Command('extract')\n .description('Extract structured fields from a page using AI (wraps /ai/fields)')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .requiredOption(\n '-f, --fields <json>',\n 'fields to extract — JSON object {name: description} or @file.json',\n );\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: ExtractFlags) => {\n if (!opts.fields) throw new Error('--fields is required');\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const fields = await parseFields(opts.fields);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildFieldsOptions(target, fields, opts, { headers });\n const result = await client.fields(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n","import { Command, Option } from 'commander';\n\nimport { buildHtmlOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface HtmlFlags extends CommonRawFlags {\n returnScriptResult?: boolean;\n}\n\nexport function htmlCommand(): Command {\n const cmd = new Command('html')\n .description('Fetch the full HTML of a page')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .addOption(\n new Option('--return-script-result', 'return the result of --js-script instead of HTML'),\n )\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: HtmlFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildHtmlOptions(\n target,\n opts,\n { headers },\n {\n returnScriptResult: opts.returnScriptResult,\n },\n );\n const result = await client.html(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n","import { Command, Option } from 'commander';\n\nimport { buildSelectedOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface SelectedFlags extends CommonRawFlags {\n selector?: string;\n}\n\nexport function selectedCommand(): Command {\n const cmd = new Command('selected')\n .description('Fetch HTML of one CSS-selected page area')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .option(\n '-s, --selector <css>',\n 'CSS selector for the target element(s); omit for whole-page HTML',\n )\n .addOption(new Option('--format <fmt>', 'response wrapping').choices(['text', 'json']));\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: SelectedFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildSelectedOptions(target, opts.selector, opts, { headers });\n const result = await client.selected(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n","import { Command } from 'commander';\n\nimport { buildSelectedMultipleOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface SelectedMultipleFlags extends CommonRawFlags {\n selector?: string[];\n}\n\nexport function selectedMultipleCommand(): Command {\n const cmd = new Command('selected-multiple')\n .description('Fetch HTML of multiple CSS-selected page areas')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .option(\n '-s, --selector <css>',\n 'CSS selector — pass multiple times for multiple selectors',\n collectMulti,\n [] as string[],\n );\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: SelectedMultipleFlags) => {\n // `--selector` is optional: omitting it returns whole-page HTML.\n const selectors = opts.selector ?? [];\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildSelectedMultipleOptions(target, selectors, opts, { headers });\n const result = await client.selectedMultiple(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n\nfunction collectMulti(value: string, previous: string[]): string[] {\n return [...previous, value];\n}\n","import { Command, Option } from 'commander';\n\nimport { buildSerpOptions } from '../lib/buildRequest.js';\nimport type { SerpRawFlags } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { EXIT_CODES } from '../lib/errors.js';\nimport { isPositiveIntegerString } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { queryIterator, urlIsStdin } from '../lib/stdin.js';\n\ninterface SerpFlags extends Omit<SerpRawFlags, 'page'> {\n /** Raw `--page` string; validated strictly in the action. */\n page?: string;\n apiKey?: string;\n output?: string;\n pretty?: boolean;\n}\n\n/**\n * `serp <query...>` — parsed search engine results (wraps /serp).\n *\n * Query-shaped, not URL-shaped, so none of the common scrape flags\n * (`--js`, `--proxy`, `--country`, …) apply and they're deliberately not\n * registered here. Multiple words are joined with spaces, so quoting the\n * query is optional. A lone `-` reads one query per line from stdin; only\n * blank lines are skipped (a leading `#` is part of the query, not a comment).\n *\n * Usage errors (bad `--page`, empty query) exit with code 2 before stdin is\n * read or the API key is resolved.\n */\nexport function serpCommand(): Command {\n const cmd = new Command('serp')\n .alias('search')\n .description('Search Google and print parsed results as JSON (wraps /serp, 15 credits/search)')\n .argument('<query...>', 'search query (or `-` to read queries from stdin, one per line)')\n .addOption(new Option('--engine <name>', 'search engine (default: google)').choices(['google']))\n .option('--gl <code>', 'two-letter country code for search geolocation (default: us)')\n .option('--hl <code>', 'two-letter language code for the results (default: en)')\n .option(\n '--page <n>',\n 'results page number, integer >= 1, 10 results per page (default: 1; server rejects > 100 with a 400)',\n )\n .option('-k, --api-key <key>', 'API key (overrides $WEBSCRAPING_AI_API_KEY and config file)')\n .option('-o, --output <file>', 'write result to FILE instead of stdout')\n .option('--pretty', 'pretty-print JSON output (default on TTY)')\n .option('--no-pretty', 'force single-line JSON output');\n\n cmd.action(async (words: string[], opts: SerpFlags) => {\n const query = words.join(' ');\n\n let page: number | undefined;\n if (opts.page !== undefined) {\n if (!isPositiveIntegerString(opts.page)) {\n cmd.error(`error: --page must be an integer >= 1, got: ${opts.page}`, {\n exitCode: EXIT_CODES.usage,\n code: 'wsai.invalidPage',\n });\n }\n page = Number(opts.page);\n }\n if (!urlIsStdin(query) && query.trim() === '') {\n cmd.error('error: serp requires a non-empty query', {\n exitCode: EXIT_CODES.usage,\n code: 'wsai.emptyQuery',\n });\n }\n\n const flags: SerpRawFlags = { engine: opts.engine, gl: opts.gl, hl: opts.hl, page };\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const client = createClient({ apiKey });\n const write = createEmitter(opts);\n\n for await (const q of queryIterator(query)) {\n const result = await client.serp(buildSerpOptions(q, flags));\n await write(result);\n }\n });\n\n return cmd;\n}\n","/**\n * Exit-code mapping for SDK errors.\n *\n * The SDK throws typed subclasses of `WebScrapingAIError`. Map each to a\n * stable exit code so shell scripts can branch on the failure mode without\n * parsing stderr.\n */\n\nimport {\n APIConnectionError,\n APIError,\n APITimeoutError,\n AuthenticationError,\n BadRequestError,\n GatewayTimeoutError,\n PaymentRequiredError,\n RateLimitError,\n ServerError,\n WebScrapingAIError,\n} from 'webscraping-ai';\n\nimport { redactSecrets } from './redact.js';\n\nexport const EXIT_CODES = {\n ok: 0,\n generic: 1,\n usage: 2,\n badRequest: 3,\n auth: 4,\n paymentRequired: 5,\n rateLimit: 6,\n serverError: 7,\n timeout: 8,\n connection: 9,\n} as const;\n\nexport function exitCodeFor(err: unknown): number {\n if (err instanceof BadRequestError) return EXIT_CODES.badRequest;\n if (err instanceof AuthenticationError) return EXIT_CODES.auth;\n if (err instanceof PaymentRequiredError) return EXIT_CODES.paymentRequired;\n if (err instanceof RateLimitError) return EXIT_CODES.rateLimit;\n if (err instanceof ServerError) return EXIT_CODES.serverError;\n if (err instanceof GatewayTimeoutError) return EXIT_CODES.timeout;\n if (err instanceof APITimeoutError) return EXIT_CODES.timeout;\n if (err instanceof APIConnectionError) return EXIT_CODES.connection;\n if (err instanceof APIError) return EXIT_CODES.serverError;\n if (err instanceof WebScrapingAIError) return EXIT_CODES.generic;\n return EXIT_CODES.generic;\n}\n\n/**\n * Human-readable one-liner for stderr. Always redacted: the API key and any\n * `api_key=...` pattern are replaced, since error bodies can echo the URL.\n */\nexport function formatError(err: unknown): string {\n return redactSecrets(rawMessage(err));\n}\n\nfunction rawMessage(err: unknown): string {\n if (err instanceof APIError) {\n const parts = [err.message];\n if (err.status) parts.push(`(HTTP ${err.status})`);\n return parts.join(' ');\n }\n if (err instanceof Error) return err.message;\n return String(err);\n}\n","import { Command } from 'commander';\n\nimport { EDITORS, editorById, installSkill, resolveTarget } from '../lib/skill.js';\nimport type { EditorAdapter } from '../lib/skill.js';\n\ninterface SetupSkillFlags {\n editor?: string[];\n all?: boolean;\n force?: boolean;\n}\n\nexport function setupCommand(): Command {\n const cmd = new Command('setup').description('Install the AI agent skill into your editor(s)');\n\n cmd\n .command('skill')\n .description('Install the WebScraping.AI skill into one or more AI coding editors')\n .option(\n '-e, --editor <id>',\n `editor id (repeatable). Supported: ${EDITORS.map((e) => e.id).join(', ')}`,\n collect,\n [] as string[],\n )\n .option('-a, --all', 'install into every supported editor')\n .option('-f, --force', 'overwrite existing skill files')\n .action(async (opts: SetupSkillFlags) => {\n const selected = resolveSelection(opts);\n if (selected.length === 0) {\n throw new Error(\n 'no editor selected — pass --all or --editor <id> (e.g. --editor claude-code)',\n );\n }\n\n for (const editor of selected) {\n const target = resolveTarget(editor);\n const result = await installSkill(target, { force: opts.force });\n process.stdout.write(`${editor.label}: ${result.status} → ${target.destination}\\n`);\n }\n });\n\n cmd\n .command('list', { hidden: false })\n .description('List supported editors for `setup skill`')\n .action(() => {\n for (const e of EDITORS) {\n const target = resolveTarget(e);\n process.stdout.write(`${e.id.padEnd(14)} ${e.label.padEnd(18)} ${target.destination}\\n`);\n }\n });\n\n return cmd;\n}\n\nfunction collect(value: string, previous: string[]): string[] {\n return [...previous, value];\n}\n\nfunction resolveSelection(opts: SetupSkillFlags): EditorAdapter[] {\n if (opts.all) return [...EDITORS];\n const ids = opts.editor ?? [];\n return ids.map((id) => {\n const editor = editorById(id);\n if (!editor) throw new Error(`unknown editor: ${id}`);\n return editor;\n });\n}\n","/**\n * Multi-editor skill installer.\n *\n * The canonical skill lives in this package at\n * `plugins/webscraping-ai/skills/webscraping-ai/SKILL.md`. Each supported\n * editor reads skill content from a different location and may want either\n * a single Markdown file or a whole skill directory. The `editors` registry\n * captures those differences so `setup skill` is a small driver over it.\n *\n * `mode: 'dir'` copies the entire skill directory (Claude Code-shaped\n * editors that scan a folder per skill). `mode: 'file'` writes the SKILL.md\n * contents to a single file (editors that read flat instructions, like\n * GitHub Copilot's `.github/copilot-instructions.md`).\n *\n * `kind: 'user'` writes to `$HOME`; `kind: 'project'` writes to the current\n * working directory. We prefer user-scope where the editor supports it so\n * the skill becomes globally available without polluting individual repos.\n */\n\nimport { promises as fs } from 'node:fs';\nimport { homedir } from 'node:os';\nimport { dirname, join, resolve } from 'node:path';\nimport { fileURLToPath } from 'node:url';\n\nexport type EditorMode = 'dir' | 'file';\nexport type EditorKind = 'user' | 'project';\n\nexport interface EditorAdapter {\n /** Stable identifier used by `--editor <name>`. */\n id: string;\n /** Human-readable label printed in summaries. */\n label: string;\n mode: EditorMode;\n kind: EditorKind;\n /**\n * Where the skill ends up. For `mode: 'dir'`, the skill directory copies\n * to `{root}/{path}`. For `mode: 'file'`, the SKILL.md contents write to\n * `{root}/{path}` (i.e. `path` ends in a filename).\n *\n * `root` is `$HOME` (user) or CWD (project).\n */\n path: string;\n}\n\n/**\n * Registry of supported editors. Order matters: it's also the order the\n * `setup skill` summary lists them in.\n */\nexport const EDITORS: readonly EditorAdapter[] = [\n // Claude Code: user-scope skills live under ~/.claude/skills/<name>/\n {\n id: 'claude-code',\n label: 'Claude Code',\n mode: 'dir',\n kind: 'user',\n path: '.claude/skills/webscraping-ai',\n },\n // Cursor: project-scope rule file. Cursor doesn't have a user-wide skill\n // dir, so this is the editor's \"install\" surface.\n {\n id: 'cursor',\n label: 'Cursor',\n mode: 'file',\n kind: 'project',\n path: '.cursor/rules/webscraping-ai.mdc',\n },\n // Windsurf: project-scope rules. Same shape as Cursor.\n {\n id: 'windsurf',\n label: 'Windsurf',\n mode: 'file',\n kind: 'project',\n path: '.windsurf/rules/webscraping-ai.md',\n },\n // Kiro: user-scope steering directory.\n { id: 'kiro', label: 'Kiro', mode: 'dir', kind: 'user', path: '.kiro/steering/webscraping-ai' },\n // OpenCode: project-scope skills (mirrors Claude shape).\n {\n id: 'opencode',\n label: 'OpenCode',\n mode: 'dir',\n kind: 'project',\n path: '.opencode/skills/webscraping-ai',\n },\n // Gemini CLI: user-scope extension dir with a GEMINI.md.\n {\n id: 'gemini',\n label: 'Gemini CLI',\n mode: 'file',\n kind: 'user',\n path: '.gemini/extensions/webscraping-ai/GEMINI.md',\n },\n // GitHub Copilot: project-scope custom instructions file. Single file.\n {\n id: 'copilot',\n label: 'GitHub Copilot',\n mode: 'file',\n kind: 'project',\n path: '.github/copilot-instructions.md',\n },\n // Augment: project-scope rules file.\n {\n id: 'augment',\n label: 'Augment',\n mode: 'file',\n kind: 'project',\n path: '.augment/rules/webscraping-ai.md',\n },\n // Factory: project-scope rules file.\n {\n id: 'factory',\n label: 'Factory',\n mode: 'file',\n kind: 'project',\n path: '.factory/rules/webscraping-ai.md',\n },\n] as const;\n\nexport function editorById(id: string): EditorAdapter | undefined {\n return EDITORS.find((e) => e.id === id);\n}\n\n/**\n * Where this package keeps the canonical skill. Resolved relative to the\n * compiled `dist/cli.js` so it works after `npm install -g`.\n */\nexport function bundledSkillDir(): string {\n const here = dirname(fileURLToPath(import.meta.url));\n // dist/ sits at the package root, so the plugins dir is two levels up\n // from any compiled file: dist/cli.js → package root → plugins/...\n return resolve(here, '..', 'plugins', 'webscraping-ai', 'skills', 'webscraping-ai');\n}\n\nexport interface InstallTarget {\n editor: EditorAdapter;\n /** Absolute destination path (file or directory depending on `mode`). */\n destination: string;\n}\n\nexport interface ResolveOptions {\n cwd?: string;\n home?: string;\n}\n\nexport function resolveTarget(editor: EditorAdapter, opts: ResolveOptions = {}): InstallTarget {\n const root = editor.kind === 'user' ? (opts.home ?? homedir()) : (opts.cwd ?? process.cwd());\n return { editor, destination: join(root, editor.path) };\n}\n\nexport interface InstallOptions {\n /** If true, overwrite existing files/dirs. */\n force?: boolean;\n /** Override skill source dir (for tests). */\n sourceDir?: string;\n}\n\nexport type InstallStatus = 'installed' | 'skipped-exists' | 'overwritten';\n\nexport interface InstallResult {\n target: InstallTarget;\n status: InstallStatus;\n}\n\nexport async function installSkill(\n target: InstallTarget,\n opts: InstallOptions = {},\n): Promise<InstallResult> {\n const source = opts.sourceDir ?? bundledSkillDir();\n const exists = await pathExists(target.destination);\n\n if (exists && !opts.force) {\n return { target, status: 'skipped-exists' };\n }\n\n if (target.editor.mode === 'dir') {\n if (exists) await fs.rm(target.destination, { recursive: true, force: true });\n await copyDirectory(source, target.destination);\n } else {\n const skillBody = await fs.readFile(join(source, 'SKILL.md'), 'utf8');\n await fs.mkdir(dirname(target.destination), { recursive: true });\n await fs.writeFile(target.destination, skillBody);\n }\n\n return { target, status: exists ? 'overwritten' : 'installed' };\n}\n\nasync function pathExists(path: string): Promise<boolean> {\n try {\n await fs.access(path);\n return true;\n } catch {\n return false;\n }\n}\n\nasync function copyDirectory(source: string, destination: string): Promise<void> {\n // `fs.cp` with `recursive` exists on Node 16.7+; we target Node 20.\n await fs.cp(source, destination, { recursive: true, force: true });\n}\n","import { Command, Option } from 'commander';\n\nimport { buildTextOptions } from '../lib/buildRequest.js';\nimport { resolveApiKey } from '../lib/config.js';\nimport { addCommonScrapeOptions, parseHeaders } from '../lib/options.js';\nimport type { CommonRawFlags } from '../lib/options.js';\nimport { createEmitter } from '../lib/output.js';\nimport { createClient } from '../lib/sdk.js';\nimport { urlIterator } from '../lib/stdin.js';\n\ninterface TextFlags extends CommonRawFlags {\n textFormat?: 'plain' | 'json' | 'xml';\n returnLinks?: boolean;\n}\n\nexport function textCommand(): Command {\n const cmd = new Command('text')\n .description('Fetch the visible text of a page')\n .argument('<url>', 'page URL (or `-` to read URLs from stdin)')\n .addOption(\n new Option('--text-format <fmt>', 'text response format').choices(['plain', 'json', 'xml']),\n )\n .option('--return-links', 'include page links (text-format=json only)');\n\n addCommonScrapeOptions(cmd);\n\n cmd.action(async (url: string, opts: TextFlags) => {\n const apiKey = await resolveApiKey({ flag: opts.apiKey });\n const headers = await parseHeaders(opts.headers);\n const client = createClient({ apiKey, requestTimeoutMs: opts.timeout });\n\n const write = createEmitter(opts);\n\n for await (const target of urlIterator(url)) {\n const sdkOptions = buildTextOptions(\n target,\n opts,\n { headers },\n {\n textFormat: opts.textFormat,\n returnLinks: opts.returnLinks,\n },\n );\n const result = await client.text(sdkOptions);\n await write(result);\n }\n });\n\n return cmd;\n}\n"],"mappings":";;;AAQA,SAAS,WAAAA,iBAAe;AACxB,SAAS,oBAAoB;AAC7B,SAAS,WAAAC,UAAS,WAAAC,gBAAe;AACjC,SAAS,iBAAAC,sBAAqB;;;ACX9B,SAAS,eAAe;;;ACYxB,SAAS,YAAY,UAAU;AAC/B,SAAS,eAAe;AACxB,SAAS,SAAS,YAAY;AAE9B,IAAM,kBAAkB;AAEjB,SAAS,YAAoB;AAClC,QAAM,MAAM,QAAQ,IAAI;AACxB,MAAI,OAAO,IAAI,KAAK,MAAM,GAAI,QAAO,KAAK,KAAK,gBAAgB;AAC/D,SAAO,KAAK,QAAQ,GAAG,WAAW,gBAAgB;AACpD;AAEO,SAAS,aAAqB;AACnC,SAAO,KAAK,UAAU,GAAG,eAAe;AAC1C;AAMA,eAAsB,sBAAgD;AACpE,MAAI;AACF,UAAM,OAAO,MAAM,GAAG,SAAS,WAAW,GAAG,MAAM;AACnD,UAAM,SAAS,KAAK,MAAM,IAAI;AAC9B,QAAI,UAAU,OAAO,WAAW,YAAY,CAAC,MAAM,QAAQ,MAAM,GAAG;AAClE,aAAO;AAAA,IACT;AACA,WAAO,CAAC;AAAA,EACV,SAAS,KAAK;AACZ,QAAK,IAA8B,SAAS,SAAU,QAAO,CAAC;AAC9D,UAAM;AAAA,EACR;AACF;AAEA,eAAsB,qBAAqB,QAAwC;AACjF,QAAM,OAAO,WAAW;AACxB,QAAM,GAAG,MAAM,QAAQ,IAAI,GAAG,EAAE,WAAW,KAAK,CAAC;AACjD,QAAM,GAAG,UAAU,MAAM,KAAK,UAAU,QAAQ,MAAM,CAAC,IAAI,MAAM,EAAE,MAAM,IAAM,CAAC;AAKhF,QAAM,GAAG,MAAM,MAAM,GAAK;AAC5B;AAEA,eAAsB,uBAAyC;AAC7D,MAAI;AACF,UAAM,GAAG,OAAO,WAAW,CAAC;AAC5B,WAAO;AAAA,EACT,SAAS,KAAK;AACZ,QAAK,IAA8B,SAAS,SAAU,QAAO;AAC7D,UAAM;AAAA,EACR;AACF;AAOA,eAAsB,cAAc,OAA6B,CAAC,GAAoB;AACpF,MAAI,KAAK,QAAQ,KAAK,KAAK,KAAK,MAAM,GAAI,QAAO,KAAK,KAAK,KAAK;AAEhE,QAAM,MAAM,QAAQ,IAAI;AACxB,MAAI,OAAO,IAAI,KAAK,MAAM,GAAI,QAAO,IAAI,KAAK;AAE9C,QAAM,YAAY,MAAM,oBAAoB;AAC5C,MAAI,UAAU,UAAU,UAAU,OAAO,KAAK,MAAM,GAAI,QAAO,UAAU,OAAO,KAAK;AAErF,QAAM,IAAI;AAAA,IACR;AAAA,EACF;AACF;AAEO,SAAS,WAAW,KAAqB;AAC9C,MAAI,IAAI,UAAU,EAAG,QAAO,IAAI,OAAO,IAAI,MAAM;AACjD,SAAO,GAAG,IAAI,MAAM,GAAG,CAAC,CAAC,GAAG,IAAI,OAAO,IAAI,SAAS,CAAC,CAAC,GAAG,IAAI,MAAM,EAAE,CAAC;AACxE;;;AC5EA,SAAS,YAAYC,WAAU;AAY/B,eAAsB,KACpB,OACA,OAAoB,CAAC,GACrB,QAA0B,CAAC,GACZ;AACf,QAAM,OAAO,UAAU,OAAO,IAAI;AAElC,MAAI,KAAK,UAAU,KAAK,OAAO,KAAK,MAAM,IAAI;AAC5C,UAAM,OAAO,QAAQ,KAAK,SAAS,IAAI,IAAI,KAAK;AAChD,QAAI,MAAM,OAAQ,OAAMA,IAAG,WAAW,KAAK,QAAQ,IAAI;AAAA,QAClD,OAAMA,IAAG,UAAU,KAAK,QAAQ,IAAI;AACzC;AAAA,EACF;AAEA,UAAQ,OAAO,MAAM,IAAI;AACzB,MAAI,CAAC,KAAK,SAAS,IAAI,EAAG,SAAQ,OAAO,MAAM,IAAI;AACrD;AAMO,SAAS,cAAc,OAAoB,CAAC,GAAsC;AACvF,MAAI,QAAQ;AACZ,SAAO,OAAO,UAAmB;AAC/B,UAAM,SAAS,CAAC;AAChB,YAAQ;AACR,UAAM,KAAK,OAAO,MAAM,EAAE,OAAO,CAAC;AAAA,EACpC;AACF;AAEO,SAAS,UAAU,OAAgB,OAAoB,CAAC,GAAW;AACxE,MAAI,OAAO,UAAU,SAAU,QAAO;AAEtC,QAAM,SAAS,KAAK,UAAU,QAAQ,OAAO,SAAS;AACtD,SAAO,KAAK,UAAU,OAAO,MAAM,SAAS,IAAI,CAAC;AACnD;;;ACjDA,SAAS,qBAAqB;;;ACJ9B,IAAM,UAAU,oBAAI,IAAY;AAGzB,SAAS,eAAe,OAAiC;AAC9D,MAAI,SAAS,MAAM,KAAK,MAAM,GAAI,SAAQ,IAAI,KAAK;AACrD;AAOA,IAAM,gBAAgB;AAEf,SAAS,cAAc,MAAsB;AAClD,MAAI,MAAM,KAAK,QAAQ,eAAe,cAAc;AACpD,QAAM,aAAa,IAAI,IAAI,OAAO;AAClC,QAAM,MAAM,QAAQ,IAAI,wBAAwB,KAAK;AACrD,MAAI,IAAK,YAAW,IAAI,GAAG;AAC3B,aAAW,UAAU,YAAY;AAC/B,eAAW,QAAQ,oBAAI,IAAI,CAAC,QAAQ,mBAAmB,MAAM,CAAC,CAAC,GAAG;AAChE,YAAM,IAAI,MAAM,IAAI,EAAE,KAAK,YAAY;AAAA,IACzC;AAAA,EACF;AACA,SAAO;AACT;;;ADEO,SAAS,aAAa,SAA6C;AACxE,iBAAe,QAAQ,MAAM;AAC7B,QAAM,QAAQ,IAAI,cAAc;AAAA,IAC9B,QAAQ,QAAQ;AAAA,IAChB,WAAW,QAAQ;AAAA,EACrB,CAAC;AAKD,QAAM,MAAM,EAAE,UAAU,KAAK;AAE7B,SAAO,IAAI,MAAM,OAAO;AAAA,IACtB,IAAI,QAAQ,MAAM,UAAU;AAC1B,YAAM,QAAQ,QAAQ,IAAI,QAAQ,MAAM,QAAQ;AAChD,UAAI,OAAO,UAAU,WAAY,QAAO;AACxC,cAAQ,MAAM;AAAA,QACZ,KAAK;AACH,iBAAO,CAAC,MAAmB,OAAO,KAAK,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACzD,KAAK;AACH,iBAAO,CAAC,MAAmB,OAAO,KAAK,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACzD,KAAK;AACH,iBAAO,CAAC,MAAuB,OAAO,SAAS,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjE,KAAK;AACH,iBAAO,CAAC,MAA+B,OAAO,iBAAiB,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjF,KAAK;AACH,iBAAO,CAAC,MAAuB,OAAO,SAAS,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACjE,KAAK;AACH,iBAAO,CAAC,MAAqB,OAAO,OAAO,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QAC7D,KAAK;AAIH,iBAAO,CAAC,MAAmB,OAAO,KAAK,EAAE,GAAG,GAAG,GAAG,IAAI,CAAC;AAAA,QACzD,KAAK;AAGH,iBAAO,MAAM,KAAK,MAAM;AAAA,QAC1B;AACE,iBAAO,OAAO,UAAU,aAAa,MAAM,KAAK,MAAM,IAAI;AAAA,MAC9D;AAAA,IACF;AAAA,EACF,CAAC;AACH;;;AHlEO,SAAS,iBAA0B;AACxC,SAAO,IAAI,QAAQ,SAAS,EACzB,YAAY,0CAA0C,EACtD,OAAO,uBAAuB,6DAA6D,EAC3F,OAAO,uBAAuB,wCAAwC,EACtE,OAAO,YAAY,2CAA2C,EAC9D,OAAO,eAAe,+BAA+B,EACrD,OAAO,OAAO,SAAuB;AACpC,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,SAAS,aAAa,EAAE,OAAO,CAAC;AACtC,UAAM,SAAS,MAAM,OAAO,QAAQ;AACpC,UAAM,KAAK,QAAQ,IAAI;AAAA,EACzB,CAAC;AACL;;;AKzBA,SAAS,WAAAC,UAAS,UAAAC,eAAc;;;AC0BzB,SAAS,mBACd,OACA,WAAkC,CAAC,GACb;AACtB,QAAM,MAA4B,CAAC;AAInC,MAAI,MAAM,OAAO,OAAW,KAAI,KAAK,MAAM;AAC3C,MAAI,MAAM,UAAU,OAAW,KAAI,QAAQ,MAAM;AACjD,MAAI,MAAM,YAAY,OAAW,KAAI,UAAU,MAAM;AACrD,MAAI,MAAM,WAAW,OAAW,KAAI,SAAS,MAAM;AACnD,MAAI,MAAM,YAAY,OAAW,KAAI,UAAU,MAAM;AACrD,MAAI,MAAM,cAAc,OAAW,KAAI,aAAa,MAAM;AAC1D,MAAI,MAAM,YAAY,OAAW,KAAI,WAAW,MAAM;AACtD,MAAI,MAAM,aAAa,OAAW,KAAI,YAAY,MAAM;AACxD,MAAI,MAAM,gBAAgB,OAAW,KAAI,eAAe,MAAM;AAC9D,MAAI,MAAM,WAAY,KAAI,eAAe;AACzC,MAAI,MAAM,gBAAiB,KAAI,oBAAoB;AACnD,MAAI,SAAS,QAAS,KAAI,UAAU,SAAS;AAC7C,SAAO;AACT;AAMO,SAAS,iBACd,KACA,OACA,UACA,SAAqB,CAAC,GACT;AACb,QAAM,MAAmB,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACvE,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,MAAI,OAAO,mBAAoB,KAAI,uBAAuB;AAC1D,SAAO;AACT;AAOO,SAAS,iBACd,KACA,OACA,UACA,SAAqB,CAAC,GACT;AACb,QAAM,MAAmB,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACvE,MAAI,OAAO,WAAY,KAAI,cAAc,OAAO;AAChD,MAAI,OAAO,YAAa,KAAI,eAAe;AAC3C,SAAO;AACT;AAEO,SAAS,qBACd,KACA,UACA,OACA,UACiB;AAIjB,QAAM,MAAgC,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACpF,MAAI,aAAa,OAAW,KAAI,WAAW;AAC3C,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,SAAO;AACT;AAEO,SAAS,6BACd,KACA,WACA,OACA,UACyB;AAIzB,QAAM,MAAwC,EAAE,KAAK,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AAC5F,MAAI,UAAU,SAAS,EAAG,KAAI,YAAY;AAC1C,SAAO;AACT;AAEO,SAAS,qBACd,KACA,UACA,OACA,UACiB;AACjB,QAAM,MAAuB,EAAE,KAAK,UAAU,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AACrF,MAAI,MAAM,OAAQ,KAAI,SAAS,MAAM;AACrC,SAAO;AACT;AAEO,SAAS,mBACd,KACA,QACA,OACA,UACe;AACf,SAAO,EAAE,KAAK,QAAQ,GAAG,mBAAmB,OAAO,QAAQ,EAAE;AAC/D;AAgBO,SAAS,iBAAiB,OAAe,QAAsB,CAAC,GAAgB;AAErF,MAAI,OAAO,UAAU,YAAY,MAAM,KAAK,MAAM,IAAI;AACpD,UAAM,IAAI,MAAM,iCAAiC;AAAA,EACnD;AACA,QAAM,MAAmB,EAAE,GAAG,MAAM;AACpC,MAAI,MAAM,WAAW,OAAW,KAAI,SAAS,MAAM;AACnD,MAAI,MAAM,GAAI,KAAI,KAAK,MAAM;AAC7B,MAAI,MAAM,GAAI,KAAI,KAAK,MAAM;AAC7B,MAAI,MAAM,SAAS,QAAW;AAC5B,QAAI,CAAC,OAAO,cAAc,MAAM,IAAI,KAAK,MAAM,OAAO,GAAG;AACvD,YAAM,IAAI,MAAM,wCAAwC,MAAM,IAAI,EAAE;AAAA,IACtE;AACA,QAAI,OAAO,MAAM;AAAA,EACnB;AACA,SAAO;AACT;;;ACzJA,SAAS,YAAYC,WAAU;AAC/B,SAAS,cAAc;AA0BhB,SAAS,uBAAuB,KAAuB;AAC5D,SAAO,IACJ,OAAO,uBAAuB,6DAA6D,EAC3F,OAAO,uBAAuB,wCAAwC,EACtE,OAAO,YAAY,2CAA2C,EAC9D,OAAO,eAAe,+BAA+B,EACrD,OAAO,QAAQ,sDAAsD,EACrE,OAAO,WAAW,wCAAwC,EAC1D;AAAA,IACC,IAAI,OAAO,kBAAkB,YAAY,EAAE,QAAQ,CAAC,cAAc,eAAe,SAAS,CAAC;AAAA,EAC7F,EACC,OAAO,oBAAoB,gDAAgD,EAC3E;AAAA,IACC,IAAI,OAAO,mBAAmB,kBAAkB,EAAE,QAAQ,CAAC,WAAW,UAAU,QAAQ,CAAC;AAAA,EAC3F,EACC,OAAO,kBAAkB,4CAA4C,aAAa,EAClF,OAAO,qBAAqB,kDAAkD,aAAa,EAC3F,OAAO,yBAAyB,mDAAmD,EACnF;AAAA,IACC;AAAA,IACA;AAAA,EACF,EACC,OAAO,sBAAsB,0CAA0C,EACvE,OAAO,wBAAwB,8CAA8C,EAC7E,OAAO,kBAAkB,qBAAqB,EAC9C,OAAO,uBAAuB,0BAA0B;AAC7D;AAQO,SAAS,wBAAwB,OAAwB;AAG9D,SAAO,aAAa,KAAK,KAAK,KAAK,OAAO,cAAc,OAAO,KAAK,CAAC;AACvE;AAEO,SAAS,cAAc,OAAuB;AACnD,QAAM,IAAI,OAAO,SAAS,OAAO,EAAE;AACnC,MAAI,CAAC,OAAO,SAAS,CAAC,EAAG,OAAM,IAAI,MAAM,0BAA0B,KAAK,EAAE;AAC1E,SAAO;AACT;AAMA,eAAsB,aACpB,OAC6C;AAC7C,MAAI,CAAC,SAAS,MAAM,KAAK,MAAM,GAAI,QAAO;AAC1C,QAAM,OAAO,MAAM,WAAW,GAAG,IAAI,MAAMA,IAAG,SAAS,MAAM,MAAM,CAAC,GAAG,MAAM,IAAI;AACjF,SAAO,gBAAgB,MAAM,SAAS;AACxC;AAMA,eAAsB,YAAY,OAAgD;AAChF,QAAM,OAAO,MAAM,WAAW,GAAG,IAAI,MAAMA,IAAG,SAAS,MAAM,MAAM,CAAC,GAAG,MAAM,IAAI;AACjF,QAAM,SAAS,gBAAgB,MAAM,QAAQ;AAC7C,MAAI,CAAC,UAAU,OAAO,KAAK,MAAM,EAAE,WAAW,GAAG;AAC/C,UAAM,IAAI,MAAM,sEAAsE;AAAA,EACxF;AACA,SAAO;AACT;AAEA,SAAS,gBAAgB,MAAc,OAAuC;AAC5E,MAAI;AACJ,MAAI;AACF,aAAS,KAAK,MAAM,IAAI;AAAA,EAC1B,SAAS,KAAK;AACZ,UAAM,IAAI,MAAM,GAAG,KAAK,yBAAqB,IAAc,OAAO,EAAE;AAAA,EACtE;AACA,MAAI,CAAC,UAAU,OAAO,WAAW,YAAY,MAAM,QAAQ,MAAM,GAAG;AAClE,UAAM,IAAI,MAAM,GAAG,KAAK,wBAAwB;AAAA,EAClD;AACA,QAAM,MAA8B,CAAC;AACrC,aAAW,CAAC,GAAG,CAAC,KAAK,OAAO,QAAQ,MAAiC,GAAG;AACtE,QAAI,OAAO,MAAM,SAAU,OAAM,IAAI,MAAM,GAAG,KAAK,IAAI,CAAC,0BAA0B;AAClF,QAAI,CAAC,IAAI;AAAA,EACX;AACA,SAAO;AACT;;;AC/GA,OAAO,cAAc;AAEd,SAAS,WAAW,KAAsB;AAC/C,SAAO,QAAQ;AACjB;AAEA,gBAAuB,oBAA2C;AAChE,QAAM,KAAK,SAAS,gBAAgB,EAAE,OAAO,QAAQ,OAAO,WAAW,SAAS,CAAC;AACjF,mBAAiB,OAAO,IAAI;AAC1B,UAAM,OAAO,IAAI,KAAK;AACtB,QAAI,SAAS,MAAM,KAAK,WAAW,GAAG,EAAG;AACzC,UAAM;AAAA,EACR;AACF;AAMA,gBAAuB,qBACrB,QAA+B,QAAQ,OAChB;AACvB,QAAM,KAAK,SAAS,gBAAgB,EAAE,OAAO,WAAW,SAAS,CAAC;AAClE,mBAAiB,QAAQ,IAAI;AAC3B,QAAI,KAAK,KAAK,MAAM,GAAI;AACxB,UAAM;AAAA,EACR;AACF;AAKA,gBAAuB,YAAY,KAAoC;AACrE,MAAI,WAAW,GAAG,GAAG;AACnB,WAAO,kBAAkB;AACzB;AAAA,EACF;AACA,QAAM;AACR;AAKA,gBAAuB,cAAc,KAAoC;AACvE,MAAI,WAAW,GAAG,GAAG;AACnB,WAAO,qBAAqB;AAC5B;AAAA,EACF;AACA,QAAM;AACR;;;AH9CO,SAAS,aAAsB;AACpC,QAAM,MAAM,IAAIC,SAAQ,KAAK,EAC1B,YAAY,sDAAsD,EAClE,SAAS,SAAS,2CAA2C,EAC7D,eAAe,yBAAyB,2CAA2C,EACnF,UAAU,IAAIC,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAmB;AAChD,QAAI,CAAC,KAAK,SAAU,OAAM,IAAI,MAAM,wBAAwB;AAC5D,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,qBAAqB,QAAQ,KAAK,UAAU,MAAM,EAAE,QAAQ,CAAC;AAChF,YAAM,SAAS,MAAM,OAAO,SAAS,UAAU;AAC/C,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AIvCA,SAAS,WAAAC,gBAAe;AAUjB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAAE,YAAY,4BAA4B;AAExE,MACG,QAAQ,WAAW,EACnB,YAAY,sBAAsB,WAAW,CAAC,EAAE,EAChD,OAAO,OAAO,QAAgB;AAC7B,QAAI,CAAC,OAAO,IAAI,KAAK,MAAM,GAAI,OAAM,IAAI,MAAM,kBAAkB;AACjE,UAAM,qBAAqB,EAAE,QAAQ,IAAI,KAAK,EAAE,CAAC;AACjD,YAAQ,OAAO,MAAM,oBAAoB,WAAW,CAAC;AAAA,CAAI;AAAA,EAC3D,CAAC;AAEH,MACG,QAAQ,MAAM,EACd,YAAY,+CAA+C,EAC3D,OAAO,YAAY;AAClB,UAAM,YAAY,MAAM,oBAAoB;AAC5C,UAAM,MAAM,QAAQ,IAAI;AACxB,QAAI,OAAO,IAAI,KAAK,MAAM,IAAI;AAC5B,cAAQ,OAAO;AAAA,QACb,4CAA4C,WAAW,IAAI,KAAK,CAAC,CAAC;AAAA;AAAA,MACpE;AAAA,IACF;AACA,QAAI,UAAU,UAAU,UAAU,OAAO,KAAK,MAAM,IAAI;AACtD,cAAQ,OAAO;AAAA,QACb,eAAe,WAAW,CAAC,MAAM,WAAW,UAAU,OAAO,KAAK,CAAC,CAAC;AAAA;AAAA,MACtE;AAAA,IACF;AACA,QAAI,CAAC,OAAO,CAAC,UAAU,QAAQ;AAC7B,cAAQ,OAAO,MAAM,+DAA+D;AAAA,IACtF;AAAA,EACF,CAAC;AAEH,MACG,QAAQ,OAAO,EACf,YAAY,gCAAgC,EAC5C,OAAO,YAAY;AAClB,UAAM,UAAU,MAAM,qBAAqB;AAC3C,YAAQ,OAAO;AAAA,MACb,UAAU,WAAW,WAAW,CAAC;AAAA,IAAO;AAAA,IAC1C;AAAA,EACF,CAAC;AAEH,SAAO;AACT;;;ACtDA,SAAS,WAAAC,gBAAe;AAcjB,SAAS,iBAA0B;AACxC,QAAM,MAAM,IAAIC,SAAQ,SAAS,EAC9B,YAAY,mEAAmE,EAC/E,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,EACF;AAEF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAuB;AACpD,QAAI,CAAC,KAAK,OAAQ,OAAM,IAAI,MAAM,sBAAsB;AACxD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,MAAM,YAAY,KAAK,MAAM;AAC5C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,mBAAmB,QAAQ,QAAQ,MAAM,EAAE,QAAQ,CAAC;AACvE,YAAM,SAAS,MAAM,OAAO,OAAO,UAAU;AAC7C,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AC1CA,SAAS,WAAAC,UAAS,UAAAC,eAAc;AAczB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAC3B,YAAY,+BAA+B,EAC3C,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC,IAAIC,QAAO,0BAA0B,kDAAkD;AAAA,EACzF,EACC,UAAU,IAAIA,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAoB;AACjD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa;AAAA,QACjB;AAAA,QACA;AAAA,QACA,EAAE,QAAQ;AAAA,QACV;AAAA,UACE,oBAAoB,KAAK;AAAA,QAC3B;AAAA,MACF;AACA,YAAM,SAAS,MAAM,OAAO,KAAK,UAAU;AAC3C,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AC/CA,SAAS,WAAAC,UAAS,UAAAC,eAAc;AAczB,SAAS,kBAA2B;AACzC,QAAM,MAAM,IAAIC,SAAQ,UAAU,EAC/B,YAAY,0CAA0C,EACtD,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,EACF,EACC,UAAU,IAAIC,QAAO,kBAAkB,mBAAmB,EAAE,QAAQ,CAAC,QAAQ,MAAM,CAAC,CAAC;AAExF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAwB;AACrD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,qBAAqB,QAAQ,KAAK,UAAU,MAAM,EAAE,QAAQ,CAAC;AAChF,YAAM,SAAS,MAAM,OAAO,SAAS,UAAU;AAC/C,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;ACzCA,SAAS,WAAAC,gBAAe;AAcjB,SAAS,0BAAmC;AACjD,QAAM,MAAM,IAAIC,SAAQ,mBAAmB,EACxC,YAAY,gDAAgD,EAC5D,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC;AAAA,IACA;AAAA,IACA;AAAA,IACA,CAAC;AAAA,EACH;AAEF,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAgC;AAE7D,UAAM,YAAY,KAAK,YAAY,CAAC;AACpC,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa,6BAA6B,QAAQ,WAAW,MAAM,EAAE,QAAQ,CAAC;AACpF,YAAM,SAAS,MAAM,OAAO,iBAAiB,UAAU;AACvD,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;AAEA,SAAS,aAAa,OAAe,UAA8B;AACjE,SAAO,CAAC,GAAG,UAAU,KAAK;AAC5B;;;AChDA,SAAS,WAAAC,UAAS,UAAAC,eAAc;;;ACQhC;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,OACK;AAIA,IAAM,aAAa;AAAA,EACxB,IAAI;AAAA,EACJ,SAAS;AAAA,EACT,OAAO;AAAA,EACP,YAAY;AAAA,EACZ,MAAM;AAAA,EACN,iBAAiB;AAAA,EACjB,WAAW;AAAA,EACX,aAAa;AAAA,EACb,SAAS;AAAA,EACT,YAAY;AACd;AAEO,SAAS,YAAY,KAAsB;AAChD,MAAI,eAAe,gBAAiB,QAAO,WAAW;AACtD,MAAI,eAAe,oBAAqB,QAAO,WAAW;AAC1D,MAAI,eAAe,qBAAsB,QAAO,WAAW;AAC3D,MAAI,eAAe,eAAgB,QAAO,WAAW;AACrD,MAAI,eAAe,YAAa,QAAO,WAAW;AAClD,MAAI,eAAe,oBAAqB,QAAO,WAAW;AAC1D,MAAI,eAAe,gBAAiB,QAAO,WAAW;AACtD,MAAI,eAAe,mBAAoB,QAAO,WAAW;AACzD,MAAI,eAAe,SAAU,QAAO,WAAW;AAC/C,MAAI,eAAe,mBAAoB,QAAO,WAAW;AACzD,SAAO,WAAW;AACpB;AAMO,SAAS,YAAY,KAAsB;AAChD,SAAO,cAAc,WAAW,GAAG,CAAC;AACtC;AAEA,SAAS,WAAW,KAAsB;AACxC,MAAI,eAAe,UAAU;AAC3B,UAAM,QAAQ,CAAC,IAAI,OAAO;AAC1B,QAAI,IAAI,OAAQ,OAAM,KAAK,SAAS,IAAI,MAAM,GAAG;AACjD,WAAO,MAAM,KAAK,GAAG;AAAA,EACvB;AACA,MAAI,eAAe,MAAO,QAAO,IAAI;AACrC,SAAO,OAAO,GAAG;AACnB;;;ADnCO,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,SAAQ,MAAM,EAC3B,MAAM,QAAQ,EACd,YAAY,iFAAiF,EAC7F,SAAS,cAAc,gEAAgE,EACvF,UAAU,IAAIC,QAAO,mBAAmB,iCAAiC,EAAE,QAAQ,CAAC,QAAQ,CAAC,CAAC,EAC9F,OAAO,eAAe,8DAA8D,EACpF,OAAO,eAAe,wDAAwD,EAC9E;AAAA,IACC;AAAA,IACA;AAAA,EACF,EACC,OAAO,uBAAuB,6DAA6D,EAC3F,OAAO,uBAAuB,wCAAwC,EACtE,OAAO,YAAY,2CAA2C,EAC9D,OAAO,eAAe,+BAA+B;AAExD,MAAI,OAAO,OAAO,OAAiB,SAAoB;AACrD,UAAM,QAAQ,MAAM,KAAK,GAAG;AAE5B,QAAI;AACJ,QAAI,KAAK,SAAS,QAAW;AAC3B,UAAI,CAAC,wBAAwB,KAAK,IAAI,GAAG;AACvC,YAAI,MAAM,+CAA+C,KAAK,IAAI,IAAI;AAAA,UACpE,UAAU,WAAW;AAAA,UACrB,MAAM;AAAA,QACR,CAAC;AAAA,MACH;AACA,aAAO,OAAO,KAAK,IAAI;AAAA,IACzB;AACA,QAAI,CAAC,WAAW,KAAK,KAAK,MAAM,KAAK,MAAM,IAAI;AAC7C,UAAI,MAAM,0CAA0C;AAAA,QAClD,UAAU,WAAW;AAAA,QACrB,MAAM;AAAA,MACR,CAAC;AAAA,IACH;AAEA,UAAM,QAAsB,EAAE,QAAQ,KAAK,QAAQ,IAAI,KAAK,IAAI,IAAI,KAAK,IAAI,KAAK;AAClF,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,SAAS,aAAa,EAAE,OAAO,CAAC;AACtC,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,KAAK,cAAc,KAAK,GAAG;AAC1C,YAAM,SAAS,MAAM,OAAO,KAAK,iBAAiB,GAAG,KAAK,CAAC;AAC3D,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AEhFA,SAAS,WAAAC,gBAAe;;;ACmBxB,SAAS,YAAYC,WAAU;AAC/B,SAAS,WAAAC,gBAAe;AACxB,SAAS,WAAAC,UAAS,QAAAC,OAAM,eAAe;AACvC,SAAS,qBAAqB;AA0BvB,IAAM,UAAoC;AAAA;AAAA,EAE/C;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA;AAAA,EAGA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA,EAAE,IAAI,QAAQ,OAAO,QAAQ,MAAM,OAAO,MAAM,QAAQ,MAAM,gCAAgC;AAAA;AAAA,EAE9F;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AAAA;AAAA,EAEA;AAAA,IACE,IAAI;AAAA,IACJ,OAAO;AAAA,IACP,MAAM;AAAA,IACN,MAAM;AAAA,IACN,MAAM;AAAA,EACR;AACF;AAEO,SAAS,WAAW,IAAuC;AAChE,SAAO,QAAQ,KAAK,CAAC,MAAM,EAAE,OAAO,EAAE;AACxC;AAMO,SAAS,kBAA0B;AACxC,QAAM,OAAOD,SAAQ,cAAc,YAAY,GAAG,CAAC;AAGnD,SAAO,QAAQ,MAAM,MAAM,WAAW,kBAAkB,UAAU,gBAAgB;AACpF;AAaO,SAAS,cAAc,QAAuB,OAAuB,CAAC,GAAkB;AAC7F,QAAM,OAAO,OAAO,SAAS,SAAU,KAAK,QAAQD,SAAQ,IAAM,KAAK,OAAO,QAAQ,IAAI;AAC1F,SAAO,EAAE,QAAQ,aAAaE,MAAK,MAAM,OAAO,IAAI,EAAE;AACxD;AAgBA,eAAsB,aACpB,QACA,OAAuB,CAAC,GACA;AACxB,QAAM,SAAS,KAAK,aAAa,gBAAgB;AACjD,QAAM,SAAS,MAAM,WAAW,OAAO,WAAW;AAElD,MAAI,UAAU,CAAC,KAAK,OAAO;AACzB,WAAO,EAAE,QAAQ,QAAQ,iBAAiB;AAAA,EAC5C;AAEA,MAAI,OAAO,OAAO,SAAS,OAAO;AAChC,QAAI,OAAQ,OAAMH,IAAG,GAAG,OAAO,aAAa,EAAE,WAAW,MAAM,OAAO,KAAK,CAAC;AAC5E,UAAM,cAAc,QAAQ,OAAO,WAAW;AAAA,EAChD,OAAO;AACL,UAAM,YAAY,MAAMA,IAAG,SAASG,MAAK,QAAQ,UAAU,GAAG,MAAM;AACpE,UAAMH,IAAG,MAAME,SAAQ,OAAO,WAAW,GAAG,EAAE,WAAW,KAAK,CAAC;AAC/D,UAAMF,IAAG,UAAU,OAAO,aAAa,SAAS;AAAA,EAClD;AAEA,SAAO,EAAE,QAAQ,QAAQ,SAAS,gBAAgB,YAAY;AAChE;AAEA,eAAe,WAAW,MAAgC;AACxD,MAAI;AACF,UAAMA,IAAG,OAAO,IAAI;AACpB,WAAO;AAAA,EACT,QAAQ;AACN,WAAO;AAAA,EACT;AACF;AAEA,eAAe,cAAc,QAAgB,aAAoC;AAE/E,QAAMA,IAAG,GAAG,QAAQ,aAAa,EAAE,WAAW,MAAM,OAAO,KAAK,CAAC;AACnE;;;AD3LO,SAAS,eAAwB;AACtC,QAAM,MAAM,IAAII,SAAQ,OAAO,EAAE,YAAY,gDAAgD;AAE7F,MACG,QAAQ,OAAO,EACf,YAAY,qEAAqE,EACjF;AAAA,IACC;AAAA,IACA,sCAAsC,QAAQ,IAAI,CAAC,MAAM,EAAE,EAAE,EAAE,KAAK,IAAI,CAAC;AAAA,IACzE;AAAA,IACA,CAAC;AAAA,EACH,EACC,OAAO,aAAa,qCAAqC,EACzD,OAAO,eAAe,gCAAgC,EACtD,OAAO,OAAO,SAA0B;AACvC,UAAM,WAAW,iBAAiB,IAAI;AACtC,QAAI,SAAS,WAAW,GAAG;AACzB,YAAM,IAAI;AAAA,QACR;AAAA,MACF;AAAA,IACF;AAEA,eAAW,UAAU,UAAU;AAC7B,YAAM,SAAS,cAAc,MAAM;AACnC,YAAM,SAAS,MAAM,aAAa,QAAQ,EAAE,OAAO,KAAK,MAAM,CAAC;AAC/D,cAAQ,OAAO,MAAM,GAAG,OAAO,KAAK,KAAK,OAAO,MAAM,WAAM,OAAO,WAAW;AAAA,CAAI;AAAA,IACpF;AAAA,EACF,CAAC;AAEH,MACG,QAAQ,QAAQ,EAAE,QAAQ,MAAM,CAAC,EACjC,YAAY,0CAA0C,EACtD,OAAO,MAAM;AACZ,eAAW,KAAK,SAAS;AACvB,YAAM,SAAS,cAAc,CAAC;AAC9B,cAAQ,OAAO,MAAM,GAAG,EAAE,GAAG,OAAO,EAAE,CAAC,IAAI,EAAE,MAAM,OAAO,EAAE,CAAC,IAAI,OAAO,WAAW;AAAA,CAAI;AAAA,IACzF;AAAA,EACF,CAAC;AAEH,SAAO;AACT;AAEA,SAAS,QAAQ,OAAe,UAA8B;AAC5D,SAAO,CAAC,GAAG,UAAU,KAAK;AAC5B;AAEA,SAAS,iBAAiB,MAAwC;AAChE,MAAI,KAAK,IAAK,QAAO,CAAC,GAAG,OAAO;AAChC,QAAM,MAAM,KAAK,UAAU,CAAC;AAC5B,SAAO,IAAI,IAAI,CAAC,OAAO;AACrB,UAAM,SAAS,WAAW,EAAE;AAC5B,QAAI,CAAC,OAAQ,OAAM,IAAI,MAAM,mBAAmB,EAAE,EAAE;AACpD,WAAO;AAAA,EACT,CAAC;AACH;;;AEjEA,SAAS,WAAAC,WAAS,UAAAC,eAAc;AAezB,SAAS,cAAuB;AACrC,QAAM,MAAM,IAAIC,UAAQ,MAAM,EAC3B,YAAY,kCAAkC,EAC9C,SAAS,SAAS,2CAA2C,EAC7D;AAAA,IACC,IAAIC,QAAO,uBAAuB,sBAAsB,EAAE,QAAQ,CAAC,SAAS,QAAQ,KAAK,CAAC;AAAA,EAC5F,EACC,OAAO,kBAAkB,4CAA4C;AAExE,yBAAuB,GAAG;AAE1B,MAAI,OAAO,OAAO,KAAa,SAAoB;AACjD,UAAM,SAAS,MAAM,cAAc,EAAE,MAAM,KAAK,OAAO,CAAC;AACxD,UAAM,UAAU,MAAM,aAAa,KAAK,OAAO;AAC/C,UAAM,SAAS,aAAa,EAAE,QAAQ,kBAAkB,KAAK,QAAQ,CAAC;AAEtE,UAAM,QAAQ,cAAc,IAAI;AAEhC,qBAAiB,UAAU,YAAY,GAAG,GAAG;AAC3C,YAAM,aAAa;AAAA,QACjB;AAAA,QACA;AAAA,QACA,EAAE,QAAQ;AAAA,QACV;AAAA,UACE,YAAY,KAAK;AAAA,UACjB,aAAa,KAAK;AAAA,QACpB;AAAA,MACF;AACA,YAAM,SAAS,MAAM,OAAO,KAAK,UAAU;AAC3C,YAAM,MAAM,MAAM;AAAA,IACpB;AAAA,EACF,CAAC;AAED,SAAO;AACT;;;AnBxBA,SAAS,iBAAyB;AAGhC,QAAM,OAAOC,SAAQC,eAAc,YAAY,GAAG,CAAC;AACnD,QAAM,UAAUC,SAAQ,MAAM,MAAM,cAAc;AAClD,QAAM,MAAM,KAAK,MAAM,aAAa,SAAS,MAAM,CAAC;AACpD,SAAO,IAAI;AACb;AAEO,SAAS,eAAwB;AACtC,QAAM,UAAU,IAAIC,UAAQ;AAC5B,UACG,KAAK,gBAAgB,EACrB;AAAA,IACC;AAAA,EAEF,EACC,QAAQ,eAAe,GAAG,iBAAiB,2BAA2B,EACtE,mBAAmB;AAEtB,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,gBAAgB,CAAC;AACpC,UAAQ,WAAW,wBAAwB,CAAC;AAC5C,UAAQ,WAAW,WAAW,CAAC;AAC/B,UAAQ,WAAW,eAAe,CAAC;AACnC,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,eAAe,CAAC;AACnC,UAAQ,WAAW,YAAY,CAAC;AAChC,UAAQ,WAAW,aAAa,CAAC;AAEjC,SAAO;AACT;AAEA,eAAe,KAAK,MAA+B;AACjD,QAAM,UAAU,aAAa;AAC7B,MAAI;AACF,UAAM,QAAQ,WAAW,IAAI;AAAA,EAC/B,SAAS,KAAK;AACZ,YAAQ,OAAO,MAAM,UAAU,YAAY,GAAG,CAAC;AAAA,CAAI;AACnD,YAAQ,KAAK,YAAY,GAAG,CAAC;AAAA,EAC/B;AACF;AAIA,IAAM,YAAY,QAAQ,KAAK,CAAC,IAAIF,eAAc,YAAY,GAAG,IAAI;AACrE,IAAI,aAAa,QAAQ,KAAK,CAAC,MAAM,WAAW;AAC9C,OAAK,QAAQ,IAAI,EAAE,MAAM,CAAC,QAAiB;AACzC,YAAQ,OAAO,MAAM,UAAU,YAAY,GAAG,CAAC;AAAA,CAAI;AACnD,YAAQ,KAAK,YAAY,GAAG,CAAC;AAAA,EAC/B,CAAC;AACH;","names":["Command","dirname","resolve","fileURLToPath","fs","Command","Option","fs","Command","Option","Command","Command","Command","Command","Command","Option","Command","Option","Command","Option","Command","Option","Command","Command","Command","Option","Command","Option","Command","fs","homedir","dirname","join","Command","Command","Option","Command","Option","dirname","fileURLToPath","resolve","Command"]}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "webscraping-ai-cli",
|
|
3
|
-
"version": "1.0
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "Official command-line client for the WebScraping.AI API. Includes a Claude Code skill so AI agents can use it too.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -53,7 +53,7 @@
|
|
|
53
53
|
"homepage": "https://webscraping.ai",
|
|
54
54
|
"dependencies": {
|
|
55
55
|
"commander": "^12.1.0",
|
|
56
|
-
"webscraping-ai": "^4.
|
|
56
|
+
"webscraping-ai": "^4.1.0"
|
|
57
57
|
},
|
|
58
58
|
"overrides": {
|
|
59
59
|
"esbuild": "^0.28.1"
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "webscraping-ai",
|
|
3
|
-
"description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — via the WebScraping.AI CLI.",
|
|
4
|
-
"version": "1.
|
|
3
|
+
"description": "Fetch HTML, text, selected elements, or AI-extracted data from any URL — including JS-rendered pages — and parsed Google search results, via the WebScraping.AI CLI.",
|
|
4
|
+
"version": "1.1.0",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "WebScraping.AI",
|
|
7
7
|
"url": "https://webscraping.ai"
|
|
@@ -9,5 +9,15 @@
|
|
|
9
9
|
"homepage": "https://github.com/webscraping-ai/webscraping-ai-cli#readme",
|
|
10
10
|
"repository": "https://github.com/webscraping-ai/webscraping-ai-cli",
|
|
11
11
|
"license": "MIT",
|
|
12
|
-
"keywords": [
|
|
12
|
+
"keywords": [
|
|
13
|
+
"web-scraping",
|
|
14
|
+
"html",
|
|
15
|
+
"text-extraction",
|
|
16
|
+
"ai-extraction",
|
|
17
|
+
"scraping",
|
|
18
|
+
"search",
|
|
19
|
+
"serp",
|
|
20
|
+
"google-search",
|
|
21
|
+
"cli"
|
|
22
|
+
]
|
|
13
23
|
}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: webscraping-ai
|
|
3
|
-
description: Use the WebScraping.AI CLI to fetch HTML, plain text, or AI-extracted data from any URL. Reach for this when you need real page content (including JS-rendered sites) instead of guessing or relying on stale knowledge.
|
|
3
|
+
description: Use the WebScraping.AI CLI to fetch HTML, plain text, or AI-extracted data from any URL, or to get parsed Google search results for a query. Reach for this when you need real page content (including JS-rendered sites) or current search results instead of guessing or relying on stale knowledge.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# WebScraping.AI skill
|
|
@@ -25,6 +25,7 @@ Reach for `webscraping-ai` when you need to:
|
|
|
25
25
|
- **Answer a question** about a specific page without scraping the whole thing first.
|
|
26
26
|
- Render **JavaScript-heavy** pages (SPAs, lazy-loaded content) that a plain `curl` won't see.
|
|
27
27
|
- Bypass **anti-bot protection** that's blocking `curl` / `fetch`.
|
|
28
|
+
- Get **current Google search results** for a query (to find candidate URLs, check rankings, or see what's out there) as clean JSON.
|
|
28
29
|
|
|
29
30
|
Don't use it for:
|
|
30
31
|
|
|
@@ -34,21 +35,22 @@ Don't use it for:
|
|
|
34
35
|
|
|
35
36
|
## Quick reference
|
|
36
37
|
|
|
37
|
-
The CLI exposes
|
|
38
|
+
The CLI exposes 8 subcommands, one per API endpoint:
|
|
38
39
|
|
|
39
|
-
| Command
|
|
40
|
-
|
|
|
41
|
-
| `html`
|
|
42
|
-
| `text`
|
|
43
|
-
| `selected`
|
|
44
|
-
| `selected-multiple
|
|
45
|
-
| `ask`
|
|
46
|
-
| `extract`
|
|
47
|
-
| `
|
|
40
|
+
| Command | What it does |
|
|
41
|
+
| ------------------- | ------------------------------------------------- |
|
|
42
|
+
| `html` | Fetch the full rendered HTML of a page |
|
|
43
|
+
| `text` | Fetch only the visible text (plain / JSON / XML) |
|
|
44
|
+
| `selected` | Fetch HTML of one CSS-selected area |
|
|
45
|
+
| `selected-multiple` | Fetch HTML of multiple CSS-selected areas at once |
|
|
46
|
+
| `ask` | Ask a natural-language question about a page (AI) |
|
|
47
|
+
| `extract` | Extract structured fields with descriptions (AI) |
|
|
48
|
+
| `serp` / `search` | Parsed Google search results for a query (JSON) |
|
|
49
|
+
| `account` | Show remaining API credits / quota |
|
|
48
50
|
|
|
49
51
|
Authentication: set `WEBSCRAPING_AI_API_KEY`, pass `--api-key`, or run `webscraping-ai auth set <key>` once.
|
|
50
52
|
|
|
51
|
-
Every command supports common flags: `--js/--no-js`, `--proxy {datacenter|residential|stealth}`, `--country <cc>`, `--device {desktop|mobile|tablet}`, `--timeout <ms>`, `--js-timeout <ms>`, `--wait-for <selector>`, `--headers '{"Cookie":"..."}'`, `--output <file>`.
|
|
53
|
+
Every page command supports common flags: `--js/--no-js`, `--proxy {datacenter|residential|stealth}`, `--country <cc>`, `--device {desktop|mobile|tablet}`, `--timeout <ms>`, `--js-timeout <ms>`, `--wait-for <selector>`, `--headers '{"Cookie":"..."}'`, `--output <file>`. `serp` is the exception: it takes a query, not a URL, and only `--engine`, `--gl <cc>`, `--hl <lang>`, `--page <n>`, `--output`.
|
|
52
54
|
|
|
53
55
|
## How to pick the right command
|
|
54
56
|
|
|
@@ -64,6 +66,8 @@ Every command supports common flags: `--js/--no-js`, `--proxy {datacenter|reside
|
|
|
64
66
|
|
|
65
67
|
**Need a JSON object of typed fields**? → `extract --fields '{"title":"Main product title","price":"Current price including currency"}'`. Returns `{ result: { title: "...", price: "..." } }`.
|
|
66
68
|
|
|
69
|
+
**Starting from a query, not a URL** (find pages about X, check who ranks for a term, see what Google shows in a given country)? → `serp "<query>"`. Returns JSON with `organic_results` (`position`, `title`, `link`, `domain`, `snippet`), `related_searches`, and `pagination`. Use `--gl`/`--hl` for country/language and `--page 2` for the next 10 results. Don't `html` a google.com search URL and parse it yourself — `serp` is cheaper, more reliable, and already parsed. Then feed the `link`s you care about into `text`/`ask`/`extract`.
|
|
70
|
+
|
|
67
71
|
**Need to check available credits**? → `account`.
|
|
68
72
|
|
|
69
73
|
## Example invocations
|
|
@@ -89,6 +93,13 @@ webscraping-ai ask https://shop.example.com/item -q "Is this product in stock?"
|
|
|
89
93
|
webscraping-ai extract https://shop.example.com/item \
|
|
90
94
|
--fields '{"title":"Product title","price":"Current price with currency","rating":"Average star rating as a number"}'
|
|
91
95
|
|
|
96
|
+
# Google results for a query (German results from Germany, page 2)
|
|
97
|
+
webscraping-ai serp coffee machines --gl de --hl de --page 2
|
|
98
|
+
|
|
99
|
+
# Search, then read the top result
|
|
100
|
+
webscraping-ai serp "rust async runtime comparison" --no-pretty | jq -r '.organic_results[0].link' \
|
|
101
|
+
| webscraping-ai text -
|
|
102
|
+
|
|
92
103
|
# Pipe a list of URLs through the same command
|
|
93
104
|
cat urls.txt | webscraping-ai text - --output ./articles.ndjson
|
|
94
105
|
|
|
@@ -103,6 +114,7 @@ webscraping-ai html https://geo.example.com \
|
|
|
103
114
|
- **`--js` is on by default.** Pass `--no-js` for static pages — it's significantly faster and cheaper.
|
|
104
115
|
- **`selected-multiple` returns nested arrays** (`Array<Array<string>>`) — known API response shape. The outer wrapper holds all matches concatenated; flatten in your script if needed.
|
|
105
116
|
- **`extract` wraps its output** under a `result` key: the parsed shape is `{ "result": { ... } }`.
|
|
117
|
+
- **`serp` costs a flat 15 credits per search** (failed searches aren't charged) and returns at most 10 organic results per page. Optional fields (`snippet`, `date`, `related_searches`, `pagination.next`, `search_information.total_results`) may be absent.
|
|
106
118
|
- **API key in URL.** The CLI sends the key as a query-string param. Don't log full requests in shared logs.
|
|
107
119
|
- **Exit codes** are stable per error class (auth → 4, payment → 5, rate-limit → 6, server → 7, timeout → 8, connection → 9). Script around them.
|
|
108
120
|
|
|
@@ -110,7 +122,8 @@ webscraping-ai html https://geo.example.com \
|
|
|
110
122
|
|
|
111
123
|
- Text/HTML responses print verbatim. JSON responses pretty-print on a TTY and single-line on a pipe.
|
|
112
124
|
- `--output FILE` writes the result to a file instead of stdout.
|
|
113
|
-
- For batched stdin input, each URL's response is emitted in order, separated by newlines.
|
|
125
|
+
- For batched stdin input, each URL's response is emitted in order, separated by newlines. `-o FILE` collects all of them in FILE. URL lists skip `#` comment lines; `serp -` skips only blank lines, so `#hashtag` queries are searched.
|
|
126
|
+
- `serp --page` must be an integer >= 1 (server rejects > 100 with a 400); anything else exits with code 2 before any request.
|
|
114
127
|
|
|
115
128
|
## See also
|
|
116
129
|
|