@opencraw/cli 0.1.43 → 0.1.45
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -41,8 +41,8 @@ input recipes) and reports what happened.
|
|
|
41
41
|
|
|
42
42
|
| Flag | Meaning |
|
|
43
43
|
|---|---|
|
|
44
|
-
| `--out <file>` | Write records to this
|
|
45
|
-
| `--append` | Keep what `--out` holds and add to it; each line carries `_key`. |
|
|
44
|
+
| `--out <file>` | Write records to this file; **its extension is the format** (see below). Without it, records print to stdout as JSON Lines. |
|
|
45
|
+
| `--append` | Keep what `--out` holds and add to it; each line carries `_key`. `.jsonl` only. |
|
|
46
46
|
| `--resume` | Skip records `--out` already has (needs `--append`). |
|
|
47
47
|
| `--only <id>` | Run only this input recipe; repeat for more than one. |
|
|
48
48
|
| `--dry-run` | One record per input recipe, printed with the scope it was mapped from — for checking a recipe under construction without a full run. |
|
|
@@ -54,6 +54,23 @@ input recipes) and reports what happened.
|
|
|
54
54
|
| `--host-delay <ms>`, `--host-concurrency <n>` | Per-site politeness across every recipe: at least `ms` between two requests to one site, at most `n` in flight. They override the access config's `throttle` defaults. See [access.md](https://github.com/Open-Craw/OpenCraw/blob/main/docs/recipes/access.md#throttling-per-site). |
|
|
55
55
|
| `--plugins <file>` (or `--hooks`) | A plugins module: named exports `hooks` (the recipes' hook steps and transforms), `accessPlugins` (for `{ kind: "plugin" }` access profiles) and `captchaSolvers`. A module whose default export is `{ name: function }` is read as hooks alone. `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` when not given; `probe` reads it too. It runs as your code; load only files you trust. See [§6 of the authoring guide](https://github.com/Open-Craw/OpenCraw/blob/main/docs/recipes/authoring.md#6-hooks). |
|
|
56
56
|
|
|
57
|
+
### Output formats
|
|
58
|
+
|
|
59
|
+
The extension of `--out` chooses the format. Any other extension is an error, raised before the crawl starts.
|
|
60
|
+
|
|
61
|
+
| Extension | What is written |
|
|
62
|
+
|---|---|
|
|
63
|
+
| `.jsonl` | One JSON object per line (the record's fields and a `_source`), written as records arrive. The only format `--append`, `--resume` and `--diff` work with. |
|
|
64
|
+
| `.json` | A JSON array of the same objects. |
|
|
65
|
+
| `.csv` | A header row of the output recipe's fields, then one row per record (RFC 4180, UTF-8). |
|
|
66
|
+
| `.xlsx` | An Excel workbook: one sheet named after the output, a violet header row with the filter on and frozen, numbers and booleans kept as numbers and booleans. |
|
|
67
|
+
| `.yml` / `.yaml` | A YAML list of the same objects. |
|
|
68
|
+
| `.html` | One self-contained page that looks like Studio: the OpenCraw logo, the output's name and description, the record count, and the records in a table. Light or dark by the reader's setting; no script. |
|
|
69
|
+
| `.pdf` | That page, printed (A4, landscape, header repeated on each page). Needs a Chromium, as a `web` recipe does: Playwright's, or `OPENCRAW_CHROMIUM` / `--browser-path`. |
|
|
70
|
+
|
|
71
|
+
The tabular formats (`.csv`, `.xlsx`, `.html`, `.pdf`) carry the output recipe's fields as columns, in the order the recipe declares them; a list or an
|
|
72
|
+
object becomes its JSON in one cell. Everything but `.jsonl` is written when the run closes.
|
|
73
|
+
|
|
57
74
|
```sh
|
|
58
75
|
opencraw run recipes/ --out out/products.jsonl --trace
|
|
59
76
|
opencraw run recipes/movie.output.json recipes/tmdb.input.json --dry-run
|
package/dist/index.esm.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { parseArgs } from 'node:util';
|
|
2
2
|
import { writeFile } from 'node:fs/promises';
|
|
3
|
-
import { readableKey, readRecordsFile, diffRecords, loadAccessConfig, HostAllowlist, AccessBroker, HttpClient, parseXml, BrowserClient, readRecipeSource, parseOutputRecipe, parseInputRecipe, bindRecipeSet, memorySink,
|
|
3
|
+
import { readableKey, readRecordsFile, diffRecords, loadAccessConfig, HostAllowlist, AccessBroker, HttpClient, parseXml, BrowserClient, readRecipeSource, parseOutputRecipe, parseInputRecipe, bindRecipeSet, memorySink, fileSink, createCrawler, RecipeSet, variantLabel, diffOptionsFor, RecipeValidationError, RecipeBindingError, traceLine } from '@opencraw/core';
|
|
4
4
|
import { resolve } from 'node:path';
|
|
5
5
|
import { pathToFileURL } from 'node:url';
|
|
6
6
|
import { describePdf, findData, describeWorkbook, describeDeck, describeJson, describeXml, describeHtml } from '@opencraw/probe';
|
|
@@ -11,7 +11,7 @@ const USAGE = `opencraw <command> [options]
|
|
|
11
11
|
|
|
12
12
|
Commands
|
|
13
13
|
validate <recipe files or directories...> Parse and bind the recipes; list every problem with its path.
|
|
14
|
-
run <recipe files or directories...> Crawl. Records go to --out
|
|
14
|
+
run <recipe files or directories...> Crawl. Records go to --out (its extension picks the format), or to stdout as JSON Lines.
|
|
15
15
|
probe <url> Fetch a page and report where its data lives.
|
|
16
16
|
diff <previous.jsonl> <current.jsonl> Compare two runs' records by key: added, removed, changed.
|
|
17
17
|
studio [folder] [--open <path>] Open a recipe folder in the browser: run samples, read
|
|
@@ -20,8 +20,10 @@ Commands
|
|
|
20
20
|
document straight away.
|
|
21
21
|
|
|
22
22
|
Options for run
|
|
23
|
-
--out <file> Write records to this
|
|
24
|
-
|
|
23
|
+
--out <file> Write records to this file; the extension is the format: .jsonl, .json, .csv, .xlsx, .yml/.yaml,
|
|
24
|
+
.html (a page that looks like Studio) or .pdf (that page, printed; needs Chromium). Any other
|
|
25
|
+
extension is an error. Default: stdout as JSON Lines.
|
|
26
|
+
--append Keep what --out holds and add to it (each line carries _key); .jsonl only.
|
|
25
27
|
--resume Skip records --out already has (needs --append).
|
|
26
28
|
--only <id> Run only this input recipe (repeatable).
|
|
27
29
|
--dry-run One record per input, printed with the scope it was mapped from.
|
|
@@ -793,6 +795,19 @@ async function runRecipes(command, terminal) {
|
|
|
793
795
|
terminal.err(command.only.length > 0 ? `no input recipe matches --only ${command.only.join(', ')}` : 'no input recipes found');
|
|
794
796
|
return 1;
|
|
795
797
|
}
|
|
798
|
+
// The extension of --out names the format: one that names none is refused here, before the crawl, not after it.
|
|
799
|
+
let sink;
|
|
800
|
+
try {
|
|
801
|
+
sink = command.dryRun || command.out === undefined ? memorySink() : fileSink(command.out, {
|
|
802
|
+
append: command.append,
|
|
803
|
+
browser: {
|
|
804
|
+
executablePath: command.options.browserPath
|
|
805
|
+
}
|
|
806
|
+
});
|
|
807
|
+
} catch (error) {
|
|
808
|
+
terminal.err(error instanceof Error ? error.message : String(error));
|
|
809
|
+
return 1;
|
|
810
|
+
}
|
|
796
811
|
let access;
|
|
797
812
|
let plugins;
|
|
798
813
|
let previous;
|
|
@@ -806,9 +821,6 @@ async function runRecipes(command, terminal) {
|
|
|
806
821
|
terminal.err(error instanceof Error ? error.message : String(error));
|
|
807
822
|
return 1;
|
|
808
823
|
}
|
|
809
|
-
const sink = command.dryRun || command.out === undefined ? memorySink() : jsonLinesSink(command.out, {
|
|
810
|
-
append: command.append
|
|
811
|
-
});
|
|
812
824
|
const crawler = createCrawler({
|
|
813
825
|
sink,
|
|
814
826
|
access,
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import type { Command } from './command.contract.js';
|
|
2
|
-
export declare const USAGE = "opencraw <command> [options]\n\nCommands\n validate <recipe files or directories...> Parse and bind the recipes; list every problem with its path.\n run <recipe files or directories...> Crawl. Records go to --out
|
|
2
|
+
export declare const USAGE = "opencraw <command> [options]\n\nCommands\n validate <recipe files or directories...> Parse and bind the recipes; list every problem with its path.\n run <recipe files or directories...> Crawl. Records go to --out (its extension picks the format), or to stdout as JSON Lines.\n probe <url> Fetch a page and report where its data lives.\n diff <previous.jsonl> <current.jsonl> Compare two runs' records by key: added, removed, changed.\n studio [folder] [--open <path>] Open a recipe folder in the browser: run samples, read\n the trace next to the page (needs @opencraw/studio).\n --open prints a link that opens a folder, recipe or\n document straight away.\n\nOptions for run\n --out <file> Write records to this file; the extension is the format: .jsonl, .json, .csv, .xlsx, .yml/.yaml,\n .html (a page that looks like Studio) or .pdf (that page, printed; needs Chromium). Any other\n extension is an error. Default: stdout as JSON Lines.\n --append Keep what --out holds and add to it (each line carries _key); .jsonl only.\n --resume Skip records --out already has (needs --append).\n --only <id> Run only this input recipe (repeatable).\n --dry-run One record per input, printed with the scope it was mapped from.\n --trace Print the crawl trace to stderr.\n --headed Show the browser.\n --parallel <n> Run this many input recipes at once (default 1). Iterations inside a recipe\n follow its limits.concurrency.\n --retries <n> Tries per request that fails in passing (connection, timeout, 429, 5xx), for\n recipes whose limits.retry says nothing. Default 3; 1 turns retrying off.\n --profiles <dir> Where recipes' session.browserProfile profiles live (or OPENCRAW_PROFILES;\n default .opencraw/profiles).\n --diff <file> Compare this run's records with a previous run's JSON Lines file (it may be\n --out itself: it is read before the crawl). Keys come from the output recipe.\n --host-delay <ms> At least this long between two requests to one site, across every recipe.\n --host-concurrency <n>\n At most this many requests to one site in flight, across every recipe.\n Both override the access config's \"throttle\" defaults.\n\nOptions for diff\n --key <a,b> The fields that identify a record (default: each line's _key).\n --ignore <a,b> Fields left out of the comparison (a date the crawl stamps, say).\n --changes <file> Also write every change as a JSON line (run --diff takes it too).\n\nOptions for probe\n --browser Also render the page in a browser and list the JSON it fetches.\n\nOptions for both\n --browser-path <p> A browser binary other than the one Playwright installed (or OPENCRAW_CHROMIUM).\n --insecure-tls Accept an intercepting proxy's certificate (or OPENCRAW_INSECURE_TLS=1).\n --user-agent <ua> The user agent to send.\n --access <file> An access config: proxy profiles, credentials as {{env.NAME}} (or OPENCRAW_ACCESS).\n --access-profile <name>\n The access profile to use when a recipe names none.\n --plugins <file> A JavaScript module exporting hooks (the recipes' hook steps and transforms),\n accessPlugins (for { kind: \"plugin\" } access profiles) and captchaSolvers\n (for session.captcha and captcha steps); --hooks, or OPENCRAW_PLUGINS /\n OPENCRAW_HOOKS. It runs as your code: trust it.\n --help, --version";
|
|
3
3
|
/**
|
|
4
4
|
* Turns argv into a command.
|
|
5
5
|
*
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@opencraw/cli",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.45",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"main": "./dist/index.esm.js",
|
|
6
6
|
"module": "./dist/index.esm.js",
|
|
@@ -24,12 +24,12 @@
|
|
|
24
24
|
"!**/*.js.map"
|
|
25
25
|
],
|
|
26
26
|
"dependencies": {
|
|
27
|
-
"@opencraw/core": "^0.1.
|
|
28
|
-
"@opencraw/probe": "^0.0.
|
|
27
|
+
"@opencraw/core": "^0.1.41",
|
|
28
|
+
"@opencraw/probe": "^0.0.24",
|
|
29
29
|
"playwright": "^1.63.0"
|
|
30
30
|
},
|
|
31
31
|
"peerDependencies": {
|
|
32
|
-
"@opencraw/studio": "^0.0.
|
|
32
|
+
"@opencraw/studio": "^0.0.34"
|
|
33
33
|
},
|
|
34
34
|
"peerDependenciesMeta": {
|
|
35
35
|
"@opencraw/studio": {
|