@opencraw/cli 0.1.9 → 0.1.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +14 -9
  2. package/package.json +2 -2
package/README.md CHANGED
@@ -1,13 +1,17 @@
1
+ <p align="center">
2
+ <img src="https://raw.githubusercontent.com/russoedu/open.craw/main/docs/assets/opencraw-logo.svg" alt="OpenCraw" width="360">
3
+ </p>
4
+
1
5
  # @opencraw/cli
2
6
 
3
- Command-line tools for [`@opencraw/core`](../core): probe a page for scrapable data, validate recipes,
7
+ Command-line tools for [`@opencraw/core`](https://github.com/russoedu/open.craw/blob/main/packages/core): probe a page for scrapable data, validate recipes,
4
8
  and run a crawl from the terminal.
5
9
 
6
10
  ## Install
7
11
 
8
12
  ```sh
9
13
  npm install -g @opencraw/cli
10
- # or, inside this workspace: node packages/cli/bin/opencraw.mjs <command>
14
+ # or, inside this workspace, after `npm run cli:build`: node packages/cli/bin/opencraw.mjs <command>
11
15
  ```
12
16
 
13
17
  ## Commands
@@ -16,6 +20,7 @@ npm install -g @opencraw/cli
16
20
  opencraw validate <recipe files or directories...>
17
21
  opencraw run <recipe files or directories...> [options]
18
22
  opencraw probe <url> [options]
23
+ opencraw diff <previous.jsonl> <current.jsonl> [--key a,b] [--ignore a,b] [--changes <file>]
19
24
  ```
20
25
 
21
26
  ### `validate`
@@ -43,11 +48,11 @@ input recipes) and reports what happened.
43
48
  | `--dry-run` | One record per input recipe, printed with the scope it was mapped from — for checking a recipe under construction without a full run. |
44
49
  | `--trace` | Print the crawl trace to stderr. |
45
50
  | `--headed` | Show the browser instead of running headless. |
46
- | `--parallel <n>` | Run this many input recipes at once (default 1). Iterations inside a recipe follow its `limits.concurrency`. See [§3.9 of the authoring guide](../../docs/recipes/authoring.md#39-concurrency). |
47
- | `--retries <n>` | Tries per request that fails in passing (a dropped connection, a timeout, a 429, a 5xx), for recipes whose `limits.retry` says nothing. Default 3; `1` turns retrying off. See [§7 of the authoring guide](../../docs/recipes/authoring.md#7-policies-what-happens-when-something-is-missing-or-fails). |
48
- | `--profiles <dir>` | Where the persistent browser profiles of `session.browserProfile` live (or `OPENCRAW_PROFILES`; default `.opencraw/profiles`). See [access.md](../../docs/recipes/access.md#persistent-browser-profiles). |
49
- | `--host-delay <ms>`, `--host-concurrency <n>` | Per-site politeness across every recipe: at least `ms` between two requests to one site, at most `n` in flight. They override the access config's `throttle` defaults. See [access.md](../../docs/recipes/access.md#throttling-per-site). |
50
- | `--plugins <file>` (or `--hooks`) | A plugins module: named exports `hooks` (the recipes' hook steps and transforms), `accessPlugins` (for `{ kind: "plugin" }` access profiles) and `captchaSolvers`. A module whose default export is `{ name: function }` is read as hooks alone. `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` when not given; `probe` reads it too. It runs as your code; load only files you trust. See [§6 of the authoring guide](../../docs/recipes/authoring.md#6-hooks). |
51
+ | `--parallel <n>` | Run this many input recipes at once (default 1). Iterations inside a recipe follow its `limits.concurrency`. See [§3.9 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#39-concurrency). |
52
+ | `--retries <n>` | Tries per request that fails in passing (a dropped connection, a timeout, a 429, a 5xx), for recipes whose `limits.retry` says nothing. Default 3; `1` turns retrying off. See [§7 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#7-policies-what-happens-when-something-is-missing-or-fails). |
53
+ | `--profiles <dir>` | Where the persistent browser profiles of `session.browserProfile` live (or `OPENCRAW_PROFILES`; default `.opencraw/profiles`). See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md#persistent-browser-profiles). |
54
+ | `--host-delay <ms>`, `--host-concurrency <n>` | Per-site politeness across every recipe: at least `ms` between two requests to one site, at most `n` in flight. They override the access config's `throttle` defaults. See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md#throttling-per-site). |
55
+ | `--plugins <file>` (or `--hooks`) | A plugins module: named exports `hooks` (the recipes' hook steps and transforms), `accessPlugins` (for `{ kind: "plugin" }` access profiles) and `captchaSolvers`. A module whose default export is `{ name: function }` is read as hooks alone. `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` when not given; `probe` reads it too. It runs as your code; load only files you trust. See [§6 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#6-hooks). |
51
56
 
52
57
  ```sh
53
58
  opencraw run recipes/ --out out/products.jsonl --trace
@@ -113,7 +118,7 @@ opencraw probe ./decks/incentivi.pptx
113
118
  ```
114
119
 
115
120
  Use it before writing an input recipe, to find the shape a site's data actually takes (§9 of
116
- [the authoring guide](../../docs/recipes/authoring.md)).
121
+ [the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md)).
117
122
 
118
123
  ### Options common to `run` and `probe`
119
124
 
@@ -121,7 +126,7 @@ Use it before writing an input recipe, to find the shape a site's data actually
121
126
  |---|---|
122
127
  | `--browser-path <path>` | A browser binary other than the one Playwright installed (or `OPENCRAW_CHROMIUM`). |
123
128
  | `--insecure-tls` | Accept an intercepting proxy's certificate (or `OPENCRAW_INSECURE_TLS=1`). |
124
- | `--access <file>` | An access config: proxy profiles, with credentials as `{{env.NAME}}` (or `OPENCRAW_ACCESS`). See [access.md](../../docs/recipes/access.md). |
129
+ | `--access <file>` | An access config: proxy profiles, with credentials as `{{env.NAME}}` (or `OPENCRAW_ACCESS`). See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md). |
125
130
  | `--access-profile <name>` | The profile used by recipes that name none, overriding the config's `default`. `probe` uses it for its fetch. |
126
131
  | `--user-agent <ua>` | The user agent to send. |
127
132
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@opencraw/cli",
3
- "version": "0.1.9",
3
+ "version": "0.1.10",
4
4
  "type": "module",
5
5
  "main": "./dist/index.esm.js",
6
6
  "module": "./dist/index.esm.js",
@@ -24,7 +24,7 @@
24
24
  "!**/*.js.map"
25
25
  ],
26
26
  "dependencies": {
27
- "@opencraw/core": "^0.1.9",
27
+ "@opencraw/core": "^0.1.10",
28
28
  "playwright": "^1.63.0"
29
29
  },
30
30
  "publishConfig": {