@opencraw/cli 0.1.9 → 0.1.11

package/README.md

@@ -1,13 +1,17 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/russoedu/open.craw/main/docs/assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
 # @opencraw/cli
 
-Command-line tools for [`@opencraw/core`](../core): probe a page for scrapable data, validate recipes,
+Command-line tools for [`@opencraw/core`](https://github.com/russoedu/open.craw/blob/main/packages/core): probe a page for scrapable data, validate recipes,
 and run a crawl from the terminal.
 
 ## Install
 
 ```sh
 npm install -g @opencraw/cli
-# or, inside this workspace: node packages/cli/bin/opencraw.mjs <command>
+# or, inside this workspace, after `npm run cli:build`: node packages/cli/bin/opencraw.mjs <command>
 ```
 
 ## Commands
@@ -16,6 +20,7 @@ npm install -g @opencraw/cli
 opencraw validate <recipe files or directories...>
 opencraw run <recipe files or directories...> [options]
 opencraw probe <url> [options]
+opencraw diff <previous.jsonl> <current.jsonl> [--key a,b] [--ignore a,b] [--changes <file>]
 ```
 
 ### `validate`
@@ -43,11 +48,11 @@ input recipes) and reports what happened.
 | `--dry-run` | One record per input recipe, printed with the scope it was mapped from — for checking a recipe under construction without a full run. |
 | `--trace` | Print the crawl trace to stderr. |
 | `--headed` | Show the browser instead of running headless. |
-| `--parallel <n>` | Run this many input recipes at once (default 1). Iterations inside a recipe follow its `limits.concurrency`. See [§3.9 of the authoring guide](../../docs/recipes/authoring.md#39-concurrency). |
-| `--retries <n>` | Tries per request that fails in passing (a dropped connection, a timeout, a 429, a 5xx), for recipes whose `limits.retry` says nothing. Default 3; `1` turns retrying off. See [§7 of the authoring guide](../../docs/recipes/authoring.md#7-policies-what-happens-when-something-is-missing-or-fails). |
-| `--profiles <dir>` | Where the persistent browser profiles of `session.browserProfile` live (or `OPENCRAW_PROFILES`; default `.opencraw/profiles`). See [access.md](../../docs/recipes/access.md#persistent-browser-profiles). |
-| `--host-delay <ms>`, `--host-concurrency <n>` | Per-site politeness across every recipe: at least `ms` between two requests to one site, at most `n` in flight. They override the access config's `throttle` defaults. See [access.md](../../docs/recipes/access.md#throttling-per-site). |
-| `--plugins <file>` (or `--hooks`) | A plugins module: named exports `hooks` (the recipes' hook steps and transforms), `accessPlugins` (for `{ kind: "plugin" }` access profiles) and `captchaSolvers`. A module whose default export is `{ name: function }` is read as hooks alone. `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` when not given; `probe` reads it too. It runs as your code; load only files you trust. See [§6 of the authoring guide](../../docs/recipes/authoring.md#6-hooks). |
+| `--parallel <n>` | Run this many input recipes at once (default 1). Iterations inside a recipe follow its `limits.concurrency`. See [§3.9 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#39-concurrency). |
+| `--retries <n>` | Tries per request that fails in passing (a dropped connection, a timeout, a 429, a 5xx), for recipes whose `limits.retry` says nothing. Default 3; `1` turns retrying off. See [§7 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#7-policies-what-happens-when-something-is-missing-or-fails). |
+| `--profiles <dir>` | Where the persistent browser profiles of `session.browserProfile` live (or `OPENCRAW_PROFILES`; default `.opencraw/profiles`). See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md#persistent-browser-profiles). |
+| `--host-delay <ms>`, `--host-concurrency <n>` | Per-site politeness across every recipe: at least `ms` between two requests to one site, at most `n` in flight. They override the access config's `throttle` defaults. See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md#throttling-per-site). |
+| `--plugins <file>` (or `--hooks`) | A plugins module: named exports `hooks` (the recipes' hook steps and transforms), `accessPlugins` (for `{ kind: "plugin" }` access profiles) and `captchaSolvers`. A module whose default export is `{ name: function }` is read as hooks alone. `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` when not given; `probe` reads it too. It runs as your code; load only files you trust. See [§6 of the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md#6-hooks). |
 
 ```sh
 opencraw run recipes/ --out out/products.jsonl --trace
@@ -113,7 +118,7 @@ opencraw probe ./decks/incentivi.pptx
 ```
 
 Use it before writing an input recipe, to find the shape a site's data actually takes (§9 of
-[the authoring guide](../../docs/recipes/authoring.md)).
+[the authoring guide](https://github.com/russoedu/open.craw/blob/main/docs/recipes/authoring.md)).
 
 ### Options common to `run` and `probe`
 
@@ -121,7 +126,7 @@ Use it before writing an input recipe, to find the shape a site's data actually
 |---|---|
 | `--browser-path <path>` | A browser binary other than the one Playwright installed (or `OPENCRAW_CHROMIUM`). |
 | `--insecure-tls` | Accept an intercepting proxy's certificate (or `OPENCRAW_INSECURE_TLS=1`). |
-| `--access <file>` | An access config: proxy profiles, with credentials as `{{env.NAME}}` (or `OPENCRAW_ACCESS`). See [access.md](../../docs/recipes/access.md). |
+| `--access <file>` | An access config: proxy profiles, with credentials as `{{env.NAME}}` (or `OPENCRAW_ACCESS`). See [access.md](https://github.com/russoedu/open.craw/blob/main/docs/recipes/access.md). |
 | `--access-profile <name>` | The profile used by recipes that name none, overriding the config's `default`. `probe` uses it for its fetch. |
 | `--user-agent <ua>` | The user agent to send. |
 

package/dist/index.esm.js

@@ -558,6 +558,41 @@ function tryParse(text) {
   }
 }
 
+/**
+ * Whether a cell holds a figure (an amount, a count, a date written in
+ * digits) rather than words: it starts with a number, or its digits are at
+ * least as many as its letters. `2,820`, `$560`, `2659 $560`, `(£0.58m)`,
+ * `-3 days` and `2026` are figures; `FY26 Lodging Rate`,
+ * `FM Classes A, B1 & C3` and `(APRIL 1, 2020)` are words that happen to
+ * contain digits.
+ *
+ * @param cell - The cell's text.
+ * @returns Whether it is a figure.
+ */
+function isFigure(cell) {
+  if (/^[\s(+\-–−<>=~$€£¥]*\d/u.test(cell)) return true;
+  const digits = cell.match(/\d/g)?.length ?? 0;
+  return digits > 0 && digits >= (cell.match(/\p{L}/gu)?.length ?? 0);
+}
+/**
+ * A row that reads like a table's header: at least two cells, none a figure.
+ *
+ * @param cells - The row's non-empty cells.
+ * @returns Whether it could be a header.
+ */
+function isHeaderRow(cells) {
+  return cells.length >= 2 && cells.every(cell => !isFigure(cell));
+}
+/**
+ * A row that reads like a table's data: at least two cells, one a figure.
+ *
+ * @param cells - The row's non-empty cells.
+ * @returns Whether it looks like data.
+ */
+function isDataRow(cells) {
+  return cells.length >= 2 && cells.some(cell => isFigure(cell));
+}
+
 /** Rows shown per sheet in a probe, at most. */
 const ROW_LIMIT$1 = 20;
 /** How far below a header row a probe looks for the first data row. */
@@ -596,23 +631,34 @@ function describeWorkbook(document) {
 function headersOf(sheet) {
   const rows = filledRows(sheet);
   const merged = mergedRows(sheet);
-  return rows.flatMap(({
+  const headers = [];
+  // The table a row belongs to: the header it follows, until an empty row ends the block.
+  let table;
+  for (const [position, {
     index,
     cells
-  }, position) => {
-    if (cells.length < 2 || cells.some(cell => /\d/.test(cell))) return [];
-    if (rows.slice(position + 1, position + 1 + LOOKAHEAD).every(row => !(row.cells.length >= 2 && row.cells.some(cell => /\d/.test(cell))))) return [];
+  }] of rows.entries()) {
+    const previous = rows[position - 1]?.index;
+    if (previous === undefined || index > previous + 1) table = undefined;
+    const secondHeaderRow = table?.merged === true && index === table.header + 1;
+    if (table !== undefined && !secondHeaderRow) continue;
+    if (!isHeaderRow(cells) || rows.slice(position + 1, position + 1 + LOOKAHEAD).every(row => !isDataRow(row.cells))) continue;
     const header = {
       sheet: sheet.name,
       row: index + 1,
       text: cells.join(' | '),
       selector: `^${escape$1(cells[0])}`
     };
-    return [merged.has(index) ? {
+    headers.push(merged.has(index) ? {
       ...header,
       hint: 'merged header cells: try "headerRows": 2'
-    } : header];
-  });
+    } : header);
+    table ??= {
+      header: index,
+      merged: merged.has(index)
+    };
+  }
+  return headers;
 }
 /** The non-empty rows, as their non-empty cells, whitespace collapsed. */
 function filledRows(sheet) {
@@ -826,7 +872,7 @@ function describePdf(document) {
   const headers = rows.filter(({
     row,
     next
-  }) => row.cells.length >= 2 && row.cells.every(cell => !/\d/.test(cell.text)) && next.some(candidate => dataRow(candidate)));
+  }) => isHeaderRow(row.cells.map(cell => cell.text)) && next.some(candidate => isDataRow(candidate.cells.map(cell => cell.text))));
   return {
     pages: document.pages.length,
     rows: rows.slice(0, ROW_LIMIT).map(({
@@ -846,10 +892,6 @@ function describePdf(document) {
     }))
   };
 }
-/** A row of at least two cells, one with a digit: data, not a heading. */
-function dataRow(row) {
-  return row.cells.length >= 2 && row.cells.some(cell => /\d/.test(cell.text));
-}
 function escape(text) {
   return text.replaceAll(/[$()*+.?[\\\]^{|}]/g, String.raw`\$&`);
 }

package/dist/src/probe/table-header.policy.d.ts

@@ -0,0 +1,27 @@
+/**
+ * Whether a cell holds a figure (an amount, a count, a date written in
+ * digits) rather than words: it starts with a number, or its digits are at
+ * least as many as its letters. `2,820`, `$560`, `2659 $560`, `(£0.58m)`,
+ * `-3 days` and `2026` are figures; `FY26 Lodging Rate`,
+ * `FM Classes A, B1 & C3` and `(APRIL 1, 2020)` are words that happen to
+ * contain digits.
+ *
+ * @param cell - The cell's text.
+ * @returns Whether it is a figure.
+ */
+export declare function isFigure(cell: string): boolean;
+/**
+ * A row that reads like a table's header: at least two cells, none a figure.
+ *
+ * @param cells - The row's non-empty cells.
+ * @returns Whether it could be a header.
+ */
+export declare function isHeaderRow(cells: readonly string[]): boolean;
+/**
+ * A row that reads like a table's data: at least two cells, one a figure.
+ *
+ * @param cells - The row's non-empty cells.
+ * @returns Whether it looks like data.
+ */
+export declare function isDataRow(cells: readonly string[]): boolean;
+//# sourceMappingURL=table-header.policy.d.ts.map

package/dist/src/probe/workbook-findings.mapper.d.ts

@@ -19,9 +19,10 @@ export interface WorkbookFindings {
         text: string;
     }[];
     /**
-     * Rows of text cells (no digit) with a data row (a digit) just below: likely
-     * table headers, with the `sheet` and `selector` a `table` extract needs, and
-     * a hint when the header has merged cells (a header over two rows).
+     * Rows of words (no cell a figure) with a data row (a figure) just below:
+     * likely table headers, with the `sheet` and `selector` a `table` extract
+     * needs, and a hint when the header has merged cells (a header over two
+     * rows). The rows right under a header are its table's body, not headers.
      */
     headers: {
         sheet: string;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@opencraw/cli",
-  "version": "0.1.9",
+  "version": "0.1.11",
   "type": "module",
   "main": "./dist/index.esm.js",
   "module": "./dist/index.esm.js",
@@ -24,7 +24,7 @@
     "!**/*.js.map"
   ],
   "dependencies": {
-    "@opencraw/core": "^0.1.9",
+    "@opencraw/core": "^0.1.11",
     "playwright": "^1.63.0"
   },
   "publishConfig": {