polite-crawler 0.1.3 → 0.1.4

package/README.md

@@ -103,6 +103,22 @@ npx polite-crawler watch list
 - `watch run` takes `--budget` (default `10m`) and `--max-pages` (default 100). It prints each check, then a summary. A scheduler of the operator starts it.
 - `daemon` checks the watch list on a schedule, until you stop it. With `--mcp-http`, it also serves the MCP tools over local HTTP. [The daemon](#the-daemon) tells how.
 
+To read the hours of a venue or a product page, use `venue` and `product`. They use the recipes of the package:
+
+```sh
+npx polite-crawler venue https://forno-esempio.example/ --name "Forno Esempio" --pages 2
+npx polite-crawler venue https://forno-esempio.example/ --compare kept-week.json
+npx polite-crawler product "https://bottega-campione.example/products/olio-esempio?variant=222"
+```
+
+- `venue` prints the week of the site and each reading with its URL, its rank and its quote. It also prints the days where the readings disagree, the menus, the phone, the pages and the notes.
+- `venue` takes `--name`, `--require-name`, `--address`, `--lat` and `--lon`, `--osm <value>` and `--pages <0-8>`. [Read the hours of a venue](#read-the-hours-of-a-venue) tells what each one does.
+- Write a number below zero after `=`, for example `--lon=-20.5`. Else the command line reads it as an option.
+- `venue --compare <file>` compares the week of the site with a week that you keep, as JSON. It prints `same`, `changed`, `new` or `not read` for each day. The file is a week, for example `{"mon": [["07:00", "15:00"]], "sun": []}`, or the `from_site` of `venue --json`.
+- `product` prints the name, the brand, the SKU and the GTIN, and the price as money with the range of the offers. It also prints the availability, the variant, the seller, and a wall or a held-back page.
+- The two commands take `--render`, `--max-age`, `--budget`, `--user-provided`, `--archive-fallback` and `--archive-despite-robots`. `--on-behalf-of <agent>` applies the robots.txt rules of an agent too, for example `Claude-User`.
+- `--json` prints the `VenueResult` or the `ProductResult` of the library.
+
 The command gives these exit codes:
 
 | Code | Meaning |
@@ -110,6 +126,8 @@ The command gives these exit codes:
 | `0` | The command did its work. |
 | `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. `discover` gives 1 when it finds no URL, and `crawl` when it gets no page. `read --at` and `verify --at` give 1 for a time before the first capture. `watch run` gives 1 when a check is `refused` or an `error`, or when entries stay due. `watch remove` gives 1 for a URL that is not on the list. With `--archive-fallback`, a page with an archived copy counts as done. `extract` gives 1 when a field is not `verified` or `normalized`. `embed` gives 1 when its budget stops it before each page has vectors. |
 | `2` | The command line is wrong. The command opens no store. |
+| `3` | `venue` and `product` read nothing: no opening hours of the site or no Product data, a page that is held back, or a page that did not come. |
+| `4` | `venue` and `product`: the site refused the crawler (a wall). The causes are a 401, a 402 or a 403, a challenge, a bot check, robots.txt or a stop of the host. The crawler does not try a different identity. |
 
 ## Use it from Claude (MCP)
 
@@ -935,6 +953,8 @@ const then = venueReadingsOf(version!);
 - With `skipSplit`, it does not compare a day with two or more intervals, for a store that keeps one interval for each day.
 - One reading can be an error of the site. Confirm a change with a second reading, as a watch does.
 
+The command line has `polite-crawler venue`, and `venue --compare <file>` does this comparison. [Use it from the command line](#use-it-from-the-command-line) gives the options.
+
 ## Read a product page
 
 `readProduct` reads the schema.org Product data of a product page. Most shop systems write this data on each product page.
@@ -977,6 +997,8 @@ const check = await crawler.extract(product.doc!, {
 });
 ```
 
+The command line has `polite-crawler product`. It exits with 4 at a wall, and with 3 when it reads no Product data.
+
 ## Writing a recipe
 
 A recipe tells the crawler what to fetch and how to read it into documents. The venue and product recipes read pages with the page reader of the package. A recipe of your own reads a source that the page reader does not know, for example an API.

package/build/{cli-daemon--cTmq0Z5.mjs → cli-daemon-ZYRdy6EW.mjs}

@@ -1,4 +1,4 @@
-import { t as describeDaemonStatus } from "./cli.mjs";
+import { n as describeDaemonStatus } from "./cli.mjs";
 //#region src/cli-daemon.ts
 /**
 * This function does the command `daemon`, and gives the exit code. `daemon status` gives 0 when a
@@ -6,7 +6,7 @@ import { t as describeDaemonStatus } from "./cli.mjs";
 * error at start rejects, so the command exits with 1.
 */
 const daemonCommand = async (line, options, io) => {
-	const { changeCommand, readDaemonState, startDaemon } = await import("./daemon-fZTE24P5.mjs").then((n) => n.a);
+	const { changeCommand, readDaemonState, startDaemon } = await import("./daemon-DK6U51_v.mjs").then((n) => n.a);
 	if (line.action === "status") {
 		const state = readDaemonState(options.dir);
 		io.out(io.json ? JSON.stringify(state ?? null, null, 2) : describeDaemonStatus(state, options.dir));
@@ -35,4 +35,4 @@ const daemonCommand = async (line, options, io) => {
 //#endregion
 export { daemonCommand };
 
-//# sourceMappingURL=cli-daemon--cTmq0Z5.mjs.map
+//# sourceMappingURL=cli-daemon-ZYRdy6EW.mjs.map

package/build/{cli-daemon--cTmq0Z5.mjs.map → cli-daemon-ZYRdy6EW.mjs.map}

@@ -1 +1 @@
-{"version":3,"file":"cli-daemon--cTmq0Z5.mjs","names":[],"sources":["../src/cli-daemon.ts"],"sourcesContent":["/*\n * The command `daemon` of the command line: `daemon` runs the daemon until `SIGINT` or `SIGTERM`, and\n * `daemon status` prints the state file. `cli.ts` loads this file only for the command `daemon`.\n */\nimport type { DaemonLine } from \"./cli-line.ts\";\nimport { describeDaemonStatus } from \"./cli-line.ts\";\nimport type { OpenOptions } from \"./open.ts\";\n\n/** These are the parts of the command line that the command `daemon` uses. */\nexport interface DaemonIo {\n  /** The value true lets `user_provided` work when the MCP client cannot ask the user. */\n  readonly allowUserProvided: boolean;\n  /** The value true prints the status as JSON. */\n  readonly json: boolean;\n  /** This function prints a line on the standard output. */\n  readonly out: (text: string) => void;\n}\n\n/**\n * This function does the command `daemon`, and gives the exit code. `daemon status` gives 0 when a\n * daemon operates on the store, and 1 when none operates. `daemon` gives 0 after a clean stop. An\n * error at start rejects, so the command exits with 1.\n */\nexport const daemonCommand = async (\n  line: DaemonLine,\n  options: OpenOptions,\n  io: DaemonIo,\n): Promise<number> => {\n  const { changeCommand, readDaemonState, startDaemon } = await import(\"./daemon.ts\");\n  if (line.action === \"status\") {\n    const state = readDaemonState(options.dir);\n    io.out(\n      io.json ? JSON.stringify(state ?? null, null, 2) : describeDaemonStatus(state, options.dir),\n    );\n    return state?.alive === true ? 0 : 1;\n  }\n  const daemon = await startDaemon({\n    ...options,\n    tick: line.tickMs,\n    jitter: line.jitterMs,\n    budget: line.budgetMs,\n    maxPages: line.maxPages,\n    grace: line.graceMs,\n    events: line.events,\n    onChange: line.onChange === undefined ? undefined : changeCommand(line.onChange),\n    mcp:\n      line.mcp === undefined\n        ? undefined\n        : {\n            port: line.mcp.port,\n            host: line.mcp.host,\n            allowRemote: line.mcp.allowRemote,\n            tokenFile: line.mcp.tokenFile,\n            allowUserProvided: io.allowUserProvided,\n          },\n    signals: true,\n  });\n  await daemon.done;\n  return 0;\n};\n"],"mappings":";;;;;;;AAuBA,MAAa,gBAAgB,OAC3B,MACA,SACA,OACoB;CACpB,MAAM,EAAE,eAAe,iBAAiB,gBAAgB,MAAM,OAAO,wBAAc,CAAA,MAAA,MAAA,EAAA,CAAA;CACnF,IAAI,KAAK,WAAW,UAAU;EAC5B,MAAM,QAAQ,gBAAgB,QAAQ,GAAG;EACzC,GAAG,IACD,GAAG,OAAO,KAAK,UAAU,SAAS,MAAM,MAAM,CAAC,IAAI,qBAAqB,OAAO,QAAQ,GAAG,CAC5F;EACA,OAAO,OAAO,UAAU,OAAO,IAAI;CACrC;CAsBA,OAAM,MArBe,YAAY;EAC/B,GAAG;EACH,MAAM,KAAK;EACX,QAAQ,KAAK;EACb,QAAQ,KAAK;EACb,UAAU,KAAK;EACf,OAAO,KAAK;EACZ,QAAQ,KAAK;EACb,UAAU,KAAK,aAAa,KAAA,IAAY,KAAA,IAAY,cAAc,KAAK,QAAQ;EAC/E,KACE,KAAK,QAAQ,KAAA,IACT,KAAA,IACA;GACE,MAAM,KAAK,IAAI;GACf,MAAM,KAAK,IAAI;GACf,aAAa,KAAK,IAAI;GACtB,WAAW,KAAK,IAAI;GACpB,mBAAmB,GAAG;EACxB;EACN,SAAS;CACX,CAAC,EAAA,CACY;CACb,OAAO;AACT"}
+{"version":3,"file":"cli-daemon-ZYRdy6EW.mjs","names":[],"sources":["../src/cli-daemon.ts"],"sourcesContent":["/*\n * The command `daemon` of the command line: `daemon` runs the daemon until `SIGINT` or `SIGTERM`, and\n * `daemon status` prints the state file. `cli.ts` loads this file only for the command `daemon`.\n */\nimport type { DaemonLine } from \"./cli-line.ts\";\nimport { describeDaemonStatus } from \"./cli-line.ts\";\nimport type { OpenOptions } from \"./open.ts\";\n\n/** These are the parts of the command line that the command `daemon` uses. */\nexport interface DaemonIo {\n  /** The value true lets `user_provided` work when the MCP client cannot ask the user. */\n  readonly allowUserProvided: boolean;\n  /** The value true prints the status as JSON. */\n  readonly json: boolean;\n  /** This function prints a line on the standard output. */\n  readonly out: (text: string) => void;\n}\n\n/**\n * This function does the command `daemon`, and gives the exit code. `daemon status` gives 0 when a\n * daemon operates on the store, and 1 when none operates. `daemon` gives 0 after a clean stop. An\n * error at start rejects, so the command exits with 1.\n */\nexport const daemonCommand = async (\n  line: DaemonLine,\n  options: OpenOptions,\n  io: DaemonIo,\n): Promise<number> => {\n  const { changeCommand, readDaemonState, startDaemon } = await import(\"./daemon.ts\");\n  if (line.action === \"status\") {\n    const state = readDaemonState(options.dir);\n    io.out(\n      io.json ? JSON.stringify(state ?? null, null, 2) : describeDaemonStatus(state, options.dir),\n    );\n    return state?.alive === true ? 0 : 1;\n  }\n  const daemon = await startDaemon({\n    ...options,\n    tick: line.tickMs,\n    jitter: line.jitterMs,\n    budget: line.budgetMs,\n    maxPages: line.maxPages,\n    grace: line.graceMs,\n    events: line.events,\n    onChange: line.onChange === undefined ? undefined : changeCommand(line.onChange),\n    mcp:\n      line.mcp === undefined\n        ? undefined\n        : {\n            port: line.mcp.port,\n            host: line.mcp.host,\n            allowRemote: line.mcp.allowRemote,\n            tokenFile: line.mcp.tokenFile,\n            allowUserProvided: io.allowUserProvided,\n          },\n    signals: true,\n  });\n  await daemon.done;\n  return 0;\n};\n"],"mappings":";;;;;;;AAuBA,MAAa,gBAAgB,OAC3B,MACA,SACA,OACoB;CACpB,MAAM,EAAE,eAAe,iBAAiB,gBAAgB,MAAM,OAAO,wBAAc,CAAA,MAAA,MAAA,EAAA,CAAA;CACnF,IAAI,KAAK,WAAW,UAAU;EAC5B,MAAM,QAAQ,gBAAgB,QAAQ,GAAG;EACzC,GAAG,IACD,GAAG,OAAO,KAAK,UAAU,SAAS,MAAM,MAAM,CAAC,IAAI,qBAAqB,OAAO,QAAQ,GAAG,CAC5F;EACA,OAAO,OAAO,UAAU,OAAO,IAAI;CACrC;CAsBA,OAAM,MArBe,YAAY;EAC/B,GAAG;EACH,MAAM,KAAK;EACX,QAAQ,KAAK;EACb,QAAQ,KAAK;EACb,UAAU,KAAK;EACf,OAAO,KAAK;EACZ,QAAQ,KAAK;EACb,UAAU,KAAK,aAAa,KAAA,IAAY,KAAA,IAAY,cAAc,KAAK,QAAQ;EAC/E,KACE,KAAK,QAAQ,KAAA,IACT,KAAA,IACA;GACE,MAAM,KAAK,IAAI;GACf,MAAM,KAAK,IAAI;GACf,aAAa,KAAK,IAAI;GACtB,WAAW,KAAK,IAAI;GACpB,mBAAmB,GAAG;EACxB;EACN,SAAS;CACX,CAAC,EAAA,CACY;CACb,OAAO;AACT"}

package/build/cli-recipes-Bci6HREA.mjs

@@ -0,0 +1,59 @@
+import { compareHours, fromCompact } from "./hours.mjs";
+import { d as readVenue, o as readProduct } from "./recipe-product-Dellqudu.mjs";
+import { a as keptWeekOf, i as describeVenue, o as productCode, r as describeProduct, s as venueCode, t as UsageError } from "./cli.mjs";
+//#region src/cli-recipes.ts
+/**
+* This function checks the parts of `venue` that need a file or a reader, before the store opens.
+* They are the kept week of `--compare` and the value of `--osm`. An error is a usage error.
+*/
+const prepareRecipe = (line) => {
+	if (line.osm !== void 0 && fromCompact(line.osm, "osm") === void 0) throw new UsageError(`--osm: the hours reader cannot read "${line.osm}". Write an opening_hours value, such as "Mo-Fr 08:00-18:00; Sa 09:00-13:00"`);
+	return { kept: line.compare === void 0 ? void 0 : keptWeekOf(line.compare) };
+};
+/** This function gives the options of the recipe that `venue` and `product` share. */
+const sharedOptions = (line) => ({
+	render: line.render,
+	maxAge: line.maxAge,
+	budget: line.budget,
+	onBehalfOf: line.onBehalfOf,
+	userProvided: line.userProvided,
+	fallback: line.fallback,
+	archiveDespiteRobots: line.archiveDespiteRobots
+});
+/**
+* This function does the command `venue`, and gives the exit code. It reads the hours of the venue
+* with `readVenue`. With `--compare`, it compares the week of the site (`from_site`) with the kept
+* week through `compareHours`.
+*/
+const venueCommand = async (target, line, input, io) => {
+	const options = {
+		...sharedOptions(line),
+		name: line.name,
+		requireName: line.requireName,
+		location: line.location,
+		osm: line.osm,
+		pages: line.pages
+	};
+	const result = await readVenue(target, line.url, options);
+	const site = result.from_site;
+	const compare = input.kept === void 0 ? void 0 : {
+		file: input.kept.file,
+		kept: input.kept.week,
+		comparison: site === null ? null : compareHours(input.kept.week, site.week)
+	};
+	io.out(io.json ? JSON.stringify(compare === void 0 ? result : {
+		...result,
+		compare
+	}, null, 2) : describeVenue(result, compare, io.renderOff).join("\n"));
+	return venueCode(result);
+};
+/** This function does the command `product`, and gives the exit code. It reads the page with `readProduct`. */
+const productCommand = async (target, line, io) => {
+	const result = await readProduct(target, line.url, sharedOptions(line));
+	io.out(io.json ? JSON.stringify(result, null, 2) : describeProduct(result, io.renderOff).join("\n"));
+	return productCode(result);
+};
+//#endregion
+export { prepareRecipe, productCommand, venueCommand };
+
+//# sourceMappingURL=cli-recipes-Bci6HREA.mjs.map

package/build/cli-recipes-Bci6HREA.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli-recipes-Bci6HREA.mjs","names":[],"sources":["../src/cli-recipes.ts"],"sourcesContent":["/*\n * The commands `venue` and `product` of the command line. They read a venue or a product page with\n * the recipes of the package (`readVenue` and `readProduct`). `cli.ts` loads this file only for these\n * two commands, so the other commands do not load the recipes.\n */\nimport type { KeptWeek, RecipeLine, VenueCompare } from \"./cli-line.ts\";\nimport {\n  UsageError,\n  describeProduct,\n  describeVenue,\n  keptWeekOf,\n  productCode,\n  venueCode,\n} from \"./cli-line.ts\";\nimport { compareHours, fromCompact } from \"./hours.ts\";\nimport type { RecipeTarget } from \"./recipe.ts\";\nimport type { ProductOptions } from \"./recipe-product.ts\";\nimport { readProduct } from \"./recipe-product.ts\";\nimport type { VenueOptions } from \"./recipe-venue.ts\";\nimport { readVenue } from \"./recipe-venue.ts\";\n\n/** These are the parts of the command line that `venue` and `product` use. */\nexport interface RecipeIo {\n  /** The value true prints the result as JSON. */\n  readonly json: boolean;\n  /** This function prints a line on the standard output. */\n  readonly out: (text: string) => void;\n  /** This text tells why rendering is off, when the computer has no browser. */\n  readonly renderOff?: string | undefined;\n}\n\n/** These are the inputs of `venue` that the command reads before the store opens. */\nexport interface RecipeInput {\n  /** This value is the kept week of `--compare`, or undefined with no `--compare`. */\n  readonly kept: KeptWeek | undefined;\n}\n\n/**\n * This function checks the parts of `venue` that need a file or a reader, before the store opens.\n * They are the kept week of `--compare` and the value of `--osm`. An error is a usage error.\n */\nexport const prepareRecipe = (line: RecipeLine): RecipeInput => {\n  if (line.osm !== undefined && fromCompact(line.osm, \"osm\") === undefined) {\n    throw new UsageError(\n      `--osm: the hours reader cannot read \"${line.osm}\". Write an opening_hours value, such as \"Mo-Fr 08:00-18:00; Sa 09:00-13:00\"`,\n    );\n  }\n  return { kept: line.compare === undefined ? undefined : keptWeekOf(line.compare) };\n};\n\n/** This function gives the options of the recipe that `venue` and `product` share. */\nconst sharedOptions = (line: RecipeLine): ProductOptions => ({\n  render: line.render,\n  maxAge: line.maxAge,\n  budget: line.budget,\n  onBehalfOf: line.onBehalfOf,\n  userProvided: line.userProvided,\n  fallback: line.fallback,\n  archiveDespiteRobots: line.archiveDespiteRobots,\n});\n\n/**\n * This function does the command `venue`, and gives the exit code. It reads the hours of the venue\n * with `readVenue`. With `--compare`, it compares the week of the site (`from_site`) with the kept\n * week through `compareHours`.\n */\nexport const venueCommand = async (\n  target: RecipeTarget,\n  line: RecipeLine,\n  input: RecipeInput,\n  io: RecipeIo,\n): Promise<number> => {\n  const options: VenueOptions = {\n    ...sharedOptions(line),\n    name: line.name,\n    requireName: line.requireName,\n    location: line.location,\n    osm: line.osm,\n    pages: line.pages,\n  };\n  const result = await readVenue(target, line.url, options);\n  const site = result.from_site;\n  const compare: VenueCompare | undefined =\n    input.kept === undefined\n      ? undefined\n      : {\n          file: input.kept.file,\n          kept: input.kept.week,\n          comparison: site === null ? null : compareHours(input.kept.week, site.week),\n        };\n  io.out(\n    io.json\n      ? JSON.stringify(compare === undefined ? result : { ...result, compare }, null, 2)\n      : describeVenue(result, compare, io.renderOff).join(\"\\n\"),\n  );\n  return venueCode(result);\n};\n\n/** This function does the command `product`, and gives the exit code. It reads the page with `readProduct`. */\nexport const productCommand = async (\n  target: RecipeTarget,\n  line: RecipeLine,\n  io: RecipeIo,\n): Promise<number> => {\n  const result = await readProduct(target, line.url, sharedOptions(line));\n  io.out(\n    io.json ? JSON.stringify(result, null, 2) : describeProduct(result, io.renderOff).join(\"\\n\"),\n  );\n  return productCode(result);\n};\n"],"mappings":";;;;;;;;AAyCA,MAAa,iBAAiB,SAAkC;CAC9D,IAAI,KAAK,QAAQ,KAAA,KAAa,YAAY,KAAK,KAAK,KAAK,MAAM,KAAA,GAC7D,MAAM,IAAI,WACR,wCAAwC,KAAK,IAAI,6EACnD;CAEF,OAAO,EAAE,MAAM,KAAK,YAAY,KAAA,IAAY,KAAA,IAAY,WAAW,KAAK,OAAO,EAAE;AACnF;;AAGA,MAAM,iBAAiB,UAAsC;CAC3D,QAAQ,KAAK;CACb,QAAQ,KAAK;CACb,QAAQ,KAAK;CACb,YAAY,KAAK;CACjB,cAAc,KAAK;CACnB,UAAU,KAAK;CACf,sBAAsB,KAAK;AAC7B;;;;;;AAOA,MAAa,eAAe,OAC1B,QACA,MACA,OACA,OACoB;CACpB,MAAM,UAAwB;EAC5B,GAAG,cAAc,IAAI;EACrB,MAAM,KAAK;EACX,aAAa,KAAK;EAClB,UAAU,KAAK;EACf,KAAK,KAAK;EACV,OAAO,KAAK;CACd;CACA,MAAM,SAAS,MAAM,UAAU,QAAQ,KAAK,KAAK,OAAO;CACxD,MAAM,OAAO,OAAO;CACpB,MAAM,UACJ,MAAM,SAAS,KAAA,IACX,KAAA,IACA;EACE,MAAM,MAAM,KAAK;EACjB,MAAM,MAAM,KAAK;EACjB,YAAY,SAAS,OAAO,OAAO,aAAa,MAAM,KAAK,MAAM,KAAK,IAAI;CAC5E;CACN,GAAG,IACD,GAAG,OACC,KAAK,UAAU,YAAY,KAAA,IAAY,SAAS;EAAE,GAAG;EAAQ;CAAQ,GAAG,MAAM,CAAC,IAC/E,cAAc,QAAQ,SAAS,GAAG,SAAS,CAAC,CAAC,KAAK,IAAI,CAC5D;CACA,OAAO,UAAU,MAAM;AACzB;;AAGA,MAAa,iBAAiB,OAC5B,QACA,MACA,OACoB;CACpB,MAAM,SAAS,MAAM,YAAY,QAAQ,KAAK,KAAK,cAAc,IAAI,CAAC;CACtE,GAAG,IACD,GAAG,OAAO,KAAK,UAAU,QAAQ,MAAM,CAAC,IAAI,gBAAgB,QAAQ,GAAG,SAAS,CAAC,CAAC,KAAK,IAAI,CAC7F;CACA,OAAO,YAAY,MAAM;AAC3B"}