polite-crawler 0.1.2 → 0.1.4

package/README.md

@@ -18,6 +18,7 @@ The site has interactive demos with the real code: <https://mark1russell7.github
 - **Watch and compare.** `watch` checks pages again on a schedule and tells what changed. `diff` compares two versions, and `read` takes `at`: the page as it was at a time.
 - **Archived copies, on request.** When a site refuses a page or is down, `fallback: "archive"` gives a copy from the Wayback Machine or Common Crawl. The result marks the copy as archived, with its date.
 - **Signed requests.** With an Ed25519 key, the crawler signs each request (Web Bot Auth). A site can then verify that a request comes from your crawler.
+- **Recipes.** `readVenue` reads the opening hours of a venue from its site, and `readProduct` reads a product page. `defineRecipe` makes a recipe of your own.
 
 ## Install
 
@@ -34,6 +35,8 @@ npx playwright-core install chromium-headless-shell
 
 Without the browser, the crawler reads each page as plain HTML. It checks for the browser one time, so a page that needs JavaScript gets a note at once, with no wait.
 
+Hybrid search (search by meaning and by words) is optional too. It uses a local model: transformers.js in the process (`npm install @huggingface/transformers`), or a local Ollama server. [Hybrid search](#hybrid-search) tells how.
+
 The package is ESM only, and it includes its TypeScript declarations. It has these entry points:
 
 | Entry point | What it gives | In a browser |
@@ -67,6 +70,7 @@ The store is in `~/.polite-crawler`. Use `--dir` or `POLITE_CRAWLER_DIR` for a d
 - `fetch --archive-fallback` gives an archived copy when a site refuses a page or is down. [Archived copies after a refusal](#archived-copies-after-a-refusal) tells the rules.
 - `--sign-key <file>` and `--signature-agent <url>` sign each request. `key-directory --sign-key <file>` prints the key directory to host. [Signed requests](#signed-requests-web-bot-auth) tells how.
 - `find --rank fast` makes a long question faster, and its results can be different. [Fast ranking](#fast-ranking) gives the measured difference.
+- `find --rank hybrid --embedder transformers` (or `--embedder ollama`) also finds the pages with a similar meaning. `embed --embedder <name>` gives vectors to each stored page first. [Hybrid search](#hybrid-search) tells the options.
 - `reindex` builds the index again. The documents of a recipe that the command cannot read (the recipes of another application) stay. The output counts them, and the captures that it skipped.
 - `export --out <file>` writes the stored captures as a WARC or a WACZ file. [Export to a web archive](#export-to-a-web-archive) tells what goes into the file.
 - `extract <doc> --field name=value[:kind]` checks the values that you propose for the fields of a stored page. Repeat `--field` for each field. `--capture` checks the text of one capture, and `--at` the version that was current at a time. [Check field values against a page](#check-field-values-against-a-page) tells the statuses.
@@ -96,15 +100,34 @@ npx polite-crawler watch list
 - `read --at` and `verify --at` use the version that was current at that time. [Read a page as it was at a time](#read-a-page-as-it-was-at-a-time) gives the rules.
 - `diff` prints the changed lines as a unified diff. A changed line marks its words with `[-removed-]` and `{+added+}`. `--from` and `--to` take a capture id or a time.
 - `watch add` takes `--every` (1 hour or more, `24h` by default), `--confirm 1` or `2`, `--render`, `--scope` and `--label <name=value>`.
-- `watch run` takes `--budget` (default `10m`) and `--max-pages` (default 100). It prints each check, then a summary. A scheduler of the operator starts it: the package has no daemon.
+- `watch run` takes `--budget` (default `10m`) and `--max-pages` (default 100). It prints each check, then a summary. A scheduler of the operator starts it.
+- `daemon` checks the watch list on a schedule, until you stop it. With `--mcp-http`, it also serves the MCP tools over local HTTP. [The daemon](#the-daemon) tells how.
+
+To read the hours of a venue or a product page, use `venue` and `product`. They use the recipes of the package:
+
+```sh
+npx polite-crawler venue https://forno-esempio.example/ --name "Forno Esempio" --pages 2
+npx polite-crawler venue https://forno-esempio.example/ --compare kept-week.json
+npx polite-crawler product "https://bottega-campione.example/products/olio-esempio?variant=222"
+```
+
+- `venue` prints the week of the site and each reading with its URL, its rank and its quote. It also prints the days where the readings disagree, the menus, the phone, the pages and the notes.
+- `venue` takes `--name`, `--require-name`, `--address`, `--lat` and `--lon`, `--osm <value>` and `--pages <0-8>`. [Read the hours of a venue](#read-the-hours-of-a-venue) tells what each one does.
+- Write a number below zero after `=`, for example `--lon=-20.5`. Else the command line reads it as an option.
+- `venue --compare <file>` compares the week of the site with a week that you keep, as JSON. It prints `same`, `changed`, `new` or `not read` for each day. The file is a week, for example `{"mon": [["07:00", "15:00"]], "sun": []}`, or the `from_site` of `venue --json`.
+- `product` prints the name, the brand, the SKU and the GTIN, and the price as money with the range of the offers. It also prints the availability, the variant, the seller, and a wall or a held-back page.
+- The two commands take `--render`, `--max-age`, `--budget`, `--user-provided`, `--archive-fallback` and `--archive-despite-robots`. `--on-behalf-of <agent>` applies the robots.txt rules of an agent too, for example `Claude-User`.
+- `--json` prints the `VenueResult` or the `ProductResult` of the library.
 
 The command gives these exit codes:
 
 | Code | Meaning |
 | --- | --- |
 | `0` | The command did its work. |
-| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. `discover` gives 1 when it finds no URL, and `crawl` when it gets no page. `read --at` and `verify --at` give 1 for a time before the first capture. `watch run` gives 1 when a check is `refused` or an `error`, or when entries stay due. `watch remove` gives 1 for a URL that is not on the list. With `--archive-fallback`, a page with an archived copy counts as done. `extract` gives 1 when a field is not `verified` or `normalized`. |
+| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. `discover` gives 1 when it finds no URL, and `crawl` when it gets no page. `read --at` and `verify --at` give 1 for a time before the first capture. `watch run` gives 1 when a check is `refused` or an `error`, or when entries stay due. `watch remove` gives 1 for a URL that is not on the list. With `--archive-fallback`, a page with an archived copy counts as done. `extract` gives 1 when a field is not `verified` or `normalized`. `embed` gives 1 when its budget stops it before each page has vectors. |
 | `2` | The command line is wrong. The command opens no store. |
+| `3` | `venue` and `product` read nothing: no opening hours of the site or no Product data, a page that is held back, or a page that did not come. |
+| `4` | `venue` and `product`: the site refused the crawler (a wall). The causes are a 401, a 402 or a 403, a challenge, a bot check, robots.txt or a stop of the host. The crawler does not try a different identity. |
 
 ## Use it from Claude (MCP)
 
@@ -149,7 +172,7 @@ The server gives nine tools:
 | Tool | What it does |
 | --- | --- |
 | `crawl_fetch` | It fetches pages into the store, and a page in the store answers with no request. It gives the age of each copy. With `archive_fallback`, a page that the site refuses gets an archived copy, marked `archived`. |
-| `crawl_find` | It searches the stored pages and gives ranked passages. |
+| `crawl_find` | It searches the stored pages and gives ranked passages. When the server has an embedder (`--embedder`), it also takes `rank: "hybrid"`, and each hit gives its `passage`. |
 | `crawl_read` | It gives a range of a stored document, with the capture ids to cite and the position of the next range. With `at`, it reads the version that was current at that time. |
 | `crawl_verify` | It finds a quote in a stored document and gives the exact span. It takes `at` too. |
 | `crawl_extract` | It checks the values that the assistant proposes for the fields of a stored page, and gives a status and a span for each field. It does not make values. With `at`, it checks the version that was current at that time. |
@@ -162,6 +185,8 @@ The MCP server applies the robots.txt rules for `Claude-User` too, because an as
 
 No tool adds a watch. A watch commits future requests to a site, so the operator adds it with `polite-crawler watch add`. An assistant can read the list (`crawl_watch_list`) and compare the versions of a watched page (`crawl_diff`).
 
+The daemon serves the same tools over local HTTP, with a bearer token (`polite-crawler daemon --mcp-http`). There, `crawl_site` and `crawl_discover` are MCP tasks: a client gets a task id at once, and gets the result later. [MCP over HTTP](#mcp-over-http) tells how.
+
 ### Web text is data, not instructions
 
 A page can contain text that tries to give orders to the assistant. Thus each tool gives the text of the pages in an envelope:
@@ -184,7 +209,7 @@ text (characters 0 to 4000 of 25000):
 
 When a page asks not to be AI input (`ai-input=no`), the server holds back its text. The tool `crawl_read` has the parameter `user_provided` for a URL that the user gave. A page can tell the assistant to set this parameter, so the server needs a signal from a person:
 
-1. When the client can ask the user (MCP elicitation), the server asks the user to confirm. The text comes back only when the user confirms.
+1. When the client can ask the user (MCP elicitation), the server asks the user to confirm. The text comes back only when the user confirms. When the question fails, for example at a timeout, the text stays held back. The opt-in of the operator does not answer for a user who does not answer.
 2. When the client cannot ask, the server releases the text only when the operator started it with `POLITE_CRAWLER_ALLOW_USER_PROVIDED=1` or `--allow-user-provided`.
 3. Else the text stays held back, and the note tells how to permit it.
 
@@ -201,6 +226,16 @@ The crawler records each release in the capture log.
 | `--no-browser` | | Do not open a headless browser. |
 | `--sign-key <file>` | `POLITE_CRAWLER_SIGN_KEY` | Sign each request with this Ed25519 key (PKCS#8 PEM or JWK). [Signed requests](#signed-requests-web-bot-auth) tells how. |
 | `--signature-agent <url>` | `POLITE_CRAWLER_SIGNATURE_AGENT` | Where a site finds the public key of `--sign-key`: an https origin, or the URL of a JSON Web Key Set. |
+| `--embedder <name>` | `POLITE_CRAWLER_EMBEDDER` | The local model of hybrid search: `transformers` or `ollama`. `crawl_find` then takes `rank: "hybrid"`. |
+| `--embed-model <name>` | `POLITE_CRAWLER_EMBED_MODEL` | The model. The defaults are `Xenova/bge-small-en-v1.5` (transformers) and `nomic-embed-text` (Ollama). |
+| `--ollama-url <url>` | `POLITE_CRAWLER_OLLAMA_URL` | The Ollama server. The default is `http://127.0.0.1:11434`. |
+| `--allow-remote-ollama` | `POLITE_CRAWLER_ALLOW_REMOTE_OLLAMA=1` | Permit an Ollama server on a different computer. The page text goes to that server. |
+
+For hybrid search with transformers.js, add the package to the `npx` command, as for the browser:
+
+```sh
+claude mcp add polite-crawler -- npx -y -p polite-crawler -p @huggingface/transformers polite-crawler mcp --embedder transformers
+```
 
 ## Use it as a library
 
@@ -235,7 +270,7 @@ await crawler.close();
 - `diff(doc, { from, to })` compares two versions. [Compare two versions](#compare-two-versions) tells what it gives.
 - `watch` is the watch list: `add`, `remove`, `list` and `run`. [Watch pages](#watch-pages) tells the rules.
 - `find` reads a query syntax. Text in double quotes is a phrase, and `bake*` is a prefix. The documents that hold each word come first. Case, accents and Unicode forms do not change a match, and Chinese, Japanese and Thai words match too.
-- `find` takes `rank`. The default is `exact`, and `fast` makes a long question faster. [Fast ranking](#fast-ranking) tells the difference.
+- `find` takes `rank`. The default is `exact`, and `fast` makes a long question faster. [Fast ranking](#fast-ranking) tells the difference. `hybrid` also finds by meaning, with the option `embedder` of `openCrawler`. [Hybrid search](#hybrid-search) tells how.
 - A page that asks not to be AI input is held back in each result. `fetch` gives no title, and `read` gives no title, no text and only the structural metadata. `verify` gives `found: false`, and `find` leaves the page out. Only `userProvided` lifts the hold, and the crawler logs it.
 - The hold applies to each version too. When a later capture does not ask it, the current text is not held back, but the earlier versions that asked it stay held back. `verify` with no `capture` does not examine them, `read` and `verify` of their capture hold them back, and `versions` shows no title for them.
 
@@ -286,7 +321,7 @@ The root entry point also exports the parts of the runtime, for an application t
 - `extractDocument` and `checkFields`: the check of field values over a runtime, and over a text and its metadata with no store.
 - `extractPage`, `typesOf`, `decodeBody` and `bodyEncoding`: the page reader and the decoder of a body in its declared encoding.
 - `readPdf` and `isPdf`: the PDF reader. `PdfOptions` sets limits on the pages, the characters, the time and the inflated bytes.
-- `fromText`, `fromSchemaOrg`, `fromCompact` and `describeWeek`: the opening-hours reader.
+- `fromText`, `fromSchemaOrg`, `fromCompact` and `describeWeek`: the opening-hours reader. `compareHours` compares a reading with a week that you keep.
 - `parseRobots` and `decide`: the robots.txt parser and matcher.
 - `RenderRefused`: the error of a renderer that did not render a page, with the reason `refused`, `download`, `scheme` or `browser`. A crawler does not count it as a failure of the host. After the reason `browser`, the crawler renders no more pages.
 
@@ -309,7 +344,7 @@ The readers give more facts than before:
 - `signer`: the signer of each request (Web Bot Auth). [Signed requests](#signed-requests-web-bot-auth) tells its options.
 - `scope`, `labels` and `scopeRank`: the labels of each record. One fetch of a robots.txt can serve requests of different scopes. Its record takes the labels of each request and the scope with the highest `scopeRank`, the most private one.
 
-`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent` and `extractors`. Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.
+`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent`, `extractors` and `recipes` ([the recipes of an application](#writing-a-recipe)). Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.
 
 `FetchRequest.maxAgeMs` can be `"auto"`: the crawler then uses the freshness that the server gave (`Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`). `crawler.resetHosts(host?)` removes the stops and the failure counts of one host or of each host, for a process that runs for a long time.
 
@@ -325,6 +360,8 @@ These parts are the stable API:
 - The diff: `diffDocument`, `diffTexts`, `DIFF_LIMITS`, `DiffOptions`, `DiffResult`, `DiffSide`, `DiffSummary`, `DiffHunk`, `DiffLine`, `WordPart`, `MetaChange`, `VersionChoice`, `DiffLimits`, `DiffInput` and `TextDiff`.
 - The time travel: `versionAtTime`, `isoTime` and `VersionAt`.
 - The watch list: `WatchList`, `WatchOptions`, `WatchEntry`, `WatchPending`, `WatchKind`, `WatchRun`, `WatchRunOptions`, `WatchEvent`, `WatchSummary`, `WatchStop`, `WATCH_LIMITS` and the file `watch.sqlite` (`WATCH_SCHEMA`). For a runtime of your own, it also has `WatchStore`, `WatchStoreOptions`, `runWatch` and `WatchRunnerOptions`. The lease steps of `WatchStore` (`claim`, `release` and `settle`) are the engine.
+- The daemon: `startDaemon`, `scheduleWatches`, `changeCommand`, `readDaemonState`, `daemonStateFile`, `DAEMON_DEFAULTS`, their options and results, and the state file (`DAEMON_STATE_SCHEMA`). `polite-crawler/mcp` adds `startMcpHttpServer`, `mcpHttpHost`, `CrawlTaskStore`, `TASK_LIMITS`, `MCP_HTTP_DEFAULTS`, `sameToken` and `makeToken`.
+- The MCP tasks follow the task API of the MCP SDK, which the SDK marks as experimental. When the SDK changes that API, the tasks change with it, and `CHANGELOG.md` says so.
 - The options and the results of `discover` and `crawl`: `DiscoverOptions`, `DiscoverResult`, `DiscoveredUrl`, `DiscoverSource`, `DiscoverStop`, `CrawlOptions`, `CrawlRun`, `CrawlPage`, `CrawlSummary`, `CrawlStop`, `UrlPattern` and `ScopeOptions`.
 - `polite-crawler/mcp`: `createMcpServer`, `startMcpServer` and its tools.
 - The readers: `polite-crawler/robots`, `/hours`, `/html`, `/classify`, `/politeness` and `/signals`. The site and its tools use them in the browser.
@@ -333,6 +370,12 @@ These parts are the stable API:
 - The archived fallback: `fallback` and `archiveDespiteRobots` of `FetchOptions`, the field `archived` of the results, and the types `Archived` and `WebArchive`.
 - Signed requests: `createWebBotAuthSigner`, `keyDirectory`, `verifySignature`, `KEY_DIRECTORY_PATH`, `KEY_DIRECTORY_TYPE`, the option `signer`, and the types `WebBotAuthOptions`, `WebBotAuthSigner`, `RequestSigner`, `KeyDirectory`, `DirectoryKey`, `KeyInput`, `SignatureAgentType`, `SignedRequest` and `SignatureCheck`.
 - The headers of a signature follow the draft. When the draft changes, the headers change with it, and `CHANGELOG.md` says so.
+- Hybrid search: the option `embedder` of `OpenOptions`, the interface `Embedder` (the extension point) with `EmbedCallOptions`, and the adapters `transformersEmbedder` and `ollamaEmbedder` with their options (`TransformersEmbedderOptions`, `OllamaEmbedderOptions`, `Pooling`, `EmbeddingModel`).
+- Also stable: `HybridCrawler` (the result of `openCrawler`), `rank: "hybrid"` (`SearchRank`, `SearchOptions`), `embed`, `rankWithin` with `RankOptions` and `Ranked`, and `hybrid` (`HybridSearch`). The three members are optional in `PoliteCrawler`, and `OpenedCrawler` does not change.
+- The options and the results of hybrid search are stable too: `HybridOptions`, `HybridFindOptions`, `EmbedOptions`, `EmbedResult`, `EmbedStop`, `Passage` and `Found.passage`. So are `HYBRID_DEFAULTS`, `OLLAMA_MODELS`, `TRANSFORMERS_MODELS`, `NO_EMBEDDER`, `NO_OLLAMA` and `NO_TRANSFORMERS`.
+
+- The recipe SDK: `defineRecipe`, `runRecipe`, `registerRecipe`, `readJson`, the option `recipes` of `openCrawler` and `createRuntime`, and `RESERVED_RECIPES`. Its types are `Recipe`, `RecipeDefinition`, `RecipeExtract`, `RecipeCapture`, `RecipeRequest`, `RecipeRequestOptions`, `RecipeRunOptions`, `RecipeResult`, `RecipeDoc` and `RecipeTarget`.
+- The recipes: `readVenue`, `venueReadingsOf`, `compareHours`, `readProduct` and `productOf`. Their options and results are stable too: `VenueOptions`, `VenueResult`, `VenueReading`, `VenuePage`, `VenueLink`, `VenueLocation`, `VenueOsm`, `VenueSource`, `VenueRead`, `HoursComparison`, `ProductOptions`, `ProductResult`, `ProductRead`, `ProductVariant` and `Money`.
 
 The other exports of the root entry point are the engine. Examples are `createRuntime`, `catchUpIndex`, `Crawler`, `CaptureLog`, `DocIndex`, `FileBlobStore`, `SqliteHostStateStore`, `Extractors`, `fetchAndIndex` and `makeRecord`. An application can build on them. A minor version before 1.0.0 can change them, and its section in `CHANGELOG.md` says so.
 
@@ -342,8 +385,137 @@ The engine of discovery and of the crawl is `discoverSite`, `crawlSite`, `readSi
 
 The engine of the time travel and of the watch list is the optional method `ReadingIndex.versionAt`, `DocIndex.versionAt` and `Crawler.hostStop`.
 
+The engine of hybrid search is `createHybridSearch`, `HybridSearchOptions`, `VectorStore`, `VECTORS_SCHEMA` and the types of the vector store. It also includes `chunkText`, `chunkTokensOf`, `estimateTokens`, `CHUNK_LIMITS`, `fuseRanks`, `unitVector`, `isLoopback`, `FeatureExtractor`, `TransformersModule`, the optional method `ReadingIndex.digests` and `DocIndex.digests`. The file `vectors.sqlite` is a projection, as the index is: it is not part of the stable store.
+
+The engine of the recipes is `rankHoursLinks`, `namesVenue`, `locationMatches`, `pickProduct`, `pickVariant`, `readAmount`, `VENUE_LIMITS`, `PRODUCT_LIMITS`, `RECIPE_LIMITS` and the `steps` of `VenueRead` and `ProductRead`. The words that rank a link or a label are the engine too, so a version can improve them.
+
 The engine of the archived fallback and of the signatures is `archiveExtractor`, `archivedDocId`, and the `fallback` and `archiveDespiteRobots` of `PageOptions`. It also includes `range`, `signalLines` and `originalHeaderPrefix` of `FetchRequest`, `ai_input` of `ExtractedDoc`, `sign` of `RenderOptions` and `signer` of `CrawlerOptions`.
 
+## Hybrid search
+
+Hybrid search finds the pages that hold the words of a question, and also the pages with a similar meaning. It is opt-in: the default ranking stays exact BM25.
+
+```ts
+import { openCrawler, transformersEmbedder } from "polite-crawler";
+
+const crawler = openCrawler({ dir: "./crawl", embedder: transformersEmbedder() });
+await crawler.embed({ budget: "10m" }); // vectors for each stored page
+
+for (const hit of await crawler.find("a loaf with a dark crust", { rank: "hybrid" })) {
+  console.log(hit.doc, hit.score, hit.passage, hit.snippet);
+}
+```
+
+- `rank: "hybrid"` needs an embedder. Without one, the search rejects with `NO_EMBEDDER`. It does not change to BM25 in silence.
+- `crawler.rankWithin(q, docs, { rank: "hybrid" })` ranks a set of stored documents in the same way.
+- `openCrawler` gives a `HybridCrawler`: an `OpenedCrawler` with `hybrid`, `embed` and `rankWithin`, and a `find` that takes `rank: "hybrid"`.
+- The command line has `--embedder`, `find --rank hybrid` and `embed`. The MCP tool `crawl_find` takes `rank: "hybrid"` when the server has an embedder.
+
+```sh
+npx polite-crawler embed --embedder ollama --budget 30m
+npx polite-crawler find "a loaf with a dark crust" --rank hybrid --embedder ollama
+```
+
+### transformers.js
+
+This embedder uses the model in the process of the crawler. The package `@huggingface/transformers` is an optional peer, and the embedder loads it at its first text.
+
+```sh
+npm install @huggingface/transformers
+```
+
+```ts
+import { transformersEmbedder } from "polite-crawler";
+
+const embedder = transformersEmbedder({ model: "Xenova/bge-small-en-v1.5", quantized: true });
+```
+
+- The default model is `Xenova/bge-small-en-v1.5`: 384 dimensions and an input of 512 tokens. The 8-bit model (`quantized: true`, the default) is approximately 34 MB.
+- The first use downloads the model from the Hugging Face Hub (`huggingface.co`). This download does not go through the polite crawler, so robots.txt, the pace and the capture log do not apply to it.
+- transformers.js keeps the files in `cacheDir`. Its default is `node_modules/@huggingface/transformers/.cache/`. The command line uses `<store>/models`. With `localOnly: true`, the embedder uses only the cache and downloads nothing.
+- The embedder pools the vectors of the tokens as the model expects: `cls` for BGE and `mean` for the other models. The option `pooling` changes it.
+
+### Ollama
+
+This embedder sends the texts to a local Ollama server (`POST /api/embed`). The package has no dependency for it.
+
+1. Install Ollama from <https://ollama.com>, and start it with `ollama serve`.
+2. Pull a model: `ollama pull nomic-embed-text`.
+
+```ts
+import { ollamaEmbedder } from "polite-crawler";
+
+const embedder = ollamaEmbedder({ model: "nomic-embed-text", baseUrl: "http://127.0.0.1:11434" });
+```
+
+| Model | Size | Dimensions | Input in tokens |
+| --- | ---: | ---: | ---: |
+| `nomic-embed-text` (the default) | approximately 274 MB | 768 | 8,192 |
+| `all-minilm` | approximately 46 MB | 384 | 256 |
+| `mxbai-embed-large` | approximately 670 MB | 1,024 | 512 |
+
+- The embedder sends 32 texts in each request (`batch`). It adds the prefixes that `nomic-embed-text` and `mxbai-embed-large` expect.
+- An error tells what to do: `ollama serve` when no server answers, and `ollama pull <model>` when the model is not there.
+- The embedder accepts only a server on this computer (a loopback address). `allowRemote: true` (`--allow-remote-ollama`) permits a different computer.
+
+### An embedder of your own
+
+An application can give an embedder of its own, as it can give a renderer:
+
+```ts
+import type { Embedder } from "polite-crawler";
+
+const embedder: Embedder = {
+  id: "my-model-v1", // the model and its version: the store keeps it with each vector
+  dimensions: 384,
+  maxInputTokens: 512,
+  embed: async (texts, { signal, purpose }) => myModel.embed(texts, { signal, purpose }),
+};
+```
+
+`purpose` is `"document"` for the chunks of a page and `"query"` for a question. Give a new `id` when the vectors of the model change.
+
+### What goes where
+
+- The embedder gets the text of the stored pages and the words of each question. With transformers.js, the text stays in the process. With Ollama, the text goes to the Ollama server.
+- A page that asks not to be AI input does not go to the embedder, and no hybrid result gives it. The rule applies to each version, as for `find`.
+- Only the download of a transformers.js model goes to the internet, to the Hugging Face Hub.
+
+### How a hybrid search ranks
+
+1. Full-text search gives the BM25 candidates (`candidates`, 50 by default). A candidate whose text has no vectors gets them first.
+2. The vector search gives the documents with the closest chunks (`neighbors`, 50 by default). It compares the vector of the question with each stored vector (a brute-force cosine).
+3. Reciprocal rank fusion makes one list. Each document gets the sum of 1 / (k + rank) over the two lists, with the rank from 1. The constant k is `fusionK`, 60 by default.
+
+- The snippet and the `passage` of each hit come from its best chunk. `passage.start` and `passage.end` are positions in the text of the capture of the hit, for `read`.
+- A search gives vectors only to its candidates. Thus a page with the meaning and none of the words comes back only after `embed` gave it vectors.
+- `embed` takes `budget` (`10m` by default), `maxDocs` and `kind`. Start it again to continue: it does not send a text with vectors again.
+- `find` and `rankWithin` take `candidates`, `neighbors`, `fusionK` and `signal`. `HYBRID_DEFAULTS` has the defaults.
+
+### Chunks and storage
+
+- A chunk is a window of whole sentences. It ends at the end of a paragraph when it can, and a longer sentence is cut at its spaces.
+- A chunk has the input limit of the model in tokens with a margin of 10 percent, and 512 tokens at most. Two chunks next to each other share approximately 10 percent of their tokens. A text gives 64 chunks at most.
+- The vectors go into `vectors.sqlite` in the store. The key of a row is the SHA-256 of the text, the number of the chunk and the `id` of the embedder. Thus a reindex keeps each vector, and the same text gets its vectors one time for each model.
+- The file is a projection, as `index.sqlite` is. Delete it, and `embed` makes the vectors again.
+- A vector takes 4 bytes for each dimension: 1.5 KiB for 384 dimensions and 3 KiB for 768. With the keys and the indexes of SQLite, a chunk of 384 dimensions took 2.4 KiB of the file in the benchmark.
+- The first hybrid search reads the vectors of the embedder into memory. Thus the memory is approximately the size of the vectors.
+
+The benchmark `bench/find.mts --hybrid` measured the time in CI ([the results](https://github.com/mark1russell7/crawler/actions/runs/39082851222)). A fake embedder of almost no cost gave the vectors, so the time of a real model comes on top. The index had 30,000 documents of 900 words, which gave 89,100 chunks of 384 dimensions. Each time is the median of 12 questions on Node.js 24:
+
+| Step | Time |
+| --- | ---: |
+| The cosine of a question and each of the 89,100 vectors | 82 ms |
+| `find`, `rank: "hybrid"`, 1 word | 88 ms |
+| `find`, `rank: "hybrid"`, 16 words | 122 ms |
+| `rankWithin` of 40,000 ids, `rank: "hybrid"`, 16 words | 528 ms |
+| The first search: it reads 131 MiB of vectors into memory | 714 ms |
+| `embed` of the 30,000 documents, with no time for a model | 17 s |
+
+- The brute-force cosine is linear in the number of chunks. It is a good fit up to approximately 100,000 chunks (approximately 30,000 pages of this size). A search then takes approximately 0.1 s, and the vectors take approximately 150 MiB of memory at 384 dimensions.
+- One million chunks take approximately 1 s for each search and 1.5 GiB of memory. A larger store needs an index of the vectors, and the package does not have one.
+- `vectors.sqlite` was 210 MiB for the 89,100 chunks.
+
 ## Read a page as it was at a time
 
 Each capture of a document is a version, and the index keeps each version. `read` and `verify` take `at`, a time. They then use the version that was current at that time.
@@ -461,10 +633,181 @@ The `diff` of an event of the kind `changed` gives the counts of the diff from t
 - Each step is one SQLite transaction (`BEGIN IMMEDIATE`) in WAL mode, and it waits for the lock of a different process (`busy_timeout`).
 - A run claims each entry with a lease before its check. The lease holds until the end of the budget and 2 minutes more. Thus two processes that run at the same time check each entry one time. A process that stops holds its leases until they end.
 - The list stays after a restart, with the state of each entry: the last confirmed capture, the suspected change and the counts.
-- The package has no daemon. A scheduler of the operator (cron, a systemd timer or a job of an application) starts `watch.run`.
+- A scheduler of the operator (cron, a systemd timer or a job of an application) starts `watch.run`. Or the daemon starts it on a schedule: [The daemon](#the-daemon).
 - The command line has `watch add <url...>`, `watch remove <url...>`, `watch list` and `watch run`. The MCP tool `crawl_watch_list` lists the entries.
 - No MCP tool adds a watch. A watch commits future requests to a site, so it is an action of the operator, not of an assistant.
 
+## The daemon
+
+The daemon is a local process that runs for a long time. It checks the watch list on a schedule, and it can serve the MCP tools over local HTTP.
+
+```sh
+npx polite-crawler watch add https://example.com/hours --every 6h
+npx polite-crawler daemon --events ~/changes.jsonl --on-change "jq -c . >> ~/seen.jsonl"
+npx polite-crawler daemon status
+```
+
+```ts
+import { startDaemon } from "polite-crawler";
+
+const daemon = await startDaemon({
+  dir: "./crawl",
+  tick: "1m",
+  onChange: (event) => console.log(event.url, event.kind, event.capture),
+});
+console.log(daemon.status().next_due);
+await daemon.stop(); // or SIGINT and SIGTERM
+```
+
+### The schedule
+
+- Each tick is one `watch run`, with the budget and the maximum of checks of a tick. Thus each polite rule applies: robots.txt, the egress policy, one request at a time to each host, the stops of a host and `Retry-After`.
+- The next tick comes when the first entry is due, and one tick (`--tick`, `1m` by default) after the last tick at the latest.
+- A random jitter from 0 to `--jitter` (`10s` by default) comes after it. Thus a check does not come before its time.
+- When a tick stops at its budget (`--tick-budget`, `10m` by default) or at its maximum (`--tick-pages`, 100 by default), the entries that stay due wait for the next tick.
+- An entry that a different process holds waits for the end of its lease. Only one daemon operates on a store: a second daemon does not start.
+- `scheduleWatches(crawler.watch, options)` gives the same schedule over the watch list of a crawler of your own.
+
+### The changes
+
+A change is an event of `watch run` with `confirmed: true`: the last confirmed version of the page changed. The first version of a page is not a change.
+
+- The daemon writes each change as one JSON line, to the standard output or to `--events <file>`. The library takes `events` (a file or a function) and `onChange` (a function).
+- `--on-change <command>` starts your command for each change. The command gets the event as one JSON line on its standard input.
+- The commands run one at a time, in the order of the changes. A command that runs for more than 1 minute stops. A command that fails becomes the last error of the status.
+
+`--on-change` is a command of the operator. Obey these rules:
+
+- The daemon starts the command in the shell of the system, as the user of the daemon, with the environment of the daemon. Give only a command that you wrote or trust.
+- Do not take the command from a page, from an assistant or from a different user.
+- The daemon gives the event only on the standard input, not in the command line. Thus no text of a page goes into a shell command.
+- The event holds URLs and notes from sites. In your command, read the event as data, for example with `jq`. Do not give its text to `eval` or to a shell.
+
+### The stop
+
+- At `SIGINT` or `SIGTERM`, the daemon starts no new check. The checks in progress get `--grace` (`10s` by default) to end and write their state.
+- After the grace, the checks in progress abort, and the daemon releases their leases. The entries stay due for the next start. A second signal aborts the checks at once.
+- The daemon stops its MCP server and cancels its tasks. It writes the state file, closes the store and exits with code 0.
+- A daemon that stops with no clean stop (a crash, or a forced stop on Windows) keeps its leases until they end: the budget of the tick and 2 minutes.
+
+### The status
+
+`polite-crawler daemon status` reads the state file `<store>/daemon/state.json`. It sends no request, and it opens no store.
+
+- It shows the next due watches, the last tick, the next tick, the counts, the MCP endpoint, the tasks that run and the last error.
+- It exits with code 0 when a daemon operates on the store, and with 1 when no daemon operates. `--json` prints the JSON of the state file.
+- The daemon writes the state file after each tick and at each change of a task. The file is for the owner only.
+- With `--mcp-http`, `GET /status` with the bearer token gives the same JSON.
+
+### MCP over HTTP
+
+```sh
+npx polite-crawler daemon --mcp-http
+claude mcp add --transport http polite-crawler http://127.0.0.1:9307/mcp --header "Authorization: Bearer $(cat ~/.polite-crawler/daemon/mcp-token)"
+```
+
+- `--mcp-http` serves the nine tools over the Streamable HTTP transport of MCP, at `http://127.0.0.1:9307/mcp`. `--mcp-port` and `--mcp-host` change the address.
+- The tools, the envelope, elicitation and `--allow-user-provided` are the same as for `polite-crawler mcp`.
+- `crawl_site` and `crawl_discover` are MCP tasks (the task primitive of the MCP specification 2025-11-25). The task support is native: it uses the tasks of the MCP SDK.
+- A client that asks for a task gets a task id at once. It polls the status (`tasks/get`), gets the result (`tasks/result`) and can cancel the task (`tasks/cancel`). A cancel stops the requests in progress.
+- A client with no tasks gets the result in the call, with a progress notification for each page, as from `polite-crawler mcp`.
+- The status message of a task holds only counts, for example `3 of 10 done`, and no text of a page.
+- A task and its result stay for 1 hour. The server has 8 tasks in progress at most: a task above the limit fails at once, with the reason.
+- Each session has its own tasks. When a session closes, its tasks stop.
+
+The library has `startMcpHttpServer(crawler, options)` in `polite-crawler/mcp`. The daemon reaches it through a small seam, `DaemonOptions.mcpHost`, so a different MCP host can replace it.
+
+#### The token
+
+- Each request needs `Authorization: Bearer <token>`. The server compares the SHA-256 digests of the two tokens in constant time.
+- By default, the daemon makes a new random token at each start. It writes the token to `<store>/daemon/mcp-token`, for the owner only where the system has file modes. On Windows, the folder of the store in your profile protects it.
+- To keep one token across starts, write it to a file of your own and give `--mcp-token-file <file>`. The daemon reads the file and writes nothing. It warns when other users can read the file.
+
+```sh
+node -e "console.log(require('node:crypto').randomBytes(32).toString('base64url'))" > ~/.polite-crawler-token
+chmod 600 ~/.polite-crawler-token
+npx polite-crawler daemon --mcp-http --mcp-token-file ~/.polite-crawler-token
+```
+
+#### The address
+
+- The server binds to `127.0.0.1` by default, so only your computer can reach it.
+- On a loopback address, the server accepts only a loopback name in the `Host` header. An `Origin` header must name the same host. Thus a web page cannot reach the server by DNS rebinding.
+- A host that is not loopback, for example `--mcp-host 0.0.0.0`, needs `--allow-remote`. Without it, the command line is wrong.
+
+> **Warning:** With `--allow-remote`, each computer that reaches the port can try the token. The server speaks plain HTTP, so a computer on the path can read the token and the pages. The tools also fetch pages from the address of your computer. Use `--allow-remote` only on a private network or behind a TLS proxy of your own. Do not open the port to the internet.
+
+### Start the daemon as a service
+
+The package installs no service. These examples start the daemon at login, and they stop it with `SIGTERM`. Install the package one time (`npm install -g polite-crawler`), and change the paths.
+
+On Linux, with a user service of systemd:
+
+```ini
+# ~/.config/systemd/user/polite-crawler.service
+[Unit]
+Description=polite-crawler daemon
+After=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/polite-crawler daemon --events %h/.polite-crawler/changes.jsonl
+Restart=on-failure
+KillSignal=SIGTERM
+TimeoutStopSec=30
+
+[Install]
+WantedBy=default.target
+```
+
+```sh
+systemctl --user daemon-reload
+systemctl --user enable --now polite-crawler
+journalctl --user -u polite-crawler -f
+npx polite-crawler daemon status
+```
+
+On macOS, with a launch agent of launchd:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!-- ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plist -->
+<plist version="1.0">
+<dict>
+  <key>Label</key><string>io.github.polite-crawler.daemon</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>/usr/local/bin/polite-crawler</string>
+    <string>daemon</string>
+    <string>--events</string>
+    <string>/Users/you/.polite-crawler/changes.jsonl</string>
+  </array>
+  <key>RunAtLoad</key><true/>
+  <key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>
+  <key>StandardErrorPath</key><string>/Users/you/.polite-crawler/daemon.log</string>
+</dict>
+</plist>
+```
+
+```sh
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plist
+launchctl bootout gui/$(id -u)/io.github.polite-crawler.daemon
+```
+
+On Windows, with the Task Scheduler (PowerShell):
+
+```powershell
+$action = New-ScheduledTaskAction -Execute "$env:APPDATA\npm\polite-crawler.cmd" `
+  -Argument "daemon --events `"$env:USERPROFILE\.polite-crawler\changes.jsonl`""
+$trigger = New-ScheduledTaskTrigger -AtLogOn
+# no time limit: the default limit of 3 days stops the daemon
+$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit ([TimeSpan]::Zero) `
+  -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)
+Register-ScheduledTask -TaskName "polite-crawler daemon" -Action $action -Trigger $trigger -Settings $settings
+```
+
+The Task Scheduler stops a task with no signal. Thus the checks in progress stop at once, and their leases end after the budget of the tick and 2 minutes. `daemon status` then says that the daemon stopped with no clean stop.
+
 ## Check field values against a page
 
 A model or a person often reads a value from a page: a price, a date, opening hours, an author or a DOI. `extract` checks each proposed value against the stored copy, and it locates the value. It does not make values. It only checks and locates the values that you give, so each value has its proof.
@@ -543,6 +886,210 @@ The time of the check is linear in the length of the text, for a fixed set of fi
 
 The MCP tool `crawl_extract` gives the statuses and the positions in its JSON. The stored text, the readings and the excerpts come in the `<untrusted-web-content>` envelope. The command line has `polite-crawler extract`.
 
+## Read the hours of a venue
+
+`readVenue` reads the opening hours of a venue (a restaurant, a shop or a museum) from its own site. It fetches a few pages, and it gives each reading with its source.
+
+```ts
+import { compareHours, openCrawler, readVenue } from "polite-crawler";
+
+const crawler = openCrawler({ dir: "./crawl" });
+const venue = await readVenue(crawler, "https://forno-esempio.example/", {
+  name: "Forno Esempio",
+  osm: "Mo-Fr 07:00-15:00; Sa 08:00-13:00; Su off", // a value that you got from OpenStreetMap
+});
+console.log(venue.best?.hours, venue.best?.capture, venue.disagree);
+for (const reading of venue.readings) console.log(reading.from, reading.label, reading.hours, reading.url);
+
+const change = compareHours(keptWeek, venue.from_site!.week);
+console.log(change.status, change.days); // changed [ 'sat' ]
+```
+
+The recipe reads these sources:
+
+1. The home page: its JSON-LD (`openingHoursSpecification` and `openingHours`) and its text.
+2. The pages that most likely hold hours or a menu. `rankHoursLinks` ranks the links of the home page by their words, for example "hours", "menu" and "visit". `pages` sets the number of these pages: 3 by default, and 8 at most.
+3. An OpenStreetMap `opening_hours` value, when you give `osm`. The recipe sends no request to OpenStreetMap.
+
+- Each page is a fetch of the crawler. Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A page that needs JavaScript gets the browser, as for `fetch`.
+- Each reading has `from`, `label`, `week`, `hours` (the week as words), `inferred`, `partial`, `season` and `raw`. It names its `url`, `doc` and `capture`, and a reading of JSON-LD names its `path`.
+- The `raw` of a reading of the text is a quote: `verify(reading.doc, reading.raw, { capture: reading.capture })` finds it.
+- `rank` orders the readings. JSON-LD is 0, a text under an hours heading is 1, OpenStreetMap is 2, and a text with no other label is 3. A different schedule (a brunch, a kitchen, a happy hour or an office) is 4.
+- `best` is the first reading of the opening hours. `from_site` is the best reading of the site itself, with no guessed am or pm. A check for a change compares `from_site`.
+- `disagree` names the days where two readings of the whole schedule (a rank of 2 or less) disagree.
+- The result also gives `menus` (the menu links that are not on a review or social site), `phone` (from the JSON-LD) and `captures`.
+
+| Option | What it does |
+| --- | --- |
+| `name` | The name of the venue. `names_venue` tells if the home page names it. A domain that a venue let go can show a different business. |
+| `requireName` | With `true`, a home page that does not name the venue gives no reading, and the recipe fetches no more pages. |
+| `location` | The `address`, or `lat` and `lon`, of the venue. On a page that lists more than one location, the recipe reads only this location. With no `location`, it reads none of them. |
+| `osm` | An OpenStreetMap `opening_hours` value, or `{ value, url, doc, capture }` with its source. |
+| `pages` | The pages after the home page, from 0 to 8. The default is 3. |
+| `render`, `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels`, `job` | As for `fetch`. |
+| `budget` | No request starts after this time, for example `"2m"`. |
+| `userProvided` | Read a page that asked not to be AI input. Use it only for a URL that a person gave. The crawler logs it. |
+| `fallback`, `archiveDespiteRobots` | Read an archived copy of a page that the site refuses. The reading then has `archived`. |
+
+- A location of a page matches when it has the same street number and a word of the street, or when it is within 150 m.
+- `moved` names the new site when the home page landed on a different host.
+- A page that asked not to be AI input gives no reading, and `held_back` is true.
+- The recipe writes nothing but the capture log. It does not decide which reading is true. That decision belongs to your application.
+
+`venueReadingsOf(doc, { location })` reads one stored document with no request. Thus it also reads an earlier version:
+
+```ts
+import { venueReadingsOf, versionAtTime } from "polite-crawler";
+
+const { version } = versionAtTime(crawler.runtime, "url:https://forno-esempio.example/", "2026-10-01T00:00:00Z");
+const then = venueReadingsOf(version!);
+```
+
+`compareHours(kept, reading, { skipSplit })` compares a reading with the week that your application keeps. It is in `polite-crawler/hours` too.
+
+- It gives `status` (`same`, `changed` or `new`), the changed `days`, the `added` days and the `skipped` days.
+- It compares only the days that the two weeks state. A day that only the reading states is `added`, not a change.
+- `24:00` and `00:00` are the same time, and `9:00` is `09:00`. The sequence of the intervals of a day does not matter.
+- With `skipSplit`, it does not compare a day with two or more intervals, for a store that keeps one interval for each day.
+- One reading can be an error of the site. Confirm a change with a second reading, as a watch does.
+
+The command line has `polite-crawler venue`, and `venue --compare <file>` does this comparison. [Use it from the command line](#use-it-from-the-command-line) gives the options.
+
+## Read a product page
+
+`readProduct` reads the schema.org Product data of a product page. Most shop systems write this data on each product page.
+
+```ts
+import { readProduct } from "polite-crawler";
+
+const product = await readProduct(crawler, "https://bottega-campione.example/products/olio-esempio?variant=222");
+if (product.wall) console.log("the shop refused the crawler:", product.note);
+else console.log(product.name, product.price, product.price_to, product.availability, product.variant);
+// Olio Esempio { amount: 9.9, currency: 'EUR', raw: '9.90' } { amount: 18, ... } InStock { sku: 'OE-500', ... }
+```
+
+- The result gives `name`, `brand`, `sku`, `gtin`, `price`, `price_to`, `availability`, `availabilities`, `variant`, `seller` and `image`. It names the `doc` and the `capture` of the page.
+- A price is `{ amount, currency, raw }`, and the currency is an ISO 4217 code. `€` and `£` count too, because each one names one currency. A price with no currency, or with `$`, is not money, and a note says so.
+- `price` is the lowest price. `price_to` is the highest price, when the offers give a range (sizes, colors, variants). The prices are in the first currency of the offers.
+- A list price and a struck-through price (`priceType` `ListPrice` or `StrikethroughPrice`) are not the price.
+- `variant` is the variant that the URL names with the parameter `variant`, `sku` or `gtin`. The recipe reads its price alone.
+- `availability` is a schema.org name, for example `InStock`. When the offers do not agree, it is `InStock` if one offer is in stock.
+- The options are `render`, `maxAge`, `onBehalfOf`, `signal`, `budget`, `scope`, `labels`, `job`, `userProvided`, `fallback` and `archiveDespiteRobots`, as for `readVenue`.
+
+A page often has more than one Product record: the related products, and the products that others bought. The recipe picks the product of the page:
+
+1. The product whose `url` or `@id` is the page. An `@id` that is only a fragment (`#product`) does not count, because a related product can have one too.
+2. Else the largest product with offers or variants.
+3. Else the largest product. The recipe also reads the `mainEntity` of a `WebPage` or an `ItemPage`.
+
+- The index keeps 30,000 characters of the JSON-LD of each page. A product with many variants can be larger, so the recipe reads the stored body again.
+- When the plain page has no Product data, the recipe renders the page once, because some shops write that data with JavaScript. `render: "never"` stops this.
+- A shop that refuses the crawler gives `wall: true`, with a note. The causes are a 401, a 402 or a 403, a challenge, a bot check, a stop of the host or robots.txt. The recipe does not try a different identity.
+- A page with no Product data gives `found: false`. It is often a list of products.
+
+`readAmount` reads an amount as a shop writes it. The last separator is the decimal point. One kind of separator before groups of three digits separates thousands, so `1.299` is 1299. A currency with three decimal digits (`KWD`) reads `1.299` as 1.299. A range, a negative amount and zero give no amount.
+
+`productOf(records, url)` reads the JSON-LD records of a page with no request, so it also reads an earlier version. `extract` checks each value against the stored page:
+
+```ts
+const check = await crawler.extract(product.doc!, {
+  price: { value: product.variant!.price!.raw, kind: "json-ld:Product.offers.price" },
+});
+```
+
+The command line has `polite-crawler product`. It exits with 4 at a wall, and with 3 when it reads no Product data.
+
+## Writing a recipe
+
+A recipe tells the crawler what to fetch and how to read it into documents. The venue and product recipes read pages with the page reader of the package. A recipe of your own reads a source that the page reader does not know, for example an API.
+
+```ts
+import { defineRecipe, openCrawler, readJson, runRecipe } from "polite-crawler";
+
+interface Item {
+  readonly id: string;
+  readonly name: string;
+  readonly price: string;
+}
+
+const menu = defineRecipe({
+  name: "forno-menu", // it goes into each capture record: do not change it
+  mode: "api", // a documented API: robots.txt does not apply, and the pace does
+  minInterval: "2s", // the documented pace of the API
+  maxAge: "6h",
+  fetch: (day: string) => `https://api.forno-esempio.example/menu?day=${encodeURIComponent(day)}`,
+  extract: (body, capture) => {
+    const { items } = readJson(body, capture) as { items: Item[] }; // a body that is not JSON throws
+    return items.map((item) => ({
+      doc: `menu:forno-esempio/${item.id}`,
+      kind: "menu-item",
+      title: item.name,
+      text: `${item.name}: ${item.price}`,
+      url: capture.url,
+      meta: { price: item.price },
+    }));
+  },
+});
+
+const crawler = openCrawler({ dir: "./crawl", recipes: [menu] });
+const run = await runRecipe(crawler, menu, "mon");
+console.log(run.outcome, run.capture, run.docs.map((doc) => doc.doc));
+
+await crawler.find("focaccia"); // the documents of the recipe are in the index
+await crawler.read("menu:forno-esempio/focaccia", { at: "2026-10-01T00:00:00Z" }); // each version stays
+```
+
+`defineRecipe` takes these values:
+
+| Value | Default | What it does |
+| --- | --- | --- |
+| `name` | (necessary) | The name of the recipe: 1 to 64 letters, digits and `. _ : -`. The names of the crawler (`RESERVED_RECIPES`) are not permitted. |
+| `extract` | (necessary) | The function that reads a body into documents. |
+| `fetch` | the input is the URL | The function that gives the request for an input: a URL, or `{ url, accept, headers, secretParams }`. |
+| `mode` | `page` | `page` is a web page, and robots.txt applies. `api` is a documented API with terms of its own. |
+| `accept` | HTML for a page, JSON for an API | The `Accept` header. |
+| `minInterval` | the policy of the host | The shortest time between two requests to the source. |
+| `maxAge` | `24h` | A stored copy younger than this age answers with no request. `auto` uses the freshness that the server gave. |
+| `maxBytes` | 5 MiB | The most bytes of a body that the crawler keeps. |
+| `description` | none | A text that tells what the recipe reads. |
+
+`runRecipe(crawler, recipe, input, options)` fetches the request of an input through the crawler, and reads its capture with the extractor.
+
+- The result gives `outcome`, `status`, `url`, `docs`, `capture`, `fetched_at`, `from_cache`, `held_back`, `error` and `note`. The `url` shows each secret parameter as `REDACTED`.
+- Each document gives `doc`, `kind`, `title`, `url`, `meta`, `chars`, `capture` and `fetched_at`. `read` gives its text. A held-back document shows no title and only its structural metadata.
+- The options are `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels`, `job`, `budget`, `userProvided`, `fallback` and `archiveDespiteRobots`.
+- Give each recipe to `openCrawler` (or `createRuntime`) with `recipes`. Then `reindex` reads its captures from the start. `runRecipe` also registers a recipe the first time that it gets it.
+- `registerRecipe(runtime.extractors, recipe)` registers a recipe in a runtime of your own. A different extractor with the same name throws an error.
+- `recipe.request(input, options)` gives the request of the crawler, for `fetchAndIndex` in an application that builds its own steps.
+- `readJson(body, capture)` reads a body as JSON, in the encoding that the response declares. A body that is not JSON throws an error that names the URL and the content type.
+- `runRecipe`, `readVenue` and `readProduct` take a crawler or a runtime (`createRuntime`). The option `onBehalfOf` of `openCrawler` does not apply to them, so give `onBehalfOf` to each call.
+
+### The rules of an extractor
+
+1. **An extractor is pure.** The same body and capture always give the same documents. Do not read the clock, a random number or a state from an earlier call.
+2. **An extractor sends no request.** It gets the body of the capture. To read a second source, start a second recipe. Its fetch is then polite, and it has its capture.
+3. **The runtime reports an error, and the work continues.** An extractor that throws gives no document. `runRecipe` gives the error in `error`, and `reindex` counts it in `failed` and `errors`. The fetch and the rebuild continue.
+4. **The SDK checks each document.** Each document needs `doc`, `kind`, `title`, `text`, `url` and `meta`, and `meta` must be JSON. Two documents with one id, the prefix `archive:` and the kind `archived` throw an error. The archived fallback uses that prefix and that kind.
+5. **A held-back body stays held back.** Set `ai_input: "no"` when the body asks not to be AI input. An extractor cannot lift a `no` of the capture.
+
+### The rules of the capture ids
+
+- The crawler makes the capture id, not the extractor. The id is a hash of the source, the kind, the external id and the payload of the capture record.
+- The payload holds the URL, the name of the recipe, the outcome, the hash of the body and the time of the fetch.
+- Thus the name of a recipe is part of each capture id. Do not change the name after the first capture. With a new name, `reindex` finds no extractor for the old captures, and it keeps their documents as they are.
+- Each version of a document names its capture. A citation names that capture, and `read`, `verify` and `extract` take `capture` to use that version.
+- `reindex` reads each capture again with the extractor. A pure extractor thus gives the same versions with the same capture ids.
+- A new version of an extractor changes the documents at the next `reindex`, not the capture ids. Keep it able to read the old captures.
+- The scope and the labels of a fetch are not part of the capture id.
+
+### The documents, the time travel and the archived fallback
+
+- Give each document an id with a prefix of your own, for example `menu:`. The page reader uses `url:`. A recipe that gives a `url:` id adds versions to the document of that page, and `merge` chooses the current version.
+- Each capture of a recipe adds a version of each of its documents. `read`, `verify` and `extract` take `at`, and `diff` compares two versions.
+- With `fallback: "archive"`, a recipe of the mode `page` gets an archived copy after a refusal. The page reader reads the copy as the document `archive:<url>`, with `archived`.
+- The extractor of a recipe reads only the live captures. Thus its documents do not mix a live answer and an old copy of a third party. A recipe of the mode `api` takes no `fallback`.
+- When the extractor cannot read a stored copy, `runRecipe` asks the source one more time. Thus a bad answer (a page about maintenance) does not stay until `maxAge` ends.
+
 ## Find the pages of a site
 
 `discover` finds the pages of a site from its sitemaps and its feeds. It fetches the files that list the pages, not the pages.