polite-crawler 0.1.1 → 0.1.3

package/README.md

@@ -12,7 +12,13 @@ The site has interactive demos with the real code: <https://mark1russell7.github
 - **Fetch one time.** Each response goes into a capture log and a store of bodies by hash. A repeated question reads the disk, so it costs no request.
 - **Read the cheap way.** The crawler reads JSON-LD, the visible text and the links with no browser. It opens a headless browser only for a page that shows almost nothing without JavaScript.
 - **Cite exact text.** Full-text search (SQLite FTS5) gives ranked passages. A quote check finds the exact span in the stored text, with the capture id to cite.
-- **An MCP server.** One command gives Claude and other MCP clients four tools: fetch, find, read and verify.
+- **An MCP server.** One command gives Claude and other MCP clients nine tools. They fetch, find, read and verify pages, check field values, discover and crawl a site, compare versions and list the watched pages.
+- **Web archives.** An export writes the stored captures as WARC or WACZ. Archive tools read them, for example `warcio`, pywb and ReplayWeb.page.
+- **Find and crawl a site.** `discover` lists the pages of a site from its sitemaps and feeds. `crawl` follows the links of a site within a budget, at the pace of the site.
+- **Watch and compare.** `watch` checks pages again on a schedule and tells what changed. `diff` compares two versions, and `read` takes `at`: the page as it was at a time.
+- **Archived copies, on request.** When a site refuses a page or is down, `fallback: "archive"` gives a copy from the Wayback Machine or Common Crawl. The result marks the copy as archived, with its date.
+- **Signed requests.** With an Ed25519 key, the crawler signs each request (Web Bot Auth). A site can then verify that a request comes from your crawler.
+- **Recipes.** `readVenue` reads the opening hours of a venue from its site, and `readProduct` reads a product page. `defineRecipe` makes a recipe of your own.
 
 ## Install
 
@@ -29,6 +35,8 @@ npx playwright-core install chromium-headless-shell
 
 Without the browser, the crawler reads each page as plain HTML. It checks for the browser one time, so a page that needs JavaScript gets a note at once, with no wait.
 
+Hybrid search (search by meaning and by words) is optional too. It uses a local model: transformers.js in the process (`npm install @huggingface/transformers`), or a local Ollama server. [Hybrid search](#hybrid-search) tells how.
+
 The package is ESM only, and it includes its TypeScript declarations. It has these entry points:
 
 | Entry point | What it gives | In a browser |
@@ -51,6 +59,7 @@ npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
 npx polite-crawler find "robots exclusion"
 npx polite-crawler read "url:https://en.wikipedia.org/wiki/Web_crawler" --length 2000
 npx polite-crawler verify "url:https://en.wikipedia.org/wiki/Web_crawler" "robots.txt protocol"
+npx polite-crawler extract "url:https://en.wikipedia.org/wiki/Web_crawler" --field "name=Web crawler"
 ```
 
 The store is in `~/.polite-crawler`. Use `--dir` or `POLITE_CRAWLER_DIR` for a different folder. A path that starts with `~` is in your home folder, and a relative path starts at the working folder. Use `--json` for output that a program reads. `npx polite-crawler --help` shows all the options.
@@ -58,15 +67,48 @@ The store is in `~/.polite-crawler`. Use `--dir` or `POLITE_CRAWLER_DIR` for a d
 - `--allow-private` (or `POLITE_CRAWLER_ALLOW_PRIVATE=1`) lets the crawler fetch loopback, private and link-local addresses. By default, the crawler refuses them.
 - `read --user-provided` reads a page that asked not to be AI input. Use it only for a URL that you gave yourself. The crawler records each such read.
 - `--max-age 0` fetches a page again, also when the store has a copy.
+- `fetch --archive-fallback` gives an archived copy when a site refuses a page or is down. [Archived copies after a refusal](#archived-copies-after-a-refusal) tells the rules.
+- `--sign-key <file>` and `--signature-agent <url>` sign each request. `key-directory --sign-key <file>` prints the key directory to host. [Signed requests](#signed-requests-web-bot-auth) tells how.
 - `find --rank fast` makes a long question faster, and its results can be different. [Fast ranking](#fast-ranking) gives the measured difference.
+- `find --rank hybrid --embedder transformers` (or `--embedder ollama`) also finds the pages with a similar meaning. `embed --embedder <name>` gives vectors to each stored page first. [Hybrid search](#hybrid-search) tells the options.
 - `reindex` builds the index again. The documents of a recipe that the command cannot read (the recipes of another application) stay. The output counts them, and the captures that it skipped.
+- `export --out <file>` writes the stored captures as a WARC or a WACZ file. [Export to a web archive](#export-to-a-web-archive) tells what goes into the file.
+- `extract <doc> --field name=value[:kind]` checks the values that you propose for the fields of a stored page. Repeat `--field` for each field. `--capture` checks the text of one capture, and `--at` the version that was current at a time. [Check field values against a page](#check-field-values-against-a-page) tells the statuses.
+
+To find the pages of a site, or to fetch a part of a site, use `discover` and `crawl`:
+
+```sh
+npx polite-crawler discover https://example.com/
+npx polite-crawler crawl https://example.com/ --max-pages 50 --max-depth 2 --budget 2m
+```
+
+- `discover` prints each URL of the sitemaps and feeds of the site, with its `lastmod`. `--max-urls` limits the list (the default is 1000), and `--feed <url>` reads one more feed.
+- `crawl` prints each page when its fetch ends, and then a summary. The options are `--max-pages` (default 100), `--max-depth` (default 3) and `--budget` (default `10m`).
+- `crawl` also takes `--include` and `--exclude` (patterns, which you can repeat), `--sitemaps`, `--render` and `--max-age`.
+- `discover` and `crawl` take `--allow-host <host>` for more hosts. The first `Ctrl+C` stops a crawl, and the summary prints.
+
+To read a page as it was, to compare two versions, or to watch pages, use `read --at`, `diff` and `watch`:
+
+```sh
+npx polite-crawler read "url:https://example.com/" --at 2026-10-01T12:00:00Z
+npx polite-crawler diff "url:https://example.com/" --from 2026-10-01 --to fetch.5d41402abc4b
+npx polite-crawler watch add https://example.com/hours --every 6h
+npx polite-crawler watch run --budget 2m
+npx polite-crawler watch list
+```
+
+- `read --at` and `verify --at` use the version that was current at that time. [Read a page as it was at a time](#read-a-page-as-it-was-at-a-time) gives the rules.
+- `diff` prints the changed lines as a unified diff. A changed line marks its words with `[-removed-]` and `{+added+}`. `--from` and `--to` take a capture id or a time.
+- `watch add` takes `--every` (1 hour or more, `24h` by default), `--confirm 1` or `2`, `--render`, `--scope` and `--label <name=value>`.
+- `watch run` takes `--budget` (default `10m`) and `--max-pages` (default 100). It prints each check, then a summary. A scheduler of the operator starts it.
+- `daemon` checks the watch list on a schedule, until you stop it. With `--mcp-http`, it also serves the MCP tools over local HTTP. [The daemon](#the-daemon) tells how.
 
 The command gives these exit codes:
 
 | Code | Meaning |
 | --- | --- |
 | `0` | The command did its work. |
-| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. |
+| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. `discover` gives 1 when it finds no URL, and `crawl` when it gets no page. `read --at` and `verify --at` give 1 for a time before the first capture. `watch run` gives 1 when a check is `refused` or an `error`, or when entries stay due. `watch remove` gives 1 for a URL that is not on the list. With `--archive-fallback`, a page with an archived copy counts as done. `extract` gives 1 when a field is not `verified` or `normalized`. `embed` gives 1 when its budget stops it before each page has vectors. |
 | `2` | The command line is wrong. The command opens no store. |
 
 ## Use it from Claude (MCP)
@@ -107,17 +149,26 @@ claude mcp add polite-crawler -- npx -y -p polite-crawler -p playwright-core pol
 
 When a new version of `playwright-core` comes out, install its browser again with the first command.
 
-The server gives four tools:
+The server gives nine tools:
 
 | Tool | What it does |
 | --- | --- |
-| `crawl_fetch` | It fetches pages into the store, and a page in the store answers with no request. It gives the age of each copy. |
-| `crawl_find` | It searches the stored pages and gives ranked passages. |
-| `crawl_read` | It gives a range of a stored document, with the capture ids to cite and the position of the next range. |
-| `crawl_verify` | It finds a quote in a stored document and gives the exact span. |
+| `crawl_fetch` | It fetches pages into the store, and a page in the store answers with no request. It gives the age of each copy. With `archive_fallback`, a page that the site refuses gets an archived copy, marked `archived`. |
+| `crawl_find` | It searches the stored pages and gives ranked passages. When the server has an embedder (`--embedder`), it also takes `rank: "hybrid"`, and each hit gives its `passage`. |
+| `crawl_read` | It gives a range of a stored document, with the capture ids to cite and the position of the next range. With `at`, it reads the version that was current at that time. |
+| `crawl_verify` | It finds a quote in a stored document and gives the exact span. It takes `at` too. |
+| `crawl_extract` | It checks the values that the assistant proposes for the fields of a stored page, and gives a status and a span for each field. It does not make values. With `at`, it checks the version that was current at that time. |
+| `crawl_discover` | It lists the page URLs of a site from its sitemaps and feeds, 1,000 or fewer. It fetches no page of the list. |
+| `crawl_site` | It fetches a few pages of a site and follows their links on the site: 10 pages and 1 minute by default, and 50 pages and 5 minutes at most. It sends progress for each page. |
+| `crawl_diff` | It compares two stored versions of a document, by capture id or by time. The changed lines, the changed words and the changes of the metadata come in the envelope. |
+| `crawl_watch_list` | It lists the watched pages: each URL, the next check, the last result and a suspected change. It leaves out the scope and the labels. |
 
 The MCP server applies the robots.txt rules for `Claude-User` too, because an assistant asks for these pages.
 
+No tool adds a watch. A watch commits future requests to a site, so the operator adds it with `polite-crawler watch add`. An assistant can read the list (`crawl_watch_list`) and compare the versions of a watched page (`crawl_diff`).
+
+The daemon serves the same tools over local HTTP, with a bearer token (`polite-crawler daemon --mcp-http`). There, `crawl_site` and `crawl_discover` are MCP tasks: a client gets a task id at once, and gets the result later. [MCP over HTTP](#mcp-over-http) tells how.
+
 ### Web text is data, not instructions
 
 A page can contain text that tries to give orders to the assistant. Thus each tool gives the text of the pages in an envelope:
@@ -140,7 +191,7 @@ text (characters 0 to 4000 of 25000):
 
 When a page asks not to be AI input (`ai-input=no`), the server holds back its text. The tool `crawl_read` has the parameter `user_provided` for a URL that the user gave. A page can tell the assistant to set this parameter, so the server needs a signal from a person:
 
-1. When the client can ask the user (MCP elicitation), the server asks the user to confirm. The text comes back only when the user confirms.
+1. When the client can ask the user (MCP elicitation), the server asks the user to confirm. The text comes back only when the user confirms. When the question fails, for example at a timeout, the text stays held back. The opt-in of the operator does not answer for a user who does not answer.
 2. When the client cannot ask, the server releases the text only when the operator started it with `POLITE_CRAWLER_ALLOW_USER_PROVIDED=1` or `--allow-user-provided`.
 3. Else the text stays held back, and the note tells how to permit it.
 
@@ -155,6 +206,18 @@ The crawler records each release in the capture log.
 | `--allow-private` | `POLITE_CRAWLER_ALLOW_PRIVATE=1` | Fetch loopback, private and link-local addresses too. |
 | `--allow-user-provided` | `POLITE_CRAWLER_ALLOW_USER_PROVIDED=1` | Let `user_provided` work when the client cannot ask the user. |
 | `--no-browser` | | Do not open a headless browser. |
+| `--sign-key <file>` | `POLITE_CRAWLER_SIGN_KEY` | Sign each request with this Ed25519 key (PKCS#8 PEM or JWK). [Signed requests](#signed-requests-web-bot-auth) tells how. |
+| `--signature-agent <url>` | `POLITE_CRAWLER_SIGNATURE_AGENT` | Where a site finds the public key of `--sign-key`: an https origin, or the URL of a JSON Web Key Set. |
+| `--embedder <name>` | `POLITE_CRAWLER_EMBEDDER` | The local model of hybrid search: `transformers` or `ollama`. `crawl_find` then takes `rank: "hybrid"`. |
+| `--embed-model <name>` | `POLITE_CRAWLER_EMBED_MODEL` | The model. The defaults are `Xenova/bge-small-en-v1.5` (transformers) and `nomic-embed-text` (Ollama). |
+| `--ollama-url <url>` | `POLITE_CRAWLER_OLLAMA_URL` | The Ollama server. The default is `http://127.0.0.1:11434`. |
+| `--allow-remote-ollama` | `POLITE_CRAWLER_ALLOW_REMOTE_OLLAMA=1` | Permit an Ollama server on a different computer. The page text goes to that server. |
+
+For hybrid search with transformers.js, add the package to the `npx` command, as for the browser:
+
+```sh
+claude mcp add polite-crawler -- npx -y -p polite-crawler -p @huggingface/transformers polite-crawler mcp --embedder transformers
+```
 
 ## Use it as a library
 
@@ -183,9 +246,13 @@ await crawler.close();
 - A result of `fetch` gives `capture`, `fetched_at` and `from_cache`. A result of `read` gives the `capture` and the `fetched_at` of its text.
 - `verify` gives the `capture`, the `blob` and the `fetched_at` of the text that holds the quote, with the positions in that text. `stale` is true when only an earlier capture holds the quote.
 - `read` and `verify` take `capture`: they then use the text of that capture, not the current text. A citation of a capture thus stays true after the page changes.
+- `extract` checks the values that you propose for the fields of a page, and gives the span and the capture of each value. [Check field values against a page](#check-field-values-against-a-page) tells how.
 - `versions(doc)` gives the versions of a document, the newest first.
+- `read` and `verify` take `at`: the version that was current at that time. [Read a page as it was at a time](#read-a-page-as-it-was-at-a-time) tells the rules.
+- `diff(doc, { from, to })` compares two versions. [Compare two versions](#compare-two-versions) tells what it gives.
+- `watch` is the watch list: `add`, `remove`, `list` and `run`. [Watch pages](#watch-pages) tells the rules.
 - `find` reads a query syntax. Text in double quotes is a phrase, and `bake*` is a prefix. The documents that hold each word come first. Case, accents and Unicode forms do not change a match, and Chinese, Japanese and Thai words match too.
-- `find` takes `rank`. The default is `exact`, and `fast` makes a long question faster. [Fast ranking](#fast-ranking) tells the difference.
+- `find` takes `rank`. The default is `exact`, and `fast` makes a long question faster. [Fast ranking](#fast-ranking) tells the difference. `hybrid` also finds by meaning, with the option `embedder` of `openCrawler`. [Hybrid search](#hybrid-search) tells how.
 - A page that asks not to be AI input is held back in each result. `fetch` gives no title, and `read` gives no title, no text and only the structural metadata. `verify` gives `found: false`, and `find` leaves the page out. Only `userProvided` lifts the hold, and the crawler logs it.
 - The hold applies to each version too. When a later capture does not ask it, the current text is not held back, but the earlier versions that asked it stay held back. `verify` with no `capture` does not examine them, `read` and `verify` of their capture hold them back, and `versions` shows no title for them.
 
@@ -233,15 +300,17 @@ The root entry point also exports the parts of the runtime, for an application t
 - `catchUpIndex(runtime)`: the step of `openCrawler` at open. It rebuilds a stale index, and gives an index that is behind the log the captures that it missed. Use it after `createRuntime`.
 - `Crawler`, `CaptureLog`, `DocIndex`, `FileBlobStore`, `SqliteHostStateStore`, `Extractors`, `fetchAndIndex`, `fetchPage` and `reindex`.
 - `readDocument`, `verifyQuote`, `findDocuments`, `isHeldBack`, `heldBackMeta` and `recordOverride`: the read, the verification and the rule for a held-back page, over a runtime.
+- `extractDocument` and `checkFields`: the check of field values over a runtime, and over a text and its metadata with no store.
 - `extractPage`, `typesOf`, `decodeBody` and `bodyEncoding`: the page reader and the decoder of a body in its declared encoding.
 - `readPdf` and `isPdf`: the PDF reader. `PdfOptions` sets limits on the pages, the characters, the time and the inflated bytes.
-- `fromText`, `fromSchemaOrg`, `fromCompact` and `describeWeek`: the opening-hours reader.
+- `fromText`, `fromSchemaOrg`, `fromCompact` and `describeWeek`: the opening-hours reader. `compareHours` compares a reading with a week that you keep.
 - `parseRobots` and `decide`: the robots.txt parser and matcher.
 - `RenderRefused`: the error of a renderer that did not render a page, with the reason `refused`, `download`, `scheme` or `browser`. A crawler does not count it as a failure of the host. After the reason `browser`, the crawler renders no more pages.
 
 The readers give more facts than before:
 
 - `PageExtract` has `canonical` (the canonical URL on the same site), `redirect` (the target of a quick `<meta http-equiv="refresh">`), `blocked` (the name of a bot check or a firewall that sent its page in place of the content) and `notes` (what the reader left out).
+- `PageExtract` also has `feeds` (the RSS and Atom feeds that the page links). `extractPage` always gives it. The field is optional, so a `PageExtract` of 0.1.1 stays correct.
 - `Hours` has `partial` (the reader left out a rule or a line) and `season` (the season or the dates of the hours). The text reader reads English only.
 
 ### The options of `Crawler`
@@ -254,9 +323,10 @@ The readers give more facts than before:
 - `onEvent`: a function that gets an event for each request, response, wait, stop and skip.
 - `obeyDefaultToken`: a crawler with a token of your own also obeys the group of `PoliteCrawler` when its `User-Agent` links the bot page of polite-crawler. Site owners read that page. The stricter rules apply. Set false when you host a bot page of your own.
 - `maxRenders`: the pages that the browser renders at one time. The default is 2, because each render holds a browser context. A render waits for a slot before the lease of its host, so the wait is not time of the host.
+- `signer`: the signer of each request (Web Bot Auth). [Signed requests](#signed-requests-web-bot-auth) tells its options.
 - `scope`, `labels` and `scopeRank`: the labels of each record. One fetch of a robots.txt can serve requests of different scopes. Its record takes the labels of each request and the scope with the highest `scopeRank`, the most private one.
 
-`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent` and `extractors`. Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.
+`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent`, `extractors` and `recipes` ([the recipes of an application](#writing-a-recipe)). Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.
 
 `FetchRequest.maxAgeMs` can be `"auto"`: the crawler then uses the freshness that the server gave (`Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`). `crawler.resetHosts(host?)` removes the stops and the failure counts of one host or of each host, for a process that runs for a long time.
 
@@ -266,14 +336,951 @@ Before version 1.0.0, a minor version can change the API, and `CHANGELOG.md` tel
 
 These parts are the stable API:
 
-- `openCrawler` and the methods of `PoliteCrawler`: `fetch`, `find`, `read`, `verify`, `versions`, `reindex` and `close`.
-- The options and the results of these methods: `OpenOptions`, `FetchOptions`, `PageResult`, `Found`, `ReadResult`, `VerifyResult`, `DocVersion` and `ReindexResult`.
-- `polite-crawler/mcp`: `createMcpServer`, `startMcpServer` and the four tools.
+- `openCrawler` and the methods of `PoliteCrawler`: `fetch`, `find`, `read`, `verify`, `versions`, `discover`, `crawl`, `reindex`, `close`, `diff`, `watch` and `extract`. `openCrawler` gives an `OpenedCrawler`: a `PoliteCrawler` that has `discover`, `crawl`, `diff`, `watch` and `extract`. The five are optional in `PoliteCrawler`, so an object of your own that implements it stays correct.
+- The options and the results of these methods: `OpenOptions`, `FetchOptions`, `PageResult`, `Found`, `ReadResult`, `VerifyResult`, `DocVersion` and `ReindexResult`. `at` of `ReadOptions` and `VerifyOptions`, and `at` and `before_first` of their results, are stable too.
+- The options and the results of `extract`: `ExtractOptions`, `ExtractResult`, `ExtractSummary`, `FieldClaim`, `FieldKind`, `FieldStatus`, `FieldResult`, `FieldSpan` and `FieldCandidate`, with the names of the normalizations.
+- The diff: `diffDocument`, `diffTexts`, `DIFF_LIMITS`, `DiffOptions`, `DiffResult`, `DiffSide`, `DiffSummary`, `DiffHunk`, `DiffLine`, `WordPart`, `MetaChange`, `VersionChoice`, `DiffLimits`, `DiffInput` and `TextDiff`.
+- The time travel: `versionAtTime`, `isoTime` and `VersionAt`.
+- The watch list: `WatchList`, `WatchOptions`, `WatchEntry`, `WatchPending`, `WatchKind`, `WatchRun`, `WatchRunOptions`, `WatchEvent`, `WatchSummary`, `WatchStop`, `WATCH_LIMITS` and the file `watch.sqlite` (`WATCH_SCHEMA`). For a runtime of your own, it also has `WatchStore`, `WatchStoreOptions`, `runWatch` and `WatchRunnerOptions`. The lease steps of `WatchStore` (`claim`, `release` and `settle`) are the engine.
+- The daemon: `startDaemon`, `scheduleWatches`, `changeCommand`, `readDaemonState`, `daemonStateFile`, `DAEMON_DEFAULTS`, their options and results, and the state file (`DAEMON_STATE_SCHEMA`). `polite-crawler/mcp` adds `startMcpHttpServer`, `mcpHttpHost`, `CrawlTaskStore`, `TASK_LIMITS`, `MCP_HTTP_DEFAULTS`, `sameToken` and `makeToken`.
+- The MCP tasks follow the task API of the MCP SDK, which the SDK marks as experimental. When the SDK changes that API, the tasks change with it, and `CHANGELOG.md` says so.
+- The options and the results of `discover` and `crawl`: `DiscoverOptions`, `DiscoverResult`, `DiscoveredUrl`, `DiscoverSource`, `DiscoverStop`, `CrawlOptions`, `CrawlRun`, `CrawlPage`, `CrawlSummary`, `CrawlStop`, `UrlPattern` and `ScopeOptions`.
+- `polite-crawler/mcp`: `createMcpServer`, `startMcpServer` and its tools.
 - The readers: `polite-crawler/robots`, `/hours`, `/html`, `/classify`, `/politeness` and `/signals`. The site and its tools use them in the browser.
 - The command line, its exit codes, the store on disk and `polite-crawler/record.schema.json`.
+- The export: `exportWarc` and `exportWacz`, their options (`ArchiveOptions`, `ArchiveFilter`, `ArchiveSource`), their results (`WarcResult`, `WaczResult`, `ArchiveCounts`) and the records of the archive.
+- The archived fallback: `fallback` and `archiveDespiteRobots` of `FetchOptions`, the field `archived` of the results, and the types `Archived` and `WebArchive`.
+- Signed requests: `createWebBotAuthSigner`, `keyDirectory`, `verifySignature`, `KEY_DIRECTORY_PATH`, `KEY_DIRECTORY_TYPE`, the option `signer`, and the types `WebBotAuthOptions`, `WebBotAuthSigner`, `RequestSigner`, `KeyDirectory`, `DirectoryKey`, `KeyInput`, `SignatureAgentType`, `SignedRequest` and `SignatureCheck`.
+- The headers of a signature follow the draft. When the draft changes, the headers change with it, and `CHANGELOG.md` says so.
+- Hybrid search: the option `embedder` of `OpenOptions`, the interface `Embedder` (the extension point) with `EmbedCallOptions`, and the adapters `transformersEmbedder` and `ollamaEmbedder` with their options (`TransformersEmbedderOptions`, `OllamaEmbedderOptions`, `Pooling`, `EmbeddingModel`).
+- Also stable: `HybridCrawler` (the result of `openCrawler`), `rank: "hybrid"` (`SearchRank`, `SearchOptions`), `embed`, `rankWithin` with `RankOptions` and `Ranked`, and `hybrid` (`HybridSearch`). The three members are optional in `PoliteCrawler`, and `OpenedCrawler` does not change.
+- The options and the results of hybrid search are stable too: `HybridOptions`, `HybridFindOptions`, `EmbedOptions`, `EmbedResult`, `EmbedStop`, `Passage` and `Found.passage`. So are `HYBRID_DEFAULTS`, `OLLAMA_MODELS`, `TRANSFORMERS_MODELS`, `NO_EMBEDDER`, `NO_OLLAMA` and `NO_TRANSFORMERS`.
+
+- The recipe SDK: `defineRecipe`, `runRecipe`, `registerRecipe`, `readJson`, the option `recipes` of `openCrawler` and `createRuntime`, and `RESERVED_RECIPES`. Its types are `Recipe`, `RecipeDefinition`, `RecipeExtract`, `RecipeCapture`, `RecipeRequest`, `RecipeRequestOptions`, `RecipeRunOptions`, `RecipeResult`, `RecipeDoc` and `RecipeTarget`.
+- The recipes: `readVenue`, `venueReadingsOf`, `compareHours`, `readProduct` and `productOf`. Their options and results are stable too: `VenueOptions`, `VenueResult`, `VenueReading`, `VenuePage`, `VenueLink`, `VenueLocation`, `VenueOsm`, `VenueSource`, `VenueRead`, `HoursComparison`, `ProductOptions`, `ProductResult`, `ProductRead`, `ProductVariant` and `Money`.
 
 The other exports of the root entry point are the engine. Examples are `createRuntime`, `catchUpIndex`, `Crawler`, `CaptureLog`, `DocIndex`, `FileBlobStore`, `SqliteHostStateStore`, `Extractors`, `fetchAndIndex` and `makeRecord`. An application can build on them. A minor version before 1.0.0 can change them, and its section in `CHANGELOG.md` says so.
 
+The engine of the check of fields is `checkFields`, `extractDocument`, `FIELD_KINDS`, `FieldCheck`, `ExtractSource` and `FieldsChecked`.
+
+The engine of discovery and of the crawl is `discoverSite`, `crawlSite`, `readSitemap`, `feedDate`, `DISCOVER_LIMITS`, `CRAWL_LIMITS`, `SITEMAP_LIMITS` and `DEADLINE_NOTE`. It also includes `Crawler.robotsAllow` and the `deadline` of `FetchRequest`, `RenderRequest` and `PageOptions`.
+
+The engine of the time travel and of the watch list is the optional method `ReadingIndex.versionAt`, `DocIndex.versionAt` and `Crawler.hostStop`.
+
+The engine of hybrid search is `createHybridSearch`, `HybridSearchOptions`, `VectorStore`, `VECTORS_SCHEMA` and the types of the vector store. It also includes `chunkText`, `chunkTokensOf`, `estimateTokens`, `CHUNK_LIMITS`, `fuseRanks`, `unitVector`, `isLoopback`, `FeatureExtractor`, `TransformersModule`, the optional method `ReadingIndex.digests` and `DocIndex.digests`. The file `vectors.sqlite` is a projection, as the index is: it is not part of the stable store.
+
+The engine of the recipes is `rankHoursLinks`, `namesVenue`, `locationMatches`, `pickProduct`, `pickVariant`, `readAmount`, `VENUE_LIMITS`, `PRODUCT_LIMITS`, `RECIPE_LIMITS` and the `steps` of `VenueRead` and `ProductRead`. The words that rank a link or a label are the engine too, so a version can improve them.
+
+The engine of the archived fallback and of the signatures is `archiveExtractor`, `archivedDocId`, and the `fallback` and `archiveDespiteRobots` of `PageOptions`. It also includes `range`, `signalLines` and `originalHeaderPrefix` of `FetchRequest`, `ai_input` of `ExtractedDoc`, `sign` of `RenderOptions` and `signer` of `CrawlerOptions`.
+
+## Hybrid search
+
+Hybrid search finds the pages that hold the words of a question, and also the pages with a similar meaning. It is opt-in: the default ranking stays exact BM25.
+
+```ts
+import { openCrawler, transformersEmbedder } from "polite-crawler";
+
+const crawler = openCrawler({ dir: "./crawl", embedder: transformersEmbedder() });
+await crawler.embed({ budget: "10m" }); // vectors for each stored page
+
+for (const hit of await crawler.find("a loaf with a dark crust", { rank: "hybrid" })) {
+  console.log(hit.doc, hit.score, hit.passage, hit.snippet);
+}
+```
+
+- `rank: "hybrid"` needs an embedder. Without one, the search rejects with `NO_EMBEDDER`. It does not change to BM25 in silence.
+- `crawler.rankWithin(q, docs, { rank: "hybrid" })` ranks a set of stored documents in the same way.
+- `openCrawler` gives a `HybridCrawler`: an `OpenedCrawler` with `hybrid`, `embed` and `rankWithin`, and a `find` that takes `rank: "hybrid"`.
+- The command line has `--embedder`, `find --rank hybrid` and `embed`. The MCP tool `crawl_find` takes `rank: "hybrid"` when the server has an embedder.
+
+```sh
+npx polite-crawler embed --embedder ollama --budget 30m
+npx polite-crawler find "a loaf with a dark crust" --rank hybrid --embedder ollama
+```
+
+### transformers.js
+
+This embedder uses the model in the process of the crawler. The package `@huggingface/transformers` is an optional peer, and the embedder loads it at its first text.
+
+```sh
+npm install @huggingface/transformers
+```
+
+```ts
+import { transformersEmbedder } from "polite-crawler";
+
+const embedder = transformersEmbedder({ model: "Xenova/bge-small-en-v1.5", quantized: true });
+```
+
+- The default model is `Xenova/bge-small-en-v1.5`: 384 dimensions and an input of 512 tokens. The 8-bit model (`quantized: true`, the default) is approximately 34 MB.
+- The first use downloads the model from the Hugging Face Hub (`huggingface.co`). This download does not go through the polite crawler, so robots.txt, the pace and the capture log do not apply to it.
+- transformers.js keeps the files in `cacheDir`. Its default is `node_modules/@huggingface/transformers/.cache/`. The command line uses `<store>/models`. With `localOnly: true`, the embedder uses only the cache and downloads nothing.
+- The embedder pools the vectors of the tokens as the model expects: `cls` for BGE and `mean` for the other models. The option `pooling` changes it.
+
+### Ollama
+
+This embedder sends the texts to a local Ollama server (`POST /api/embed`). The package has no dependency for it.
+
+1. Install Ollama from <https://ollama.com>, and start it with `ollama serve`.
+2. Pull a model: `ollama pull nomic-embed-text`.
+
+```ts
+import { ollamaEmbedder } from "polite-crawler";
+
+const embedder = ollamaEmbedder({ model: "nomic-embed-text", baseUrl: "http://127.0.0.1:11434" });
+```
+
+| Model | Size | Dimensions | Input in tokens |
+| --- | ---: | ---: | ---: |
+| `nomic-embed-text` (the default) | approximately 274 MB | 768 | 8,192 |
+| `all-minilm` | approximately 46 MB | 384 | 256 |
+| `mxbai-embed-large` | approximately 670 MB | 1,024 | 512 |
+
+- The embedder sends 32 texts in each request (`batch`). It adds the prefixes that `nomic-embed-text` and `mxbai-embed-large` expect.
+- An error tells what to do: `ollama serve` when no server answers, and `ollama pull <model>` when the model is not there.
+- The embedder accepts only a server on this computer (a loopback address). `allowRemote: true` (`--allow-remote-ollama`) permits a different computer.
+
+### An embedder of your own
+
+An application can give an embedder of its own, as it can give a renderer:
+
+```ts
+import type { Embedder } from "polite-crawler";
+
+const embedder: Embedder = {
+  id: "my-model-v1", // the model and its version: the store keeps it with each vector
+  dimensions: 384,
+  maxInputTokens: 512,
+  embed: async (texts, { signal, purpose }) => myModel.embed(texts, { signal, purpose }),
+};
+```
+
+`purpose` is `"document"` for the chunks of a page and `"query"` for a question. Give a new `id` when the vectors of the model change.
+
+### What goes where
+
+- The embedder gets the text of the stored pages and the words of each question. With transformers.js, the text stays in the process. With Ollama, the text goes to the Ollama server.
+- A page that asks not to be AI input does not go to the embedder, and no hybrid result gives it. The rule applies to each version, as for `find`.
+- Only the download of a transformers.js model goes to the internet, to the Hugging Face Hub.
+
+### How a hybrid search ranks
+
+1. Full-text search gives the BM25 candidates (`candidates`, 50 by default). A candidate whose text has no vectors gets them first.
+2. The vector search gives the documents with the closest chunks (`neighbors`, 50 by default). It compares the vector of the question with each stored vector (a brute-force cosine).
+3. Reciprocal rank fusion makes one list. Each document gets the sum of 1 / (k + rank) over the two lists, with the rank from 1. The constant k is `fusionK`, 60 by default.
+
+- The snippet and the `passage` of each hit come from its best chunk. `passage.start` and `passage.end` are positions in the text of the capture of the hit, for `read`.
+- A search gives vectors only to its candidates. Thus a page with the meaning and none of the words comes back only after `embed` gave it vectors.
+- `embed` takes `budget` (`10m` by default), `maxDocs` and `kind`. Start it again to continue: it does not send a text with vectors again.
+- `find` and `rankWithin` take `candidates`, `neighbors`, `fusionK` and `signal`. `HYBRID_DEFAULTS` has the defaults.
+
+### Chunks and storage
+
+- A chunk is a window of whole sentences. It ends at the end of a paragraph when it can, and a longer sentence is cut at its spaces.
+- A chunk has the input limit of the model in tokens with a margin of 10 percent, and 512 tokens at most. Two chunks next to each other share approximately 10 percent of their tokens. A text gives 64 chunks at most.
+- The vectors go into `vectors.sqlite` in the store. The key of a row is the SHA-256 of the text, the number of the chunk and the `id` of the embedder. Thus a reindex keeps each vector, and the same text gets its vectors one time for each model.
+- The file is a projection, as `index.sqlite` is. Delete it, and `embed` makes the vectors again.
+- A vector takes 4 bytes for each dimension: 1.5 KiB for 384 dimensions and 3 KiB for 768. With the keys and the indexes of SQLite, a chunk of 384 dimensions took 2.4 KiB of the file in the benchmark.
+- The first hybrid search reads the vectors of the embedder into memory. Thus the memory is approximately the size of the vectors.
+
+The benchmark `bench/find.mts --hybrid` measured the time in CI ([the results](https://github.com/mark1russell7/crawler/actions/runs/39082851222)). A fake embedder of almost no cost gave the vectors, so the time of a real model comes on top. The index had 30,000 documents of 900 words, which gave 89,100 chunks of 384 dimensions. Each time is the median of 12 questions on Node.js 24:
+
+| Step | Time |
+| --- | ---: |
+| The cosine of a question and each of the 89,100 vectors | 82 ms |
+| `find`, `rank: "hybrid"`, 1 word | 88 ms |
+| `find`, `rank: "hybrid"`, 16 words | 122 ms |
+| `rankWithin` of 40,000 ids, `rank: "hybrid"`, 16 words | 528 ms |
+| The first search: it reads 131 MiB of vectors into memory | 714 ms |
+| `embed` of the 30,000 documents, with no time for a model | 17 s |
+
+- The brute-force cosine is linear in the number of chunks. It is a good fit up to approximately 100,000 chunks (approximately 30,000 pages of this size). A search then takes approximately 0.1 s, and the vectors take approximately 150 MiB of memory at 384 dimensions.
+- One million chunks take approximately 1 s for each search and 1.5 GiB of memory. A larger store needs an index of the vectors, and the package does not have one.
+- `vectors.sqlite` was 210 MiB for the 89,100 chunks.
+
+## Read a page as it was at a time
+
+Each capture of a document is a version, and the index keeps each version. `read` and `verify` take `at`, a time. They then use the version that was current at that time.
+
+```ts
+const then = await crawler.read("url:https://example.com/", { at: "2026-10-01T12:00:00Z" });
+console.log(then.capture, then.fetched_at, then.before_first, then.text);
+
+const check = await crawler.verify("url:https://example.com/", "open on Sundays", { at: new Date("2026-10-01") });
+```
+
+- `at` is an ISO 8601 text or a `Date`. A date with no time is the start of that day in UTC.
+- The rules that choose the current version apply to the versions of that time. Thus a render stays current while the plain body under it stays the same, and the richer recipe wins.
+- The result names the `capture` and the `fetched_at` of the version, and it gives `at`.
+- When `at` is before the first capture, no version answers. `before_first` is true, and the note gives the time of the first capture. `read` then gives no text, and `verify` gives `found: false`.
+- `verify` with `at` examines that version only, as `verify` with `capture` does. A match is not `stale`.
+- Give `capture` or `at`, not the two. The two together are an error.
+- The hold of a page that asked not to be AI input applies as before. A version that was held back at that time stays held back. When the document is held back at this time, each of its versions is held back too.
+- The command line has `read --at <time>` and `verify --at <time>`. The MCP tools `crawl_read` and `crawl_verify` have `at`.
+
+`find` has no `at`. The search index (FTS5) holds the current text of each document. A search at a time needs an index entry for each version, or a scan of each stored text. Neither is cheap, so the package does not give it.
+
+## Compare two versions
+
+`diff` compares two versions of a stored document. It sends no request.
+
+```ts
+const diff = await crawler.diff("url:https://example.com/");
+console.log(diff.from.capture, diff.to.capture, diff.summary);
+
+const since = await crawler.diff("url:https://example.com/", { from: { at: "2026-10-01" }, to: "fetch.5d41402abc4b" });
+```
+
+- `from` and `to` are a capture id, or `{ at }`: the version that was current at that time. The default `to` is the newest version. The default `from` is the version before `to` of the same recipe and kind (plain or rendered). Thus a render is not compared with the plain body under it.
+- `hunks` give the text as a line diff, in the form of a unified diff, with 2 lines of context. A `changed` line pairs a removed line with an added line. Its `words` give the words that stay, the added words and the removed words.
+- `meta` gives the changes of the title, the URL and the metadata at their paths, for example `meta.jsonld[0].openingHours`. Each change has the JSON of the old and the new value, cut to 300 characters.
+- `summary` counts the added, removed and changed lines, the added and removed words, and the changes of the metadata. `same` is true when the two versions are the same.
+- The command line has `diff <doc> [--from <version>] [--to <version>]`. A version is a capture id or a time. The MCP tool `crawl_diff` gives the changes in the envelope.
+
+The line diff uses the algorithm of Myers. Its time is linear when the change is small. `DIFF_LIMITS` gives the bounds:
+
+| Limit | Value | What it bounds |
+| --- | ---: | --- |
+| `maxChars` | 1,000,000 | The characters of each text. A longer text gives its first part. |
+| `maxLines` | 50,000 | The lines of each text. A longer text gives its first lines. |
+| `maxSteps` | 1,000,000 | The steps of the line diff. After them, the rest of the change is one removal and one addition. |
+| `maxWordSteps` | 1,000,000 | The steps of the word diffs of all the changed lines together. |
+| `maxLineWords` | 5,000 | The words of one line. A longer line gets no word diff. |
+| `maxHunkLines` | 2,000 | The lines that the hunks show. The summary counts each line. |
+| `maxMetaChanges` | 200 | The changes of the metadata that the result shows. The summary counts each change. |
+
+- When a limit applies, the note tells which limit. `exact` is false when the line diff is not minimal or a changed line has no word diff. `truncated` is true when the result leaves out a part.
+- `steps` counts the steps of the diff. The tests use it, because a count of steps does not change with the computer.
+- `limits` in the options changes the bounds of one diff.
+- When a side is held back, or the document is held back, the diff gives no text and no title. It compares only the structural metadata, and `same` is null.
+
+## Watch pages
+
+A watch list keeps the pages that the crawler checks again on a schedule. The list is in `watch.sqlite` in the store.
+
+```ts
+await crawler.watch.add("https://example.com/hours", { every: "6h", labels: { lineage: "venue-17" } });
+
+const run = crawler.watch.run({ budgetMs: 120_000, maxPages: 50 });
+for await (const event of run) console.log(event.kind, event.suspected, event.url, event.diff);
+console.log((await run.summary).due);
+
+console.log(await crawler.watch.list());
+await crawler.watch.remove("https://example.com/hours");
+```
+
+| Option of `add` | Default | What it does |
+| --- | --- | --- |
+| `every` | `24h` | The time between two checks. The floor is `1h`, so a watch cannot hammer a site. `watchFloor` of `openCrawler` changes the floor. |
+| `confirm` | `2` | The number of checks that must see a change before it is confirmed. |
+| `render` | `auto` | As for `fetch`. |
+| `labels`, `scope`, `onBehalfOf` | none | As for `fetch`. They apply to each check. |
+
+`add` of a page on the list changes its options and keeps its state. A new entry is due at once.
+
+`run` checks the entries that are due, the first due first. It gives an async iterable of events, and a `summary`. Its options are `budgetMs` (10 minutes by default), `maxPages` (100 by default) and `signal`.
+
+- Each check is a fetch of the crawler with no stored copy (`maxAge: 0`). Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A stored `ETag` or `Last-Modified` date makes the check a conditional request.
+- A run sends requests to 8 hosts at one time at most, and to each host one request at a time.
+- When the budget ends before the request of a check, the entry stays due for the next run.
+- When a check finds a stopped host (a 403, a challenge, or a `Retry-After` of more than 1 hour), the entry is due again at the end of the stop, or later.
+
+Each event has the URL, the document, the capture of the check, the last confirmed capture (`previous`), the outcome and the kind of the check:
+
+| Kind | What the check saw |
+| --- | --- |
+| `new` | The first version of the page, or a page that came back after it was gone. |
+| `changed` | A version with a different title, text or JSON-LD than the last confirmed version. |
+| `unchanged` | The last confirmed version. |
+| `gone` | A 404 or a 410. |
+| `refused` | No version: robots.txt, the egress policy, a stop of the host or the site refused the request. |
+| `error` | No version: a server error, no answer, or a page that gives no document. |
+
+### The confirm rule
+
+One read can be a glitch: a page of an error, a test of the site, or a cache that is not current. Thus a change counts only when two checks see it:
+
+1. The first check that sees a change gives it with `suspected: true`. The last confirmed version does not change.
+2. The next check confirms the change when it sees the same new version. It gives `confirmed: true`, and the new version becomes the last confirmed version.
+3. When the next check sees the last confirmed version again, it gives `unchanged`, and the note says that the change did not repeat.
+4. When it sees a third version, that version is the new suspected change.
+
+A confirmation needs a new capture: a stored copy does not confirm a change. `gone` and a page that comes back follow the same rule. With `confirm: 1`, each change is confirmed at once. A `refused` or an `error` check changes no state.
+
+The `diff` of an event of the kind `changed` gives the counts of the diff from the last confirmed version. A page that asked not to be AI input gives no diff.
+
+### The list in the store
+
+- The list is the decision of the operator, not a projection of the capture log. Thus it has its own file, `watch.sqlite`. A deletion of the index or of `hosts.sqlite` does not lose it. The file opens at the first use of `crawler.watch`.
+- Each step is one SQLite transaction (`BEGIN IMMEDIATE`) in WAL mode, and it waits for the lock of a different process (`busy_timeout`).
+- A run claims each entry with a lease before its check. The lease holds until the end of the budget and 2 minutes more. Thus two processes that run at the same time check each entry one time. A process that stops holds its leases until they end.
+- The list stays after a restart, with the state of each entry: the last confirmed capture, the suspected change and the counts.
+- A scheduler of the operator (cron, a systemd timer or a job of an application) starts `watch.run`. Or the daemon starts it on a schedule: [The daemon](#the-daemon).
+- The command line has `watch add <url...>`, `watch remove <url...>`, `watch list` and `watch run`. The MCP tool `crawl_watch_list` lists the entries.
+- No MCP tool adds a watch. A watch commits future requests to a site, so it is an action of the operator, not of an assistant.
+
+## The daemon
+
+The daemon is a local process that runs for a long time. It checks the watch list on a schedule, and it can serve the MCP tools over local HTTP.
+
+```sh
+npx polite-crawler watch add https://example.com/hours --every 6h
+npx polite-crawler daemon --events ~/changes.jsonl --on-change "jq -c . >> ~/seen.jsonl"
+npx polite-crawler daemon status
+```
+
+```ts
+import { startDaemon } from "polite-crawler";
+
+const daemon = await startDaemon({
+  dir: "./crawl",
+  tick: "1m",
+  onChange: (event) => console.log(event.url, event.kind, event.capture),
+});
+console.log(daemon.status().next_due);
+await daemon.stop(); // or SIGINT and SIGTERM
+```
+
+### The schedule
+
+- Each tick is one `watch run`, with the budget and the maximum of checks of a tick. Thus each polite rule applies: robots.txt, the egress policy, one request at a time to each host, the stops of a host and `Retry-After`.
+- The next tick comes when the first entry is due, and one tick (`--tick`, `1m` by default) after the last tick at the latest.
+- A random jitter from 0 to `--jitter` (`10s` by default) comes after it. Thus a check does not come before its time.
+- When a tick stops at its budget (`--tick-budget`, `10m` by default) or at its maximum (`--tick-pages`, 100 by default), the entries that stay due wait for the next tick.
+- An entry that a different process holds waits for the end of its lease. Only one daemon operates on a store: a second daemon does not start.
+- `scheduleWatches(crawler.watch, options)` gives the same schedule over the watch list of a crawler of your own.
+
+### The changes
+
+A change is an event of `watch run` with `confirmed: true`: the last confirmed version of the page changed. The first version of a page is not a change.
+
+- The daemon writes each change as one JSON line, to the standard output or to `--events <file>`. The library takes `events` (a file or a function) and `onChange` (a function).
+- `--on-change <command>` starts your command for each change. The command gets the event as one JSON line on its standard input.
+- The commands run one at a time, in the order of the changes. A command that runs for more than 1 minute stops. A command that fails becomes the last error of the status.
+
+`--on-change` is a command of the operator. Obey these rules:
+
+- The daemon starts the command in the shell of the system, as the user of the daemon, with the environment of the daemon. Give only a command that you wrote or trust.
+- Do not take the command from a page, from an assistant or from a different user.
+- The daemon gives the event only on the standard input, not in the command line. Thus no text of a page goes into a shell command.
+- The event holds URLs and notes from sites. In your command, read the event as data, for example with `jq`. Do not give its text to `eval` or to a shell.
+
+### The stop
+
+- At `SIGINT` or `SIGTERM`, the daemon starts no new check. The checks in progress get `--grace` (`10s` by default) to end and write their state.
+- After the grace, the checks in progress abort, and the daemon releases their leases. The entries stay due for the next start. A second signal aborts the checks at once.
+- The daemon stops its MCP server and cancels its tasks. It writes the state file, closes the store and exits with code 0.
+- A daemon that stops with no clean stop (a crash, or a forced stop on Windows) keeps its leases until they end: the budget of the tick and 2 minutes.
+
+### The status
+
+`polite-crawler daemon status` reads the state file `<store>/daemon/state.json`. It sends no request, and it opens no store.
+
+- It shows the next due watches, the last tick, the next tick, the counts, the MCP endpoint, the tasks that run and the last error.
+- It exits with code 0 when a daemon operates on the store, and with 1 when no daemon operates. `--json` prints the JSON of the state file.
+- The daemon writes the state file after each tick and at each change of a task. The file is for the owner only.
+- With `--mcp-http`, `GET /status` with the bearer token gives the same JSON.
+
+### MCP over HTTP
+
+```sh
+npx polite-crawler daemon --mcp-http
+claude mcp add --transport http polite-crawler http://127.0.0.1:9307/mcp --header "Authorization: Bearer $(cat ~/.polite-crawler/daemon/mcp-token)"
+```
+
+- `--mcp-http` serves the nine tools over the Streamable HTTP transport of MCP, at `http://127.0.0.1:9307/mcp`. `--mcp-port` and `--mcp-host` change the address.
+- The tools, the envelope, elicitation and `--allow-user-provided` are the same as for `polite-crawler mcp`.
+- `crawl_site` and `crawl_discover` are MCP tasks (the task primitive of the MCP specification 2025-11-25). The task support is native: it uses the tasks of the MCP SDK.
+- A client that asks for a task gets a task id at once. It polls the status (`tasks/get`), gets the result (`tasks/result`) and can cancel the task (`tasks/cancel`). A cancel stops the requests in progress.
+- A client with no tasks gets the result in the call, with a progress notification for each page, as from `polite-crawler mcp`.
+- The status message of a task holds only counts, for example `3 of 10 done`, and no text of a page.
+- A task and its result stay for 1 hour. The server has 8 tasks in progress at most: a task above the limit fails at once, with the reason.
+- Each session has its own tasks. When a session closes, its tasks stop.
+
+The library has `startMcpHttpServer(crawler, options)` in `polite-crawler/mcp`. The daemon reaches it through a small seam, `DaemonOptions.mcpHost`, so a different MCP host can replace it.
+
+#### The token
+
+- Each request needs `Authorization: Bearer <token>`. The server compares the SHA-256 digests of the two tokens in constant time.
+- By default, the daemon makes a new random token at each start. It writes the token to `<store>/daemon/mcp-token`, for the owner only where the system has file modes. On Windows, the folder of the store in your profile protects it.
+- To keep one token across starts, write it to a file of your own and give `--mcp-token-file <file>`. The daemon reads the file and writes nothing. It warns when other users can read the file.
+
+```sh
+node -e "console.log(require('node:crypto').randomBytes(32).toString('base64url'))" > ~/.polite-crawler-token
+chmod 600 ~/.polite-crawler-token
+npx polite-crawler daemon --mcp-http --mcp-token-file ~/.polite-crawler-token
+```
+
+#### The address
+
+- The server binds to `127.0.0.1` by default, so only your computer can reach it.
+- On a loopback address, the server accepts only a loopback name in the `Host` header. An `Origin` header must name the same host. Thus a web page cannot reach the server by DNS rebinding.
+- A host that is not loopback, for example `--mcp-host 0.0.0.0`, needs `--allow-remote`. Without it, the command line is wrong.
+
+> **Warning:** With `--allow-remote`, each computer that reaches the port can try the token. The server speaks plain HTTP, so a computer on the path can read the token and the pages. The tools also fetch pages from the address of your computer. Use `--allow-remote` only on a private network or behind a TLS proxy of your own. Do not open the port to the internet.
+
+### Start the daemon as a service
+
+The package installs no service. These examples start the daemon at login, and they stop it with `SIGTERM`. Install the package one time (`npm install -g polite-crawler`), and change the paths.
+
+On Linux, with a user service of systemd:
+
+```ini
+# ~/.config/systemd/user/polite-crawler.service
+[Unit]
+Description=polite-crawler daemon
+After=network-online.target
+
+[Service]
+ExecStart=/usr/local/bin/polite-crawler daemon --events %h/.polite-crawler/changes.jsonl
+Restart=on-failure
+KillSignal=SIGTERM
+TimeoutStopSec=30
+
+[Install]
+WantedBy=default.target
+```
+
+```sh
+systemctl --user daemon-reload
+systemctl --user enable --now polite-crawler
+journalctl --user -u polite-crawler -f
+npx polite-crawler daemon status
+```
+
+On macOS, with a launch agent of launchd:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!-- ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plist -->
+<plist version="1.0">
+<dict>
+  <key>Label</key><string>io.github.polite-crawler.daemon</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>/usr/local/bin/polite-crawler</string>
+    <string>daemon</string>
+    <string>--events</string>
+    <string>/Users/you/.polite-crawler/changes.jsonl</string>
+  </array>
+  <key>RunAtLoad</key><true/>
+  <key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>
+  <key>StandardErrorPath</key><string>/Users/you/.polite-crawler/daemon.log</string>
+</dict>
+</plist>
+```
+
+```sh
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plist
+launchctl bootout gui/$(id -u)/io.github.polite-crawler.daemon
+```
+
+On Windows, with the Task Scheduler (PowerShell):
+
+```powershell
+$action = New-ScheduledTaskAction -Execute "$env:APPDATA\npm\polite-crawler.cmd" `
+  -Argument "daemon --events `"$env:USERPROFILE\.polite-crawler\changes.jsonl`""
+$trigger = New-ScheduledTaskTrigger -AtLogOn
+# no time limit: the default limit of 3 days stops the daemon
+$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit ([TimeSpan]::Zero) `
+  -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)
+Register-ScheduledTask -TaskName "polite-crawler daemon" -Action $action -Trigger $trigger -Settings $settings
+```
+
+The Task Scheduler stops a task with no signal. Thus the checks in progress stop at once, and their leases end after the budget of the tick and 2 minutes. `daemon status` then says that the daemon stopped with no clean stop.
+
+## Check field values against a page
+
+A model or a person often reads a value from a page: a price, a date, opening hours, an author or a DOI. `extract` checks each proposed value against the stored copy, and it locates the value. It does not make values. It only checks and locates the values that you give, so each value has its proof.
+
+```ts
+const page = await crawler.fetch("https://example.com/focaccia");
+const result = await crawler.extract(page.doc!, {
+  price: { value: "4.50 EUR", kind: "money" },
+  since: { value: "2026-03-04", kind: "date" },
+  phone: { value: "+1 617 555 1234", kind: "phone" },
+  offer: { value: "4.50", kind: "json-ld:Product.offers.price" },
+  name: { value: "Focaccia" },
+});
+for (const [name, field] of Object.entries(result.fields)) {
+  console.log(name, field.status, field.span ?? field.path, field.normalization, field.capture);
+}
+```
+
+Each field gets one status:
+
+| Status | What it means |
+| --- | --- |
+| `verified` | The stored text holds the exact value. The result gives the span (`start` and `end`) or the JSON-LD path, the stored text (`raw`) and the capture. |
+| `normalized` | The stored text holds the value in a different form. `normalization` names the form, and the span is the span of the stored text. |
+| `ambiguous` | The value matches, but the stored text does not give one meaning. `candidates` gives each place, and the check does not choose one. |
+| `not-found` | The stored text does not hold the value. |
+| `held-back` | The page asked not to be AI input. The result tells nothing about the value. |
+| `invalid` | The value is not a value of its kind, so the check did not search for it. |
+
+The kinds are `text` (the default for a text), `number` (the default for a number), `money`, `date`, `url`, `email`, `phone` and `json-ld:<path>`. A kind that is not a kind throws a `RangeError`. The result also gives `summary`, the count of each status.
+
+### The normalizations
+
+Each normalization has a name. A normalized field gives the names that it used, joined with `+`, for example `thousands-separator+number-form`.
+
+| Kind | Name | What it permits |
+| --- | --- | --- |
+| `text` | `unicode`, `whitespace`, `case` | NFKC forms, soft hyphens, typographic quotes and dashes, runs of spaces, and capital letters. A match starts and ends at the edge of a word. |
+| `number` | `thousands-separator` | A comma, a point, an apostrophe, or a thin or no-break space between groups of three digits. |
+| `number` | `decimal-comma` | A comma in place of the decimal point. |
+| `number` | `number-form` | The same value with other digits: zeros at the start or the end, or a plus sign. |
+| `number` | `unicode` | The same digits in other characters: fullwidth digits, or the minus sign. |
+| `money` | `currency-symbol`, `currency-position`, `amount-only` | A symbol in place of a code (`€` and `EUR`), the currency before or after the amount, and a value with no currency. The names of a number apply to the amount. |
+| `date` | `date-iso`, `date-ymd`, `date-written`, `date-dmy`, `date-mdy`, `date-numeric`, `date-form` | The form of the stored date. `date-written` has an English month. `date-dmy` and `date-mdy` are numeric dates with one possible order. |
+| `date` | `date-partial` | The value has no year or no day, and the stored date has them. |
+| `url` | `url-resolved`, `url-form`, `url-scheme` | A relative URL resolved against the page URL, a different form of the same canonical URL, and a stored URL with no scheme (`www.example.com`). |
+| `email` | `email-case` | Capital letters. |
+| `phone` | `phone-digits`, `phone-suffix` | Only the digits count. With `phone-suffix`, one number ends with the other: a country code, or a number with less detail. |
+
+An amount of money has a currency symbol or a code of ISO 4217 next to it, with one space or none. A code counts only in capital letters, so `15 try` is not an amount.
+
+### The rule for ambiguity
+
+A normalized match must have one meaning. The check does not guess:
+
+- `1,234` and `1.234` can be 1234 or 1.234. Thus a number value that needs a normalization to match them is `ambiguous`. `1,234.5` and `12,50` have one meaning.
+- `03/04/2026` can be 3 April or 4 March. `17/10/2026` and `10/17/2026` have one meaning.
+- `$` is the symbol of more than one currency, so `$12` does not confirm `USD 12`. `US$12` confirms it.
+- `March 4` has no year, so it does not confirm `2026-03-04`.
+- When the value matches two places with different meanings, the field is `ambiguous`, with each place. An example is two phone numbers that end in `555-1234`.
+- A value with two meanings, for example `03/04/2026`, is `ambiguous`, unless the stored text holds its exact text.
+
+An exact match is always `verified`: the stored text holds the text of the value, and the caller gives it its meaning.
+
+### Where the check looks
+
+- The kinds of text read the text of the version, the text that `read` gives. The positions are positions in that text. When the reader cut the text, the check also reads the whole text, and the span then has `in: "full_text"`.
+- `url` also reads the stored links of the page, with the path `links[3]`. The store keeps the canonical form of each link.
+- `json-ld:<path>` reads the JSON-LD of the page. The first part of the path names a schema.org type, and the path starts at each object of that type, at any depth. `*` starts at each top-level record.
+- Each array on a JSON-LD path gives each of its values. The result gives the concrete path, for example `jsonld[0].offers[1].price`, and the raw value. It compares as a number, a date, a URL and a text, in that sequence.
+- With no option, the check reads the current version. `capture` reads the version of one capture, and `at` reads the version that was current at a time. Give `capture` or `at`, not the two.
+
+A page that asked not to be AI input gives `held-back` for each field, with no position and no capture. The result is the same for a value that is in the page and for a value that is not. Thus the check is not an oracle. `userProvided` lifts the hold for a URL that a person gave directly, as for `read` and `verify`. The crawler logs it.
+
+The time of the check is linear in the length of the text, for a fixed set of fields. The tests count the steps of the readers on texts of different lengths.
+
+The MCP tool `crawl_extract` gives the statuses and the positions in its JSON. The stored text, the readings and the excerpts come in the `<untrusted-web-content>` envelope. The command line has `polite-crawler extract`.
+
+## Read the hours of a venue
+
+`readVenue` reads the opening hours of a venue (a restaurant, a shop or a museum) from its own site. It fetches a few pages, and it gives each reading with its source.
+
+```ts
+import { compareHours, openCrawler, readVenue } from "polite-crawler";
+
+const crawler = openCrawler({ dir: "./crawl" });
+const venue = await readVenue(crawler, "https://forno-esempio.example/", {
+  name: "Forno Esempio",
+  osm: "Mo-Fr 07:00-15:00; Sa 08:00-13:00; Su off", // a value that you got from OpenStreetMap
+});
+console.log(venue.best?.hours, venue.best?.capture, venue.disagree);
+for (const reading of venue.readings) console.log(reading.from, reading.label, reading.hours, reading.url);
+
+const change = compareHours(keptWeek, venue.from_site!.week);
+console.log(change.status, change.days); // changed [ 'sat' ]
+```
+
+The recipe reads these sources:
+
+1. The home page: its JSON-LD (`openingHoursSpecification` and `openingHours`) and its text.
+2. The pages that most likely hold hours or a menu. `rankHoursLinks` ranks the links of the home page by their words, for example "hours", "menu" and "visit". `pages` sets the number of these pages: 3 by default, and 8 at most.
+3. An OpenStreetMap `opening_hours` value, when you give `osm`. The recipe sends no request to OpenStreetMap.
+
+- Each page is a fetch of the crawler. Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A page that needs JavaScript gets the browser, as for `fetch`.
+- Each reading has `from`, `label`, `week`, `hours` (the week as words), `inferred`, `partial`, `season` and `raw`. It names its `url`, `doc` and `capture`, and a reading of JSON-LD names its `path`.
+- The `raw` of a reading of the text is a quote: `verify(reading.doc, reading.raw, { capture: reading.capture })` finds it.
+- `rank` orders the readings. JSON-LD is 0, a text under an hours heading is 1, OpenStreetMap is 2, and a text with no other label is 3. A different schedule (a brunch, a kitchen, a happy hour or an office) is 4.
+- `best` is the first reading of the opening hours. `from_site` is the best reading of the site itself, with no guessed am or pm. A check for a change compares `from_site`.
+- `disagree` names the days where two readings of the whole schedule (a rank of 2 or less) disagree.
+- The result also gives `menus` (the menu links that are not on a review or social site), `phone` (from the JSON-LD) and `captures`.
+
+| Option | What it does |
+| --- | --- |
+| `name` | The name of the venue. `names_venue` tells if the home page names it. A domain that a venue let go can show a different business. |
+| `requireName` | With `true`, a home page that does not name the venue gives no reading, and the recipe fetches no more pages. |
+| `location` | The `address`, or `lat` and `lon`, of the venue. On a page that lists more than one location, the recipe reads only this location. With no `location`, it reads none of them. |
+| `osm` | An OpenStreetMap `opening_hours` value, or `{ value, url, doc, capture }` with its source. |
+| `pages` | The pages after the home page, from 0 to 8. The default is 3. |
+| `render`, `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels`, `job` | As for `fetch`. |
+| `budget` | No request starts after this time, for example `"2m"`. |
+| `userProvided` | Read a page that asked not to be AI input. Use it only for a URL that a person gave. The crawler logs it. |
+| `fallback`, `archiveDespiteRobots` | Read an archived copy of a page that the site refuses. The reading then has `archived`. |
+
+- A location of a page matches when it has the same street number and a word of the street, or when it is within 150 m.
+- `moved` names the new site when the home page landed on a different host.
+- A page that asked not to be AI input gives no reading, and `held_back` is true.
+- The recipe writes nothing but the capture log. It does not decide which reading is true. That decision belongs to your application.
+
+`venueReadingsOf(doc, { location })` reads one stored document with no request. Thus it also reads an earlier version:
+
+```ts
+import { venueReadingsOf, versionAtTime } from "polite-crawler";
+
+const { version } = versionAtTime(crawler.runtime, "url:https://forno-esempio.example/", "2026-10-01T00:00:00Z");
+const then = venueReadingsOf(version!);
+```
+
+`compareHours(kept, reading, { skipSplit })` compares a reading with the week that your application keeps. It is in `polite-crawler/hours` too.
+
+- It gives `status` (`same`, `changed` or `new`), the changed `days`, the `added` days and the `skipped` days.
+- It compares only the days that the two weeks state. A day that only the reading states is `added`, not a change.
+- `24:00` and `00:00` are the same time, and `9:00` is `09:00`. The sequence of the intervals of a day does not matter.
+- With `skipSplit`, it does not compare a day with two or more intervals, for a store that keeps one interval for each day.
+- One reading can be an error of the site. Confirm a change with a second reading, as a watch does.
+
+## Read a product page
+
+`readProduct` reads the schema.org Product data of a product page. Most shop systems write this data on each product page.
+
+```ts
+import { readProduct } from "polite-crawler";
+
+const product = await readProduct(crawler, "https://bottega-campione.example/products/olio-esempio?variant=222");
+if (product.wall) console.log("the shop refused the crawler:", product.note);
+else console.log(product.name, product.price, product.price_to, product.availability, product.variant);
+// Olio Esempio { amount: 9.9, currency: 'EUR', raw: '9.90' } { amount: 18, ... } InStock { sku: 'OE-500', ... }
+```
+
+- The result gives `name`, `brand`, `sku`, `gtin`, `price`, `price_to`, `availability`, `availabilities`, `variant`, `seller` and `image`. It names the `doc` and the `capture` of the page.
+- A price is `{ amount, currency, raw }`, and the currency is an ISO 4217 code. `€` and `£` count too, because each one names one currency. A price with no currency, or with `$`, is not money, and a note says so.
+- `price` is the lowest price. `price_to` is the highest price, when the offers give a range (sizes, colors, variants). The prices are in the first currency of the offers.
+- A list price and a struck-through price (`priceType` `ListPrice` or `StrikethroughPrice`) are not the price.
+- `variant` is the variant that the URL names with the parameter `variant`, `sku` or `gtin`. The recipe reads its price alone.
+- `availability` is a schema.org name, for example `InStock`. When the offers do not agree, it is `InStock` if one offer is in stock.
+- The options are `render`, `maxAge`, `onBehalfOf`, `signal`, `budget`, `scope`, `labels`, `job`, `userProvided`, `fallback` and `archiveDespiteRobots`, as for `readVenue`.
+
+A page often has more than one Product record: the related products, and the products that others bought. The recipe picks the product of the page:
+
+1. The product whose `url` or `@id` is the page. An `@id` that is only a fragment (`#product`) does not count, because a related product can have one too.
+2. Else the largest product with offers or variants.
+3. Else the largest product. The recipe also reads the `mainEntity` of a `WebPage` or an `ItemPage`.
+
+- The index keeps 30,000 characters of the JSON-LD of each page. A product with many variants can be larger, so the recipe reads the stored body again.
+- When the plain page has no Product data, the recipe renders the page once, because some shops write that data with JavaScript. `render: "never"` stops this.
+- A shop that refuses the crawler gives `wall: true`, with a note. The causes are a 401, a 402 or a 403, a challenge, a bot check, a stop of the host or robots.txt. The recipe does not try a different identity.
+- A page with no Product data gives `found: false`. It is often a list of products.
+
+`readAmount` reads an amount as a shop writes it. The last separator is the decimal point. One kind of separator before groups of three digits separates thousands, so `1.299` is 1299. A currency with three decimal digits (`KWD`) reads `1.299` as 1.299. A range, a negative amount and zero give no amount.
+
+`productOf(records, url)` reads the JSON-LD records of a page with no request, so it also reads an earlier version. `extract` checks each value against the stored page:
+
+```ts
+const check = await crawler.extract(product.doc!, {
+  price: { value: product.variant!.price!.raw, kind: "json-ld:Product.offers.price" },
+});
+```
+
+## Writing a recipe
+
+A recipe tells the crawler what to fetch and how to read it into documents. The venue and product recipes read pages with the page reader of the package. A recipe of your own reads a source that the page reader does not know, for example an API.
+
+```ts
+import { defineRecipe, openCrawler, readJson, runRecipe } from "polite-crawler";
+
+interface Item {
+  readonly id: string;
+  readonly name: string;
+  readonly price: string;
+}
+
+const menu = defineRecipe({
+  name: "forno-menu", // it goes into each capture record: do not change it
+  mode: "api", // a documented API: robots.txt does not apply, and the pace does
+  minInterval: "2s", // the documented pace of the API
+  maxAge: "6h",
+  fetch: (day: string) => `https://api.forno-esempio.example/menu?day=${encodeURIComponent(day)}`,
+  extract: (body, capture) => {
+    const { items } = readJson(body, capture) as { items: Item[] }; // a body that is not JSON throws
+    return items.map((item) => ({
+      doc: `menu:forno-esempio/${item.id}`,
+      kind: "menu-item",
+      title: item.name,
+      text: `${item.name}: ${item.price}`,
+      url: capture.url,
+      meta: { price: item.price },
+    }));
+  },
+});
+
+const crawler = openCrawler({ dir: "./crawl", recipes: [menu] });
+const run = await runRecipe(crawler, menu, "mon");
+console.log(run.outcome, run.capture, run.docs.map((doc) => doc.doc));
+
+await crawler.find("focaccia"); // the documents of the recipe are in the index
+await crawler.read("menu:forno-esempio/focaccia", { at: "2026-10-01T00:00:00Z" }); // each version stays
+```
+
+`defineRecipe` takes these values:
+
+| Value | Default | What it does |
+| --- | --- | --- |
+| `name` | (necessary) | The name of the recipe: 1 to 64 letters, digits and `. _ : -`. The names of the crawler (`RESERVED_RECIPES`) are not permitted. |
+| `extract` | (necessary) | The function that reads a body into documents. |
+| `fetch` | the input is the URL | The function that gives the request for an input: a URL, or `{ url, accept, headers, secretParams }`. |
+| `mode` | `page` | `page` is a web page, and robots.txt applies. `api` is a documented API with terms of its own. |
+| `accept` | HTML for a page, JSON for an API | The `Accept` header. |
+| `minInterval` | the policy of the host | The shortest time between two requests to the source. |
+| `maxAge` | `24h` | A stored copy younger than this age answers with no request. `auto` uses the freshness that the server gave. |
+| `maxBytes` | 5 MiB | The most bytes of a body that the crawler keeps. |
+| `description` | none | A text that tells what the recipe reads. |
+
+`runRecipe(crawler, recipe, input, options)` fetches the request of an input through the crawler, and reads its capture with the extractor.
+
+- The result gives `outcome`, `status`, `url`, `docs`, `capture`, `fetched_at`, `from_cache`, `held_back`, `error` and `note`. The `url` shows each secret parameter as `REDACTED`.
+- Each document gives `doc`, `kind`, `title`, `url`, `meta`, `chars`, `capture` and `fetched_at`. `read` gives its text. A held-back document shows no title and only its structural metadata.
+- The options are `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels`, `job`, `budget`, `userProvided`, `fallback` and `archiveDespiteRobots`.
+- Give each recipe to `openCrawler` (or `createRuntime`) with `recipes`. Then `reindex` reads its captures from the start. `runRecipe` also registers a recipe the first time that it gets it.
+- `registerRecipe(runtime.extractors, recipe)` registers a recipe in a runtime of your own. A different extractor with the same name throws an error.
+- `recipe.request(input, options)` gives the request of the crawler, for `fetchAndIndex` in an application that builds its own steps.
+- `readJson(body, capture)` reads a body as JSON, in the encoding that the response declares. A body that is not JSON throws an error that names the URL and the content type.
+- `runRecipe`, `readVenue` and `readProduct` take a crawler or a runtime (`createRuntime`). The option `onBehalfOf` of `openCrawler` does not apply to them, so give `onBehalfOf` to each call.
+
+### The rules of an extractor
+
+1. **An extractor is pure.** The same body and capture always give the same documents. Do not read the clock, a random number or a state from an earlier call.
+2. **An extractor sends no request.** It gets the body of the capture. To read a second source, start a second recipe. Its fetch is then polite, and it has its capture.
+3. **The runtime reports an error, and the work continues.** An extractor that throws gives no document. `runRecipe` gives the error in `error`, and `reindex` counts it in `failed` and `errors`. The fetch and the rebuild continue.
+4. **The SDK checks each document.** Each document needs `doc`, `kind`, `title`, `text`, `url` and `meta`, and `meta` must be JSON. Two documents with one id, the prefix `archive:` and the kind `archived` throw an error. The archived fallback uses that prefix and that kind.
+5. **A held-back body stays held back.** Set `ai_input: "no"` when the body asks not to be AI input. An extractor cannot lift a `no` of the capture.
+
+### The rules of the capture ids
+
+- The crawler makes the capture id, not the extractor. The id is a hash of the source, the kind, the external id and the payload of the capture record.
+- The payload holds the URL, the name of the recipe, the outcome, the hash of the body and the time of the fetch.
+- Thus the name of a recipe is part of each capture id. Do not change the name after the first capture. With a new name, `reindex` finds no extractor for the old captures, and it keeps their documents as they are.
+- Each version of a document names its capture. A citation names that capture, and `read`, `verify` and `extract` take `capture` to use that version.
+- `reindex` reads each capture again with the extractor. A pure extractor thus gives the same versions with the same capture ids.
+- A new version of an extractor changes the documents at the next `reindex`, not the capture ids. Keep it able to read the old captures.
+- The scope and the labels of a fetch are not part of the capture id.
+
+### The documents, the time travel and the archived fallback
+
+- Give each document an id with a prefix of your own, for example `menu:`. The page reader uses `url:`. A recipe that gives a `url:` id adds versions to the document of that page, and `merge` chooses the current version.
+- Each capture of a recipe adds a version of each of its documents. `read`, `verify` and `extract` take `at`, and `diff` compares two versions.
+- With `fallback: "archive"`, a recipe of the mode `page` gets an archived copy after a refusal. The page reader reads the copy as the document `archive:<url>`, with `archived`.
+- The extractor of a recipe reads only the live captures. Thus its documents do not mix a live answer and an old copy of a third party. A recipe of the mode `api` takes no `fallback`.
+- When the extractor cannot read a stored copy, `runRecipe` asks the source one more time. Thus a bad answer (a page about maintenance) does not stay until `maxAge` ends.
+
+## Find the pages of a site
+
+`discover` finds the pages of a site from its sitemaps and its feeds. It fetches the files that list the pages, not the pages.
+
+```ts
+const found = await crawler.discover("https://example.com/");
+for (const url of found.urls) console.log(url.url, url.lastmod, url.via, url.source);
+console.log(found.stopped, found.dropped);
+```
+
+Discovery reads these sources, in sequence:
+
+1. The page at the URL that you give, for the RSS and Atom feeds that it links with `<link rel="alternate">`. With `linkedFeeds: false`, discovery does not fetch the page.
+2. The feeds in `feeds`, and the sitemaps in `sitemaps`.
+3. The `Sitemap:` lines of robots.txt. When robots.txt names no sitemap and `sitemaps` is empty, discovery reads `/sitemap.xml`.
+4. The sitemaps of each sitemap index, the newest `lastmod` first.
+
+Each URL of the result has its `lastmod` and `changefreq` when the file gives them. Its `source` is the file that listed it, and `via` tells if that file is a sitemap or a feed. The newest `lastmod` comes first.
+
+- Discovery keeps a URL only when it is on the same site and robots.txt permits it. The rules for the crawler and for the agents of `onBehalfOf` apply. A host and its `www.` form are one site.
+- `allowHosts` adds more hosts (`*.example.org` names the hosts under a name). `sameSite: false` keeps the URLs of each host.
+- Discovery reads XML sitemaps, sitemap indexes, gzip sitemaps (`.xml.gz`), text sitemaps, RSS 2.0, RSS 1.0 and Atom.
+- Each file is a fetch of the crawler, with the recipe `sitemap` or `feed`, and the capture log records it. A stored copy younger than `maxAge` (24 hours by default) answers with no request.
+- `sources` tells what discovery did with each file. `dropped` counts the URLs that it did not keep, and `stopped` tells why it stopped.
+
+These options limit a discovery. `DISCOVER_LIMITS` gives the defaults.
+
+| Option | Default | What it limits |
+| --- | --- | --- |
+| `maxFiles` | 20 | The sitemap and feed files that discovery reads. |
+| `maxUrls` | 10,000 | The URLs of the result. |
+| `maxBytes` | 10 MiB | The bytes of one file, after decompression. A longer file gives its first part, and its source says `truncated`. |
+| `maxNesting` | 3 | The levels of sitemap index under the first file. |
+| `budgetMs` | 60,000 (1 minute) | The time of the discovery. No request starts after it. |
+
+Discovery also reads robots.txt for 10 origins at most, and 5 feeds at most from the links of the page.
+
+The sitemap reader (`readSitemap`) is strict, because a file from the web can be hostile:
+
+- It expands no entity. It knows only the five entities of XML and the character references, and it ignores a DTD. Thus a "billion laughs" file gives no text.
+- It stops at a depth of 32 elements, and at 50,000 entries (the limit of the sitemap protocol). A value of more than 4,096 characters is not valid.
+- It reads the text one time and builds no tree, so its time is linear in the length of the file.
+- It stops at the first error of the markup, and it keeps the entries before the error.
+
+## Crawl a site
+
+`crawl` fetches the seeds, and then the links of each page on the same site, to a budget. A `for await` loop gets each page when its fetch ends.
+
+```ts
+const run = crawler.crawl("https://example.com/", { maxPages: 50, maxDepth: 2, budgetMs: 120_000 });
+for await (const page of run) console.log(page.depth, page.source, page.outcome, page.url);
+
+const summary = await run.summary;
+console.log(summary.stopped, summary.reason, summary.fetched, summary.refused, summary.skipped);
+```
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `maxPages` | 100 | The most pages that the crawl gives. |
+| `maxDepth` | 3 | The most links from a seed to a page. The value 0 fetches only the seeds. |
+| `budgetMs` | 600,000 (10 minutes) | The time of the crawl. No request starts after it. |
+| `sameSite`, `allowHosts` | the sites of the seeds | The hosts that the crawl follows, as for `discover`. |
+| `include`, `exclude` | none | The patterns of the URLs that the crawl follows. |
+| `useSitemaps` | `false` | The value `true` adds the pages of the sitemaps of each site of the seeds. |
+| `render`, `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels` | as for `fetch` | These options apply to each page. |
+
+- The seeds come first, then the pages of the sitemaps (the newest `lastmod` first), then the links. A lower depth comes before a higher depth.
+- A pattern that starts with `/` matches the path and the query. A different pattern matches the whole URL. A `*` matches any text, and a `$` at the end matches the end, as in robots.txt. A `RegExp` tests the whole URL.
+- The scope and the patterns apply to the links and to the pages of the sitemaps, not to the seeds.
+- The crawl follows only http(s) links, and it fetches each page one time. A fragment, a tracking parameter or a default port does not make a different page.
+- The crawl does not fetch a URL that is the final URL or the canonical URL of a page that it fetched.
+- The links come from the stored document of each page, 100 or fewer for each page. The crawl does not follow the links of a page that asked not to be AI input.
+
+The summary counts the pages (`fetched`, `from_cache`, `refused` and `failed`) and the links that the crawl did not follow (`skipped`, with the reasons in `skips`). `queued` counts the URLs that waited at the stop. `stopped` and `reason` tell why the crawl stopped: `done`, `max-pages`, `budget`, `cancelled`, `closed` or `error`.
+
+### The crawl is polite
+
+The frontier does not change a rule of the crawler, because each page is a fetch of the crawler:
+
+- robots.txt applies to each page. A link that robots.txt refuses costs no request, and the summary counts it in `refused`.
+- One request at a time goes to each host, at the pace of the host. By default, a host rests 3 seconds or more between two pages, so a crawl of one site gets approximately 20 pages in each minute.
+- The frontier sends requests to 16 hosts at one time at most. The crawler also limits the hosts in progress (`maxActive`).
+- The stops, the backoff, the egress policy, the redaction of secrets and the capture log apply as for `fetch`. The `User-Agent` stays the honest default.
+- When the budget ends, no new request starts, and a request in progress ends normally. A request that waits for its host past the budget does not start. Its skip record says `deadline reached`.
+- When `signal` aborts, or a `break` ends the loop, the crawl stops the requests in progress and ends.
+
+The frontier keeps its state in memory, for one call. It keeps 10,000 URLs that wait and 100,000 URLs that it saw, at most (`CRAWL_LIMITS`). A later version can make the frontier resumable.
+
+## Archived copies after a refusal
+
+When a site refuses a page, or the site is down, the crawler can give a copy from a public web archive. The crawler does not show the copy as a fresh fetch, and each result marks it.
+
+```ts
+const page = await crawler.fetch("https://example.com/hours", { fallback: "archive" });
+console.log(page.outcome, page.archived?.source, page.archived?.timestamp, page.doc);
+// blocked wayback 2024-01-15T09:30:00Z archive:https://example.com/hours
+```
+
+The fallback is off by default. The command line has `fetch --archive-fallback`, and the tool `crawl_fetch` has `archive_fallback`.
+
+### When the crawler gives a copy
+
+| The live fetch | An archived copy |
+| --- | --- |
+| robots.txt disallows the page | Only with `archiveDespiteRobots` |
+| A stop of the host, after a 401, a 402, a 403 or a challenge | Yes |
+| A 401, a 402, a 403, or a challenge (also a page that is a bot check) | Yes |
+| A 502, a 503, a 504, or a different 5xx error | Yes |
+| No response (a dead host), or a robots.txt that the crawler cannot read and has no recent copy of | Yes |
+| A 404, a 410, a 429, a cancel, or a refusal of the egress policy | No |
+
+### Where the copy comes from
+
+1. The Wayback Machine. Its availability API (`archive.org/wayback/available`) gives the newest copy, and the crawler fetches the raw copy (`web.archive.org/web/<time>id_/<url>`).
+2. Common Crawl, when the Wayback Machine has no copy. The crawler reads the list of indexes (`index.commoncrawl.org/collinfo.json`) and asks the 3 newest indexes. It fetches the WARC record of the newest copy with a range request (`data.commoncrawl.org`).
+
+- The archives are services of third parties, with their own politeness. Each request to them is a fetch of the same crawler. Their robots.txt, the pace of each host, the egress policy and the identity of the crawler apply.
+- The capture log records each request to an archive, with the recipe `archive-index` (an index) or `archive-copy` (a copy). `reindex` reads the copies again from the log.
+- A stored copy younger than `maxAge` answers with no request.
+- An archive learns the URL of the page. A URL that holds a secret (a password or a secret parameter) does not go to an archive.
+
+### How a result shows a copy
+
+- A copy is a document of its own, `archive:<url>`, of the kind `archived`. The document of the live page (`url:<url>`) does not change.
+- `fetch` keeps the `outcome` and the `status` of the live fetch, for example `blocked` and 403. It adds `archived`: `source` (`wayback` or `commoncrawl`), `timestamp` (when the archive captured the page), `url` (the copy in the archive), `original` (the URL that the archive captured), and `record` for Common Crawl.
+- `read`, `verify` and `find` give `archived` for a copy. The note of `read` and `verify` names the archive and the date. Thus a citation of a copy names the archive and its date, not the date of a fresh fetch.
+- `fetched_at` stays the time of the fetch from the archive. `archived.timestamp` is the date of the copy.
+- `find` with `kind: "page"` leaves the copies out.
+- In the MCP envelope, a copy has the attribute `archived`, for example `archived="wayback 2024-01-15T09:30:00Z"`. Its `age` starts at the date of the copy.
+- With `--archive-fallback`, `fetch` gives the exit code 0 for a page with a copy, and the line of the page starts with `~`.
+
+### robots.txt and the usage signals of a copy
+
+An archive keeps copies of pages that the site refuses today. To read such a copy or not is a decision of policy, and the default is strict:
+
+- With no `archiveDespiteRobots`, the crawler gives no copy of a page that the current robots.txt of its site disallows for this agent. The rules for the agents of `onBehalfOf` apply too.
+- For a stopped host, the crawler reads the rules again: from memory, or from the stored copy of robots.txt of the last 30 days. Thus only a stop, an error or a dead host gets a copy.
+- The reason: robots.txt tells what the owner of the site wants today. A copy of a disallowed page goes around that wish. An owner who disallows a page today does not expect an agent to read it from an archive.
+- `archiveDespiteRobots: true` (`--archive-despite-robots`) gives a copy of these pages too. The robots.txt of each archive still applies to the requests to it. The MCP server has no such parameter, because this decision belongs to the operator.
+- The usage signals of a copy apply as for a live page. A copy is held back from the AI reader when the content signals of the current robots.txt of its site say `ai-input=no`.
+- The headers of the archived response count too: `X-Archive-Orig-Content-Signal` of the Wayback Machine, and the HTTP headers in the WARC record of Common Crawl. The signals of the archive itself count as well.
+
+## Signed requests (Web Bot Auth)
+
+With an Ed25519 key, the crawler signs each request with HTTP Message Signatures (RFC 9421), as the IETF Web Bot Auth draft tells. A site can then verify that a request comes from the holder of your key. The package follows `draft-ietf-webbotauth-httpsig-protocol-00` (1 September 2026).
+
+```ts
+import { readFileSync } from "node:fs";
+import { openCrawler } from "polite-crawler";
+
+const crawler = openCrawler({
+  dir: "./crawl",
+  signer: {
+    privateKey: readFileSync("bot-key.pem", "utf8"), // PKCS#8 PEM, a JWK or a KeyObject
+    signatureAgent: "https://bot.example.com", // the origin of your key directory
+  },
+});
+```
+
+Each request gets three headers:
+
+```text
+Signature-Agent: sig1="https://bot.example.com"
+Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1760702400;keyid="poqkLGiy...";alg="ed25519";expires=1760702700;nonce="...";tag="web-bot-auth"
+Signature: sig1=:...:
+```
+
+- The signature covers `@authority` and the member of `Signature-Agent`. `keyid` is the JWK thumbprint of the key (RFC 8037). Each signature has a new random nonce, and it expires after 5 minutes.
+- The crawler signs each request: robots.txt, each redirect, the sitemaps, the archives of the fallback and each request of a rendered page. Without a signer, it signs nothing. The browser does not sign a WebSocket connection.
+- A signature adds an identity. The `User-Agent` and `From` headers do not change. With a signer, the crawler replaces the signature headers of a caller.
+- A signer that fails sends no request: the result is a skip with a note.
+- The signature headers are not secrets. The capture log records `signed`, with the key id and the agent. It does not hold the key. The signer keeps the key in a closure, and no property of it shows the key.
+
+`createWebBotAuthSigner(options)` makes a signer. `signer` also takes these options, or a `RequestSigner` of your own (for example, a signer with a key in a hardware module).
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `privateKey` | | The Ed25519 private key: a `KeyObject`, PKCS#8 (PEM text or DER bytes) or a JWK. |
+| `signatureAgent` | | Where a site finds your public key. For the type `directory`, an https origin. |
+| `agentType` | `directory` | `directory`: the key directory is at the well-known path of the origin. `jwks_uri`: the URL is a JSON Web Key Set on any path. |
+| `keyId` | the thumbprint | A check: a value that is not the thumbprint of the key throws an error. |
+| `lifetimeMs` | 300,000 (5 minutes) | The time that a signature stays valid, 24 hours at most. |
+| `label` | `sig1` | The label of the signature and the key of the member of `Signature-Agent`. |
+
+`verifySignature(request, directory)` verifies the signature of a request with a key directory, and tells why a request does not verify. Use it to test your setup.
+
+Make a key with OpenSSL, and give it to the command line. Do not use the test keys of RFC 9421 for a real crawler.
+
+```sh
+openssl genpkey -algorithm ed25519 -out bot-key.pem
+npx polite-crawler key-directory --sign-key bot-key.pem --signature-agent https://bot.example.com > http-message-signatures-directory
+npx polite-crawler fetch https://example.com/ --sign-key bot-key.pem --signature-agent https://bot.example.com
+```
+
+The command line takes `--sign-key` and `--signature-agent` (or `POLITE_CRAWLER_SIGN_KEY` and `POLITE_CRAWLER_SIGNATURE_AGENT`) for `fetch`, `discover`, `crawl`, `watch run` and `mcp`. A `--signature-agent` with a path gives the type `jwks_uri`.
+
+### Host the key directory
+
+A site finds your public key from `Signature-Agent`. With the type `directory`, the key directory is at `/.well-known/http-message-signatures-directory` on the origin of `signatureAgent`.
+
+1. Make the JSON with `keyDirectory([publicKey])` or `polite-crawler key-directory`. It holds only the public part of each key, with its thumbprint as `kid`.
+2. Serve it at `https://<your origin>/.well-known/http-message-signatures-directory`. Send the status 200 and `Content-Type: application/http-message-signatures-directory+json`. Do not redirect it, because a verifier follows no redirect.
+3. When you change the key, keep the old key in the directory until its last signature expires.
+
+The draft recommends that the server of the directory also signs its response with each key, as a proof of possession. `keyDirectory` gives only the JSON, so a static host gives no such proof.
+
+GitHub Pages cannot serve a key directory of the type `directory` for a project site:
+
+- A project site (`<user>.github.io/<repository>`) serves files only below its path, so it cannot serve the well-known path at the root of the host.
+- GitHub Pages sets the content type from the extension of the file, so it cannot send the type of the directory.
+- On a project site, put the JSON in a file, for example `jwks.json`. Give its URL as `signatureAgent`, with `agentType: "jwks_uri"`. The draft then binds the key to that URL, not to the domain.
+- A host that sets headers (the `_headers` file of Cloudflare Pages or Netlify, for example) can serve the type `directory` at the root of a domain.
+
 ## How the crawler is polite
 
 1. **robots.txt.** The crawler reads the robots.txt of each site and obeys it, as RFC 9309 specifies. A 401 or a 403 for robots.txt means that the site has no file. When the crawler cannot read robots.txt (a dead domain, a server error or a body that does not decode), it fetches nothing from the site. Then it tries again after some minutes, with a backoff from `Retry-After` up to 1 hour. A good copy of the file from the last 30 days applies in that time.
@@ -283,6 +1290,8 @@ The other exports of the root entry point are the engine. Examples are `createRu
 5. **An honest `User-Agent`.** The default is `PoliteCrawler/0.1.0 (+https://mark1russell7.github.io/crawler/bot/)`. The `contact` option adds an address. An email address also goes into the `From` header, and a URL does not, because `From` holds only a mailbox. The constructor of `Crawler` throws an error for a bad option. Examples are a token that is not a product token, a `User-Agent` or a `From` that HTTP cannot send, and a `maxActive` less than 1.
 6. **Public addresses only.** By default, the crawler fetches only `http:` and `https:` URLs on public addresses. It checks the address after DNS and at each redirect. In the headless browser, it checks each request of the page and each redirect of that request. A refusal is a skipped fetch with the note `egress policy: <address> is a loopback address; set allowPrivate to fetch it`.
 7. **Usage signals.** When a page asks not to be AI input (`ai-input=no` content signals), the crawler keeps it but holds back its text from the AI reader. The crawler also records the RSL `License` lines of robots.txt and the TDMRep reservation (the `tdm-reservation` and `tdm-policy` headers and `<meta>` tags) with each capture. These records change nothing, because a reservation of text and data mining is not a refusal of AI input.
+8. **Signed requests, as an option.** With a `signer`, each request has a Web Bot Auth signature, so a site can verify where it comes from. [Signed requests](#signed-requests-web-bot-auth) tells how.
+9. **Archived copies, as an option.** `fallback: "archive"` reads a copy from a web archive only after a refusal. By default, it gives no copy of a page that the current robots.txt disallows.
 
 To block the crawler, a site owner adds this group to robots.txt:
 
@@ -293,7 +1302,7 @@ Disallow: /
 
 ## Stored copies and conditional requests
 
-A fetch uses the stored copy of a URL, with no request, while the copy is younger than `maxAge` (24 hours by default). With `maxAge: "auto"`, the crawler uses the freshness that the server gave: `Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`. `no-store` and `no-cache` make a copy stale at once. A response that varies on a request header (`Vary`) has a stored copy for each value of that header.
+A fetch uses the stored copy of a URL, with no request, while the copy is younger than `maxAge` (24 hours by default). With `maxAge: "auto"`, the crawler uses the freshness that the server gave: `Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`. With `"auto"`, `no-store` and `no-cache` make a copy stale at once. A fixed `maxAge` ignores them. A response that varies on a request header (`Vary`) is used again only for a request with the same value of that header. The store keeps one copy for each URL, so a request with a different value replaces it.
 
 When the stored copy is too old, the crawler sends a conditional request. `If-None-Match` sends the `ETag` of the copy, and `If-Modified-Since` sends its `Last-Modified` date. A `304 Not Modified` sends no body, and the crawler uses the stored body again. The capture log records each request, with the validators that it sent (`validators`).
 
@@ -314,6 +1323,7 @@ The memo is in the host state store. `openCrawler` keeps it in `hosts.sqlite`, s
 | `blobs/` | The bodies of the responses, by SHA-256 hash. |
 | `index.sqlite` | The search index. It is a projection: `polite-crawler reindex` builds it again from the other two. |
 | `hosts.sqlite` | The pace, the counts, the stops and the validator memo of each host. The processes on one store share it. |
+| `watch.sqlite` | The watch list and the state of each watch. The processes on one store share it. The file opens at the first use of `crawler.watch`. |
 
 The package publishes the JSON Schema of a line of the log: `polite-crawler/record.schema.json`. A program that reads the log can check each record with it.
 
@@ -327,6 +1337,68 @@ The package publishes the JSON Schema of a line of the log: `polite-crawler/reco
 - A capture does not hold the secrets of a URL. The crawler writes `REDACTED` in place of a user name and a password. It does the same for the values of the usual secret query parameters (`DEFAULT_SECRET_PARAMS`) and of each name in `secretParams`.
 - The crawler keeps no more than `maxBytes` of a body (5 MiB by default) after decompression. A longer body keeps its first part, and the capture says `truncated`. A body that does not download or decode gives the outcome `error`.
 
+## Export to a web archive
+
+The store is private evidence, and it stays on your disk. An export is a decision of the operator: it copies the captures that you select into a standard web archive. Archive tools read it, for example `warcio`, pywb and ReplayWeb.page.
+
+```sh
+npx polite-crawler export --out crawl.wacz
+npx polite-crawler export --out site.warc.gz --host example.com --since 2026-10-01
+```
+
+```ts
+import { exportWacz, exportWarc } from "polite-crawler";
+
+await exportWarc("./crawl", { out: "crawl.warc.gz", host: "example.com" });
+
+const result = await exportWacz(crawler.runtime, { out: "crawl.wacz", since: "2026-10-01" });
+console.log(result.responses, result.revisits, result.pages);
+```
+
+- `exportWarc` writes WARC/1.1 (ISO 28500:2017). Each record is one gzip member, so the file is a `.warc.gz`.
+- `exportWacz` writes WACZ 1.1.1: a ZIP with the WARC file, a sorted CDXJ index, a page list and `datapackage.json`.
+- The source is the folder of a store, or `crawler.runtime`. The export only reads the store. It makes no store, and it sends no request.
+- These filters select the captures: `since` (this time or later), `until` (before this time), `host` (the host and its subdomains), `urlPrefix` and `ids`. The command line has `--since`, `--until`, `--host`, `--prefix` and `--id`.
+- The export reads the capture log as a stream and holds one body at a time. It sorts the index and the page list in files on the disk. Thus its memory does not grow with the store.
+- The export writes a temporary file and gives it its name at the end. A failed export leaves no part of a file. When the file exists, the export fails. Set `overwrite` (`--overwrite`) to replace it.
+- The MCP server has no export tool. An export writes files on the disk of the operator, so it is an action of the operator, not of an assistant.
+
+### What goes into the archive
+
+| Capture | Records |
+| --- | --- |
+| A response with a stored body | A `response` record with the status line, the headers and the body. Then a `metadata` record. |
+| A `304 Not Modified` | A `revisit` record with the profile `server-not-modified`, and a `metadata` record. |
+| A page that the headless browser rendered | A `conversion` record with the page after JavaScript, and a `metadata` record. |
+| A capture with no body: a refusal, a skip of robots.txt, an error or a redirect | No record. With `includeRefusals` (`--include-refusals`), a `metadata` record alone. |
+
+- The first record is a `warcinfo` record. It names the software, its version and the filters of the export.
+- The `metadata` record of a capture (`WARC-Concurrent-To`) holds the JSON of the capture. It has the id, the kind, the URL, the times, the content hash and the payload. The payload has the outcome, the note, the robots.txt check and the usage signals. A reader can calculate the content hash again and check the capture id.
+- Each record of a capture has the field `WARC-Polite-Crawler-Capture` with the capture id. The record ids come from the hash of the capture, so a second export gives the same ids.
+- The digests are SHA-256 in base 32: `sha256:` and 56 characters, the form that `warcio` uses. `WARC-Payload-Digest` is the digest of the stored body.
+- A revisit names its original (`WARC-Refers-To` and `WARC-Refers-To-Date`) when the original is in the same export. Its `WARC-Payload-Digest` is the digest of the body that the 304 confirmed.
+- A body that the crawler cut at `maxBytes` has `WARC-Truncated: length`. A compressed body that ended early has `WARC-Truncated: disconnect`.
+- The pages that ask not to be AI input go into the archive too. An archive is not AI input. Their usage signals are in the `metadata` record, so a reader of the archive can obey them.
+
+### What does not go into the archive, and why
+
+- **The raw headers.** A capture keeps only some headers of the response: `Content-Type`, `ETag`, `Last-Modified`, `Cache-Control`, `Expires`, `Age`, `Date` and `Vary`. Thus the export rebuilds the status line and the header block from them, and it adds the `Content-Length` of the stored body. Such a record has the field `WARC-Polite-Crawler-Headers: rebuilt`, and its `metadata` record says `"http_headers": "rebuilt"`.
+  - The stored body is the decoded body. Thus the block has no `Content-Encoding` and no `Transfer-Encoding`.
+  - The status line says `HTTP/1.1`, also when the server answered over HTTP/2.
+  - The export writes a `response` record and not a `resource` record. Replay tools (pywb, ReplayWeb.page) replay a `response` with its status and its headers, and a `resource` record has neither. Browser crawlers also write the headers that they have, with a decoded body.
+- **The request.** A capture does not keep the headers of the request (`User-Agent` and `From`). Thus the export writes no `request` record. The `metadata` record gives the validators that the request sent.
+- **The IP address.** A capture does not keep the address of the server. Thus no record has `WARC-IP-Address`.
+- **The labels and the scope of the application.** They can name a private lineage, so the `metadata` record leaves them out.
+- **The other records.** The overrides of a held-back page and the records of the other sources of the log stay in the store.
+
+### The index and the pages of a WACZ file
+
+- `archive/data.warc.gz` is stored with no more compression, so a replay tool reads a record at its offset.
+- `indexes/index.cdxj` has one line for each `response` and `revisit` record, sorted by SURT key and time. The `conversion` records are not in the index, so a replay tool shows only the responses of the servers.
+- `pages/pages.jsonl` has one page for each document: each URL with a `200` response, except robots.txt files. The page names its latest capture.
+- `datapackage.json` gives the SHA-256 and the size of each part. `datapackage-digest.json` gives the SHA-256 of `datapackage.json`.
+- The ZIP has Zip64 records only when a part or the file is larger than 4 GiB.
+
 ## Contributing
 
 [`CONTRIBUTING.md`](https://github.com/mark1russell7/crawler/blob/main/CONTRIBUTING.md) tells how to set up the repository and test a change. [`CHANGELOG.md`](https://github.com/mark1russell7/crawler/blob/main/CHANGELOG.md) records the changes in each version. Report a vulnerability privately, as [`SECURITY.md`](https://github.com/mark1russell7/crawler/blob/main/SECURITY.md) tells.