polite-crawler 0.1.0 → 0.1.2

package/README.md

@@ -12,7 +12,12 @@ The site has interactive demos with the real code: <https://mark1russell7.github
 - **Fetch one time.** Each response goes into a capture log and a store of bodies by hash. A repeated question reads the disk, so it costs no request.
 - **Read the cheap way.** The crawler reads JSON-LD, the visible text and the links with no browser. It opens a headless browser only for a page that shows almost nothing without JavaScript.
 - **Cite exact text.** Full-text search (SQLite FTS5) gives ranked passages. A quote check finds the exact span in the stored text, with the capture id to cite.
-- **An MCP server.** One command gives Claude and other MCP clients four tools: fetch, find, read and verify.
+- **An MCP server.** One command gives Claude and other MCP clients nine tools. They fetch, find, read and verify pages, check field values, discover and crawl a site, compare versions and list the watched pages.
+- **Web archives.** An export writes the stored captures as WARC or WACZ. Archive tools read them, for example `warcio`, pywb and ReplayWeb.page.
+- **Find and crawl a site.** `discover` lists the pages of a site from its sitemaps and feeds. `crawl` follows the links of a site within a budget, at the pace of the site.
+- **Watch and compare.** `watch` checks pages again on a schedule and tells what changed. `diff` compares two versions, and `read` takes `at`: the page as it was at a time.
+- **Archived copies, on request.** When a site refuses a page or is down, `fallback: "archive"` gives a copy from the Wayback Machine or Common Crawl. The result marks the copy as archived, with its date.
+- **Signed requests.** With an Ed25519 key, the crawler signs each request (Web Bot Auth). A site can then verify that a request comes from your crawler.
 
 ## Install
 
@@ -51,6 +56,7 @@ npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
 npx polite-crawler find "robots exclusion"
 npx polite-crawler read "url:https://en.wikipedia.org/wiki/Web_crawler" --length 2000
 npx polite-crawler verify "url:https://en.wikipedia.org/wiki/Web_crawler" "robots.txt protocol"
+npx polite-crawler extract "url:https://en.wikipedia.org/wiki/Web_crawler" --field "name=Web crawler"
 ```
 
 The store is in `~/.polite-crawler`. Use `--dir` or `POLITE_CRAWLER_DIR` for a different folder. A path that starts with `~` is in your home folder, and a relative path starts at the working folder. Use `--json` for output that a program reads. `npx polite-crawler --help` shows all the options.
@@ -58,14 +64,46 @@ The store is in `~/.polite-crawler`. Use `--dir` or `POLITE_CRAWLER_DIR` for a d
 - `--allow-private` (or `POLITE_CRAWLER_ALLOW_PRIVATE=1`) lets the crawler fetch loopback, private and link-local addresses. By default, the crawler refuses them.
 - `read --user-provided` reads a page that asked not to be AI input. Use it only for a URL that you gave yourself. The crawler records each such read.
 - `--max-age 0` fetches a page again, also when the store has a copy.
+- `fetch --archive-fallback` gives an archived copy when a site refuses a page or is down. [Archived copies after a refusal](#archived-copies-after-a-refusal) tells the rules.
+- `--sign-key <file>` and `--signature-agent <url>` sign each request. `key-directory --sign-key <file>` prints the key directory to host. [Signed requests](#signed-requests-web-bot-auth) tells how.
+- `find --rank fast` makes a long question faster, and its results can be different. [Fast ranking](#fast-ranking) gives the measured difference.
 - `reindex` builds the index again. The documents of a recipe that the command cannot read (the recipes of another application) stay. The output counts them, and the captures that it skipped.
+- `export --out <file>` writes the stored captures as a WARC or a WACZ file. [Export to a web archive](#export-to-a-web-archive) tells what goes into the file.
+- `extract <doc> --field name=value[:kind]` checks the values that you propose for the fields of a stored page. Repeat `--field` for each field. `--capture` checks the text of one capture, and `--at` the version that was current at a time. [Check field values against a page](#check-field-values-against-a-page) tells the statuses.
+
+To find the pages of a site, or to fetch a part of a site, use `discover` and `crawl`:
+
+```sh
+npx polite-crawler discover https://example.com/
+npx polite-crawler crawl https://example.com/ --max-pages 50 --max-depth 2 --budget 2m
+```
+
+- `discover` prints each URL of the sitemaps and feeds of the site, with its `lastmod`. `--max-urls` limits the list (the default is 1000), and `--feed <url>` reads one more feed.
+- `crawl` prints each page when its fetch ends, and then a summary. The options are `--max-pages` (default 100), `--max-depth` (default 3) and `--budget` (default `10m`).
+- `crawl` also takes `--include` and `--exclude` (patterns, which you can repeat), `--sitemaps`, `--render` and `--max-age`.
+- `discover` and `crawl` take `--allow-host <host>` for more hosts. The first `Ctrl+C` stops a crawl, and the summary prints.
+
+To read a page as it was, to compare two versions, or to watch pages, use `read --at`, `diff` and `watch`:
+
+```sh
+npx polite-crawler read "url:https://example.com/" --at 2026-10-01T12:00:00Z
+npx polite-crawler diff "url:https://example.com/" --from 2026-10-01 --to fetch.5d41402abc4b
+npx polite-crawler watch add https://example.com/hours --every 6h
+npx polite-crawler watch run --budget 2m
+npx polite-crawler watch list
+```
+
+- `read --at` and `verify --at` use the version that was current at that time. [Read a page as it was at a time](#read-a-page-as-it-was-at-a-time) gives the rules.
+- `diff` prints the changed lines as a unified diff. A changed line marks its words with `[-removed-]` and `{+added+}`. `--from` and `--to` take a capture id or a time.
+- `watch add` takes `--every` (1 hour or more, `24h` by default), `--confirm 1` or `2`, `--render`, `--scope` and `--label <name=value>`.
+- `watch run` takes `--budget` (default `10m`) and `--max-pages` (default 100). It prints each check, then a summary. A scheduler of the operator starts it: the package has no daemon.
 
 The command gives these exit codes:
 
 | Code | Meaning |
 | --- | --- |
 | `0` | The command did its work. |
-| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. |
+| `1` | The command did not do its work: a page that the crawler did not fetch, no search result, a quote that is not in the document, a document that is not stored or that is held back, or an error. `discover` gives 1 when it finds no URL, and `crawl` when it gets no page. `read --at` and `verify --at` give 1 for a time before the first capture. `watch run` gives 1 when a check is `refused` or an `error`, or when entries stay due. `watch remove` gives 1 for a URL that is not on the list. With `--archive-fallback`, a page with an archived copy counts as done. `extract` gives 1 when a field is not `verified` or `normalized`. |
 | `2` | The command line is wrong. The command opens no store. |
 
 ## Use it from Claude (MCP)
@@ -106,17 +144,24 @@ claude mcp add polite-crawler -- npx -y -p polite-crawler -p playwright-core pol
 
 When a new version of `playwright-core` comes out, install its browser again with the first command.
 
-The server gives four tools:
+The server gives nine tools:
 
 | Tool | What it does |
 | --- | --- |
-| `crawl_fetch` | It fetches pages into the store, and a page in the store answers with no request. It gives the age of each copy. |
+| `crawl_fetch` | It fetches pages into the store, and a page in the store answers with no request. It gives the age of each copy. With `archive_fallback`, a page that the site refuses gets an archived copy, marked `archived`. |
 | `crawl_find` | It searches the stored pages and gives ranked passages. |
-| `crawl_read` | It gives a range of a stored document, with the capture ids to cite and the position of the next range. |
-| `crawl_verify` | It finds a quote in a stored document and gives the exact span. |
+| `crawl_read` | It gives a range of a stored document, with the capture ids to cite and the position of the next range. With `at`, it reads the version that was current at that time. |
+| `crawl_verify` | It finds a quote in a stored document and gives the exact span. It takes `at` too. |
+| `crawl_extract` | It checks the values that the assistant proposes for the fields of a stored page, and gives a status and a span for each field. It does not make values. With `at`, it checks the version that was current at that time. |
+| `crawl_discover` | It lists the page URLs of a site from its sitemaps and feeds, 1,000 or fewer. It fetches no page of the list. |
+| `crawl_site` | It fetches a few pages of a site and follows their links on the site: 10 pages and 1 minute by default, and 50 pages and 5 minutes at most. It sends progress for each page. |
+| `crawl_diff` | It compares two stored versions of a document, by capture id or by time. The changed lines, the changed words and the changes of the metadata come in the envelope. |
+| `crawl_watch_list` | It lists the watched pages: each URL, the next check, the last result and a suspected change. It leaves out the scope and the labels. |
 
 The MCP server applies the robots.txt rules for `Claude-User` too, because an assistant asks for these pages.
 
+No tool adds a watch. A watch commits future requests to a site, so the operator adds it with `polite-crawler watch add`. An assistant can read the list (`crawl_watch_list`) and compare the versions of a watched page (`crawl_diff`).
+
 ### Web text is data, not instructions
 
 A page can contain text that tries to give orders to the assistant. Thus each tool gives the text of the pages in an envelope:
@@ -154,6 +199,8 @@ The crawler records each release in the capture log.
 | `--allow-private` | `POLITE_CRAWLER_ALLOW_PRIVATE=1` | Fetch loopback, private and link-local addresses too. |
 | `--allow-user-provided` | `POLITE_CRAWLER_ALLOW_USER_PROVIDED=1` | Let `user_provided` work when the client cannot ask the user. |
 | `--no-browser` | | Do not open a headless browser. |
+| `--sign-key <file>` | `POLITE_CRAWLER_SIGN_KEY` | Sign each request with this Ed25519 key (PKCS#8 PEM or JWK). [Signed requests](#signed-requests-web-bot-auth) tells how. |
+| `--signature-agent <url>` | `POLITE_CRAWLER_SIGNATURE_AGENT` | Where a site finds the public key of `--sign-key`: an https origin, or the URL of a JSON Web Key Set. |
 
 ## Use it as a library
 
@@ -182,11 +229,41 @@ await crawler.close();
 - A result of `fetch` gives `capture`, `fetched_at` and `from_cache`. A result of `read` gives the `capture` and the `fetched_at` of its text.
 - `verify` gives the `capture`, the `blob` and the `fetched_at` of the text that holds the quote, with the positions in that text. `stale` is true when only an earlier capture holds the quote.
 - `read` and `verify` take `capture`: they then use the text of that capture, not the current text. A citation of a capture thus stays true after the page changes.
+- `extract` checks the values that you propose for the fields of a page, and gives the span and the capture of each value. [Check field values against a page](#check-field-values-against-a-page) tells how.
 - `versions(doc)` gives the versions of a document, the newest first.
+- `read` and `verify` take `at`: the version that was current at that time. [Read a page as it was at a time](#read-a-page-as-it-was-at-a-time) tells the rules.
+- `diff(doc, { from, to })` compares two versions. [Compare two versions](#compare-two-versions) tells what it gives.
+- `watch` is the watch list: `add`, `remove`, `list` and `run`. [Watch pages](#watch-pages) tells the rules.
 - `find` reads a query syntax. Text in double quotes is a phrase, and `bake*` is a prefix. The documents that hold each word come first. Case, accents and Unicode forms do not change a match, and Chinese, Japanese and Thai words match too.
+- `find` takes `rank`. The default is `exact`, and `fast` makes a long question faster. [Fast ranking](#fast-ranking) tells the difference.
 - A page that asks not to be AI input is held back in each result. `fetch` gives no title, and `read` gives no title, no text and only the structural metadata. `verify` gives `found: false`, and `find` leaves the page out. Only `userProvided` lifts the hold, and the crawler logs it.
 - The hold applies to each version too. When a later capture does not ask it, the current text is not held back, but the earlier versions that asked it stay held back. `verify` with no `capture` does not examine them, `read` and `verify` of their capture hold them back, and `versions` shows no title for them.
 
+### Fast ranking
+
+The default ranking (`rank: "exact"`) gives a BM25 score to each document that matches. A long question is the slowest search, because its common words match almost each document. `rank: "fast"` changes only the second part of a search, the documents that hold one word of the question:
+
+- It leaves out each word that more than 20 percent of the documents hold (`FAST_RANK_SHARE`). When each word is that common, the rarest word stays.
+- A document that holds only common words is not in that part of the results. The common words do not add to the score of that part.
+- The documents that hold each word come first in the two modes, with the same rank.
+
+The command line has `find --rank fast`, and the tool `crawl_find` has the parameter `rank`. `DocIndex.rankWithin` takes `{ rank: "fast" }` too. It then leaves out each document that holds only common words.
+
+The benchmark `bench/find.mts` measured the two modes in CI ([the results](https://github.com/mark1russell7/crawler/actions/runs/39057146797)). The index had 30,000 documents of 900 words, with a Zipf spread of the words. Each time is the median of 12 questions on Node.js 24:
+
+| Search | `exact` | `fast` | Speed | Same top 10 | Same order |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| `find`, 1 word | 3.5 ms | 3.4 ms | 1.0 times | 100 % | 1.00 |
+| `find`, 4 words | 4.4 ms | 4.7 ms | 0.9 times | 98 % | 1.00 |
+| `find`, 16 words | 37 ms | 15 ms | 2.5 times | 85 % | 0.62 |
+| `rankWithin` of 40,000 ids, 16 words | 87 ms | 39 ms | 2.2 times | 85 % | 0.62 |
+
+- "Same top 10" is the share of the exact top 10 that `fast` also gives (overlap at 10). "Same order" is the Kendall tau of the documents that the two lists share. A value of 1 is the same order.
+- In the corpus of the review, almost each word is in almost each document. There, `fast` was 2.2 times faster for 16 words, but its top 10 had only 43 percent of the exact top 10.
+- For 16 words, `rankWithin` with `fast` gave 22 percent of the documents that `exact` gave.
+
+Use `fast` only when the speed of a long question is more important than its exact rank. To measure again, start the workflow `bench.yml`.
+
 The MCP server is in its own entry point:
 
 ```ts
@@ -206,6 +283,7 @@ The root entry point also exports the parts of the runtime, for an application t
 - `catchUpIndex(runtime)`: the step of `openCrawler` at open. It rebuilds a stale index, and gives an index that is behind the log the captures that it missed. Use it after `createRuntime`.
 - `Crawler`, `CaptureLog`, `DocIndex`, `FileBlobStore`, `SqliteHostStateStore`, `Extractors`, `fetchAndIndex`, `fetchPage` and `reindex`.
 - `readDocument`, `verifyQuote`, `findDocuments`, `isHeldBack`, `heldBackMeta` and `recordOverride`: the read, the verification and the rule for a held-back page, over a runtime.
+- `extractDocument` and `checkFields`: the check of field values over a runtime, and over a text and its metadata with no store.
 - `extractPage`, `typesOf`, `decodeBody` and `bodyEncoding`: the page reader and the decoder of a body in its declared encoding.
 - `readPdf` and `isPdf`: the PDF reader. `PdfOptions` sets limits on the pages, the characters, the time and the inflated bytes.
 - `fromText`, `fromSchemaOrg`, `fromCompact` and `describeWeek`: the opening-hours reader.
@@ -215,6 +293,7 @@ The root entry point also exports the parts of the runtime, for an application t
 The readers give more facts than before:
 
 - `PageExtract` has `canonical` (the canonical URL on the same site), `redirect` (the target of a quick `<meta http-equiv="refresh">`), `blocked` (the name of a bot check or a firewall that sent its page in place of the content) and `notes` (what the reader left out).
+- `PageExtract` also has `feeds` (the RSS and Atom feeds that the page links). `extractPage` always gives it. The field is optional, so a `PageExtract` of 0.1.1 stays correct.
 - `Hours` has `partial` (the reader left out a rule or a line) and `season` (the season or the dates of the hours). The text reader reads English only.
 
 ### The options of `Crawler`
@@ -223,13 +302,14 @@ The readers give more facts than before:
 
 - `egress`: the addresses that the crawler fetches. By default, the crawler fetches only public addresses. Set `allowPrivate: true` for loopback, private and link-local addresses, or name hosts and ranges in `allowHosts`. With a dispatcher of your own, put `egressLookup` in its connection options.
 - `politeness`: the limits of the policy, for each host or for some hosts (`hosts: { "*.example.org": { pageIntervalMs: 10_000 } }`). `DEFAULT_POLICY` has the defaults. A value that is not valid throws a `RangeError`.
-- `hostState`: the store of the pace, the counts and the stops of each host (`HostStateStore`). The default keeps them in memory (`MemoryHostStateStore`). `SqliteHostStateStore` keeps them in a file that processes share.
+- `hostState`: the store of the pace, the counts, the stops and the validator memo of each host (`HostStateStore`). The default keeps them in memory (`MemoryHostStateStore`). `SqliteHostStateStore` keeps them in a file that processes share.
 - `onEvent`: a function that gets an event for each request, response, wait, stop and skip.
 - `obeyDefaultToken`: a crawler with a token of your own also obeys the group of `PoliteCrawler` when its `User-Agent` links the bot page of polite-crawler. Site owners read that page. The stricter rules apply. Set false when you host a bot page of your own.
 - `maxRenders`: the pages that the browser renders at one time. The default is 2, because each render holds a browser context. A render waits for a slot before the lease of its host, so the wait is not time of the host.
+- `signer`: the signer of each request (Web Bot Auth). [Signed requests](#signed-requests-web-bot-auth) tells its options.
 - `scope`, `labels` and `scopeRank`: the labels of each record. One fetch of a robots.txt can serve requests of different scopes. Its record takes the labels of each request and the scope with the highest `scopeRank`, the most private one.
 
-`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent` and `extractors`. Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace and the stops of each host.
+`openCrawler` takes `egress`, `politeness` and `hostState` too, and also `clock`, `scope`, `onEvent` and `extractors`. Its default `hostState` is the file `hosts.sqlite` in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.
 
 `FetchRequest.maxAgeMs` can be `"auto"`: the crawler then uses the freshness that the server gave (`Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`). `crawler.resetHosts(host?)` removes the stops and the failure counts of one host or of each host, for a process that runs for a long time.
 
@@ -239,14 +319,443 @@ Before version 1.0.0, a minor version can change the API, and `CHANGELOG.md` tel
 
 These parts are the stable API:
 
-- `openCrawler` and the methods of `PoliteCrawler`: `fetch`, `find`, `read`, `verify`, `versions`, `reindex` and `close`.
-- The options and the results of these methods: `OpenOptions`, `FetchOptions`, `PageResult`, `Found`, `ReadResult`, `VerifyResult`, `DocVersion` and `ReindexResult`.
-- `polite-crawler/mcp`: `createMcpServer`, `startMcpServer` and the four tools.
+- `openCrawler` and the methods of `PoliteCrawler`: `fetch`, `find`, `read`, `verify`, `versions`, `discover`, `crawl`, `reindex`, `close`, `diff`, `watch` and `extract`. `openCrawler` gives an `OpenedCrawler`: a `PoliteCrawler` that has `discover`, `crawl`, `diff`, `watch` and `extract`. The five are optional in `PoliteCrawler`, so an object of your own that implements it stays correct.
+- The options and the results of these methods: `OpenOptions`, `FetchOptions`, `PageResult`, `Found`, `ReadResult`, `VerifyResult`, `DocVersion` and `ReindexResult`. `at` of `ReadOptions` and `VerifyOptions`, and `at` and `before_first` of their results, are stable too.
+- The options and the results of `extract`: `ExtractOptions`, `ExtractResult`, `ExtractSummary`, `FieldClaim`, `FieldKind`, `FieldStatus`, `FieldResult`, `FieldSpan` and `FieldCandidate`, with the names of the normalizations.
+- The diff: `diffDocument`, `diffTexts`, `DIFF_LIMITS`, `DiffOptions`, `DiffResult`, `DiffSide`, `DiffSummary`, `DiffHunk`, `DiffLine`, `WordPart`, `MetaChange`, `VersionChoice`, `DiffLimits`, `DiffInput` and `TextDiff`.
+- The time travel: `versionAtTime`, `isoTime` and `VersionAt`.
+- The watch list: `WatchList`, `WatchOptions`, `WatchEntry`, `WatchPending`, `WatchKind`, `WatchRun`, `WatchRunOptions`, `WatchEvent`, `WatchSummary`, `WatchStop`, `WATCH_LIMITS` and the file `watch.sqlite` (`WATCH_SCHEMA`). For a runtime of your own, it also has `WatchStore`, `WatchStoreOptions`, `runWatch` and `WatchRunnerOptions`. The lease steps of `WatchStore` (`claim`, `release` and `settle`) are the engine.
+- The options and the results of `discover` and `crawl`: `DiscoverOptions`, `DiscoverResult`, `DiscoveredUrl`, `DiscoverSource`, `DiscoverStop`, `CrawlOptions`, `CrawlRun`, `CrawlPage`, `CrawlSummary`, `CrawlStop`, `UrlPattern` and `ScopeOptions`.
+- `polite-crawler/mcp`: `createMcpServer`, `startMcpServer` and its tools.
 - The readers: `polite-crawler/robots`, `/hours`, `/html`, `/classify`, `/politeness` and `/signals`. The site and its tools use them in the browser.
 - The command line, its exit codes, the store on disk and `polite-crawler/record.schema.json`.
+- The export: `exportWarc` and `exportWacz`, their options (`ArchiveOptions`, `ArchiveFilter`, `ArchiveSource`), their results (`WarcResult`, `WaczResult`, `ArchiveCounts`) and the records of the archive.
+- The archived fallback: `fallback` and `archiveDespiteRobots` of `FetchOptions`, the field `archived` of the results, and the types `Archived` and `WebArchive`.
+- Signed requests: `createWebBotAuthSigner`, `keyDirectory`, `verifySignature`, `KEY_DIRECTORY_PATH`, `KEY_DIRECTORY_TYPE`, the option `signer`, and the types `WebBotAuthOptions`, `WebBotAuthSigner`, `RequestSigner`, `KeyDirectory`, `DirectoryKey`, `KeyInput`, `SignatureAgentType`, `SignedRequest` and `SignatureCheck`.
+- The headers of a signature follow the draft. When the draft changes, the headers change with it, and `CHANGELOG.md` says so.
 
 The other exports of the root entry point are the engine. Examples are `createRuntime`, `catchUpIndex`, `Crawler`, `CaptureLog`, `DocIndex`, `FileBlobStore`, `SqliteHostStateStore`, `Extractors`, `fetchAndIndex` and `makeRecord`. An application can build on them. A minor version before 1.0.0 can change them, and its section in `CHANGELOG.md` says so.
 
+The engine of the check of fields is `checkFields`, `extractDocument`, `FIELD_KINDS`, `FieldCheck`, `ExtractSource` and `FieldsChecked`.
+
+The engine of discovery and of the crawl is `discoverSite`, `crawlSite`, `readSitemap`, `feedDate`, `DISCOVER_LIMITS`, `CRAWL_LIMITS`, `SITEMAP_LIMITS` and `DEADLINE_NOTE`. It also includes `Crawler.robotsAllow` and the `deadline` of `FetchRequest`, `RenderRequest` and `PageOptions`.
+
+The engine of the time travel and of the watch list is the optional method `ReadingIndex.versionAt`, `DocIndex.versionAt` and `Crawler.hostStop`.
+
+The engine of the archived fallback and of the signatures is `archiveExtractor`, `archivedDocId`, and the `fallback` and `archiveDespiteRobots` of `PageOptions`. It also includes `range`, `signalLines` and `originalHeaderPrefix` of `FetchRequest`, `ai_input` of `ExtractedDoc`, `sign` of `RenderOptions` and `signer` of `CrawlerOptions`.
+
+## Read a page as it was at a time
+
+Each capture of a document is a version, and the index keeps each version. `read` and `verify` take `at`, a time. They then use the version that was current at that time.
+
+```ts
+const then = await crawler.read("url:https://example.com/", { at: "2026-10-01T12:00:00Z" });
+console.log(then.capture, then.fetched_at, then.before_first, then.text);
+
+const check = await crawler.verify("url:https://example.com/", "open on Sundays", { at: new Date("2026-10-01") });
+```
+
+- `at` is an ISO 8601 text or a `Date`. A date with no time is the start of that day in UTC.
+- The rules that choose the current version apply to the versions of that time. Thus a render stays current while the plain body under it stays the same, and the richer recipe wins.
+- The result names the `capture` and the `fetched_at` of the version, and it gives `at`.
+- When `at` is before the first capture, no version answers. `before_first` is true, and the note gives the time of the first capture. `read` then gives no text, and `verify` gives `found: false`.
+- `verify` with `at` examines that version only, as `verify` with `capture` does. A match is not `stale`.
+- Give `capture` or `at`, not the two. The two together are an error.
+- The hold of a page that asked not to be AI input applies as before. A version that was held back at that time stays held back. When the document is held back at this time, each of its versions is held back too.
+- The command line has `read --at <time>` and `verify --at <time>`. The MCP tools `crawl_read` and `crawl_verify` have `at`.
+
+`find` has no `at`. The search index (FTS5) holds the current text of each document. A search at a time needs an index entry for each version, or a scan of each stored text. Neither is cheap, so the package does not give it.
+
+## Compare two versions
+
+`diff` compares two versions of a stored document. It sends no request.
+
+```ts
+const diff = await crawler.diff("url:https://example.com/");
+console.log(diff.from.capture, diff.to.capture, diff.summary);
+
+const since = await crawler.diff("url:https://example.com/", { from: { at: "2026-10-01" }, to: "fetch.5d41402abc4b" });
+```
+
+- `from` and `to` are a capture id, or `{ at }`: the version that was current at that time. The default `to` is the newest version. The default `from` is the version before `to` of the same recipe and kind (plain or rendered). Thus a render is not compared with the plain body under it.
+- `hunks` give the text as a line diff, in the form of a unified diff, with 2 lines of context. A `changed` line pairs a removed line with an added line. Its `words` give the words that stay, the added words and the removed words.
+- `meta` gives the changes of the title, the URL and the metadata at their paths, for example `meta.jsonld[0].openingHours`. Each change has the JSON of the old and the new value, cut to 300 characters.
+- `summary` counts the added, removed and changed lines, the added and removed words, and the changes of the metadata. `same` is true when the two versions are the same.
+- The command line has `diff <doc> [--from <version>] [--to <version>]`. A version is a capture id or a time. The MCP tool `crawl_diff` gives the changes in the envelope.
+
+The line diff uses the algorithm of Myers. Its time is linear when the change is small. `DIFF_LIMITS` gives the bounds:
+
+| Limit | Value | What it bounds |
+| --- | ---: | --- |
+| `maxChars` | 1,000,000 | The characters of each text. A longer text gives its first part. |
+| `maxLines` | 50,000 | The lines of each text. A longer text gives its first lines. |
+| `maxSteps` | 1,000,000 | The steps of the line diff. After them, the rest of the change is one removal and one addition. |
+| `maxWordSteps` | 1,000,000 | The steps of the word diffs of all the changed lines together. |
+| `maxLineWords` | 5,000 | The words of one line. A longer line gets no word diff. |
+| `maxHunkLines` | 2,000 | The lines that the hunks show. The summary counts each line. |
+| `maxMetaChanges` | 200 | The changes of the metadata that the result shows. The summary counts each change. |
+
+- When a limit applies, the note tells which limit. `exact` is false when the line diff is not minimal or a changed line has no word diff. `truncated` is true when the result leaves out a part.
+- `steps` counts the steps of the diff. The tests use it, because a count of steps does not change with the computer.
+- `limits` in the options changes the bounds of one diff.
+- When a side is held back, or the document is held back, the diff gives no text and no title. It compares only the structural metadata, and `same` is null.
+
+## Watch pages
+
+A watch list keeps the pages that the crawler checks again on a schedule. The list is in `watch.sqlite` in the store.
+
+```ts
+await crawler.watch.add("https://example.com/hours", { every: "6h", labels: { lineage: "venue-17" } });
+
+const run = crawler.watch.run({ budgetMs: 120_000, maxPages: 50 });
+for await (const event of run) console.log(event.kind, event.suspected, event.url, event.diff);
+console.log((await run.summary).due);
+
+console.log(await crawler.watch.list());
+await crawler.watch.remove("https://example.com/hours");
+```
+
+| Option of `add` | Default | What it does |
+| --- | --- | --- |
+| `every` | `24h` | The time between two checks. The floor is `1h`, so a watch cannot hammer a site. `watchFloor` of `openCrawler` changes the floor. |
+| `confirm` | `2` | The number of checks that must see a change before it is confirmed. |
+| `render` | `auto` | As for `fetch`. |
+| `labels`, `scope`, `onBehalfOf` | none | As for `fetch`. They apply to each check. |
+
+`add` of a page on the list changes its options and keeps its state. A new entry is due at once.
+
+`run` checks the entries that are due, the first due first. It gives an async iterable of events, and a `summary`. Its options are `budgetMs` (10 minutes by default), `maxPages` (100 by default) and `signal`.
+
+- Each check is a fetch of the crawler with no stored copy (`maxAge: 0`). Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A stored `ETag` or `Last-Modified` date makes the check a conditional request.
+- A run sends requests to 8 hosts at one time at most, and to each host one request at a time.
+- When the budget ends before the request of a check, the entry stays due for the next run.
+- When a check finds a stopped host (a 403, a challenge, or a `Retry-After` of more than 1 hour), the entry is due again at the end of the stop, or later.
+
+Each event has the URL, the document, the capture of the check, the last confirmed capture (`previous`), the outcome and the kind of the check:
+
+| Kind | What the check saw |
+| --- | --- |
+| `new` | The first version of the page, or a page that came back after it was gone. |
+| `changed` | A version with a different title, text or JSON-LD than the last confirmed version. |
+| `unchanged` | The last confirmed version. |
+| `gone` | A 404 or a 410. |
+| `refused` | No version: robots.txt, the egress policy, a stop of the host or the site refused the request. |
+| `error` | No version: a server error, no answer, or a page that gives no document. |
+
+### The confirm rule
+
+One read can be a glitch: a page of an error, a test of the site, or a cache that is not current. Thus a change counts only when two checks see it:
+
+1. The first check that sees a change gives it with `suspected: true`. The last confirmed version does not change.
+2. The next check confirms the change when it sees the same new version. It gives `confirmed: true`, and the new version becomes the last confirmed version.
+3. When the next check sees the last confirmed version again, it gives `unchanged`, and the note says that the change did not repeat.
+4. When it sees a third version, that version is the new suspected change.
+
+A confirmation needs a new capture: a stored copy does not confirm a change. `gone` and a page that comes back follow the same rule. With `confirm: 1`, each change is confirmed at once. A `refused` or an `error` check changes no state.
+
+The `diff` of an event of the kind `changed` gives the counts of the diff from the last confirmed version. A page that asked not to be AI input gives no diff.
+
+### The list in the store
+
+- The list is the decision of the operator, not a projection of the capture log. Thus it has its own file, `watch.sqlite`. A deletion of the index or of `hosts.sqlite` does not lose it. The file opens at the first use of `crawler.watch`.
+- Each step is one SQLite transaction (`BEGIN IMMEDIATE`) in WAL mode, and it waits for the lock of a different process (`busy_timeout`).
+- A run claims each entry with a lease before its check. The lease holds until the end of the budget and 2 minutes more. Thus two processes that run at the same time check each entry one time. A process that stops holds its leases until they end.
+- The list stays after a restart, with the state of each entry: the last confirmed capture, the suspected change and the counts.
+- The package has no daemon. A scheduler of the operator (cron, a systemd timer or a job of an application) starts `watch.run`.
+- The command line has `watch add <url...>`, `watch remove <url...>`, `watch list` and `watch run`. The MCP tool `crawl_watch_list` lists the entries.
+- No MCP tool adds a watch. A watch commits future requests to a site, so it is an action of the operator, not of an assistant.
+
+## Check field values against a page
+
+A model or a person often reads a value from a page: a price, a date, opening hours, an author or a DOI. `extract` checks each proposed value against the stored copy, and it locates the value. It does not make values. It only checks and locates the values that you give, so each value has its proof.
+
+```ts
+const page = await crawler.fetch("https://example.com/focaccia");
+const result = await crawler.extract(page.doc!, {
+  price: { value: "4.50 EUR", kind: "money" },
+  since: { value: "2026-03-04", kind: "date" },
+  phone: { value: "+1 617 555 1234", kind: "phone" },
+  offer: { value: "4.50", kind: "json-ld:Product.offers.price" },
+  name: { value: "Focaccia" },
+});
+for (const [name, field] of Object.entries(result.fields)) {
+  console.log(name, field.status, field.span ?? field.path, field.normalization, field.capture);
+}
+```
+
+Each field gets one status:
+
+| Status | What it means |
+| --- | --- |
+| `verified` | The stored text holds the exact value. The result gives the span (`start` and `end`) or the JSON-LD path, the stored text (`raw`) and the capture. |
+| `normalized` | The stored text holds the value in a different form. `normalization` names the form, and the span is the span of the stored text. |
+| `ambiguous` | The value matches, but the stored text does not give one meaning. `candidates` gives each place, and the check does not choose one. |
+| `not-found` | The stored text does not hold the value. |
+| `held-back` | The page asked not to be AI input. The result tells nothing about the value. |
+| `invalid` | The value is not a value of its kind, so the check did not search for it. |
+
+The kinds are `text` (the default for a text), `number` (the default for a number), `money`, `date`, `url`, `email`, `phone` and `json-ld:<path>`. A kind that is not a kind throws a `RangeError`. The result also gives `summary`, the count of each status.
+
+### The normalizations
+
+Each normalization has a name. A normalized field gives the names that it used, joined with `+`, for example `thousands-separator+number-form`.
+
+| Kind | Name | What it permits |
+| --- | --- | --- |
+| `text` | `unicode`, `whitespace`, `case` | NFKC forms, soft hyphens, typographic quotes and dashes, runs of spaces, and capital letters. A match starts and ends at the edge of a word. |
+| `number` | `thousands-separator` | A comma, a point, an apostrophe, or a thin or no-break space between groups of three digits. |
+| `number` | `decimal-comma` | A comma in place of the decimal point. |
+| `number` | `number-form` | The same value with other digits: zeros at the start or the end, or a plus sign. |
+| `number` | `unicode` | The same digits in other characters: fullwidth digits, or the minus sign. |
+| `money` | `currency-symbol`, `currency-position`, `amount-only` | A symbol in place of a code (`€` and `EUR`), the currency before or after the amount, and a value with no currency. The names of a number apply to the amount. |
+| `date` | `date-iso`, `date-ymd`, `date-written`, `date-dmy`, `date-mdy`, `date-numeric`, `date-form` | The form of the stored date. `date-written` has an English month. `date-dmy` and `date-mdy` are numeric dates with one possible order. |
+| `date` | `date-partial` | The value has no year or no day, and the stored date has them. |
+| `url` | `url-resolved`, `url-form`, `url-scheme` | A relative URL resolved against the page URL, a different form of the same canonical URL, and a stored URL with no scheme (`www.example.com`). |
+| `email` | `email-case` | Capital letters. |
+| `phone` | `phone-digits`, `phone-suffix` | Only the digits count. With `phone-suffix`, one number ends with the other: a country code, or a number with less detail. |
+
+An amount of money has a currency symbol or a code of ISO 4217 next to it, with one space or none. A code counts only in capital letters, so `15 try` is not an amount.
+
+### The rule for ambiguity
+
+A normalized match must have one meaning. The check does not guess:
+
+- `1,234` and `1.234` can be 1234 or 1.234. Thus a number value that needs a normalization to match them is `ambiguous`. `1,234.5` and `12,50` have one meaning.
+- `03/04/2026` can be 3 April or 4 March. `17/10/2026` and `10/17/2026` have one meaning.
+- `$` is the symbol of more than one currency, so `$12` does not confirm `USD 12`. `US$12` confirms it.
+- `March 4` has no year, so it does not confirm `2026-03-04`.
+- When the value matches two places with different meanings, the field is `ambiguous`, with each place. An example is two phone numbers that end in `555-1234`.
+- A value with two meanings, for example `03/04/2026`, is `ambiguous`, unless the stored text holds its exact text.
+
+An exact match is always `verified`: the stored text holds the text of the value, and the caller gives it its meaning.
+
+### Where the check looks
+
+- The kinds of text read the text of the version, the text that `read` gives. The positions are positions in that text. When the reader cut the text, the check also reads the whole text, and the span then has `in: "full_text"`.
+- `url` also reads the stored links of the page, with the path `links[3]`. The store keeps the canonical form of each link.
+- `json-ld:<path>` reads the JSON-LD of the page. The first part of the path names a schema.org type, and the path starts at each object of that type, at any depth. `*` starts at each top-level record.
+- Each array on a JSON-LD path gives each of its values. The result gives the concrete path, for example `jsonld[0].offers[1].price`, and the raw value. It compares as a number, a date, a URL and a text, in that sequence.
+- With no option, the check reads the current version. `capture` reads the version of one capture, and `at` reads the version that was current at a time. Give `capture` or `at`, not the two.
+
+A page that asked not to be AI input gives `held-back` for each field, with no position and no capture. The result is the same for a value that is in the page and for a value that is not. Thus the check is not an oracle. `userProvided` lifts the hold for a URL that a person gave directly, as for `read` and `verify`. The crawler logs it.
+
+The time of the check is linear in the length of the text, for a fixed set of fields. The tests count the steps of the readers on texts of different lengths.
+
+The MCP tool `crawl_extract` gives the statuses and the positions in its JSON. The stored text, the readings and the excerpts come in the `<untrusted-web-content>` envelope. The command line has `polite-crawler extract`.
+
+## Find the pages of a site
+
+`discover` finds the pages of a site from its sitemaps and its feeds. It fetches the files that list the pages, not the pages.
+
+```ts
+const found = await crawler.discover("https://example.com/");
+for (const url of found.urls) console.log(url.url, url.lastmod, url.via, url.source);
+console.log(found.stopped, found.dropped);
+```
+
+Discovery reads these sources, in sequence:
+
+1. The page at the URL that you give, for the RSS and Atom feeds that it links with `<link rel="alternate">`. With `linkedFeeds: false`, discovery does not fetch the page.
+2. The feeds in `feeds`, and the sitemaps in `sitemaps`.
+3. The `Sitemap:` lines of robots.txt. When robots.txt names no sitemap and `sitemaps` is empty, discovery reads `/sitemap.xml`.
+4. The sitemaps of each sitemap index, the newest `lastmod` first.
+
+Each URL of the result has its `lastmod` and `changefreq` when the file gives them. Its `source` is the file that listed it, and `via` tells if that file is a sitemap or a feed. The newest `lastmod` comes first.
+
+- Discovery keeps a URL only when it is on the same site and robots.txt permits it. The rules for the crawler and for the agents of `onBehalfOf` apply. A host and its `www.` form are one site.
+- `allowHosts` adds more hosts (`*.example.org` names the hosts under a name). `sameSite: false` keeps the URLs of each host.
+- Discovery reads XML sitemaps, sitemap indexes, gzip sitemaps (`.xml.gz`), text sitemaps, RSS 2.0, RSS 1.0 and Atom.
+- Each file is a fetch of the crawler, with the recipe `sitemap` or `feed`, and the capture log records it. A stored copy younger than `maxAge` (24 hours by default) answers with no request.
+- `sources` tells what discovery did with each file. `dropped` counts the URLs that it did not keep, and `stopped` tells why it stopped.
+
+These options limit a discovery. `DISCOVER_LIMITS` gives the defaults.
+
+| Option | Default | What it limits |
+| --- | --- | --- |
+| `maxFiles` | 20 | The sitemap and feed files that discovery reads. |
+| `maxUrls` | 10,000 | The URLs of the result. |
+| `maxBytes` | 10 MiB | The bytes of one file, after decompression. A longer file gives its first part, and its source says `truncated`. |
+| `maxNesting` | 3 | The levels of sitemap index under the first file. |
+| `budgetMs` | 60,000 (1 minute) | The time of the discovery. No request starts after it. |
+
+Discovery also reads robots.txt for 10 origins at most, and 5 feeds at most from the links of the page.
+
+The sitemap reader (`readSitemap`) is strict, because a file from the web can be hostile:
+
+- It expands no entity. It knows only the five entities of XML and the character references, and it ignores a DTD. Thus a "billion laughs" file gives no text.
+- It stops at a depth of 32 elements, and at 50,000 entries (the limit of the sitemap protocol). A value of more than 4,096 characters is not valid.
+- It reads the text one time and builds no tree, so its time is linear in the length of the file.
+- It stops at the first error of the markup, and it keeps the entries before the error.
+
+## Crawl a site
+
+`crawl` fetches the seeds, and then the links of each page on the same site, to a budget. A `for await` loop gets each page when its fetch ends.
+
+```ts
+const run = crawler.crawl("https://example.com/", { maxPages: 50, maxDepth: 2, budgetMs: 120_000 });
+for await (const page of run) console.log(page.depth, page.source, page.outcome, page.url);
+
+const summary = await run.summary;
+console.log(summary.stopped, summary.reason, summary.fetched, summary.refused, summary.skipped);
+```
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `maxPages` | 100 | The most pages that the crawl gives. |
+| `maxDepth` | 3 | The most links from a seed to a page. The value 0 fetches only the seeds. |
+| `budgetMs` | 600,000 (10 minutes) | The time of the crawl. No request starts after it. |
+| `sameSite`, `allowHosts` | the sites of the seeds | The hosts that the crawl follows, as for `discover`. |
+| `include`, `exclude` | none | The patterns of the URLs that the crawl follows. |
+| `useSitemaps` | `false` | The value `true` adds the pages of the sitemaps of each site of the seeds. |
+| `render`, `maxAge`, `onBehalfOf`, `signal`, `scope`, `labels` | as for `fetch` | These options apply to each page. |
+
+- The seeds come first, then the pages of the sitemaps (the newest `lastmod` first), then the links. A lower depth comes before a higher depth.
+- A pattern that starts with `/` matches the path and the query. A different pattern matches the whole URL. A `*` matches any text, and a `$` at the end matches the end, as in robots.txt. A `RegExp` tests the whole URL.
+- The scope and the patterns apply to the links and to the pages of the sitemaps, not to the seeds.
+- The crawl follows only http(s) links, and it fetches each page one time. A fragment, a tracking parameter or a default port does not make a different page.
+- The crawl does not fetch a URL that is the final URL or the canonical URL of a page that it fetched.
+- The links come from the stored document of each page, 100 or fewer for each page. The crawl does not follow the links of a page that asked not to be AI input.
+
+The summary counts the pages (`fetched`, `from_cache`, `refused` and `failed`) and the links that the crawl did not follow (`skipped`, with the reasons in `skips`). `queued` counts the URLs that waited at the stop. `stopped` and `reason` tell why the crawl stopped: `done`, `max-pages`, `budget`, `cancelled`, `closed` or `error`.
+
+### The crawl is polite
+
+The frontier does not change a rule of the crawler, because each page is a fetch of the crawler:
+
+- robots.txt applies to each page. A link that robots.txt refuses costs no request, and the summary counts it in `refused`.
+- One request at a time goes to each host, at the pace of the host. By default, a host rests 3 seconds or more between two pages, so a crawl of one site gets approximately 20 pages in each minute.
+- The frontier sends requests to 16 hosts at one time at most. The crawler also limits the hosts in progress (`maxActive`).
+- The stops, the backoff, the egress policy, the redaction of secrets and the capture log apply as for `fetch`. The `User-Agent` stays the honest default.
+- When the budget ends, no new request starts, and a request in progress ends normally. A request that waits for its host past the budget does not start. Its skip record says `deadline reached`.
+- When `signal` aborts, or a `break` ends the loop, the crawl stops the requests in progress and ends.
+
+The frontier keeps its state in memory, for one call. It keeps 10,000 URLs that wait and 100,000 URLs that it saw, at most (`CRAWL_LIMITS`). A later version can make the frontier resumable.
+
+## Archived copies after a refusal
+
+When a site refuses a page, or the site is down, the crawler can give a copy from a public web archive. The crawler does not show the copy as a fresh fetch, and each result marks it.
+
+```ts
+const page = await crawler.fetch("https://example.com/hours", { fallback: "archive" });
+console.log(page.outcome, page.archived?.source, page.archived?.timestamp, page.doc);
+// blocked wayback 2024-01-15T09:30:00Z archive:https://example.com/hours
+```
+
+The fallback is off by default. The command line has `fetch --archive-fallback`, and the tool `crawl_fetch` has `archive_fallback`.
+
+### When the crawler gives a copy
+
+| The live fetch | An archived copy |
+| --- | --- |
+| robots.txt disallows the page | Only with `archiveDespiteRobots` |
+| A stop of the host, after a 401, a 402, a 403 or a challenge | Yes |
+| A 401, a 402, a 403, or a challenge (also a page that is a bot check) | Yes |
+| A 502, a 503, a 504, or a different 5xx error | Yes |
+| No response (a dead host), or a robots.txt that the crawler cannot read and has no recent copy of | Yes |
+| A 404, a 410, a 429, a cancel, or a refusal of the egress policy | No |
+
+### Where the copy comes from
+
+1. The Wayback Machine. Its availability API (`archive.org/wayback/available`) gives the newest copy, and the crawler fetches the raw copy (`web.archive.org/web/<time>id_/<url>`).
+2. Common Crawl, when the Wayback Machine has no copy. The crawler reads the list of indexes (`index.commoncrawl.org/collinfo.json`) and asks the 3 newest indexes. It fetches the WARC record of the newest copy with a range request (`data.commoncrawl.org`).
+
+- The archives are services of third parties, with their own politeness. Each request to them is a fetch of the same crawler. Their robots.txt, the pace of each host, the egress policy and the identity of the crawler apply.
+- The capture log records each request to an archive, with the recipe `archive-index` (an index) or `archive-copy` (a copy). `reindex` reads the copies again from the log.
+- A stored copy younger than `maxAge` answers with no request.
+- An archive learns the URL of the page. A URL that holds a secret (a password or a secret parameter) does not go to an archive.
+
+### How a result shows a copy
+
+- A copy is a document of its own, `archive:<url>`, of the kind `archived`. The document of the live page (`url:<url>`) does not change.
+- `fetch` keeps the `outcome` and the `status` of the live fetch, for example `blocked` and 403. It adds `archived`: `source` (`wayback` or `commoncrawl`), `timestamp` (when the archive captured the page), `url` (the copy in the archive), `original` (the URL that the archive captured), and `record` for Common Crawl.
+- `read`, `verify` and `find` give `archived` for a copy. The note of `read` and `verify` names the archive and the date. Thus a citation of a copy names the archive and its date, not the date of a fresh fetch.
+- `fetched_at` stays the time of the fetch from the archive. `archived.timestamp` is the date of the copy.
+- `find` with `kind: "page"` leaves the copies out.
+- In the MCP envelope, a copy has the attribute `archived`, for example `archived="wayback 2024-01-15T09:30:00Z"`. Its `age` starts at the date of the copy.
+- With `--archive-fallback`, `fetch` gives the exit code 0 for a page with a copy, and the line of the page starts with `~`.
+
+### robots.txt and the usage signals of a copy
+
+An archive keeps copies of pages that the site refuses today. To read such a copy or not is a decision of policy, and the default is strict:
+
+- With no `archiveDespiteRobots`, the crawler gives no copy of a page that the current robots.txt of its site disallows for this agent. The rules for the agents of `onBehalfOf` apply too.
+- For a stopped host, the crawler reads the rules again: from memory, or from the stored copy of robots.txt of the last 30 days. Thus only a stop, an error or a dead host gets a copy.
+- The reason: robots.txt tells what the owner of the site wants today. A copy of a disallowed page goes around that wish. An owner who disallows a page today does not expect an agent to read it from an archive.
+- `archiveDespiteRobots: true` (`--archive-despite-robots`) gives a copy of these pages too. The robots.txt of each archive still applies to the requests to it. The MCP server has no such parameter, because this decision belongs to the operator.
+- The usage signals of a copy apply as for a live page. A copy is held back from the AI reader when the content signals of the current robots.txt of its site say `ai-input=no`.
+- The headers of the archived response count too: `X-Archive-Orig-Content-Signal` of the Wayback Machine, and the HTTP headers in the WARC record of Common Crawl. The signals of the archive itself count as well.
+
+## Signed requests (Web Bot Auth)
+
+With an Ed25519 key, the crawler signs each request with HTTP Message Signatures (RFC 9421), as the IETF Web Bot Auth draft tells. A site can then verify that a request comes from the holder of your key. The package follows `draft-ietf-webbotauth-httpsig-protocol-00` (1 September 2026).
+
+```ts
+import { readFileSync } from "node:fs";
+import { openCrawler } from "polite-crawler";
+
+const crawler = openCrawler({
+  dir: "./crawl",
+  signer: {
+    privateKey: readFileSync("bot-key.pem", "utf8"), // PKCS#8 PEM, a JWK or a KeyObject
+    signatureAgent: "https://bot.example.com", // the origin of your key directory
+  },
+});
+```
+
+Each request gets three headers:
+
+```text
+Signature-Agent: sig1="https://bot.example.com"
+Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1760702400;keyid="poqkLGiy...";alg="ed25519";expires=1760702700;nonce="...";tag="web-bot-auth"
+Signature: sig1=:...:
+```
+
+- The signature covers `@authority` and the member of `Signature-Agent`. `keyid` is the JWK thumbprint of the key (RFC 8037). Each signature has a new random nonce, and it expires after 5 minutes.
+- The crawler signs each request: robots.txt, each redirect, the sitemaps, the archives of the fallback and each request of a rendered page. Without a signer, it signs nothing. The browser does not sign a WebSocket connection.
+- A signature adds an identity. The `User-Agent` and `From` headers do not change. With a signer, the crawler replaces the signature headers of a caller.
+- A signer that fails sends no request: the result is a skip with a note.
+- The signature headers are not secrets. The capture log records `signed`, with the key id and the agent. It does not hold the key. The signer keeps the key in a closure, and no property of it shows the key.
+
+`createWebBotAuthSigner(options)` makes a signer. `signer` also takes these options, or a `RequestSigner` of your own (for example, a signer with a key in a hardware module).
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `privateKey` | | The Ed25519 private key: a `KeyObject`, PKCS#8 (PEM text or DER bytes) or a JWK. |
+| `signatureAgent` | | Where a site finds your public key. For the type `directory`, an https origin. |
+| `agentType` | `directory` | `directory`: the key directory is at the well-known path of the origin. `jwks_uri`: the URL is a JSON Web Key Set on any path. |
+| `keyId` | the thumbprint | A check: a value that is not the thumbprint of the key throws an error. |
+| `lifetimeMs` | 300,000 (5 minutes) | The time that a signature stays valid, 24 hours at most. |
+| `label` | `sig1` | The label of the signature and the key of the member of `Signature-Agent`. |
+
+`verifySignature(request, directory)` verifies the signature of a request with a key directory, and tells why a request does not verify. Use it to test your setup.
+
+Make a key with OpenSSL, and give it to the command line. Do not use the test keys of RFC 9421 for a real crawler.
+
+```sh
+openssl genpkey -algorithm ed25519 -out bot-key.pem
+npx polite-crawler key-directory --sign-key bot-key.pem --signature-agent https://bot.example.com > http-message-signatures-directory
+npx polite-crawler fetch https://example.com/ --sign-key bot-key.pem --signature-agent https://bot.example.com
+```
+
+The command line takes `--sign-key` and `--signature-agent` (or `POLITE_CRAWLER_SIGN_KEY` and `POLITE_CRAWLER_SIGNATURE_AGENT`) for `fetch`, `discover`, `crawl`, `watch run` and `mcp`. A `--signature-agent` with a path gives the type `jwks_uri`.
+
+### Host the key directory
+
+A site finds your public key from `Signature-Agent`. With the type `directory`, the key directory is at `/.well-known/http-message-signatures-directory` on the origin of `signatureAgent`.
+
+1. Make the JSON with `keyDirectory([publicKey])` or `polite-crawler key-directory`. It holds only the public part of each key, with its thumbprint as `kid`.
+2. Serve it at `https://<your origin>/.well-known/http-message-signatures-directory`. Send the status 200 and `Content-Type: application/http-message-signatures-directory+json`. Do not redirect it, because a verifier follows no redirect.
+3. When you change the key, keep the old key in the directory until its last signature expires.
+
+The draft recommends that the server of the directory also signs its response with each key, as a proof of possession. `keyDirectory` gives only the JSON, so a static host gives no such proof.
+
+GitHub Pages cannot serve a key directory of the type `directory` for a project site:
+
+- A project site (`<user>.github.io/<repository>`) serves files only below its path, so it cannot serve the well-known path at the root of the host.
+- GitHub Pages sets the content type from the extension of the file, so it cannot send the type of the directory.
+- On a project site, put the JSON in a file, for example `jwks.json`. Give its URL as `signatureAgent`, with `agentType: "jwks_uri"`. The draft then binds the key to that URL, not to the domain.
+- A host that sets headers (the `_headers` file of Cloudflare Pages or Netlify, for example) can serve the type `directory` at the root of a domain.
+
 ## How the crawler is polite
 
 1. **robots.txt.** The crawler reads the robots.txt of each site and obeys it, as RFC 9309 specifies. A 401 or a 403 for robots.txt means that the site has no file. When the crawler cannot read robots.txt (a dead domain, a server error or a body that does not decode), it fetches nothing from the site. Then it tries again after some minutes, with a backoff from `Retry-After` up to 1 hour. A good copy of the file from the last 30 days applies in that time.
@@ -256,6 +765,8 @@ The other exports of the root entry point are the engine. Examples are `createRu
 5. **An honest `User-Agent`.** The default is `PoliteCrawler/0.1.0 (+https://mark1russell7.github.io/crawler/bot/)`. The `contact` option adds an address. An email address also goes into the `From` header, and a URL does not, because `From` holds only a mailbox. The constructor of `Crawler` throws an error for a bad option. Examples are a token that is not a product token, a `User-Agent` or a `From` that HTTP cannot send, and a `maxActive` less than 1.
 6. **Public addresses only.** By default, the crawler fetches only `http:` and `https:` URLs on public addresses. It checks the address after DNS and at each redirect. In the headless browser, it checks each request of the page and each redirect of that request. A refusal is a skipped fetch with the note `egress policy: <address> is a loopback address; set allowPrivate to fetch it`.
 7. **Usage signals.** When a page asks not to be AI input (`ai-input=no` content signals), the crawler keeps it but holds back its text from the AI reader. The crawler also records the RSL `License` lines of robots.txt and the TDMRep reservation (the `tdm-reservation` and `tdm-policy` headers and `<meta>` tags) with each capture. These records change nothing, because a reservation of text and data mining is not a refusal of AI input.
+8. **Signed requests, as an option.** With a `signer`, each request has a Web Bot Auth signature, so a site can verify where it comes from. [Signed requests](#signed-requests-web-bot-auth) tells how.
+9. **Archived copies, as an option.** `fallback: "archive"` reads a copy from a web archive only after a refusal. By default, it gives no copy of a page that the current robots.txt disallows.
 
 To block the crawler, a site owner adds this group to robots.txt:
 
@@ -264,6 +775,21 @@ User-agent: PoliteCrawler
 Disallow: /
 ```
 
+## Stored copies and conditional requests
+
+A fetch uses the stored copy of a URL, with no request, while the copy is younger than `maxAge` (24 hours by default). With `maxAge: "auto"`, the crawler uses the freshness that the server gave: `Cache-Control`, `Expires`, or 10 percent of the age of `Last-Modified`. With `"auto"`, `no-store` and `no-cache` make a copy stale at once. A fixed `maxAge` ignores them. A response that varies on a request header (`Vary`) is used again only for a request with the same value of that header. The store keeps one copy for each URL, so a request with a different value replaces it.
+
+When the stored copy is too old, the crawler sends a conditional request. `If-None-Match` sends the `ETag` of the copy, and `If-Modified-Since` sends its `Last-Modified` date. A `304 Not Modified` sends no body, and the crawler uses the stored body again. The capture log records each request, with the validators that it sent (`validators`).
+
+A server examines `If-None-Match` first, and it ignores `If-Modified-Since` when the request has the two (RFC 9110, section 13.1.3). Some hosts give a different `ETag` at each response, for example one for each edge server. Such a host does not answer 304 to the two validators, also when its `Last-Modified` date is stable. Thus the crawler keeps a validator memo for each host:
+
+1. After 3 responses in sequence that send the stored body again, with `If-None-Match` and a `Last-Modified` date, the crawler stops the `ETag` for the host. It then sends only `If-Modified-Since`.
+2. After 20 such requests, or after 7 days, the next request sends the `ETag` again (a probe). A 304 to the probe starts the `ETag` again. A 200 keeps it stopped until the next probe.
+3. When the date alone also fails 3 times in sequence, the crawler sends the two validators again, because they cost nothing. The memo records that neither validator works.
+4. A changed body tells nothing about a validator. The robots.txt of a host is not part of its memo, because it is often a static file with validators of its own.
+
+The memo is in the host state store. `openCrawler` keeps it in `hosts.sqlite`, so the processes on one store share it, and it stays after a restart. `MemoryHostStateStore` keeps it in memory. A `HostStateStore` of your own gets a memo when it has the optional methods `validators` and `noteValidators`. `nextValidatorMemo` and `VALIDATOR_MEMO` give the rules and the limits.
+
 ## The store
 
 | Path | What it holds |
@@ -271,7 +797,8 @@ Disallow: /
 | `captures/fetch/<yyyy>/<mm>.ndjson` | The capture log: one line for each fetch decision, and the declined fetches too. |
 | `blobs/` | The bodies of the responses, by SHA-256 hash. |
 | `index.sqlite` | The search index. It is a projection: `polite-crawler reindex` builds it again from the other two. |
-| `hosts.sqlite` | The pace, the counts and the stops of each host. The processes on one store share it. |
+| `hosts.sqlite` | The pace, the counts, the stops and the validator memo of each host. The processes on one store share it. |
+| `watch.sqlite` | The watch list and the state of each watch. The processes on one store share it. The file opens at the first use of `crawler.watch`. |
 
 The package publishes the JSON Schema of a line of the log: `polite-crawler/record.schema.json`. A program that reads the log can check each record with it.
 
@@ -285,6 +812,68 @@ The package publishes the JSON Schema of a line of the log: `polite-crawler/reco
 - A capture does not hold the secrets of a URL. The crawler writes `REDACTED` in place of a user name and a password. It does the same for the values of the usual secret query parameters (`DEFAULT_SECRET_PARAMS`) and of each name in `secretParams`.
 - The crawler keeps no more than `maxBytes` of a body (5 MiB by default) after decompression. A longer body keeps its first part, and the capture says `truncated`. A body that does not download or decode gives the outcome `error`.
 
+## Export to a web archive
+
+The store is private evidence, and it stays on your disk. An export is a decision of the operator: it copies the captures that you select into a standard web archive. Archive tools read it, for example `warcio`, pywb and ReplayWeb.page.
+
+```sh
+npx polite-crawler export --out crawl.wacz
+npx polite-crawler export --out site.warc.gz --host example.com --since 2026-10-01
+```
+
+```ts
+import { exportWacz, exportWarc } from "polite-crawler";
+
+await exportWarc("./crawl", { out: "crawl.warc.gz", host: "example.com" });
+
+const result = await exportWacz(crawler.runtime, { out: "crawl.wacz", since: "2026-10-01" });
+console.log(result.responses, result.revisits, result.pages);
+```
+
+- `exportWarc` writes WARC/1.1 (ISO 28500:2017). Each record is one gzip member, so the file is a `.warc.gz`.
+- `exportWacz` writes WACZ 1.1.1: a ZIP with the WARC file, a sorted CDXJ index, a page list and `datapackage.json`.
+- The source is the folder of a store, or `crawler.runtime`. The export only reads the store. It makes no store, and it sends no request.
+- These filters select the captures: `since` (this time or later), `until` (before this time), `host` (the host and its subdomains), `urlPrefix` and `ids`. The command line has `--since`, `--until`, `--host`, `--prefix` and `--id`.
+- The export reads the capture log as a stream and holds one body at a time. It sorts the index and the page list in files on the disk. Thus its memory does not grow with the store.
+- The export writes a temporary file and gives it its name at the end. A failed export leaves no part of a file. When the file exists, the export fails. Set `overwrite` (`--overwrite`) to replace it.
+- The MCP server has no export tool. An export writes files on the disk of the operator, so it is an action of the operator, not of an assistant.
+
+### What goes into the archive
+
+| Capture | Records |
+| --- | --- |
+| A response with a stored body | A `response` record with the status line, the headers and the body. Then a `metadata` record. |
+| A `304 Not Modified` | A `revisit` record with the profile `server-not-modified`, and a `metadata` record. |
+| A page that the headless browser rendered | A `conversion` record with the page after JavaScript, and a `metadata` record. |
+| A capture with no body: a refusal, a skip of robots.txt, an error or a redirect | No record. With `includeRefusals` (`--include-refusals`), a `metadata` record alone. |
+
+- The first record is a `warcinfo` record. It names the software, its version and the filters of the export.
+- The `metadata` record of a capture (`WARC-Concurrent-To`) holds the JSON of the capture. It has the id, the kind, the URL, the times, the content hash and the payload. The payload has the outcome, the note, the robots.txt check and the usage signals. A reader can calculate the content hash again and check the capture id.
+- Each record of a capture has the field `WARC-Polite-Crawler-Capture` with the capture id. The record ids come from the hash of the capture, so a second export gives the same ids.
+- The digests are SHA-256 in base 32: `sha256:` and 56 characters, the form that `warcio` uses. `WARC-Payload-Digest` is the digest of the stored body.
+- A revisit names its original (`WARC-Refers-To` and `WARC-Refers-To-Date`) when the original is in the same export. Its `WARC-Payload-Digest` is the digest of the body that the 304 confirmed.
+- A body that the crawler cut at `maxBytes` has `WARC-Truncated: length`. A compressed body that ended early has `WARC-Truncated: disconnect`.
+- The pages that ask not to be AI input go into the archive too. An archive is not AI input. Their usage signals are in the `metadata` record, so a reader of the archive can obey them.
+
+### What does not go into the archive, and why
+
+- **The raw headers.** A capture keeps only some headers of the response: `Content-Type`, `ETag`, `Last-Modified`, `Cache-Control`, `Expires`, `Age`, `Date` and `Vary`. Thus the export rebuilds the status line and the header block from them, and it adds the `Content-Length` of the stored body. Such a record has the field `WARC-Polite-Crawler-Headers: rebuilt`, and its `metadata` record says `"http_headers": "rebuilt"`.
+  - The stored body is the decoded body. Thus the block has no `Content-Encoding` and no `Transfer-Encoding`.
+  - The status line says `HTTP/1.1`, also when the server answered over HTTP/2.
+  - The export writes a `response` record and not a `resource` record. Replay tools (pywb, ReplayWeb.page) replay a `response` with its status and its headers, and a `resource` record has neither. Browser crawlers also write the headers that they have, with a decoded body.
+- **The request.** A capture does not keep the headers of the request (`User-Agent` and `From`). Thus the export writes no `request` record. The `metadata` record gives the validators that the request sent.
+- **The IP address.** A capture does not keep the address of the server. Thus no record has `WARC-IP-Address`.
+- **The labels and the scope of the application.** They can name a private lineage, so the `metadata` record leaves them out.
+- **The other records.** The overrides of a held-back page and the records of the other sources of the log stay in the store.
+
+### The index and the pages of a WACZ file
+
+- `archive/data.warc.gz` is stored with no more compression, so a replay tool reads a record at its offset.
+- `indexes/index.cdxj` has one line for each `response` and `revisit` record, sorted by SURT key and time. The `conversion` records are not in the index, so a replay tool shows only the responses of the servers.
+- `pages/pages.jsonl` has one page for each document: each URL with a `200` response, except robots.txt files. The page names its latest capture.
+- `datapackage.json` gives the SHA-256 and the size of each part. `datapackage-digest.json` gives the SHA-256 of `datapackage.json`.
+- The ZIP has Zip64 records only when a part or the file is larger than 4 GiB.
+
 ## Contributing
 
 [`CONTRIBUTING.md`](https://github.com/mark1russell7/crawler/blob/main/CONTRIBUTING.md) tells how to set up the repository and test a change. [`CHANGELOG.md`](https://github.com/mark1russell7/crawler/blob/main/CHANGELOG.md) records the changes in each version. Report a vulnerability privately, as [`SECURITY.md`](https://github.com/mark1russell7/crawler/blob/main/SECURITY.md) tells.