@opencraw/mcp 0.1.45 → 0.1.47

package/docs/recipes/authoring.md

@@ -0,0 +1,1503 @@
+<p align="center">
+  <img src="../assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
+# Authoring a recipe: the reference
+
+A crawl is described by JSON files, never code (you may still generate that JSON from code: see [programmatic-authoring.md](./programmatic-authoring.md)). **One output recipe** declares the records you want.
+**One or more input recipes** declare how to get them from a site. The engine runs the inputs one after
+another and writes records that all match the output. Anything the JSON cannot express goes into a
+**hook**, a named function you register in code.
+
+The mental model: every step fills one named cell (its `id`), later steps read earlier cells by name, and the
+mapping at the end says which cells become which output fields. Navigation is the order of `goto` /
+`request` steps; the link between two pages is an id rendered into the next URL.
+
+Point `$schema` at `packages/core/schemas/output-recipe.schema.json` or `input-recipe.schema.json` and the
+editor validates and completes as you type. Every recipe is validated on load; unknown keys are errors, so a
+typo never silently does nothing.
+
+## File names
+
+Name a recipe file for its kind: `<id>.ocw-in.json` for an input recipe and `<id>.ocw-out.json` for an
+output recipe (`shop-web.ocw-in.json`, `shop.ocw-out.json`). They stay plain JSON, so every editor
+and tool still reads them, and the VS Code extension validates each against its schema with no `$schema`
+line. The name is a hint that has a rule behind it: `kind` inside the file is what the engine goes by, and a
+file named for one kind that holds the other is an error.
+
+A folder that holds any `.ocw-in.json` or `.ocw-out.json` file is read for those, plus any other `.json` or
+`.jsonl` file whose `kind` is `input` or `output`; everything else beside them (`package.json`, data) is left
+alone. A folder with none is read as it always was: every `.json` and `.jsonl` file is a recipe. Existing
+`*.input.json` and `*.output.json` names keep working.
+
+This page says what each key means. [How OpenCraw works](../how-it-works/README.md) runs each of them on real
+sites, APIs and documents and shows what the engine did: the scope, the mapping steps, the trace and the record.
+
+---
+
+## 1. The output recipe
+
+```json
+{
+  "$schema": "../../packages/core/schemas/output-recipe.schema.json",
+  "kind": "output",
+  "id": "filmography",
+  "version": 1,
+  "description": "One film per record from the lead actor's filmography.",
+  "onMissing": "null",
+  "fields": {
+    "actor":      { "type": "string", "required": true, "key": true },
+    "title":      { "type": "string", "required": true, "key": true },
+    "year":       { "type": "integer", "nullable": true, "min": 1880, "max": 2100 },
+    "role":       { "type": "string", "nullable": true },
+    "filmUrl":    { "type": "url", "required": true },
+    "source":     { "type": "string", "generated": "recipeId", "key": true },
+    "scrapedAt":  { "type": "datetime", "generated": "now" }
+  }
+}
+```
+
+| Key | Meaning |
+|---|---|
+| `id` | Lowercase letters, digits, hyphens. Input recipes name it in their `output`. |
+| `version` | A positive integer you bump when the shape changes. |
+| `fields` | Name → field spec. A name is letters (either case), digits, `_` and `-`, not starting with a digit. Names have no dots; nesting is expressed with `type: "object"`, whose member names follow the same rule. A dotted name fails loading with `fields["stock.count"]: a field name has no dots`. |
+| `onMissing` | Recipe-wide default for a missing value: `fail`, `skip-record` or `null`. See §7. |
+
+### 1.1 Field spec
+
+| Key | Meaning |
+|---|---|
+| `type` | `string`, `number`, `integer`, `boolean`, `date`, `datetime`, `currency`, `url`, `enum`, `array`, `object`, `json`. |
+| `required` | Missing value fails the recipe (unless a policy says otherwise). Default `false`. |
+| `nullable` | Allows `null` on a `required` field when the resolved policy is `null` (§7). A field that is not required is `null` when missing anyway. |
+| `default` | Used when the value is missing and the resolved policy is `default` (it is, automatically, when a default exists and nothing else is said). Coerced and validated like a mapped value when the recipes load, so `"default": "0"` on a `number` field is `0`, and a default that can't be the field's type is a load error. |
+| `onMissing` | Per-field policy: `fail`, `skip-record`, `null`, `default`. `default` on a field without a `default` is a load error. |
+| `key` | Part of the record identity. All `key` fields together form the key; repeated keys are dropped as duplicates. |
+| `generated` | `now` (ISO instant), `uuid`, `sourceUrl` (the page the record was emitted from), `recipeId`. Engine-supplied; mapping it is a binding error. |
+| `format` | Input format for `date` / `datetime` (tokens `YYYY MM DD HH mm ss`); output is always ISO 8601. |
+| `currency` | ISO 4217 code for `currency` fields. When set, it overrides any code or symbol in the value. |
+| `values` | Allowed values for `enum`. |
+| `items` | Element spec for `array`. |
+| `fields` | Member specs for `object`, named like the top-level fields (no dots). A `json` field takes none. |
+| `min`, `max` | Numeric bounds. |
+| `minLength`, `maxLength` | String length or array length bounds. |
+| `pattern` | A regular expression the string must match. |
+
+### 1.2 What each type accepts (coercion)
+
+After the transform chain, the value is converted to the field's type. Conversion is forgiving where a site's
+text is predictable, strict where a wrong value would poison the data.
+
+| Type | Accepts | Produces |
+|---|---|---|
+| `string` | text, numbers, booleans | text. A list is an error: extract one value or `join` it. |
+| `number` | text holding **one** number, with a sign, a currency symbol or code, words and thousands separators around it (`"1.299,00 €"`, `"$1,299"`, `"In stock (22 available)"`), numbers | a number. Text with two numbers or more (`"2 for 10,00"`, an id like `"a897fe39b1053632"`) is an error that names the text: pick the one you want with a `regex` transform first. Without a `number` transform's `locale` the decimal separator is guessed: of two kinds of separator, the last one (`1.299,50` → 1299.5); a single separator is a thousands separator only when exactly three digits follow it and one to three digits other than a lone `0` precede it (`1.299` → 1299, but `0.125` → 0.125 and `10,00` → 10); a repeated one groups thousands (`1,234,567`). Separators that make no number (`1.2.3`) are an error. |
+| `integer` | as `number` | truncated. Errors name `integer`. |
+| `boolean` | booleans, numbers, text | `true` when the whole text is `true`, `yes`, `y`, `1` or `on`, or it contains the words `in stock` or `available` (case-insensitive). So `"none"`, `"2021"` and `"unavailable"` are `false`. Negation is not read: `"not available"` is `true`. For other phrases, use the `boolean` transform with `truthy`. |
+| `date` | ISO text, `Date.parse`-able text, epoch ms, or text matching `format` | `YYYY-MM-DD` (UTC). |
+| `datetime` | same | ISO 8601 instant. Text that names no zone (`Z`, `+02:00`, `GMT`) is read as UTC, or in the `date` transform's `timezone`: never in the zone of the machine running the crawl. |
+| `currency` | text with **one** amount and a symbol or code, a number, or `{ amount, currency }` | `{ "amount": 1299, "currency": "EUR" }`. The code comes from the field, else the transform, else an ISO 4217 code next to the amount (`USD 12.50`), else a symbol, else an ISO 4217 code anywhere in the text; three capitals that are not an ISO 4217 code (`OTR £34,000`) are ignored. No code at all is an error. |
+| `url` | an absolute URL string | normalised. A relative link is an error: use the `absoluteUrl` transform first. |
+| `enum` | text among `values` | the text. |
+| `array` | a list, or a single value (wrapped) | a list; each item coerced and validated by `items`. The members of an object item (an `each` rule's) get the missing-value policy, `default` and validation like any field, reported at their index (`variants[1].size`); a missing scalar item is `null`. |
+| `object` | an object | only the declared `fields`, each coerced. **Undeclared keys are dropped.** |
+| `json` | any JSON value: an object of any shape, a list, text, a number, a boolean | the value, **verbatim**: every key kept, however nested, nothing coerced. Only what JSON cannot hold (`NaN`, `Infinity`, a `Date`, a function) is an error. |
+
+Blank text (`""`, or only spaces) in a field of any type but `string` and `json` is no value rather than a
+value that fails to convert: it follows the missing-value policy (§7), so `default` and `null` apply. In a
+`string` or `json` field, `""` is missing too.
+
+`object` is for a shape you know: it validates it and keeps nothing else. `json` is for a payload whose
+shape you don't know or don't want to fix: archival, a schema that evolves, "land it now, parse it later".
+It still takes `required`, `nullable`, `onMissing`, `default` and `key` like any other field, and it takes
+no `fields` or `items`. Landing a whole API response as one field:
+
+```json
+{ "kind": "output", "id": "configuration-raw", "version": 1, "fields": {
+  "configurationId": { "type": "string", "key": true, "required": true },
+  "fetchedFrom":     { "type": "url", "generated": "sourceUrl" },
+  "payload":         { "type": "json", "required": true }
+}}
+```
+
+```json
+"steps": [
+  { "type": "request", "id": "configuration", "url": "{{start.url}}", "as": "json" },
+  { "type": "emit" }
+],
+"mapping": {
+  "configurationId": { "from": "configuration.id" },
+  "payload":         { "from": "configuration" }
+}
+```
+
+### 1.3 Keys and duplicates
+
+The key is the JSON of the key fields' values in declaration order. De-duplication scope is a run option:
+`run` (default: across every input recipe, first record wins), `recipe`, or `off`. A record without key
+fields is never de-duplicated. Duplicates are reported (`record:duplicate`) and counted, not written.
+
+---
+
+## 2. The input recipe
+
+```json
+{
+  "$schema": "../../packages/core/schemas/input-recipe.schema.json",
+  "kind": "input",
+  "id": "tmdb",
+  "output": "filmography",
+  "mode": "api",
+  "description": "why this route works on this site",
+  "start": [{ "url": "https://www.themoviedb.org/movie/949-heat" }],
+  "vars": { "lang": "en" },
+  "session": { "userAgent": "...", "headers": { "accept-language": "en-US,en;q=0.9" } },
+  "limits": { "maxRecords": 10, "delayMs": 1000, "timeoutMs": 30000 },
+  "onError": { "policy": "fail" },
+  "steps": [ ... ],
+  "mapping": { ... }
+}
+```
+
+| Key | Meaning |
+|---|---|
+| `id` | Lowercase letters, digits, hyphens; unique in a run. Also the value of `generated: "recipeId"`. |
+| `output` | The output recipe id this recipe feeds. Loading fails if it does not match. |
+| `mode` | `web`: a real browser page (Playwright). `api`: HTTP requests through Playwright's request context, no browser. See §2.1. |
+| `start` | One or more start points `{ url, vars? }`. Each runs the whole step list from a fresh scope with `start.url` and its `vars`. A start `url` must be absolute (`https://…`, or a `file:` URL, §3.2): there is no page before it to resolve a relative one against, so `./listino.csv` fails loading with a message saying so. |
+| `vars` | Values templates read as `{{vars.name}}`. Start-point `vars` override recipe `vars`. |
+| `window` | Worker mode only ([worker-mode.md](./worker-mode.md)): `{ check?, maxItems? }`. `check` is an element a reused window's page must still show before an item runs on it, else the window is replaced; after `maxItems` items the window is replaced anyway. |
+| `matrix` | Runs the recipe once per set of vars, each run with its own report. An object of lists runs every combination, the first var slowest: `{ "state": ["Delhi", "Goa"], "year": [2025, 2026] }` is four runs. A list of objects runs those sets as given: `[{ "state": "Delhi" }, { "state": "Goa", "year": 2026 }]`. Each set overrides `vars`, so every name must be declared in `vars` (with the value a run without the matrix uses). The runs of one recipe go one after the other; `parallel` spreads different recipes. Reports and the trace name each run's vars (`▶ report [state=Delhi, year=2025] (web)`). |
+| `session` | §2.2. |
+| `limits` | `maxRecords` stops the walk after that many records, exactly, whatever runs in parallel. `delayMs` is the minimum interval between two request starts across the recipe. `timeoutMs` bounds navigations, requests and `wait` steps. `concurrency` (default `1`) is how many `forEach` iterations over a list may run at once: requests in `api` mode, tabs in `web` mode (§3.9). `retry` sends a request that fails in passing again (§7; on by default). |
+| `onError` | The default policy for every step. §7. |
+| `steps` | The acquisition recipe. §3. |
+| `mapping` | Output field → mapping rule. §5. |
+| `description` | Free text. Write down *why the route works*: which page is public, which element is the lead actor, which table is the acting one. Future you will thank you when the site changes. |
+
+### 2.1 Choosing the mode
+
+Try `api` first. It is faster, needs no browser, and most sites still serve their data in the HTML
+(server-rendered markup, `<script type="application/ld+json">`, inline JSON). Use `web` when the content is
+built by JavaScript after load, when you must click or scroll to reveal it, or when the site challenges plain
+HTTP clients (bot walls). A recipe cannot mix modes in its steps, but an `api` recipe can run **browser
+steps in its bootstrap** to obtain a session (§2.2).
+
+### 2.2 Session
+
+| Key | Meaning |
+|---|---|
+| `headers` | Extra HTTP headers, both modes. |
+| `userAgent` | Sent as the user agent, both modes. Sites answer a Chrome UA more readily than a bare one. |
+| `cookies` | Cookies to start with (Playwright's shape: `name`, `value`, `domain`, `path?`, ...). |
+| `viewport` | `{ width, height }`, web mode. |
+| `storageStatePath` | A file saved by a previous bootstrap (`saveTo`); the recipe starts from it and skips the bootstrap. |
+| `bootstrap` | `{ steps, keep, saveTo? }`. Runs `steps` in a browser **before** the crawl, then captures what `keep` lists (`cookies`, `localStorage`). A `web` recipe starts its page from that state; an `api` recipe sends those cookies with every request. Bootstrap steps run in a browser, with the steps a `web` recipe allows, and never emit. |
+| `access` | `{ profile?, country?, sticky? }`: what the site needs from the network. `profile` names a profile of the runner's access config (its default when omitted), `country` is a two-letter code for profiles that target by country, `sticky: false` lets the provider change IP per request. Never credentials: those live in the access config. See [access.md](./access.md). |
+| `blockedWhen` | `{ status?, header?, text? }`: what counts as the site refusing the crawl, checked on every navigation and request. Default: 403, 429, or an AWS WAF challenge header. A block fails the step with the URL and reason instead of a later selector miss. |
+| `onBlock` | `{ rotate?, solve?, attempts? }`. `rotate: true`: on a block, take a new access lease (a new IP), reopen the session (re-running the bootstrap) and retry the step, up to `attempts` times per run (default 2). `solve: true` (web mode, needs `captcha`): a block page showing a captcha is solved on the spot; rotation, if on, applies only when solving fails. |
+| `browserProfile` | A persistent browser profile of the runner (cookies, storage, cache kept between runs): web recipes and bootstraps run in it, api recipes start from its cookies. See [access.md](access.md#persistent-browser-profiles). |
+| `captcha` | `{ solver, detect?, verify?, attempts?, timeoutMs?, maxSolves? }`, web mode and bootstraps: after every navigation, click and key press, a visible challenge is solved by the named solver before the next step runs. §6.1. |
+
+A login looks like this and works for both modes:
+
+```json
+"session": { "bootstrap": { "keep": ["cookies"], "saveTo": "state/shop.json", "steps": [
+  { "type": "goto", "url": "https://shop.example/login" },
+  { "type": "fill", "selector": "#user", "value": "{{vars.user}}" },
+  { "type": "fill", "selector": "#pass", "value": "{{vars.pass}}" },
+  { "type": "click", "selector": "button[type=submit]" },
+  { "type": "wait", "selector": "#logged-in" }
+]}}
+```
+
+---
+
+## 3. Steps
+
+Every step has `type`, and may have:
+
+| Key | Meaning |
+|---|---|
+| `id` | The name of the value the step produces. A word: letters (either case), digits and underscores, not starting with a digit. Unique along any path, together with the `as` of the enclosing `forEach` loops; never `page`, `start` or `vars`. |
+| `onError` | `{ "policy": "fail" }`, `{ "policy": "skip" }` or `{ "policy": "retry", "attempts": 3, "backoffMs": 500 }`. §7. |
+| `when` | A template; the step runs only when it renders truthy (§3.4). |
+| `keep` | Worker mode only ([worker-mode.md](./worker-mode.md)): the window remembers the step as it last ran it and skips it on the next item while it would do the same. A kept step that runs again makes the window forget the kept steps after it. Top-level page actions (`goto`, `click`, `fill`, `press`, `select`, `scroll`, `wait`, `evaluate`) without an `id`. |
+
+### 3.1 Web steps (browser page)
+
+| Step | Fields | Notes |
+|---|---|---|
+| `goto` | `url` (template), `waitUntil?` (`load`, `domcontentloaded`, `networkidle`, `commit`), `ready?` | Relative URLs resolve against the current page. Records `page.url`. `ready: { selector, timeoutMs?, reloads? }` is an element the page must show: a site that sometimes serves its shell without the content is loaded again, up to `reloads` times (default 2), each reload reported as `request:retry`. |
+| `click` | `selector` or `target`, `optional?`, `download?` | First match. With `optional: true` a missing element is skipped after a 2 s wait. With `download: { as?, saveTo?, encoding?, delimiter?, timeoutMs? }` the click downloads a file (an "Export to Excel" button): it is read like a fetched document (CSV, spreadsheet, PDF, Word, JSON…, by `as`, else the file's name, else its bytes), becomes the current document, and the step `id` holds it; `saveTo` (a template) keeps a copy. Read it with `from: <id>` (a `table` extract without `from` reads the page's HTML). |
+| `fill` | `selector` or `target`, `value` (template) | |
+| `press` | `key`, `selector?` or `target?` | A key on an element, or on the page. |
+| `select` | `selector` or `target`, one of `value`, `label`, `index`, `values`; `multiple?`, `force?`, `ignoreCase?`, `timeoutMs?` | Picks an option of a `<select>`; `value` and `label` are templates. Fires the page's `change` handlers. `values` picks several, each matched by value or label (an item rendering a list, `{{ split(vars.states) }}`, adds each); `multiple` adds to what is chosen. `force` sets the options on the element itself, visible or not, and fires `input` and `change`: a hidden `<select multiple>` behind a script-built widget, when the page listens to the select (it usually does: that is how the widget's choice reaches the form). The step waits up to `timeoutMs` (`limits.timeoutMs`, else 30 s) for options the page loads after another pick, then names what matched nothing. `values` that render to nothing (a blank filter var) leave the control alone. `clear: true` makes `values` that render to nothing clear the control instead (a worker window reused for the next item must not keep the last item's filter). `search: { input, open?, close? }` is for a widget that loads its options as you type (a maker list of thousands): each value missing from the options is typed into `input` key by key (after clicking `open` when the box is hidden), picked once the page lists it, and the picks add up; `close` is clicked at the end. A search usually replaces the widget's list, so check that earlier picks survive a later search on the site at hand. |
+| `scroll` | `to` (`bottom` or a selector), `times?`, `untilStable?`, `maxScrolls?`, `settleMs?` | Scrolls `times` times (default 1), waiting `settleMs` (default 300) after each for the page to load more. `untilStable` keeps scrolling until the page stops growing: infinite lists. It stops at `maxScrolls` scrolls (default 50) on a feed that never ends, without failing: a `warning` event (`!` in the trace) says so. Raise `settleMs` for a site whose next batch takes longer to arrive: `untilStable` measures the page, and a slow API makes it stop early. |
+| `wait` | one of `selector`, `ms`, `state: "networkidle"`; `timeoutMs?` | `selector` waits for visibility. Put a `wait` after `goto` on script-heavy pages before extracting. `timeoutMs` bounds the `selector` and `state` forms; default `limits.timeoutMs`, else 30 s. Give a long one to wait for a slow report, or for a person in a headed run. |
+| `evaluate` | `script`, `args?` | JavaScript evaluated in the page; the result is bound under `id`. `script` is a template. With `args`, `script` is a function expression called with them, each string rendered (a lone placeholder keeps its type): `{ "script": "(a) => a.states.length", "args": { "states": "{{ split(vars.state) }}" } }`. **Trusted recipes only.** |
+| `screenshot` | `path` (template) | Full page. A debugging aid. |
+| `captcha` | `solver?`, `selector?`, `verify?`, `attempts?`, `timeoutMs?`; a form captcha: `image`, `refresh?`, `field?`, `submit?` | Solves the challenge on the page, if there is one (none is fine), reCAPTCHA v3 included. Missing fields come from `session.captcha`. With `image`, a captcha checked when its form is posted: the solver fills `field`, `submit` posts the form, `verify.selector` / `verify.failure` say yes or no. §6.1. |
+
+Clicks and key presses can navigate; the engine re-reads the page URL after every web step. A `click`,
+`press` or `select` that navigates the page is checked like a `goto`: the engine waits for the new page to
+load, retries a load that fails in passing (`limits.retry`: it loads the URL the step navigated to again; a
+form the step posted is never sent twice), checks the answer against `session.blockedWhen`, and reports the
+page as `page:visit` with its status, so it counts in `pages`. A navigation that fails, or still answers a
+retried status (a 503) once the tries are spent, fails the step with its cause (`net::ERR_… at <url>`,
+`HTTP 503 at <url>`): the browser's error page is never taken for the page. A click that only changes the
+page in place counts nothing; one whose script changes the URL (`history.pushState`) counts a page without
+a status. A key press on the page itself and a `select` do not wait for a navigation their handlers start;
+they give it 150 ms to begin.
+
+`target` is a template instead of a selector: it renders either to a **live element** (the variable of a
+`forEach` over `selector`, §3.7) or to a selector string. Give one of `selector` and `target`, never both.
+A plain `selector` is never rendered, so a recipe with `{{` in one is refused at load time: put that selector
+in `target`.
+
+### 3.2 Requests (HTTP)
+
+`request` runs in both modes. In `api` mode it goes through the recipe's request context. In `web` mode it goes
+through **the page's own session** (its cookies: a login, a consent, a solved captcha), the way a page's scripts
+call their JSON endpoints: a report whose table the page reads from an API is read the same way, with no
+`evaluate` and `fetch` code.
+
+| Step | Fields | Notes |
+|---|---|---|
+| `request` | `url` (template), `method?`, `query?`, `headers?`, `body?` or `form?`, `as?` (`json`, `jsonl`, `html`, `text`, `pdf`, `csv`, `xlsx`, `pptx`, `yaml`, `markdown`, `xml`, `docx`), `encoding?`, `delimiter?`, `scalars?` | The response becomes the current document and, if `id` is set, the id holds the parsed JSON, the read PDF, workbook or deck, the markup or the text. Relative URLs resolve against the current page; one with no page to resolve against fails with a message saying so. Without `as`, the content type decides (`application/pdf` is a PDF, `text/csv` a CSV, a spreadsheet type a workbook, a presentation type a deck, `application/yaml` YAML, `application/x-ndjson` JSON Lines, `text/markdown` Markdown, an XML type XML: §4.12, a Word type a document read as HTML: §4.13); a type that says nothing (below) is sniffed. A `file:` URL reads a local file, its kind from `as`, else the extension (`.pdf`, `.csv`, `.tsv`, `.xlsx`, `.xlsm`, `.pptx`, `.yaml`, `.yml`, `.json`, `.jsonl`, `.ndjson`, `.md`, `.html`, `.xml`, `.rss`, `.atom`, `.xml.gz`, `.docx`), else its bytes; a relative one (`file:data/listino.csv`) is read from the recipe file's folder (below). 4xx/5xx fail the step. |
+
+**A body served as anything** (`application/octet-stream`, `binary/octet-stream`, `text/plain`, `application/zip`
+and the like, or no content type: a file behind a bucket or a CDN, a download link, a raw file) is read by what it
+is. Its first bytes decide: `%PDF-` is a PDF; a zip whose parts are under `xl/`, `ppt/` or `word/` is a workbook,
+a deck or a Word document; gzip is a sitemap when the URL ends `.xml.gz`. Otherwise the URL path's extension
+decides, as for a `file:` URL, and only then is it text. So a GitHub raw `.csv` or `.json` (served as
+`text/plain`) reads as a CSV or JSON. An error page (4xx/5xx) is not sniffed by its URL's extension. `as` wins
+over all of it; a specific content type wins over the bytes. A local file or a download with no known extension is
+sniffed the same way.
+
+**Local files.** `file:///srv/listini/listino.csv` reads that file. A relative `file:` URL
+(`file:data/listino.csv`, `file:./x.pdf`, `file:../shared/x.xlsx`) is read from **the folder of the recipe file**
+that names it, when the recipe was loaded from a file (`loadRecipeSet`, `loadRecipes`, the cli), so a recipe and
+its documents move together. That holds for start URLs, start and recipe `vars`, `matrix` values, and every step's
+`url` (a template keeps its placeholders: `file:data/{{vars.month}}.csv`). A recipe given as an object or text,
+and a URL that only a template makes relative, read from the process working directory. `allowedHosts` refuses
+every `file:` URL, relative or not (§8).
+
+Text bodies are decoded from, in order: a byte-order mark, `encoding` (any WHATWG label: `windows-1252`,
+`iso-8859-15`, `shift_jis`), the charset the server declares, UTF-8, and Windows-1252 for text that is not
+UTF-8 (the usual European export). `delimiter` (one character) overrides a CSV's detected delimiter (§4.7).
+
+**A body from a form** (web mode): `form: { selector, omit?, set? }` posts the page's form as the browser would
+(its `FormData`: every chosen option of a multi-select, no disabled field, no unticked box), url-encoded, without
+the `omit` fields, with each `set` field (a template) replacing or adding one. The form's values are sent as
+read, never rendered. A report endpoint paged with a cursor:
+
+```json
+{ "type": "paginate", "next": { "jsonpath": "$.next", "as": "cursor" }, "until": "{{ !page_rows.hasMore }}", "steps": [
+  { "type": "request", "id": "page_rows", "url": "/report/rows", "method": "POST",
+    "form": { "selector": "#reportForm", "omit": ["captcha"], "set": { "pageSize": "25", "after": "{{ default(cursor, '') }}" } } },
+  { "type": "extract", "id": "rows", "selector": "$.rows[*]", "kind": "jsonpath", "take": "json", "many": true },
+  { "type": "forEach", "over": "rows", "as": "row", "emit": true, "steps": [] }
+] }
+```
+
+`until` stops on the endpoint's own "no more" flag: some keep returning a cursor on the last page.
+
+In web mode, a `jsonpath` extract reads the JSON the last `request` fetched (the live page is read by `css` and
+`xpath`), and `next.jsonpath` pages on it.
+
+`body` is templated **all the way down**: a string body is one template, and in an object or list body
+every string inside it is one, at any depth. A string that is exactly one placeholder keeps the value's type
+(`"limit": "{{vars.limit}}"` sends a number). Numbers, booleans and `null` go as they are. An object or list
+body is sent as JSON:
+
+```json
+{ "type": "request", "id": "token", "url": "https://auth.example/oauth2/token", "method": "POST", "as": "json",
+  "body": { "grant_type": "client_credentials", "client_id": "{{vars.clientId}}", "client_secret": "{{vars.clientSecret}}" } }
+```
+
+### 3.3 Steps of both modes
+
+| Step | Fields | Notes |
+|---|---|---|
+| `extract` | `selector`, `kind` (`css`, `xpath`, `jsonpath`, `regex`, `table`, `region`), `take?`, `many?`, `from?`; `namespaces?`, `ignoreNamespaces?` (`xpath` on XML) | §4. |
+| `set` | `value` | A literal, or a template when it is a string. In an object or list every string, at any depth, is a template (a lone placeholder keeps its type), as in a request `body`. |
+| `collect` | `into` (an id), `value` | Appends `value` (a literal, or a template when it is a string, rendered all the way down like `set`) to the list `into` holds, in whichever enclosing scope binds it; a list value is appended item by item, a missing one adds nothing. `into` must be bound first, usually `{ "type": "set", "id": "all", "value": [] }` before the loop: the binding validator checks. The way to carry values out of `forEach` iterations or `paginate` pages (§3.6). |
+| `forEach` | `over` (a list id) **or** `selector` (web), `as` (variable), `steps`, `emit?` (`true`, or `{ output }` naming this recipe's output) | Runs `steps` once per item in a fresh child scope with the item bound as `as`. `as` follows the rules of a step `id`: it may not name `page`, `start`, `vars` or an id already bound on the path (§3.5). `emit: true` produces one record per iteration. `over` may name a single value; it is treated as a one-item list. `selector` iterates the live elements it matches (§3.7). |
+| `if` | `test` (template), `steps`, `else?` | Runs `steps` when `test` renders truthy, otherwise `else`, **in the current scope**: ids bound in a branch are visible after it. §3.8. |
+| `paginate` | `next`, `until?` (template), `maxPages?`, `steps` | Runs `steps` per page in a fresh child scope, then follows `next`. §3.6. |
+| `emit` | `output?` (must be this recipe's output id) | Produces a record from everything in scope. |
+| `hook` | `name`, `args?` | Calls the registered hook; `args` strings are templates. The result is bound under `id`. |
+
+### 3.4 Templates
+
+`{{path}}` reads: any id in scope, the current `forEach` variable and paths into it (`{{item.url}}`),
+`{{vars.name}}`, `{{start.url}}`, `{{page.url}}`, `{{page.number}}`. Dotted paths walk into objects and
+lists (`{{item.images[0]}}`). A template that is **exactly one placeholder** yields the value with its type
+(a list stays a list); inside longer text values are stringified, missing ones as empty text.
+
+A placeholder may also be an **expression** over such paths:
+
+| Kind | Syntax |
+|---|---|
+| literals | `12`, `1.5`, `'text'`, `"text"`, `true`, `false`, `null` |
+| arithmetic | `+ - * / %`, parentheses. `+` adds two numbers and concatenates anything else (`'p-' + id`). Division by zero and arithmetic on non-numbers give a missing value. Numeric text counts as a number (`qty * price` with `qty` = `"3"`). |
+| comparison | `== != < <= > >=`. `==` is loose (`3 == '3'`, `null == missing`); ordering is numeric when both sides are numbers, else textual. |
+| logic | `&& || !` with recipe truthiness (below); `a ?? b` gives `b` only when `a` is missing (`null` / undefined) |
+| choice | `test ? a : b` |
+| functions | `upper(s)`, `lower(s)`, `trim(s)`, `len(list or text)`, `default(v, fallback)` (missing or `''`), `round(n, digits?)`, `number(text)`, `join(list, sep?)`, `first(list)`, `last(list)`, `replace(s, pattern, replacement)` (a regular expression, global), `contains(list or text, needle)`, `split(text, sep?)` (pieces trimmed, blank ones dropped: a blank text gives `[]`, so a `forEach` over it runs no iteration), `urlEncode(s)` (percent-encodes one URL component: `{{start.url}}/config/{{urlEncode(id)}}` turns `#` into `%23`, a space into `%20`), `entries(object)` (`[{ key, value }]` in key order: rows keyed by name become a list `forEach` walks; a list gives its indexes), `keys(object)`, `values(object)` |
+
+```json
+{ "type": "set", "id": "next_url", "value": "{{ start.url }}?page={{ page.number + 1 }}" },
+{ "type": "set", "id": "label", "value": "{{ stock > 0 ? 'in stock' : 'sold out' }}" },
+{ "type": "if", "test": "{{ len(links) > 0 && !wall }}", "steps": [] }
+```
+
+Rules that keep this safe and predictable:
+
+- A placeholder made only of path characters (`item.display-name`, `ld.@type`, `price-1`) is a **path**,
+  never an expression: write `{{ price - 1 }}` with spaces for subtraction.
+- Expressions never run code. There is no member call, no `new`, no assignment; a value that is a function
+  (a hook could bind one) reads as missing, and paths read own data only, so `constructor`, `__proto__`,
+  `toString` and the like are missing too. Unknown functions and stray characters fail at parse time with
+  the position.
+- Deep nesting and very long expressions are refused (64 levels, 512 tokens).
+
+Truthiness for `when`, `until`, `test` and the logical operators: `false`, `0`, `null`, `undefined`, an empty list,
+and text that is, trimmed and in any case, `""`, `"false"`, `"0"`, `"null"` or `"undefined"` are false; everything else
+is true.
+
+### 3.5 Scope: what a step can see
+
+- Ids are bound in the scope where the step ran. Reads fall through to parent scopes.
+- `forEach` opens a **fresh child scope per iteration**; `paginate` opens one **per page**. A child scope is
+  dropped when its iteration or page ends, so nothing from page 1 is visible on page 2 and a row without a
+  value cannot inherit the previous row's.
+- `emit` snapshots the whole chain, child values shadowing parents. That is why a record produced inside a
+  `forEach` sees its own row's values *and* the actor name extracted outside the loop.
+- A name is bound once along a path: a step `id` or a `forEach`'s `as` that repeats `page`, `start`, `vars` or
+  an id bound earlier on the path (an outer step, an enclosing loop's `as`) fails binding with
+  `id "item" is already bound on this path`. So a loop over tags inside a loop over products needs its own
+  name (`as: "tag"`, not a second `as: "item"`). Sibling loops, and the two branches of an `if`, are separate
+  paths and may reuse a name.
+- `page.url` / `page.number` and the current document are scope state, bound in the innermost scope that
+  navigated. In `web` mode `page.url` is the real page URL; in `api` mode it is the final URL of the nearest
+  `request`, and `start.url` before any request.
+- A value found inside a child scope dies with it, unless a `collect` step appends it to a list bound in an
+  enclosing scope (§3.6).
+- Exactly **one emitting construct per path**: an emitting `forEach` may not contain an `emit` or another
+  emitting `forEach`. The binding validator rejects it.
+- `limits.maxRecords` stops the walk cleanly, wherever it is in the tree.
+
+### 3.6 Pagination
+
+`next` is evaluated **after** the page body ran:
+
+| `next` | Mode | Behaviour |
+|---|---|---|
+| `{ "selector": "a.next" }` | web | Clicks it. If the body navigated away (a `forEach` visiting every item), the engine returns to the listing page first. No visible element within 2 s means no next page. The navigation the click causes is checked like a `goto`'s: a load that fails in passing is retried (`limits.retry`) by loading the URL the click led to again, not by clicking again; the answer is checked against the block rule; a failure fails the `paginate` step with its cause (`HTTP 503 at …`, `net::ERR_… at …`), never running the body on the browser's error page. A next that swaps the content in place (no navigation) still works: the click gets 500 ms to start one, and the page counts either way. |
+| `{ "url": "{{start.url}}?page={{ page.number + 1 }}" }` | both | The rendered value is the next `page.url` (web mode navigates to it). Empty means no next page. |
+| `{ "jsonpath": "$.nextPage" }` | api, web (on a `request`'s JSON) | Evaluated on the current document; the value is the next URL, relative allowed. `null`, `false` or empty means no next page. |
+| `{ "jsonpath": "$.cursor", "as": "cursor" }` | api, web (on a `request`'s JSON) | The value is bound under `cursor` in the next page's scope and the body builds the URL itself (`?cursor={{cursor}}`); on page 1 it is unset. |
+
+Pagination also stops when `until` renders truthy or at `maxPages`. `page.number` increments per page.
+
+Two shapes cover most sites:
+
+```json
+{ "type": "paginate", "next": { "jsonpath": "$.nextPage" }, "steps": [
+  { "type": "request", "id": "list", "url": "{{page.url}}", "as": "json" },
+  { "type": "extract", "id": "items", "from": "list", "selector": "$.items[*]", "kind": "jsonpath", "take": "json", "many": true },
+  { "type": "forEach", "over": "items", "as": "item", "emit": true, "steps": [] }
+]}
+```
+
+```json
+{ "type": "goto", "url": "{{start.url}}" },
+{ "type": "paginate", "next": { "selector": "a.next" }, "maxPages": 3, "steps": [
+  { "type": "extract", "id": "links", "selector": "a.product", "kind": "css", "take": "attr:href", "many": true },
+  { "type": "forEach", "over": "links", "as": "link", "emit": true, "steps": [
+    { "type": "goto", "url": "{{link}}" },
+    { "type": "extract", "id": "raw_title", "selector": "h1", "kind": "css" }
+  ]}
+]}
+```
+
+**Collect first, then fan out.** Some sources enumerate ids page by page and only then fetch one detail
+per id. Page scopes are dropped, so the ids have to be carried out: bind an empty list before the
+`paginate`, `collect` each page's ids into it, and loop over it once pagination has finished:
+
+```json
+{ "type": "set", "id": "allIds", "value": [] },
+{ "type": "paginate", "next": { "jsonpath": "$.responseDetails.next", "as": "cursor" }, "until": "{{ !list.responseDetails.existsMore }}", "steps": [
+  { "type": "request", "id": "list", "url": "{{start.url}}?cursor={{cursor}}", "as": "json" },
+  { "type": "extract", "id": "ids", "from": "list", "selector": "$.items[*].id", "kind": "jsonpath", "take": "json", "many": true },
+  { "type": "collect", "into": "allIds", "value": "{{ids}}" }
+]},
+{ "type": "forEach", "over": "allIds", "as": "id", "emit": true, "steps": [
+  { "type": "request", "id": "detail", "url": "{{start.url}}/{{urlEncode(id)}}", "as": "json" }
+]}
+```
+
+The list is held in memory until the loop ends: fine for thousands of ids. For millions, or when records
+should start flowing before the enumeration ends, **fan out per page** instead: put the `forEach` inside
+the `paginate` body, as in the first shape above, so each page's details are fetched and emitted before the
+next page is read.
+
+### 3.7 Live elements: driving a configurator
+
+`forEach` with `selector` instead of `over` runs its body once per element the selector matches **on the
+current page**, in web mode only. Each iteration binds a snapshot of the element under `as`:
+
+| Path | Holds |
+|---|---|
+| `{{option.text}}` | text content, whitespace collapsed |
+| `{{option.html}}` | inner HTML |
+| `{{option.attrs.value}}`, `{{option.attrs.data-id}}` | any attribute |
+| `{{option.value}}` | the `value` of an input, option or select |
+| `{{option.selector}}`, `{{option.index}}` | where to find it again |
+
+The snapshot is taken once, when the loop starts, and the engine never holds on to the element itself: a
+`target: "{{option}}"` on `click`, `fill`, `press` or `select` re-resolves the element by selector and
+index at that moment. That is what makes the loop survive a page that re-renders after every interaction,
+which is what a car configurator, a size picker or a tabbed spec sheet does.
+
+The shape for "one record per option of a `<select>`":
+
+```json
+{ "type": "goto", "url": "{{start.url}}" },
+{ "type": "extract", "id": "model", "selector": "h1", "kind": "css" },
+{ "type": "forEach", "selector": "select#trim option", "as": "option", "emit": true, "steps": [
+  { "type": "select", "selector": "select#trim", "value": "{{option.attrs.value}}" },
+  { "type": "extract", "id": "trim", "selector": ".trim", "kind": "css" },
+  { "type": "extract", "id": "price", "selector": ".price", "kind": "css" }
+]}
+```
+
+For a row of buttons or tabs use `{ "type": "click", "target": "{{option}}" }` instead of `select`. Put a
+`wait` after the interaction when the page fetches the new state instead of rewriting it in place.
+
+Limits: the loop iterates the matches present when it started; elements a later interaction adds are not
+visited (nest a second `forEach` for that). Navigating away inside the body and relying on `target` on the
+next iteration fails, because the selector no longer matches on the new page.
+
+### 3.8 Decisions
+
+`when` skips one step. `if` chooses between two step lists and is the tool for "do this only on the
+first page", "the layout differs for sold-out items", "dismiss the wall if it is up":
+
+```json
+{ "type": "extract", "id": "wall", "selector": "#consent", "kind": "css", "take": "html", "many": true },
+{ "type": "if", "test": "{{wall}}", "steps": [{ "type": "click", "selector": "#accept" }] }
+```
+
+`test` uses the same truthiness as `when` (§3.4); an extract with `many: true` never fails, so its empty
+list is the clean way to test for presence. Both branches run in the scope of the `if`, not a child: a
+value set in `steps` or `else` is visible to the steps after it, and each branch may bind the same id (they
+never both run). A branch may `emit`; the one-emit-per-path rule treats the branches as separate paths.
+The trace shows the branch taken as `⑂ steps.N  then`.
+
+### 3.9 Concurrency
+
+Two dials, one inside a recipe and one across recipes.
+
+**Inside a recipe**, `limits.concurrency: 3` lets a `forEach` over a list run three iterations at once: a
+listing whose body fetches every item's page fans out, and records come out in completion order rather than
+list order.
+
+- **api mode:** three requests in flight on one HTTP session.
+- **web mode:** three **tabs** of the recipe's browser context, one per iteration: they share cookies (the
+  login), each has its own page, and each closes when its iteration ends. A tab opens blank, so the body must
+  start with a `goto` (`{{item.url}}`). A `forEach` over `selector` stays sequential: its live elements
+  belong to one page.
+- The limit is **per recipe run**, not per loop: nested loops share it. The outermost concurrent loop takes
+  the permits; a loop inside one of its iterations runs its body sequentially, so nesting never multiplies
+  the number of requests in flight.
+- `delayMs` is a **rate**, not a per-request pause: the minimum time between two request starts across
+  every iteration. `concurrency: 4, delayMs: 250` means at most four requests per second, in flight or not.
+- `maxRecords` is exact: once reached, iterations still in flight finish but emit nothing more.
+- Politeness towards a **site** across recipes is the runner's `throttle` ([access.md](access.md#throttling-per-site)),
+  on top of these per-recipe limits.
+- A failing step under the `fail` policy stops new iterations; the ones in flight settle, then the recipe
+  fails as usual.
+- A block under `onBlock.rotate` rotates once for the whole recipe; every tab reopens on the new lease.
+
+**Across recipes**, `createCrawler({ parallel: 3 })` (CLI `--parallel 3`, MCP `run` with `parallel: 3`) runs
+three input recipes of a set at once, each in its own browser context or HTTP session. They share the browser,
+the sink and the per-site throttle. Reports keep the set's order. Under `onRecipeError: 'stop'`, a failure stops
+the recipes not started yet. De-duplication under `dedupe: 'recipe'` stays per recipe; under `run`, the recipe
+that emits a key first keeps it.
+
+Parallel is only faster if the site lets it be. Pair it with the per-site `throttle`
+([access.md](access.md#throttling-per-site)), which bounds what all of this adds up to on one site.
+
+---
+
+## 4. `extract` in depth
+
+`extract` reads **a document** with **a selector** and **takes** something from each match.
+
+### 4.1 Which document
+
+| `from` | Web mode | Api mode |
+|---|---|---|
+| absent | the live page | the last `request`'s response (nearest scope that has one) |
+| an id holding **text** | with `css`: the text as HTML (a fragment such as a `<tr>` is parsed as a fragment, so cells survive); with `table`: the HTML's `<table>`s (a Word or Markdown document, a page a `request` fetched: §4.11); with `jsonpath`: the text parsed as JSON | same |
+| an id holding a **list of texts** | with `jsonpath`: every entry that parses as JSON becomes one element of an array and the path runs over the array; with `table`: the tables of every fragment (`take: "html"`, `many: true`) | same |
+| an id holding **data** (an object, a list of objects) | with `jsonpath` | same |
+| an id holding a **read PDF** (a `request` with `as: "pdf"`) | with `table`, `region`, `regex` or `jsonpath` (§4.6) | same |
+| an id holding a **read workbook** (a spreadsheet or a CSV) | with `table`, `regex` or `jsonpath` (§4.7) | same |
+| an id holding a **read deck** (a presentation) | with `table`, `regex` or `jsonpath` (§4.8) | same |
+
+**JSON Lines** (NDJSON: one JSON value per line, as bulk exports and Shopify bulk operations give) read with
+`as: "jsonl"`, or by the `application/x-ndjson` / `application/jsonl` content types or the `.jsonl` / `.ndjson`
+extensions, into an **array** of the lines' values: `$[*]` walks them. A line that does not parse fails the step
+with its number. JSON Lines served as `application/json` fail with a hint to use `as: "jsonl"`.
+
+**Wrapped JSON** is unwrapped, both in a response read as JSON and in text a `jsonpath` extract parses (`from`):
+
+- comment guards around JSON-LD: `/* <![CDATA[ */ … /* ]]> */`, `<!-- … -->`;
+- anti-hijacking prefixes: `)]}'` (with or without a comma), `while(1);`, `for(;;);`;
+- JSONP: `callback({…});`, so an endpoint served as `text/javascript` reads with `as: "json"`;
+- an assignment in an inline script: `window.__INITIAL_STATE__ = {…};` (with `var`, `let` or `const`, too).
+
+Valid JSON is always read as it is: a wrapper is only removed when the text does not parse, and what is inside
+must be strict JSON. Nothing is evaluated, so a JavaScript literal (`{ a: undefined }`, unquoted keys) still
+fails: read it with `regex`, or a hook. The common JSON-LD pattern, which both example sites use:
+
+```json
+{ "type": "extract", "id": "ld", "selector": "script[type=\"application/ld+json\"]", "kind": "css", "take": "text", "many": true },
+{ "type": "extract", "id": "name", "from": "ld", "selector": "$[?(@['@type']=='Movie')].name", "kind": "jsonpath", "take": "json" },
+{ "type": "extract", "id": "actors", "from": "ld", "selector": "$[*].actors[*].name", "kind": "jsonpath", "take": "json", "many": true }
+```
+
+### 4.2 Which selector
+
+| `kind` | Works on | Notes |
+|---|---|---|
+| `css` | live page, fetched HTML, HTML fragments | Standard CSS through Playwright (live) or cheerio (static). |
+| `xpath` | live page, fetched XML, fetched HTML, XML or HTML fragments | XPath 1.0. Fetched HTML is parsed the way a browser parses it (`<tbody>` added), so a query that works in the browser works on the fetched page. On namespaced XML, see §4.12. |
+| `jsonpath` | JSON data, JSON text, lists of JSON texts | jsonpath-plus syntax: `$.items[*].url`, `$[?(@.actors)]`, `$[?(@['@type']=='Movie')].name`. |
+| `regex` | any document as text: markup, text, JSON re-serialised, a list of texts joined by newlines | A JavaScript regular expression (flags `gs`); group 1 is taken when the pattern has one, else the whole match. For values that live in inline scripts (`"carPath":"([^"]+)"`), attributes, or table prose (`Boot capacity</td>\\s*<td>([^<]+)`). |
+
+**Several keys into one list.** To merge same-shaped arrays kept under different keys
+(`availableExteriors`, `availableInteriors`, `availableWheels`...), name them in an **unquoted** union, or
+match the key names with a filter on `@property`:
+
+```json
+{ "type": "extract", "id": "options", "from": "config", "kind": "jsonpath", "take": "json", "many": true,
+  "selector": "$.model[availableExteriors,availableInteriors,availableWheels,standardEquipments][*]" }
+```
+
+`$.model[?(@property.startsWith('available'))][*]` takes every key that starts with `available`, and
+`$..[?(@property==='availableExteriors' || @property==='availableWheels')][*]` finds them at any depth. The
+quoted union `$.model['availableExteriors','availableInteriors'][*]` matches **nothing** in jsonpath-plus:
+leave the names unquoted.
+
+Every selector is a template: `{{ }}` placeholders render against the scope first, so one step can pick the
+colour set of the current trim (`$[?(@.trimname=='{{trim}}')].colors[*].displayName`) or the row of the
+current item.
+
+### 4.3 What to take
+
+| `take` | From an element | From a JSON node |
+|---|---|---|
+| `text` (default) | text content with whitespace collapsed | scalars as text, objects as JSON text |
+| `html` | inner HTML | – |
+| `json` | outer HTML (the element itself, for a later `from`) | the node as data |
+| `value` | an input's value | – |
+| `attr:<name>` | the attribute, `undefined` when absent | – |
+
+Take `html` (inner) when you will extract *inside* the element again; take `json` (outer) when the
+information sits on the element's own attributes (`<li title="...">`).
+
+### 4.4 One or many
+
+`many: true` binds a list, empty when nothing matches. Without it the **first** match is bound, and **no match
+fails the step**. That is deliberate: a missing element is either a site change (let it fail) or expected
+(say so with `onError: { "policy": "skip" }`, and the id stays unset for the missing-value policy).
+
+### 4.5 Nested markup: the shifted-field trap
+
+A selector that matches an ancestor first returns the ancestor's whole inner HTML, which contains every
+descendant. Extracting "the first title" from that wrapper gives the right title and "the first role" gives
+the *next* row's role: a shifted field, no error. When a site nests tables or lists, select the innermost
+repeating element (`table.credit_group tr`, not `tr`) and check the first record by hand.
+
+### 4.6 PDFs
+
+`request` with `as: "pdf"` (or a response served as `application/pdf`, or a local `file:…pdf`) reads the
+PDF's text layer with pdf.js into pages of **rows**: text that sits side by side becomes a cell, cells whose
+vertical extents overlap become a row. A scanned page has no text layer and gives no rows; a PDF where no page
+has one fails the step (no OCR). Five extract kinds read it:
+
+| `kind` | Reads | Use for |
+|---|---|---|
+| `table` | tables, found by their header row | price lists, discount sheets, spec tables |
+| `region` | the text inside a box on a page: `page=1 x=72..252 y=640..664` (points from the page's bottom-left corner; `page=*` for every page) | a title, a date, a total — anything that sits at the same place on every issue of a document; what the studio's PDF canvas writes when you click a line |
+| `section` | the text between a heading and the next one, wherever they sit (below) | a description, a list of terms, a price block: content under a title |
+| `regex` | the text: one line per row, cells separated by a tab, pages by a blank line | a value next to a label: `Valid until\t(\S+)` |
+| `jsonpath` | the structure: `{ kind: "pdf", pages: [{ number, width, height, rows: [{ text, top, bottom, cells: [{ x, y, width, height, text }] }] }] }` | positions, a given page |
+
+**Prefer structure to position.** A `region` is a place on the page: it breaks when a description is longer, a
+table has one more row or a column is added. Two sheets of the same layout and different content are read with
+`section` (the text between two headings) and `regex` (the value after a label), which find their text wherever it
+sits. The studio writes those whenever the line you click has a heading above it or a label before it, and a
+`region` only for a box you draw.
+
+A **section** reads the text under a heading: the selector matches the heading row (case-insensitive, the cells
+of the row joined by a space), the text runs to the row `until` matches or, without `until`, to the next row set
+in text as large as the heading (the next heading of its level), or to the end of the document, over page breaks.
+`many: true` gives one text per matching heading.
+
+```json
+{ "type": "extract", "id": "description", "kind": "section", "selector": "^Details$", "until": "^Price$" },
+{ "type": "extract", "id": "price", "kind": "section", "selector": "^Price$" },
+{ "type": "extract", "id": "terms", "kind": "section", "selector": "^Terms$", "ignoreHeadersFooters": true }
+```
+
+The text is read the way a person reads it: the lines of a paragraph are joined with a space (a word broken at
+a hyphen with none), paragraphs are separated by a newline, a numbered or bulleted item starts a paragraph, and
+a page laid out in columns is read one column after the other, left to right.
+
+A section can be narrowed to part of its text. `line` takes one printed line of it (1 is the first, -1 the last;
+the lines of a wrapped paragraph count apart), and `match` keeps the part a pattern matches (its first group,
+else the whole match; case-insensitive), after `line` when both are given. A section the pattern does not match
+in is no match:
+
+```json
+{ "type": "extract", "id": "date", "kind": "section", "selector": "^SUBSCRIPTION SERVICES$", "until": "^COMMERCIAL TERMS$", "line": 1 },
+{ "type": "extract", "id": "email", "kind": "section", "selector": "^Customer Details$", "match": "([\\w.]+@[\\w.]+)" }
+```
+
+`ignoreHeadersFooters: true` (kinds `section` and `table`) leaves out the running headers and footers: rows in
+the top or bottom 15% of a page whose text, digits aside (`Page 1 of 2`), shows at the same height on more than
+one page. Without it they are content like any other row, which is what a `regex` or a `region` over them wants.
+A section that runs over a page break needs it, or the footer lands in the middle of the sentence.
+
+A **region** reads every cell at least half inside its box, row by row: a row's cells joined by a space, rows by
+a newline — so a box around a wrapped paragraph gives its lines. A page with nothing in the box is no match (the
+step fails without `many`); with `page=*` and `many: true` you get one text per page that has something there, a
+running footer for instance. The box is in PDF points, which a `jsonpath` extract of a cell's `x`/`y`/`width`/
+`height` shows you — or the studio draws it for you.
+
+A **table** extract takes the header row's pattern as its `selector`, and returns one table per match, `many`
+for all of them:
+
+```json
+{ "type": "request", "id": "sheet", "url": "{{start.url}}", "as": "pdf" },
+{ "type": "extract", "id": "tables", "selector": "^MODELLI", "kind": "table", "many": true,
+  "until": "^(NOTA BENE|N\\.B\\.)",
+  "columns": { "model": "^MODELLI", "discount": "^Sconto", "excluded": "^Versioni", "extra": "^Azion" } },
+{ "type": "forEach", "over": "tables", "as": "table", "steps": [
+  { "type": "set", "id": "rows", "value": "{{table.rows}}" },
+  { "type": "forEach", "over": "rows", "as": "row", "emit": true, "steps": [] }
+]}
+```
+
+Each table is `{ page, title, header, rows }`: `title` is the first header cell (`MODELLI FIAT`, a brand), and
+each row an object keyed by `columns` (output key → a pattern for that column's header cell), or by the header
+texts without `columns`. A cell's lines are joined by spaces. All patterns are case-insensitive.
+
+| Field | Meaning |
+|---|---|
+| `selector` | Matches a table's header row, its cells joined by spaces. |
+| `until` | Matches the row that ends a table: a footnote, a "Note:" line. A table also ends at the next row the `selector` matches, and at the page's end. With a narrow `selector` (one table among several), set `until` to the start of the next one. |
+| `columns` | Output key → header pattern. Unmatched columns are dropped. |
+| `align` | How a row's values sit against a cell wrapped over several lines: `top`, `center`, `bottom`, or `auto` (default). |
+
+How it reads a table, which is what makes real-world sheets work:
+
+- **Columns come from the body.** A header is often centred over a column whose cells are left-aligned, so the
+  left edges of the body's cells cluster into bands, and bands map to header cells left to right. A header
+  spanning two columns reads both.
+- **Rows are regrouped.** A long name wraps over several lines, a note runs onto a second line, a list of
+  versions sits above and below its row. A line with a name and a value, or with a value in the first value
+  column, anchors a row; the other lines join it. With `auto`, a table that centres values beside wrapped
+  names shares the wrapped lines evenly around each anchor; otherwise a line joins the nearest anchor, and a
+  line halfway between two joins the one below.
+
+Find the selectors with `opencraw probe <url or file.pdf>`: for a PDF it lists the rows and every row that
+looks like a table header, with a ready `selector`. The Stellantis example
+(`examples/stellantis-it-discounts/`) reads eight monthly discount sheets this way.
+
+`fillDown` (below, §4.7) works on PDF tables too.
+
+### 4.7 Spreadsheets and CSV
+
+Both read into a **workbook**: sheets of cells that the same three extract kinds read.
+
+**Spreadsheets.** `request` with `as: "xlsx"` (or a response served with a spreadsheet content type, or a
+local `file:…xlsx` / `…xlsm`) reads every worksheet with
+[`@opencraw/office-reader`](../../packages/office-reader). Numbers and booleans keep their type, so
+`13955.625` is a number that no locale guess can misread. Dates become ISO text (`2026-06-01`,
+`2026-06-01T09:30:00`), errors their text (`#DIV/0!`), and empty cells `''`. Formulas give their cached
+value; nothing is evaluated, and display formats are not applied (a percentage stays `0.125`). Hidden sheets
+and rows are read, and `table` skips them unless `includeHidden`. Merged ranges are recorded, and `table`
+fills them. A legacy `.xls`, a password-protected file or an `.ods` fails the step, with what to do.
+
+**CSV.** `request` with `as: "csv"` (or a response served as `text/csv`, `application/csv` or
+`text/tab-separated-values`, or a local `file:…csv` / `…tsv`) reads the file into a **workbook**: a CSV is a
+workbook with one sheet, named after the file (`…/prezzo_alle_8.csv` → `prezzo_alle_8`). Every cell stays
+text; `number` with a `locale` and `date` with a `format` convert them in the mapping.
+
+- **Encoding** as in §3.2: a Windows-1252 export with no charset reads right without `encoding`.
+- **Delimiter** is detected among `,` `;` tab `|`: the one that splits the first lines into the most
+  consistent number of fields, so a title line above the header does not mislead it, and `;` with decimal
+  commas (`Panda;15.950,00`) is read as `;`. Set `delimiter` on the `request` when detection gets it wrong.
+- **Quotes**: a quoted field may hold the delimiter, line breaks and `""`; a quote inside an unquoted field is
+  kept as is (`1.0 Hybrid "Cross"`). Rows are kept ragged, nothing is trimmed.
+
+The same three extract kinds read it:
+
+| `kind` | Reads | Use for |
+|---|---|---|
+| `table` | tables, found by their header row | the list itself, below any title lines |
+| `regex` | the text: cells separated by a tab, rows by a newline, sheets by a blank line | a date in a title line: `Estrazione del (\S+)` |
+| `jsonpath` | the structure: `{ kind: "workbook", sheets: [{ name, rows: [["cell", …], …] }], csv: { encoding, delimiter } }` | a file with no header row: `$.sheets[0].rows[*]`; one cell by position, a title or a date: `$.sheets[?(@.name=='Prices')].rows[0][0]` (a rectangle: `rows[1:3][0:2]`, read row by row — what OpenCraw Studio writes from a click on the grid); `merges`, `hidden` and `hiddenRows` too, for a spreadsheet |
+
+A grid needs no geometry, so a workbook table is simpler than a PDF one: column *i* of a row belongs to header
+cell *i*. The `selector` matches the header row (its non-empty cells joined by spaces, whitespace collapsed),
+and each table is `{ sheet, title, header, rows }`. Before reading, every merged range's value is copied
+into the cells it covers: a brand merged down its models reads on every row, and a group header merged
+across its sub-columns names each of them.
+
+```json
+{ "type": "request", "url": "{{start.url}}" },
+{ "type": "extract", "id": "table", "selector": "^Marca Modello", "kind": "table", "until": "^Totale",
+  "fillDown": ["brand", "model"],
+  "columns": { "brand": "^Marca$", "model": "^Modello$", "version": "^Versione$", "price": "^Prezzo" } },
+{ "type": "set", "id": "rows", "value": "{{table.rows}}" },
+{ "type": "forEach", "over": "rows", "as": "row", "emit": true, "steps": [] }
+```
+
+| Field | Meaning |
+|---|---|
+| `selector` | Matches the header row. Title lines above it and empty rows below it are skipped. |
+| `until` | Matches the row that ends a table. A table also ends at the next header and at the sheet's end. `"^$"` ends it at the first empty row: several tables on one sheet, separated by blank rows. |
+| `columns` | Output key → header pattern. Without it the header texts are the keys; a column with data but no header text is keyed by its letter (`A`, `B`…), and a repeated header gets a counter (`Price 2`). |
+| `fillDown` | Output keys whose empty cells take the value of the row above: exports that write a brand or a model once over its versions. |
+| `headerRows` | How many rows the header spans (default 1): a column's key joins its header texts (`Total` over `August` → `Total August`). |
+| `sheet` | A pattern for the names of the sheets to read (default: all). |
+| `includeHidden` | Read hidden sheets and rows too. |
+
+A spreadsheet table under a two-row merged header (the German KBA registration statistics):
+
+```json
+{ "type": "extract", "id": "table", "kind": "table", "sheet": "^FZ 10\\.1$", "selector": "^Marke Modellreihe",
+  "headerRows": 2, "until": "^NEUZULASSUNGEN INSGESAMT",
+  "columns": { "brand": "^Marke$", "model": "^Modellreihe$", "month": "^Insgesamt August", "bev": "^mit Elektroantrieb \\(BEV\\) August" } }
+```
+
+`Insgesamt` is merged over *August 2026 · Jan.–August 2026 · Anteil in %*, so the three columns read as
+`Insgesamt August 2026` and so on, and the brand merged down its models' rows fills every row.
+
+`opencraw probe <url or file>` lists a workbook's sheets, the first rows, and every row that looks like a
+header, with a ready `selector` (and a `headerRows` hint when the header has merged cells). For a CSV it
+also reports the encoding and the delimiter it used.
+
+### 4.8 Presentations
+
+`request` with `as: "pptx"` (or a response served with a presentation content type, or a local
+`file:…pptx`) reads the deck with [`@opencraw/office-reader`](../../packages/office-reader) into slides:
+
+```
+{ kind: "deck", width, height, slides: [{ number, title, hidden,
+    shapes: [{ x, y, width, height, text, placeholder }],   // points from the top-left corner, reading order
+    tables: [{ name, rows, merges }],                        // native tables, as sheets
+    charts: [{ type, title, series: [{ name, categories, values }] }],
+    notes }] }
+```
+
+Charts are read from the data the chart caches, the 2016 kinds too (waterfall, treemap, sunburst, histogram, box
+and whisker, funnel, region map), so the embedded workbook is never opened. A title placeholder that PowerPoint places through the slide's layout gets the layout's position, and boxes
+inside a group are placed through its scaling. Slide numbers, dates and footers are left out. A legacy `.ppt`,
+a password-protected file or an `.odp` fails the step, with what to do.
+
+| `kind` | Reads | Use for |
+|---|---|---|
+| `table` | native tables, or with `shapes: true` text boxes laid out as a table | price and incentive tables |
+| `regex` | per visible slide: its text boxes, its tables' rows (cells separated by a tab), then `Notes: …`; slides separated by a blank line | a validity date in the notes: `Notes: .*fino al (\d+ \w+)` |
+| `jsonpath` | the structure above | chart data: `$.slides[?(@.title=='Vendite')].charts[*].series[*]` |
+| `region` | the text boxes at least half inside a box on one slide (`slide=3 x=60..900 y=30..90`, points from the top-left corner) or on every visible slide (`slide=*`), in reading order, one line per box | a source line under a chart, a title in a fixed place on every slide |
+
+A `region` on a deck is the PDF one (§4.6) with `slide=` instead of `page=` and y counted down from the top,
+as the deck's shapes are: `slide=*` reads the same box off every visible slide, one value per slide that has
+something in it. OpenCraw Studio writes the selector from a click on the slide.
+
+A **native table** reads like a spreadsheet table (§4.7). Merged cells are filled, `headerRows` joins a header
+spread over several rows, and `slide` (a pattern on slide titles) picks the slides:
+
+```json
+{ "type": "extract", "id": "table", "kind": "table", "slide": "^Incentivi", "selector": "^Modello Prezzo", "headerRows": 2,
+  "columns": { "model": "^Modello$", "list": "^Prezzo Listino$", "discount": "^Sconto$" } }
+```
+
+**Text boxes laid out as a table** read with `shapes: true`, through the PDF table reader (§4.6): each box is a
+cell, boxes whose heights overlap form a row, and columns come from where the body's boxes start (`align`
+applies). A very tall box can pull two rows together.
+
+```json
+{ "type": "extract", "id": "table", "kind": "table", "shapes": true, "slide": "^Griglia", "selector": "^Modello Prezzo",
+  "columns": { "model": "^Modello", "price": "^Prezzo" } }
+```
+
+Each table is `{ slide, slideTitle, title, header, rows }`, and hidden slides are skipped unless
+`includeHidden`. `opencraw probe <url or file.pptx>` lists the slides, each native table's header with a ready
+`selector`, the charts' series, and the slides whose short text boxes look like a table.
+
+### 4.9 YAML
+
+`request` with `as: "yaml"` (or a response served as `application/yaml`, `text/yaml` or their `x-` forms, or a
+local `file:…yaml` / `…yml`) parses the YAML into **JSON data**: `jsonpath`, `forEach` and `json` fields work
+exactly as on a JSON response. Several documents (`---`) become an array of them.
+
+- **YAML 1.2, always.** Under YAML 1.1, `country: NO` is `false`, `y` is `true` and `0123` is octal `83`. The
+  parser is pinned to 1.2 even when the file declares `%YAML 1.1`, so `NO` stays `"NO"`.
+- **`scalars: "text"`** on the `request` keeps every scalar as written. Even 1.2 reads `zip: 0123` as `123` and
+  `version: 1.10` as `1.1`; with `"text"`, they stay `"0123"` and `"1.10"`, and transforms convert what needs
+  converting.
+- **Merge keys** (`<<: *defaults`) are applied, **anchors** expand with a cap (a "billion laughs" file fails),
+  and **duplicate keys** fail the step with their line.
+- **Custom tags** (`!!js/function`, `!custom`) never build values: the tagged value is read as a plain one, and
+  the trace shows a warning.
+
+### 4.10 Markdown
+
+`request` with `as: "markdown"` (or a response served as `text/markdown`, or a local `file:…md`) renders
+GitHub-flavoured Markdown (tables, task lists, strikethrough, autolinks) to an **HTML document**, so every
+`css` selector works on it. GitHub raw and most CDNs serve Markdown as `text/plain`: set `as`. Two additions
+make it easier to aim at:
+
+- **Sections.** Each heading and everything up to the next heading of the same or a higher level is wrapped
+  in `<section data-heading="…" data-level="…">`, sections nesting: the table under *Prezzi* is
+  `section[data-heading='Prezzi' i] table`. Headings get GitHub's slug ids (`<h2 id="prezzi">`).
+- **Front matter.** A leading `---` YAML block is parsed (YAML 1.2, as §4.9) and put in the head as
+  `<script type="application/json" data-front-matter>`: read it the way JSON-LD is read.
+
+```json
+{ "type": "request", "url": "{{start.url}}", "as": "markdown" },
+{ "type": "extract", "id": "front", "selector": "script[data-front-matter]", "kind": "css", "take": "text" },
+{ "type": "extract", "id": "updated", "from": "front", "selector": "$.updated", "kind": "jsonpath" },
+{ "type": "extract", "id": "table", "selector": "^Modello Versione", "kind": "table", "columns": { "model": "^Modello$", "price": "^Prezzo$" } }
+```
+
+In a GFM table a `|` inside a code span still splits the cell: the Markdown must write it `\|`. Raw HTML in the
+Markdown is kept and selectable, and never runs (it is parsed, not loaded in a browser). To run a `regex` over
+the Markdown source instead, request it with `as: "text"`.
+
+### 4.11 HTML tables
+
+`table` also reads an HTML document's `<table>`s: a fetched page, rendered Markdown, a Word document, or, in web
+mode, the live page. With `from`, it reads the HTML an id holds: a Word or Markdown document fetched earlier, or,
+in web mode, a page a `request` returned (without `from`, a web recipe's `table` reads the live page). Every table
+becomes a grid: `thead`, `tbody` and `tfoot` rows in order, `th` and `td` alike, cell text with whitespace
+collapsed (a `<br>` or a paragraph inside a cell reads as a space: `2659<br>$560` is `2659 $560`), and `colspan` /
+`rowspan` as merged ranges. From there it is the grid table of §4.7: the `selector` matches the header row, merged
+cells are filled, `headerRows` joins a header over two rows, `fillDown` and `columns` work the same. A table
+inside a table is read on its own.
+
+```json
+{ "type": "extract", "id": "table", "kind": "table", "selector": "^Model Version", "headerRows": 2,
+  "columns": { "model": "^Model$", "version": "^Version$", "urban": "^Consumption Urban$" } }
+```
+
+A model merged down its versions (`rowspan`) reads on every row, and *Consumption* over *Urban · Mixed*
+(`colspan`) names both columns. It replaces a `css` extract per `tr` and a `take` per `td`. `probe` lists
+every table's header row with a ready `selector`.
+
+### 4.12 XML: feeds, sitemaps, open data
+
+`application/xml`, `text/xml`, any `+xml` type (`application/atom+xml`, `application/rss+xml`), `.xml`, `.rss`,
+`.atom` files and gzipped `.xml.gz` sitemaps are read as XML (`as: "xml"` forces it). Query them with `xpath`:
+
+```json
+{ "type": "request", "url": "https://shop.example/feed.xml" },
+{ "type": "extract", "id": "entries", "selector": "//a:entry", "kind": "xpath", "namespaces": { "a": "http://www.w3.org/2005/Atom" }, "take": "json", "many": true },
+{ "type": "forEach", "over": "entries", "as": "entry", "emit": true, "steps": [
+  { "type": "extract", "id": "title", "from": "entry", "selector": "/a:entry/a:title", "kind": "xpath", "namespaces": { "a": "http://www.w3.org/2005/Atom" } },
+  { "type": "extract", "id": "link", "from": "entry", "selector": "/entry/link/@href", "kind": "xpath", "ignoreNamespaces": true, "take": "value" }
+] }
+```
+
+- **Namespaces.** An element in a namespace only matches a prefixed name. Prefixes the root element declares
+  (`xmlns:media="…"`) are known already. A **default** namespace (`<feed xmlns="http://www.w3.org/2005/Atom">`) has
+  no prefix, so give it one in `namespaces`, or set `ignoreNamespaces: true` and write plain names. `probe` prints
+  the namespaces and the line to paste. `//*[local-name()='entry']` also works, but reads worse.
+- **Take.** `text` collapses whitespace, `value` keeps a node's text as is (an attribute: `@href` with `take:
+  "value"`), `attr:<name>` reads an element's attribute, `html` is the element's inner markup, `json` its outer
+  markup: an entry to query again with `from`, its namespace declarations included. A function result
+  (`count(//a:entry)`, `string(/a:feed/a:title)`) is taken as it is.
+- **CDATA and entities.** `<![CDATA[600e <La Prima>]]>` and `&amp;` read as text. Entities a DOCTYPE declares are
+  never expanded and nothing external is fetched: an XML bomb or an XXE stays inert text.
+- **Sitemaps.** `//loc` with `ignoreNamespaces: true` lists a sitemap's pages; a `forEach` over them fetches each.
+  A sitemap index lists sitemaps: nest a second `forEach`.
+- **Other kinds on XML.** `css` works too (tag names keep their case; escape a prefix: `media\\:thumbnail`),
+  `regex` reads the markup; `jsonpath` and `table` do not.
+
+`xpath` also reads **fetched HTML** in api mode, parsed like a browser does: `//table[@class='variants']/tbody/tr`
+matches a page whose markup has no `<tbody>`.
+
+### 4.13 Word documents
+
+A `.docx` (a Word content type, `.docx`/`.docm`/`.dotx`, or `as: "docx"`) is read by `@opencraw/office-reader`
+and handed to the recipe as **HTML**, built like rendered Markdown (§4.10), so every tool for pages works on it:
+
+- headings (`Heading 1`, a localized `Titolo 2`, `Title`) become `<h1>`…`<h6>`, each in a
+  `<section data-heading="…" data-level="…">` with what follows it: `section[data-heading='Prezzi' i] table`;
+- bullets and numbering become nested `<ul>` / `<ol>`;
+- tables become `<table>`s with merged cells as `colspan` / `rowspan`, so a `table` extract reads a two-row
+  header (`headerRows: 2`) the way it reads a spreadsheet's (§4.11);
+- links become `<a href>`, a paragraph's Word style is `data-style` (`p[data-style='Prezzo']`);
+- page headers come before the body, footers and notes after it: `header[data-part=header]`, `footer[data-part=footer]`,
+  `aside[data-part=notes] li#footnote-1`; the document's title is `<title>`.
+
+```json
+{ "type": "request", "url": "https://dealer.example/circolare.docx" },
+{ "type": "extract", "id": "prices", "selector": "^Modello", "kind": "table", "headerRows": 2,
+  "columns": { "model": "^Modello$", "list": "Listino", "net": "Netto" } },
+{ "type": "extract", "id": "conditions", "selector": "section[data-heading='Condizioni'] li", "kind": "css", "many": true }
+```
+
+Its `id` holds the HTML, so `table` with `from: <id>` reads its tables after another `request` has replaced the
+current document. Tracked changes read as accepted (insertions in, deletions out). A legacy `.doc` is refused with
+what to do: save it as `.docx`, or export it as PDF (§4.6). `probe` lists a document's sections and tables.
+
+## 5. Mapping
+
+```json
+"mapping": {
+  "actor":   { "from": "actor_name", "transform": [{ "op": "trim" }] },
+  "year":    { "from": "year", "transform": [{ "op": "regex", "pattern": "(\\d{4})" }, { "op": "integer" }] },
+  "price":   { "from": ["price_int", "price_cents"], "transform": [{ "op": "join", "separator": "." }, { "op": "currency" }] },
+  "seller.name": { "from": "seller" },
+  "variants": { "each": "variant_rows", "fields": {
+    "size":  { "from": ".", "transform": [{ "op": "regex", "pattern": "<td class=\"size\">([^<]+)" }] }
+  }}
+}
+```
+
+- The key is an output field, dotted for members of `object` fields.
+- `from` is an id or a path into one (`item.href`, `page.url`). Several sources give the chain a **list** of
+  values (then `join`, `coalesce`, `sum`...).
+- `each` builds an `array` of `object` from a list id; inside `fields`, `from` is relative to each item and
+  `.` is the item itself. Each key of `fields` names a field of the items (a typo is a load error), and each
+  member gets its rule's `onMissing`, its field's policy, `default` and validation, reported at its index
+  (`mapping failed: variants[1].size: missing`).
+- `onMissing` on a rule overrides the field's policy for this recipe. `onMissing: "default"` needs a `default`
+  on the field.
+- Generated fields are not mapped; required fields, and required members of an `each` item, must be mapped,
+  defaulted or generated. The binding validator enforces these at load time.
+
+### 5.1 Transforms
+
+Applied in order. A **scalar** op applied to a list runs on every item; a **list** op runs on the list. A
+missing value (`undefined`, `null`) passes through every op except `default`, `template` and `hook`, so it
+reaches the missing-value policy untouched.
+
+| Op | Args | Kind | Does |
+|---|---|---|---|
+| `trim`, `lowercase`, `uppercase` | – | scalar | text ops |
+| `replace` | `pattern`, `replacement`, `flags?` (default `g`) | scalar | regular-expression replace |
+| `regex` | `pattern`, `group?`, `flags?` | scalar | the first match; group 1 by default when the pattern has one, else the whole match; **no match gives a missing value** |
+| `split` | `separator` | scalar | text → list |
+| `join`, `concat` | `separator` | list | list → text (`concat` defaults to no separator) |
+| `first`, `last` | – | list | one item |
+| `nth` | `index` (negative from the end) | list | one item |
+| `slice` | `start`, `end?` | list | a sub-list |
+| `coalesce` | – | list | the first item that is not missing or blank |
+| `default` | `value` | list | replaces `undefined`, `null` or `''` |
+| `number` | `locale?` | scalar | reads the one number in the text (`"1.299,00"` with `de-DE` → 1299, `"£51.77"` → 51.77); two numbers or more is an error, so `regex` first. Separators as for the `number` type (§1.2) |
+| `integer` | – | scalar | as `number`, truncated |
+| `boolean` | `truthy?` | scalar | `truthy`: `true` when the text contains a phrase (substring, case-insensitive). Without it, the field type's default phrases (§1.2). |
+| `toString` | – | scalar | coerces any value to its plain string form (`String(value)`), unlike the text ops above which reject anything but text, a number, or a boolean |
+| `padStart`, `padEnd` | `length`, `char?` (default a space, exactly one character) | scalar | pads text to `length` from the left or right |
+| `currency` | `locale?`, `currency?` | scalar | `{ amount, currency }`; the amount as `number` reads it; the code from the arg, else an ISO 4217 code next to the amount, else a symbol, else an ISO 4217 code anywhere in the text |
+| `date` | `format?`, `timezone?` | scalar | a Date; `format` tokens `YYYY MM DD HH mm ss`; `timezone` an IANA zone for text without an offset, UTC without one (never the host's zone); a date alone (`2026-03-04`) is midnight UTC |
+| `absoluteUrl` | `base?` | scalar | resolve against `base`, else `page.url` |
+| `urlEncode` | – | scalar | percent-encodes one URL component (`encodeURIComponent`): `#` → `%23`, space → `%20`, `&` → `%26`, `+` → `%2B`, `é` → `%C3%A9` |
+| `flatten`, `unique` | – | list | nested lists flattened; duplicates removed |
+| `sum`, `count` | – | list | numbers |
+| `template` | `value` | list | renders a template against the scope, ignoring the input |
+| `jsonpath` | `path` | list | runs a JSONPath on the input data |
+| `lookup` | `in` (an id or path in scope), `key`, `pick?` | scalar | finds the first item of the table `in` whose `key` path equals the value (compared as text) and yields `pick` from it, or the whole item; no match gives a missing value. The table may be data, JSON text or a list of JSON texts (one `data-*` attribute per element): §5.2 |
+| `group` | `by` | list | `[{ key, items }]` in first-seen order; items without the path group under `null` |
+| `hook` | `name`, `args?` | list | calls a registered hook with the value so far |
+
+### 5.2 Joining two extractions
+
+Sites split one record across places: the price table lists versions, a colour picker elsewhere on the
+page carries a JSON per trim. `lookup` joins them at mapping time, so the body of the loop stays a plain
+row read:
+
+```json
+{ "type": "extract", "id": "colour_data", "selector": "[data-vrdata]", "kind": "css", "take": "attr:data-vrdata", "many": true },
+{ "type": "forEach", "over": "rows", "as": "row", "emit": true, "steps": [
+  { "type": "extract", "id": "trim_base", "from": "label", "selector": "^'(GT-Line S|GT-Line|Air)", "kind": "regex" }
+]}
+```
+
+```json
+"colours": { "from": "trim_base", "transform": [
+  { "op": "lookup", "in": "colour_data", "key": "trimname", "pick": "colors" },
+  { "op": "jsonpath", "path": "$[*].displayName" }
+]}
+```
+
+`in` is resolved against the scope of the record like a template path, so `colour_data` extracted outside
+the loop is visible. Inside an `each` rule's `fields` the same holds: `in` (and a `template` transform's
+paths) look in the current item first, then in the record, so one table extracted once per record can
+annotate every row of a nested list:
+
+```json
+"options": { "each": "option_rows", "fields": {
+  "code":    { "from": "code" },
+  "content": { "from": "code", "transform": [{ "op": "lookup", "in": "componentsInfo", "key": "id", "pick": "text" }] }
+}}
+```
+
+An item key with the same name as a record id wins. `from` inside `fields` stays relative to the item. Applied to a list, `lookup` runs per item. `group` goes the other way: one list of
+rows becomes one `{ key, items }` per distinct key. Map the result straight into an `array` of `object` field
+(members `key` and `items`) or a `json` field: `each` can't walk it, since `each` reads a list bound in the
+scope, not a transform's output.
+
+---
+
+## 6. Hooks
+
+The one extension point. Register functions when creating the crawler and reference them by name:
+
+```ts
+const crawler = createCrawler({ hooks: {
+  positive: (input) => Number(input) > 0,
+  geocode:  async (input, args, context) => { context.log('info', 'geocoding', { input }); return await lookup(String(input), args.country) },
+}})
+```
+
+`(input, args, context) => value`, sync or async. From a `hook` **step**, `input` is `undefined` and the
+result is bound under the step's `id`; from a `hook` **transform**, `input` is the value so far. `context`
+gives `recipeId`, the scope snapshot and a `log`. A recipe naming an unregistered hook fails the run before
+its first request: `crawler.run` and `crawler.work` reject with `UnknownHookError`, naming the recipe and where
+it calls the hook (`shop mapping.inStock.transform.1: unknown hook "positive"`).
+
+**From the cli and the MCP server**, hooks come from a JavaScript module whose default export is the map
+(named function exports work too):
+
+```js
+// hooks.mjs
+export default {
+  positive: (input) => Number(input) > 0,
+}
+```
+
+```sh
+opencraw run recipes/ --hooks hooks.mjs      # or OPENCRAW_HOOKS=hooks.mjs
+```
+
+The same file becomes a **plugins module** when it names its exports: `hooks` (the map), `accessPlugins`
+([access.md](access.md#plugins)) and `captchaSolvers` (§6.1). `--plugins` / `OPENCRAW_PLUGINS` are the same
+option under their own name.
+
+```js
+// plugins.mjs
+export const hooks = { positive: input => Number(input) > 0 }
+export const accessPlugins = [myProxyList]
+export const captchaSolvers = [capsolver({ apiKey: process.env.CAPSOLVER_KEY })]
+```
+
+The MCP server reads `OPENCRAW_PLUGINS` / `OPENCRAW_HOOKS` from its own environment, never from a tool call: an agent can run
+recipes that call your hooks but cannot make the server load a module of its choosing. The module runs as
+your code, with your privileges, like anything you `import`: load only files you trust.
+
+**In the Studio**, start it with the same option: `opencraw studio recipes/ --hooks hooks.mjs` (or
+`OPENCRAW_HOOKS`). The command loads the file, so the Studio runs sample runs and login snapshots with your
+hooks, access plugins and captcha solvers, and shows a bar naming the file and the hooks it provides. Started
+without it, a run that calls a hook stops with a message saying so. No request or recipe can make the Studio
+load a module; only the command line that started it can.
+
+The editor uses the loaded names. **Add step → Hook** adds a `hook` step named after the first loaded hook, and
+its form has a list of the hooks to pick from. In a field's transform chain, **+ transform → hook (your code)**
+adds a hook transform, and clicking it opens the same list. A name the recipe already has that is not loaded
+stays selectable and is marked. With no hooks loaded the name is a text field (a new hook is called `myHook`).
+
+**Hooks that call out, in the Studio.** A hook made with `commandHook` or `httpHook`, a solver made with
+`commandCaptchaSolver` or `httpCaptchaSolver`, and a plugin made with `commandAccessPlugin` or
+`httpAccessPlugin` call outside the process: a sample run waits for them and they may cost. The bar under the
+toolbar lists them with what they call, a sample run's trace says `☎ hook "price" calls POST https://…` for each
+call, and each hook has **stub with**: a JSON value (`{"amount": 9}`, `"abc"`, `null`) the run answers with
+instead of calling it. A stub lasts for the session and is never written to a recipe; a value that is not JSON
+is ignored, so the hook is called. Solvers and plugins are listed but cannot be stubbed.
+
+**TypeScript:** the module may be a `.ts` file on Node 22.18 or newer (Node 24 included), which strips the
+types as it loads; only syntax that is just types works (no `enum`, no parameter properties). On older Node,
+compile it first.
+
+**Hooks in any language, or behind an API.** A hook does not have to be JavaScript. `commandHook` runs a
+program, and `httpHook` calls a service; both answer to the same request and response, so the recipe still
+says `{ "op": "hook", "name": "slug" }` and the module decides where the code runs.
+
+```js
+// hooks.mjs
+import { commandHook, httpHook } from '@opencraw/core'
+
+export const hooks = {
+  slug:  commandHook('slug', ['python3', 'hooks/slug.py']),
+  price: httpHook('price', 'https://rates.example.com/price', { tokenEnv: 'RATES_TOKEN', allowedHosts: ['rates.example.com'] }),
+}
+```
+
+The program (or the service) gets one JSON request and answers with one JSON response. The shapes are published
+as `schemas/callout-request.schema.json` and `schemas/callout-response.schema.json`:
+
+```json
+{ "kind": "hook", "name": "slug", "input": "Hello World", "args": {}, "context": { "recipeId": "books", "scope": {} }, "idempotencyKey": "9f2c…" }
+```
+
+```json
+{ "status": "ok", "output": "hello-world" }
+```
+
+```python
+# hooks/slug.py: the request is on stdin, the answer goes to stdout
+import json, re, sys
+
+request = json.load(sys.stdin)
+slug = re.sub(r"[^a-z0-9]+", "-", str(request["input"]).lower()).strip("-")
+print(json.dumps({"status": "ok", "output": slug}))
+```
+
+`{ "status": "error", "error": "why" }` fails the step with that reason. Anything the program writes to stderr is
+logged at `debug`. `idempotencyKey` is the same for the same call, so a service can answer a repeat with the result
+it already has. A command is an argument list, never a shell string; both kinds time out (30 s by default). An
+endpoint is named in the module, not in a recipe: `tokenEnv` and `signingKeyEnv` name environment variables (a
+bearer token, and a secret that signs the body as `x-opencraw-signature: sha256=…`), and `allowedHosts` refuses an
+endpoint outside the hosts you list.
+
+**Work that takes a while.** A handler that is not done answers `{ "status": "pending", "retryAfterMs": 5000 }`.
+OpenCraw waits that long (a second when it says nothing) and asks again with the same request and the same
+`idempotencyKey`, until the handler answers `ok` or `error`. `maxWaitMs` bounds the whole wait (120 s by default) and
+the call fails with "still pending" when it passes. The handler starts the work on the first request and looks it
+up by the key on later ones, which is what makes the repeat safe:
+
+```js
+price: httpHook('price', 'https://rates.example.com/price', { maxWaitMs: 300_000 }),
+```
+
+```python
+from opencraw import HookPending, serve_hook
+
+def price(value, args, request):
+    job = jobs.get(request["idempotencyKey"]) or jobs.start(value)
+    if not job.done:
+        raise HookPending(retry_after_ms=2000)
+    return job.result
+
+serve_hook(price)
+```
+
+**Pushing the result back.** On a host that can receive it (the durable host), the request also carries
+`callback: { "url": "…" }`, a URL valid for that one call. A handler that answers `pending` may then post
+its result there instead of being asked again: the same `{ "status": "ok", "output": … }` or
+`{ "status": "error", "error": … }` (published as `schemas/callout-resolution.schema.json`), never `pending`
+again. The durable host takes it with its `callouts` option (see
+[azure-durable.md](azure-durable.md#8-hooks-that-take-a-while)); captcha solvers and access plugins behind the
+same schema are tracked in issue #201.
+
+### 6.1 Captcha solvers
+
+A captcha solver gets past one challenge on the live page. The engine does the rest: finding challenges,
+checking the page afterwards, retrying, rotating, and capping what a run spends.
+
+```ts
+import type { CaptchaSolver } from '@opencraw/core'
+
+const solver: CaptchaSolver = {
+  name:  'my-solver',
+  solve: async (challenge, { page, lease, attempt, signal, log }) => {
+    // challenge: { kind, url, siteKey?, action?, selector?, field?, refresh? }
+    const token = await myService.solve(challenge.kind, challenge.siteKey, challenge.url, { signal })
+    await page.locator('[name="g-recaptcha-response"]').evaluate((field, value) => { field.value = value }, token)
+    await page.locator('form').first().evaluate(form => form.submit())
+    return { status: 'solved' }            // or { status: 'failed', reason: 'balance is zero' }
+  },
+}
+const crawler = createCrawler({ captchaSolvers: [solver] })
+```
+
+A recipe names it in `session.captcha` (checked after every navigation, click and key press) or in a
+`captcha` step (one known point, such as a login form):
+
+```json
+"session": { "captcha": { "solver": "my-solver", "verify": { "selector": "#results" }, "maxSolves": 5 },
+             "onBlock": { "solve": true, "rotate": true } }
+```
+
+| Key | Default | Meaning |
+|---|---|---|
+| `solver` | | A registered solver's name. An unknown name fails the recipe before its first page. |
+| `detect.selector` | the common widgets | Where challenges are: visible `.g-recaptcha`, `.h-captcha`, `.cf-turnstile`, or their iframes. |
+| `verify` | `{ gone: true }` | How a solve is confirmed: the challenge is `gone` and/or an element (`selector`) appears; a `failure` element showing refuses it at once; `timeoutMs` (10000) is how long the page has. The solver's `solved` is only a claim. |
+| `attempts` | 3 | Solves tried per challenge. |
+| `timeoutMs` | 120000 | Time one solve may take; then `signal` aborts and the attempt fails. |
+| `maxSolves` | 10 | Solves the whole run may spend, rotations and bootstrap included. `0` detects without paying. |
+
+What happens:
+
+1. **Detect.** The first visible match gives `kind` (`recaptcha-v2`, `hcaptcha`, `turnstile`, `image`,
+   `unknown`; `recaptcha-v3` in a `captcha` step only), the `siteKey` (`data-sitekey`, or the iframe's `k` /
+   `sitekey`) and a `selector` for the widget.
+2. **Solve.** Each attempt spends one solve of `maxSolves`. The solver applies its answer on the page
+   (injects a token and submits, types a text). A throw or a timeout is a failed attempt.
+3. **Verify.** Up to 10 s for the page to confirm. A failed attempt re-detects (a widget re-renders after a
+   wrong answer), or reloads when the widget is gone without confirming.
+4. **Give up.** After `attempts`, or when `maxSolves` is spent, the step fails with a `CaptchaError`. It is a
+   block, so `onBlock.rotate` retries the step on a new IP (often an easier challenge, or none), then the step's
+   `onError` applies.
+
+**Form captchas.** A `captcha` step with `image` is a captcha the site checks only when its form is posted
+(an image code, a field, the form's button). `refresh` and `field` go to the solver with the challenge;
+`submit` (clicks, fills, key presses, selects, waits, scripts) posts the form after the solver on every
+attempt; `verify.selector` is required, and `verify.failure` names the refusal message. The solver fills the
+field and returns `solved` (or `{ status: 'solved', submitted: true }` when it, or a person, posted the form);
+its optional `verdict` hook hears the page's answer. `"solver": "manual"` is built in: a person solves it in a
+headed browser. See [captcha.md](captcha.md#form-captchas-checked-when-the-form-is-posted).
+
+Solvers get the access `lease`: token services solve faster, and more often correctly, through the same proxy
+as the browser. Api recipes cannot solve (there is no page): with `session.captcha`, a fetched page showing a
+widget fails as a block that says so. Solve it in `session.bootstrap` and keep the cookies.
+
+Solving captchas can break a site's terms of service. [captcha.md](captcha.md) has a working solver for
+CapSolver, the costs, and when not to.
+
+---
+
+## 7. Policies: what happens when something is missing or fails
+
+**A step fails** (selector without match, HTTP 404, timeout, hook throws). The policy is the step's
+`onError`, else the recipe's, else `fail`:
+
+| Policy | Effect |
+|---|---|
+| `fail` | The recipe stops with a `StepFailure` naming the step path. The next input recipe still runs unless the crawler was created with `onRecipeError: 'stop'`. |
+| `skip` | The id stays unset, the walk continues, a `step:skip` event is reported. |
+| `retry` | The step is tried up to `attempts` times in all, the first included, waiting `backoffMs`, then 2 × `backoffMs`, and so on (linear) between tries; then it is treated as `fail`. |
+
+**A request fails in passing** (a dropped connection, a timeout, a 503, a 429). Before any of the above, the
+engine sends the same request again: that is `limits.retry`, and it is **on by default**.
+
+```json
+"limits": { "retry": { "attempts": 4, "backoffMs": 500, "maxDelayMs": 20000, "statuses": [429, 502, 503, 504] } }
+```
+
+| Key | Default | Meaning |
+|---|---|---|
+| `attempts` | `3` | Tries per request, the first included. `1` turns it off. |
+| `backoffMs` | `1000` | The first pause. It doubles on each retry, give or take 25% so parallel requests do not retry in step. |
+| `maxDelayMs` | `30000` | The longest pause. A server whose `Retry-After` asks for longer is not retried: it means "come back much later". |
+| `statuses` | `[408, 425, 429, 500, 502, 503, 504]` | Answers retried. Connection failures always are: resets, refusals, timeouts, a DNS lookup that could not run, a proxy that dropped the tunnel. A host that does not exist is not. |
+| `forMs` | none | A time budget: keep retrying until this long after the first try. Without `attempts`, tries are not counted; with it, whichever runs out first. No pause runs past the budget. |
+
+- It covers every `goto`, `request` and `next.url` page, and the page a `next.selector`, `click`, `press` or
+  `select` navigates to (retried by loading that URL again; a form post is not retried). A `Retry-After` (seconds or a date) is honoured, and it
+  holds back **every** request to that site, not only the one that got it.
+- A retry is the same request again, on the same access lease. It does not spend the step's `onError` retries,
+  and each one is a `request:retry` event (`↺` in the trace).
+- When the tries run out, the last answer counts. A 429 or 403 that is still there becomes a block
+  (`session.blockedWhen`), so `onBlock.rotate` takes over; a 503 fails the step, and the step's `onError` applies.
+- The crawler sets the default for every recipe: `createCrawler({ retry: { attempts: 5 } })`, CLI `--retries <n>`.
+  A recipe's `limits.retry` wins over it.
+
+Like redialling a busy number: wait a moment, dial again, give up after a few tries. If the other end said "call
+back in a minute", wait that minute.
+
+**A site that goes down for a while.** Some sites fail every request for minutes at a time, then come back. For
+an unattended run, give the retry a time budget instead of a number of tries. This waits out up to 30 minutes of
+404s, trying about once a minute:
+
+```json
+"limits": { "retry": { "forMs": 1800000, "backoffMs": 5000, "maxDelayMs": 60000, "statuses": [404, 429, 500, 502, 503, 504] } }
+```
+
+Retry 404 only for a site known to answer it for an outage: on most sites a 404 means the page is not there.
+
+**A mapped value is missing** (`undefined`, `null`, `""`, and in a field that is not `string` or `json` text
+with nothing but spaces; an empty list is a value). This holds for every field type: an empty table cell
+mapped to an `integer` is missing, not a number that failed to parse. The policy is the
+mapping rule's `onMissing`, else the field's, else `default` when the field has a `default`, else the
+recipe's, else `fail` for required fields and `null` otherwise:
+
+| Policy | Effect |
+|---|---|
+| `fail` | The recipe stops (`MappingFailedError`). |
+| `skip-record` | This record is dropped and reported (`record:reject`); the walk continues. |
+| `null` | The field is `null` (the field must be `nullable` or not `required`). |
+| `default` | The field's `default`, coerced to the field's type. |
+
+**A value cannot be coerced** (`"call us"` into a `number`, a relative link into a `url`) follows the same
+missing-value policy: `skip-record` drops the record, anything else stops the recipe. The message names the
+field and its type, not a transform the rule doesn't have: `mapping failed: price: number field: no number in
+"call us"`.
+
+**A transform throws** (a `number` op on `"OTR £"`, a hook error): the recipe stops, unless the mapping rule
+itself says `onMissing: "skip-record"`, which reads as "without this field the record is worthless" and
+drops just that record. Use it on the price of a table whose header row you cannot select away.
+
+---
+
+## 8. Loading, binding and running
+
+```ts
+import { createCrawler, jsonLinesSink, loadRecipeSet, traceLine } from '@opencraw/core'
+
+const recipes = await loadRecipeSet({ output: 'recipes/movie.output.json', inputs: ['recipes/'] })
+const crawler = createCrawler({
+  sink:    jsonLinesSink('out/movies.jsonl'),          // default: memorySink()
+  hooks:   { positive: input => Number(input) > 0 },
+  dedupe:  'run',                                       // 'run' | 'recipe' | 'off'; worker mode defaults to 'recipe' (per item)
+  onRecipeError: 'continue',                            // or 'stop'
+  browser: { headless: true },                          // Playwright launch settings
+  onEvent: event => { const line = traceLine(event); if (line) console.log(line) },
+})
+const report = await crawler.run(recipes)
+await crawler.close()
+```
+
+`loadRecipeSet` takes the output recipe apart from the inputs; `loadRecipes(source)` takes one source holding
+all of them and finds the output recipe by its `kind`. Either reads recipes from anywhere, not only files:
+a path to a `.json` or `.jsonl` file or a directory of them, JSON or JSON Lines text, the same as bytes
+(`Buffer`, `ArrayBuffer`, `Blob`, `File`, a stream), decoded objects, or an array mixing these. A server
+holding uploaded recipes in memory calls `loadRecipes(blob)` and never writes a file. The cli takes paths
+only.
+
+`loadRecipeSet` parses every file against its schema (`RecipeValidationError` lists every problem with its
+JSON path) and then **binds** the inputs to the output (`RecipeBindingError`): every mapping key names an
+output field and every `each` field a field of the items, every `from` starts with a known id, required fields
+are covered, every `default` coerces to its field's type and passes its rules, `onMissing: "default"` has a
+`default`, web steps stay in web recipes, `next.selector` only in web mode, one emitting construct per path, and
+no step `id` or `forEach` `as` rebinds a name already bound on its path. Hook names are checked when the crawler
+runs the set, before any request (§6).
+
+The report gives, per recipe: `emitted`, `rejected`, `duplicates`, `skipped` (records a resumed run already
+had), `stepsSkipped` (steps whose `onError: skip` swallowed a failure: a check that found nothing), `pages`,
+`durationMs`, and `error` when the recipe stopped. `pages` counts every `page:visit`: each page a `goto`, a
+`next` or a navigating `click`, `press` or `select` reached, and each `request` that got an answer. An answer
+that fails the step still counts (a 404 is a page that was fetched); a request or navigation that got no
+answer at all (a dropped connection, a refused one) does not. The sink summary says how many records were written and where.
+
+**Resuming.** A long crawl that dies halfway does not have to start over. Open the sink in append mode and
+ask the crawler to resume:
+
+```ts
+const crawler = createCrawler({ sink: jsonLinesSink('out/movies.jsonl', { append: true }), resume: true })
+```
+
+In append mode every line carries the record key as `_key`, and `open` reads the keys already in the file.
+With `resume`, a record whose key the sink has is not written again: it is counted as `skipped` and
+reported as `record:skipped`. The steps still run (the engine has to reach the record to know its key), so
+a resumed run costs the requests but not the duplicates. Any sink can support this by implementing `has(key)`;
+`memorySink` does. `resume` with a sink that cannot answer throws at `createCrawler`.
+
+**Running recipes you don't trust.** A service that runs other people's recipes must not become a way into
+its own network or files. `allowedHosts` limits every request the crawler makes:
+
+```ts
+const crawler = createCrawler({ allowedHosts: ['example.com', '*.shop.example', 'api.example.org:8443'] })
+```
+
+- A pattern is a host, `*.host` (its subdomains and the host itself), `host:port`, or `*` for any host.
+- It holds wherever a request leaves: navigations, the page's own requests (a script's `fetch`, images,
+  frames), web sockets, `request` steps, and every redirect hop, which is checked before it is followed.
+- `file:` is refused whatever the list says, a relative `file:` URL too (after it resolves); `data:`, `blob:` and
+  `about:` never leave the page and pass.
+- Service workers are blocked, since their requests would bypass the check. A remote browser's own service
+  workers (`cdp` profiles) can't be blocked.
+- A refused request fails the step: `HostNotAllowedError` in api mode, `net::ERR_BLOCKED_BY_CLIENT` in a
+  browser, both reported as `errorKind: 'host'`.
+- The check is by name. A public name that resolves to a private address is not caught, so run the service
+  in a network that can't reach anything it shouldn't.
+
+### 8.1 Change detection
+
+A price list crawled every week answers "what is there", but the question is usually "what changed". Records
+have keys, so two runs compare **record by record**, like a spreadsheet's "compare versions" rather than a text
+diff: a reordered file is no change, and a changed price shows as that field, before and after.
+
+```sh
+opencraw run recipes/ --out prices.jsonl --diff prices.jsonl --changes changes.jsonl
+```
+
+reads the previous `prices.jsonl` before crawling, overwrites it, and reports `added` / `removed` / `changed`
+(nested fields dotted: `price.amount: 24950 → 23950`), ignoring the fields the engine stamps (`generated:
+now`, `uuid`). From code: `diffRecords(previous, current, diffOptionsFor(output))`; from the MCP server, the
+`diff` tool.
+
+It also watches the recipe: a run with under half the previous run's records is flagged. A site that changed
+its markup rarely makes a recipe fail; it makes it find less.
+
+### 8.2 Events and the trace
+
+Everything the engine does is an event: `recipe:start` / `recipe:finish`, `page:visit` (with the HTTP status; one per page counted in `pages`),
+`access:lease` / `access:blocked` / `access:rotate`, `request:retry`, `captcha:detected` / `captcha:solve` / `captcha:solved` /
+`captcha:failed` / `captcha:budget`, `step:start` /
+`step:finish` / `step:retry` / `step:skip` (with the step type, its id and its path such as
+`steps.8.steps.2`), `step:branch`, `step:kept`, in worker mode `window:open` / `window:close` / `windows:change` /
+`browser:restart` / `item:finish`, `record:emit` / `record:reject` / `record:duplicate` / `record:skipped`,
+`warning`, `error`. `traceLine`
+turns an event into one indented line, so a recipe's route reads as a tree:
+
+```text
+▶ tmdb (api)
+  ⇢ page 1  https://www.themoviedb.org/movie/949-heat
+  · steps.0  request entry  312 ms
+  · steps.1  extract entry_ld  4 ms
+  · steps.3  extract actor_name  2 ms
+  ⇢ page 1  https://www.themoviedb.org/person/1158-al-pacino
+  · steps.5  request actor_page  401 ms
+    · steps.8.steps.0  extract title  1 ms
+    ↷ steps.8.steps.2  extract year  skipped: no match for td.year
+  ✚ record ["Al Pacino","St. Vincent","tmdb"]
+■ tmdb: 10 emitted, 0 rejected, 0 duplicates, 2 pages, 3006 ms
+```
+
+The example runners print it with `--trace`.
+
+---
+
+## 9. Method: how to write a recipe for a new site
+
+A quick way to see where a site's data lives before writing selectors by hand: `@opencraw/cli`'s
+`opencraw probe <url>` (add `--browser` to also watch the JSON a script fetches after load) lists JSON-LD
+blocks, inline JSON objects, `.json` URLs, script hosts and API-looking links found in the page.
+`@opencraw/mcp` gives an agent the same `probe`, plus `validate`, `run`, `list_recipes` and `diff`, as MCP tools
+instead of a terminal command.
+
+1. **Find the data before the markup.** Fetch the page with a browser user agent and look for
+   `application/ld+json`, inline JSON (`__NEXT_DATA__`, `window.__STATE__`) or an XHR the page calls. Data
+   beats selectors: it does not move when the design changes.
+2. **Decide the mode.** Data in the HTML → `api`. Data rendered by scripts, interactions needed, or a bot
+   wall on plain HTTP → `web`.
+3. **Map the route.** Listing → detail, or entry → link → list. Each hop is a `goto` / `request` whose URL
+   is a template over an id extracted on the previous page.
+4. **Write the output first.** What is required, what is a key, what may be null.
+5. **Extract with ids named after what they hold**, not where they come from (`actor_href`, not `link3`).
+6. **Say what is optional.** `onError: skip` on extracts that legitimately miss; `nullable` on their fields.
+   Leave everything else strict so a site change fails loudly.
+7. **Run with `maxRecords: 3` and `--trace`.** Check the first record by hand against the page. Then lift
+   the limit.
+8. **Write the `description`.** The route, the reason it works, the element that is the pivot.
+
+---
+
+## 10. Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `no match for <selector>` on the first extract of a page | wrong selector, or the page is a challenge / login wall | print `page:visit` URLs; fetch the page and look; on a bot wall switch to `web` or run through a proxy ([access.md](./access.md)) |
+| `none of the N texts bound to "x" is JSON` | the scripts are not JSON-LD, or hold JavaScript | check the block; `evaluate` in web mode is the fallback |
+| a field carries the *next* row's value | the row selector matched a wrapper element first | select the innermost repeating element (§4.5) |
+| `"…" is not an absolute URL, and there is no page to resolve it against` | a relative URL before any navigation, or a start URL that is a plain path | make the first step `goto` / `request` with `{{start.url}}`; give a start point a full URL, or `file:` for a local file (§3.2) |
+| `mapping.x.from: "y" does not start with a known id` | typo in an id, or the id is bound only in a bootstrap | ids are per recipe; bootstraps produce a session, not ids |
+| `record rejected: title: missing` on every record | the id is bound in a sibling scope, not the emitting one | extract inside the `forEach` body, or before it |
+| the crawl stops after page 1 in web mode | the body navigated away and `next.selector` is not on the page | the engine returns to the listing page; if the listing is itself reached by clicking, use `next.url` |
+| `HTTP 202` with an empty body | AWS WAF challenge | a real browser from a non-flagged IP: a residential proxy profile ([access.md](./access.md)) |