@opencraw/mcp 0.1.45 → 0.1.48

package/docs/recipes/azure-durable.md

@@ -0,0 +1,180 @@
+<p align="center">
+  <img src="../assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
+# Running OpenCraw as an Azure Durable Functions app
+
+`@opencraw/azure-durable` turns a Function App into an OpenCraw service: callers send recipes (or the name of a
+stored set) over HTTP and read the result from the Durable status response. This page covers deploying it. The
+API itself is in the [package README](../../packages/azure-durable/README.md).
+
+Think of it as a central kitchen. Every crawler used to own its ovens: Playwright, Chromium, the captcha reader,
+the proxies. Now they send orders to one kitchen that keeps its ovens hot, and pick up the dish.
+
+## 1. The app
+
+```text
+my-crawler-host/
+  host.json
+  package.json          "main": "dist/index.js", dependencies: @opencraw/azure-durable, @azure/functions, durable-functions
+  src/index.ts          registerOpenCraw({ ... })
+  recipes/              the named recipe sets it serves
+  Dockerfile
+```
+
+`src/index.ts` is the whole app: see [Register it](../../packages/azure-durable/README.md#register-it).
+
+## 2. `host.json`
+
+```json
+{
+  "version": "2.0",
+  "functionTimeout": "-1",
+  "extensions": {
+    "durableTask": {
+      "maxConcurrentActivityFunctions": 12,
+      "maxConcurrentOrchestratorFunctions": 50,
+      "overridableExistingInstanceStates": "NonRunningStates"
+    }
+  }
+}
+```
+
+- **`maxConcurrentActivityFunctions`**: at least the pools' `maxWindows` plus two. An item waiting in a pool
+  holds an activity slot, so with fewer slots than windows a pool can never fill. The host warns at startup when
+  it sees fewer.
+- **`overridableExistingInstanceStates: NonRunningStates`**: an item id can run again once it has finished (a
+  `neutral` item sent back), and the starter answers a second start while it runs with that instance's status, so the item runs once.
+- **`functionTimeout: -1`**: activities run as long as a crawl needs. The Consumption plan caps them at 10
+  minutes and can't run Chromium anyway.
+
+## 3. The container
+
+[`apps/azure-host`](../../apps/azure-host/README.md) is a complete app built this way. Its
+[Dockerfile](../../apps/azure-host/Dockerfile) installs the pruned build (`npx nx run @opencraw/azure-host:prune`:
+the app's own `package.json`, lockfile and the workspace packages it uses) on the Functions Node 24 image, then
+Chromium:
+
+```dockerfile
+FROM mcr.microsoft.com/azure-functions/node:4-node24
+ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
+    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
+    PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
+WORKDIR /home/site/wwwroot
+COPY dist/package.json dist/package-lock.json ./
+COPY dist/workspace_modules ./workspace_modules
+RUN npm ci --omit=dev --ignore-scripts && npx playwright install --with-deps chromium
+COPY host.json ./
+COPY dist/*.js ./dist/
+COPY dist/functions ./dist/functions
+COPY dist/assets ./dist/assets
+```
+
+The build happens before the image, so the image needs no compiler. Run it on an **Elastic Premium** or
+**App Service** plan (Linux, custom container). Chromium needs about 150 to 300 MB per window: size the plan for
+`maxWindows` windows plus the host.
+
+## 4. Scale: one pool per instance
+
+A warm pool lives in one process's memory. Azure sends each activity to whichever instance is free, so a
+Function App scaled to three instances has up to three pools for the same crawl id. Each is warm and adapts on
+its own, but together they run three times the windows, and three times the traffic from the same outbound IPs.
+
+For a job that must be a singleton (a site that blocks an IP that asks too much), pin the app to one instance:
+`functionAppScaleLimit: 1` on Premium, or a fixed single instance on App Service. Scale up (a bigger instance),
+not out. The per-site `throttle` is per process too, for the same reason.
+
+## 5. Security
+
+- **Authentication.** Routes need a function key by default (`authLevel: 'function'`). For per-team access, turn
+  on App Service authentication (Entra ID) and use `identify: request =>
+  request.headers.get('x-ms-client-principal-name') ?? undefined`. That header is trustworthy only because App
+  Service authentication sets it and strips any a client sends. Without it, anyone can send that header.
+- **Allowed hosts.** The host fetches whatever a recipe points at, so without a limit it is a way into your
+  network. `allowedHosts` is enforced on every request, redirect and web socket, and `file:` is always refused.
+  The check is by host name: also run the app where it can't reach anything it shouldn't (a VNet with no route
+  to internal services).
+- **Code stays in the host.** Hooks, captcha solvers and access profiles are registered in `registerOpenCraw`.
+  A caller can only use them by name.
+
+## 6. Writing recipes on the host
+
+With `mcp: true`, the host also serves `/api/mcp`: probe, validate, sample runs, full runs and `publish`, over MCP.
+Recipes written there are written where they will run. A recipe that works in the MCP session works in
+production, because it is the same IP, proxies and browser.
+
+Keep published versions in Blob storage so they survive restarts:
+
+```ts
+registerOpenCraw({
+  // ...
+  recipes:    blobRecipes({ connectionString: process.env.AzureWebJobsStorage ?? '', shipped: [/* the sets in the repo */] }),
+  mcp:        { sampleRecords: 20, sampleMs: 60_000 },
+  identify:   request => request.headers.get('x-ms-client-principal-name') ?? undefined,
+  canPromote: caller => caller === 'lead@example.com',
+})
+```
+
+`publish` stores a draft, which production refuses. Someone whom `canPromote` allows then promotes it with
+`POST /api/recipes/{name}/{version}/promote`. A version never changes once written; the next change is the next
+version. A pool running version 3 keeps running it until it closes.
+
+## 7. A caller that keeps the pool busy
+
+```ts
+const inFlight = new Set<Promise<void>>()
+let wanted = 2
+for (const item of items) {
+  while (inFlight.size >= wanted) await Promise.race(inFlight)
+  const task = (async () => {
+    const started = await fetch(`${host}/api/jobs/${crawlId}/items?code=${key}`, { method: 'POST', body: JSON.stringify({ recipe, item, jobSize: items.length }) })
+    const { statusQueryGetUri } = await started.json()
+    const result = await poll(statusQueryGetUri)             // until runtimeStatus is Completed or Failed
+    const { output } = result
+    if (output.outcome === 'neutral') items.push(item)       // send it back later
+    // One more in flight while windows sit idle, up to the pool's maximum.
+    wanted = Math.max(1, output.pool.windows + (output.pool.idle === 0 ? 1 : 0))
+    await save(item, output)
+  })()
+  inFlight.add(task)
+  void task.finally(() => inFlight.delete(task))
+}
+await Promise.all(inFlight)
+```
+
+The pool grows only while every window is busy, and lets a window go once it has waited `windows.idle.afterMs`.
+So a caller that keeps one more item in flight than the pool has windows lets it grow, and a caller that slows
+down lets it shrink.
+
+## 8. Hooks that take a while
+
+A hook behind a service (`httpHook`, or a `commandHook` that starts a job) can answer `pending`. Without
+`callouts`, the activity asks again until the service settles (`maxWaitMs` on the hook, 120 s by default),
+which holds the activity for that long. With `callouts` the run does not wait inside the activity:
+
+```ts
+registerOpenCraw({
+  allowedHosts: ['example.com'],
+  hooks:        { price: httpHook('price', 'https://rates.example.com/price', { tokenEnv: 'RATES_TOKEN' }) },
+  callouts:     { signingKeyEnv: 'OPENCRAW_CALLOUT_KEY', waitMs: 2 * 60 * 60 * 1000 },
+})
+```
+
+- The request to the service carries `callback: { "url": "https://app/api/callouts/<token>/resolve" }`. The
+  service answers `{ "status": "pending" }` and, when it is done, posts `{ "status": "ok", "output": … }` (or
+  `{ "status": "error", "error": "…" }`) to that URL. The shapes are `callout-request.schema.json` and
+  `callout-resolution.schema.json` in `@opencraw/core`.
+- The recipe stops, and the orchestration waits for that result with a durable timer beside it. When the
+  result is posted, the recipe **runs again from the start**, and the call that was waiting returns the result
+  without being made: steps before it run twice, so this suits fetch and extract steps, and a service must
+  answer a repeat of a call (same `idempotencyKey`) with what it already has.
+- A recipe whose result does not arrive within `waitMs` (default 1 hour) fails with the handler named; the
+  other recipes of the job are not affected. A recipe is resumed at most 25 times.
+- **The token is the credential.** The resolve route is registered with `authLevel: 'anonymous'` because the
+  services that post back hold no function key. The token is an HMAC (with the key in `signingKeyEnv`) of the
+  job, the recipe and the call, with an expiry, so a token settles that one call and nothing else, and a
+  second post finds the job no longer waiting and gets 410. Keep the key in the Function App settings, like the
+  other secrets; rotating it refuses the tokens in flight.
+- `publicUrl` is where services post to (`https://app.example.com/api`). Left out, it is the start URL of the
+  job without `/crawl`: set it behind a proxy or a custom domain.
+- Only `POST /crawl` takes it. Jobs started through `/mcp` and items sent to `/jobs` poll a pending hook, as without `callouts`.

package/docs/recipes/captcha.md

@@ -0,0 +1,204 @@
+<p align="center">
+  <img src="../assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
+# Captchas
+
+A captcha is the site asking whether a human is there. OpenCraw can hand the question to a **solver** you
+register (usually a paid service) and carry on once the page accepts the answer. The engine does the parts
+that are the same for every service: finding the challenge, checking the answer worked, retrying, rotating the
+IP, and capping what a run may spend. The solver does one thing: get past one challenge on the live page.
+
+Think of the engine as the driver and the solver as the passenger with the toll money: the driver spots the
+toll gate, stops, and only drives on once the barrier is up. The passenger pays.
+
+The recipe reference is [authoring §6.1](authoring.md#61-captcha-solvers). This page is the why and the how.
+
+## First: should you?
+
+Solving a captcha can break the site's terms of service, and in some places it matters legally. Before adding
+a solver:
+
+- **Look for the data another way.** An official API, a bulk export, a feed, a partner agreement.
+  [authoring §9](authoring.md#9-method-how-to-write-a-recipe-for-a-new-site) finds JSON the page loads
+  anyway, which often skips the page that shows the captcha.
+- **Slow down first.** Many challenges only appear under load: `limits.delayMs` and a residential proxy
+  (`session.access`, [access.md](access.md)) often make them go away.
+- **Log in, if the data is behind an account.** A `session.bootstrap` login, solved once and saved with
+  `saveTo`, costs one solve instead of one per page.
+
+## The four places a challenge appears
+
+| Where | Recipe | What the engine does |
+|---|---|---|
+| After a navigation, click or key press | `session.captcha` | Looks for a visible widget after every `goto`, `click`, `press` and pagination click, and solves it before the next step. |
+| As the block page itself (a 403 with a challenge) | `session.onBlock: { solve: true }` | A block whose page shows a widget is solved instead of failing; with `rotate: true` too, rotation is the fallback. |
+| At one known point (a login form, an invisible reCAPTCHA v3) | a `captcha` step | Solves whatever challenge is on the page, v3 included; none is fine. |
+| In a form, checked only when the form is posted (an image code) | a `captcha` step with `image` | The solver fills the answer, the step's `submit` posts the form, the page says yes or no. [Below](#form-captchas-checked-when-the-form-is-posted). |
+
+```json
+{
+  "session": {
+    "captcha": { "solver": "capsolver", "verify": { "selector": "#results" }, "attempts": 2, "maxSolves": 5 },
+    "onBlock": { "solve": true, "rotate": true, "attempts": 2 }
+  },
+  "steps": [
+    { "type": "goto", "url": "https://shop.example/search?q={{vars.q}}" },
+    { "type": "extract", "id": "items", "selector": "#results .item", "kind": "css", "many": true }
+  ]
+}
+```
+
+## What a solve looks like
+
+```text
+  ⇢ page 1  https://shop.example/search?q=pandina  [403]
+  ⚿ captcha recaptcha-v2 on https://shop.example/search?q=pandina
+  ✗ captcha attempt 1 failed: the page still shows the challenge
+  ✓ captcha solved by capsolver (attempt 2, 21480 ms)
+  · steps.0  goto  23105 ms
+```
+
+1. **Detect.** The first *visible* match of `detect.selector` (default: `.g-recaptcha`, `.h-captcha`,
+   `.cf-turnstile` and their iframes). The solver gets `kind`, `url`, `siteKey`, `action` and a `selector`.
+2. **Solve.** One attempt spends one of `maxSolves`. The solver has `timeoutMs` (default 2 minutes), then its
+   `signal` aborts.
+3. **Verify.** The solver's `solved` is only a claim. The engine waits up to 10 s for the challenge to be gone
+   (`verify.gone`, default) and for `verify.selector` to appear, if given. A token the site rejects is a
+   failed attempt, and the next one starts from what the page shows then.
+4. **Give up.** After `attempts` (default 3), the step fails with a `CaptchaError`. That is a block: with
+   `onBlock.rotate`, the run takes a new IP, reopens the session and retries the step. A fresh IP often gets an
+   easier challenge, or none.
+
+The report counts them: `captchas: { detected, solved, failed }`.
+
+## Form captchas: checked when the form is posted
+
+Some sites have no "check" for their captcha: it is a code drawn as an image, a text field, and the form's
+own button. Whether the code was right is only known once the form is posted and the page comes back, with
+the result or with "Invalid CAPTCHA." and a new image. The public Vahan registrations report works this way,
+and so does this step:
+
+```json
+{ "type": "captcha", "solver": "tesseract",
+  "image":   "#captchaImage",
+  "refresh": "#captchaImg",
+  "field":   "#externalCaptcha",
+  "submit":  [ { "type": "click", "selector": "#applyTrigger" } ],
+  "verify":  { "selector": "#makerDynamicReportHeader", "failure": "#captchaMsg", "timeoutMs": 30000 },
+  "attempts": 5 }
+```
+
+[`@opencraw/captcha-tesseract`](../../packages/captcha-tesseract) is a ready reader for such image codes
+(Tesseract, bundled, nothing downloaded). Put the step after the steps that fill the form. Each attempt:
+
+1. **The solver does its part.** It gets the image (`challenge.selector`), the answer field (`challenge.field`)
+   and the refresh control (`challenge.refresh`). A reader reads the image, clicks refresh when it is unsure
+   (that costs nothing: no form is posted), and fills the field.
+2. **The engine runs `submit`**: the button, or the fills a form that empties itself after a wrong code needs
+   first. Clicks, fills, key presses, selects, waits and scripts; each may have a `when`.
+3. **The page answers.** `verify.selector` showing is a yes. `verify.failure` showing is a no, at once, not
+   after a timeout, even when the page already showed the message from the attempt before. The engine tells
+   the new page, or a newly drawn element, from the old one. The same holds for the yes: on a page that
+   already shows the last report (a [worker window](./worker-mode.md) on its next item, a site that swaps the
+   report in without a reload), only a report drawn after this submit counts. The page keeps a captcha after a success (the next report needs a new
+   one), so `gone` is not asked.
+4. **The solver hears the verdict** (`verdict`, below), and the next attempt starts on the new image.
+
+A form captcha needs `verify.selector`. Every report costs one captcha: a crawl of many reports is many
+solves, and `maxSolves` caps them.
+
+### The manual solver
+
+`"solver": "manual"` is built in, for headed runs: a person types the code in the browser window and presses
+the button. The solver waits (10 minutes, or the step's `timeoutMs`) until the page navigates or the success
+element shows, and the engine checks the page as for any solver; the `submit` steps do not run, since the
+person submitted. Paired with a reader's audit, it collects what people typed and whether the site took it:
+the labelled images a reader is tuned on.
+
+## Money
+
+Every attempt is paid, successful or not, and a detector that matches the wrong element would drain a balance.
+Hence `maxSolves` (default 10 per recipe run, rotations and bootstrap included; `0` detects without paying).
+When it runs out, the challenge is left unsolved with a `captcha:budget` event, and the step fails.
+
+## Writing a solver
+
+```ts
+import type { CaptchaSolver } from '@opencraw/core'
+
+export const mySolver: CaptchaSolver = {
+  name: 'my-solver',
+  async solve (challenge, { page, lease, attempt, signal, log }) {
+    const token = await myService.solve({ kind: challenge.kind, siteKey: challenge.siteKey, url: challenge.url, signal })
+    await page.locator('[name="g-recaptcha-response"]').evaluate((field, value) => { field.value = value }, token)
+    await page.locator('form').first().evaluate(form => form.submit())
+    return { status: 'solved' }
+  },
+}
+```
+
+- **Do your part, then return `solved`.** A reader fills `challenge.field`; a widget solver puts the token
+  where the widget would (`g-recaptcha-response`, `cf-turnstile-response`) and runs its `data-callback` or
+  submits its form. When the recipe has `submit` steps, the engine runs them after you; return
+  `{ status: 'solved', submitted: true }` when you submitted yourself (or a person did).
+- **`verdict(challenge, { status: 'solved' | 'rejected', reason? })`**, optional, tells you what the page said
+  after a `solved`: for an audit log, or to report a bad token to a service that refunds them.
+- **`close()`**, optional, runs when the crawler closes: let a worker or a connection go.
+- **`timeoutMs`**, optional, is your default time per solve when the recipe does not set one.
+- **Throw or return `failed`** with a reason: both count as a failed attempt, and the reason reaches the trace.
+- **Honour `signal`.** Stop polling your service when it aborts.
+- **Use `lease` when the service accepts a proxy**: tokens are often tied to the IP that asked for them.
+
+[`examples/captcha-solver/capsolver.mjs`](../../examples/captcha-solver/capsolver.mjs) is a complete solver for
+CapSolver: reCAPTCHA v2 and v3, Turnstile and image captchas, with tests against a faked API.
+
+## A solver in any language, or behind an API
+
+A solver does not have to be JavaScript. `commandCaptchaSolver` runs a program and `httpCaptchaSolver` calls a
+service; both put the answer on the page themselves, so the program or the service never needs a browser.
+
+```js
+// solvers.mjs
+import { commandCaptchaSolver, httpCaptchaSolver } from '@opencraw/core'
+
+export const captchaSolvers = [
+  httpCaptchaSolver('capsolver', 'https://solver.example.com/solve', { tokenEnv: 'SOLVER_TOKEN', allowedHosts: ['solver.example.com'] }),
+  commandCaptchaSolver('reader', ['python3', 'solvers/read_image.py']),
+]
+```
+
+A recipe names it as any solver (`"solver": "capsolver"`). The request is the callout request of
+[authoring §6](authoring.md#6-hooks) with `kind: "captcha"`, and its `input` is published as
+`schemas/callout-captcha-input.schema.json`:
+
+```json
+{ "kind": "captcha", "name": "capsolver", "idempotencyKey": "…", "args": {}, "context": { "recipeId": "shop", "scope": {} },
+  "input": { "challenge": { "kind": "recaptcha-v2", "url": "https://shop.example/gate", "siteKey": "6Lc…" }, "attempt": 1 } }
+```
+
+It answers with a `token` (a widget) or the `text` of an image captcha
+(`schemas/callout-captcha-output.schema.json`):
+
+```json
+{ "status": "ok", "output": { "token": "03AGdBq…" } }
+```
+
+- **A token** goes into the widget's response fields (`g-recaptcha-response`, `h-captcha-response`,
+  `cf-turnstile-response`), then the widget's `data-callback` runs, or its form is submitted when it has none.
+  `submit: "none"` stops there, for a recipe whose `submit` steps do it.
+- **`text`** is typed into the challenge's answer field (a `captcha` step with `image` names it in `field`).
+- **An image challenge** carries its picture in `input.image` (PNG, base64) when it names the image's selector.
+- **The proxy** of the access lease is sent in `input.proxy` only with `sendProxy: true`, since it carries the
+  proxy's credentials; services that tie a token to an IP need it.
+- **Slow services** may answer `{ "status": "pending", "retryAfterMs": 3000 }`: the solver asks again with the
+  same request until `maxWaitMs` or the solve's timeout (`timeoutMs`, 120 s). A captcha is always polled, never
+  parked on a durable host, because the page it belongs to has to stay open.
+- `{ "status": "error", "error": "out of balance" }` fails that attempt with the reason in the trace; the engine
+  retries, rotates and counts it against `maxSolves` as for any solver.
+
+## Api recipes
+
+An api recipe has no page to solve on. With `session.captcha`, a fetched page that shows a widget fails as a
+block that says so. Put the part that meets the captcha in `session.bootstrap` (a browser), keep the cookies,
+and let the requests reuse them.

package/docs/recipes/movies-example.md

@@ -0,0 +1,12 @@
+<p align="center">
+  <img src="../assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
+# Worked example: movies
+
+This write-up now lives with its example, so the recipes and the reasoning behind them sit together:
+
+- [Movies from Netflix and IMDb](../../examples/movies/README.md#how-the-recipes-were-written): api mode on JSON-LD, web mode behind a WAF.
+- [From a movie to its lead actor's films](../../examples/filmography/README.md#how-the-recipes-were-written): navigation across pages, the shifted-field trap, reading the trace.
+
+Every example is listed in [examples/README.md](../../examples/README.md).

package/docs/recipes/programmatic-authoring.md

@@ -0,0 +1,101 @@
+# Building recipes in code
+
+A recipe is JSON, and a crawl never runs code you wrote into a recipe. That does not mean you have to
+type every recipe by hand. When you have fifty similar sites, a list of regions or a shared login, build
+the recipe objects in JavaScript or TypeScript, and let the engine see only the final JSON shape.
+
+The loader accepts a plain object as a recipe source, so there is no file round trip:
+
+```js
+import { createCrawler, loadRecipes, memorySink } from '@opencraw/core'
+
+const output = {
+  kind: 'output',
+  id: 'product',
+  version: 1,
+  fields: {
+    url:   { type: 'url', required: true, key: true },
+    title: { type: 'string', required: true },
+  },
+}
+
+// One input recipe per shop: same steps, different start URL and selector.
+const shops = [
+  { id: 'shop-a', url: 'https://a.example/catalog', title: 'h1.name' },
+  { id: 'shop-b', url: 'https://b.example/products', title: '.product-title' },
+]
+
+const inputs = shops.map(shop => ({
+  kind:   'input',
+  id:     shop.id,
+  output: 'product',
+  mode:   'web',
+  start:  [{ url: shop.url }],
+  steps: [
+    { type: 'goto', url: '{{start.url}}' },
+    { type: 'extract', id: 'title', selector: shop.title, kind: 'css', take: 'text' },
+    { type: 'emit' },
+  ],
+  mapping: { url: { from: 'start.url' }, title: { from: 'title' } },
+}))
+
+const recipes = await loadRecipes([output, ...inputs])
+const report = await createCrawler({ sink: memorySink() }).run(recipes)
+```
+
+`loadRecipes` takes the output recipe and the inputs from one array and tells them apart by `kind`. If
+you keep them apart, `loadRecipeSet({ output, inputs })` takes each part separately. Both accept any
+mix of objects, JSON text, JSON Lines text, file paths and directories, so you can build some recipes in
+code and keep the rest as files.
+
+## Type checking in TypeScript
+
+`@opencraw/core` exports the recipe types, so no builder function is needed: `satisfies` checks an object
+literal against the type and keeps its own narrow type.
+
+```ts
+import type { InputRecipe } from '@opencraw/core'
+
+const recipe = {
+  kind:   'input',
+  id:     'shop-a',
+  output: 'product',
+  mode:   'web',
+  start:  [{ url: 'https://a.example/catalog' }],
+  steps:  [{ type: 'goto', url: '{{start.url}}' }, { type: 'emit' }],
+  mapping: {},
+} satisfies InputRecipe
+```
+
+A misspelt step type (`'gotoo'`) is a compile error that suggests `'goto'`, and the editor completes step
+types and keys. `OutputRecipe` does the same for an output recipe. Template expressions and bindings are
+still checked at load time, as above.
+
+## From Python
+
+`python-packages/opencraw` is a small standard-library package for building the same recipe dicts in Python
+and driving the real CLI: `input_recipe`, `output_recipe` and `step` save typing, `write_recipes` writes the
+files, `validate` and `run` call `opencraw validate` and `opencraw run` and hand back the records as dicts.
+It restates no schema, so the engine stays the one place that says what is valid. See its
+[README](../../python-packages/opencraw/README.md). It is published to PyPI as `opencraw`.
+
+## What is checked, and when
+
+Objects get exactly the validation a file gets: the schema, the template expressions, the binding of each
+input to its output. A mistake in a generated recipe throws a `RecipeValidationError` or
+`RecipeBindingError` that names the recipe by its `id` (and by its position, such as `recipes[2]`, where
+there is no id yet).
+Nothing is checked while you build the objects, only when you load them.
+
+To keep the generated recipes, write them out with `JSON.stringify(recipe, null, 2)`, which is what the CLI,
+the MCP server and the Studio read.
+
+## Boundaries
+
+- The objects are authoring-time only. Functions, class instances and anything that does not survive
+  `JSON.stringify` are not recipe content.
+- The template language stays sandboxed (no calls, no `new`). Real logic during a crawl belongs in a
+  [`hook` step](./authoring.md), the one sanctioned place for it.
+- Other languages (Python, Go, a shell script) can generate recipes the same way: write JSON, then hand
+  the files to the CLI or the MCP server. `packages/core/schemas/` has the JSON Schemas to generate
+  against. There is no Python package, because Python cannot hand a live object to the engine.

package/docs/recipes/quick-guide.md

@@ -0,0 +1,157 @@
+<p align="center">
+  <img src="../assets/opencraw-logo.svg" alt="OpenCraw" width="360">
+</p>
+
+# Writing recipes
+
+A crawl is two kinds of JSON file: **one output recipe** that says what a record looks like, and **input
+recipes** that say how to get there from a site or an API. The engine runs the inputs one after another and
+every record, whatever its source, matches the output. The formal specification is
+[requirements.md](../requirements.md); this is the working guide. To see what the engine does with each part, on
+real sites with the traces and records it produced, read [How OpenCraw works](../how-it-works/README.md).
+
+Point `$schema` at `packages/core/schemas/output-recipe.schema.json` or `input-recipe.schema.json` and your
+editor validates and completes the file as you type.
+
+## 1. The output recipe
+
+```json
+{
+  "kind": "output", "id": "product", "version": 1,
+  "fields": {
+    "url":     { "type": "url", "required": true, "key": true },
+    "title":   { "type": "string", "required": true },
+    "price":   { "type": "currency", "currency": "EUR", "required": true },
+    "inStock": { "type": "boolean", "default": false, "onMissing": "default" },
+    "images":  { "type": "array", "items": { "type": "url" } },
+    "seller":  { "type": "object", "fields": { "name": { "type": "string" } } },
+    "scrapedAt": { "type": "datetime", "generated": "now" }
+  }
+}
+```
+
+- `key: true` fields identify a record; a repeated key is dropped as a duplicate.
+- `required` fields fail the recipe when missing, unless `onMissing` says `skip-record` (drop the record),
+  `null`, or `default` (use `default`).
+- `generated` fields come from the engine: `now`, `uuid`, `sourceUrl`, `recipeId`. Do not map them.
+- Types: `string`, `number`, `integer`, `boolean`, `date` (`YYYY-MM-DD`), `datetime` (ISO 8601), `currency`
+  (`{ amount, currency }`), `url` (absolute), `enum` (`values`), `array` (`items`), `object` (`fields`), `json` (any value, kept as is).
+
+## 2. The input recipe
+
+```json
+{
+  "kind": "input", "id": "shop-web", "output": "product", "mode": "web",
+  "start": [{ "url": "https://shop.example/catalog?page=1" }],
+  "steps": [ ... ],
+  "mapping": { ... }
+}
+```
+
+`mode` is `web` (a real browser) or `api` (HTTP requests, no browser). `start` lists where to begin; each
+start point runs the whole step list with `start.url` and its `vars` in scope.
+
+### Steps
+
+Every step can have an `id` (the name of the value it produces), an `onError` policy and a `when` template.
+
+| Step | Mode | What it does |
+|---|---|---|
+| `goto` | web | Navigates to `url` (a template, relative to the current page). |
+| `click`, `fill`, `press`, `select`, `scroll`, `wait`, `screenshot` | web | Interacts with the page. `click` with `optional: true` skips a missing element. |
+| `evaluate` | web | Runs `script` in the page and binds the result. Trusted recipes only. |
+| `request` | both | Sends an HTTP request (in web mode, through the page's own session); the response becomes the current document (and the `id`, if given). `as` forces the body kind: `json`, `jsonl`, `html`, `text`, `xml`, `yaml`, `markdown`, `csv`, `xlsx`, `pptx`, `docx` or `pdf`. A relative `url` resolves against the current page, like a link. |
+| `extract` | both | Selects from the current document (or the `from` id): `kind` is `css`, `xpath` (live pages, fetched HTML and XML), `jsonpath`, `regex` (group 1 of the text) or `table` (PDF, workbook and deck tables); `take` is `text` (default), `html`, `value`, `json` or `attr:href`; `many: true` gives a list. A single `extract` that matches nothing fails, so give it `onError: { "policy": "skip" }` when the element is optional. |
+| | | A `jsonpath` extract whose `from` holds **text** parses it as JSON, and a **list of texts** (every `script[type="application/ld+json"]` of a page, extracted with `many`) becomes an array of the entries that parse: `$[*].actors[*].name` then finds the block that has actors wherever it sits. |
+| `set` | both | Binds a literal or a rendered template. |
+| `forEach` | both | Runs `steps` once per item of the list `over` (or per live element matching `selector`, web), with the item bound as `as`. `emit: true` produces one record per iteration. |
+| `paginate` | both | Runs `steps` per page, then follows `next`: `{ "selector" }` (web, clicks it), `{ "url" }` (a template), `{ "jsonpath" }` (api, a URL from the document; add `"as"` to bind a cursor instead). Stops when there is no next page, when `until` is truthy, or at `maxPages`. |
+| `if` | both | Runs `steps` when `test` renders truthy, else `else`. |
+| `collect` | both | Appends `value` to a list bound in an enclosing scope (`into`). |
+| `captcha` | web | Solves a captcha on the page, if there is one (see [captcha.md](./captcha.md)). |
+| `emit` | both | Produces a record from everything in scope. |
+| `hook` | both | Calls a handler registered in code by `name`, with `args`. |
+
+### Templates
+
+`{{path}}` reads any id in scope, the `forEach` variable (`{{item.href}}`), `{{vars.name}}`,
+`{{start.url}}`, `{{page.url}}` and `{{page.number}}`. A template that is exactly one placeholder keeps the
+value's type, so `"value": "{{links}}"` in a `set` binds the list itself, not its text (`forEach.over` takes the bare id: `"over": "links"`).
+
+### Scope, in one paragraph
+
+`forEach` and `paginate` open a fresh scope per iteration or page and drop it afterwards, so page 2 never
+sees page 1's values. Navigations inside a `forEach` (visiting every product) are fine: `paginate` returns to
+the listing page before looking for the next link. `page.url` is the current page in web mode and the last
+request's URL in api mode.
+
+### Errors
+
+`onError` on a step, else on the recipe, else `fail`: `{ "policy": "skip" }` leaves the id unset and
+continues, `{ "policy": "retry", "attempts": 3, "backoffMs": 500 }` retries, `fail` stops the recipe. A
+stopped recipe never stops the run unless the crawler was created with `onRecipeError: 'stop'`.
+
+## 3. Mapping
+
+```json
+"mapping": {
+  "url":         { "from": "page.url" },
+  "title":       { "from": "raw_title", "transform": [{ "op": "trim" }] },
+  "price":       { "from": "raw_price", "transform": [{ "op": "regex", "pattern": "([\\d.,]+)" }, { "op": "currency", "locale": "de-DE" }] },
+  "price2":      { "from": ["price_int", "price_cents"], "transform": [{ "op": "join", "separator": "." }, { "op": "currency" }] },
+  "images":      { "from": "imgs", "transform": [{ "op": "absoluteUrl" }, { "op": "unique" }] },
+  "variants":    { "each": "variant_rows", "fields": { "size": { "from": ".", "transform": [{ "op": "regex", "pattern": "<td class=\"size\">([^<]+)" }] } } },
+  "seller.name": { "from": "seller" }
+}
+```
+
+- The key is an output field, dotted for nested objects.
+- `from` is an id or a path into one; several sources give the chain a list.
+- `each` builds an array of objects from a list; inside, `from` is relative to each item (`.` is the item).
+- A scalar transform applied to a list runs on every item. A missing value passes through every transform
+  except `default`, `template` and `hook`, then meets the field's missing policy.
+- The ops: `trim`, `lowercase`, `uppercase`, `replace`, `regex`, `split`, `join`, `first`, `last`, `nth`,
+  `slice`, `concat`, `coalesce`, `default`, `number`, `integer`, `boolean`, `currency`, `date`,
+  `absoluteUrl`, `urlEncode`, `flatten`, `unique`, `sum`, `count`, `template`, `jsonpath`, `lookup`, `group`, `hook`.
+
+## 4. Sessions and logins
+
+```json
+"session": {
+  "bootstrap": {
+    "keep": ["cookies"],
+    "steps": [
+      { "type": "goto", "url": "https://shop.example/login" },
+      { "type": "fill", "selector": "#user", "value": "{{vars.user}}" },
+      { "type": "fill", "selector": "#pass", "value": "{{vars.pass}}" },
+      { "type": "click", "selector": "button[type=submit]" },
+      { "type": "wait", "selector": "#logged-in" }
+    ],
+    "saveTo": "state/shop.json"
+  }
+}
+```
+
+The bootstrap always runs in a browser. What it keeps is handed to the recipe: a `web` recipe starts its
+page from it, an `api` recipe sends its cookies with every request. `storageStatePath` reuses a state saved
+by `saveTo` on a later run. `headers`, `cookies`, `userAgent` and `viewport` apply to both modes.
+
+## 5. Running
+
+```ts
+import { createCrawler, jsonLinesSink, loadRecipeSet } from '@opencraw/core'
+
+const recipes = await loadRecipeSet({ output: 'recipes/product.output.json', inputs: ['recipes/'] })
+const crawler = createCrawler({
+  hooks:   { positive: input => Number(input) > 0 },
+  sink:    jsonLinesSink('out/products.jsonl'),
+  onEvent: event => console.log(event.type, event),
+})
+const report = await crawler.run(recipes)
+await crawler.close()
+```
+
+`loadRecipeSet` validates every file and checks that each input can feed the output (mapped fields exist,
+ids are bound, required fields are covered, web steps stay in web recipes). Errors list every problem with
+its JSON path. The report says, per recipe, how many records were emitted, rejected or dropped as
+duplicates, how many pages were visited, and the error that stopped it, if any.