@enricai/barnacle 1.12.17 → 1.12.19

package/README.md

@@ -1,32 +1,16 @@
 # Barnacle
 
+## How it works
+
 Point Barnacle at a site, describe the user flow in plain English, and run three
 recon commands. Barnacle drives a real browser through the flow, captures every
 API call, replays them with plain HTTP to prove which ones work without a browser,
-probes rate-limit ceilings, and then generates a complete plugin — Zod schemas
+probes rate-limit ceilings, and generates a complete plugin — Zod schemas
 inferred from captured JSON, load-bearing headers, rate-limit ceiling, hot-path
 HTTP client, and Stagehand browser fallback. Register the plugin in one line;
 Barnacle handles sessions, retries, fallback routing, audit persistence, and
 response envelope wrapping.
 
-## How it works
-
-### The mental model
-
-Stagehand drives a real browser through your described user flow. Its only job is
-to trigger the site's network traffic — not to extract DOM data. While it clicks,
-a response listener wiretaps every API call to disk. Once that recon run is done,
-a separate script replays those captures via plain `fetch()` — no browser, no AI —
-to prove the endpoints work standalone. The surviving queries and headers become
-committed constants. In production, the runtime hits those endpoints directly:
-fast, cheap, deterministic. The browser only re-engages if the direct path breaks.
-
-A nightly smoke test tells you the moment a contract drifts. When it fires, you
-re-run the same recon command you ran the first time and diff the captures.
-Human involvement is one recon run up front and a small PR when things change.
-
-### The pipeline at a glance
-
 | Phase | What runs | What you get |
 |-------|-----------|--------------|
 | **1 — Browser recon** | `pnpm run recon:browser` | Every API call the site makes, captured to `<run-dir>/graphql/*.json` |
@@ -34,945 +18,9 @@ Human involvement is one recon run up front and a small PR when things change.
 | **4 — Plugin generation** | `pnpm run recon:generate` | A complete plugin: Zod schemas, headers, Bottleneck config, hot-path client, Stagehand fallback |
 | **5+ — Runtime** | `pnpm start` | Direct HTTP hot path, automatic browser fallback, nightly smoke test, drift detection |
 
-### Why this approach
-
-You will get asked why not just use the browser for every request, or scrape HTML,
-or reverse-engineer endpoints by hand. Here is the honest comparison:
-
-| Approach | Cost/req | Latency | Fragile to UI | Fragile to API | Effort |
-|----------|----------|---------|---------------|----------------|--------|
-| Browser on every request | High | 5–15 s | Medium | Low | Low |
-| HTML screen scraper | Low | Low | **High** | Low | Medium |
-| Manual DevTools recon | Low | Low | Low | High (human redo) | High (ongoing) |
-| HAR replay (mitmproxy) | Low | Low | Medium | **High** | Medium |
-| **Recon → codify → direct HTTP + fallback (this)** | **Low** | **Low** | **Low** (re-runnable) | **Low** (fallback covers) | Medium, front-loaded |
-
-The browser-on-every-call approach uses Steel minutes and Anthropic tokens on
-every production call — orders of magnitude more expensive at scale. HTML
-scrapers break on every UI redesign, and the API response usually carries richer
-data than what the UI renders anyway. Manual DevTools recon is exactly what this
-pipeline automates, but committed and re-runnable. Front-loaded recon work buys
-an integration as cheap as direct HTTP, as robust as a browser fallback, and
-maintainable in a way none of the hand-rolled options are.
-
----
-
-## Adding a New Site — The Recon Playbook
-
-Every new site follows the same pipeline (Phases 0–6). The only human-authored
-input is the flow definition you write once in Phase 0. After that, the scripts
-run unattended — recon captures, HTTP replay proves endpoints, the generator
-writes the plugin. When the site changes months later, you re-run the same
-command and diff the captures. Human time is front-loaded to one recon run and
-a small PR.
-
-### Phase 0 — Define the user flow
-
-Commit the flow steps to a file first — this makes recon re-runnable in one command without retyping it. When the site changes and you need to re-run recon months later, you `git pull` and run the same command you ran the first time:
-
-```bash
-# src/sites/my-site/recon-flow.json
-["click the Electronics category filter", "open the first product result"]
-```
-
-### Phase 1 — Run the browser recon harness
-
-```bash
-# Preferred: load flow from committed file
-pnpm run recon:browser -- \
-  --url https://example.com \
-  --flow-file src/sites/my-site/recon-flow.json
-
-# Or inline (ephemeral — must be re-typed each recon run):
-pnpm run recon:browser -- \
-  --url https://example.com \
-  --flow '["click the Electronics category filter", "open the first product result"]'
-
-# For sites whose API paths don't match /graph, /api/, /graphql, /v1/, or *.json:
-pnpm run recon:browser -- \
-  --url https://example.com \
-  --flow-file src/sites/my-site/recon-flow.json \
-  --capture-all
-
-# Capture page-load XHRs only (no interaction — useful for pure GET-style SPAs):
-pnpm run recon:browser -- --url https://example.com
-```
-
-Drives a real Stagehand + Steel browser through your flow. Captures are wired via a single CDP session-level listener (`page.getSessionForFrame().on(...)`) — Stagehand V3 enables the Network domain internally, so attaching our `Network.requestWillBeSent` / `responseReceived` / `loadingFinished` listeners on the main session catches every response, including the early ones that fire before any page-level handler could be wired.
-
-Captures every network call matching `/graph`, `/api/`, `/graphql`, `/v1/`, or `*.json` to `<run-dir>/graphql/<NNN>-<phase>-<operationName>.json` — one file per call, diffable and greppable. Use `--capture-all` for sites with non-standard API paths; it captures every response, producing more noise but missing nothing. Omitting both `--flow` and `--flow-file` runs zero interaction steps and captures only the network activity that fires during page navigation — useful for pure GET-style SPAs that fetch everything they need on load.
-
-Each step runs through a self-healing cascade (`act` → `observe + act` → `observe + act` with `ignoreSelectors` → LLM rephrase) verified by network-counter delta or URL change. The script's `main()` attempts up to two global flow replans before giving up; terminal failures dump a diagnostic bundle to `<run-dir>/step-failures/`. See [docs/playbook.md#1c--self-healing-cascade](./docs/playbook.md#1c--self-healing-cascade) for the full design.
-
-Total runtime: 20–40 minutes for a typical flow (longer if healing or replans fire), fully unattended.
-
-Every artifact — captures, cookie-jar snapshots, step-failure dumps, DOM dumps — is rooted under one run-scoped directory resolved once at startup: `<run-dir>` defaults to `/tmp/recon/<runId>`, where `<runId>` is a timestamp + random suffix generated per process (e.g. `20260718-120326-a1b2`). Set `RECON_RUN_ID` to pin a deterministic runId (e.g. for tests or replaying a known run) and `RECON_OUT_DIR` to override the base directory runs are rooted under. This keeps concurrent or repeated runs from intermixing files — the startup log line prints both the resolved `runId` and `out=<run-dir>`.
-
-#### Cookie-jar snapshots
-
-Alongside the network captures, every run snapshots the browser's complete cookie jar (via CDP `Network.getAllCookies`, which returns the whole-browser jar regardless of the current page's URL — unlike `document.cookie` or `Page.getCookies`, it also sees HttpOnly cookies) at each phase boundary: the initial goto, immediately before each flow step (`pre-step`), immediately after each flow step completes (`post-step`), and once more at run completion (`run-complete`).
-
-Snapshots land in `<run-dir>/cookies/<NNN>-<label>-<phase>.json` — one file per boundary, using the same zero-padded chronological index convention as the network captures. `<label>` is the boundary kind (`goto`, `pre-step`, `post-step`, `run-complete`); `<phase>` is the current step's slugified instruction (e.g. `click-the-apply-button`), or `home` before the first step starts.
-
-Each file is a JSON object:
-
-```json
-{
-  "label": "post-step",
-  "phase": "click-the-apply-button",
-  "stepIndex": 2,
-  "timestamp": "2026-07-18T12:34:56.789Z",
-  "cookies": [
-    {
-      "name": "_acme_attr",
-      "value": "abc123",
-      "domain": ".acme.example",
-      "path": "/",
-      "expires": 1234567890,
-      "size": 20,
-      "httpOnly": true,
-      "secure": true,
-      "session": false,
-      "sameSite": "Lax"
-    }
-  ]
-}
-```
-
-Field reference (matches CDP's `Network.Cookie` type verbatim — no remapping between capture and disk):
-
-| Field | Meaning |
-| --- | --- |
-| `name` / `value` | The cookie's name and value. |
-| `domain` | Scope, e.g. `.acme.example` (all subdomains) vs. `apply.acme.example` (exact host) — the detail needed to tell a click-domain cookie from an apply-domain cookie. |
-| `path` | Cookie path scope. |
-| `expires` | Raw CDP epoch-seconds number; `-1` means a session cookie (also reflected in `session: true`). Not reformatted — read it as CDP reports it. |
-| `size` | Cookie size in bytes, as reported by CDP. |
-| `httpOnly` / `secure` | Standard cookie flags. |
-| `session` | `true` for a session cookie (no persistent expiry). |
-| `sameSite` | `"Strict" \| "Lax" \| "None" \| null` — `null` when the cookie doesn't set the attribute. |
-
-If the CDP call fails, the file still writes but with an empty `cookies` array and an `error` string field carrying the failure message — cookie telemetry is best-effort and never aborts the run.
-
-**Diffing what a phase established:** to isolate what a specific traversal (e.g. a tracking-click redirect) minted, diff its `post-step` snapshot against the `pre-step` snapshot for the *next* step — cookies present in the later file but absent from the earlier one were established during that step:
-
-```bash
-diff <(jq -S .cookies <run-dir>/cookies/004-post-step-click-the-apply-button.json) \
-     <(jq -S .cookies <run-dir>/cookies/005-pre-step-fill-in-your-name.json)
-```
-
-**Cookies actually sent per request:** the jar snapshot shows what's *available*, not what's *sent*. Each network capture in `<run-dir>/graphql/` separately carries the outgoing `Cookie` header in its `requestHeaders` (recovered via CDP's `Network.requestWillBeSentExtraInfo`, since `requestWillBeSent` omits it by design) — cross-reference that capture's `requestHeaders.Cookie` against a jar snapshot to see which of the available cookies a given request, e.g. the application submit, actually sent.
-
-**Caveat on `Set-Cookie`:** response captures fold `responseReceivedExtraInfo` headers (which is where `Set-Cookie` actually appears — `responseReceived` omits it) into `responseHeaders` as a flat `Record<string, string>`. CDP does not guarantee multiple `Set-Cookie` values on one response stay distinguishable once folded into that shape — if a single response mints more than one cookie, treat the jar snapshot (not the response capture's `Set-Cookie` header) as the source of truth for what actually landed.
-
-### Phase 2–3 — Replay and probe
-
-```bash
-pnpm run recon:http
-```
-
-Replays every capture via plain `fetch()` — no browser, no AI — to prove endpoints work standalone. Every replay returning 200 proves the browser is unnecessary for production. Also runs GraphQL introspection, auxiliary fixture detection (static JSON to commit as fixtures), and a rate-limit probe at 1→3→5 rps (run last — if it triggers a ban, all captures are already saved). Results land under the run-scoped root resolved by `resolveReconRunDir()` — `/tmp/recon/<runId>/replays/` by default, rooted elsewhere via `--out-dir <path>` or `RECON_OUT_DIR`.
-
-See [docs/playbook.md](./docs/playbook.md#interpreting-replay-failures) for the full troubleshooting decision matrix when replays fail.
-
-### Phase 4 — Generate the plugin
-
-```bash
-pnpm run recon:generate -- --site-id my-site
-```
-
-Reads every artifact from Phases 1–3 — `<run-dir>/graphql/*.json` (captures), `<run-dir>/replays/*.json` (replay results), `<run-dir>/replays/rate-limit.json` (probe findings), `<run-dir>/aux/*.json` (static fixtures), and `src/sites/my-site/recon-flow.json` — and writes a complete plugin to `src/sites/my-site/`. Pass `--run-dir <path>` to read a specific run's artifacts instead of the most recently modified run root under `/tmp/recon` (or `RECON_OUT_DIR`, if set):
-
-- `contract.ts` — Zod schemas inferred from captured JSON, load-bearing headers, Bottleneck ceiling, and `executeHttp` / `execute` implementations
-- `flows/browser-flow.ts` — Stagehand fallback wired to your `recon-flow.json` steps
-- `index.ts` — barrel export
-- `fixtures/` — any static JSON found by the auxiliary probe, already copied in
-
-Then review the generated files: trim UI-only fields from the GraphQL query, narrow any `z.unknown()` entries in the schema you care about, and verify the headers. If you need to regenerate after making changes to the recon flow, pass `--force`.
-
-#### Telling the generator which fields carry your caller's data
-
-Barnacle cannot know what your site's forms mean. `"Select the neighborhood from the Country dropdown"` and `"…select the test candidate's country"` are the same sentence shape; only you know that the first is a search facet and the second is your caller's address. So the vocabulary is yours to supply, with `--vocabulary`:
-
-```ts
-// src/recon/my-vocabulary.ts
-import type { ReconVocabulary } from "@enricai/barnacle/recon/vocabulary";
-
-export const vocabulary: ReconVocabulary = {
-  // Only a step naming the SUBJECT may splice off a bare dropdown — a dropdown
-  // step carries no quoted constant, so the label alone cannot tell your
-  // caller's data from a filter that happens to say "Country".
-  subject: /\b(the\s+)?(test\s+)?customer'?s\b/i,
-  // Labels that must stay literal even when a table row matches.
-  exclusions: [/\bbilling contact\b/i],
-  // Ordered label → payload field. First match wins, so specific precedes broad.
-  table: [
-    [/\bfirst name\b/i, "FirstName"],
-    [/\bcity\b/i, "City"],
-  ],
-};
-```
-
-```bash
-pnpm run recon:generate -- --site-id my-site --vocabulary ./src/recon/my-vocabulary.ts
-```
-
-- The specifier follows the same rule as `BARNACLE_PLUGINS`: a leading `.` or `/` is a filesystem path; anything else resolves from your `node_modules`. The module may export `vocabulary` or a default.
-- **`--vocabulary none`** for a site that splices no caller data at all (read-only inventory, search, pricing). Explicit, so it can't happen by accident.
-- Every regex must be free of the `g`/`y` flags and every field name must be a valid JS identifier — the loader rejects both. A stateful regex matches only every other step; a non-identifier emits `payload.<name>` as a syntax error.
-- Run the generator with a `.ts` vocabulary under `tsx`, or point it at compiled `.js` — plain `node` cannot import TypeScript before v22.18.
-
-> Omitting `--vocabulary` disables splicing entirely — every step's recon constant is emitted literally, with no caller-data substitution.
-
-#### Telling the generator your ATS's form-schema wire keys
-
-Where `--vocabulary` matches instruction *prose*, `--form-schema` names the JSON *keys* the generator reads out of an ATS's form-definition responses when recovering field ids, option ids, and submitted values. The engine ships no vendor's format; a site whose ATS exposes a form definition declares its keys with `--form-schema`:
-
-```ts
-// src/recon/my-form-schema.ts
-import type { ReconFormSchema } from "@enricai/barnacle/recon/form-schema";
-
-export const formSchema: ReconFormSchema = {
-  fieldIdKey: "fieldId",                    // UUID-valued field identity
-  fieldNameKeys: ["code", "label"],         // code, then human label — code preferred
-  fieldOptionsKey: "options",
-  optionIdKey: "optionId",                  // option id, inside the options array
-  optionValueKey: "optionLabel",            // option label, inside the options array
-  responseValueKey: "submittedValue",       // submitted free value
-  responseOptionIdKey: "submittedOptionId", // submitted option reference
-};
-```
-
-```bash
-pnpm run recon:generate -- --site-id my-site --form-schema ./src/recon/my-form-schema.ts
-```
-
-- The specifier follows the same rule as `--vocabulary`: a leading `.` or `/` is a filesystem path; anything else resolves from your `node_modules`. The module may export `formSchema` or a default.
-- **`--form-schema none`** for a site with no ATS form definition (a search API, a real-estate listing site) — same as omitting it. "none" is the explicit form.
-- Wire keys anchor `"key":"uuid"` markers, so they may be any non-empty string without a quote or backslash — the JS-identifier rule that governs vocabulary field names does **not** apply here.
-- `fieldNameKeys` models two roles: the first key is a machine code (PascalCased directly), the second is a human label (run through the section-heading heuristic). Supply one key for a label-only ATS, or two for one that exposes both. Additional keys are unused.
-- Omit `--form-schema` (or pass `none`) and ATS form-key recovery does not run — the engine hardcodes no vendor's wire format. A site whose ATS exposes a form definition must supply one to recover its option fields. See issue #57.
-
-#### Filtering analytics noise
-
-| Env var | Default | Description |
-| --- | --- | --- |
-| `RECON_TELEMETRY_URL_PATTERNS` | *(empty)* | Comma-separated extra URL fragments to treat as analytics noise, on top of the built-in list. Put your site's trackers here. |
-
-Screening-question answers with coded options (dropdowns) are recovered generically: when a flow step selects an answer by its label and the site's response carries the option list (JSON-Schema `enum`/`enumNames`, or `{label,value}` option objects), the generator maps the label to the wire code and emits a `z.enum` payload field, so the caller supplies the human-readable answer.
-
-Optionally generate the human-readable findings doc alongside:
-
-```bash
-pnpm run recon:summarize -- --site-id my-site
-```
-
-Writes `docs/my-site-recon.md` with: endpoints found, replay status, rate-limit ceiling, header frequency table, and hazards (Akamai, Cloudflare). Without `--site-id`, the default output path is `docs/target-recon.md`. Accepts `--run-dir <path>` the same way `recon:generate` does.
-
-### Phase 5 — Register the plugin
-
-See **[Plugin Authoring Guide → Register the plugin](#register-the-plugin)** below for plugin registration options.
-
-### Phase 6 — Wire up drift detection
-
-See **[Plugin Authoring Guide → Wire up the nightly smoke test](#wire-up-the-nightly-smoke-test)** below for the CI step that runs the smoke test nightly.
-
-See [docs/playbook.md](./docs/playbook.md#phase-6--drift-detection) for the full detection ladder and maintenance loop.
-
-### The whole loop, in one picture
-
-![Barnacle end-to-end workflow: Setup (recon) feeds a dashed Deploys edge into Runtime (dispatch + cache + hot path), Heal catches errors and runs nightly drift detection, and a solid orange arrow sweeps back from smoke-test.ts into Phase 1 to close the self-healing loop.](docs/images/workflow.svg)
-
-The dashed `deploys` edge is the human-in-the-loop step (the contract PR merges and ships to Runtime). The solid orange edge from `smoke-test.ts` back into Phase 1 is the self-healing loop: when the contract drifts, recon reruns unattended (~20–40 min) and the next PR is a diff of captures, not a hand-rewrite. See [docs/architecture.md](./docs/architecture.md) for the design rationale behind each lane.
-
-## Plugin Authoring Guide
-
-A site plugin is a single TypeScript module that satisfies `SitePlugin<TInput, TOutput>`
-from `src/site-plugin.ts`. Core registers built-in plugins via `BUILTIN_SITE_PLUGINS` in
-`src/plugins/discover.ts`; out-of-tree plugins are loaded at startup via `BARNACLE_PLUGINS`.
-
-### The SitePlugin interface
-
-```ts
-interface SitePlugin<TPayload, TResult> {
-  meta: SitePluginMeta;
-  // Optional direct-HTTP hot path — no browser, no LLM tokens, millisecond latency.
-  // Core tries this first; falls back to execute() on HttpSchemaError / HttpBotChallengeError / HttpServerError.
-  executeHttp?: (
-    payload: TPayload,
-    context: SitePluginContext
-  ) => Promise<SitePluginResult<TResult>>;
-  // Browser fallback — Stagehand + Steel session, acquired from the pool by core.
-  execute(
-    payload: TPayload,
-    session: BrowserSession,
-    context: SitePluginContext
-  ): Promise<SitePluginResult<TResult>>;
-  // Async work is supported. Note: NOT called on CaptchaError or EmptyResultsError —
-  // p-retry skips onFailedAttempt for AbortError, so those abort paths bypass this hook.
-  onRetry?: (error: ScraperError, attempt: number) => void | Promise<void>;
-}
-```
-
-### SitePluginMeta — required fields
-
-| Field | Type | Purpose |
-|---|---|---|
-| `siteId` | `string` | Stable key used for routing (`/v1/<siteId>/run`) and audit rows |
-| `displayName?` | `string` | Human-readable label for logs and Swagger docs. `recon:generate` does not derive this — plugin authors set it explicitly |
-| `bodySchema` | `ZodTypeAny` | Request body schema — core validates before calling `execute()` |
-| `responseSchema` | `ZodTypeAny` | Success response schema — drives Swagger output shape |
-| `routeOverride?` | `string` | Override the full route path (legacy compatibility only) |
-| `defaultBaseUrl?` | `string` | Fallback base URL when `config.scraper.siteBaseUrls[siteId]` is absent |
-| `taskTimeoutMs?` | `number` | Override the pool's 60-minute per-task hang ceiling for this plugin only — set when the site's normal latency is well below the default and a faster failure is preferable |
-| `maxAttempts?` | `number` | Override the retry policy's default of 3 attempts (including the first try). Without this, the per-run ceiling is `3 × taskTimeoutMs`; set to `1` so `taskTimeoutMs` is the real per-run cap |
-| `apiVersion?` | `string` | Semver range targeting a plugin API version (e.g. `"^1.0.0"`); core disables the plugin on a major-version mismatch. Absent means "accept any version." |
-| `extraRoutes?` | `readonly SitePluginExtraRoute[]` | Extra non-run routes (OTP trigger, resume, etc.) that core registers as authenticated Fastify routes at startup. See `SitePluginExtraRoute` in `src/site-plugin.ts`. |
-| `onShutdown?` | `() => Promise<void>` | Optional cleanup for background work the plugin launched fire-and-forget, awaited during graceful shutdown so in-flight work is not abandoned and sessions are not leaked. Parallels the engine's own drain functions. Bounded by a per-plugin timeout, so a hanging drain cannot stall shutdown. Module plugins only — config-only `*.plugin.json` manifests are pure JSON and cannot declare a function. |
-
-### Full plugin skeleton (hot path + browser fallback)
-
-`pnpm run recon:generate` produces this structure automatically. Use `createRateLimitedJsonClient()` for REST endpoints that send Chromium client-hint headers (the common case) and `createGraphqlClient()` for GraphQL endpoints — `recon:generate` selects the right one based on what it captured. The skeleton below illustrates the REST hot-path pattern; for GraphQL sites, `recon-generate` uses `createGraphqlClient` instead. A GraphQL target with a single captured query inlines it as a constant; a read-only GraphQL flow with several 2xx `query` candidates and no `submitStep` instead ranks them by response size, `payloadField` correlation, capture phase, and recurrence to pick the primary data operation; a GraphQL target whose captures form a multi-operation mutation sequence (a submission flow) gets the same state-threaded, multi-step `executeHttp` REST submission flows get.
-
-```ts
-// src/sites/my-site/contract.ts
-import { z } from "zod/v4";
-import { createRateLimitedJsonClient } from "@/scraper/rate-limited-json-client";
-import type { BrowserSession } from "@/scraper/session";
-import type { SitePlugin, SitePluginContext, SitePluginResult } from "@/site-plugin";
-import { runMySiteBrowserFlow } from "@/sites/my-site/flows/browser-flow";
-
-// Generated: Zod schemas inferred from captured JSON — tighten z.unknown() fields as needed.
-export const MySiteResponseSchema = z.object({ data: z.object({ items: z.array(z.object({ id: z.string() })) }) });
-const MySitePayloadSchema = z.object({ query: z.string().min(1) });
-
-type MySitePayload = z.infer<typeof MySitePayloadSchema>;
-type MySiteResponse = z.infer<typeof MySiteResponseSchema>;
-
-// Generated: rate-limit ceiling (5 rps) + Chromium hints + site-specific headers from recon.
-// Use createHttpClient() directly only when you need manual Bottleneck or header control.
-const httpClient = createRateLimitedJsonClient({
-  minTimeMs: 200,
-  userAgent: "Mozilla/5.0 ...",
-  secChUa: '"Chromium";v="..."',
-  platform: "Linux",
-  extraHeaders: {
-    "Content-Type": "application/json",
-    Accept: "application/json, */*",
-  },
-  schema: MySiteResponseSchema,
-});
-
-export const mySitePlugin: SitePlugin<MySitePayload, MySiteResponse> = {
-  meta: {
-    siteId: "my-site",
-    displayName: "My Site",
-    bodySchema: MySitePayloadSchema,
-    responseSchema: MySiteResponseSchema,
-    defaultBaseUrl: "https://my-site.com",
-  },
-  // Hot path: direct HTTP — no browser, no LLM tokens.
-  async executeHttp(payload: MySitePayload, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
-    const data = await httpClient(`${context.baseUrl}/api/search`, {
-      method: "POST",
-      body: JSON.stringify({ query: payload.query }),
-    });
-    return { data };
-  },
-  // Browser fallback: Stagehand + Steel — invoked automatically when hot path fails.
-  async execute(payload: MySitePayload, session: BrowserSession, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
-    const raw = await runMySiteBrowserFlow(session.stagehand, context.baseUrl, payload.query);
-    return { data: raw };
-  },
-};
-```
-
-### The auditPayload hook
-
-`SitePluginResult` accepts an optional `auditPayload` field alongside `data`:
-
-```ts
-return {
-  data: responseData,
-  auditPayload: { query: payload.query, resultCount: responseData.items.length },
-};
-```
-
-When `auditPayload` is present, core writes it — not `data` — to the submission-envelope telemetry record. Use this to strip PII or large blobs from the audit trail while keeping the full response in the API reply. When absent, `data` is written as-is.
-
-### Reconciliation join keys (`extractJoinKeys`)
-
-Core has no opinion on what a reconciliation join key is named or how it's
-shaped — that's site-specific vocabulary (an attribution vendor's click ID,
-a job-reference composition rule, whatever the site needs). A plugin that
-needs its submission and beacon-fire telemetry to be joinable back to its own
-attribution provider declares an optional `extractJoinKeys` hook on its
-`SitePlugin`:
-
-```ts
-export const myPlugin: SitePlugin<MyPayload, MyResponse> = {
-  extractJoinKeys: (payload) =>
-    payload.someVendorClickId ? { vendorClickId: payload.someVendorClickId } : null,
-  // ...
-};
-```
-
-`dispatch()` (`src/plugins/loader.ts`) calls this once per submission,
-resolving `extractJoinKeys(payload)` from the inbound payload alone — core
-never inspects the result's contents. A plugin with no reconciliation needs
-simply omits `extractJoinKeys`.
-
-#### Mid-run attach point (`context.telemetry.addJoinKeys`)
-
-`extractJoinKeys` only ever sees the payload a plugin received up front, so
-it has no way to attach a field the plugin only discovers *during* the run
-— a token minted mid-flow, a value read from the page after navigation, a
-value observed on a response. For that, call the mid-run attach point,
-`context.telemetry.addJoinKeys()`, from anywhere inside `execute()` or
-`executeHttp()`:
-
-```ts
-async execute(payload: MyPayload, session, context: SitePluginContext) {
-  const mintedToken = await readTokenFromPage(session);
-  context.telemetry.addJoinKeys({ mintedToken });
-  // ...
-},
-```
-
-`context.telemetry` is a per-dispatch `RunTelemetry` accumulator
-(`src/lib/telemetry/run-telemetry.ts`), constructed fresh for every
-dispatch by `buildPluginContext` (`src/plugins/loader.ts`) alongside
-`recordBeaconOutcome` below. Successive `addJoinKeys()` calls within the
-same run merge, later calls winning on key collision. Once the plugin call
-resolves — on both the success and error paths — `dispatch()` snapshots the
-accumulator and merges it over the earlier `extractJoinKeys(payload)`
-result, run-discovered keys winning on collision, before stamping the
-combined bag onto the submission envelope's and beacon-fire record's
-`joinKeys` field. `joinKeys` stays `null` only when neither source ever
-produced anything.
-
-**A config-only `*.plugin.json` manifest can reach
-`context.telemetry.addJoinKeys()` only through the same `spec.httpModule`
-escape hatch documented below for `context.recordBeaconOutcome`** —
-`executeHttp(payload, context)` receives the same `SitePluginContext`, so an
-`httpModule` can call it exactly like `execute()` does above; the
-manifest's declarative browser flow cannot, since `runHealingFlow` is
-data-driven with no imperative call site for either seam to live in.
-
-**Declaring `extractJoinKeys` also opts the plugin out of core's automatic
-`TrackingUrl` fire.** If the site returns a post-submission click-tracking
-URL, declare it on the plugin's `bodySchema` by composing `JobTrackingSchema`
-(`src/lib/job-tracking.ts`) — `MySitePayloadSchema.extend(JobTrackingSchema.shape)`.
-When a plugin has no `extractJoinKeys`, `dispatch()` fires that `TrackingUrl`
-itself via `fireTrackingClick`, site-agnostically, after a successful submit.
-When a plugin *does* declare `extractJoinKeys`, core assumes the plugin fires
-its own post-submit tracking navigation (e.g. because the click and apply
-navs must share one browser session for a vendor's device-cookie
-attribution to work) and skips its own fire — firing both would open two
-independent sessions against the same URL.
-
-By default a self-managing plugin's beacon-fire telemetry is stuck at
-`beaconStatus: "skipped"`, since core has no visibility into a navigation the
-plugin drives itself. To report the real outcome, call
-`context.recordBeaconOutcome` — passed on `SitePluginContext` alongside
-`baseUrl`/`logger`/`requestId`, bound to this run — from `execute()`,
-`executeHttp()`, or an extra-route handler:
-
-```ts
-import type { SitePlugin, SitePluginContext } from "@enricai/barnacle/site-plugin";
-
-export const myPlugin: SitePlugin<MyPayload, MyResponse> = {
-  extractJoinKeys: (payload) =>
-    payload.someVendorClickId ? { vendorClickId: payload.someVendorClickId } : null,
-  async execute(payload, session, context: SitePluginContext) {
-    const t0 = Date.now();
-    const fired = await runMySiteBeaconNav(session, payload.TrackingUrl);
-    await context.recordBeaconOutcome({
-      beaconStatus: fired ? "fired" : "failed",
-      joinKeys: { vendorClickId: payload.someVendorClickId },
-      trackingUrl: payload.TrackingUrl,
-      durationMs: Date.now() - t0,
-    });
-    // ...
-  },
-};
-```
-
-Core binds the run's `requestId` and the plugin's own `siteId` for you, so
-`recordBeaconOutcome`'s input carries only `beaconStatus` (`"fired"` |
-`"failed"` — `"skipped"` stays an engine-owned outcome), the opaque `joinKeys`
-bag (same shape returned from `extractJoinKeys`), and optional `trackingUrl`/
-`durationMs`. It never throws — a telemetry-sink hiccup cannot fail the
-request. A `fired`/`failed` line recorded this way outranks the automatic
-`skipped` line for the same `requestId` when the two are folded together
-(see [Telemetry & LLM judging](docs/telemetry-and-judging.md)). A plugin that
-never calls it keeps today's unchanged `skipped` default. Import
-`BeaconOutcomeInput` from `@enricai/barnacle/site-plugin` if you want to type
-the input object explicitly — that's the published subpath an out-of-tree
-plugin resolves against its own `node_modules`; in-tree code under `src/`
-uses the `@/site-plugin` alias instead.
-
-**A config-only `*.plugin.json` manifest can reach `context.recordBeaconOutcome`
-only through the `spec.httpModule` escape hatch** — `executeHttp(payload,
-context)` receives the same `SitePluginContext` a module plugin's does, so an
-`httpModule` can call it exactly like `execute()` does above. The manifest's
-declarative browser flow cannot: `runHealingFlow` is data-driven, with no
-imperative call site for a call like this to live in. One consequence to know
-before adopting it: `buildConfigPlugin` never synthesizes `extractJoinKeys`,
-so a config-only plugin is never `managesOwnTracking` — when the response
-carries a `TrackingUrl`, core still fires it itself via `fireTrackingClick`,
-and a manifest-recorded `fired`/`failed` line for that `requestId` ranks
-equal to core's own line under `beaconRank()`, so the fold resolves by write
-order (last line wins) rather than the manifest's line automatically
-outranking core's. Only when no `TrackingUrl` is present — so core's own
-write is the `skipped` default — does the manifest's recorded line
-deterministically outrank it.
-
-### Static fixtures
-
-If Phase 3b (auxiliary fixture detection) found static JSON endpoints (markets, currencies, labels), `recon:generate` copies them to `src/sites/<id>/fixtures/`. Load them at module init via `loadFixture()` — zero per-request overhead, fails fast on deploy if the fixture is missing or stale:
-
-```ts
-import { z } from "zod/v4";
-import { loadFixture } from "@/scraper/fixtures";
-
-const MarketsSchema = z.array(z.object({ id: z.string(), name: z.string() }));
-
-// Loaded synchronously at module init. Throws at startup if file is missing
-// or shape drifted — surface fixture breakage on deploy, not on the first request.
-const markets = loadFixture("my-site", "markets.json", MarketsSchema);
-```
-
-See [docs/playbook.md — Phase 3b](./docs/playbook.md#3b--auxiliary-fixture-detection) for how fixtures are detected and when to use them.
-
-### Register the plugin
-
-**Out-of-tree (recommended for operator-owned plugins):** point `BARNACLE_PLUGINS` at the compiled plugin module — no core edits required:
-
-```bash
-BARNACLE_PLUGINS=./plugins/my-site/dist/index.js pnpm start
-```
-
-Barnacle validates the export at startup and registers `POST /v1/my-site/run` automatically. See the [Out-of-tree plugins](#out-of-tree-plugins) env var table for `BARNACLE_PLUGINS_STRICT` and `BARNACLE_PLUGINS_DIR`. A copyable, runnable template lives at [`examples/plugins/hello-site/`](./examples/plugins/hello-site/).
-
-**Config-only (no TypeScript, no compile step):** a browser-flow plugin can be a single JSON manifest. Point `BARNACLE_PLUGINS` at a `*.plugin.json` file, or drop manifests into a directory named by `BARNACLE_PLUGINS_CONFIG_DIR`:
-
-```bash
-BARNACLE_PLUGINS=./plugins/acme-jobs.plugin.json pnpm start
-# or, for directory-drop discovery of every *.plugin.json:
-BARNACLE_PLUGINS_CONFIG_DIR=./plugins pnpm start
-```
-
-The manifest wears the Kubernetes-style `apiVersion` / `kind` / `metadata` / `spec` envelope, declares its request/response/extract shapes as **JSON Schema**, and lists the browser flow as data (the same self-heal step format the recon toolchain authors). Core reads it at startup and registers `POST /v1/acme-jobs/run` — no per-site code. A site needing the direct-HTTP hot path can reference a compiled `executeHttp` module via `spec.httpModule`. A copyable manifest lives at [`examples/plugins/acme-jobs.plugin.json`](./examples/plugins/acme-jobs.plugin.json).
-
-The JSON Schema converter accepts a deliberately small subset — `object`, `string`, `number`, `integer`, `boolean`, `array` (with `items`), string `enum`, and `required` — and rejects anything else (e.g. `pattern`, `minLength`, `$ref`, `format` constraints) at load time. Flow steps interpolate request values with `{{ .request.FieldName }}`; a reference to a field the request schema does not declare fails loudly, while an optional declared field the caller omits splices as an empty string.
-
-**In-tree (bundled built-ins only):** push to `BUILTIN_SITE_PLUGINS` in `src/plugins/discover.ts`:
-
-```ts
-import { mySitePlugin } from "@/sites/my-site";
-import { BUILTIN_SITE_PLUGINS } from "@/plugins/discover";
-
-BUILTIN_SITE_PLUGINS.push(mySitePlugin as SitePlugin<unknown, unknown>);
-```
-
-Core registers `POST /v1/my-site/run` automatically at startup.
-
-### Wire up the nightly smoke test
-
-Add a step to `.github/workflows/smoke.yml`:
-
-```yaml
-- name: Run smoke test — my-site
-  if: steps.check-secrets.outputs.skip == 'false'
-  run: |
-    pnpm run smoke -- \
-      --site my-site \
-      --payload '{"query":"test"}' \
-      --host "$SMOKE_HOST" \
-      --fallback \
-      --response-schema src/sites/my-site/contract.ts
-  env:
-    API_KEY: ${{ secrets.SMOKE_API_KEY }}
-    SMOKE_HOST: ${{ secrets.SMOKE_HOST }}
-    NODE_ENV: production
-```
-
-`--response-schema` points to a module whose **default export is a Zod schema**. The smoke test validates the full response body against it — not just the envelope shape — so any schema drift on the data payload fails the pipeline immediately.
-
-`--fallback` additionally runs a second request via the Stagehand browser path. This catches Stagehand cache staleness: if the page DOM changed and the cached selector now points at the wrong element, the hot-path test passes but the fallback test fails — alerting you before the fallback is invoked in production.
-
-### Maintenance loop
-
-When the smoke test fails: re-run `pnpm run recon:browser` → diff `<run-dir>/graphql/*<operationName>*.json` against `src/sites/<id>/contract.ts` → update query / headers / Zod schema → ship. See [docs/playbook.md](./docs/playbook.md#phase-6--drift-detection) for the full maintenance loop and change severity table.
-
-## Runtime internals
-
-### Hot-path fallback triggers
-
-`dispatch()` (`src/plugins/loader.ts`) tries `executeHttp()` first. Which errors trigger the browser fallback and which don't:
-
-| Hot-path error | Status | Triggers browser fallback? | Reason |
-|---------------|--------|--------------------------|--------|
-| `HttpSchemaError` | Any | **Yes** | Response shape drifted; browser may still work |
-| `HttpBotChallengeError` | 401 / 403 | **Yes** | Residential proxy IP may get through |
-| `HttpServerError` | 5xx | **Yes** | Server-side outage; recovery strategy is the same |
-| `HttpRateLimitError` | 429 | **No** | A 429 means the configured rps ceiling is too high. Routing to the browser path would just hit the same ceiling and waste a Steel session. The right response is to lower the Bottleneck `minTime` in `contract.ts` and re-deploy. |
-| `HttpUrlLockedError` | 429 | **No** | A plugin's `classifyResponseBody` detected a terminal resource-lock sentinel — the URL is locked at the target's end. Neither a retry nor a browser session can succeed; the caller must back off and surface a "retry later" state. |
-| `UnknownScraperError` | Any | **No** | Transient network failure or unclassified non-JSON response. `createHttpClient` retries up to 2 times internally; if all attempts fail, the error propagates as `ScrapeFailureError`. |
-
-### Cache deduplication
-
-`getCachedResponse()` checks the LRU cache first. On a miss, `getOrCreateInFlight()` registers a promise in an `inFlight` map before awaiting it — meaning concurrent identical requests all await the same origin call rather than fanning out. First caller wins; all others coalesce onto its promise.
-
-Cache key: `<endpoint>:<sha256(canonical payload)[:32]>` — the endpoint is a literal prefix; the hash covers only the canonical payload. Object key order and primitive array element order are normalized so `{a:1,b:2}` and `{b:2,a:1}` hit the same entry. Default TTL: 15 minutes (`CACHE_TTL_MS`). Max entries: 1000 (`CACHE_MAX_ENTRIES`). Only successful responses are cached; errors propagate and never poison the cache.
-
-### Session pool and timeouts
-
-`runWithSession()` (`src/scraper/pool.ts`) queues tasks through a `p-queue` bounded by `SESSION_POOL_SIZE` (default: 3). Sessions are created on demand — not pre-warmed — so Steel billing stays proportional to actual traffic.
-
-**Per-task hang ceiling:** each queued task races against `TASK_TIMEOUT_MS` (`src/scraper/pool.ts`, **60 minutes** by default). A hung `execute()` — frozen CDP connection, infinite network wait — converts to `SessionTimeoutError`, which the retry policy handles by tearing down the broken session and creating a fresh one. The default is sized for long browser flows; shorten per-plugin via `SitePluginMeta.taskTimeoutMs`. This is a hang-recovery floor, not a p99 latency budget.
-
-**Retry policy:** `withScraperRetry` (`src/scraper/retry.ts`) uses p-retry with `factor: 2`, `minTimeout: 500ms`, `maxTimeout: 5000ms`, `randomize: true`, and default `maxAttempts: 3`. `EmptyResultsError`, `CaptchaError`, and `StepVerificationError` short-circuit retries (abort after the first attempt — a deterministic verification failure won't resolve by re-running the whole flow); `SessionTimeoutError` and `CdpTransportClosedError` both trigger a session restart before every retry attempt, not just the first.
-
-`src/scripts/recon-browser.ts`'s `main()` applies the same policy at the whole-flow level: it wraps a single-attempt `runFlowAttempt()` (session creation → step loop → replan write-back) in `withScraperRetry`, bounded by `config.scraper.maxTransportRetries` (default 3, `RECON_MAX_TRANSPORT_RETRIES`). Only a `CdpTransportClosedError` — a CDP transport teardown detected mid-flow while traffic is still active — retries on a brand-new session; every other failure mode (`isFlowTruncated`, `StepVerificationError`, etc.) still exits/rejects after exactly one attempt, exactly as before this retry loop was added.
-
-**Graceful shutdown:** `drainPool()` is called during graceful shutdown — `SIGTERM`/`SIGINT` triggers `app.close()`, which fires Fastify's `onClose` hook, which calls `drainPool()`. It pauses new intake, waits up to 20 seconds for in-flight tasks to close their Steel sessions, then resolves. Without this, process exit leaves live sessions billing until Steel's own timeout.
-
-### Viewport rotation
-
-`createBrowserSession()` (`src/scraper/session.ts`) picks a random desktop viewport per session from: `1280×720`, `1366×768`, `1440×900`, `1920×1080`. A fixed pixel size is an easy bot-detection fingerprint; rotating it makes sessions harder to cluster.
-
-### LLM routing: Anthropic vs. AWS Bedrock
-
-By default, Stagehand calls the Anthropic API directly (`ANTHROPIC_API_KEY`). Set `USE_BEDROCK=true` to route through AWS Bedrock instead:
-
-```bash
-USE_BEDROCK=true
-AWS_REGION=us-east-1
-AWS_ACCESS_KEY_ID=...
-AWS_SECRET_ACCESS_KEY=...
-BEDROCK_MODEL=us.anthropic.claude-sonnet-4-6[1m]   # default
-```
-
-The `[1m]` suffix selects the 1-million-token context variant on Bedrock. Both paths run Stagehand with `serverCache: true` (server-side action cache to skip LLM inference on replay) and `selfHeal: false` (recon-browser owns its own verify-and-retry cascade; see `src/scraper/session.ts` for the rationale).
-
-When using Anthropic directly (not Bedrock), the model is controlled by `STAGEHAND_MODEL` (default: `anthropic/claude-sonnet-4-6`).
-
-## Observability
-
-`GET /readyz` returns readiness status plus per-site drift-detection metrics exposed by `src/scraper/metrics.ts`:
-
-```json
-{
-  "status": "ready",
-  "checks": {
-    "database": { "ok": true },
-    "scraperCredentials": { "ok": true },
-    "scraperPool": { "ok": true, "detail": "depth=0" }
-  },
-  "stats": {
-    "scraperPool": { "size": 0, "pending": 0, "concurrency": 3 },
-    "cache": { "size": 12, "max": 1000, "inFlight": 0 },
-    "metrics": {
-      "my-site": {
-        "hotPathSuccess": 4821,
-        "fallbackActivations": 3,
-        "rateLimitRejections": 0,
-        "p95LatencyMs": 187
-      }
-    }
-  },
-  "telemetry": {
-    "currentRunFile": "/path/to/.barnacle/events/run-123.ndjson",
-    "currentRunFileSizeBytes": 4096,
-    "orphansRecovered": 0
-  },
-  "heal": {
-    "my-site": { "verdict": "SUCCESS", "bestPassRate": 0.95, "reportPath": "heal-out/my-site/healing-my-site.md" }
-  }
-}
-```
-
-**What rising `fallbackActivations` means:** the hot path is failing and the browser fallback is absorbing traffic. Cost and latency rise while error rate stays flat — users don't notice yet, but you will on your bill. This is your signal to re-run recon.
-
-**`p95LatencyMs`** is reservoir-sampled (Vitter's Algorithm R, capped at 1000 samples) over actual origin round-trips. Cache hits are excluded — they're memory reads and must not bias the origin latency signal.
-
-See [docs/playbook.md](./docs/playbook.md#6b--metrics-signals-the-detection-ladder) for the full detection ladder.
-
-### NDJSON telemetry files
-
-Barnacle writes three append-only NDJSON files alongside its metrics:
-
-| File | Default path | Purpose |
-|------|-------------|---------|
-| LLM call samples | `.barnacle/calls.ndjson` | One line per LLM/Stagehand call; feed to the `judge:llm` and `slm-self-heal` skills |
-| Run event stream | `.barnacle/events/<runId>.ndjson` | Per-run event stream written by the event-stream subsystem; path surfaced in `/readyz` `telemetry.currentRunFile` |
-| Submission reconciliation records | `.barnacle/submissions.ndjson` | One line per dispatch submit outcome or beacon-fire event; the durable, queryable join-key record — see [Submission record schema](#submission-record-schema) below and `GET /v1/submissions` under [Endpoints](#endpoints) |
-
-#### LLM call sample schema
-
-Every line in `.barnacle/calls.ndjson` is a JSON object with these fields (source: `src/api/schemas/telemetry.ts`):
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `callId` | `string` | UUID generated per call |
-| `callType` | `string` | Which LLM call site produced this sample — see table below |
-| `model` | `string` | Model identifier string passed to the SDK |
-| `systemPrompt` | `string \| null` | System-prompt text, or `null` when absent |
-| `userContent` | `string` | Full user-turn content |
-| `responseContent` | `string \| null` | Raw response text, or `null` on SDK error |
-| `parsedOk` | `boolean` | Whether the response was successfully parsed into the expected schema |
-| `inputTokens` | `number \| null` | Input token count from SDK usage metadata |
-| `outputTokens` | `number \| null` | Output token count from SDK usage metadata |
-| `latencyMs` | `number \| null` | Wall-clock latency of the SDK call in milliseconds |
-| `success` | `boolean` | Whether the call site considered the call successful end-to-end |
-| `ts` | `string` | ISO-8601 timestamp at write time |
-
-#### Call types
-
-`callType` is a stable string constant defined in `src/lib/telemetry/call-types.ts`:
-
-| `callType` | Source | When emitted |
-|------------|--------|--------------|
-| `recon-rephrase` | `src/scripts/recon-browser.ts` | Attempt-5 rephrase inside the recon-browser step-healing cascade — the ai-SDK model (Anthropic-direct or Bedrock-backed) is asked to reword the failing step |
-| `recon-replan` | `src/scripts/recon-browser.ts` | Global replan after a step terminally fails — Claude rewrites the remaining flow tail |
-| `recon-flow-patch` | `src/scripts/recon-heal.ts` | Patch proposal from the recon-flow-patch-generator during the `recon-heal` self-healing loop |
-| `llm-prompt-patch` | `src/scripts/llm-heal.ts` | Patch proposal from the llm-call-patch-generator during the `llm-heal` self-healing loop |
-
-#### Submission record schema
-
-Every `"submit"`-kind line in `.barnacle/submissions.ndjson` is a JSON object
-validated against `submissionEnvelopeSampleSchema`
-(`src/lib/telemetry/submission-capture.ts`, an alias of `submitRecordSchema`
-in `src/lib/telemetry/reconciliation-record.ts`):
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `kind` | `"submit"` | Discriminates this record from a `"beacon"` conversion-event record sharing the same sink; defaults to `"submit"` so lines written before this field existed still parse. |
-| `siteId` | `string` | Which plugin handled the request — the cohort dimension for reconciliation. |
-| `requestId` | `string` | Fastify-issued correlation ID; joins a later `"beacon"` record to this one by matching `requestId`. |
-| `joinKeys` | `Record<string, unknown> \| null` | Opaque, plugin-owned reconciliation join keys: the plugin's `extractJoinKeys` hook resolved once from the inbound payload, merged with anything the plugin attached mid-run via `context.telemetry.addJoinKeys()` (run-discovered keys win on collision — see [Reconciliation join keys](#reconciliation-join-keys-extractjoinkeys) above); `null` when neither source produced anything. |
-| `inboundPayload` | `unknown` | The request body the caller posted, unredacted. |
-| `status` | `"submitted" \| "error"` | Submit outcome. |
-| `auditPayload` | `unknown` | The plugin's `SitePluginResult.auditPayload`, or `data` when absent; `null` on errors. |
-| `errorMessage` | `string \| null` | Failure message on errors; `null` on success. |
-| `durationMs` | `number` | Total dispatch wall time in milliseconds. |
-| `ts` | `string` | ISO-8601 timestamp at write time. |
-| `session` | `{ id, provider, ip, ipCapturedAt } \| null` | Identity and outbound IP of the Browserbase session that served this run; `null` on the direct-HTTP hot path where no session is ever acquired. See [Submission-envelope sink](docs/telemetry-and-judging.md#submission-envelope-sink) for capture details. |
-
-A `"beacon"`-kind record shares the same sink to record a later, independent
-beacon-fire outcome for the same `requestId`. Core writes one itself — either
-a `fired`/`failed` line once `fireTrackingClick` resolves, or a `skipped` line
-when there is no `TrackingUrl` to fire or the plugin declared
-`extractJoinKeys` — but a plugin managing its own tracking nav can also emit
-one directly via `context.recordBeaconOutcome` (see
-[Reconciliation join keys](#reconciliation-join-keys-extractjoinkeys) above).
-See [Submission-envelope sink](docs/telemetry-and-judging.md#submission-envelope-sink)
-for the full schema and the `GET /v1/submissions` read path.
-
-#### Tailing call samples with jq
-
-```bash
-# Stream all LLM call samples as they arrive
-tail -f .barnacle/calls.ndjson | jq '.'
-
-# Filter to a specific call type
-tail -f .barnacle/calls.ndjson | jq 'select(.callType == "recon-rephrase")'
-
-# Show only failures
-tail -f .barnacle/calls.ndjson | jq 'select(.success == false) | {callId, callType, latencyMs}'
-
-# Token usage summary by call type
-jq -s 'group_by(.callType) | map({callType: .[0].callType, totalInputTokens: map(.inputTokens // 0) | add, totalOutputTokens: map(.outputTokens // 0) | add, n: length})' .barnacle/calls.ndjson
-
-# Tail the current run event stream (path from /readyz telemetry.currentRunFile)
-tail -f .barnacle/events/<runId>.ndjson | jq '.'
-```
-
-## Environment variables
-
-All variables are read once at process start. Required variables cause the
-process to exit on missing values; optional ones have safe defaults.
-
-### Application
-
-| Variable | Default | Required | Purpose |
-|----------|---------|----------|---------|
-| `APP_NAME` | `barnacle` | No | Application name used in logs |
-| `NODE_ENV` | `development` | No | `development` / `production` / `test` |
-| `PORT` | `3000` | No | HTTP listen port |
-| `HOST` | `0.0.0.0` | No | HTTP listen address |
-| `LOG_LEVEL` | `info` | No | Pino log level (`debug`, `info`, `warn`, `error`) |
-
-### Auth
-
-| Variable | Default | Required | Purpose |
-|----------|---------|----------|---------|
-| `API_KEYS_HASHED` | `""` | Yes (prod) | Comma-separated bcrypt hashes of plaintext bearer tokens. See [Generating an API key](#generating-an-api-key). |
-| `DEV_BYPASS_AUTH` | `false` | No | Skip auth entirely. Local dev only — **never set in production**. |
-
-### Browser automation (Steel + Stagehand)
-
-| Variable | Default | Required | Purpose |
-|----------|---------|----------|---------|
-| `STEEL_API_KEY` | — | **Yes** | Steel account API key. Required for all browser automation. |
-| `ANTHROPIC_API_KEY` | — | Yes (if not using Bedrock) | Anthropic API key for Stagehand's LLM calls. |
-| `STAGEHAND_MODEL` | `anthropic/claude-sonnet-4-6` | No | Stagehand model. Use the `anthropic/` prefix — Stagehand 2.x's model map is stale and the prefix routes through AI-SDK's fallback path. |
-| `SCRAPER_PROXY_TYPE` | `residential` | No | `residential` (paid Steel tiers) or `none` (free tier — Steel rejects `useProxy=true` on hobby plans). |
-| `SCRAPER_SOLVE_CAPTCHA` | `true` | No | Enable Steel's built-in CAPTCHA solver. Requires a paid plan; set `false` on the free tier. |
-| `SESSION_POOL_SIZE` | `3` | No | Maximum concurrent Steel browser sessions. |
-| `SCRAPER_MIN_ACTION_DELAY_MS` | `500` | No | Minimum delay between scraper actions (ms). Jitter applied on top. |
-| `SCRAPER_MAX_ACTION_DELAY_MS` | `1500` | No | Maximum delay between scraper actions (ms). |
-| `STAGEHAND_API_TIMEOUT_MS` | `120000` | No | Anthropic SDK request timeout (ms). Raise on slow network paths to `api.anthropic.com`. |
-| `STAGEHAND_CONNECT_TIMEOUT_MS` | `120000` | No | TCP connect timeout for all outbound fetch calls (ms). Raised from the undici default of 10 s to match `STAGEHAND_API_TIMEOUT_MS`. |
-| `STEEL_SESSION_TIMEOUT_MS` | `3600000` | No | Steel session wall-clock timeout (ms). Default is 1 hour; lower on plans that enforce shorter maximum session durations. |
-| `FRAME_READY_TIMEOUT_MS` | `20000` | No | How long `resolveFrameTarget` polls for a child iframe to attach before falling back to the main frame (ms). Raise further for cross-origin OOPIFs that attach slowly under advancedStealth + proxied CDP. |
-| `FRAME_DOCUMENT_READY_TIMEOUT_MS` | `5000` | No | How long `waitForChildFrameReady` polls a resolved child frame's `document.readyState` before proceeding anyway (ms). Independent of `FRAME_READY_TIMEOUT_MS` — this wait settles in well under a second once attached. |
-| `FRAME_EVALUATE_TIMEOUT_MS` | `30000` | No | Watchdog budget for a single frame-scoped evaluate/candidate-probe call (ms), so a call against a racy frame fails the attempt instead of hanging indefinitely. |
-| `FRAME_PRESENCE_PROBE_FLOOR_MS` | `3000` | No | Per-probe watchdog floor for `probeAttachedFrameTarget`'s single non-polling presence check (ms) — a real budget a genuine CDP round-trip can land within, instead of the `timeoutMs: 0` zero-budget pattern that always loses that race. |
-| `SCRAPER_CAPTURE_SESSION_IP` | `true` | No | Master switch for the outbound-IP echo navigation; `false` yields `session: null` / `sessionIp: null` everywhere without touching the rest of the submit/beacon record. |
-| `SCRAPER_SESSION_IP_ECHO_URL` | `https://api.ipify.org?format=json` | No | The IP-echo endpoint the session's own short-lived tab navigates to. Operators can point this at a self-hosted echo endpoint. |
-| `SCRAPER_SESSION_IP_TIMEOUT_MS` | `10000` | No | Watchdog bound on the echo navigation; a page that never resolves is cut off and yields `null` rather than blocking the submission. |
-
-### AWS Bedrock (alternative LLM provider)
-
-Set `USE_BEDROCK=true` to route Stagehand's LLM calls through AWS Bedrock
-instead of the Anthropic API. When enabled, `ANTHROPIC_API_KEY` is not needed.
-AWS credentials resolve in standard SDK order: explicit vars → ECS task role →
-EC2 instance profile → `~/.aws/credentials`.
-
-| Variable | Default | Required | Purpose |
-|----------|---------|----------|---------|
-| `USE_BEDROCK` | `false` | No | Master switch — routes LLM calls through Bedrock when `true`. |
-| `AWS_REGION` | `us-east-1` | No | AWS region for Bedrock calls. |
-| `AWS_ACCESS_KEY_ID` | — | No | Explicit AWS access key (leave blank for ambient IAM). |
-| `AWS_SECRET_ACCESS_KEY` | — | No | Explicit AWS secret key. |
-| `AWS_SESSION_TOKEN` | — | No | Required only for temporary STS credentials. |
-| `BEDROCK_MODEL` | `us.anthropic.claude-sonnet-4-6[1m]` | No | Bedrock cross-region inference profile ID. The `us.` prefix enables automatic cross-region routing; the `[1m]` suffix selects the 1M-token context variant. |
-
-### Cache
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `CACHE_TTL_MS` | `900000` (15 min) | LRU response cache TTL. Cached responses skip the target API entirely. |
-| `CACHE_MAX_ENTRIES` | `1000` | Maximum entries in the LRU cache. |
-
-### Rate limiting (inbound)
-
-These limit traffic *to* Barnacle's own API. See per-plugin Bottleneck config
-in each `contract.ts` for outbound rate limits to target sites.
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `RATE_LIMIT_MAX` | `120` | Max requests per window per API key (or IP for unauthenticated traffic). |
-| `RATE_LIMIT_WINDOW_MS` | `60000` (1 min) | Rate limit window duration. |
-| `TRUST_PROXY` | `true` | Trust `X-Forwarded-For` when behind a reverse proxy. Set `false` for bare-metal deploys to prevent spoofing. |
-
-### Readiness / observability
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `READINESS_QUEUE_THRESHOLD` | `20` | `/readyz` returns 503 when scraper queue depth exceeds this. Lets orchestrators shed load before the pool is saturated. |
-| `ENABLE_DOCS` | `false` | Serve Swagger UI at `/docs`. Disable in production. |
-
-### Datadog (opt-in)
-
-APM tracing and DogStatsD metrics are **opt-in**: `dd-trace` and `hot-shots` are
-optional peer dependencies, so a plain `npm i @enricai/barnacle` installs neither
-and Barnacle runs without them. Enable either half independently — install the
-package and flip its flag. If a flag is on but the package is missing, Barnacle
-warns and carries on with that feature disabled; it never fails to boot.
-
-```bash
-# APM tracing
-pnpm add dd-trace
-DD_TRACE_ENABLED=true node --import dd-trace/initialize dist/server.js
-
-# DogStatsD metrics
-pnpm add hot-shots
-DD_METRICS_ENABLED=true node dist/server.js
-```
-
-Tracing needs `--import dd-trace/initialize` for full auto-instrumentation:
-Datadog requires the tracer to load before any other module so it can patch
-http/net/dns. Metrics have no such constraint.
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `DD_TRACE_ENABLED` | `false` | Enable APM tracing. Requires the `dd-trace` peer dependency. |
-| `DD_METRICS_ENABLED` | `false` | Enable DogStatsD metrics. Requires the `hot-shots` peer dependency. Independent of `DD_TRACE_ENABLED`. |
-| `DD_AGENT_HOST` | `localhost` | Datadog agent hostname (the sidecar, in ECS Fargate). |
-| `DD_DOGSTATSD_PORT` | `8125` | DogStatsD UDP port on the agent host. |
-| `DD_SERVICE` | `barnacle` | Service name tagged on spans and metrics. |
-| `DD_ENV` | `NODE_ENV` | Deployment environment tag. |
-| `DD_VERSION` | `0.1.0` | Application version tag — git SHA or package version. |
-
-### Telemetry
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `TELEMETRY_ENABLED` | `true` | Master switch — set `false` to disable all NDJSON telemetry writes. |
-| `TELEMETRY_EVENTS_DIR` | `.barnacle/events` | Directory for per-run NDJSON event stream files (`<eventsDir>/<runId>.ndjson`). |
-| `CALLS_NDJSON_PATH` | `.barnacle/calls.ndjson` | Append-only NDJSON sink for LLM/Stagehand call samples. One line per call; feed to the judge and self-heal skills. |
-| `SUBMISSIONS_NDJSON_PATH` | `.barnacle/submissions.ndjson` | Append-only NDJSON sink for dispatch submission envelopes and beacon-fire outcomes. `kind:"submit"` lines (null/`"submit"`-defaulted on legacy lines) capture siteId, requestId, inbound payload, status, audit payload, and duration, plus the opaque `joinKeys` bag a plugin's `extractJoinKeys` hook resolved, merged with anything attached mid-run via `context.telemetry.addJoinKeys()` — the durable source-of-truth for "what did we submit for jobId X and did it succeed." `kind:"beacon"` lines record a later (or, for `beaconStatus: "skipped"`, immediate) independent beacon-fire outcome (`beaconStatus`: `fired`/`failed`/`skipped`, truncated `trackingUrl`) for the same `requestId`, so "submitted but the beacon did not fire" is measurable — the `skipped` line is always written by `dispatch()` itself, but a plugin managing its own tracking nav can call `context.recordBeaconOutcome` to append a real `fired`/`failed` line for the same `requestId`, which outranks `skipped` when the two are folded (see [Reconciliation join keys](#reconciliation-join-keys-extractjoinkeys)). A reader folds both kinds together by `requestId`, so a plugin can join runs to its own attribution provider's report without re-parsing `inboundPayload`. |
-| `TELEMETRY_MAX_FILE_SIZE_BYTES` | `104857600` (100 MB) | Rotate/drop the calls NDJSON once it exceeds this byte count. |
-| `TELEMETRY_MAX_RETENTION_MS` | `2592000000` (30 days) | Drop event-stream files older than this many milliseconds. |
-| `TELEMETRY_S3_BUCKET` | — | Optional — destination bucket for the buffered S3 telemetry replica. Sink is entirely inert (no client, no network calls) when unset. Credentials/region resolve the same way as Bedrock (`AWS_REGION`, standard SDK credential order). |
-| `TELEMETRY_S3_PREFIX` | `telemetry` | Key prefix for uploaded NDJSON objects (`<prefix>/<calls\|submissions>/<date>/...`). |
-| `TELEMETRY_S3_FLUSH_INTERVAL_MS` | `60000` | How often buffered lines are flushed to S3. |
-| `TELEMETRY_S3_MAX_BUFFER_LINES` | `500` | Threshold-flush trigger — flush early if either buffer exceeds this many lines, ahead of the next scheduled interval. |
-| `TELEMETRY_S3_READ_MAX_OBJECTS` | `200` | Upper bound on the number of S3 objects a single reconciliation read-path query is allowed to scan. |
-| `TELEMETRY_S3_READ_CONCURRENCY` | `8` | Max concurrent object fetches for a single reconciliation read-path query. |
-
-### LLM judging
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `JUDGE_MODEL` | `us.anthropic.claude-sonnet-4-6[1m]` | Anthropic model used by the judge script. Reuses Bedrock creds via the cross-region inference profile. |
-| `JUDGE_TEMPERATURE` | `0.2` | Sampling temperature for judge LLM calls. Keep low (≤ 0.3) for deterministic verdicts. |
-| `JUDGE_BATCH_SIZE` | `10` | Number of call samples sent to the judge in one LLM request. |
-| `JUDGE_TIMEOUT_MS` | `120000` (2 min) | Anthropic SDK request timeout for judge calls. |
-
-### Self-heal
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `SELFHEAL_MAX_ITERATIONS` | `5` | Maximum patch→replay→score iterations before BUDGET_EXHAUSTED. |
-| `SELFHEAL_N_REPLAYS` | `5` | Number of replay runs per iteration arm. |
-| `SELFHEAL_SUCCESS_THRESHOLD` | `0.9` | Minimum pass rate (0–1) to declare SUCCESS and stop iterating. |
-| `SELFHEAL_PLATEAU_WINDOW` | `3` | Consecutive iterations below `SELFHEAL_PLATEAU_DELTA` that triggers PLATEAUED. |
-| `SELFHEAL_PLATEAU_DELTA` | `0.03` | Minimum absolute pass-rate improvement per iteration to count as progress. |
-| `SELFHEAL_TIMEOUT_MS` | `60000` (1 min) | Per-replay LLM request timeout. |
-
-### Per-site base URL overrides
-
-Set `BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL` to override a plugin's
-`defaultBaseUrl` without source changes. Underscores in the env key map to
-hyphens in the `siteId`:
-
-```bash
-BARNACLE_SITE_MY_SHOP_BASE_URL="https://staging.my-shop.com"  # overrides plugin `my-shop`
-```
-
-### Out-of-tree plugins
-
-| Variable | Default | Purpose |
-|----------|---------|---------|
-| `BARNACLE_PLUGINS` | `""` | Comma-separated list of plugin specifiers to load at startup — relative paths (`./plugins/acme`) or package names (`@acme/barnacle-plugin`). Empty by default (built-ins only). |
-| `BARNACLE_PLUGINS_STRICT` | `false` | When `true`, any plugin that fails to load aborts the process instead of producing a disabled record. |
-| `BARNACLE_PLUGINS_DIR` | `process.cwd()` | Base directory used to resolve relative specifiers and locate the operator's `node_modules`. Defaults to wherever the binary is run — not the installed Barnacle package root. |
-| `BARNACLE_PLUGINS_CONFIG_DIR` | _(unset)_ | Directory scanned at startup for `*.plugin.json` config manifests, each loaded as a config-only plugin. Lets operators register sites by dropping a JSON file in a directory instead of editing `BARNACLE_PLUGINS`. An unreadable directory is logged and skipped — it never crashes boot. |
-
-**Resolution rule:** a specifier starting with `.` or `/` is treated as a filesystem path resolved relative to `BARNACLE_PLUGINS_DIR`. Anything else is treated as an npm package name and resolved via `require.resolve` against the operator's own `node_modules` inside `BARNACLE_PLUGINS_DIR`.
-
-**Failure policy:** by default (non-strict), a plugin that fails to load is logged at `warn` level and recorded as `"disabled"` in the load report — the server still boots with the remaining plugins. Set `BARNACLE_PLUGINS_STRICT=true` to abort startup on any load failure instead.
-
-**`zod/v4` requirement for plugin authors:** import Zod as `import { z } from "zod/v4"` in your plugin, not as bare `"zod"`. Barnacle uses `fastify-type-provider-zod` which compiles routes against core's own zod instance; a plugin schema built against a different zod import may pass load-time validation but fail at route registration.
-
-`GET /v1/plugins` (authenticated) returns the full plugin load report — one record per built-in and out-of-tree specifier — including `siteId`, `displayName`, `route`, `specifier`, `resolvedPath`, `apiVersion`, `status` (`"loaded"` or `"disabled"`), and an optional `reason` when disabled. Requires a valid `Authorization: Bearer <token>` header (reveals filesystem paths, so it is separate from the auth-free `/healthz`/`/readyz` probes).
-
----
+See [docs/architecture.md](./docs/architecture.md) for the design rationale and
+[docs/playbook.md](./docs/playbook.md) for the full step-by-step guide to
+adding a new site.
 
 ## Usage
 
@@ -980,33 +28,18 @@ BARNACLE_SITE_MY_SHOP_BASE_URL="https://staging.my-shop.com"  # overrides plugin
 
 - Node.js 22+
 - pnpm 10.4.1
-- A Steel account (`STEEL_API_KEY`) for managed browser sessions
-- An Anthropic key (`ANTHROPIC_API_KEY`) for Stagehand's LLM calls, **or** AWS Bedrock (`USE_BEDROCK=true` + AWS credentials) — see `.env.example` for details
+- A Browserbase account (`BROWSERBASE_API_KEY` + `BROWSERBASE_PROJECT_ID`) for managed browser sessions — the default provider. A Steel account (`STEEL_API_KEY`) is an alternative via `SCRAPER_PROVIDER=steel`
+- An Anthropic key (`ANTHROPIC_API_KEY`) for Stagehand's LLM calls, **or** AWS Bedrock (`USE_BEDROCK=true` + AWS credentials) — see [docs/configuration.md](./docs/configuration.md) for details
 
 ### Install
 
 ```bash
 pnpm install
-cp .env.example .env   # fill in STEEL_API_KEY and either ANTHROPIC_API_KEY or Bedrock creds
-```
-
-### Generating an API key
-
-Barnacle validates every request using bcrypt-hashed bearer tokens stored in
-`API_KEYS_HASHED`. To create one:
-
-```bash
-# 1. Generate a random plaintext key — save this, you'll send it as Authorization: Bearer <key>
-node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
-
-# 2. Hash it (bcrypt cost factor 10) — paste the output into API_KEYS_HASHED
-node -e "const b=require('bcryptjs');b.hash(process.argv[1],10,(e,h)=>console.log(h))" <your-key>
+cp .env.example .env   # fill in BROWSERBASE_API_KEY + BROWSERBASE_PROJECT_ID (or STEEL_API_KEY for SCRAPER_PROVIDER=steel) and either ANTHROPIC_API_KEY or Bedrock creds
 ```
 
-Comma-separate multiple hashes in `API_KEYS_HASHED` to support key rotation.
-
-For local development, set `DEV_BYPASS_AUTH=true` in `.env` to skip auth
-entirely — never set this in production.
+Generating and hashing an API key, plus the full env-var reference, live in
+[docs/configuration.md](./docs/configuration.md).
 
 ### Dev
 
@@ -1023,106 +56,31 @@ pnpm start
 
 ### Try it
 
-Barnacle boots with the built-in plugins registered (see `BUILTIN_SITE_PLUGINS` in `src/plugins/discover.ts`). Follow [Adding a New Site](#adding-a-new-site--the-recon-playbook) above to build and register a plugin; core will register `POST /v1/<your-siteId>/run` automatically at startup.
-
-With the dev server running (`pnpm run dev`), confirm the server is up:
+With the dev server running, confirm it's up:
 
 ```bash
 curl -s http://localhost:3000/health | jq .
 ```
 
-Once a plugin is registered, every response follows the same envelope shape. The status block is always present; the plugin's `responseSchema` fields are spread alongside it at the root:
-
-```json
-{
-  "status": {
-    "httpStatus": "OK",
-    "dateTime": "2025-05-16T12:00:00.000Z",
-    "details": []
-  },
-  "items": []
-}
-```
-
-The envelope is a **flat merge**, not nested — `status` lives at the root and the plugin's response fields are spread alongside it (`src/api/helpers/envelope.ts:8-25`). Parse as `{ status, ...pluginData }`, not `{ status, data: pluginData }`.
-
-Every response — success or error — uses the same envelope shape so clients share a single parser. Error details appear in `status.details[]` with numeric codes:
-
-| Code | Name | When |
-|------|------|------|
-| 1000 | `PARTIAL_CONTENT_SUCCESS` | Partial data returned |
-| 1001 | `DECODING_ERROR` | Request body could not be parsed |
-| 1002 | `FIELD_VIOLATION` | Schema validation failure on a field |
-| 1003 | `EMPTY_REQUEST` | Request body was missing or empty |
-| 1004 | `AUTHORIZATION_ERROR` | Bearer token missing or invalid |
-| 1005 | `RESOURCE_NOT_FOUND` | Requested resource does not exist |
-| 1006 | `INDEX_NOT_FOUND` | Internal index lookup failed |
-| 1007 | `CLIENT_CALL_ERROR` | Downstream client call failed |
-| 1008 | `GENERIC_ERROR` | Unclassified server error |
-| 1009 | `EXTRA_DETAIL` | Supplemental detail entry (informational) |
-| 1010 | `THROTTLED_REQUEST` | Rate limit exceeded (hot path 429) |
-| 1011 | `TIME_OUT` | Request timed out |
-| 2003 | `SCRAPE_FAILURE` | Browser automation failed after retries |
-| 2004 | `CAPTCHA_ENCOUNTERED` | CAPTCHA challenge could not be resolved |
-| 2005 | `EMPTY_RESULTS` | Scrape succeeded but returned no data |
-| 2006 | `VERIFICATION_TRIGGER_FAILED` | OTP trigger to the target site failed |
-| 2007 | `RESUME_INVALID_OTP` | Provided OTP was rejected by the target site |
-| 2008 | `URL_LOCKED` | Target site locked the target URL; back off and retry later |
-
-Full definitions: `src/api/schemas/common.ts`.
-
-**How scraper exceptions map to API codes** (`src/plugins/loader.ts:88-92`):
-
-- `CaptchaError` → `2004 CAPTCHA_ENCOUNTERED`
-- `EmptyResultsError` → `2005 EMPTY_RESULTS`
-- `HttpRateLimitError` → `1010 THROTTLED_REQUEST` (no browser fallback)
-- `HttpUrlLockedError` → `2008 URL_LOCKED` (no browser fallback; distinct from rate-limit for metrics)
-- Any other `ScraperError` → `2003 SCRAPE_FAILURE`
-- Task exceeded `TASK_TIMEOUT_MS` → `1011 TIME_OUT`
-
-## Endpoints
-
-Each registered plugin exposes a POST route following the default convention:
-`POST /v1/<siteId>/run`. When the hot path detects that required applicant
-answers are absent (e.g. Gender, Degree, EducationLevel, SignatureFullName) or
-a repeat-applicant OTP challenge, `/run` returns HTTP 200 with
-`{ needsUserInfo: true, missingFields: [{ field, question }], requiresOtp }`
-instead of a submission result, so the caller can collect the gaps and hand back.
-
-Plugins declare their own extra routes via `meta.extraRoutes`, which core registers
-uniformly — the engine has no per-site knowledge. Route paths are declared as
-`:siteId` templates, so the concrete path is whatever the plugin's `siteId` is. Two
-conventional shapes a plugin may add:
-
-- `POST /v1/<siteId>/resume` — body = the full original candidate payload plus
-  `collectedData` (and `otpCode` where the site issues an OTP challenge); re-runs the
-  hot path with the collected answers merged in; returns the same `{ verified }`
-  envelope as `/run`, or `2007 RESUME_INVALID_OTP` if the OTP is rejected
-- `POST /v1/<siteId>/trigger-otp` — body `{ offerId, email }`; asks the target site to
-  email an OTP to a repeat applicant; returns `{ success: true }` or a
-  `2006 VERIFICATION_TRIGGER_FAILED` error envelope
-
-See `examples/plugins/acme-jobs.plugin.json` for a runnable declaration.
-
-Operational routes:
-- `GET /healthz` — liveness probe
-- `GET /readyz`  — readiness probe (checks scraper credentials, queue depth)
-- `GET /docs`    — Swagger UI (when `ENABLE_DOCS=true`)
-- `GET /v1/plugins` — authenticated plugin load report (see [Out-of-tree plugins](#out-of-tree-plugins))
-- `GET /v1/submissions` — authenticated, queryable submit+beacon reconciliation rows (filter by `siteId`, `requestId`, `status`, `beaconStatus`, `from`/`to`; each row also carries the submit session block `session` (`{ id, provider, ip, ipCapturedAt }`) and the beacon-fire `beaconSessionIp`, neither of which is filterable at this layer, same as the opaque `joinKeys` bag; see [Submission-envelope sink](docs/telemetry-and-judging.md#submission-envelope-sink))
+Barnacle boots with the built-in plugins registered (see `BUILTIN_SITE_PLUGINS`
+in `src/plugins/discover.ts`). Each registered plugin exposes
+`POST /v1/<siteId>/run`; every response uses the same envelope shape
+(`{ status, ...pluginData }`) so clients share a single parser. See
+[docs/plugin-authoring.md#endpoints](./docs/plugin-authoring.md#endpoints)
+for the full endpoint reference.
 
 ## Commands
 
 | Command | What it does |
 |---------|--------------|
 | `pnpm run dev` | `tsx watch --env-file=.env src/server.ts` with hot reload |
-| `pnpm run build` | compile to `dist/` (tsc + path alias rewriting + copy `src/sites/` fixtures and `src/testing/fixtures`) |
+| `pnpm run build` | compile to `dist/` (tsc + path alias rewriting + copy fixtures) |
 | `pnpm start` | `node dist/server.js` |
 | `pnpm run typecheck` | strict TS noEmit |
 | `pnpm run lint` / `lint:fix` | Biome |
 | `pnpm run test` | Vitest unit + integration |
 | `pnpm test src/scraper/fixtures.test.ts` | Run a single test file (NEVER use `--` before the filter) |
-| `pnpm run test:watch` | Vitest in watch mode (re-runs on file changes) |
+| `pnpm run test:watch` | Vitest in watch mode |
 | `pnpm run test:coverage` | Vitest with v8 coverage report |
 | `pnpm run format` | Biome format write |
 | `pnpm run recon:browser` | Phase 1 — drive browser + capture API calls |
@@ -1131,165 +89,28 @@ Operational routes:
 | `pnpm run recon:summarize -- --site-id <id>` | Phase 4 (optional) — write human-readable findings doc |
 | `pnpm run recon:heal -- --site-id <id> --url <url>` | Self-heal a failing recon flow without modifying the source file |
 | `pnpm run smoke -- --site <id> --payload '...'` | Phase 6 — run nightly drift-detection smoke test |
-| `pnpm run judge:llm -- --calls-ndjson <path> --call-type <type>` | Score captured LLM calls on a three-dimensional rubric; writes a verdict JSON to `judge-out/` |
-| `pnpm run heal:llm -- --verdict-path <path> --call-type <type>` | Self-heal a failing prompt template: iterate patch→replay→score, write `healing-<callType>.md` with the best patch — production prompts are never modified |
-
-## Architecture
-
-```
-src/
-├── server.ts                  # Fastify bootstrap — calls loadAllPlugins(), registerRoutes(), site-agnostic
-├── site-plugin.ts             # SitePlugin<TInput,TOutput> interface (engine contract)
-├── config.ts                  # frozen env-typed config singleton
-├── plugins/
-│   ├── loader.ts              # dispatch(), registerRoutes(app, cfg, plugins)
-│   └── discover.ts            # BUILTIN_SITE_PLUGINS, loadAllPlugins(), loadPlugins()
-├── sites/
-│   ├── _shared/               # branch-local cross-plugin guards (coverage-expectations.test.ts)
-│   └── <site-id>/             # one directory per registered plugin
-├── api/
-│   ├── plugins/               # auth, error-handler, request-context
-│   ├── routes/                # health
-│   ├── schemas/               # common envelope schemas; LLM telemetry + judge-verdict schemas
-│   ├── helpers/envelope.ts    # success envelope builder
-│   └── errors.ts              # error hierarchy + envelope builder
-├── scraper/
-│   ├── session.ts             # Steel + Stagehand session factory
-│   ├── pool.ts                # p-queue over createBrowserSession
-│   ├── throttle.ts            # Bottleneck limiter + jitter
-│   ├── retry.ts               # p-retry + failure classification
-│   ├── errors.ts              # typed scraper error hierarchy
-│   ├── http-client.ts         # typed fetch wrapper (hot path)
-│   ├── rate-limited-json-client.ts # factory: Bottleneck + chromiumClientHints + createHttpClient in one call — prefer this over the three-step scaffold for Chromium-hint plugins
-│   ├── http-status-classifier.ts # pure status→ScraperError classifier for raw-fetch callers
-│   ├── raw-fetch.ts           # site-agnostic undici scaffold: network-error wrap, onResponse hook, optional classifyHttpStatus (skipClassify for callers that classify manually)
-│   ├── graphql-client.ts      # GraphQL POST wrapper
-│   ├── metrics.ts             # drift-detection counters
-│   ├── fixtures.ts            # static JSON fixture loader
-│   ├── navigate.ts            # shared awaitActivePage + goto(networkidle) helper
-│   ├── behavioral-signals.ts  # CDP synthetic mouse-move + scroll dispatcher for bot-detection warmup
-│   ├── session-warmup.ts      # generic pRetry browser-session runner: acquire → callback → close, with caller-supplied exhaustion mapping
-│   ├── session-ip.ts          # resolves a session's outbound IP via a throwaway tab + IP-echo navigation
-│   └── require-response-field.ts # shared helpers for extracting required fields from HTTP response objects (HttpSchemaError on missing/null)
-├── cache/
-│   ├── response-cache.ts      # lru-cache wrapper for deduplicating concurrent identical scraper requests
-│   └── keyed-ttl-cache.ts     # generic per-key TTL + single-flight coalescing cache factory
-├── lib/                       # logging, env, bedrock, db client, multipart, option-matcher, chromium-client-hints, telemetry/
-├── scripts/                   # recon-browser, recon-http, recon-generate, recon-summarize, recon-heal, recon-shared, smoke-test, judge-llm-batch, llm-heal
-├── testing/
-│   ├── integration-runner.ts              # site-agnostic scaffold for integration tests (allocate inbox → dispatch → poll)
-│   ├── mock-fetch-response.ts             # shared undici-compatible Response stub factory for flow tests that mock fetch
-│   ├── replay-integration-suite.ts        # generic describe.skipIf/it.each scaffold; eliminates per-site integration boilerplate
-│   ├── contract-parity-suite.ts           # offline schema-parity scaffold; one-call drop-in for accept + rejection-case coverage
-│   ├── coverage-guard-suite.ts            # registry-driven structural guard; asserts contract.parity.test.ts exists per registered plugin
-│   ├── batch-email-confirmation.ts        # two-phase batch runner: submit jobs → poll inboxes (site-agnostic)
-│   └── batch-report.ts                    # markdown table renderer for batch-test verdicts
-└── types/
-```
-
-**Library choices** (battle-tested — no custom reinventions):
-
-- API server: [`fastify`](https://fastify.dev/) + helmet + compress + rate-limit + swagger
-- Schema: [`zod`](https://zod.dev/) via `fastify-type-provider-zod`
-- Browser automation: [`@browserbasehq/stagehand`](https://github.com/browserbase/stagehand) + [`steel-sdk`](https://steel.dev)
-- Concurrency: [`p-queue`](https://github.com/sindresorhus/p-queue), [`p-retry`](https://github.com/sindresorhus/p-retry), [`bottleneck`](https://github.com/SGrondin/bottleneck)
-- Caching: [`lru-cache`](https://github.com/isaacs/node-lru-cache)
-- Logging: [`pino`](https://github.com/pinojs/pino) with CloudWatch 256KB splitting + sensitive-field redaction
-
-**Per-site base URL overrides:** set `BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL` to override a plugin's `defaultBaseUrl` without source changes. Underscores in the env key map to hyphens in the `siteId` (e.g. `BARNACLE_SITE_MY_SHOP_BASE_URL` → plugin `my-shop`).
-
-**Execution header:** send `x-barnacle-execution: browser` on any plugin request to skip the hot path and go directly to the Stagehand browser path. Omit the header (or send any other value) to use the default hot path. Useful for debugging or when you know the hot path is broken. (Fastify lowercases incoming header keys; the dispatcher reads `request.headers["x-barnacle-execution"]` — supply lowercase to match.)
+| `pnpm run judge:llm -- --calls-ndjson <path> --call-type <type>` | Score captured LLM calls on a three-dimensional rubric |
+| `pnpm run heal:llm -- --verdict-path <path> --call-type <type>` | Self-heal a failing prompt template |
 
----
-
-## Deployment
-
-### Production checklist
-
-```bash
-# .env (production)
-NODE_ENV=production
-ENABLE_DOCS=false         # never expose Swagger in prod
-TRUST_PROXY=true          # set false if deploying directly to the internet (no ALB/nginx)
-DEV_BYPASS_AUTH=false     # this is the default — confirm it's not set to true
-API_KEYS_HASHED="<bcrypt-hash>,<bcrypt-hash>"  # at least one key
-STEEL_API_KEY="..."
-ANTHROPIC_API_KEY="..."   # or USE_BEDROCK=true + AWS creds
-```
+## Writing a plugin
 
-### Process management
-
-Barnacle is a plain Node.js process. Use pm2 or systemd to keep it alive and
-restart it on crash:
-
-```bash
-# pm2
-pm2 start dist/server.js --name barnacle --env production
-pm2 save && pm2 startup
-
-# systemd (example unit)
-[Service]
-ExecStart=/usr/bin/node /srv/barnacle/dist/server.js
-WorkingDirectory=/srv/barnacle
-EnvironmentFile=/srv/barnacle/.env
-Restart=on-failure
-```
-
-### Reverse proxy
-
-Route traffic through nginx or an Application Load Balancer (ALB). Set
-`TRUST_PROXY=true` so Fastify uses `X-Forwarded-For` for the client IP
-(needed for rate limiting on unauthenticated traffic).
-
-```nginx
-location / {
-    proxy_pass         http://127.0.0.1:3000;
-    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header   X-Forwarded-Proto $scheme;
-    proxy_set_header   Host $host;
-}
-```
-
-### Health probes
-
-Wire `/healthz` as the liveness probe and `/readyz` as the readiness probe:
-
-```yaml
-# Kubernetes example
-livenessProbe:
-  httpGet: { path: /healthz, port: 3000 }
-  initialDelaySeconds: 5
-readinessProbe:
-  httpGet: { path: /readyz, port: 3000 }
-  initialDelaySeconds: 10
-```
-
-`/readyz` returns 503 when the scraper pool queue is saturated (depth >
-`READINESS_QUEUE_THRESHOLD`) or when required scraper credentials are missing.
-
----
-
-## Common issues
-
-| Symptom | Cause | Fix |
-|---------|-------|-----|
-| `Error: STEEL_API_KEY is required` | Missing env var | Add `STEEL_API_KEY` to `.env` |
-| `useProxy rejected` / `402` from Steel | Free-tier plan doesn't support residential proxies | Set `SCRAPER_PROXY_TYPE=none` and `SCRAPER_SOLVE_CAPTCHA=false` |
-| `401 Unauthorized` on every request | No API key configured or wrong plaintext key | Verify `API_KEYS_HASHED` is set; double-check the plaintext key. For dev, set `DEV_BYPASS_AUTH=true` |
-| Stagehand throws `model not found` | Wrong model name format | Use the `anthropic/` prefix: `STAGEHAND_MODEL=anthropic/claude-sonnet-4-6` |
-| `/readyz` returns 503 on `scraperCredentials` | `STEEL_API_KEY` or LLM key missing | Set the missing credential |
-| Build succeeds but `dist/sites/` is empty | `tsc` ran but `cp -r src/sites dist/sites` was skipped | Run `pnpm run build` (not `tsc` directly) — the build script copies site sources after compilation |
-
----
+A site plugin is a single TypeScript module that satisfies `SitePlugin<TInput, TOutput>`
+from `src/site-plugin.ts` — a `meta` block, an optional `executeHttp` hot path, and a
+Stagehand `execute` fallback. `pnpm run recon:generate` writes this structure for you.
+Plugins can also be config-only JSON manifests with no TypeScript at all. Full
+reference in [docs/plugin-authoring.md](./docs/plugin-authoring.md).
 
 ## Reference
 
 - Coding standards: [CLAUDE.md](./CLAUDE.md)
 - Architecture & design rationale: [docs/architecture.md](./docs/architecture.md)
 - Recon playbook (step-by-step): [docs/playbook.md](./docs/playbook.md)
+- Configuration (all env vars, deployment, common issues): [docs/configuration.md](./docs/configuration.md)
+- Plugin authoring guide: [docs/plugin-authoring.md](./docs/plugin-authoring.md) (runnable example: [examples/plugins/hello-site/](./examples/plugins/hello-site/README.md))
 - Testing guide: [docs/testing.md](./docs/testing.md)
+- Security policy & vulnerability reporting: [SECURITY.md](./SECURITY.md)
 - Telemetry & LLM judging concept guide: [docs/telemetry-and-judging.md](./docs/telemetry-and-judging.md)
-- Submission reconciliation runbook (join Barnacle runs to a plugin's own attribution provider's report): [docs/submission-reconciliation.md](./docs/submission-reconciliation.md)
+- Submission reconciliation runbook: [docs/submission-reconciliation.md](./docs/submission-reconciliation.md)
 - Per-site recon findings: [docs/target-recon.md](./docs/target-recon.md) (populated after first `pnpm run recon:summarize`)
 
 ## License