imprint-mcp 0.4.7 → 0.4.8

package/README.md

@@ -169,8 +169,8 @@ When an API call gets blocked, Imprint doesn't jump to DOM replay. It escalates
     │               + API
     ▼
   cdp-replay       ~2-35s    API calls run inside a live, trusted Chrome —
-    │                        reuses browser-observed request state and refreshes
-    │                        anti-bot tokens between protected POSTs
+    │                        a protected POST refreshes its anti-bot token
+    │                        between calls (multi-step state-changing flows)
     ▼
   stealth-fetch    ~1-12s    Defeats Akamai, Cloudflare, DataDome
     │
@@ -178,9 +178,9 @@ When an API call gets blocked, Imprint doesn't jump to DOM replay. It escalates
   playbook         ~9s       Full DOM replay — universal fallback
 ```
 
-The full order is `fetch → fetch-bootstrap → cdp-replay → stealth-fetch → playbook`; `auto` mode walks it and stops at the first backend that works. Workflows that declare browser-observed request captures can start at `cdp-replay`, so MCP sessions reuse the same Chrome instead of paying a cold bootstrap on each route/date.
+The full order is `fetch → fetch-bootstrap → cdp-replay → stealth-fetch → playbook`; `auto` mode walks it and stops at the first backend that works.
 
-For bot-protected sites, `imprint probe-backends <site> --tool <toolName>` writes a `backends.json` preference cache so cron and MCP start from the known-good backend instead of rediscovering blocked rungs. Use `imprint probe-backends <site> --all` to refresh every tool in a multi-tool site; `imprint mcp status` reports stale or invalid backend caches before they quietly fall back to the default ladder. Execution-only workflow changes keep the cached backend order when the backend capability hash still matches. CDP replay records both cold and warm timings when it succeeds: a timeout-safe cold start may rank by its fast warm runtime, but a cold start above the preferred threshold stays behind cold-safe backends in durable cache order.
+For bot-protected sites, `imprint probe-backends <site> --tool <toolName>` writes a `backends.json` preference cache so cron and MCP start from the known-good backend instead of rediscovering blocked rungs. Use `imprint probe-backends <site> --all` to refresh every tool in a multi-tool site; `imprint mcp status` reports stale or invalid backend caches before they quietly fall back to the default ladder. CDP replay records both cold and warm timings when it succeeds: a timeout-safe cold start may rank by its fast warm runtime, but a cold start above the preferred threshold stays behind cold-safe backends in durable cache order.
 
 Every recording compiles to *both* `workflow.json` and `playbook.yaml`, so the ladder always has a DOM fallback.
 

package/examples/google-flights/README.md

@@ -17,8 +17,6 @@ A 4-tool MCP server for Google Flights, compiled from a recording of a normal fl
 
 - **Protocol**: Google's `/_/FlightsFrontendUi` **`batchexecute`** endpoint returns a nested-array (protobuf-ish) payload. The compiler reverse-engineered the encoding into `_shared/batchexecute.ts` (shared decoder) + per-tool `parser.ts`, and the `f.req` request shape into `_shared/flights_request.ts` + per-tool `request-transform.ts`.
 - **Anti-bot**: the per-page `f.sid` / `bl` tokens are bootstrapped at runtime (`${state.f_sid}` placeholders), and calls run on the **cdp-replay** rung (requests issued inside a live, trusted Chrome) with a **stealth-fetch** fallback.
-- **MCP pacing**: `search_flights` declares `execution.minCallSpacingMs: 2000` because Google Flights can return fast empty result sets when warm CDP searches are fired back-to-back with no breathing room.
-- **Bounded fallback**: these tools declare `execution.skipPlaybookFallback` so MCP calls fail fast after the API/browser-backed rungs are exhausted instead of spending the rest of the agent timeout in an unstructured DOM replay.
 - **Artifacts per tool**: `workflow.json` (API replay), `playbook.yaml` (DOM fallback), `index.ts` (MCP tool), `parser.ts` + `request-transform.ts` (codecs).
 
 ## Install

package/examples/google-flights/_shared/flights_request.ts

@@ -8,11 +8,9 @@
 // with the date range living in the outer wrapper; Shopping/Booking use the full
 // 15-slot leg with DATE at [6]. Verified by decoding seq 97 vs seq 111.
 
-// Fresh searches emit wrapper `...,0,0,0,1]` and use leg[14]=3 for normal
-// shopping legs. Return leg [14]=1 appears in booking / selected-leg flows, not
-// in the initial search request used by this tool.
-// In-page-refined searches use `...,0,1,0,1]` — a UI freshness flag, not a user
-// param; we always emit the fresh form for shopping.
+// Fresh searches emit wrapper `...,0,0,0,1]` and leg[14]=3 (proven seq 111/140).
+// In-page-refined searches use `...,0,1,0,1]` with return-leg[14]=1 (seq 194/425) —
+// a UI freshness flag, not a user param; we always emit the fresh form for shopping.
 // Booking outbound legs use [14]=3, return legs [14]=1 (seq 764/811).
 
 function buildLeg(leg: any): any[] {
@@ -76,11 +74,7 @@ export function transform(
   let payload: any;
 
   if (rpc === 'GetShoppingResults') {
-    const searchContext =
-      typeof p.searchContextToken === 'string' && p.searchContextToken
-        ? [null, null, null, p.searchContextToken]
-        : [];
-    payload = [searchContext, sp, 0, 0, 0, 1];
+    payload = [[], sp, 0, 0, 0, 1];
   } else if (rpc === 'GetCalendarPicker') {
     const legs = sp[13];
     if (Array.isArray(legs)) sp[13] = legs.map((l: any) => (Array.isArray(l) ? l.slice(0, 4) : l));

package/examples/google-flights/get_flight_booking_details/index.ts

@@ -94,7 +94,7 @@ const WORKFLOW: Workflow = {
     "captures": [
       {
         "name": "f_sid",
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap",
         "source": "html_regex",
         "pattern": "\"FdrFJe\":\"([^\"]+)\"",
@@ -102,7 +102,7 @@ const WORKFLOW: Workflow = {
       },
       {
         "name": "bl",
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap",
         "source": "html_regex",
         "pattern": "\"cfb2h\":\"([^\"]+)\"",
@@ -112,9 +112,6 @@ const WORKFLOW: Workflow = {
   },
   "parserModule": "./parser.ts",
   "requestTransformModule": "./request-transform.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 };
 

package/examples/google-flights/get_flight_booking_details/parser.ts

@@ -142,9 +142,6 @@ export function extract(
   let frames: Array<{ rpcid: string | null; payload: any }> = [];
   if (typeof rawResponse === 'string') {
     frames = decodeBatchExecute(rawResponse);
-    if (frames.length === 0) {
-      throw new Error('Google Flights GetBookingResults response did not contain a batchexecute payload');
-    }
   } else if (rawResponse != null) {
     frames = [{ rpcid: null, payload: rawResponse }];
   }
@@ -173,11 +170,6 @@ export function extract(
 
   const segments = [...segMap.values()];
   const fareOptions = [...fareMap.values()];
-  if (segments.length === 0 && fareOptions.length === 0) {
-    throw new Error(
-      'Google Flights GetBookingResults payload did not contain recognizable booking details',
-    );
-  }
   const prices = fareOptions.map((f) => f.priceUSD);
 
   return {

package/examples/google-flights/get_flight_booking_details/workflow.json

@@ -65,7 +65,7 @@
         "name": "f_sid",
         "pattern": "\"FdrFJe\":\"([^\"]+)\"",
         "group": 1,
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap"
       },
       {
@@ -73,7 +73,7 @@
         "name": "bl",
         "pattern": "\"cfb2h\":\"([^\"]+)\"",
         "group": 1,
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap"
       }
     ]
@@ -94,8 +94,5 @@
   ],
   "requestTransformModule": "./request-transform.ts",
   "parserModule": "./parser.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 }

package/examples/google-flights/get_flight_calendar_prices/index.ts

@@ -74,7 +74,7 @@ const WORKFLOW: Workflow = {
     "captures": [
       {
         "name": "f_sid",
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap",
         "source": "html_regex",
         "pattern": "\"FdrFJe\":\"([^\"]+)\"",
@@ -82,7 +82,7 @@ const WORKFLOW: Workflow = {
       },
       {
         "name": "bl",
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap",
         "source": "html_regex",
         "pattern": "\"cfb2h\":\"([^\"]+)\"",
@@ -92,9 +92,6 @@ const WORKFLOW: Workflow = {
   },
   "parserModule": "./parser.ts",
   "requestTransformModule": "./request-transform.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 };
 

package/examples/google-flights/get_flight_calendar_prices/parser.ts

@@ -54,9 +54,6 @@ export function extract(
 ): unknown {
   const raw = typeof rawResponse === 'string' ? rawResponse : JSON.stringify(rawResponse ?? '');
   const frames = decodeBatchExecute(raw);
-  if (frames.length === 0) {
-    throw new Error('Google Flights GetCalendarPicker response did not contain a batchexecute payload');
-  }
 
   let payload: unknown = null;
   for (const f of frames) {
@@ -66,14 +63,13 @@ export function extract(
       break;
     }
   }
+  // If no frame produced entries, still attempt the first frame's payload so an
+  // empty (zero-result) response yields an empty calendar rather than throwing.
+  if (payload == null && frames.length > 0) payload = frames[0]?.payload ?? null;
+
   const entries = collectEntries(payload).sort((a, b) =>
     a.departureDate < b.departureDate ? -1 : a.departureDate > b.departureDate ? 1 : 0,
   );
-  if (entries.length === 0) {
-    throw new Error(
-      'Google Flights GetCalendarPicker payload did not contain recognizable calendar prices',
-    );
-  }
 
   const prices: Record<string, number> = {};
   for (const e of entries) prices[e.departureDate] = e.lowestPriceUSD;

package/examples/google-flights/get_flight_calendar_prices/workflow.json

@@ -44,7 +44,7 @@
         "name": "f_sid",
         "pattern": "\"FdrFJe\":\"([^\"]+)\"",
         "group": 1,
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap"
       },
       {
@@ -52,7 +52,7 @@
         "name": "bl",
         "pattern": "\"cfb2h\":\"([^\"]+)\"",
         "group": 1,
-        "required": true,
+        "required": false,
         "capability": "browser_bootstrap"
       }
     ]
@@ -74,8 +74,5 @@
   ],
   "requestTransformModule": "./request-transform.ts",
   "parserModule": "./parser.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 }

package/examples/google-flights/lookup_airport/index.ts

@@ -71,9 +71,6 @@ const WORKFLOW: Workflow = {
   },
   "parserModule": "./parser.ts",
   "requestTransformModule": "./request-transform.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 };
 

package/examples/google-flights/lookup_airport/parser.ts

@@ -45,15 +45,8 @@ export function extract(
 ): unknown {
   const raw = typeof rawResponse === 'string' ? rawResponse : JSON.stringify(rawResponse);
   const payload = extractRpcPayload(raw, 'tDoGIe');
-  if (payload == null) {
-    throw new Error('Google Flights tDoGIe response did not contain a batchexecute payload');
-  }
 
-  if (!Array.isArray(payload) || !Array.isArray(payload[1])) {
-    throw new Error('Google Flights tDoGIe payload did not contain a recognizable match list');
-  }
-
-  const matchesRaw = payload[1];
+  const matchesRaw = Array.isArray(payload) && Array.isArray(payload[1]) ? payload[1] : [];
   const matches = matchesRaw
     .map(parseItem)
     .filter((m): m is AirportMatch => m !== null);

package/examples/google-flights/lookup_airport/workflow.json

@@ -53,8 +53,5 @@
   ],
   "parserModule": "./parser.ts",
   "requestTransformModule": "./request-transform.ts",
-  "execution": {
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 }

package/examples/google-flights/search_flights/index.ts

@@ -120,13 +120,11 @@ const WORKFLOW: Workflow = {
   "requests": [
     {
       "method": "POST",
-      "url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=${state.reqid}&rt=c",
+      "url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=1708023&rt=c",
       "headers": {
         "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
         "X-Same-Domain": "1",
-        "Referer": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
-        "Accept-Language": "en-US,en;q=0.9",
-        "X-Goog-BatchExecute-Bgr": "${state.bgr}",
+        "Referer": "https://www.google.com/travel/flights",
         "x-goog-ext-259736195-jspb": "[\"en-US\",\"US\",\"USD\",2,null,[420],null,null,7,[]]"
       },
       "body": "f.req=${param.origin}|${param.destination}|${param.departure_date}|${param.return_date}|${param.trip_type}|${param.max_stops}|${param.airlines}|${param.max_price}|${param.outbound_times}|${param.return_times}|${param.max_duration}|${param.carry_on_bags}&",
@@ -135,7 +133,7 @@ const WORKFLOW: Workflow = {
   ],
   "site": "google-flights",
   "bootstrap": {
-    "url": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
+    "url": "https://www.google.com/travel/flights",
     "waitUntil": "domcontentloaded",
     "timeoutMs": 30000,
     "captures": [
@@ -143,75 +141,22 @@ const WORKFLOW: Workflow = {
         "name": "f_sid",
         "required": true,
         "capability": "browser_bootstrap",
-        "source": "request_url_regex",
-        "pattern": "[?&]f\\.sid=([^&]+)",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
+        "source": "html_regex",
+        "pattern": "\"FdrFJe\":\"([^\"]+)\"",
         "group": 1
       },
       {
         "name": "bl",
         "required": true,
         "capability": "browser_bootstrap",
-        "source": "request_url_regex",
-        "pattern": "[?&]bl=([^&]+)",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "group": 1
-      },
-      {
-        "name": "bgr",
-        "required": true,
-        "capability": "browser_bootstrap",
-        "source": "request_header",
-        "header": "X-Goog-BatchExecute-Bgr",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last"
-      },
-      {
-        "name": "reqid",
-        "required": true,
-        "capability": "browser_bootstrap",
-        "source": "request_url_regex",
-        "pattern": "[?&]_reqid=([^&]+)",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "group": 1
-      },
-      {
-        "name": "search_context_token",
-        "required": false,
-        "capability": "browser_bootstrap",
-        "source": "request_body_regex",
-        "pattern": "%5B%5Bnull%2Cnull%2Cnull%2C%5C%22(.+?)%5C%22%5D",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "group": 1
-      },
-      {
-        "name": "observed_search_body",
-        "required": false,
-        "capability": "browser_bootstrap",
-        "source": "request_body_regex",
-        "pattern": "^(f\\.req=.*)$",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
+        "source": "html_regex",
+        "pattern": "\"cfb2h\":\"([^\"]+)\"",
         "group": 1
       }
     ]
   },
   "parserModule": "./parser.ts",
   "requestTransformModule": "./request-transform.ts",
-  "execution": {
-    "minCallSpacingMs": 2000,
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 };
 

package/examples/google-flights/search_flights/request-transform.ts

@@ -62,56 +62,13 @@ function num(v: unknown): number | undefined {
   return Number.isFinite(n) && n > 0 ? n : undefined;
 }
 
-function buildBootstrapQuery(params: Params): string {
-  const origin = params.origin != null ? String(params.origin) : '';
-  const destination = params.destination != null ? String(params.destination) : '';
-  const departureDate = params.departure_date != null ? String(params.departure_date) : '';
-  const tripType = String(params.trip_type ?? 'round_trip').toLowerCase();
-  if (tripType === 'one_way' || tripType === 'oneway' || tripType === '2') {
-    return `One way flights from ${origin} to ${destination} on ${departureDate}`;
-  }
-  if (params.return_date) {
-    return `Round trip flights from ${origin} to ${destination} departing ${departureDate} returning ${params.return_date}`;
-  }
-  return `Flights from ${origin} to ${destination} on ${departureDate}`;
-}
-
-export function prepareParams(params?: Params): Params {
-  const p: Params = params ?? {};
-  return {
-    ...p,
-    bootstrap_query: buildBootstrapQuery(p),
-  };
-}
-
-function hasNonDefaultFilters(params: Params): boolean {
-  if (params.max_stops != null && params.max_stops !== '' && Number(params.max_stops) !== 3) {
-    return true;
-  }
-  return Boolean(
-    params.airlines ||
-      num(params.max_price) ||
-      params.outbound_times ||
-      params.return_times ||
-      num(params.max_duration) ||
-      num(params.carry_on_bags),
-  );
-}
-
 export function transform(
   method: string,
   url: string,
   responses: Record<string, any>,
   params?: Params,
-  state?: Record<string, unknown>,
 ): { url: string; body: string } {
   const p: Params = params ?? {};
-  const observedSearchBody =
-    typeof state?.observed_search_body === 'string' ? state.observed_search_body : undefined;
-  if (observedSearchBody && !hasNonDefaultFilters(p)) {
-    return { url, body: observedSearchBody };
-  }
-
   const tripType = mapTripType(p.trip_type);
   const stops = p.max_stops != null && p.max_stops !== '' ? mapStops(p.max_stops) : 0;
   const { alliances, carriers } = parseAirlines(p.airlines);
@@ -156,8 +113,6 @@ export function transform(
     // CONFIG[10] wire form is [1, <carry-on count>]; shared builder emits
     // [carryOn, checked], so map count -> checked slot, constant 1 -> first.
     bags: carryOn != null ? { carryOn: 1, checked: carryOn } : undefined,
-    searchContextToken:
-      typeof state?.search_context_token === 'string' ? state.search_context_token : undefined,
   };
 
   return sharedTransform(method, url, responses, mapped);

package/examples/google-flights/search_flights/workflow.json

@@ -101,87 +101,36 @@
     }
   ],
   "bootstrap": {
-    "url": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
+    "url": "https://www.google.com/travel/flights",
     "waitUntil": "domcontentloaded",
     "timeoutMs": 30000,
     "captures": [
       {
-        "source": "request_url_regex",
+        "source": "html_regex",
         "name": "f_sid",
-        "pattern": "[?&]f\\.sid=([^&]+)",
+        "pattern": "\"FdrFJe\":\"([^\"]+)\"",
         "group": 1,
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
         "required": true,
         "capability": "browser_bootstrap"
       },
       {
-        "source": "request_url_regex",
+        "source": "html_regex",
         "name": "bl",
-        "pattern": "[?&]bl=([^&]+)",
+        "pattern": "\"cfb2h\":\"([^\"]+)\"",
         "group": 1,
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
         "required": true,
         "capability": "browser_bootstrap"
-      },
-      {
-        "source": "request_header",
-        "name": "bgr",
-        "header": "X-Goog-BatchExecute-Bgr",
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "required": true,
-        "capability": "browser_bootstrap"
-      },
-      {
-        "source": "request_url_regex",
-        "name": "reqid",
-        "pattern": "[?&]_reqid=([^&]+)",
-        "group": 1,
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "required": true,
-        "capability": "browser_bootstrap"
-      },
-      {
-        "source": "request_body_regex",
-        "name": "search_context_token",
-        "pattern": "%5B%5Bnull%2Cnull%2Cnull%2C%5C%22(.+?)%5C%22%5D",
-        "group": 1,
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "required": false,
-        "capability": "browser_bootstrap"
-      },
-      {
-        "source": "request_body_regex",
-        "name": "observed_search_body",
-        "pattern": "^(f\\.req=.*)$",
-        "group": 1,
-        "method": "POST",
-        "urlPattern": "FlightsFrontendService/GetShoppingResults",
-        "mode": "last",
-        "required": false,
-        "capability": "browser_bootstrap"
       }
     ]
   },
   "requests": [
     {
       "method": "POST",
-      "url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=${state.reqid}&rt=c",
+      "url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=1708023&rt=c",
       "headers": {
         "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
         "X-Same-Domain": "1",
-        "Referer": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
-        "Accept-Language": "en-US,en;q=0.9",
-        "X-Goog-BatchExecute-Bgr": "${state.bgr}",
+        "Referer": "https://www.google.com/travel/flights",
         "x-goog-ext-259736195-jspb": "[\"en-US\",\"US\",\"USD\",2,null,[420],null,null,7,[]]"
       },
       "body": "f.req=${param.origin}|${param.destination}|${param.departure_date}|${param.return_date}|${param.trip_type}|${param.max_stops}|${param.airlines}|${param.max_price}|${param.outbound_times}|${param.return_times}|${param.max_duration}|${param.carry_on_bags}&",
@@ -190,9 +139,5 @@
   ],
   "requestTransformModule": "./request-transform.ts",
   "parserModule": "./parser.ts",
-  "execution": {
-    "minCallSpacingMs": 2000,
-    "skipPlaybookFallback": true
-  },
   "liveVerified": true
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "imprint-mcp",
-  "version": "0.4.7",
+  "version": "0.4.8",
   "description": "Teach an AI agent how to use any website. Once. Records a real browser session + narration; generates a deterministic MCP tool plus a DOM-replay playbook fallback.",
   "type": "module",
   "exports": {

package/prompts/build-planning.md

@@ -59,7 +59,7 @@ You receive:
 
 1. **Emit exactly one `perTool` entry per `selectedTools` entry**, using the same `toolName`. Do not invent or drop tools.
 2. **Only hoist a shared module when ≥2 selected tools genuinely share it.** Single-use logic stays inside that tool's own parser.ts / request-transform.ts — do NOT create a `_shared/` module for it.
-3. **`request-transform`** — URL signing, body construction, or bootstrap-param preparation shared across tools. Wire-up: the consuming tool sets `requestTransformModule: "../_shared/<name>.ts"`. Ground it in `ephemeralValues` (browser_minted, high-entropy query param) and `sourceSeqs`. The exported `transform(method, url, responses, params?, state?)` returns the signed URL (or `{ url, body?, headers? }`); an optional `prepareParams(params)` can return primitive params used before bootstrap URL substitution.
+3. **`request-transform`** — URL signing or body construction shared across tools. Wire-up: the consuming tool sets `requestTransformModule: "../_shared/<name>.ts"`. Ground it in `ephemeralValues` (browser_minted, high-entropy query param) and `sourceSeqs`. The exported `transform(method, url, responses, params?)` returns the signed URL (or `{ url, body? }`).
 4. **`parser-helper`** — a decoder/normalizer ≥2 tools' parsers call (e.g. a shared JSPB walker, a shared field mapper). The consuming tool's parser.ts does `import { ... } from '../_shared/<name>.ts'`. Ground it in a captured response body (`sourceSeqs`).
 5. **`types`** — shared TypeScript interfaces used by ≥2 parsers. Type-only; no runtime behavior.
 6. **Auth is NEVER a shared module.** Login is request data, and the runtime cannot run a shared sub-workflow. Put the exact recipe in each tool's `authRecipe` (login seqs, credential names, captures with `${state.X}` wiring) and set `required: false` with empty arrays when a tool needs no login. Every authed tool replicates the same recipe inline.

package/prompts/compile-agent.md

@@ -52,7 +52,7 @@ Follow these steps to compile the session:
      - *Session-scoped state* (minted once per page load, reused across requests): add a bootstrap capture with `browser_bootstrap` capability. Pick the `source` based on where the value actually lives in the recording — these are not interchangeable:
        - **Response header** (`source: 'response_header'`, `header: '<exact name>'`): the bootstrap GET's HTTP response carries the token as a header. Enterprise CSRF tokens, anti-replay tokens, and many app-minted page nonces are returned this way. **First check** — search the bootstrap response headers for the recorded token before reaching for any HTML/DOM source. If the token appears in `requests[0].response.headers`, this is the only correct source. Do NOT synthesize an `_shared/page-tokens.ts` HTML-regex helper for it; the body will not contain the value and the regex will silently miss.
 
-       **Capture-source cross-check (verifier-enforced).** Before you declare any `required` capture, locate the matching recorded request in the session and confirm the declared source actually carries the recorded value: `response_header` → the header must exist in `response.headers`; `cookie` → `response.headers['set-cookie']` must define that cookie name; `html_regex` / `text_regex` → the pattern must match the recorded response body; `request_header` / `request_url_regex` / `request_body_regex` → a browser-observed request matching `method`/`urlPattern` must carry the value in its headers, URL, or body. The verifier rejects `done()` if the declared source does not produce a value in the recording, and it explicitly classifies a runtime `STATE_MISSING` from a declared capture as a workflow-correctness error (not infra) so the tool cannot ship waived. Picking the wrong source is the most common cause of "API rungs all silently fall to playbook" — measure twice.
+       **Capture-source cross-check (verifier-enforced).** Before you declare any `required` capture, locate the matching recorded request in the session and confirm the declared source actually carries the recorded value: `response_header` → the header must exist in `response.headers`; `cookie` → `response.headers['set-cookie']` must define that cookie name; `html_regex` / `text_regex` → the pattern must match the recorded response body. The verifier rejects `done()` if the declared source does not produce a value in the recording, and it explicitly classifies a runtime `STATE_MISSING` from a declared capture as a workflow-correctness error (not infra) so the tool cannot ship waived. Picking the wrong source is the most common cause of "API rungs all silently fall to playbook" — measure twice.
 
        **Referenced-capture cross-check — applies even to `required: false` captures (verifier-enforced).** If ANY request hard-references a capture via `${state.X}` in a header/body/url, that capture is effectively required regardless of its `required` flag, and the verifier checks its `html_regex`/`text_regex` pattern against EVERY recorded HTML page for the site (not just the bootstrap URL's own response — the bootstrap page may not even be in the recording). If the pattern matches no recorded page, `done()` is rejected (the runtime would `STATE_MISSING` the whole request). **Write the regex against the token as it ACTUALLY appears in the recorded HTML — read the recorded page first.** Common pitfall: a token embedded as `mUtil.createSecureCookie("Csrf-token", "<hex>")` is NOT matched by a pattern like `[Cc]srf[^"']{0,24}['"]([0-9a-f]{48,})['"]` because the `", "` separator between the cookie name and value falls between the two quotes — anchor on the real structure instead, e.g. `createSecureCookie\("Csrf-token",\s*"([0-9a-f]+)"`. When the live call would burn an anti-bot `.act`, the verifier SKIPS the live test entirely if a referenced capture can't resolve — so a wrong regex here costs you a whole verification cycle with no live signal. Get it right against the recording first.
 
@@ -60,7 +60,6 @@ Follow these steps to compile the session:
        - **HTML body** (`source: 'html_regex'`): the token is embedded in a `<script>` block, meta tag, or inline JSON inside the HTML. Use this only after confirming the value actually appears in the response body.
        - **DOM** (`source: 'dom_attribute'` / `source: 'dom_text'`): the token is rendered into a specific element by the page's JS — use a stable selector.
        - **Cookie / storage** (`source: 'cookie'` / `'local_storage'` / `'session_storage'`): the token is persisted client-side after bootstrap.
-       - **Browser-observed request** (`source: 'request_header'` / `'request_url_regex'` / `'request_body_regex'`): the bootstrap page's own XHR/fetch request carries the token, request id, or complete POST envelope. Use `method`, `urlPattern`, and `mode` to select the observed request, then extract from its header, URL, or body. This is the right source when the value is minted by page JavaScript but is absent from HTML, cookies, storage, and response headers.
      - *Per-request state* (unique per API call — nonces, request IDs, timestamps): write a `requestTransformModule` that generates fresh values.
      - *Bot-defense state* (sensor headers, fingerprints): use `stealth_bootstrap` capability.
    - **`constant`**: Identical across every pass the classifier compared — usually safe to hardcode. BUT: scrutinize high-entropy “constants” (UUIDs, JWTs, long hex/base64 strings). They may be slow-rotating tokens that happened to match across two runs taken minutes apart. If a constant looks like a token, treat it with suspicion and consider adding a bootstrap capture as a safety measure. **Exception — cross-recording corroboration.** The classifier diffs the recording against the automated replay AND against every other recording of this site (often captured hours or days apart), then keeps a value `constant` only if it never varied in any pass. A high-entropy value classified `constant` on this basis is *static infrastructure the server checks on every call*, NOT a rotating token: a GraphQL safelisting / persisted-query signature (`graphql-operation-signature`, `x-apollo-operation-id`, `x-apollo-operation-signature`), an API build/asset hash, a public app key. **Keep it verbatim** — dropping it gets the request 403'd or silently degraded to sentinel data. A genuinely rotating token could not be byte-identical across time-separated recordings; the classifier would have marked it `browser_minted`/`server_derived`. (The replay alone is unreliable here: anti-bot edges block the automated replay, so a protected header may be `constant` *purely* on cross-recording evidence — that evidence is sufficient; do not second-guess it as "high-entropy so probably rotating".)
@@ -98,7 +97,7 @@ Follow these steps to compile the session:
    - Keep headers minimal — drop bot-detection headers (Akamai fingerprints, DataDome, PerimeterX), drop browser-internal headers, keep `Content-Type`, `Origin`, `Referer` when needed
    - **CRITICAL — preserve FUNCTIONAL request headers (same principle as query params).** Beyond the standard set, the recorded request often carries headers the server *checks* on every call: anti-CSRF / anti-replay tokens (`X-Csrf-Token`, `X-XSRF-Token`, `RequestVerificationToken`, …), API keys, session/nonce headers, `X-*` app headers. These are part of the functional contract — dropping one usually makes a state-changing POST silently fail or get tarpitted, exactly like dropping a query param. For each non-bot, non-browser-internal header on the recorded request: keep it. If its value is a per-session/per-call token (high-entropy, rotates across the recording), do NOT hardcode it — capture it (`${state.NAME}` from a bootstrap/request capture) and template it. The litmus test mirrors query params: if the recorded request sent it and it isn't a bot fingerprint, the workflow request must send it too (literal if static, `${state.X}`/`${param.X}` if dynamic). A recorded state-changing POST (`*.act`, `/checkout`, `/book`, anything that mutates) that carried a CSRF/session header MUST template that header from captured state — never silently omit it.
    - **CRITICAL: Preserve ALL query parameters from the recorded URL.** Unlike HTTP headers — where you drop bot-detection fingerprints — query params are part of the API's functional contract. Even if a param value looks obfuscated or high-entropy (base64, hex, random-looking), it likely carries meaning the server checks (anti-bot tokens, session binding, A/B bucketing, obfuscated checksums). Preserve every param key: substitute the value with `${response[N].name}` or `${state.name}` if it came from an earlier response, `${param.NAME}` if user-variable, or keep the literal value if it's a static constant (like `search=false`). Missing a single query param can silently cause the API to return sentinel/degraded data rather than an error — the server may fall back to generic defaults instead of returning the actual results.
-   - **Per-call query params (URL signing).** If a query param has a different high-entropy value on every request to the same URL path in the session, it is likely a URL signing token computed by client-side JavaScript. Do NOT hardcode the recorded value — it is per-call and will expire. Instead: use `search_response_body` to search the session's JavaScript responses (look for `.js` URLs) for the param name. The signing function is usually simple (HMAC, MD5, XOR + base64 with a static key). Once you find it, write a `requestTransformModule` (sibling to `parser.ts`) that exports `transform(method: string, url: string, responses: unknown[], params?: Record<string, string | number | boolean>, state?: Record<string, unknown>)` — it takes the unsigned URL and returns the URL with the signing param appended. Set `"requestTransformModule": "./request-transform.ts"` in workflow.json. The runtime calls this function before each request.
+   - **Per-call query params (URL signing).** If a query param has a different high-entropy value on every request to the same URL path in the session, it is likely a URL signing token computed by client-side JavaScript. Do NOT hardcode the recorded value — it is per-call and will expire. Instead: use `search_response_body` to search the session's JavaScript responses (look for `.js` URLs) for the param name. The signing function is usually simple (HMAC, MD5, XOR + base64 with a static key). Once you find it, write a `requestTransformModule` (sibling to `parser.ts`) that exports `transform(method: string, url: string): string` — it takes the unsigned URL and returns the URL with the signing param appended. Set `"requestTransformModule": "./request-transform.ts"` in workflow.json. The runtime calls this function before each request.
    - **Complex body construction via requestTransformModule.** When the API uses a body format where simple `${param.X}` placeholder substitution cannot correctly encode values — e.g., JSPB arrays in form-encoded fields, nested JSON strings with position-dependent escaping — write a `requestTransformModule` that constructs the body programmatically. The transform receives `params` as a 4th argument and can return an object instead of a string:
      ```typescript
      export function transform(
@@ -106,13 +105,12 @@ Follow these steps to compile the session:
        url: string,
        responses: unknown[],
        params?: Record<string, string | number | boolean>,
-       state?: Record<string, unknown>,
      ): { url: string; body?: string } {
        const body = buildRequestBody(params ?? {});
        return { url, body };
      }
      ```
-     Returning a plain `string` (just the URL) still works for simple URL-signing. Use the object return when you need to build or modify the request body or headers. If the bootstrap URL itself needs a derived value, export `prepareParams(params)` from the same module; it runs before bootstrap URL substitution and can return additional primitive params. Do NOT invent URL query parameters as a workaround for body-encoding complexity — the server ignores unknown query params and the parameters will have no effect.
+     Returning a plain `string` (just the URL) still works for simple URL-signing. Use the object return when you need to build or modify the request body or headers. Do NOT invent URL query parameters as a workaround for body-encoding complexity — the server ignores unknown query params and the parameters will have no effect.
    - **`x-api-key` is normally NOT a credential.** It's an app-level identifier baked into the site's JavaScript — same for every visitor, not user-specific. Keep it as a literal string in the workflow. Only treat it as a credential if you can clearly see it varies per account (e.g., it appears in a `Set-Cookie` after login, or differs across sessions). The same applies to `x-channel-id`, `x-app-id`, `x-app-version`, and similar metadata headers — hardcode them.
    - **NEVER use `${env.NAME}` placeholders.** The `${env.X}` syntax exists in the runtime but is reserved for operator-level configuration, not for values you can see in the recording. If a value appears in the captured request, hardcode it. If multiple candidates in the same session use different API keys for different endpoints, hardcode each one — they are endpoint-specific app constants, not secrets. The only valid placeholder types for your workflow are `${param.NAME}`, `${credential.NAME}`, `${state.NAME}`, and `${response[N].NAME}`.
    - If the workflow chains multiple requests (request N+1 uses a value from request N's response), add an `extract` field to request N and reference it in request N+1 via `${response[N].name}`

package/prompts/prereq-builder.md

@@ -22,8 +22,7 @@ You receive `{ site, url, module, availableDependencies, sources, implementation
 ## Output requirements by `kind`
 
 ### `request-transform`
-- Export a `transform` function: `transform(method: string, url: string, responses: unknown[], params?: Record<string, string | number | boolean>, state?: Record<string, unknown>): string | { url: string; body?: string; headers?: Record<string, string> }`.
-- Optionally export `prepareParams(params)` when bootstrap URLs need derived primitive params before `${param.*}` substitution.
+- Export a `transform` function: `transform(method: string, url: string, responses: unknown[], params?: Record<string, string | number | boolean>): string | { url: string; body?: string }`.
 - It reproduces the site's per-request signing/body logic (e.g. HMAC/MD5/CRC32 + encoding) so the regenerated value matches what the recording sent. Derive the algorithm from `sources` (and any `.js` body included there). Return the URL with the signing param appended (or `{ url, body }` when you must build the body).
 - **The verifier re-signs a recorded URL and checks your output reproduces the recorded signing param.** A no-op that returns the URL unchanged will fail.