imprint-mcp 0.4.6 → 0.4.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -4
- package/examples/google-flights/README.md +2 -0
- package/examples/google-flights/_shared/flights_request.ts +10 -4
- package/examples/google-flights/get_flight_booking_details/index.ts +5 -2
- package/examples/google-flights/get_flight_booking_details/parser.ts +8 -0
- package/examples/google-flights/get_flight_booking_details/workflow.json +5 -2
- package/examples/google-flights/get_flight_calendar_prices/index.ts +5 -2
- package/examples/google-flights/get_flight_calendar_prices/parser.ts +8 -4
- package/examples/google-flights/get_flight_calendar_prices/workflow.json +5 -2
- package/examples/google-flights/lookup_airport/index.ts +3 -0
- package/examples/google-flights/lookup_airport/parser.ts +8 -1
- package/examples/google-flights/lookup_airport/workflow.json +3 -0
- package/examples/google-flights/search_flights/index.ts +63 -8
- package/examples/google-flights/search_flights/parser.ts +10 -0
- package/examples/google-flights/search_flights/request-transform.ts +45 -0
- package/examples/google-flights/search_flights/workflow.json +63 -8
- package/package.json +1 -1
- package/prompts/build-planning.md +1 -1
- package/prompts/compile-agent.md +5 -3
- package/prompts/prereq-builder.md +2 -1
- package/src/imprint/backend-ladder.ts +436 -43
- package/src/imprint/cdp-browser-fetch.ts +176 -6
- package/src/imprint/cdp-jar-cache.ts +105 -10
- package/src/imprint/compile-tools.ts +2 -2
- package/src/imprint/mcp-server.ts +152 -65
- package/src/imprint/probe-backends.ts +41 -10
- package/src/imprint/runtime.ts +24 -12
- package/src/imprint/stealth-fetch.ts +71 -0
- package/src/imprint/stealth-token-cache.ts +38 -1
- package/src/imprint/types.ts +45 -0
package/README.md
CHANGED
|
@@ -169,8 +169,8 @@ When an API call gets blocked, Imprint doesn't jump to DOM replay. It escalates
|
|
|
169
169
|
│ + API
|
|
170
170
|
▼
|
|
171
171
|
cdp-replay ~2-35s API calls run inside a live, trusted Chrome —
|
|
172
|
-
│
|
|
173
|
-
│
|
|
172
|
+
│ reuses browser-observed request state and refreshes
|
|
173
|
+
│ anti-bot tokens between protected POSTs
|
|
174
174
|
▼
|
|
175
175
|
stealth-fetch ~1-12s Defeats Akamai, Cloudflare, DataDome
|
|
176
176
|
│
|
|
@@ -178,9 +178,9 @@ When an API call gets blocked, Imprint doesn't jump to DOM replay. It escalates
|
|
|
178
178
|
playbook ~9s Full DOM replay — universal fallback
|
|
179
179
|
```
|
|
180
180
|
|
|
181
|
-
The full order is `fetch → fetch-bootstrap → cdp-replay → stealth-fetch → playbook`; `auto` mode walks it and stops at the first backend that works.
|
|
181
|
+
The full order is `fetch → fetch-bootstrap → cdp-replay → stealth-fetch → playbook`; `auto` mode walks it and stops at the first backend that works. Workflows that declare browser-observed request captures can start at `cdp-replay`, so MCP sessions reuse the same Chrome instead of paying a cold bootstrap on each route/date.
|
|
182
182
|
|
|
183
|
-
For bot-protected sites, `imprint probe-backends <site> --tool <toolName>` writes a `backends.json` preference cache so cron and MCP start from the known-good backend instead of rediscovering blocked rungs. Use `imprint probe-backends <site> --all` to refresh every tool in a multi-tool site; `imprint mcp status` reports stale or invalid backend caches before they quietly fall back to the default ladder. CDP replay records both cold and warm timings when it succeeds: a timeout-safe cold start may rank by its fast warm runtime, but a cold start above the preferred threshold stays behind cold-safe backends in durable cache order.
|
|
183
|
+
For bot-protected sites, `imprint probe-backends <site> --tool <toolName>` writes a `backends.json` preference cache so cron and MCP start from the known-good backend instead of rediscovering blocked rungs. Use `imprint probe-backends <site> --all` to refresh every tool in a multi-tool site; `imprint mcp status` reports stale or invalid backend caches before they quietly fall back to the default ladder. Execution-only workflow changes keep the cached backend order when the backend capability hash still matches. CDP replay records both cold and warm timings when it succeeds: a timeout-safe cold start may rank by its fast warm runtime, but a cold start above the preferred threshold stays behind cold-safe backends in durable cache order.
|
|
184
184
|
|
|
185
185
|
Every recording compiles to *both* `workflow.json` and `playbook.yaml`, so the ladder always has a DOM fallback.
|
|
186
186
|
|
|
@@ -17,6 +17,8 @@ A 4-tool MCP server for Google Flights, compiled from a recording of a normal fl
|
|
|
17
17
|
|
|
18
18
|
- **Protocol**: Google's `/_/FlightsFrontendUi` **`batchexecute`** endpoint returns a nested-array (protobuf-ish) payload. The compiler reverse-engineered the encoding into `_shared/batchexecute.ts` (shared decoder) + per-tool `parser.ts`, and the `f.req` request shape into `_shared/flights_request.ts` + per-tool `request-transform.ts`.
|
|
19
19
|
- **Anti-bot**: the per-page `f.sid` / `bl` tokens are bootstrapped at runtime (`${state.f_sid}` placeholders), and calls run on the **cdp-replay** rung (requests issued inside a live, trusted Chrome) with a **stealth-fetch** fallback.
|
|
20
|
+
- **MCP pacing**: `search_flights` declares `execution.minCallSpacingMs: 2000` because Google Flights can return fast empty result sets when warm CDP searches are fired back-to-back with no breathing room.
|
|
21
|
+
- **Bounded fallback**: these tools declare `execution.skipPlaybookFallback` so MCP calls fail fast after the API/browser-backed rungs are exhausted instead of spending the rest of the agent timeout in an unstructured DOM replay.
|
|
20
22
|
- **Artifacts per tool**: `workflow.json` (API replay), `playbook.yaml` (DOM fallback), `index.ts` (MCP tool), `parser.ts` + `request-transform.ts` (codecs).
|
|
21
23
|
|
|
22
24
|
## Install
|
|
@@ -8,9 +8,11 @@
|
|
|
8
8
|
// with the date range living in the outer wrapper; Shopping/Booking use the full
|
|
9
9
|
// 15-slot leg with DATE at [6]. Verified by decoding seq 97 vs seq 111.
|
|
10
10
|
|
|
11
|
-
// Fresh searches emit wrapper `...,0,0,0,1]` and leg[14]=3
|
|
12
|
-
//
|
|
13
|
-
//
|
|
11
|
+
// Fresh searches emit wrapper `...,0,0,0,1]` and use leg[14]=3 for normal
|
|
12
|
+
// shopping legs. Return leg [14]=1 appears in booking / selected-leg flows, not
|
|
13
|
+
// in the initial search request used by this tool.
|
|
14
|
+
// In-page-refined searches use `...,0,1,0,1]` — a UI freshness flag, not a user
|
|
15
|
+
// param; we always emit the fresh form for shopping.
|
|
14
16
|
// Booking outbound legs use [14]=3, return legs [14]=1 (seq 764/811).
|
|
15
17
|
|
|
16
18
|
function buildLeg(leg: any): any[] {
|
|
@@ -74,7 +76,11 @@ export function transform(
|
|
|
74
76
|
let payload: any;
|
|
75
77
|
|
|
76
78
|
if (rpc === 'GetShoppingResults') {
|
|
77
|
-
|
|
79
|
+
const searchContext =
|
|
80
|
+
typeof p.searchContextToken === 'string' && p.searchContextToken
|
|
81
|
+
? [null, null, null, p.searchContextToken]
|
|
82
|
+
: [];
|
|
83
|
+
payload = [searchContext, sp, 0, 0, 0, 1];
|
|
78
84
|
} else if (rpc === 'GetCalendarPicker') {
|
|
79
85
|
const legs = sp[13];
|
|
80
86
|
if (Array.isArray(legs)) sp[13] = legs.map((l: any) => (Array.isArray(l) ? l.slice(0, 4) : l));
|
|
@@ -94,7 +94,7 @@ const WORKFLOW: Workflow = {
|
|
|
94
94
|
"captures": [
|
|
95
95
|
{
|
|
96
96
|
"name": "f_sid",
|
|
97
|
-
"required":
|
|
97
|
+
"required": true,
|
|
98
98
|
"capability": "browser_bootstrap",
|
|
99
99
|
"source": "html_regex",
|
|
100
100
|
"pattern": "\"FdrFJe\":\"([^\"]+)\"",
|
|
@@ -102,7 +102,7 @@ const WORKFLOW: Workflow = {
|
|
|
102
102
|
},
|
|
103
103
|
{
|
|
104
104
|
"name": "bl",
|
|
105
|
-
"required":
|
|
105
|
+
"required": true,
|
|
106
106
|
"capability": "browser_bootstrap",
|
|
107
107
|
"source": "html_regex",
|
|
108
108
|
"pattern": "\"cfb2h\":\"([^\"]+)\"",
|
|
@@ -112,6 +112,9 @@ const WORKFLOW: Workflow = {
|
|
|
112
112
|
},
|
|
113
113
|
"parserModule": "./parser.ts",
|
|
114
114
|
"requestTransformModule": "./request-transform.ts",
|
|
115
|
+
"execution": {
|
|
116
|
+
"skipPlaybookFallback": true
|
|
117
|
+
},
|
|
115
118
|
"liveVerified": true
|
|
116
119
|
};
|
|
117
120
|
|
|
@@ -142,6 +142,9 @@ export function extract(
|
|
|
142
142
|
let frames: Array<{ rpcid: string | null; payload: any }> = [];
|
|
143
143
|
if (typeof rawResponse === 'string') {
|
|
144
144
|
frames = decodeBatchExecute(rawResponse);
|
|
145
|
+
if (frames.length === 0) {
|
|
146
|
+
throw new Error('Google Flights GetBookingResults response did not contain a batchexecute payload');
|
|
147
|
+
}
|
|
145
148
|
} else if (rawResponse != null) {
|
|
146
149
|
frames = [{ rpcid: null, payload: rawResponse }];
|
|
147
150
|
}
|
|
@@ -170,6 +173,11 @@ export function extract(
|
|
|
170
173
|
|
|
171
174
|
const segments = [...segMap.values()];
|
|
172
175
|
const fareOptions = [...fareMap.values()];
|
|
176
|
+
if (segments.length === 0 && fareOptions.length === 0) {
|
|
177
|
+
throw new Error(
|
|
178
|
+
'Google Flights GetBookingResults payload did not contain recognizable booking details',
|
|
179
|
+
);
|
|
180
|
+
}
|
|
173
181
|
const prices = fareOptions.map((f) => f.priceUSD);
|
|
174
182
|
|
|
175
183
|
return {
|
|
@@ -65,7 +65,7 @@
|
|
|
65
65
|
"name": "f_sid",
|
|
66
66
|
"pattern": "\"FdrFJe\":\"([^\"]+)\"",
|
|
67
67
|
"group": 1,
|
|
68
|
-
"required":
|
|
68
|
+
"required": true,
|
|
69
69
|
"capability": "browser_bootstrap"
|
|
70
70
|
},
|
|
71
71
|
{
|
|
@@ -73,7 +73,7 @@
|
|
|
73
73
|
"name": "bl",
|
|
74
74
|
"pattern": "\"cfb2h\":\"([^\"]+)\"",
|
|
75
75
|
"group": 1,
|
|
76
|
-
"required":
|
|
76
|
+
"required": true,
|
|
77
77
|
"capability": "browser_bootstrap"
|
|
78
78
|
}
|
|
79
79
|
]
|
|
@@ -94,5 +94,8 @@
|
|
|
94
94
|
],
|
|
95
95
|
"requestTransformModule": "./request-transform.ts",
|
|
96
96
|
"parserModule": "./parser.ts",
|
|
97
|
+
"execution": {
|
|
98
|
+
"skipPlaybookFallback": true
|
|
99
|
+
},
|
|
97
100
|
"liveVerified": true
|
|
98
101
|
}
|
|
@@ -74,7 +74,7 @@ const WORKFLOW: Workflow = {
|
|
|
74
74
|
"captures": [
|
|
75
75
|
{
|
|
76
76
|
"name": "f_sid",
|
|
77
|
-
"required":
|
|
77
|
+
"required": true,
|
|
78
78
|
"capability": "browser_bootstrap",
|
|
79
79
|
"source": "html_regex",
|
|
80
80
|
"pattern": "\"FdrFJe\":\"([^\"]+)\"",
|
|
@@ -82,7 +82,7 @@ const WORKFLOW: Workflow = {
|
|
|
82
82
|
},
|
|
83
83
|
{
|
|
84
84
|
"name": "bl",
|
|
85
|
-
"required":
|
|
85
|
+
"required": true,
|
|
86
86
|
"capability": "browser_bootstrap",
|
|
87
87
|
"source": "html_regex",
|
|
88
88
|
"pattern": "\"cfb2h\":\"([^\"]+)\"",
|
|
@@ -92,6 +92,9 @@ const WORKFLOW: Workflow = {
|
|
|
92
92
|
},
|
|
93
93
|
"parserModule": "./parser.ts",
|
|
94
94
|
"requestTransformModule": "./request-transform.ts",
|
|
95
|
+
"execution": {
|
|
96
|
+
"skipPlaybookFallback": true
|
|
97
|
+
},
|
|
95
98
|
"liveVerified": true
|
|
96
99
|
};
|
|
97
100
|
|
|
@@ -54,6 +54,9 @@ export function extract(
|
|
|
54
54
|
): unknown {
|
|
55
55
|
const raw = typeof rawResponse === 'string' ? rawResponse : JSON.stringify(rawResponse ?? '');
|
|
56
56
|
const frames = decodeBatchExecute(raw);
|
|
57
|
+
if (frames.length === 0) {
|
|
58
|
+
throw new Error('Google Flights GetCalendarPicker response did not contain a batchexecute payload');
|
|
59
|
+
}
|
|
57
60
|
|
|
58
61
|
let payload: unknown = null;
|
|
59
62
|
for (const f of frames) {
|
|
@@ -63,13 +66,14 @@ export function extract(
|
|
|
63
66
|
break;
|
|
64
67
|
}
|
|
65
68
|
}
|
|
66
|
-
// If no frame produced entries, still attempt the first frame's payload so an
|
|
67
|
-
// empty (zero-result) response yields an empty calendar rather than throwing.
|
|
68
|
-
if (payload == null && frames.length > 0) payload = frames[0]?.payload ?? null;
|
|
69
|
-
|
|
70
69
|
const entries = collectEntries(payload).sort((a, b) =>
|
|
71
70
|
a.departureDate < b.departureDate ? -1 : a.departureDate > b.departureDate ? 1 : 0,
|
|
72
71
|
);
|
|
72
|
+
if (entries.length === 0) {
|
|
73
|
+
throw new Error(
|
|
74
|
+
'Google Flights GetCalendarPicker payload did not contain recognizable calendar prices',
|
|
75
|
+
);
|
|
76
|
+
}
|
|
73
77
|
|
|
74
78
|
const prices: Record<string, number> = {};
|
|
75
79
|
for (const e of entries) prices[e.departureDate] = e.lowestPriceUSD;
|
|
@@ -44,7 +44,7 @@
|
|
|
44
44
|
"name": "f_sid",
|
|
45
45
|
"pattern": "\"FdrFJe\":\"([^\"]+)\"",
|
|
46
46
|
"group": 1,
|
|
47
|
-
"required":
|
|
47
|
+
"required": true,
|
|
48
48
|
"capability": "browser_bootstrap"
|
|
49
49
|
},
|
|
50
50
|
{
|
|
@@ -52,7 +52,7 @@
|
|
|
52
52
|
"name": "bl",
|
|
53
53
|
"pattern": "\"cfb2h\":\"([^\"]+)\"",
|
|
54
54
|
"group": 1,
|
|
55
|
-
"required":
|
|
55
|
+
"required": true,
|
|
56
56
|
"capability": "browser_bootstrap"
|
|
57
57
|
}
|
|
58
58
|
]
|
|
@@ -74,5 +74,8 @@
|
|
|
74
74
|
],
|
|
75
75
|
"requestTransformModule": "./request-transform.ts",
|
|
76
76
|
"parserModule": "./parser.ts",
|
|
77
|
+
"execution": {
|
|
78
|
+
"skipPlaybookFallback": true
|
|
79
|
+
},
|
|
77
80
|
"liveVerified": true
|
|
78
81
|
}
|
|
@@ -45,8 +45,15 @@ export function extract(
|
|
|
45
45
|
): unknown {
|
|
46
46
|
const raw = typeof rawResponse === 'string' ? rawResponse : JSON.stringify(rawResponse);
|
|
47
47
|
const payload = extractRpcPayload(raw, 'tDoGIe');
|
|
48
|
+
if (payload == null) {
|
|
49
|
+
throw new Error('Google Flights tDoGIe response did not contain a batchexecute payload');
|
|
50
|
+
}
|
|
48
51
|
|
|
49
|
-
|
|
52
|
+
if (!Array.isArray(payload) || !Array.isArray(payload[1])) {
|
|
53
|
+
throw new Error('Google Flights tDoGIe payload did not contain a recognizable match list');
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
const matchesRaw = payload[1];
|
|
50
57
|
const matches = matchesRaw
|
|
51
58
|
.map(parseItem)
|
|
52
59
|
.filter((m): m is AirportMatch => m !== null);
|
|
@@ -120,11 +120,13 @@ const WORKFLOW: Workflow = {
|
|
|
120
120
|
"requests": [
|
|
121
121
|
{
|
|
122
122
|
"method": "POST",
|
|
123
|
-
"url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid
|
|
123
|
+
"url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=${state.reqid}&rt=c",
|
|
124
124
|
"headers": {
|
|
125
125
|
"Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
|
|
126
126
|
"X-Same-Domain": "1",
|
|
127
|
-
"Referer": "https://www.google.com/travel/flights",
|
|
127
|
+
"Referer": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
|
|
128
|
+
"Accept-Language": "en-US,en;q=0.9",
|
|
129
|
+
"X-Goog-BatchExecute-Bgr": "${state.bgr}",
|
|
128
130
|
"x-goog-ext-259736195-jspb": "[\"en-US\",\"US\",\"USD\",2,null,[420],null,null,7,[]]"
|
|
129
131
|
},
|
|
130
132
|
"body": "f.req=${param.origin}|${param.destination}|${param.departure_date}|${param.return_date}|${param.trip_type}|${param.max_stops}|${param.airlines}|${param.max_price}|${param.outbound_times}|${param.return_times}|${param.max_duration}|${param.carry_on_bags}&",
|
|
@@ -133,30 +135,83 @@ const WORKFLOW: Workflow = {
|
|
|
133
135
|
],
|
|
134
136
|
"site": "google-flights",
|
|
135
137
|
"bootstrap": {
|
|
136
|
-
"url": "https://www.google.com/travel/flights",
|
|
138
|
+
"url": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
|
|
137
139
|
"waitUntil": "domcontentloaded",
|
|
138
140
|
"timeoutMs": 30000,
|
|
139
141
|
"captures": [
|
|
140
142
|
{
|
|
141
143
|
"name": "f_sid",
|
|
142
|
-
"required":
|
|
144
|
+
"required": true,
|
|
143
145
|
"capability": "browser_bootstrap",
|
|
144
|
-
"source": "
|
|
145
|
-
"pattern": "
|
|
146
|
+
"source": "request_url_regex",
|
|
147
|
+
"pattern": "[?&]f\\.sid=([^&]+)",
|
|
148
|
+
"method": "POST",
|
|
149
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
150
|
+
"mode": "last",
|
|
146
151
|
"group": 1
|
|
147
152
|
},
|
|
148
153
|
{
|
|
149
154
|
"name": "bl",
|
|
155
|
+
"required": true,
|
|
156
|
+
"capability": "browser_bootstrap",
|
|
157
|
+
"source": "request_url_regex",
|
|
158
|
+
"pattern": "[?&]bl=([^&]+)",
|
|
159
|
+
"method": "POST",
|
|
160
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
161
|
+
"mode": "last",
|
|
162
|
+
"group": 1
|
|
163
|
+
},
|
|
164
|
+
{
|
|
165
|
+
"name": "bgr",
|
|
166
|
+
"required": true,
|
|
167
|
+
"capability": "browser_bootstrap",
|
|
168
|
+
"source": "request_header",
|
|
169
|
+
"header": "X-Goog-BatchExecute-Bgr",
|
|
170
|
+
"method": "POST",
|
|
171
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
172
|
+
"mode": "last"
|
|
173
|
+
},
|
|
174
|
+
{
|
|
175
|
+
"name": "reqid",
|
|
176
|
+
"required": true,
|
|
177
|
+
"capability": "browser_bootstrap",
|
|
178
|
+
"source": "request_url_regex",
|
|
179
|
+
"pattern": "[?&]_reqid=([^&]+)",
|
|
180
|
+
"method": "POST",
|
|
181
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
182
|
+
"mode": "last",
|
|
183
|
+
"group": 1
|
|
184
|
+
},
|
|
185
|
+
{
|
|
186
|
+
"name": "search_context_token",
|
|
150
187
|
"required": false,
|
|
151
188
|
"capability": "browser_bootstrap",
|
|
152
|
-
"source": "
|
|
153
|
-
"pattern": "
|
|
189
|
+
"source": "request_body_regex",
|
|
190
|
+
"pattern": "%5B%5Bnull%2Cnull%2Cnull%2C%5C%22(.+?)%5C%22%5D",
|
|
191
|
+
"method": "POST",
|
|
192
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
193
|
+
"mode": "last",
|
|
194
|
+
"group": 1
|
|
195
|
+
},
|
|
196
|
+
{
|
|
197
|
+
"name": "observed_search_body",
|
|
198
|
+
"required": false,
|
|
199
|
+
"capability": "browser_bootstrap",
|
|
200
|
+
"source": "request_body_regex",
|
|
201
|
+
"pattern": "^(f\\.req=.*)$",
|
|
202
|
+
"method": "POST",
|
|
203
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
204
|
+
"mode": "last",
|
|
154
205
|
"group": 1
|
|
155
206
|
}
|
|
156
207
|
]
|
|
157
208
|
},
|
|
158
209
|
"parserModule": "./parser.ts",
|
|
159
210
|
"requestTransformModule": "./request-transform.ts",
|
|
211
|
+
"execution": {
|
|
212
|
+
"minCallSpacingMs": 2000,
|
|
213
|
+
"skipPlaybookFallback": true
|
|
214
|
+
},
|
|
160
215
|
"liveVerified": true
|
|
161
216
|
};
|
|
162
217
|
|
|
@@ -147,6 +147,11 @@ export function extract(
|
|
|
147
147
|
const frames = decodeBatchExecute(rawResponse);
|
|
148
148
|
payload = frames[0]?.payload;
|
|
149
149
|
}
|
|
150
|
+
if (payload == null) {
|
|
151
|
+
throw new Error(
|
|
152
|
+
'Google Flights GetShoppingResults response did not contain a batchexecute payload',
|
|
153
|
+
);
|
|
154
|
+
}
|
|
150
155
|
} else {
|
|
151
156
|
payload = rawResponse;
|
|
152
157
|
}
|
|
@@ -162,6 +167,11 @@ export function extract(
|
|
|
162
167
|
}
|
|
163
168
|
|
|
164
169
|
const itineraries = [...byToken.values()];
|
|
170
|
+
if (itineraries.length === 0) {
|
|
171
|
+
throw new Error(
|
|
172
|
+
'Google Flights GetShoppingResults payload did not contain recognizable itineraries',
|
|
173
|
+
);
|
|
174
|
+
}
|
|
165
175
|
return {
|
|
166
176
|
count: itineraries.length,
|
|
167
177
|
itineraries,
|
|
@@ -62,13 +62,56 @@ function num(v: unknown): number | undefined {
|
|
|
62
62
|
return Number.isFinite(n) && n > 0 ? n : undefined;
|
|
63
63
|
}
|
|
64
64
|
|
|
65
|
+
function buildBootstrapQuery(params: Params): string {
|
|
66
|
+
const origin = params.origin != null ? String(params.origin) : '';
|
|
67
|
+
const destination = params.destination != null ? String(params.destination) : '';
|
|
68
|
+
const departureDate = params.departure_date != null ? String(params.departure_date) : '';
|
|
69
|
+
const tripType = String(params.trip_type ?? 'round_trip').toLowerCase();
|
|
70
|
+
if (tripType === 'one_way' || tripType === 'oneway' || tripType === '2') {
|
|
71
|
+
return `One way flights from ${origin} to ${destination} on ${departureDate}`;
|
|
72
|
+
}
|
|
73
|
+
if (params.return_date) {
|
|
74
|
+
return `Round trip flights from ${origin} to ${destination} departing ${departureDate} returning ${params.return_date}`;
|
|
75
|
+
}
|
|
76
|
+
return `Flights from ${origin} to ${destination} on ${departureDate}`;
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
export function prepareParams(params?: Params): Params {
|
|
80
|
+
const p: Params = params ?? {};
|
|
81
|
+
return {
|
|
82
|
+
...p,
|
|
83
|
+
bootstrap_query: buildBootstrapQuery(p),
|
|
84
|
+
};
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
function hasNonDefaultFilters(params: Params): boolean {
|
|
88
|
+
if (params.max_stops != null && params.max_stops !== '' && Number(params.max_stops) !== 3) {
|
|
89
|
+
return true;
|
|
90
|
+
}
|
|
91
|
+
return Boolean(
|
|
92
|
+
params.airlines ||
|
|
93
|
+
num(params.max_price) ||
|
|
94
|
+
params.outbound_times ||
|
|
95
|
+
params.return_times ||
|
|
96
|
+
num(params.max_duration) ||
|
|
97
|
+
num(params.carry_on_bags),
|
|
98
|
+
);
|
|
99
|
+
}
|
|
100
|
+
|
|
65
101
|
export function transform(
|
|
66
102
|
method: string,
|
|
67
103
|
url: string,
|
|
68
104
|
responses: Record<string, any>,
|
|
69
105
|
params?: Params,
|
|
106
|
+
state?: Record<string, unknown>,
|
|
70
107
|
): { url: string; body: string } {
|
|
71
108
|
const p: Params = params ?? {};
|
|
109
|
+
const observedSearchBody =
|
|
110
|
+
typeof state?.observed_search_body === 'string' ? state.observed_search_body : undefined;
|
|
111
|
+
if (observedSearchBody && !hasNonDefaultFilters(p)) {
|
|
112
|
+
return { url, body: observedSearchBody };
|
|
113
|
+
}
|
|
114
|
+
|
|
72
115
|
const tripType = mapTripType(p.trip_type);
|
|
73
116
|
const stops = p.max_stops != null && p.max_stops !== '' ? mapStops(p.max_stops) : 0;
|
|
74
117
|
const { alliances, carriers } = parseAirlines(p.airlines);
|
|
@@ -113,6 +156,8 @@ export function transform(
|
|
|
113
156
|
// CONFIG[10] wire form is [1, <carry-on count>]; shared builder emits
|
|
114
157
|
// [carryOn, checked], so map count -> checked slot, constant 1 -> first.
|
|
115
158
|
bags: carryOn != null ? { carryOn: 1, checked: carryOn } : undefined,
|
|
159
|
+
searchContextToken:
|
|
160
|
+
typeof state?.search_context_token === 'string' ? state.search_context_token : undefined,
|
|
116
161
|
};
|
|
117
162
|
|
|
118
163
|
return sharedTransform(method, url, responses, mapped);
|
|
@@ -101,23 +101,72 @@
|
|
|
101
101
|
}
|
|
102
102
|
],
|
|
103
103
|
"bootstrap": {
|
|
104
|
-
"url": "https://www.google.com/travel/flights",
|
|
104
|
+
"url": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
|
|
105
105
|
"waitUntil": "domcontentloaded",
|
|
106
106
|
"timeoutMs": 30000,
|
|
107
107
|
"captures": [
|
|
108
108
|
{
|
|
109
|
-
"source": "
|
|
109
|
+
"source": "request_url_regex",
|
|
110
110
|
"name": "f_sid",
|
|
111
|
-
"pattern": "
|
|
111
|
+
"pattern": "[?&]f\\.sid=([^&]+)",
|
|
112
112
|
"group": 1,
|
|
113
|
-
"
|
|
113
|
+
"method": "POST",
|
|
114
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
115
|
+
"mode": "last",
|
|
116
|
+
"required": true,
|
|
114
117
|
"capability": "browser_bootstrap"
|
|
115
118
|
},
|
|
116
119
|
{
|
|
117
|
-
"source": "
|
|
120
|
+
"source": "request_url_regex",
|
|
118
121
|
"name": "bl",
|
|
119
|
-
"pattern": "
|
|
122
|
+
"pattern": "[?&]bl=([^&]+)",
|
|
123
|
+
"group": 1,
|
|
124
|
+
"method": "POST",
|
|
125
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
126
|
+
"mode": "last",
|
|
127
|
+
"required": true,
|
|
128
|
+
"capability": "browser_bootstrap"
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"source": "request_header",
|
|
132
|
+
"name": "bgr",
|
|
133
|
+
"header": "X-Goog-BatchExecute-Bgr",
|
|
134
|
+
"method": "POST",
|
|
135
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
136
|
+
"mode": "last",
|
|
137
|
+
"required": true,
|
|
138
|
+
"capability": "browser_bootstrap"
|
|
139
|
+
},
|
|
140
|
+
{
|
|
141
|
+
"source": "request_url_regex",
|
|
142
|
+
"name": "reqid",
|
|
143
|
+
"pattern": "[?&]_reqid=([^&]+)",
|
|
144
|
+
"group": 1,
|
|
145
|
+
"method": "POST",
|
|
146
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
147
|
+
"mode": "last",
|
|
148
|
+
"required": true,
|
|
149
|
+
"capability": "browser_bootstrap"
|
|
150
|
+
},
|
|
151
|
+
{
|
|
152
|
+
"source": "request_body_regex",
|
|
153
|
+
"name": "search_context_token",
|
|
154
|
+
"pattern": "%5B%5Bnull%2Cnull%2Cnull%2C%5C%22(.+?)%5C%22%5D",
|
|
120
155
|
"group": 1,
|
|
156
|
+
"method": "POST",
|
|
157
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
158
|
+
"mode": "last",
|
|
159
|
+
"required": false,
|
|
160
|
+
"capability": "browser_bootstrap"
|
|
161
|
+
},
|
|
162
|
+
{
|
|
163
|
+
"source": "request_body_regex",
|
|
164
|
+
"name": "observed_search_body",
|
|
165
|
+
"pattern": "^(f\\.req=.*)$",
|
|
166
|
+
"group": 1,
|
|
167
|
+
"method": "POST",
|
|
168
|
+
"urlPattern": "FlightsFrontendService/GetShoppingResults",
|
|
169
|
+
"mode": "last",
|
|
121
170
|
"required": false,
|
|
122
171
|
"capability": "browser_bootstrap"
|
|
123
172
|
}
|
|
@@ -126,11 +175,13 @@
|
|
|
126
175
|
"requests": [
|
|
127
176
|
{
|
|
128
177
|
"method": "POST",
|
|
129
|
-
"url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid
|
|
178
|
+
"url": "https://www.google.com/_/FlightsFrontendUi/data/travel.frontend.flights.FlightsFrontendService/GetShoppingResults?f.sid=${state.f_sid}&bl=${state.bl}&hl=en-US&soc-app=162&soc-platform=1&soc-device=1&_reqid=${state.reqid}&rt=c",
|
|
130
179
|
"headers": {
|
|
131
180
|
"Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
|
|
132
181
|
"X-Same-Domain": "1",
|
|
133
|
-
"Referer": "https://www.google.com/travel/flights",
|
|
182
|
+
"Referer": "https://www.google.com/travel/flights?q=${param.bootstrap_query}&curr=USD",
|
|
183
|
+
"Accept-Language": "en-US,en;q=0.9",
|
|
184
|
+
"X-Goog-BatchExecute-Bgr": "${state.bgr}",
|
|
134
185
|
"x-goog-ext-259736195-jspb": "[\"en-US\",\"US\",\"USD\",2,null,[420],null,null,7,[]]"
|
|
135
186
|
},
|
|
136
187
|
"body": "f.req=${param.origin}|${param.destination}|${param.departure_date}|${param.return_date}|${param.trip_type}|${param.max_stops}|${param.airlines}|${param.max_price}|${param.outbound_times}|${param.return_times}|${param.max_duration}|${param.carry_on_bags}&",
|
|
@@ -139,5 +190,9 @@
|
|
|
139
190
|
],
|
|
140
191
|
"requestTransformModule": "./request-transform.ts",
|
|
141
192
|
"parserModule": "./parser.ts",
|
|
193
|
+
"execution": {
|
|
194
|
+
"minCallSpacingMs": 2000,
|
|
195
|
+
"skipPlaybookFallback": true
|
|
196
|
+
},
|
|
142
197
|
"liveVerified": true
|
|
143
198
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "imprint-mcp",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.7",
|
|
4
4
|
"description": "Teach an AI agent how to use any website. Once. Records a real browser session + narration; generates a deterministic MCP tool plus a DOM-replay playbook fallback.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"exports": {
|
|
@@ -59,7 +59,7 @@ You receive:
|
|
|
59
59
|
|
|
60
60
|
1. **Emit exactly one `perTool` entry per `selectedTools` entry**, using the same `toolName`. Do not invent or drop tools.
|
|
61
61
|
2. **Only hoist a shared module when ≥2 selected tools genuinely share it.** Single-use logic stays inside that tool's own parser.ts / request-transform.ts — do NOT create a `_shared/` module for it.
|
|
62
|
-
3. **`request-transform`** — URL signing
|
|
62
|
+
3. **`request-transform`** — URL signing, body construction, or bootstrap-param preparation shared across tools. Wire-up: the consuming tool sets `requestTransformModule: "../_shared/<name>.ts"`. Ground it in `ephemeralValues` (browser_minted, high-entropy query param) and `sourceSeqs`. The exported `transform(method, url, responses, params?, state?)` returns the signed URL (or `{ url, body?, headers? }`); an optional `prepareParams(params)` can return primitive params used before bootstrap URL substitution.
|
|
63
63
|
4. **`parser-helper`** — a decoder/normalizer ≥2 tools' parsers call (e.g. a shared JSPB walker, a shared field mapper). The consuming tool's parser.ts does `import { ... } from '../_shared/<name>.ts'`. Ground it in a captured response body (`sourceSeqs`).
|
|
64
64
|
5. **`types`** — shared TypeScript interfaces used by ≥2 parsers. Type-only; no runtime behavior.
|
|
65
65
|
6. **Auth is NEVER a shared module.** Login is request data, and the runtime cannot run a shared sub-workflow. Put the exact recipe in each tool's `authRecipe` (login seqs, credential names, captures with `${state.X}` wiring) and set `required: false` with empty arrays when a tool needs no login. Every authed tool replicates the same recipe inline.
|