@mrscraper/mcp 0.2.3 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +138 -80
- package/dist/api.d.ts +7 -17
- package/dist/api.d.ts.map +1 -1
- package/dist/api.js +32 -27
- package/dist/api.js.map +1 -1
- package/dist/app.d.ts.map +1 -1
- package/dist/app.js +16 -5
- package/dist/app.js.map +1 -1
- package/dist/config.d.ts +4 -3
- package/dist/config.d.ts.map +1 -1
- package/dist/config.js +5 -4
- package/dist/config.js.map +1 -1
- package/dist/http.d.ts.map +1 -1
- package/dist/http.js +4 -1
- package/dist/http.js.map +1 -1
- package/dist/runtime.d.ts.map +1 -1
- package/dist/runtime.js +2 -2
- package/dist/runtime.js.map +1 -1
- package/dist/server.d.ts +2 -0
- package/dist/server.d.ts.map +1 -1
- package/dist/server.js +2 -2
- package/dist/server.js.map +1 -1
- package/dist/tools.d.ts +26 -11
- package/dist/tools.d.ts.map +1 -1
- package/dist/tools.js +61 -41
- package/dist/tools.js.map +1 -1
- package/dist/widgets/bundles.generated.js +2 -2
- package/dist/widgets/bundles.generated.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
MrScraper MCP is the official [MrScraper](https://mrscraper.com) server for the
|
|
4
4
|
[Model Context Protocol](https://modelcontextprotocol.io). It connects AI
|
|
5
|
-
assistants to page content and web search through two
|
|
5
|
+
assistants to page content and web search through two tools,
|
|
6
6
|
plus an account check. Retrieve HTML, Markdown, or screenshots with
|
|
7
7
|
`scraperAPI`, discover pages with `webSearch`, and check your plan and token
|
|
8
8
|
usage with `status`.
|
|
@@ -31,6 +31,10 @@ is no key to copy. Self-hosted and stdio setups use a MrScraper API key from
|
|
|
31
31
|
[app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens), which the
|
|
32
32
|
hosted server also still accepts.
|
|
33
33
|
|
|
34
|
+
ChatGPT and Codex get a `scraperAPI` with fewer options, on `/mcp` and on
|
|
35
|
+
`https://mcp.mrscraper.com/mcp/chatgpt` (see
|
|
36
|
+
[ChatGPT and Codex endpoint](#chatgpt-and-codex-endpoint)).
|
|
37
|
+
|
|
34
38
|
## Connect to the hosted server
|
|
35
39
|
|
|
36
40
|
Point your MCP client at `https://mcp.mrscraper.com/mcp` with no credential.
|
|
@@ -70,7 +74,7 @@ the address will be refused.
|
|
|
70
74
|
A tool called without its scope is refused with `403 insufficient_scope`, and
|
|
71
75
|
your client will offer to grant the missing permission.
|
|
72
76
|
|
|
73
|
-
`scraperAPI` submits a scraping job to the backend with `
|
|
77
|
+
`scraperAPI` submits a scraping job to the backend with `GET /`, which requires
|
|
74
78
|
`scrape:write`. Connections authorized before this scope was advertised need to
|
|
75
79
|
authorize again and grant it; `scrape:read` alone does not grant write access.
|
|
76
80
|
|
|
@@ -137,7 +141,7 @@ Connect MrScraper MCP to this agent. Detect the current MCP client and configure
|
|
|
137
141
|
| Tool | Choose it when |
|
|
138
142
|
| ------------ | ---------------------------------------------------------------------------- |
|
|
139
143
|
| `scraperAPI` | You have a page URL and need HTML, Markdown, a screenshot, or a combination. |
|
|
140
|
-
| `webSearch` | You have a search query
|
|
144
|
+
| `webSearch` | You have a search query and need ranked results with links. |
|
|
141
145
|
| `status` | You need the account's plan, token usage, or remaining tokens. |
|
|
142
146
|
|
|
143
147
|
A discovery workflow is `webSearch` → `scraperAPI`. Both web tools advertise
|
|
@@ -150,27 +154,57 @@ connection; no tool takes an API token argument.
|
|
|
150
154
|
Only `scraperAPI`, `webSearch`, and `status` are registered. The other old tool
|
|
151
155
|
names are removed.
|
|
152
156
|
|
|
153
|
-
| Previous call | Replacement
|
|
154
|
-
| -------------------------------------- |
|
|
155
|
-
| `fetch` | `scraperAPI`; uses
|
|
156
|
-
| `serp` | `webSearch`; rename `query_or_url` to `url` and `client_timeout` to `timeout`.
|
|
157
|
-
| `status` | `status` for account usage; the `domain`, `from`, `to`, `action`, and `api_token_name` inputs are gone.
|
|
158
|
-
| `scrape`, `rerun`, `results`, `result` | Removed from this MCP server.
|
|
157
|
+
| Previous call | Replacement |
|
|
158
|
+
| -------------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
|
159
|
+
| `fetch` | `scraperAPI`; uses GET, SDK options, and automatic browser fallback by default. |
|
|
160
|
+
| `serp` | `webSearch`; rename `query_or_url` to `url` and `client_timeout` to `timeout`. |
|
|
161
|
+
| `status` | `status` for account usage; the `domain`, `from`, `to`, `action`, and `api_token_name` inputs are gone. |
|
|
162
|
+
| `scrape`, `rerun`, `results`, `result` | Removed from this MCP server. |
|
|
159
163
|
|
|
160
164
|
`super_mode` becomes `super`. `home_page` now maps to `warmStart`, and
|
|
161
165
|
`token_cap=0` is accepted. Restart the server and reconnect the MCP client after
|
|
162
166
|
updating so it refreshes `tools/list`.
|
|
163
167
|
|
|
168
|
+
### Changes in 0.4.2
|
|
169
|
+
|
|
170
|
+
`scraperAPI` now loads pages with `GET`: `url`, `maxRetries`, and `tokenCap`
|
|
171
|
+
moved from the JSON body to the query string, and the request has no body. The
|
|
172
|
+
API replays a request body to the target page, so sites used to receive a POST
|
|
173
|
+
carrying that body, and many refused it. With `html: false`, HTML that the API
|
|
174
|
+
still returns is dropped. `geo_code` no longer defaults to `"US"`: when it is
|
|
175
|
+
omitted, the Scraper API chooses the country.
|
|
176
|
+
|
|
177
|
+
### Changes in 0.4.1
|
|
178
|
+
|
|
179
|
+
On `/mcp`, ChatGPT and Codex sign-ins now get the same `scraperAPI` as
|
|
180
|
+
`/mcp/chatgpt`. OpenAI's plugin directory doesn't allow an existing plugin to
|
|
181
|
+
move to another MCP URL, so the MrScraper plugin keeps using `/mcp`. Other
|
|
182
|
+
clients and API keys still get the full set of options.
|
|
183
|
+
|
|
184
|
+
### Changes in 0.4.0
|
|
185
|
+
|
|
186
|
+
The HTTP server adds a second endpoint, `/mcp/chatgpt`, for the MrScraper
|
|
187
|
+
plugin in ChatGPT and Codex. It is described under
|
|
188
|
+
[ChatGPT and Codex endpoint](#chatgpt-and-codex-endpoint). `/mcp` is unchanged.
|
|
189
|
+
|
|
190
|
+
### Changes in 0.3.0
|
|
191
|
+
|
|
192
|
+
`webSearch` now calls MrScraper's web search action on the Scraper API host
|
|
193
|
+
and returns a ranked list of results. It gained `num_results`, and `region`
|
|
194
|
+
must be a two-letter country code. The 0.2 options `raw`, `format`,
|
|
195
|
+
`language`, `page`, and `render_js` are no longer supported: calls that still
|
|
196
|
+
send them succeed, and the options are ignored. `MRSCRAPER_SYNC_BASE_URL` is
|
|
197
|
+
gone; `MRSCRAPER_FETCH_BASE_URL` now covers both web tools.
|
|
198
|
+
|
|
164
199
|
## Tool reference
|
|
165
200
|
|
|
166
|
-
The options and defaults of `scraperAPI`
|
|
167
|
-
|
|
168
|
-
argument names use the SDK's snake case.
|
|
201
|
+
The options and defaults of `scraperAPI` follow the Python SDK's `scraper_api`
|
|
202
|
+
method. MCP tool names use camel case; their argument names use snake case.
|
|
169
203
|
|
|
170
204
|
### `scraperAPI`
|
|
171
205
|
|
|
172
206
|
```text
|
|
173
|
-
|
|
207
|
+
GET https://api.mrscraper.com/
|
|
174
208
|
```
|
|
175
209
|
|
|
176
210
|
Start with a known URL:
|
|
@@ -201,18 +235,20 @@ and fall back to browser loading. `true` forces browser loading and `false`
|
|
|
201
235
|
disables it. A screenshot always forces browser loading, even when
|
|
202
236
|
`browser_rendering=false`. `super` selects real-device routing independently.
|
|
203
237
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
238
|
+
Every option, including `url`, `maxRetries`, and `tokenCap`, is a query
|
|
239
|
+
parameter, and the request has no body, as in the Python SDK's `scraper_api`.
|
|
240
|
+
The API replays a request body to the target page, so the request must not
|
|
241
|
+
carry one. For API keys, the request also includes the SDK-compatible `token`
|
|
242
|
+
query parameter and `Authorization`/`x-api-token` headers. OAuth JWTs are sent
|
|
243
|
+
only through `Authorization`.
|
|
208
244
|
|
|
209
245
|
| Input | Default | API mapping / purpose |
|
|
210
246
|
| ---------------------- | -------- | --------------------------------------------------------------------------------------- |
|
|
211
|
-
| `url` | required |
|
|
212
|
-
| `html` | `true` | Query `html`; request HTML.
|
|
247
|
+
| `url` | required | Query `url`; absolute HTTP(S) page URL. |
|
|
248
|
+
| `html` | `true` | Query `html`; request HTML. With `false`, HTML the API still returns is dropped. |
|
|
213
249
|
| `markdown` | `false` | Query `markdown`; request Markdown. |
|
|
214
250
|
| `timeout` | `120` | Query `timeout`; integer from 1 to 300 seconds. Transport allows 30 additional seconds. |
|
|
215
|
-
| `geo_code` |
|
|
251
|
+
| `geo_code` | omitted | Query `geoCode`; proxy country code. Omitted, the API chooses the country (US). |
|
|
216
252
|
| `proxy_country` | omitted | Query `proxyCountry`; compatibility country field. |
|
|
217
253
|
| `browser_rendering` | omitted | Query `browserRendering`; true/false, or omit/null for automatic fallback. |
|
|
218
254
|
| `super` | `false` | Query `super=true` when enabled. |
|
|
@@ -225,55 +261,36 @@ headers. OAuth JWTs are sent only through `Authorization`.
|
|
|
225
261
|
| `bypass_proxy` | `false` | Query `bypassProxy=true`; proxy-bypass/resource-blocking option. |
|
|
226
262
|
| `cookie_jar` | omitted | Query `cookieJar`; cookie configuration string. |
|
|
227
263
|
| `headers` | omitted | Query `headers`; target request headers as an object or JSON string. |
|
|
228
|
-
| `max_retries` | `3` |
|
|
229
|
-
| `token_cap` | omitted |
|
|
264
|
+
| `max_retries` | `3` | Query `maxRetries`; nonnegative integer, 0 disables retries. |
|
|
265
|
+
| `token_cap` | omitted | Query `tokenCap`; nonnegative integer retry-token budget. |
|
|
230
266
|
| `stream` | omitted | Query `stream`; response is buffered into the normal MCP envelope. |
|
|
231
267
|
| `discard_token_secret` | `""` | Optional `x-discard-token-secret` header. |
|
|
232
268
|
|
|
233
269
|
### `webSearch`
|
|
234
270
|
|
|
235
271
|
```text
|
|
236
|
-
POST https://
|
|
272
|
+
POST https://api.mrscraper.com
|
|
237
273
|
```
|
|
238
274
|
|
|
239
|
-
|
|
275
|
+
The JSON body carries `action: "websearch"`, the query, and the optional
|
|
276
|
+
country and result count. Ask for 20 results from the United Kingdom:
|
|
240
277
|
|
|
241
278
|
```json
|
|
242
279
|
{
|
|
243
|
-
"url": "
|
|
244
|
-
"
|
|
245
|
-
"
|
|
246
|
-
"language": "id",
|
|
247
|
-
"page": 1,
|
|
248
|
-
"format": "json"
|
|
280
|
+
"url": "running shoes",
|
|
281
|
+
"region": "gb",
|
|
282
|
+
"num_results": 20
|
|
249
283
|
}
|
|
250
284
|
```
|
|
251
285
|
|
|
252
|
-
|
|
253
|
-
of `format`. Set `raw=false` and `format="json"` for parsed results.
|
|
254
|
-
|
|
255
|
-
A search results URL supplies the query, locale, and page:
|
|
256
|
-
|
|
257
|
-
```json
|
|
258
|
-
{
|
|
259
|
-
"url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
|
|
260
|
-
"raw": false
|
|
261
|
-
}
|
|
262
|
-
```
|
|
286
|
+
A search results URL also works; its `q` parameter becomes the query.
|
|
263
287
|
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
|
268
|
-
|
|
|
269
|
-
| `
|
|
270
|
-
| `raw` | `true` | Forces body `format="html"`; false allows `format`. |
|
|
271
|
-
| `format` | `"json"` | Body `format`, either `json` or `html`, when raw is false. |
|
|
272
|
-
| `region` | URL value or omitted | Body `region`; overrides `gl`. |
|
|
273
|
-
| `language` | URL value or omitted | Body `language`; overrides `hl`. |
|
|
274
|
-
| `page` | URL value or omitted | Body `page`; positive one-based integer, overrides `start`. |
|
|
275
|
-
| `render_js` | `false` | Body `renderJs`; render dynamic search features. |
|
|
276
|
-
| `timeout` | `600` | Local HTTP deadline in seconds; omitted from the body. |
|
|
288
|
+
| Input | Default | Request mapping / purpose |
|
|
289
|
+
| ------------- | -------- | --------------------------------------------------------------------------- |
|
|
290
|
+
| `url` | required | Body `query`; search text, or a search results URL whose `q` value is used. |
|
|
291
|
+
| `region` | `us` | Body `country`; two-letter country code. |
|
|
292
|
+
| `num_results` | `10` | Body `numResults`; 10 to 100. The search may return fewer. |
|
|
293
|
+
| `timeout` | `600` | Local HTTP deadline in seconds; omitted from the body. |
|
|
277
294
|
|
|
278
295
|
### `status`
|
|
279
296
|
|
|
@@ -346,8 +363,29 @@ status:
|
|
|
346
363
|
}
|
|
347
364
|
```
|
|
348
365
|
|
|
349
|
-
A
|
|
350
|
-
`
|
|
366
|
+
A `webSearch` response lists the results in `data.data`, each with `rank`,
|
|
367
|
+
`title`, `url`, `description`, and `source`:
|
|
368
|
+
|
|
369
|
+
```json
|
|
370
|
+
{
|
|
371
|
+
"status_code": 200,
|
|
372
|
+
"data": {
|
|
373
|
+
"success": true,
|
|
374
|
+
"message": "Search completed",
|
|
375
|
+
"data": [
|
|
376
|
+
{
|
|
377
|
+
"rank": 1,
|
|
378
|
+
"title": "Web Scraping API | MrScraper",
|
|
379
|
+
"url": "https://mrscraper.com/",
|
|
380
|
+
"description": "Extract structured data from any website with AI.",
|
|
381
|
+
"source": "MrScraper"
|
|
382
|
+
}
|
|
383
|
+
],
|
|
384
|
+
"tokenUsage": 1
|
|
385
|
+
},
|
|
386
|
+
"headers": { "content-type": "application/json; charset=utf-8" }
|
|
387
|
+
}
|
|
388
|
+
```
|
|
351
389
|
|
|
352
390
|
Failures include `error` and set MCP `isError=true`. Network failures have
|
|
353
391
|
`status_code=null`. Credential-bearing response headers and JSON metadata
|
|
@@ -399,8 +437,8 @@ Run these commands individually from the repository directory:
|
|
|
399
437
|
npm run test:manual -- status
|
|
400
438
|
```
|
|
401
439
|
|
|
402
|
-
The webSearch sample
|
|
403
|
-
|
|
440
|
+
The webSearch sample asks for US results (`region` `us`). Each command prints
|
|
441
|
+
`Tools: scraperAPI, webSearch, status`
|
|
404
442
|
and `PASS <tool>` on success. Exit codes are 0 for success, 1 for a tool or
|
|
405
443
|
connection failure, and 2 for invalid command arguments.
|
|
406
444
|
|
|
@@ -411,7 +449,7 @@ npm run test:manual -- scraperAPI --args '{"url":"https://example.com","markdown
|
|
|
411
449
|
```
|
|
412
450
|
|
|
413
451
|
```sh
|
|
414
|
-
npm run test:manual -- webSearch --args '{"url":"web scraping","
|
|
452
|
+
npm run test:manual -- webSearch --args '{"url":"web scraping","region":"id","num_results":20}'
|
|
415
453
|
```
|
|
416
454
|
|
|
417
455
|
To test an already running HTTP MCP server, append
|
|
@@ -501,10 +539,31 @@ Browser-origin requests are accepted from trusted local origins, from
|
|
|
501
539
|
without an `Origin` header. The discovery documents are served ahead of origin
|
|
502
540
|
checks, since any client must be able to read them before it has credentials.
|
|
503
541
|
|
|
542
|
+
### ChatGPT and Codex endpoint
|
|
543
|
+
|
|
544
|
+
ChatGPT and Codex get a `scraperAPI` that accepts only `url`, `html`,
|
|
545
|
+
`markdown`, `screenshot`, `browser_rendering`, `wait_for_selector`, `timeout`,
|
|
546
|
+
and `geo_code`. Proxy, cookie, header, retry, real-device, and warm-start
|
|
547
|
+
options aren't offered; arguments that aren't in its schema are dropped, and
|
|
548
|
+
the omitted options keep their defaults. `webSearch` and `status` are the same
|
|
549
|
+
as for other clients.
|
|
550
|
+
|
|
551
|
+
The server picks this tool set in two cases:
|
|
552
|
+
|
|
553
|
+
- On `/mcp`, when the access token's `client_id` starts with
|
|
554
|
+
`https://chatgpt.com/oauth/`. ChatGPT and Codex sign in with Client ID
|
|
555
|
+
Metadata Documents at that address, so the MrScraper plugin in ChatGPT and
|
|
556
|
+
Codex, and OpenAI's tool scan, use this tool set. Other OAuth clients and API
|
|
557
|
+
keys keep the full set.
|
|
558
|
+
- On `/mcp/chatgpt`, for every caller. It uses the same sign-in as `/mcp`:
|
|
559
|
+
unauthenticated requests point to the `/mcp` protected resource metadata, and
|
|
560
|
+
an access token issued for `https://mcp.mrscraper.com/mcp` works on both
|
|
561
|
+
endpoints.
|
|
562
|
+
|
|
504
563
|
## Interactive widgets
|
|
505
564
|
|
|
506
|
-
`webSearch` carries a self-contained search results widget
|
|
507
|
-
results
|
|
565
|
+
`webSearch` carries a self-contained search results widget that lists the
|
|
566
|
+
results with their titles, links, and descriptions. `status`
|
|
508
567
|
carries an account widget with the plan, token usage, and a usage meter. Hosts
|
|
509
568
|
without MCP Apps support display the normal JSON tool response.
|
|
510
569
|
|
|
@@ -526,24 +585,23 @@ deployment and keep bearer authentication enabled.
|
|
|
526
585
|
|
|
527
586
|
## Environment variables
|
|
528
587
|
|
|
529
|
-
| Variable | Default
|
|
530
|
-
| -------------------------------- |
|
|
531
|
-
| `TRANSPORT` | `stdio`
|
|
532
|
-
| `HOST` | `127.0.0.1`
|
|
533
|
-
| `PORT` | `8000`
|
|
534
|
-
| `MRSCRAPER_API_KEY` | -
|
|
535
|
-
| `MRSCRAPER_API_TOKEN` | -
|
|
536
|
-
| `MRSCRAPER_HTTP_AUTH` | `1`
|
|
537
|
-
| `MRSCRAPER_OAUTH` | `1`
|
|
538
|
-
| `MRSCRAPER_MCP_PUBLIC_URL` | `https://mcp.mrscraper.com`
|
|
539
|
-
| `MRSCRAPER_OAUTH_ISSUER` | `https://api.app.mrscraper.com`
|
|
540
|
-
| `MRSCRAPER_OAUTH_JWKS_URL` | `<issuer>/.well-known/jwks.json`
|
|
541
|
-
| `MRSCRAPER_ALLOWED_ORIGINS` | -
|
|
542
|
-
| `MRSCRAPER_API_BASE_URL` | MrScraper platform API
|
|
543
|
-
| `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker
|
|
544
|
-
| `
|
|
545
|
-
| `
|
|
546
|
-
| `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
|
|
588
|
+
| Variable | Default | Purpose |
|
|
589
|
+
| -------------------------------- | -------------------------------- | ---------------------------------------------------------------- |
|
|
590
|
+
| `TRANSPORT` | `stdio` | Selects `stdio` or `http`. |
|
|
591
|
+
| `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. |
|
|
592
|
+
| `PORT` | `8000` | HTTP listen port. |
|
|
593
|
+
| `MRSCRAPER_API_KEY` | - | Primary stdio credential. |
|
|
594
|
+
| `MRSCRAPER_API_TOKEN` | - | Alternative stdio credential (also used by the Python SDK). |
|
|
595
|
+
| `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. |
|
|
596
|
+
| `MRSCRAPER_OAUTH` | `1` | Accepts OAuth 2.1 access tokens alongside API keys. |
|
|
597
|
+
| `MRSCRAPER_MCP_PUBLIC_URL` | `https://mcp.mrscraper.com` | Public origin; `<origin>/mcp` is the required token audience. |
|
|
598
|
+
| `MRSCRAPER_OAUTH_ISSUER` | `https://api.app.mrscraper.com` | Authorization server issuer. |
|
|
599
|
+
| `MRSCRAPER_OAUTH_JWKS_URL` | `<issuer>/.well-known/jwks.json` | Key set used to verify access tokens. |
|
|
600
|
+
| `MRSCRAPER_ALLOWED_ORIGINS` | - | Comma-separated browser origins allowed to call the HTTP server. |
|
|
601
|
+
| `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. |
|
|
602
|
+
| `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Scraper API and web search endpoint override. |
|
|
603
|
+
| `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. |
|
|
604
|
+
| `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
|
|
547
605
|
|
|
548
606
|
## Troubleshooting
|
|
549
607
|
|
|
@@ -560,7 +618,7 @@ deployment and keep bearer authentication enabled.
|
|
|
560
618
|
- **403 Forbidden Origin:** Add the exact browser origin to
|
|
561
619
|
`MRSCRAPER_ALLOWED_ORIGINS`.
|
|
562
620
|
- **Tool input error:** Compare the call with the tool's parameter table and
|
|
563
|
-
|
|
621
|
+
its defaults.
|
|
564
622
|
- **Upstream timeout:** Increase `scraperAPI.timeout` (up to 300 seconds) or `webSearch.timeout`
|
|
565
623
|
when the target requires a longer request window.
|
|
566
624
|
- **New tools are missing:** Start a new client session and inspect
|
package/dist/api.d.ts
CHANGED
|
@@ -1,5 +1,4 @@
|
|
|
1
1
|
import { type ApiResponse } from "./http.js";
|
|
2
|
-
export type SearchFormat = "json" | "html";
|
|
3
2
|
interface FetchDependency {
|
|
4
3
|
fetchFn?: typeof fetch;
|
|
5
4
|
}
|
|
@@ -10,7 +9,7 @@ export interface ScraperApiOptions extends FetchDependency {
|
|
|
10
9
|
html?: boolean;
|
|
11
10
|
markdown?: boolean;
|
|
12
11
|
timeout?: number;
|
|
13
|
-
geoCode?: string;
|
|
12
|
+
geoCode?: string | null;
|
|
14
13
|
proxyCountry?: string | null;
|
|
15
14
|
blockResources?: boolean | null;
|
|
16
15
|
browserRendering?: boolean | null;
|
|
@@ -29,25 +28,16 @@ export interface ScraperApiOptions extends FetchDependency {
|
|
|
29
28
|
discardTokenSecret?: string;
|
|
30
29
|
}
|
|
31
30
|
export declare function scraperApi({ token, url, html, markdown, timeout, geoCode, proxyCountry, blockResources, browserRendering, super: superMode, useProxy, proxy, waitForSelector, homePage, screenshot, bypassProxy, cookieJar, headers, maxRetries, tokenCap, stream, discardTokenSecret, fetchFn, }: ScraperApiOptions): Promise<ApiResponse>;
|
|
32
|
-
export interface NormalizedSearchInput {
|
|
33
|
-
query: string;
|
|
34
|
-
region: string | null;
|
|
35
|
-
language: string | null;
|
|
36
|
-
page: number | null;
|
|
37
|
-
}
|
|
38
|
-
export declare function normalizeSearchInput(input: string): NormalizedSearchInput;
|
|
39
31
|
export interface WebSearchOptions extends FetchDependency {
|
|
40
32
|
token: string;
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
language?: string | null;
|
|
45
|
-
page?: number | null;
|
|
46
|
-
format?: SearchFormat;
|
|
47
|
-
renderJs?: boolean;
|
|
33
|
+
query: string;
|
|
34
|
+
country?: string | null;
|
|
35
|
+
numResults?: number | null;
|
|
48
36
|
timeout?: number;
|
|
49
37
|
}
|
|
50
|
-
|
|
38
|
+
/** Returns the query text, or the `q` parameter when given a search results URL. */
|
|
39
|
+
export declare function searchQuery(input: string): string;
|
|
40
|
+
export declare function webSearchApi({ token, query, country, numResults, timeout, fetchFn, }: WebSearchOptions): Promise<ApiResponse>;
|
|
51
41
|
export declare function getSubscriptionAccountApi(token: string, { fetchFn }?: FetchDependency): Promise<ApiResponse>;
|
|
52
42
|
export {};
|
|
53
43
|
//# sourceMappingURL=api.d.ts.map
|
package/dist/api.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"api.d.ts","sourceRoot":"","sources":["../src/api.ts"],"names":[],"mappings":"AAEA,OAAO,EAAW,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AAGtD,
|
|
1
|
+
{"version":3,"file":"api.d.ts","sourceRoot":"","sources":["../src/api.ts"],"names":[],"mappings":"AAEA,OAAO,EAAW,KAAK,WAAW,EAAE,MAAM,WAAW,CAAC;AAGtD,UAAU,eAAe;IACvB,OAAO,CAAC,EAAE,OAAO,KAAK,CAAC;CACxB;AAED,wBAAgB,cAAc,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAUpE;AAaD,MAAM,WAAW,iBAAkB,SAAQ,eAAe;IACxD,KAAK,EAAE,MAAM,CAAC;IACd,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,YAAY,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC7B,cAAc,CAAC,EAAE,OAAO,GAAG,IAAI,CAAC;IAChC,gBAAgB,CAAC,EAAE,OAAO,GAAG,IAAI,CAAC;IAClC,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,eAAe,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAChC,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,SAAS,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1B,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM,GAAG,IAAI,CAAC;IACjD,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,MAAM,CAAC,EAAE,OAAO,GAAG,IAAI,CAAC;IACxB,kBAAkB,CAAC,EAAE,MAAM,CAAC;CAC7B;AAED,wBAAsB,UAAU,CAAC,EAC/B,KAAK,EACL,GAAG,EACH,IAAW,EACX,QAAgB,EAChB,OAAa,EACb,OAAc,EACd,YAAmB,EACnB,cAAsB,EACtB,gBAAuB,EACvB,KAAK,EAAE,SAAiB,EACxB,QAAe,EACf,KAAY,EACZ,eAAsB,EACtB,QAAgB,EAChB,UAAkB,EAClB,WAAmB,EACnB,SAAgB,EAChB,OAAc,EACd,UAAc,EACd,QAAe,EACf,MAAa,EACb,kBAAuB,EACvB,OAAO,GACR,EAAE,iBAAiB,GAAG,OAAO,CAAC,WAAW,CAAC,CAgE1C;AAED,MAAM,WAAW,gBAAiB,SAAQ,eAAe;IACvD,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,UAAU,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC3B,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,oFAAoF;AACpF,wBAAgB,WAAW,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAOjD;AAED,wBAAsB,YAAY,CAAC,EACjC,KAAK,EACL,KAAK,EACL,OAAc,EACd,UAAiB,EACjB,OAAa,EACb,OAAO,GACR,EAAE,gBAAgB,GAAG,OAAO,CAAC,WAAW,CAAC,CAYzC;AAED,wBAAsB,yBAAyB,CAC7C,KAAK,EAAE,MAAM,EACb,EAAE,OAAO,EAAE,GAAE,eAAoB,GAChC,OAAO,CAAC,WAAW,CAAC,CAKtB"}
|
package/dist/api.js
CHANGED
|
@@ -17,7 +17,7 @@ export function getAuthHeaders(token) {
|
|
|
17
17
|
function compact(values) {
|
|
18
18
|
return Object.fromEntries(Object.entries(values).filter((entry) => entry[1] !== null && entry[1] !== undefined));
|
|
19
19
|
}
|
|
20
|
-
export async function scraperApi({ token, url, html = true, markdown = false, timeout = 120, geoCode =
|
|
20
|
+
export async function scraperApi({ token, url, html = true, markdown = false, timeout = 120, geoCode = null, proxyCountry = null, blockResources = false, browserRendering = null, super: superMode = false, useProxy = true, proxy = null, waitForSelector = null, homePage = false, screenshot = false, bypassProxy = false, cookieJar = null, headers = null, maxRetries = 3, tokenCap = null, stream = null, discardTokenSecret = "", fetchFn, }) {
|
|
21
21
|
if (!Number.isInteger(timeout) || timeout < 1 || timeout > 300) {
|
|
22
22
|
throw new Error("timeout must be between 1 and 300 seconds");
|
|
23
23
|
}
|
|
@@ -28,7 +28,10 @@ export async function scraperApi({ token, url, html = true, markdown = false, ti
|
|
|
28
28
|
throw new Error("token_cap must be a nonnegative integer");
|
|
29
29
|
}
|
|
30
30
|
const apiToken = normalizeBearerToken(token);
|
|
31
|
-
|
|
31
|
+
// GET with every option in the query, as the Python SDK's scraper_api sends it. The API replays a
|
|
32
|
+
// request body to the target page, so a POST carrying {url, maxRetries} reached sites as a POST
|
|
33
|
+
// with that JSON body and failed with 404/405/422 on most of them.
|
|
34
|
+
const response = await request("GET", getApiEndpoints().scraperApiUrl, {
|
|
32
35
|
headers: {
|
|
33
36
|
...getAuthHeaders(apiToken),
|
|
34
37
|
...(discardTokenSecret
|
|
@@ -57,43 +60,45 @@ export async function scraperApi({ token, url, html = true, markdown = false, ti
|
|
|
57
60
|
? JSON.stringify(headers)
|
|
58
61
|
: headers,
|
|
59
62
|
stream,
|
|
63
|
+
url,
|
|
64
|
+
maxRetries,
|
|
65
|
+
tokenCap,
|
|
60
66
|
}),
|
|
61
|
-
json: compact({ url, maxRetries, tokenCap }),
|
|
62
67
|
timeout: timeout + 30,
|
|
63
68
|
fetchFn,
|
|
64
69
|
});
|
|
70
|
+
// On GET the API returns the page's HTML alongside Markdown even with html=false (the POST path
|
|
71
|
+
// left it out). Drop it here so a Markdown request does not hand the model megabytes of HTML.
|
|
72
|
+
const { data } = response;
|
|
73
|
+
if (!html &&
|
|
74
|
+
data &&
|
|
75
|
+
typeof data === "object" &&
|
|
76
|
+
!Array.isArray(data) &&
|
|
77
|
+
typeof data.html === "string") {
|
|
78
|
+
return { ...response, data: { ...data, html: null } };
|
|
79
|
+
}
|
|
80
|
+
return response;
|
|
65
81
|
}
|
|
66
|
-
|
|
82
|
+
/** Returns the query text, or the `q` parameter when given a search results URL. */
|
|
83
|
+
export function searchQuery(input) {
|
|
67
84
|
const value = String(input || "").trim();
|
|
68
85
|
if (!value)
|
|
69
|
-
throw new Error("A web search query
|
|
70
|
-
if (!/^https?:\/\//i.test(value))
|
|
71
|
-
return
|
|
72
|
-
|
|
73
|
-
const parsed = new URL(value);
|
|
74
|
-
const query = parsed.searchParams.get("q")?.trim();
|
|
86
|
+
throw new Error("A web search query is required");
|
|
87
|
+
if (!/^https?:\/\//i.test(value))
|
|
88
|
+
return value;
|
|
89
|
+
const query = new URL(value).searchParams.get("q")?.trim();
|
|
75
90
|
if (!query)
|
|
76
91
|
throw new Error("Search results URL must contain a q parameter");
|
|
77
|
-
|
|
78
|
-
const start = /^[+-]?\d+$/.test(startValue.trim()) ? Number(startValue) : 0;
|
|
79
|
-
return {
|
|
80
|
-
query,
|
|
81
|
-
region: parsed.searchParams.get("gl") || null,
|
|
82
|
-
language: parsed.searchParams.get("hl") || null,
|
|
83
|
-
page: Number.isFinite(start) && start > 0 ? Math.floor(start / 10) + 1 : null,
|
|
84
|
-
};
|
|
92
|
+
return query;
|
|
85
93
|
}
|
|
86
|
-
export async function webSearchApi({ token,
|
|
87
|
-
|
|
88
|
-
return request("POST", getApiEndpoints().webSearchUrl, {
|
|
94
|
+
export async function webSearchApi({ token, query, country = null, numResults = null, timeout = 600, fetchFn, }) {
|
|
95
|
+
return request("POST", getApiEndpoints().scraperApiUrl, {
|
|
89
96
|
headers: getAuthHeaders(token),
|
|
90
97
|
json: compact({
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
format: raw ? "html" : format,
|
|
96
|
-
renderJs,
|
|
98
|
+
action: "websearch",
|
|
99
|
+
query: searchQuery(query),
|
|
100
|
+
country: country ? country.toLowerCase() : null,
|
|
101
|
+
numResults,
|
|
97
102
|
}),
|
|
98
103
|
timeout,
|
|
99
104
|
fetchFn,
|
package/dist/api.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"api.js","sourceRoot":"","sources":["../src/api.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,oBAAoB,EAAE,MAAM,WAAW,CAAC;AACjD,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,EAAE,OAAO,EAAoB,MAAM,WAAW,CAAC;AACtD,OAAO,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;
|
|
1
|
+
{"version":3,"file":"api.js","sourceRoot":"","sources":["../src/api.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,oBAAoB,EAAE,MAAM,WAAW,CAAC;AACjD,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,EAAE,OAAO,EAAoB,MAAM,WAAW,CAAC;AACtD,OAAO,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAMhD,MAAM,UAAU,cAAc,CAAC,KAAa;IAC1C,MAAM,QAAQ,GAAG,oBAAoB,CAAC,KAAK,CAAC,CAAC;IAC7C,IAAI,CAAC,QAAQ;QAAE,MAAM,IAAI,KAAK,CAAC,uBAAuB,CAAC,CAAC;IACxD,OAAO;QACL,YAAY,EAAE,MAAM;QACpB,iBAAiB,EAAE,GAAG;QACtB,gBAAgB,EAAE,MAAM;QACxB,aAAa,EAAE,UAAU,QAAQ,EAAE;QACnC,GAAG,CAAC,CAAC,kBAAkB,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAE,aAAa,EAAE,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACtE,CAAC;AACJ,CAAC;AAED,SAAS,OAAO,CACd,MAAoE;IAEpE,OAAO,MAAM,CAAC,WAAW,CACvB,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,MAAM,CAC3B,CAAC,KAAK,EAAgD,EAAE,CACtD,KAAK,CAAC,CAAC,CAAC,KAAK,IAAI,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,SAAS,CAC9C,CACF,CAAC;AACJ,CAAC;AA2BD,MAAM,CAAC,KAAK,UAAU,UAAU,CAAC,EAC/B,KAAK,EACL,GAAG,EACH,IAAI,GAAG,IAAI,EACX,QAAQ,GAAG,KAAK,EAChB,OAAO,GAAG,GAAG,EACb,OAAO,GAAG,IAAI,EACd,YAAY,GAAG,IAAI,EACnB,cAAc,GAAG,KAAK,EACtB,gBAAgB,GAAG,IAAI,EACvB,KAAK,EAAE,SAAS,GAAG,KAAK,EACxB,QAAQ,GAAG,IAAI,EACf,KAAK,GAAG,IAAI,EACZ,eAAe,GAAG,IAAI,EACtB,QAAQ,GAAG,KAAK,EAChB,UAAU,GAAG,KAAK,EAClB,WAAW,GAAG,KAAK,EACnB,SAAS,GAAG,IAAI,EAChB,OAAO,GAAG,IAAI,EACd,UAAU,GAAG,CAAC,EACd,QAAQ,GAAG,IAAI,EACf,MAAM,GAAG,IAAI,EACb,kBAAkB,GAAG,EAAE,EACvB,OAAO,GACW;IAClB,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,OAAO,CAAC,IAAI,OAAO,GAAG,CAAC,IAAI,OAAO,GAAG,GAAG,EAAE,CAAC;QAC/D,MAAM,IAAI,KAAK,CAAC,2CAA2C,CAAC,CAAC;IAC/D,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,UAAU,CAAC,IAAI,UAAU,GAAG,CAAC,EAAE,CAAC;QACpD,MAAM,IAAI,KAAK,CAAC,2CAA2C,CAAC,CAAC;IAC/D,CAAC;IACD,IAAI,QAAQ,KAAK,IAAI,IAAI,CAAC,CAAC,MAAM,CAAC,SAAS,CAAC,QAAQ,CAAC,IAAI,QAAQ,GAAG,CAAC,CAAC,EAAE,CAAC;QACvE,MAAM,IAAI,KAAK,CAAC,yCAAyC,CAAC,CAAC;IAC7D,CAAC;IACD,MAAM,QAAQ,GAAG,oBAAoB,CAAC,KAAK,CAAC,CAAC;IAC7C,kGAAkG;IAClG,gGAAgG;IAChG,mEAAmE;IACnE,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,KAAK,EAAE,eAAe,EAAE,CAAC,aAAa,EAAE;QACrE,OAAO,EAAE;YACP,GAAG,cAAc,CAAC,QAAQ,CAAC;YAC3B,GAAG,CAAC,kBAAkB;gBACpB,CAAC,CAAC,EAAE,wBAAwB,EAAE,kBAAkB,EAAE;gBAClD,CAAC,CAAC,EAAE,CAAC;SACR;QACD,MAAM,EAAE,OAAO,CAAC;YACd,6EAA6E;YAC7E,KAAK,EAAE,kBAAkB,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,QAAQ;YACrD,OAAO;YACP,OAAO;YACP,YAAY;YACZ,gBAAgB,EAAE,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,gBAAgB;YACtD,KAAK,EAAE,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI;YAC9B,QAAQ,EAAE,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK;YACjC,KAAK;YACL,IAAI;YACJ,QAAQ;YACR,eAAe;YACf,SAAS,EAAE,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI;YACjC,UAAU,EAAE,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI;YACpC,cAAc,EAAE,cAAc,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI;YAC5C,WAAW,EAAE,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI;YACtC,SAAS;YACT,OAAO,EACL,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,KAAK,IAAI;gBAC7C,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC;gBACzB,CAAC,CAAC,OAAO;YACb,MAAM;YACN,GAAG;YACH,UAAU;YACV,QAAQ;SACT,CAAC;QACF,OAAO,EAAE,OAAO,GAAG,EAAE;QACrB,OAAO;KACR,CAAC,CAAC;IACH,gGAAgG;IAChG,8FAA8F;IAC9F,MAAM,EAAE,IAAI,EAAE,GAAG,QAAQ,CAAC;IAC1B,IACE,CAAC,IAAI;QACL,IAAI;QACJ,OAAO,IAAI,KAAK,QAAQ;QACxB,CAAC,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC;QACpB,OAAO,IAAI,CAAC,IAAI,KAAK,QAAQ,EAC7B,CAAC;QACD,OAAO,EAAE,GAAG,QAAQ,EAAE,IAAI,EAAE,EAAE,GAAG,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,EAAE,CAAC;IACxD,CAAC;IACD,OAAO,QAAQ,CAAC;AAClB,CAAC;AAUD,oFAAoF;AACpF,MAAM,UAAU,WAAW,CAAC,KAAa;IACvC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACzC,IAAI,CAAC,KAAK;QAAE,MAAM,IAAI,KAAK,CAAC,gCAAgC,CAAC,CAAC;IAC9D,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC;IAC/C,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC,YAAY,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,IAAI,EAAE,CAAC;IAC3D,IAAI,CAAC,KAAK;QAAE,MAAM,IAAI,KAAK,CAAC,+CAA+C,CAAC,CAAC;IAC7E,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,YAAY,CAAC,EACjC,KAAK,EACL,KAAK,EACL,OAAO,GAAG,IAAI,EACd,UAAU,GAAG,IAAI,EACjB,OAAO,GAAG,GAAG,EACb,OAAO,GACU;IACjB,OAAO,OAAO,CAAC,MAAM,EAAE,eAAe,EAAE,CAAC,aAAa,EAAE;QACtD,OAAO,EAAE,cAAc,CAAC,KAAK,CAAC;QAC9B,IAAI,EAAE,OAAO,CAAC;YACZ,MAAM,EAAE,WAAW;YACnB,KAAK,EAAE,WAAW,CAAC,KAAK,CAAC;YACzB,OAAO,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC,IAAI;YAC/C,UAAU;SACX,CAAC;QACF,OAAO;QACP,OAAO;KACR,CAAC,CAAC;AACL,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,yBAAyB,CAC7C,KAAa,EACb,EAAE,OAAO,EAAE,GAAoB,EAAE;IAEjC,OAAO,OAAO,CAAC,KAAK,EAAE,eAAe,EAAE,CAAC,oBAAoB,EAAE;QAC5D,OAAO,EAAE,EAAE,MAAM,EAAE,kBAAkB,EAAE,GAAG,cAAc,CAAC,KAAK,CAAC,EAAE;QACjE,OAAO;KACR,CAAC,CAAC;AACL,CAAC"}
|
package/dist/app.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"app.d.ts","sourceRoot":"","sources":["../src/app.ts"],"names":[],"mappings":"AAMA,OAAO,EAKL,KAAK,cAAc,
|
|
1
|
+
{"version":3,"file":"app.d.ts","sourceRoot":"","sources":["../src/app.ts"],"names":[],"mappings":"AAMA,OAAO,EAKL,KAAK,cAAc,EAEnB,KAAK,aAAa,EACnB,MAAM,8BAA8B,CAAC;AACtC,OAAO,KAAK,EAAE,OAAO,EAAmC,MAAM,SAAS,CAAC;AAExE,OAAO,EAQL,KAAK,WAAW,EAEjB,MAAM,aAAa,CAAC;AAMrB,OAAO,EAAuB,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AAE7E,MAAM,WAAW,cAAe,SAAQ,oBAAoB;IAC1D,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,OAAO,CAAC,OAAO,CAAC,CAAC;IACpD,WAAW,CAAC,EAAE,WAAW,CAAC;IAC1B,aAAa,CAAC,EAAE,aAAa,CAAC;IAC9B,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,aAAa,CAAC,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,OAAO;IACtB,GAAG,EAAE,OAAO,CAAC;IACb,OAAO,EAAE,cAAc,CAAC;CACzB;AA2ID,wBAAgB,aAAa,CAAC,OAAO,GAAE,cAAmB,GAAG,OAAO,CA2GnE"}
|
package/dist/app.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { createMcpExpressApp, mcpAuthMetadataRouter, requireBearerAuth, } from "@modelcontextprotocol/express";
|
|
2
2
|
import { toNodeHandler } from "@modelcontextprotocol/node";
|
|
3
3
|
import { buildOAuthProtectedResourceMetadata, createMcpHandler, getOAuthProtectedResourceMetadataUrl, } from "@modelcontextprotocol/server";
|
|
4
|
-
import { TOOL_NAMES, VERSION, getOAuthConfig, httpAuthEnabled, httpRuntimeConfig, } from "./config.js";
|
|
4
|
+
import { CHATGPT_MCP_PATH, OPENAI_OAUTH_CLIENT_PREFIX, TOOL_NAMES, VERSION, getOAuthConfig, httpAuthEnabled, httpRuntimeConfig, } from "./config.js";
|
|
5
5
|
import { buildAuthorizationServerMetadata, createCompositeVerifier, } from "./oauth.js";
|
|
6
6
|
import { missingScopes, requiredScopesForRequest } from "./scopes.js";
|
|
7
7
|
import { createServerFactory } from "./server.js";
|
|
@@ -111,6 +111,15 @@ function servePrmAlias(metadata) {
|
|
|
111
111
|
response.json(metadata);
|
|
112
112
|
};
|
|
113
113
|
}
|
|
114
|
+
function surfaceForRequest(context) {
|
|
115
|
+
const path = context.requestInfo
|
|
116
|
+
? new URL(context.requestInfo.url).pathname
|
|
117
|
+
.replace(/\/+$/, "")
|
|
118
|
+
.toLowerCase()
|
|
119
|
+
: "";
|
|
120
|
+
const openAIClient = context.authInfo?.clientId?.startsWith(OPENAI_OAUTH_CLIENT_PREFIX);
|
|
121
|
+
return path === CHATGPT_MCP_PATH || openAIClient ? "chatgpt" : "default";
|
|
122
|
+
}
|
|
114
123
|
export function createHttpApp(options = {}) {
|
|
115
124
|
const runtime = httpRuntimeConfig();
|
|
116
125
|
const host = options.host || runtime.host;
|
|
@@ -127,9 +136,11 @@ export function createHttpApp(options = {}) {
|
|
|
127
136
|
host,
|
|
128
137
|
allowedOrigins: allowedOrigins.hostnames,
|
|
129
138
|
});
|
|
130
|
-
const
|
|
131
|
-
|
|
132
|
-
|
|
139
|
+
const factories = {
|
|
140
|
+
default: createServerFactory(options),
|
|
141
|
+
chatgpt: createServerFactory({ ...options, surface: "chatgpt" }),
|
|
142
|
+
};
|
|
143
|
+
const handler = createMcpHandler((context) => factories[surfaceForRequest(context)](context), { onerror: (error) => console.error("MCP request error:", error) });
|
|
133
144
|
const nodeHandler = toNodeHandler(handler);
|
|
134
145
|
const logPayloads = options.logPayloads ??
|
|
135
146
|
["1", "true", "yes"].includes((process.env.MRSCRAPER_LOG_HTTP_PAYLOAD || "").toLowerCase());
|
|
@@ -174,7 +185,7 @@ export function createHttpApp(options = {}) {
|
|
|
174
185
|
: []),
|
|
175
186
|
]
|
|
176
187
|
: [];
|
|
177
|
-
app.all("/mcp", ...auth, async (request, response, next) => {
|
|
188
|
+
app.all(["/mcp", CHATGPT_MCP_PATH], ...auth, async (request, response, next) => {
|
|
178
189
|
try {
|
|
179
190
|
await nodeHandler(request, response, request.body);
|
|
180
191
|
}
|
package/dist/app.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"app.js","sourceRoot":"","sources":["../src/app.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,mBAAmB,EACnB,qBAAqB,EACrB,iBAAiB,GAClB,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,aAAa,EAAE,MAAM,4BAA4B,CAAC;AAC3D,OAAO,EACL,mCAAmC,EACnC,gBAAgB,EAChB,oCAAoC,
|
|
1
|
+
{"version":3,"file":"app.js","sourceRoot":"","sources":["../src/app.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,mBAAmB,EACnB,qBAAqB,EACrB,iBAAiB,GAClB,MAAM,+BAA+B,CAAC;AACvC,OAAO,EAAE,aAAa,EAAE,MAAM,4BAA4B,CAAC;AAC3D,OAAO,EACL,mCAAmC,EACnC,gBAAgB,EAChB,oCAAoC,GAKrC,MAAM,8BAA8B,CAAC;AAGtC,OAAO,EACL,gBAAgB,EAChB,0BAA0B,EAC1B,UAAU,EACV,OAAO,EACP,cAAc,EACd,eAAe,EACf,iBAAiB,GAGlB,MAAM,aAAa,CAAC;AACrB,OAAO,EACL,gCAAgC,EAChC,uBAAuB,GACxB,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,aAAa,EAAE,wBAAwB,EAAE,MAAM,aAAa,CAAC;AACtE,OAAO,EAAE,mBAAmB,EAA6B,MAAM,aAAa,CAAC;AAkB7E,MAAM,sBAAsB,GAAG,CAAC,WAAW,EAAE,WAAW,EAAE,OAAO,CAAC,CAAC;AAEnE,SAAS,uBAAuB,CAAC,OAAiB;IAIhD,MAAM,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE;QACxC,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,CAAC;YAC/B,IACE,CAAC,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC;gBAC9C,MAAM,CAAC,QAAQ,KAAK,GAAG;gBACvB,MAAM,CAAC,MAAM;gBACb,MAAM,CAAC,IAAI,EACX,CAAC;gBACD,MAAM,IAAI,KAAK,CAAC,oBAAoB,CAAC,CAAC;YACxC,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,IAAI,KAAK,CACb,2CAA2C,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,iEAAiE,CACnI,CAAC;QACJ,CAAC;IACH,CAAC,CAAC,CAAC;IACH,OAAO;QACL,OAAO,EAAE,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC3D,SAAS,EAAE;YACT,GAAG,IAAI,GAAG,CAAC;gBACT,GAAG,sBAAsB;gBACzB,GAAG,UAAU,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,QAAQ,CAAC;aAC/C,CAAC;SACH;KACF,CAAC;AACJ,CAAC;AAED,SAAS,qBAAqB,CAAC,cAA2B;IACxD,OAAO,CAAC,OAAgB,EAAE,QAAkB,EAAE,IAAkB,EAAQ,EAAE;QACxE,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QACrC,IAAI,CAAC,MAAM,EAAE,CAAC;YACZ,IAAI,EAAE,CAAC;YACP,OAAO;QACT,CAAC;QACD,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,CAAC;YAC/B,IACE,sBAAsB,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC;gBAChD,cAAc,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,EACjC,CAAC;gBACD,IAAI,EAAE,CAAC;gBACP,OAAO;YACT,CAAC;QACH,CAAC;QAAC,MAAM,CAAC;YACP,wEAAwE;QAC1E,CAAC;QACD,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC;IAChD,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,oBAAoB,CAAC,OAAgB,EAAE,aAAqB;IACnE,OAAO,CAAC,OAAgB,EAAE,SAAmB,EAAE,IAAkB,EAAQ,EAAE;QACzE,IACE,OAAO;YACP,CAAC,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,MAAM,CAAC;YAC3D,OAAO,CAAC,IAAI,KAAK,SAAS,EAC1B,CAAC;YACD,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;YAChD,MAAM,QAAQ,GACZ,UAAU,CAAC,MAAM,GAAG,aAAa;gBAC/B,CAAC,CAAC,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,EAAE,aAAa,CAAC,iBAAiB;gBACxD,CAAC,CAAC,UAAU,CAAC;YACjB,OAAO,CAAC,KAAK,CAAC,GAAG,OAAO,CAAC,MAAM,IAAI,OAAO,CAAC,IAAI,aAAa,QAAQ,EAAE,CAAC,CAAC;QAC1E,CAAC;QACD,IAAI,EAAE,CAAC;IACT,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,iBAAiB,CAAC,mBAA2B;IACpD,OAAO,CAAC,OAAgB,EAAE,QAAkB,EAAE,IAAkB,EAAQ,EAAE;QACxE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;QAC1B,IAAI,CAAC,IAAI,EAAE,CAAC;YACV,IAAI,EAAE,CAAC;YACP,OAAO;QACT,CAAC;QACD,MAAM,QAAQ,GAAG,wBAAwB,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;QACxD,MAAM,OAAO,GAAG,aAAa,CAAC,IAAI,CAAC,MAAM,EAAE,QAAQ,CAAC,CAAC;QACrD,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACzB,IAAI,EAAE,CAAC;YACP,OAAO;QACT,CAAC;QACD,MAAM,KAAK,GAAG,CAAC,GAAG,IAAI,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACpE,QAAQ;aACL,MAAM,CAAC,GAAG,CAAC;aACX,GAAG,CACF,kBAAkB,EAClB,6CAA6C,KAAK,yBAAyB,mBAAmB,KAAK;YACjG,6CAA6C,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAC1E;aACA,IAAI,CAAC;YACJ,KAAK,EAAE,oBAAoB;YAC3B,iBAAiB,EAAE,0BAA0B,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ;SACvE,CAAC,CAAC;IACP,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,QAAiB;IACtC,OAAO,CAAC,OAAgB,EAAE,QAAkB,EAAE,IAAkB,EAAQ,EAAE;QACxE,IAAI,OAAO,CAAC,IAAI,KAAK,GAAG,EAAE,CAAC;YACzB,IAAI,EAAE,CAAC;YACP,OAAO;QACT,CAAC;QACD,QAAQ,CAAC,GAAG,CAAC,6BAA6B,EAAE,GAAG,CAAC,CAAC;QACjD,QAAQ,CAAC,GAAG,CAAC,8BAA8B,EAAE,GAAG,CAAC,CAAC;QAClD,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;YACjC,QAAQ,CAAC,GAAG,CAAC,8BAA8B,EAAE,cAAc,CAAC,CAAC;YAC7D,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC;YAC3B,OAAO;QACT,CAAC;QACD,IAAI,OAAO,CAAC,MAAM,KAAK,KAAK,EAAE,CAAC;YAC7B,QAAQ,CAAC,GAAG,CAAC,OAAO,EAAE,cAAc,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC;YACxD,OAAO;QACT,CAAC;QACD,QAAQ,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC1B,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,iBAAiB,CAAC,OAA0B;IACnD,MAAM,IAAI,GAAG,OAAO,CAAC,WAAW;QAC9B,CAAC,CAAC,IAAI,GAAG,CAAC,OAAO,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,QAAQ;aACtC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;aACnB,WAAW,EAAE;QAClB,CAAC,CAAC,EAAE,CAAC;IACP,MAAM,YAAY,GAAG,OAAO,CAAC,QAAQ,EAAE,QAAQ,EAAE,UAAU,CACzD,0BAA0B,CAC3B,CAAC;IACF,OAAO,IAAI,KAAK,gBAAgB,IAAI,YAAY,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC;AAC3E,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,OAAO,GAAmB,EAAE;IACxD,MAAM,OAAO,GAAG,iBAAiB,EAAE,CAAC;IACpC,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAC1C,MAAM,OAAO,GAAG,OAAO,CAAC,cAAc,IAAI,OAAO,CAAC,cAAc,CAAC;IACjE,MAAM,cAAc,GAAG,uBAAuB,CAAC,OAAO,CAAC,CAAC;IACxD,MAAM,WAAW,GAAG,OAAO,CAAC,WAAW,IAAI,cAAc,EAAE,CAAC;IAC5D,MAAM,iBAAiB,GAAG,WAAW,CAAC,OAAO;QAC3C,CAAC,CAAC,IAAI,GAAG,CAAC,WAAW,CAAC,WAAW,CAAC;QAClC,CAAC,CAAC,SAAS,CAAC;IACd,MAAM,mBAAmB,GAAG,iBAAiB;QAC3C,CAAC,CAAC,oCAAoC,CAAC,iBAAiB,CAAC;QACzD,CAAC,CAAC,SAAS,CAAC;IACd,MAAM,GAAG,GAAG,mBAAmB,CAAC;QAC9B,IAAI;QACJ,cAAc,EAAE,cAAc,CAAC,SAAS;KACzC,CAAC,CAAC;IACH,MAAM,SAAS,GAAG;QAChB,OAAO,EAAE,mBAAmB,CAAC,OAAO,CAAC;QACrC,OAAO,EAAE,mBAAmB,CAAC,EAAE,GAAG,OAAO,EAAE,OAAO,EAAE,SAAS,EAAE,CAAC;KACjE,CAAC;IACF,MAAM,OAAO,GAAG,gBAAgB,CAC9B,CAAC,OAAO,EAAE,EAAE,CAAC,SAAS,CAAC,iBAAiB,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,EAC3D,EAAE,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,CAAC,oBAAoB,EAAE,KAAK,CAAC,EAAE,CACnE,CAAC;IACF,MAAM,WAAW,GAAG,aAAa,CAAC,OAAO,CAAC,CAAC;IAC3C,MAAM,WAAW,GACf,OAAO,CAAC,WAAW;QACnB,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,QAAQ,CAC3B,CAAC,OAAO,CAAC,GAAG,CAAC,0BAA0B,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,CAC7D,CAAC;IACJ,MAAM,aAAa,GAAG,MAAM,CAC1B,OAAO,CAAC,aAAa,IAAI,OAAO,CAAC,GAAG,CAAC,8BAA8B,IAAI,IAAI,CAC5E,CAAC;IACF,MAAM,aAAa,GACjB,MAAM,CAAC,SAAS,CAAC,aAAa,CAAC,IAAI,aAAa,GAAG,CAAC,CAAC,CAAC,CAAC,aAAa,CAAC,CAAC,CAAC,IAAI,CAAC;IAE9E,IAAI,iBAAiB,EAAE,CAAC;QACtB,MAAM,mBAAmB,GAAwB;YAC/C,aAAa,EACX,OAAO,CAAC,aAAa,IAAI,gCAAgC,CAAC,WAAW,CAAC;YACxE,iBAAiB;YACjB,eAAe,EAAE,CAAC,GAAG,WAAW,CAAC,eAAe,CAAC;YACjD,YAAY,EAAE,WAAW;YACzB,uBAAuB,EAAE,IAAI,GAAG,CAAC,WAAW,CAAC,gBAAgB,CAAC;SAC/D,CAAC;QACF,GAAG,CAAC,GAAG,CAAC,qBAAqB,CAAC,mBAAmB,CAAC,CAAC,CAAC;QAEpD,GAAG,CAAC,GAAG,CACL,uCAAuC,EACvC,aAAa,CAAC,mCAAmC,CAAC,mBAAmB,CAAC,CAAC,CACxE,CAAC;IACJ,CAAC;IAED,GAAG,CAAC,GAAG,CAAC,qBAAqB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC,CAAC;IACvD,GAAG,CAAC,GAAG,CAAC,oBAAoB,CAAC,WAAW,EAAE,aAAa,CAAC,CAAC,CAAC;IAC1D,GAAG,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,QAAQ,EAAE,QAAQ,EAAE,EAAE;QACxC,QAAQ,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,eAAe,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC,CAAC;IAC9E,CAAC,CAAC,CAAC;IACH,GAAG,CAAC,GAAG,CAAC,QAAQ,EAAE,CAAC,QAAQ,EAAE,QAAQ,EAAE,EAAE;QACvC,QAAQ,CAAC,IAAI,CAAC;YACZ,MAAM,EAAE,OAAO;YACf,OAAO,EAAE,eAAe;YACxB,OAAO,EAAE,OAAO;YAChB,KAAK,EAAE,UAAU,CAAC,MAAM;SACzB,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;IAEH,MAAM,WAAW,GAAG,OAAO,CAAC,WAAW,IAAI,eAAe,EAAE,CAAC;IAC7D,MAAM,IAAI,GAAG,WAAW;QACtB,CAAC,CAAC;YACE,iBAAiB,CAAC;gBAChB,QAAQ,EAAE,uBAAuB,CAAC;oBAChC,MAAM,EAAE,WAAW;oBACnB,aAAa,EAAE,OAAO,CAAC,aAAa;iBACrC,CAAC;gBACF,GAAG,CAAC,mBAAmB,CAAC,CAAC,CAAC,EAAE,mBAAmB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;aACxD,CAAC;YACF,GAAG,CAAC,mBAAmB;gBACrB,CAAC,CAAC,CAAC,iBAAiB,CAAC,mBAAmB,CAAC,CAAC;gBAC1C,CAAC,CAAC,EAAE,CAAC;SACR;QACH,CAAC,CAAC,EAAE,CAAC;IACP,GAAG,CAAC,GAAG,CACL,CAAC,MAAM,EAAE,gBAAgB,CAAC,EAC1B,GAAG,IAAI,EACP,KAAK,EAAE,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE;QAChC,IAAI,CAAC;YACH,MAAM,WAAW,CAAC,OAAO,EAAE,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,KAAK,CAAC,CAAC;QACd,CAAC;IACH,CAAC,CACF,CAAC;IACF,GAAG,CAAC,GAAG,CACL,CACE,KAAc,EACd,QAAiB,EACjB,QAAkB,EAClB,KAAmB,EACnB,EAAE;QACF,OAAO,CAAC,KAAK,CAAC,oBAAoB,EAAE,KAAK,CAAC,CAAC;QAC3C,IAAI,CAAC,QAAQ,CAAC,WAAW,EAAE,CAAC;YAC1B,QAAQ,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,uBAAuB,EAAE,CAAC,CAAC;QAChE,CAAC;IACH,CAAC,CACF,CAAC;IACF,OAAO,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC;AAC1B,CAAC"}
|