@mrscraper/mcp 0.1.2 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +230 -353
- package/dist/api.d.ts +26 -75
- package/dist/api.d.ts.map +1 -1
- package/dist/api.js +72 -171
- package/dist/api.js.map +1 -1
- package/dist/app.js +1 -1
- package/dist/app.js.map +1 -1
- package/dist/config.d.ts +5 -12
- package/dist/config.d.ts.map +1 -1
- package/dist/config.js +6 -25
- package/dist/config.js.map +1 -1
- package/dist/http.d.ts.map +1 -1
- package/dist/http.js +5 -1
- package/dist/http.js.map +1 -1
- package/dist/index.d.ts +0 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -2
- package/dist/index.js.map +1 -1
- package/dist/scopes.d.ts.map +1 -1
- package/dist/scopes.js +2 -6
- package/dist/scopes.js.map +1 -1
- package/dist/server.d.ts.map +1 -1
- package/dist/server.js +3 -16
- package/dist/server.js.map +1 -1
- package/dist/status.d.ts +0 -2
- package/dist/status.d.ts.map +1 -1
- package/dist/status.js +2 -30
- package/dist/status.js.map +1 -1
- package/dist/tools.d.ts +30 -80
- package/dist/tools.d.ts.map +1 -1
- package/dist/tools.js +184 -569
- package/dist/tools.js.map +1 -1
- package/dist/widgets/bundles.generated.d.ts.map +1 -1
- package/dist/widgets/bundles.generated.js +1 -2
- package/dist/widgets/bundles.generated.js.map +1 -1
- package/dist/widgets/index.d.ts +1 -7
- package/dist/widgets/index.d.ts.map +1 -1
- package/dist/widgets/index.js +1 -7
- package/dist/widgets/index.js.map +1 -1
- package/package.json +4 -4
- package/dist/compliance.d.ts +0 -4
- package/dist/compliance.d.ts.map +0 -1
- package/dist/compliance.js +0 -10
- package/dist/compliance.js.map +0 -1
package/README.md
CHANGED
|
@@ -1,14 +1,21 @@
|
|
|
1
|
-
# MrScraper MCP
|
|
1
|
+
# MrScraper MCP Server for Web Scraping and Google Search
|
|
2
2
|
|
|
3
|
-
MrScraper MCP
|
|
4
|
-
[Model Context Protocol](https://modelcontextprotocol.io).
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
MrScraper MCP is the official [MrScraper](https://mrscraper.com) server for the
|
|
4
|
+
[Model Context Protocol](https://modelcontextprotocol.io). It connects AI
|
|
5
|
+
assistants to page content and Google search through two SDK-compatible tools,
|
|
6
|
+
plus an account check. Retrieve HTML, Markdown, or screenshots with
|
|
7
|
+
`scraperAPI`, discover pages with `webSearch`, and check your plan and token
|
|
8
|
+
usage with `status`.
|
|
7
9
|
|
|
8
|
-
|
|
10
|
+
Use the hosted server with OAuth sign-in, or run the open-source server locally
|
|
11
|
+
with the [`@mrscraper/mcp` npm package](https://www.npmjs.com/package/@mrscraper/mcp).
|
|
12
|
+
See the [MrScraper MCP documentation](https://docs.mrscraper.com/docs/getting-started/mcp-server)
|
|
13
|
+
for setup instructions and client configurations.
|
|
14
|
+
|
|
15
|
+
The server exposes three tools:
|
|
9
16
|
|
|
10
17
|
```text
|
|
11
|
-
|
|
18
|
+
scraperAPI webSearch status
|
|
12
19
|
```
|
|
13
20
|
|
|
14
21
|
## Deployment options
|
|
@@ -54,11 +61,10 @@ the address will be refused.
|
|
|
54
61
|
|
|
55
62
|
### What you are approving
|
|
56
63
|
|
|
57
|
-
| Scope | Grants
|
|
58
|
-
| -------------- |
|
|
59
|
-
| `scrape:read` | `
|
|
60
|
-
| `
|
|
61
|
-
| `account:read` | `status`: subscription plan, quota, and request analytics |
|
|
64
|
+
| Scope | Grants |
|
|
65
|
+
| -------------- | ---------------------------- |
|
|
66
|
+
| `scrape:read` | `scraperAPI` and `webSearch` |
|
|
67
|
+
| `account:read` | `status` |
|
|
62
68
|
|
|
63
69
|
A tool called without its scope is refused with `403 insufficient_scope`, and
|
|
64
70
|
your client will offer to grant the missing permission.
|
|
@@ -118,436 +124,294 @@ matches how broadly the connection should be available.
|
|
|
118
124
|
## Copyable agent setup prompt
|
|
119
125
|
|
|
120
126
|
```text
|
|
121
|
-
Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that
|
|
127
|
+
Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that scraperAPI, webSearch, and status are the only available tools.
|
|
122
128
|
```
|
|
123
129
|
|
|
124
130
|
## Choosing a tool
|
|
125
131
|
|
|
126
|
-
| Tool
|
|
127
|
-
|
|
|
128
|
-
| `
|
|
129
|
-
| `
|
|
130
|
-
| `
|
|
131
|
-
| `status` | You need subscription, quota, token usage, or domain request outcomes. |
|
|
132
|
-
| `rerun` | You have a saved AI or manual scraper ID and want to run it again. |
|
|
133
|
-
| `results` | You need a paginated or filtered list of stored results. |
|
|
134
|
-
| `result` | You know one result ID and need its complete stored record. |
|
|
132
|
+
| Tool | Choose it when |
|
|
133
|
+
| ------------ | ---------------------------------------------------------------------------- |
|
|
134
|
+
| `scraperAPI` | You have a page URL and need HTML, Markdown, a screenshot, or a combination. |
|
|
135
|
+
| `webSearch` | You have a Google query or Google search URL and need search results. |
|
|
136
|
+
| `status` | You need the account's plan, token usage, or remaining tokens. |
|
|
135
137
|
|
|
136
|
-
A
|
|
137
|
-
|
|
138
|
+
A discovery workflow is `webSearch` → `scraperAPI`. Both web tools advertise
|
|
139
|
+
`readOnlyHint: true`, `openWorldHint: true`, and `destructiveHint: false`;
|
|
140
|
+
`status` is read-only and not open-world. Authentication comes from the MCP
|
|
141
|
+
connection; no tool takes an API token argument.
|
|
138
142
|
|
|
139
|
-
|
|
143
|
+
### Migration from the previous tool set
|
|
140
144
|
|
|
141
|
-
|
|
142
|
-
|
|
145
|
+
Only `scraperAPI`, `webSearch`, and `status` are registered. The other old tool
|
|
146
|
+
names are removed.
|
|
143
147
|
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
}
|
|
151
|
-
}
|
|
152
|
-
```
|
|
153
|
-
|
|
154
|
-
| Field | Type | Meaning |
|
|
155
|
-
| ------------- | ----------------------------- | ------------------------------------------------------------------------------------ |
|
|
156
|
-
| `status_code` | number or `null` | Upstream HTTP status, or `null` when a request ends before an HTTP response arrives. |
|
|
157
|
-
| `data` | JSON value, string, or `null` | Parsed JSON or response text supplied by the MrScraper service. |
|
|
158
|
-
| `headers` | object | Response headers with credential-bearing headers filtered out. |
|
|
159
|
-
| `error` | string, when present | Request failure summary. |
|
|
148
|
+
| Previous call | Replacement |
|
|
149
|
+
| -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
|
150
|
+
| `fetch` | `scraperAPI`; uses POST, SDK options, and automatic browser fallback by default. |
|
|
151
|
+
| `serp` | `webSearch`; rename `query_or_url` to `url` and `client_timeout` to `timeout`. Set `raw=false` for JSON. |
|
|
152
|
+
| `status` | `status` for account usage; the `domain`, `from`, `to`, `action`, and `api_token_name` inputs are gone. |
|
|
153
|
+
| `scrape`, `rerun`, `results`, `result` | Removed from this MCP server. |
|
|
160
154
|
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
sanitized while extracted scraper data remains available to the caller.
|
|
155
|
+
`super_mode` becomes `super`. `home_page` now maps to `warmStart`, and
|
|
156
|
+
`token_cap=0` is accepted. Restart the server and reconnect the MCP client after
|
|
157
|
+
updating so it refreshes `tools/list`.
|
|
165
158
|
|
|
166
159
|
## Tool reference
|
|
167
160
|
|
|
168
|
-
|
|
161
|
+
The options and defaults of `scraperAPI` and `webSearch` follow the Python SDK's
|
|
162
|
+
`scraper_api` and `web_search` methods. MCP tool names use camel case; their
|
|
163
|
+
argument names use the SDK's snake case.
|
|
169
164
|
|
|
170
|
-
`
|
|
165
|
+
### `scraperAPI`
|
|
171
166
|
|
|
172
167
|
```text
|
|
173
|
-
|
|
168
|
+
POST https://api.mrscraper.com/
|
|
174
169
|
```
|
|
175
170
|
|
|
176
|
-
Start with
|
|
171
|
+
Start with a known URL:
|
|
177
172
|
|
|
178
173
|
```json
|
|
179
174
|
{
|
|
180
|
-
"url": "https://
|
|
175
|
+
"url": "https://www.scrapethissite.com/pages/simple/"
|
|
181
176
|
}
|
|
182
177
|
```
|
|
183
178
|
|
|
184
|
-
|
|
179
|
+
Request HTML, Markdown, and a screenshot together:
|
|
185
180
|
|
|
186
181
|
```json
|
|
187
182
|
{
|
|
188
|
-
"url": "https://example.com
|
|
189
|
-
"
|
|
190
|
-
"
|
|
183
|
+
"url": "https://example.com",
|
|
184
|
+
"html": true,
|
|
185
|
+
"markdown": true,
|
|
186
|
+
"screenshot": true,
|
|
187
|
+
"super": true,
|
|
188
|
+
"home_page": true,
|
|
191
189
|
"geo_code": "ID",
|
|
192
|
-
"timeout":
|
|
190
|
+
"timeout": 120
|
|
193
191
|
}
|
|
194
192
|
```
|
|
195
193
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
`
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
|
221
|
-
|
|
|
222
|
-
| `
|
|
223
|
-
| `
|
|
224
|
-
| `
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
"agent": "general",
|
|
232
|
-
"prompt": "Extract the product name, price, availability, and image URLs"
|
|
233
|
-
}
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
Listing extraction:
|
|
237
|
-
|
|
238
|
-
```json
|
|
239
|
-
{
|
|
240
|
-
"url": "https://example.com/products",
|
|
241
|
-
"agent": "listing",
|
|
242
|
-
"prompt": "Extract every product name, price, and detail URL",
|
|
243
|
-
"max_pages": 5
|
|
244
|
-
}
|
|
245
|
-
```
|
|
246
|
-
|
|
247
|
-
Site map:
|
|
248
|
-
|
|
249
|
-
```json
|
|
250
|
-
{
|
|
251
|
-
"url": "https://example.com",
|
|
252
|
-
"agent": "map",
|
|
253
|
-
"max_depth": 2,
|
|
254
|
-
"max_pages": 50,
|
|
255
|
-
"limit": 1000,
|
|
256
|
-
"include_patterns": "/products/",
|
|
257
|
-
"exclude_patterns": "/cart/"
|
|
258
|
-
}
|
|
259
|
-
```
|
|
260
|
-
|
|
261
|
-
Best-effort schema guidance:
|
|
262
|
-
|
|
263
|
-
```json
|
|
264
|
-
{
|
|
265
|
-
"url": "https://example.com/product",
|
|
266
|
-
"prompt": "Extract the product",
|
|
267
|
-
"schema_prompt": {
|
|
268
|
-
"type": "object",
|
|
269
|
-
"properties": {
|
|
270
|
-
"name": { "type": "string" },
|
|
271
|
-
"price": { "type": "number" }
|
|
272
|
-
},
|
|
273
|
-
"required": ["name", "price"]
|
|
274
|
-
}
|
|
275
|
-
}
|
|
276
|
-
```
|
|
277
|
-
|
|
278
|
-
`schema_prompt` is appended to the natural-language instruction. Treat it as
|
|
279
|
-
shape guidance and validate returned data in the consuming application when
|
|
280
|
-
strict conformance is required.
|
|
281
|
-
|
|
282
|
-
| Input | Required | Default | Request mapping | Purpose |
|
|
283
|
-
| ------------------ | --------------- | --------------- | ---------------------- | ----------------------------------------------------------- |
|
|
284
|
-
| `url` | Yes | - | Body `url` | Target URL for every agent. |
|
|
285
|
-
| `prompt` | General/listing | - | Body `message` | Natural-language extraction instructions. |
|
|
286
|
-
| `schema_prompt` | No | omitted | Appended to `message` | Best-effort JSON Schema shape guidance for general/listing. |
|
|
287
|
-
| `agent` | No | `general` | Body `agent` | Selects `general`, `listing`, or `map`. |
|
|
288
|
-
| `proxy_country` | No | omitted | Body `proxyCountry` | Proxy country for general/listing. |
|
|
289
|
-
| `max_pages` | No | service default | Body `maxPages` | Page bound for listing/map. |
|
|
290
|
-
| `max_depth` | No | service default | Body `maxDepth` | Link-depth bound for map. |
|
|
291
|
-
| `limit` | No | service default | Body `limit` | URL-result bound for map. |
|
|
292
|
-
| `include_patterns` | No | service default | Body `includePatterns` | URL inclusion expression for map. |
|
|
293
|
-
| `exclude_patterns` | No | service default | Body `excludePatterns` | URL exclusion expression for map. |
|
|
294
|
-
|
|
295
|
-
General and listing send `url`, `message`, and `agent`, plus the supplied
|
|
296
|
-
agent inputs. Map sends `url`, `agent`, and only the supplied crawl inputs.
|
|
297
|
-
|
|
298
|
-
#### Reproduce a scrape with `rerun`
|
|
299
|
-
|
|
300
|
-
Every successful `scrape` creates a saved AI scraper configuration by default.
|
|
301
|
-
Its response run object contains `scraperId`. Pass that UUID to `rerun` as
|
|
302
|
-
`scraper_id` to apply the same saved extraction configuration to the original
|
|
303
|
-
URL or another URL without rebuilding the prompt and agent settings:
|
|
304
|
-
|
|
305
|
-
```json
|
|
306
|
-
{
|
|
307
|
-
"target": "https://example.com/product-2",
|
|
308
|
-
"type": "ai",
|
|
309
|
-
"scraper_id": "scraper-uuid"
|
|
310
|
-
}
|
|
311
|
-
```
|
|
312
|
-
|
|
313
|
-
This makes the scraper configuration reproducible, but it does not guarantee
|
|
314
|
-
identical extracted values when the page or model behavior changes.
|
|
315
|
-
|
|
316
|
-
`rerun` also supports dashboard-built manual workflows and asynchronous bulk
|
|
317
|
-
jobs across multiple target URLs. See the full [`rerun`](#rerun) section below
|
|
318
|
-
for the available modes and result-tracking workflow.
|
|
319
|
-
|
|
320
|
-
### `serp`
|
|
321
|
-
|
|
322
|
-
`serp` calls the synchronous Google endpoint:
|
|
194
|
+
Omit `browser_rendering` or use `null` to let the service try without a browser
|
|
195
|
+
and fall back to browser loading. `true` forces browser loading and `false`
|
|
196
|
+
disables it. A screenshot always forces browser loading, even when
|
|
197
|
+
`browser_rendering=false`. `super` selects real-device routing independently.
|
|
198
|
+
|
|
199
|
+
The JSON body contains `url`, `maxRetries`, and optional `tokenCap`. Page
|
|
200
|
+
options are query parameters. For API keys, the request also includes the
|
|
201
|
+
SDK-compatible `token` query parameter and `Authorization`/`x-api-token`
|
|
202
|
+
headers. OAuth JWTs are sent only through `Authorization`.
|
|
203
|
+
|
|
204
|
+
| Input | Default | API mapping / purpose |
|
|
205
|
+
| ---------------------- | -------- | --------------------------------------------------------------------------------------- |
|
|
206
|
+
| `url` | required | Body `url`; absolute HTTP(S) page URL. |
|
|
207
|
+
| `html` | `true` | Query `html`; request HTML. |
|
|
208
|
+
| `markdown` | `false` | Query `markdown`; request Markdown. |
|
|
209
|
+
| `timeout` | `120` | Query `timeout`; integer from 1 to 300 seconds. Transport allows 30 additional seconds. |
|
|
210
|
+
| `geo_code` | `"US"` | Query `geoCode`; proxy country code. |
|
|
211
|
+
| `proxy_country` | omitted | Query `proxyCountry`; compatibility country field. |
|
|
212
|
+
| `browser_rendering` | omitted | Query `browserRendering`; true/false, or omit/null for automatic fallback. |
|
|
213
|
+
| `super` | `false` | Query `super=true` when enabled. |
|
|
214
|
+
| `use_proxy` | `true` | Query `useProxy=false` when disabled. |
|
|
215
|
+
| `proxy` | omitted | Query `proxy`; custom `username:password@host:port` proxy. |
|
|
216
|
+
| `wait_for_selector` | omitted | Query `waitForSelector`; CSS selector to wait for. |
|
|
217
|
+
| `home_page` | `false` | Query `warmStart=true` when enabled. |
|
|
218
|
+
| `screenshot` | `false` | Query `screenshot=true`; also sets `browserRendering=true`. |
|
|
219
|
+
| `block_resources` | `false` | Query `blockResources=true`; block images, CSS, and fonts. |
|
|
220
|
+
| `bypass_proxy` | `false` | Query `bypassProxy=true`; proxy-bypass/resource-blocking option. |
|
|
221
|
+
| `cookie_jar` | omitted | Query `cookieJar`; cookie configuration string. |
|
|
222
|
+
| `headers` | omitted | Query `headers`; target request headers as an object or JSON string. |
|
|
223
|
+
| `max_retries` | `3` | Body `maxRetries`; nonnegative integer, 0 disables retries. |
|
|
224
|
+
| `token_cap` | omitted | Body `tokenCap`; nonnegative integer retry-token budget. |
|
|
225
|
+
| `stream` | omitted | Query `stream`; response is buffered into the normal MCP envelope. |
|
|
226
|
+
| `discard_token_secret` | `""` | Optional `x-discard-token-secret` header. |
|
|
227
|
+
|
|
228
|
+
### `webSearch`
|
|
323
229
|
|
|
324
230
|
```text
|
|
325
231
|
POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync
|
|
326
232
|
```
|
|
327
233
|
|
|
328
|
-
|
|
234
|
+
Request parsed Google results:
|
|
329
235
|
|
|
330
236
|
```json
|
|
331
237
|
{
|
|
332
|
-
"
|
|
238
|
+
"url": "MrScraper web scraping",
|
|
239
|
+
"raw": false,
|
|
333
240
|
"region": "id",
|
|
334
241
|
"language": "id",
|
|
335
|
-
"page":
|
|
242
|
+
"page": 1,
|
|
336
243
|
"format": "json"
|
|
337
244
|
}
|
|
338
245
|
```
|
|
339
246
|
|
|
340
|
-
|
|
247
|
+
`raw=true` is the Python SDK-compatible default and requests HTML regardless
|
|
248
|
+
of `format`. Set `raw=false` and `format="json"` for parsed results.
|
|
249
|
+
|
|
250
|
+
A Google search URL supplies the query, locale, and page:
|
|
341
251
|
|
|
342
252
|
```json
|
|
343
253
|
{
|
|
344
|
-
"
|
|
254
|
+
"url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
|
|
255
|
+
"raw": false
|
|
345
256
|
}
|
|
346
257
|
```
|
|
347
258
|
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
take priority over URL-derived values.
|
|
351
|
-
|
|
352
|
-
| Input | Required | Default | Request mapping | Purpose |
|
|
353
|
-
| ---------------- | -------- | -------------------- | ---------------------- | -------------------------------------------- |
|
|
354
|
-
| `query_or_url` | Yes | - | Body `query` | Google query or Google search URL. |
|
|
355
|
-
| `region` | No | URL value or omitted | Body `region` | Result country. |
|
|
356
|
-
| `language` | No | URL value or omitted | Body `language` | Result language. |
|
|
357
|
-
| `page` | No | URL value or omitted | Body `page` | One-based result page. |
|
|
358
|
-
| `format` | No | `json` | Body `format` | Selects parsed JSON or result-page HTML. |
|
|
359
|
-
| `render_js` | No | `false` | Body `renderJs` | Waits for JavaScript-rendered SERP features. |
|
|
360
|
-
| `raw` | No | `false` | Body `format=html` | Compatibility alias for HTML output. |
|
|
361
|
-
| `client_timeout` | No | `120` | Local request deadline | Sets the upstream HTTP timeout in seconds. |
|
|
259
|
+
This derives `query="web scraping"`, `region="us"`, `language="en"`, and
|
|
260
|
+
`page=3`. Explicit options override URL-derived values.
|
|
362
261
|
|
|
363
|
-
|
|
262
|
+
| Input | Default | Request mapping / purpose |
|
|
263
|
+
| ----------- | -------------------- | ----------------------------------------------------------- |
|
|
264
|
+
| `url` | required | Search text or Google URL; normalized to body `query`. |
|
|
265
|
+
| `raw` | `true` | Forces body `format="html"`; false allows `format`. |
|
|
266
|
+
| `format` | `"json"` | Body `format`, either `json` or `html`, when raw is false. |
|
|
267
|
+
| `region` | URL value or omitted | Body `region`; overrides `gl`. |
|
|
268
|
+
| `language` | URL value or omitted | Body `language`; overrides `hl`. |
|
|
269
|
+
| `page` | URL value or omitted | Body `page`; positive one-based integer, overrides `start`. |
|
|
270
|
+
| `render_js` | `false` | Body `renderJs`; render dynamic search features. |
|
|
271
|
+
| `timeout` | `600` | Local HTTP deadline in seconds; omitted from the body. |
|
|
364
272
|
|
|
365
|
-
`status`
|
|
273
|
+
### `status`
|
|
366
274
|
|
|
367
275
|
```text
|
|
368
276
|
GET https://api.app.mrscraper.com/api/v1/subscription-accounts
|
|
369
277
|
```
|
|
370
278
|
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
```json
|
|
378
|
-
{
|
|
379
|
-
"domain": "https://www.example.com/products",
|
|
380
|
-
"from": "7d",
|
|
381
|
-
"to": "now",
|
|
382
|
-
"action": "fetch",
|
|
383
|
-
"api_token_name": "production"
|
|
384
|
-
}
|
|
385
|
-
```
|
|
386
|
-
|
|
387
|
-
| Input | Required | Default | Purpose |
|
|
388
|
-
| ---------------- | -------- | ------------ | ------------------------------------------------------------------------------------------- |
|
|
389
|
-
| `domain` | No | omitted | Adds request-outcome analytics for a hostname or URL. |
|
|
390
|
-
| `from` | No | `24h` | Range start as ISO 8601, `now`, or a relative duration such as `30m`, `24h`, `7d`, or `2w`. |
|
|
391
|
-
| `to` | No | `now` | Range end using the same date syntax. |
|
|
392
|
-
| `action` | No | empty filter | Filters analytics by exact action. |
|
|
393
|
-
| `api_token_name` | No | empty filter | Filters analytics by API-token name. |
|
|
394
|
-
|
|
395
|
-
Successful output is a normalized account and analytics summary:
|
|
279
|
+
`status` takes no input. It returns the connected account's identity (name,
|
|
280
|
+
email, and verification state), subscription status, token limit, token usage,
|
|
281
|
+
remaining tokens, usage percentage, rate limit, auto-renewal, and renewal date.
|
|
282
|
+
It copies only those fields, so the API key and Stripe identifiers in the raw
|
|
283
|
+
account record never reach the client.
|
|
396
284
|
|
|
397
285
|
```json
|
|
398
286
|
{
|
|
399
287
|
"kind": "mrscraper-cli-status-summary",
|
|
400
|
-
"source_endpoints": ["/subscription-accounts"
|
|
288
|
+
"source_endpoints": ["/subscription-accounts"],
|
|
401
289
|
"status_code": 200,
|
|
402
290
|
"data": {
|
|
403
291
|
"account": {
|
|
404
292
|
"subscription_status": "active",
|
|
405
293
|
"enterprise": false,
|
|
406
|
-
"token_usage":
|
|
407
|
-
"token_limit":
|
|
408
|
-
"token_remaining":
|
|
409
|
-
"usage_percent":
|
|
410
|
-
"rate_limit":
|
|
411
|
-
"rate_ttl":
|
|
294
|
+
"token_usage": 61240,
|
|
295
|
+
"token_limit": 100000,
|
|
296
|
+
"token_remaining": 38760,
|
|
297
|
+
"usage_percent": 61.24,
|
|
298
|
+
"rate_limit": 0,
|
|
299
|
+
"rate_ttl": 0,
|
|
412
300
|
"auto_renew": true,
|
|
413
|
-
"ends_at":
|
|
301
|
+
"ends_at": "2026-12-01T00:00:00.000Z",
|
|
414
302
|
"user": {
|
|
415
303
|
"name": "Ada",
|
|
416
304
|
"email": "ada@example.com",
|
|
417
305
|
"verified": true
|
|
418
306
|
}
|
|
419
|
-
},
|
|
420
|
-
"analytics": {
|
|
421
|
-
"domain": "www.example.com",
|
|
422
|
-
"from": "2026-08-11 00:00:00 UTC",
|
|
423
|
-
"to": "2026-08-18 00:00:00 UTC"
|
|
424
307
|
}
|
|
425
308
|
}
|
|
426
309
|
}
|
|
427
310
|
```
|
|
428
311
|
|
|
429
|
-
|
|
430
|
-
URL to its hostname, and records the source endpoints used for the response.
|
|
312
|
+
An OAuth connection needs the `account:read` scope for `status`.
|
|
431
313
|
|
|
432
|
-
###
|
|
314
|
+
### Response envelope
|
|
433
315
|
|
|
434
|
-
`
|
|
316
|
+
`scraperAPI` and `webSearch` return `status_code`, `data`, and `headers` in
|
|
317
|
+
`structuredContent` and as JSON text. JSON bodies are parsed; other response
|
|
318
|
+
bodies remain text. `status` returns the summary shown above.
|
|
435
319
|
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
- Leave `bulk` as `false` for one URL. Set it to `true` to apply the same saved
|
|
440
|
-
configuration to a comma- or newline-separated URL list in one bulk request.
|
|
320
|
+
A `scraperAPI` response carries the requested representations in `data`;
|
|
321
|
+
`screenshot` is a base64-encoded JPEG and `x-status-code` is the target page's
|
|
322
|
+
status:
|
|
441
323
|
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
324
|
+
```json
|
|
325
|
+
{
|
|
326
|
+
"status_code": 200,
|
|
327
|
+
"data": {
|
|
328
|
+
"html": "<!DOCTYPE html>...",
|
|
329
|
+
"markdown": null,
|
|
330
|
+
"screenshot": null,
|
|
331
|
+
"token_usage": 3,
|
|
332
|
+
"runtime": 18.02,
|
|
333
|
+
"retry_count": 0,
|
|
334
|
+
"error": null
|
|
335
|
+
},
|
|
336
|
+
"headers": {
|
|
337
|
+
"content-type": "application/json",
|
|
338
|
+
"x-status-code": "200",
|
|
339
|
+
"token_usage": "3"
|
|
340
|
+
}
|
|
341
|
+
}
|
|
342
|
+
```
|
|
445
343
|
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
and pass it to `result` as `result_id` until the stored result is finished.
|
|
344
|
+
A parsed `webSearch` response (`raw=false`) has organic results in
|
|
345
|
+
`data.data.organic`.
|
|
449
346
|
|
|
450
|
-
`
|
|
347
|
+
Failures include `error` and set MCP `isError=true`. Network failures have
|
|
348
|
+
`status_code=null`. Credential-bearing response headers and JSON metadata
|
|
349
|
+
are filtered; network errors redact signed query credentials.
|
|
451
350
|
|
|
452
|
-
|
|
453
|
-
| ------------- | ---------------------------------- | ------------ | ---------------------------- | ------------------- |
|
|
454
|
-
| Single AI | `POST /scrapers-ai-rerun` | `scraper_id` | One URL | Available |
|
|
455
|
-
| Bulk AI | `POST /scrapers-ai-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
|
|
456
|
-
| Single manual | `POST /scrapers-manual-rerun` | `scraper_id` | One URL | Saved configuration |
|
|
457
|
-
| Bulk manual | `POST /scrapers-manual-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
|
|
351
|
+
## Manual testing: one tool at a time
|
|
458
352
|
|
|
459
|
-
|
|
353
|
+
[`scripts/test-tools.ts`](scripts/test-tools.ts) starts the compiled local MCP
|
|
354
|
+
server over stdio, verifies that exactly the three tools are exposed, calls one
|
|
355
|
+
tool, prints its response, and exits. No separate server terminal is needed.
|
|
356
|
+
The runner loads `.env`, or reads `MRSCRAPER_API_KEY` / `MRSCRAPER_API_TOKEN`
|
|
357
|
+
from your shell. Each tool call uses the live MrScraper service.
|
|
460
358
|
|
|
461
|
-
|
|
462
|
-
{
|
|
463
|
-
"target": "https://example.com/products",
|
|
464
|
-
"type": "ai",
|
|
465
|
-
"scraper_id": "scraper-uuid",
|
|
466
|
-
"max_depth": 2,
|
|
467
|
-
"max_pages": 50,
|
|
468
|
-
"limit": 1000,
|
|
469
|
-
"include_patterns": "/products/",
|
|
470
|
-
"exclude_patterns": "/cart/"
|
|
471
|
-
}
|
|
472
|
-
```
|
|
359
|
+
Run these commands individually from the repository directory:
|
|
473
360
|
|
|
474
|
-
|
|
361
|
+
1. Install dependencies:
|
|
475
362
|
|
|
476
|
-
```
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
"type": "manual",
|
|
480
|
-
"bulk": true,
|
|
481
|
-
"id": "scraper-uuid"
|
|
482
|
-
}
|
|
483
|
-
```
|
|
363
|
+
```sh
|
|
364
|
+
npm ci
|
|
365
|
+
```
|
|
484
366
|
|
|
485
|
-
|
|
486
|
-
| ------------------ | ----------- | ------------ | --------------------------------------------------------------- |
|
|
487
|
-
| `target` | Yes | - | One URL, or a comma/newline-separated URL string for bulk mode. |
|
|
488
|
-
| `type` | Yes | - | Selects `ai` or `manual`. |
|
|
489
|
-
| `bulk` | No | `false` | Selects a bulk endpoint. |
|
|
490
|
-
| `scraper_id` | Single mode | - | Saved scraper UUID for one URL. |
|
|
491
|
-
| `id` | Bulk mode | - | Saved scraper UUID for the bulk URL list. |
|
|
492
|
-
| `max_depth` | Single AI | `2` | Crawl depth. |
|
|
493
|
-
| `max_pages` | Single AI | `50` | Page bound. |
|
|
494
|
-
| `limit` | Single AI | `1000` | Result bound. |
|
|
495
|
-
| `include_patterns` | Single AI | empty string | URL inclusion expression. |
|
|
496
|
-
| `exclude_patterns` | Single AI | empty string | URL exclusion expression. |
|
|
367
|
+
2. Build the current server:
|
|
497
368
|
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
369
|
+
```sh
|
|
370
|
+
npm run build
|
|
371
|
+
```
|
|
501
372
|
|
|
502
|
-
|
|
373
|
+
3. Set your API key:
|
|
503
374
|
|
|
504
|
-
|
|
375
|
+
```sh
|
|
376
|
+
export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY"
|
|
377
|
+
```
|
|
505
378
|
|
|
506
|
-
|
|
507
|
-
GET https://api.app.mrscraper.com/api/v1/results
|
|
508
|
-
```
|
|
379
|
+
4. Test `scraperAPI`:
|
|
509
380
|
|
|
510
|
-
```
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
"sort_order": "desc",
|
|
514
|
-
"page_size": 25,
|
|
515
|
-
"page": 1,
|
|
516
|
-
"search": "example.com",
|
|
517
|
-
"date_range_column": "updatedAt",
|
|
518
|
-
"start_at": "2026-08-01T00:00:00Z",
|
|
519
|
-
"end_at": "2026-08-18T23:59:59Z"
|
|
520
|
-
}
|
|
521
|
-
```
|
|
381
|
+
```sh
|
|
382
|
+
npm run test:manual -- scraperAPI --url "https://www.scrapethissite.com/pages/simple/"
|
|
383
|
+
```
|
|
522
384
|
|
|
523
|
-
|
|
524
|
-
| ------------------- | -------- | ----------- | ----------------- | ------------------------------------------------------------- |
|
|
525
|
-
| `sort_field` | No | `updatedAt` | `sortField` | Field used by the results API for sorting. |
|
|
526
|
-
| `sort_order` | No | `desc` | `sortOrder` | Case-insensitive `asc` or `desc`; sent upstream in uppercase. |
|
|
527
|
-
| `page_size` | No | `10` | `pageSize` | Number of rows per page. |
|
|
528
|
-
| `page` | No | `1` | `page` | One-based page index. |
|
|
529
|
-
| `search` | No | omitted | `search` | Free-text result filter. |
|
|
530
|
-
| `date_range_column` | No | omitted | `dateRangeColumn` | Column used by the date range. |
|
|
531
|
-
| `start_at` | No | omitted | `startAt` | Inclusive range start. |
|
|
532
|
-
| `end_at` | No | omitted | `endAt` | Inclusive range end. |
|
|
385
|
+
5. Test `webSearch`:
|
|
533
386
|
|
|
534
|
-
|
|
387
|
+
```sh
|
|
388
|
+
npm run test:manual -- webSearch --query "MrScraper web scraping"
|
|
389
|
+
```
|
|
535
390
|
|
|
536
|
-
|
|
391
|
+
6. Test `status`:
|
|
537
392
|
|
|
538
|
-
```
|
|
539
|
-
|
|
393
|
+
```sh
|
|
394
|
+
npm run test:manual -- status
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
The webSearch sample explicitly requests `raw=false`, `format="json"`, US
|
|
398
|
+
results, and English. Each command prints `Tools: scraperAPI, webSearch, status`
|
|
399
|
+
and `PASS <tool>` on success. Exit codes are 0 for success, 1 for a tool or
|
|
400
|
+
connection failure, and 2 for invalid command arguments.
|
|
401
|
+
|
|
402
|
+
Pass a complete JSON object to test any supported options:
|
|
403
|
+
|
|
404
|
+
```sh
|
|
405
|
+
npm run test:manual -- scraperAPI --args '{"url":"https://example.com","markdown":true,"screenshot":true,"super":true}'
|
|
540
406
|
```
|
|
541
407
|
|
|
542
|
-
```
|
|
543
|
-
{
|
|
544
|
-
"result_id": "result-uuid"
|
|
545
|
-
}
|
|
408
|
+
```sh
|
|
409
|
+
npm run test:manual -- webSearch --args '{"url":"web scraping","raw":false,"region":"id","language":"id","page":1,"render_js":true}'
|
|
546
410
|
```
|
|
547
411
|
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
412
|
+
To test an already running HTTP MCP server, append
|
|
413
|
+
`--target http://127.0.0.1:8000/mcp`. Rebuild after source changes when testing
|
|
414
|
+
local stdio. `npm run test:manual -- --help` lists runner options.
|
|
551
415
|
|
|
552
416
|
## Run the server locally
|
|
553
417
|
|
|
@@ -624,7 +488,7 @@ server environment credentials are never used on the HTTP transport.
|
|
|
624
488
|
|
|
625
489
|
The server filters credential-bearing response headers. Parsed JSON credential
|
|
626
490
|
metadata and credentials embedded in generated curl commands are redacted.
|
|
627
|
-
|
|
491
|
+
Page content remains available in `data`.
|
|
628
492
|
|
|
629
493
|
Browser-origin requests are accepted from trusted local origins, from
|
|
630
494
|
`claude.ai` and `chatgpt.com`, and from exact origins configured through
|
|
@@ -634,10 +498,10 @@ checks, since any client must be able to read them before it has credentials.
|
|
|
634
498
|
|
|
635
499
|
## Interactive widgets
|
|
636
500
|
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
the JSON
|
|
501
|
+
`webSearch` carries a self-contained Google results widget for parsed JSON
|
|
502
|
+
results. HTML responses show a prompt to request parsed results. `status`
|
|
503
|
+
carries an account widget with the plan, token usage, and a usage meter. Hosts
|
|
504
|
+
without MCP Apps support display the normal JSON tool response.
|
|
641
505
|
|
|
642
506
|
Each widget is a self-contained document with its script and styles inlined,
|
|
643
507
|
because strict host CSP blocks anything external. Each one declares an empty
|
|
@@ -663,7 +527,7 @@ deployment and keep bearer authentication enabled.
|
|
|
663
527
|
| `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. |
|
|
664
528
|
| `PORT` | `8000` | HTTP listen port. |
|
|
665
529
|
| `MRSCRAPER_API_KEY` | - | Primary stdio credential. |
|
|
666
|
-
| `MRSCRAPER_API_TOKEN` | - |
|
|
530
|
+
| `MRSCRAPER_API_TOKEN` | - | Alternative stdio credential (also used by the Python SDK). |
|
|
667
531
|
| `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. |
|
|
668
532
|
| `MRSCRAPER_OAUTH` | `1` | Accepts OAuth 2.1 access tokens alongside API keys. |
|
|
669
533
|
| `MRSCRAPER_MCP_PUBLIC_URL` | `https://mcp.mrscraper.com` | Public origin; `<origin>/mcp` is the required token audience. |
|
|
@@ -671,8 +535,8 @@ deployment and keep bearer authentication enabled.
|
|
|
671
535
|
| `MRSCRAPER_OAUTH_JWKS_URL` | `<issuer>/.well-known/jwks.json` | Key set used to verify access tokens. |
|
|
672
536
|
| `MRSCRAPER_ALLOWED_ORIGINS` | - | Comma-separated browser origins allowed to call the HTTP server. |
|
|
673
537
|
| `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. |
|
|
674
|
-
| `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker |
|
|
675
|
-
| `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API |
|
|
538
|
+
| `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Scraper API endpoint override. |
|
|
539
|
+
| `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | Web search endpoint override. |
|
|
676
540
|
| `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. |
|
|
677
541
|
| `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
|
|
678
542
|
|
|
@@ -690,11 +554,11 @@ deployment and keep bearer authentication enabled.
|
|
|
690
554
|
- **403 Forbidden Origin:** Add the exact browser origin to
|
|
691
555
|
`MRSCRAPER_ALLOWED_ORIGINS`.
|
|
692
556
|
- **Tool input error:** Compare the call with the tool's parameter table and
|
|
693
|
-
the
|
|
694
|
-
- **Upstream timeout:** Increase `
|
|
557
|
+
the documented SDK-compatible defaults.
|
|
558
|
+
- **Upstream timeout:** Increase `scraperAPI.timeout` (up to 300 seconds) or `webSearch.timeout`
|
|
695
559
|
when the target requires a longer request window.
|
|
696
560
|
- **New tools are missing:** Start a new client session and inspect
|
|
697
|
-
`tools/list` for
|
|
561
|
+
`tools/list` for exactly `scraperAPI` and `webSearch`.
|
|
698
562
|
|
|
699
563
|
## Development
|
|
700
564
|
|
|
@@ -715,8 +579,21 @@ npm run test:mcp -- \
|
|
|
715
579
|
--token "$MRSCRAPER_API_KEY"
|
|
716
580
|
```
|
|
717
581
|
|
|
718
|
-
`npm run test:e2e`
|
|
719
|
-
service
|
|
582
|
+
`npm run test:e2e` runs both new tools sequentially against the live MrScraper
|
|
583
|
+
service using the manual runner. Use `-- --url URL --query QUERY` to customize
|
|
584
|
+
the targets, or `-- --target http://127.0.0.1:8000/mcp` to use HTTP.
|
|
585
|
+
|
|
586
|
+
## MCP Registry
|
|
587
|
+
|
|
588
|
+
The hosted server's MCP Registry manifest is defined in
|
|
589
|
+
[`server.json`](server.json). It describes the Streamable HTTP endpoint and
|
|
590
|
+
links to this repository and the setup documentation.
|
|
591
|
+
|
|
592
|
+
See the [official registry publishing guide](https://modelcontextprotocol.io/registry/github-actions)
|
|
593
|
+
for GitHub Actions authentication and publication. Registry versions are
|
|
594
|
+
immutable, so use a new version in `server.json` when publishing changed
|
|
595
|
+
metadata. Publishing this manifest registers the hosted server; it does not
|
|
596
|
+
publish a new npm package.
|
|
720
597
|
|
|
721
598
|
## License
|
|
722
599
|
|