@mrscraper/mcp 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +642 -0
- package/dist/api.d.ts +102 -0
- package/dist/api.d.ts.map +1 -0
- package/dist/api.js +203 -0
- package/dist/api.js.map +1 -0
- package/dist/app.d.ts +17 -0
- package/dist/app.d.ts.map +1 -0
- package/dist/app.js +123 -0
- package/dist/app.js.map +1 -0
- package/dist/auth.d.ts +7 -0
- package/dist/auth.d.ts.map +1 -0
- package/dist/auth.js +64 -0
- package/dist/auth.js.map +1 -0
- package/dist/bin.d.ts +3 -0
- package/dist/bin.d.ts.map +1 -0
- package/dist/bin.js +8 -0
- package/dist/bin.js.map +1 -0
- package/dist/compliance.d.ts +4 -0
- package/dist/compliance.d.ts.map +1 -0
- package/dist/compliance.js +10 -0
- package/dist/compliance.js.map +1 -0
- package/dist/config.d.ts +28 -0
- package/dist/config.d.ts.map +1 -0
- package/dist/config.js +63 -0
- package/dist/config.js.map +1 -0
- package/dist/http.d.ts +19 -0
- package/dist/http.d.ts.map +1 -0
- package/dist/http.js +156 -0
- package/dist/http.js.map +1 -0
- package/dist/index.d.ts +11 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +11 -0
- package/dist/index.js.map +1 -0
- package/dist/runtime.d.ts +8 -0
- package/dist/runtime.d.ts.map +1 -0
- package/dist/runtime.js +46 -0
- package/dist/runtime.js.map +1 -0
- package/dist/server.d.ts +9 -0
- package/dist/server.d.ts.map +1 -0
- package/dist/server.js +24 -0
- package/dist/server.js.map +1 -0
- package/dist/status.d.ts +21 -0
- package/dist/status.d.ts.map +1 -0
- package/dist/status.js +62 -0
- package/dist/status.js.map +1 -0
- package/dist/tools.d.ts +122 -0
- package/dist/tools.d.ts.map +1 -0
- package/dist/tools.js +664 -0
- package/dist/tools.js.map +1 -0
- package/package.json +77 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 MrScraper
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,642 @@
|
|
|
1
|
+
# MrScraper MCP
|
|
2
|
+
|
|
3
|
+
MrScraper MCP exposes the MrScraper web-data service through the
|
|
4
|
+
[Model Context Protocol](https://modelcontextprotocol.io). Agents can fetch a
|
|
5
|
+
known page, extract structured records, search Google, inspect account usage,
|
|
6
|
+
rerun saved scrapers, and read stored results.
|
|
7
|
+
|
|
8
|
+
The server exposes seven web-data tools:
|
|
9
|
+
|
|
10
|
+
```text
|
|
11
|
+
fetch scrape serp status rerun results result
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Deployment options
|
|
15
|
+
|
|
16
|
+
| Option | Endpoint or command | Best for |
|
|
17
|
+
| ----------- | --------------------------------------------- | --------------------------------------- |
|
|
18
|
+
| Hosted | `https://mcp.mrscraper.com/mcp` | A managed Streamable HTTP connection |
|
|
19
|
+
| Local HTTP | `TRANSPORT=http npx -y @mrscraper/mcp@latest` | A self-managed HTTP service |
|
|
20
|
+
| Local stdio | `npx -y @mrscraper/mcp@latest` | MCP clients that launch a local process |
|
|
21
|
+
|
|
22
|
+
All modes use a MrScraper API key from
|
|
23
|
+
[app.mrscraper.com/api-tokens](https://app.mrscraper.com/api-tokens).
|
|
24
|
+
|
|
25
|
+
## Connect to the hosted server
|
|
26
|
+
|
|
27
|
+
Configure a Streamable HTTP MCP connection and send the API key as a bearer
|
|
28
|
+
token:
|
|
29
|
+
|
|
30
|
+
```json
|
|
31
|
+
{
|
|
32
|
+
"mcpServers": {
|
|
33
|
+
"mrscraper": {
|
|
34
|
+
"type": "http",
|
|
35
|
+
"url": "https://mcp.mrscraper.com/mcp",
|
|
36
|
+
"headers": {
|
|
37
|
+
"Authorization": "Bearer YOUR_MRSCRAPER_API_KEY"
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Use the environment-variable or secret-storage feature provided by your MCP
|
|
45
|
+
client so the key remains outside project files.
|
|
46
|
+
|
|
47
|
+
### Codex
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY"
|
|
51
|
+
codex mcp add mrscraper \
|
|
52
|
+
--url https://mcp.mrscraper.com/mcp \
|
|
53
|
+
--bearer-token-env-var MRSCRAPER_API_KEY
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Start Codex from an environment containing `MRSCRAPER_API_KEY`. Codex stores
|
|
57
|
+
the environment-variable name in its MCP configuration.
|
|
58
|
+
|
|
59
|
+
### Claude Code
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
claude mcp add \
|
|
63
|
+
--transport http \
|
|
64
|
+
--scope user \
|
|
65
|
+
--header "Authorization: Bearer YOUR_MRSCRAPER_API_KEY" \
|
|
66
|
+
mrscraper https://mcp.mrscraper.com/mcp
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Claude Code also supports project and local scopes. Choose the scope that
|
|
70
|
+
matches how broadly the connection should be available.
|
|
71
|
+
|
|
72
|
+
## Copyable agent setup prompt
|
|
73
|
+
|
|
74
|
+
```text
|
|
75
|
+
Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp. Use a MrScraper API key from https://app.mrscraper.com/api-tokens, store it through the client's environment-variable or secret-storage mechanism, and send it as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that fetch, scrape, serp, status, rerun, results, and result are available.
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Choosing a tool
|
|
79
|
+
|
|
80
|
+
| Tool | Choose it when |
|
|
81
|
+
| --------- | --------------------------------------------------------------------------------- |
|
|
82
|
+
| `fetch` | You already have a URL and need the page response from Web Unblocker. |
|
|
83
|
+
| `scrape` | You already have a URL and need defined fields, records, listings, or a site map. |
|
|
84
|
+
| `serp` | You are starting from a Google query or Google search URL. |
|
|
85
|
+
| `status` | You need subscription, quota, token usage, or domain request outcomes. |
|
|
86
|
+
| `rerun` | You have a saved AI or manual scraper ID and want to run it again. |
|
|
87
|
+
| `results` | You need a paginated or filtered list of stored results. |
|
|
88
|
+
| `result` | You know one result ID and need its complete stored record. |
|
|
89
|
+
|
|
90
|
+
A common discovery workflow is `serp` → `fetch` or `scrape`. A saved
|
|
91
|
+
scraper workflow is `scrape` → `rerun` → `result`.
|
|
92
|
+
|
|
93
|
+
## Response contract
|
|
94
|
+
|
|
95
|
+
API-backed tools return a response envelope in both MCP `structuredContent`
|
|
96
|
+
and a formatted JSON text block:
|
|
97
|
+
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"status_code": 200,
|
|
101
|
+
"data": {},
|
|
102
|
+
"headers": {
|
|
103
|
+
"content-type": "application/json"
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
| Field | Type | Meaning |
|
|
109
|
+
| ------------- | ----------------------------- | ------------------------------------------------------------------------------------ |
|
|
110
|
+
| `status_code` | number or `null` | Upstream HTTP status, or `null` when a request ends before an HTTP response arrives. |
|
|
111
|
+
| `data` | JSON value, string, or `null` | Parsed JSON or response text supplied by the MrScraper service. |
|
|
112
|
+
| `headers` | object | Response headers with credential-bearing headers filtered out. |
|
|
113
|
+
| `error` | string, when present | Request failure summary. |
|
|
114
|
+
|
|
115
|
+
An API failure keeps this envelope in `structuredContent` and sets the MCP
|
|
116
|
+
result's `isError` flag. Input-contract errors are returned as MCP tool
|
|
117
|
+
errors. Parsed credential metadata and generated curl credentials are
|
|
118
|
+
sanitized while extracted scraper data remains available to the caller.
|
|
119
|
+
|
|
120
|
+
## Tool reference
|
|
121
|
+
|
|
122
|
+
### `fetch`
|
|
123
|
+
|
|
124
|
+
`fetch` calls the Web Unblocker endpoint once:
|
|
125
|
+
|
|
126
|
+
```text
|
|
127
|
+
GET https://api.mrscraper.com/
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Start with the URL alone:
|
|
131
|
+
|
|
132
|
+
```json
|
|
133
|
+
{
|
|
134
|
+
"url": "https://example.com/products"
|
|
135
|
+
}
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Use browser rendering for JavaScript-driven content:
|
|
139
|
+
|
|
140
|
+
```json
|
|
141
|
+
{
|
|
142
|
+
"url": "https://example.com/products",
|
|
143
|
+
"browser_rendering": true,
|
|
144
|
+
"wait_for_selector": ".product-card",
|
|
145
|
+
"geo_code": "ID",
|
|
146
|
+
"timeout": 45
|
|
147
|
+
}
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
| Input | Required | Default | API mapping | Purpose |
|
|
151
|
+
| ------------------- | -------- | ------- | ------------------------ | ------------------------------------------------------------------------------------ |
|
|
152
|
+
| `url` | Yes | — | Query `url` | Target page URL. |
|
|
153
|
+
| `browser_rendering` | No | `false` | Query `browserRendering` | Executes page JavaScript in a browser. |
|
|
154
|
+
| `geo_code` | No | omitted | Query `geoCode` | Selects proxy-country routing. |
|
|
155
|
+
| `wait_for_selector` | No | omitted | Query `waitForSelector` | Waits for a CSS selector together with `browser_rendering: true`. |
|
|
156
|
+
| `home_page` | No | `false` | Query `homePage` | Visits the site root before the target URL. |
|
|
157
|
+
| `block_resources` | No | `false` | Query `blockResources` | Applies resource blocking during page loading. |
|
|
158
|
+
| `max_retries` | No | `3` | Query `maxRetries` | Sets the Web Unblocker retry limit; zero is accepted. |
|
|
159
|
+
| `token_cap` | No | omitted | Query `tokenCap` | Sets the retry token budget. |
|
|
160
|
+
| `timeout` | No | `30` | Query `timeout` | Sets the page-load deadline in seconds; transport receives an additional 30 seconds. |
|
|
161
|
+
|
|
162
|
+
The response body's value is returned in `data`, commonly as HTML.
|
|
163
|
+
|
|
164
|
+
### `scrape`
|
|
165
|
+
|
|
166
|
+
`scrape` creates an AI scraper through:
|
|
167
|
+
|
|
168
|
+
```text
|
|
169
|
+
POST https://api.app.mrscraper.com/api/v1/scrapers-ai
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Choose an agent based on the extraction shape:
|
|
173
|
+
|
|
174
|
+
| Agent | Designed for | Available inputs |
|
|
175
|
+
| --------- | ------------------------------------- | ------------------------------------------------------------------------- |
|
|
176
|
+
| `general` | Defined fields from a page | `prompt`, `schema_prompt`, `proxy_country` |
|
|
177
|
+
| `listing` | Repeated records across listing pages | `prompt`, `schema_prompt`, `proxy_country`, `max_pages` |
|
|
178
|
+
| `map` | URL discovery across a site | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` |
|
|
179
|
+
|
|
180
|
+
General extraction:
|
|
181
|
+
|
|
182
|
+
```json
|
|
183
|
+
{
|
|
184
|
+
"url": "https://example.com/product",
|
|
185
|
+
"agent": "general",
|
|
186
|
+
"prompt": "Extract the product name, price, availability, and image URLs"
|
|
187
|
+
}
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Listing extraction:
|
|
191
|
+
|
|
192
|
+
```json
|
|
193
|
+
{
|
|
194
|
+
"url": "https://example.com/products",
|
|
195
|
+
"agent": "listing",
|
|
196
|
+
"prompt": "Extract every product name, price, and detail URL",
|
|
197
|
+
"max_pages": 5
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Site map:
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"url": "https://example.com",
|
|
206
|
+
"agent": "map",
|
|
207
|
+
"max_depth": 2,
|
|
208
|
+
"max_pages": 50,
|
|
209
|
+
"limit": 1000,
|
|
210
|
+
"include_patterns": "/products/",
|
|
211
|
+
"exclude_patterns": "/cart/"
|
|
212
|
+
}
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
Best-effort schema guidance:
|
|
216
|
+
|
|
217
|
+
```json
|
|
218
|
+
{
|
|
219
|
+
"url": "https://example.com/product",
|
|
220
|
+
"prompt": "Extract the product",
|
|
221
|
+
"schema_prompt": {
|
|
222
|
+
"type": "object",
|
|
223
|
+
"properties": {
|
|
224
|
+
"name": { "type": "string" },
|
|
225
|
+
"price": { "type": "number" }
|
|
226
|
+
},
|
|
227
|
+
"required": ["name", "price"]
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
`schema_prompt` is appended to the natural-language instruction. Treat it as
|
|
233
|
+
shape guidance and validate returned data in the consuming application when
|
|
234
|
+
strict conformance is required.
|
|
235
|
+
|
|
236
|
+
| Input | Required | Default | Request mapping | Purpose |
|
|
237
|
+
| ------------------ | --------------- | --------------- | ---------------------- | ----------------------------------------------------------- |
|
|
238
|
+
| `url` | Yes | — | Body `url` | Target URL for every agent. |
|
|
239
|
+
| `prompt` | General/listing | — | Body `message` | Natural-language extraction instructions. |
|
|
240
|
+
| `schema_prompt` | No | omitted | Appended to `message` | Best-effort JSON Schema shape guidance for general/listing. |
|
|
241
|
+
| `agent` | No | `general` | Body `agent` | Selects `general`, `listing`, or `map`. |
|
|
242
|
+
| `proxy_country` | No | omitted | Body `proxyCountry` | Proxy country for general/listing. |
|
|
243
|
+
| `max_pages` | No | service default | Body `maxPages` | Page bound for listing/map. |
|
|
244
|
+
| `max_depth` | No | service default | Body `maxDepth` | Link-depth bound for map. |
|
|
245
|
+
| `limit` | No | service default | Body `limit` | URL-result bound for map. |
|
|
246
|
+
| `include_patterns` | No | service default | Body `includePatterns` | URL inclusion expression for map. |
|
|
247
|
+
| `exclude_patterns` | No | service default | Body `excludePatterns` | URL exclusion expression for map. |
|
|
248
|
+
|
|
249
|
+
General and listing send `url`, `message`, and `agent`, plus the supplied
|
|
250
|
+
agent inputs. Map sends `url`, `agent`, and only the supplied crawl inputs.
|
|
251
|
+
|
|
252
|
+
#### Reproduce a scrape with `rerun`
|
|
253
|
+
|
|
254
|
+
Every successful `scrape` creates a saved AI scraper configuration by default.
|
|
255
|
+
Its response run object contains `scraperId`. Pass that UUID to `rerun` as
|
|
256
|
+
`scraper_id` to apply the same saved extraction configuration to the original
|
|
257
|
+
URL or another URL without rebuilding the prompt and agent settings:
|
|
258
|
+
|
|
259
|
+
```json
|
|
260
|
+
{
|
|
261
|
+
"target": "https://example.com/product-2",
|
|
262
|
+
"type": "ai",
|
|
263
|
+
"scraper_id": "scraper-uuid"
|
|
264
|
+
}
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
This makes the scraper configuration reproducible, but it does not guarantee
|
|
268
|
+
identical extracted values when the page or model behavior changes.
|
|
269
|
+
|
|
270
|
+
`rerun` also supports dashboard-built manual workflows and asynchronous bulk
|
|
271
|
+
jobs across multiple target URLs. See the full [`rerun`](#rerun) section below
|
|
272
|
+
for the available modes and result-tracking workflow.
|
|
273
|
+
|
|
274
|
+
### `serp`
|
|
275
|
+
|
|
276
|
+
`serp` calls the synchronous Google endpoint:
|
|
277
|
+
|
|
278
|
+
```text
|
|
279
|
+
POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
Search from a query:
|
|
283
|
+
|
|
284
|
+
```json
|
|
285
|
+
{
|
|
286
|
+
"query_or_url": "iphone 17",
|
|
287
|
+
"region": "id",
|
|
288
|
+
"language": "id",
|
|
289
|
+
"page": 2,
|
|
290
|
+
"format": "json"
|
|
291
|
+
}
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
A Google search URL can supply the query, locale, and page:
|
|
295
|
+
|
|
296
|
+
```json
|
|
297
|
+
{
|
|
298
|
+
"query_or_url": "https://www.google.com/search?q=iphone+17&gl=us&hl=en&start=20"
|
|
299
|
+
}
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
The server derives `query` from `q`, `region` from `gl`, `language`
|
|
303
|
+
from `hl`, and a one-based page number from `start). Explicit tool inputs
|
|
304
|
+
take priority over URL-derived values.
|
|
305
|
+
|
|
306
|
+
| Input | Required | Default | Request mapping | Purpose |
|
|
307
|
+
| ---------------- | -------- | -------------------- | ---------------------- | -------------------------------------------- |
|
|
308
|
+
| `query_or_url` | Yes | — | Body `query` | Google query or Google search URL. |
|
|
309
|
+
| `region` | No | URL value or omitted | Body `region` | Result country. |
|
|
310
|
+
| `language` | No | URL value or omitted | Body `language` | Result language. |
|
|
311
|
+
| `page` | No | URL value or omitted | Body `page` | One-based result page. |
|
|
312
|
+
| `format` | No | `json` | Body `format` | Selects parsed JSON or result-page HTML. |
|
|
313
|
+
| `render_js` | No | `false` | Body `renderJs` | Waits for JavaScript-rendered SERP features. |
|
|
314
|
+
| `raw` | No | `false` | Body `format=html` | Compatibility alias for HTML output. |
|
|
315
|
+
| `client_timeout` | No | `120` | Local request deadline | Sets the upstream HTTP timeout in seconds. |
|
|
316
|
+
|
|
317
|
+
### `status`
|
|
318
|
+
|
|
319
|
+
`status` always reads account data from:
|
|
320
|
+
|
|
321
|
+
```text
|
|
322
|
+
GET https://api.app.mrscraper.com/api/v1/subscription-accounts
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
With `domain`, it also reads request-outcome analytics from:
|
|
326
|
+
|
|
327
|
+
```text
|
|
328
|
+
GET https://api.app.mrscraper.com/api/v1/analytic/statuses
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
```json
|
|
332
|
+
{
|
|
333
|
+
"domain": "https://www.example.com/products",
|
|
334
|
+
"from": "7d",
|
|
335
|
+
"to": "now",
|
|
336
|
+
"action": "fetch",
|
|
337
|
+
"api_token_name": "production"
|
|
338
|
+
}
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
| Input | Required | Default | Purpose |
|
|
342
|
+
| ---------------- | -------- | ------------ | ------------------------------------------------------------------------------------------- |
|
|
343
|
+
| `domain` | No | omitted | Adds request-outcome analytics for a hostname or URL. |
|
|
344
|
+
| `from` | No | `24h` | Range start as ISO 8601, `now`, or a relative duration such as `30m`, `24h`, `7d`, or `2w`. |
|
|
345
|
+
| `to` | No | `now` | Range end using the same date syntax. |
|
|
346
|
+
| `action` | No | empty filter | Filters analytics by exact action. |
|
|
347
|
+
| `api_token_name` | No | empty filter | Filters analytics by API-token name. |
|
|
348
|
+
|
|
349
|
+
Successful output is a normalized account and analytics summary:
|
|
350
|
+
|
|
351
|
+
```json
|
|
352
|
+
{
|
|
353
|
+
"kind": "mrscraper-cli-status-summary",
|
|
354
|
+
"source_endpoints": ["/subscription-accounts", "/analytic/statuses"],
|
|
355
|
+
"status_code": 200,
|
|
356
|
+
"data": {
|
|
357
|
+
"account": {
|
|
358
|
+
"subscription_status": "active",
|
|
359
|
+
"enterprise": false,
|
|
360
|
+
"token_usage": 250,
|
|
361
|
+
"token_limit": 1000,
|
|
362
|
+
"token_remaining": 750,
|
|
363
|
+
"usage_percent": 25,
|
|
364
|
+
"rate_limit": 10,
|
|
365
|
+
"rate_ttl": 60,
|
|
366
|
+
"auto_renew": true,
|
|
367
|
+
"ends_at": null,
|
|
368
|
+
"user": {
|
|
369
|
+
"name": "Ada",
|
|
370
|
+
"email": "ada@example.com",
|
|
371
|
+
"verified": true
|
|
372
|
+
}
|
|
373
|
+
},
|
|
374
|
+
"analytics": {
|
|
375
|
+
"domain": "www.example.com",
|
|
376
|
+
"from": "2026-08-11 00:00:00 UTC",
|
|
377
|
+
"to": "2026-08-18 00:00:00 UTC"
|
|
378
|
+
}
|
|
379
|
+
}
|
|
380
|
+
}
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
The summary calculates `token_remaining` and `usage_percent`, normalizes a
|
|
384
|
+
URL to its hostname, and records the source endpoints used for the response.
|
|
385
|
+
|
|
386
|
+
### `rerun`
|
|
387
|
+
|
|
388
|
+
`type` and `bulk` describe different parts of the request:
|
|
389
|
+
|
|
390
|
+
- Set `type` to `ai` for a saved AI scraper created by the `scrape` tool.
|
|
391
|
+
- Set `type` to `manual` for a saved step-based workflow created in the
|
|
392
|
+
MrScraper dashboard. MCP reruns that workflow but does not create it.
|
|
393
|
+
- Leave `bulk` as `false` for one URL. Set it to `true` to apply the same saved
|
|
394
|
+
configuration to a comma- or newline-separated URL list in one bulk request.
|
|
395
|
+
|
|
396
|
+
Manual reruns can be single or bulk, and bulk reruns can use either scraper
|
|
397
|
+
type. Bulk describes the number of targets; manual describes how the saved
|
|
398
|
+
scraper was built.
|
|
399
|
+
|
|
400
|
+
Bulk mode submits one asynchronous backend job; MCP does not repeat the
|
|
401
|
+
single-URL tool call locally. Save `data.data.bulkResultId` from the response
|
|
402
|
+
and pass it to `result` as `result_id` until the stored result is finished.
|
|
403
|
+
|
|
404
|
+
`rerun` routes one tool contract to four saved-scraper endpoints:
|
|
405
|
+
|
|
406
|
+
| Mode | Endpoint | ID input | Target format | Crawl controls |
|
|
407
|
+
| ------------- | ---------------------------------- | ------------ | ---------------------------- | ------------------- |
|
|
408
|
+
| Single AI | `POST /scrapers-ai-rerun` | `scraper_id` | One URL | Available |
|
|
409
|
+
| Bulk AI | `POST /scrapers-ai-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
|
|
410
|
+
| Single manual | `POST /scrapers-manual-rerun` | `scraper_id` | One URL | Saved configuration |
|
|
411
|
+
| Bulk manual | `POST /scrapers-manual-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
|
|
412
|
+
|
|
413
|
+
Single AI rerun:
|
|
414
|
+
|
|
415
|
+
```json
|
|
416
|
+
{
|
|
417
|
+
"target": "https://example.com/products",
|
|
418
|
+
"type": "ai",
|
|
419
|
+
"scraper_id": "scraper-uuid",
|
|
420
|
+
"max_depth": 2,
|
|
421
|
+
"max_pages": 50,
|
|
422
|
+
"limit": 1000,
|
|
423
|
+
"include_patterns": "/products/",
|
|
424
|
+
"exclude_patterns": "/cart/"
|
|
425
|
+
}
|
|
426
|
+
```
|
|
427
|
+
|
|
428
|
+
Bulk manual rerun:
|
|
429
|
+
|
|
430
|
+
```json
|
|
431
|
+
{
|
|
432
|
+
"target": "https://example.com/a,https://example.com/b\nhttps://example.com/c",
|
|
433
|
+
"type": "manual",
|
|
434
|
+
"bulk": true,
|
|
435
|
+
"id": "scraper-uuid"
|
|
436
|
+
}
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
| Input | Required | Default | Purpose |
|
|
440
|
+
| ------------------ | ----------- | ------------ | --------------------------------------------------------------- |
|
|
441
|
+
| `target` | Yes | — | One URL, or a comma/newline-separated URL string for bulk mode. |
|
|
442
|
+
| `type` | Yes | — | Selects `ai` or `manual`. |
|
|
443
|
+
| `bulk` | No | `false` | Selects a bulk endpoint. |
|
|
444
|
+
| `scraper_id` | Single mode | — | Saved scraper UUID for one URL. |
|
|
445
|
+
| `id` | Bulk mode | — | Saved scraper UUID for the bulk URL list. |
|
|
446
|
+
| `max_depth` | Single AI | `2` | Crawl depth. |
|
|
447
|
+
| `max_pages` | Single AI | `50` | Page bound. |
|
|
448
|
+
| `limit` | Single AI | `1000` | Result bound. |
|
|
449
|
+
| `include_patterns` | Single AI | empty string | URL inclusion expression. |
|
|
450
|
+
| `exclude_patterns` | Single AI | empty string | URL exclusion expression. |
|
|
451
|
+
|
|
452
|
+
Manual reruns carry a compliance acknowledgment in the MCP server
|
|
453
|
+
instructions. MCP clients should present that acknowledgment before executing
|
|
454
|
+
the manual mode.
|
|
455
|
+
|
|
456
|
+
### `results`
|
|
457
|
+
|
|
458
|
+
`results` reads the stored result collection:
|
|
459
|
+
|
|
460
|
+
```text
|
|
461
|
+
GET https://api.app.mrscraper.com/api/v1/results
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
```json
|
|
465
|
+
{
|
|
466
|
+
"sort_field": "updatedAt",
|
|
467
|
+
"sort_order": "desc",
|
|
468
|
+
"page_size": 25,
|
|
469
|
+
"page": 1,
|
|
470
|
+
"search": "example.com",
|
|
471
|
+
"date_range_column": "updatedAt",
|
|
472
|
+
"start_at": "2026-08-01T00:00:00Z",
|
|
473
|
+
"end_at": "2026-08-18T23:59:59Z"
|
|
474
|
+
}
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
| Input | Required | Default | Query mapping | Purpose |
|
|
478
|
+
| ------------------- | -------- | ----------- | ----------------- | ------------------------------------------------------------- |
|
|
479
|
+
| `sort_field` | No | `updatedAt` | `sortField` | Field used by the results API for sorting. |
|
|
480
|
+
| `sort_order` | No | `desc` | `sortOrder` | Case-insensitive `asc` or `desc`; sent upstream in uppercase. |
|
|
481
|
+
| `page_size` | No | `10` | `pageSize` | Number of rows per page. |
|
|
482
|
+
| `page` | No | `1` | `page` | One-based page index. |
|
|
483
|
+
| `search` | No | omitted | `search` | Free-text result filter. |
|
|
484
|
+
| `date_range_column` | No | omitted | `dateRangeColumn` | Column used by the date range. |
|
|
485
|
+
| `start_at` | No | omitted | `startAt` | Inclusive range start. |
|
|
486
|
+
| `end_at` | No | omitted | `endAt` | Inclusive range end. |
|
|
487
|
+
|
|
488
|
+
### `result`
|
|
489
|
+
|
|
490
|
+
`result` reads one stored result:
|
|
491
|
+
|
|
492
|
+
```text
|
|
493
|
+
GET https://api.app.mrscraper.com/api/v1/results/{result_id}
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
```json
|
|
497
|
+
{
|
|
498
|
+
"result_id": "result-uuid"
|
|
499
|
+
}
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
| Input | Required | Purpose |
|
|
503
|
+
| ----------- | -------- | ----------------------------- |
|
|
504
|
+
| `result_id` | Yes | Stored MrScraper result UUID. |
|
|
505
|
+
|
|
506
|
+
## Run the server locally
|
|
507
|
+
|
|
508
|
+
Node.js 20 or newer is required.
|
|
509
|
+
|
|
510
|
+
```sh
|
|
511
|
+
git clone https://github.com/mrscraper-com/mrscraper-mcp.git
|
|
512
|
+
cd mrscraper-mcp
|
|
513
|
+
npm ci
|
|
514
|
+
npm run build
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
### Streamable HTTP
|
|
518
|
+
|
|
519
|
+
```sh
|
|
520
|
+
TRANSPORT=http npm start
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
The default endpoint is `http://127.0.0.1:8000/mcp`.
|
|
524
|
+
|
|
525
|
+
```json
|
|
526
|
+
{
|
|
527
|
+
"mcpServers": {
|
|
528
|
+
"mrscraper": {
|
|
529
|
+
"type": "http",
|
|
530
|
+
"url": "http://127.0.0.1:8000/mcp",
|
|
531
|
+
"headers": {
|
|
532
|
+
"Authorization": "Bearer YOUR_MRSCRAPER_API_KEY"
|
|
533
|
+
}
|
|
534
|
+
}
|
|
535
|
+
}
|
|
536
|
+
}
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
### stdio
|
|
540
|
+
|
|
541
|
+
```json
|
|
542
|
+
{
|
|
543
|
+
"mcpServers": {
|
|
544
|
+
"mrscraper": {
|
|
545
|
+
"command": "npx",
|
|
546
|
+
"args": ["-y", "@mrscraper/mcp@latest"],
|
|
547
|
+
"env": {
|
|
548
|
+
"MRSCRAPER_API_KEY": "YOUR_MRSCRAPER_API_KEY"
|
|
549
|
+
}
|
|
550
|
+
}
|
|
551
|
+
}
|
|
552
|
+
}
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
Stdio credential precedence is `MRSCRAPER_API_KEY`, then
|
|
556
|
+
`MRSCRAPER_API_TOKEN`.
|
|
557
|
+
|
|
558
|
+
## Authentication and security
|
|
559
|
+
|
|
560
|
+
| Transport | Credential source |
|
|
561
|
+
| -------------------- | ----------------------------------------------------------------- |
|
|
562
|
+
| Hosted or local HTTP | `Authorization: Bearer <MrScraper API key>` on the MCP connection |
|
|
563
|
+
| Local stdio | `MRSCRAPER_API_KEY`, then `MRSCRAPER_API_TOKEN` |
|
|
564
|
+
|
|
565
|
+
HTTP bearer tokens are validated against the MrScraper account endpoint before
|
|
566
|
+
tool execution. Each HTTP request uses its caller's credential.
|
|
567
|
+
|
|
568
|
+
The server filters credential-bearing response headers. Parsed JSON credential
|
|
569
|
+
metadata and credentials embedded in generated curl commands are redacted.
|
|
570
|
+
Scraper run extraction values remain available in `data`.
|
|
571
|
+
|
|
572
|
+
Browser-origin requests are accepted from trusted local origins and exact
|
|
573
|
+
origins configured through `MRSCRAPER_ALLOWED_ORIGINS`. Service-to-service
|
|
574
|
+
MCP clients typically connect without an `Origin` header.
|
|
575
|
+
|
|
576
|
+
## Docker
|
|
577
|
+
|
|
578
|
+
```sh
|
|
579
|
+
docker build -f docker/Dockerfile -t mrscraper-mcp .
|
|
580
|
+
docker run --rm -p 8000:8000 mrscraper-mcp
|
|
581
|
+
```
|
|
582
|
+
|
|
583
|
+
The image binds to `0.0.0.0`. Apply the network controls appropriate for the
|
|
584
|
+
deployment and keep bearer authentication enabled.
|
|
585
|
+
|
|
586
|
+
## Environment variables
|
|
587
|
+
|
|
588
|
+
| Variable | Default | Purpose |
|
|
589
|
+
| -------------------------------- | --------------------------------- | ---------------------------------------------------------------- |
|
|
590
|
+
| `TRANSPORT` | `stdio` | Selects `stdio` or `http`. |
|
|
591
|
+
| `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. |
|
|
592
|
+
| `PORT` | `8000` | HTTP listen port. |
|
|
593
|
+
| `MRSCRAPER_API_KEY` | — | Primary stdio credential. |
|
|
594
|
+
| `MRSCRAPER_API_TOKEN` | — | Legacy stdio credential alias. |
|
|
595
|
+
| `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. |
|
|
596
|
+
| `MRSCRAPER_ALLOWED_ORIGINS` | — | Comma-separated browser origins allowed to call the HTTP server. |
|
|
597
|
+
| `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. |
|
|
598
|
+
| `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Fetch endpoint override. |
|
|
599
|
+
| `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | SERP endpoint override. |
|
|
600
|
+
| `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. |
|
|
601
|
+
| `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
|
|
602
|
+
|
|
603
|
+
## Troubleshooting
|
|
604
|
+
|
|
605
|
+
- **Connection error:** Confirm the server URL ends in `/mcp` and reload the
|
|
606
|
+
MCP client after configuration changes.
|
|
607
|
+
- **401 Unauthorized:** Confirm the bearer key for HTTP or the environment key
|
|
608
|
+
for stdio.
|
|
609
|
+
- **403 Forbidden Origin:** Add the exact browser origin to
|
|
610
|
+
`MRSCRAPER_ALLOWED_ORIGINS`.
|
|
611
|
+
- **Tool input error:** Compare the call with the tool's parameter table and
|
|
612
|
+
the selected scrape or rerun mode.
|
|
613
|
+
- **Upstream timeout:** Increase `fetch.timeout` or `serp.client_timeout`
|
|
614
|
+
when the target requires a longer request window.
|
|
615
|
+
- **New tools are missing:** Start a new client session and inspect
|
|
616
|
+
`tools/list` for all seven names.
|
|
617
|
+
|
|
618
|
+
## Development
|
|
619
|
+
|
|
620
|
+
```sh
|
|
621
|
+
npm ci
|
|
622
|
+
npm run format:check
|
|
623
|
+
npm run lint
|
|
624
|
+
npm test
|
|
625
|
+
npm run build
|
|
626
|
+
npm pack --dry-run
|
|
627
|
+
```
|
|
628
|
+
|
|
629
|
+
Smoke-test a running HTTP server:
|
|
630
|
+
|
|
631
|
+
```sh
|
|
632
|
+
npm run test:mcp -- \
|
|
633
|
+
--target http://127.0.0.1:8000/mcp \
|
|
634
|
+
--token "$MRSCRAPER_API_KEY"
|
|
635
|
+
```
|
|
636
|
+
|
|
637
|
+
`npm run test:e2e` exercises all seven tools against the live MrScraper
|
|
638
|
+
service and creates stored scraper results.
|
|
639
|
+
|
|
640
|
+
## License
|
|
641
|
+
|
|
642
|
+
MIT — see [LICENSE](./LICENSE).
|