@mrscraper/mcp 0.1.3 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,14 +1,21 @@
1
- # MrScraper MCP
1
+ # MrScraper MCP Server for Web Scraping and Google Search
2
2
 
3
- MrScraper MCP exposes the MrScraper web-data service through the
4
- [Model Context Protocol](https://modelcontextprotocol.io). Agents can fetch a
5
- known page, extract structured records, search Google, inspect account usage,
6
- rerun saved scrapers, and read stored results.
3
+ MrScraper MCP is the official [MrScraper](https://mrscraper.com) server for the
4
+ [Model Context Protocol](https://modelcontextprotocol.io). It connects AI
5
+ assistants to page content and Google search through two SDK-compatible tools,
6
+ plus an account check. Retrieve HTML, Markdown, or screenshots with
7
+ `scraperAPI`, discover pages with `webSearch`, and check your plan and token
8
+ usage with `status`.
7
9
 
8
- The server exposes seven web-data tools:
10
+ Use the hosted server with OAuth sign-in, or run the open-source server locally
11
+ with the [`@mrscraper/mcp` npm package](https://www.npmjs.com/package/@mrscraper/mcp).
12
+ See the [MrScraper MCP documentation](https://docs.mrscraper.com/docs/getting-started/mcp-server)
13
+ for setup instructions and client configurations.
14
+
15
+ The server exposes three tools:
9
16
 
10
17
  ```text
11
- fetch scrape serp status rerun results result
18
+ scraperAPI webSearch status
12
19
  ```
13
20
 
14
21
  ## Deployment options
@@ -54,11 +61,10 @@ the address will be refused.
54
61
 
55
62
  ### What you are approving
56
63
 
57
- | Scope | Grants |
58
- | -------------- | --------------------------------------------------------- |
59
- | `scrape:read` | `fetch`, `serp`, `results`, `result` |
60
- | `scrape:write` | `scrape`, `rerun`: creating scrapers and running them |
61
- | `account:read` | `status`: subscription plan, quota, and request analytics |
64
+ | Scope | Grants |
65
+ | -------------- | ---------------------------- |
66
+ | `scrape:read` | `scraperAPI` and `webSearch` |
67
+ | `account:read` | `status` |
62
68
 
63
69
  A tool called without its scope is refused with `403 insufficient_scope`, and
64
70
  your client will offer to grant the missing permission.
@@ -118,62 +124,51 @@ matches how broadly the connection should be available.
118
124
  ## Copyable agent setup prompt
119
125
 
120
126
  ```text
121
- Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that fetch, scrape, serp, status, rerun, results, and result are available.
127
+ Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that scraperAPI, webSearch, and status are the only available tools.
122
128
  ```
123
129
 
124
130
  ## Choosing a tool
125
131
 
126
- | Tool | Choose it when |
127
- | --------- | --------------------------------------------------------------------------------- |
128
- | `fetch` | You already have a URL and need the page response from Web Unblocker. |
129
- | `scrape` | You already have a URL and need defined fields, records, listings, or a site map. |
130
- | `serp` | You are starting from a Google query or Google search URL. |
131
- | `status` | You need subscription, quota, token usage, or domain request outcomes. |
132
- | `rerun` | You have a saved AI or manual scraper ID and want to run it again. |
133
- | `results` | You need a paginated or filtered list of stored results. |
134
- | `result` | You know one result ID and need its complete stored record. |
132
+ | Tool | Choose it when |
133
+ | ------------ | ---------------------------------------------------------------------------- |
134
+ | `scraperAPI` | You have a page URL and need HTML, Markdown, a screenshot, or a combination. |
135
+ | `webSearch` | You have a Google query or Google search URL and need search results. |
136
+ | `status` | You need the account's plan, token usage, or remaining tokens. |
135
137
 
136
- A common discovery workflow is `serp` → `fetch` or `scrape`. A saved
137
- scraper workflow is `scrape` → `rerun` → `result`.
138
+ A discovery workflow is `webSearch` → `scraperAPI`. Both web tools advertise
139
+ `readOnlyHint: true`, `openWorldHint: true`, and `destructiveHint: false`;
140
+ `status` is read-only and not open-world. Authentication comes from the MCP
141
+ connection; no tool takes an API token argument.
138
142
 
139
- ## Response contract
143
+ ### Migration from the previous tool set
140
144
 
141
- API-backed tools return a response envelope in both MCP `structuredContent`
142
- and a formatted JSON text block:
145
+ Only `scraperAPI`, `webSearch`, and `status` are registered. The other old tool
146
+ names are removed.
143
147
 
144
- ```json
145
- {
146
- "status_code": 200,
147
- "data": {},
148
- "headers": {
149
- "content-type": "application/json"
150
- }
151
- }
152
- ```
148
+ | Previous call | Replacement |
149
+ | -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
150
+ | `fetch` | `scraperAPI`; uses POST, SDK options, and automatic browser fallback by default. |
151
+ | `serp` | `webSearch`; rename `query_or_url` to `url` and `client_timeout` to `timeout`. Set `raw=false` for JSON. |
152
+ | `status` | `status` for account usage; the `domain`, `from`, `to`, `action`, and `api_token_name` inputs are gone. |
153
+ | `scrape`, `rerun`, `results`, `result` | Removed from this MCP server. |
153
154
 
154
- | Field | Type | Meaning |
155
- | ------------- | ----------------------------- | ------------------------------------------------------------------------------------ |
156
- | `status_code` | number or `null` | Upstream HTTP status, or `null` when a request ends before an HTTP response arrives. |
157
- | `data` | JSON value, string, or `null` | Parsed JSON or response text supplied by the MrScraper service. |
158
- | `headers` | object | Response headers with credential-bearing headers filtered out. |
159
- | `error` | string, when present | Request failure summary. |
160
-
161
- An API failure keeps this envelope in `structuredContent` and sets the MCP
162
- result's `isError` flag. Input-contract errors are returned as MCP tool
163
- errors. Parsed credential metadata and generated curl credentials are
164
- sanitized while extracted scraper data remains available to the caller.
155
+ `super_mode` becomes `super`. `home_page` now maps to `warmStart`, and
156
+ `token_cap=0` is accepted. Restart the server and reconnect the MCP client after
157
+ updating so it refreshes `tools/list`.
165
158
 
166
159
  ## Tool reference
167
160
 
168
- ### `fetch`
161
+ The options and defaults of `scraperAPI` and `webSearch` follow the Python SDK's
162
+ `scraper_api` and `web_search` methods. MCP tool names use camel case; their
163
+ argument names use the SDK's snake case.
169
164
 
170
- `fetch` calls the Web Unblocker endpoint once:
165
+ ### `scraperAPI`
171
166
 
172
167
  ```text
173
- GET https://api.mrscraper.com/
168
+ POST https://api.mrscraper.com/
174
169
  ```
175
170
 
176
- Start with the URL alone:
171
+ Start with a known URL:
177
172
 
178
173
  ```json
179
174
  {
@@ -181,424 +176,242 @@ Start with the URL alone:
181
176
  }
182
177
  ```
183
178
 
184
- Browser loading and real-device routing are independent controls. These four
185
- inputs can return different results for the same URL:
186
-
187
- | Browser rendering | Super Mode | Input | Loading path |
188
- | ----------------- | ---------- | ----------------------------------------------------------------- | -------------------------------------------------------- |
189
- | Off | Off | `{ "url": "URL" }` | Standard routing with the non-browser loader. |
190
- | On | Off | `{ "url": "URL", "browser_rendering": true }` | Standard routing with browser loading and JavaScript. |
191
- | Off | On | `{ "url": "URL", "super_mode": true }` | Real-device routing with the non-browser loader. |
192
- | On | On | `{ "url": "URL", "browser_rendering": true, "super_mode": true }` | Real-device routing with browser loading and JavaScript. |
193
-
194
- Start with both controls off, inspect the response, and change one axis at a
195
- time when needed. Browser rendering is not guaranteed to work better: some
196
- sites fail with it enabled but load without it. Try the remaining combinations
197
- without repeating an identical failed request.
198
-
199
- Use browser rendering for JavaScript-driven content:
179
+ Request HTML, Markdown, and a screenshot together:
200
180
 
201
181
  ```json
202
182
  {
203
- "url": "https://www.scrapethissite.com/pages/ajax-javascript/",
204
- "browser_rendering": true,
205
- "wait_for_selector": "body",
183
+ "url": "https://example.com",
184
+ "html": true,
185
+ "markdown": true,
186
+ "screenshot": true,
187
+ "super": true,
188
+ "home_page": true,
206
189
  "geo_code": "ID",
207
- "timeout": 45
208
- }
209
- ```
210
-
211
- Use both controls for real-device browser loading:
212
-
213
- ```json
214
- {
215
- "url": "https://www.scrapethissite.com/pages/ajax-javascript/",
216
- "browser_rendering": true,
217
- "super_mode": true
218
- }
219
- ```
220
-
221
- | Input | Required | Default | API mapping | Purpose |
222
- | ------------------- | -------- | ------- | ------------------------ | ------------------------------------------------------------------------------------ |
223
- | `url` | Yes | - | Query `url` | Target page URL. |
224
- | `browser_rendering` | No | `false` | Query `browserRendering` | Executes page JavaScript in a browser. |
225
- | `super_mode` | No | `false` | Query `super` | Selects real-device routing independently of browser rendering. |
226
- | `geo_code` | No | omitted | Query `geoCode` | Selects proxy-country routing. |
227
- | `wait_for_selector` | No | omitted | Query `waitForSelector` | Waits for a CSS selector together with `browser_rendering: true`. |
228
- | `home_page` | No | `false` | Query `homePage` | Visits the site root before the target URL. |
229
- | `block_resources` | No | `false` | Query `blockResources` | Applies resource blocking during page loading. |
230
- | `max_retries` | No | `3` | Query `maxRetries` | Sets the Web Unblocker retry limit; zero is accepted. |
231
- | `token_cap` | No | omitted | Query `tokenCap` | Sets the retry token budget. |
232
- | `timeout` | No | `30` | Query `timeout` | Sets the page-load deadline in seconds; transport receives an additional 30 seconds. |
233
-
234
- The response body's value is returned in `data`, commonly as HTML. Toggle
235
- `browser_rendering` for JavaScript needs and `super_mode` for routing needs.
236
- When browser loading fails, is blocked, or returns worse content, retry the same
237
- `super_mode` value with `browser_rendering: false`. Stop after a usable response
238
- unless the task requires comparing modes.
239
-
240
- ### `scrape`
241
-
242
- `scrape` creates an AI scraper through:
243
-
244
- ```text
245
- POST https://api.app.mrscraper.com/api/v1/scrapers-ai
246
- ```
247
-
248
- Choose an agent based on the extraction shape:
249
-
250
- | Agent | Designed for | Available inputs |
251
- | --------- | ------------------------------------- | ------------------------------------------------------------------------- |
252
- | `general` | Defined fields from a page | `prompt`, `schema_prompt`, `proxy_country` |
253
- | `listing` | Repeated records across listing pages | `prompt`, `schema_prompt`, `proxy_country`, `max_pages` |
254
- | `map` | URL discovery across a site | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` |
255
-
256
- General extraction:
257
-
258
- ```json
259
- {
260
- "url": "https://www.scrapethissite.com/pages/simple/",
261
- "agent": "general",
262
- "mode": "Super",
263
- "prompt": "Extract Andorra's name, capital, population, and area"
264
- }
265
- ```
266
-
267
- Listing extraction:
268
-
269
- ```json
270
- {
271
- "url": "https://www.scrapethissite.com/pages/forms/",
272
- "agent": "listing",
273
- "prompt": "Extract every hockey team, year, wins, losses, and win percentage",
274
- "max_pages": 5
275
- }
276
- ```
277
-
278
- Site map:
279
-
280
- ```json
281
- {
282
- "url": "https://www.scrapethissite.com/",
283
- "agent": "map",
284
- "max_depth": 2,
285
- "max_pages": 50,
286
- "limit": 1000,
287
- "include_patterns": "/pages/",
288
- "exclude_patterns": "/login/"
289
- }
290
- ```
291
-
292
- Best-effort schema guidance:
293
-
294
- ```json
295
- {
296
- "url": "https://www.scrapethissite.com/pages/simple/",
297
- "prompt": "Extract Andorra's name and capital",
298
- "schema_prompt": {
299
- "type": "object",
300
- "properties": {
301
- "name": { "type": "string" },
302
- "capital": { "type": "string" }
303
- },
304
- "required": ["name", "capital"]
305
- }
306
- }
307
- ```
308
-
309
- `schema_prompt` is appended to the natural-language instruction. Treat it as
310
- shape guidance and validate returned data in the consuming application when
311
- strict conformance is required.
312
-
313
- | Input | Required | Default | Request mapping | Purpose |
314
- | ------------------ | --------------- | --------------- | ---------------------- | ----------------------------------------------------------- |
315
- | `url` | Yes | - | Body `url` | Target URL for every agent. |
316
- | `prompt` | General/listing | - | Body `message` | Natural-language extraction instructions. |
317
- | `schema_prompt` | No | omitted | Appended to `message` | Best-effort JSON Schema shape guidance for general/listing. |
318
- | `agent` | No | `general` | Body `agent` | Selects `general`, `listing`, or `map`. |
319
- | `mode` | No | service default | Body `mode` | Selects `Cheap` or `Super` independently of the agent. |
320
- | `proxy_country` | No | omitted | Body `proxyCountry` | Proxy country for general/listing. |
321
- | `max_pages` | No | service default | Body `maxPages` | Page bound for listing/map. |
322
- | `max_depth` | No | service default | Body `maxDepth` | Link-depth bound for map. |
323
- | `limit` | No | service default | Body `limit` | URL-result bound for map. |
324
- | `include_patterns` | No | service default | Body `includePatterns` | URL inclusion expression for map. |
325
- | `exclude_patterns` | No | service default | Body `excludePatterns` | URL exclusion expression for map. |
326
-
327
- General and listing send `url`, `message`, and `agent`, plus the supplied
328
- agent inputs. Map sends `url`, `agent`, and only the supplied crawl inputs.
329
-
330
- #### Reproduce a scrape with `rerun`
331
-
332
- Every successful `scrape` creates a saved AI scraper configuration by default.
333
- Its response run object contains `scraperId`. Pass that UUID to `rerun` as
334
- `scraper_id` to apply the same saved extraction configuration to the original
335
- URL or another URL without rebuilding the prompt and agent settings:
336
-
337
- ```json
338
- {
339
- "target": "https://www.scrapethissite.com/pages/forms/?page_num=2",
340
- "type": "ai",
341
- "scraper_id": "scraper-uuid"
190
+ "timeout": 120
342
191
  }
343
192
  ```
344
193
 
345
- This makes the scraper configuration reproducible, but it does not guarantee
346
- identical extracted values when the page or model behavior changes.
347
-
348
- `rerun` also supports dashboard-built manual workflows and asynchronous bulk
349
- jobs across multiple target URLs. See the full [`rerun`](#rerun) section below
350
- for the available modes and result-tracking workflow.
351
-
352
- ### `serp`
353
-
354
- `serp` calls the synchronous Google endpoint:
194
+ Omit `browser_rendering` or use `null` to let the service try without a browser
195
+ and fall back to browser loading. `true` forces browser loading and `false`
196
+ disables it. A screenshot always forces browser loading, even when
197
+ `browser_rendering=false`. `super` selects real-device routing independently.
198
+
199
+ The JSON body contains `url`, `maxRetries`, and optional `tokenCap`. Page
200
+ options are query parameters. For API keys, the request also includes the
201
+ SDK-compatible `token` query parameter and `Authorization`/`x-api-token`
202
+ headers. OAuth JWTs are sent only through `Authorization`.
203
+
204
+ | Input | Default | API mapping / purpose |
205
+ | ---------------------- | -------- | --------------------------------------------------------------------------------------- |
206
+ | `url` | required | Body `url`; absolute HTTP(S) page URL. |
207
+ | `html` | `true` | Query `html`; request HTML. |
208
+ | `markdown` | `false` | Query `markdown`; request Markdown. |
209
+ | `timeout` | `120` | Query `timeout`; integer from 1 to 300 seconds. Transport allows 30 additional seconds. |
210
+ | `geo_code` | `"US"` | Query `geoCode`; proxy country code. |
211
+ | `proxy_country` | omitted | Query `proxyCountry`; compatibility country field. |
212
+ | `browser_rendering` | omitted | Query `browserRendering`; true/false, or omit/null for automatic fallback. |
213
+ | `super` | `false` | Query `super=true` when enabled. |
214
+ | `use_proxy` | `true` | Query `useProxy=false` when disabled. |
215
+ | `proxy` | omitted | Query `proxy`; custom `username:password@host:port` proxy. |
216
+ | `wait_for_selector` | omitted | Query `waitForSelector`; CSS selector to wait for. |
217
+ | `home_page` | `false` | Query `warmStart=true` when enabled. |
218
+ | `screenshot` | `false` | Query `screenshot=true`; also sets `browserRendering=true`. |
219
+ | `block_resources` | `false` | Query `blockResources=true`; block images, CSS, and fonts. |
220
+ | `bypass_proxy` | `false` | Query `bypassProxy=true`; proxy-bypass/resource-blocking option. |
221
+ | `cookie_jar` | omitted | Query `cookieJar`; cookie configuration string. |
222
+ | `headers` | omitted | Query `headers`; target request headers as an object or JSON string. |
223
+ | `max_retries` | `3` | Body `maxRetries`; nonnegative integer, 0 disables retries. |
224
+ | `token_cap` | omitted | Body `tokenCap`; nonnegative integer retry-token budget. |
225
+ | `stream` | omitted | Query `stream`; response is buffered into the normal MCP envelope. |
226
+ | `discard_token_secret` | `""` | Optional `x-discard-token-secret` header. |
227
+
228
+ ### `webSearch`
355
229
 
356
230
  ```text
357
231
  POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync
358
232
  ```
359
233
 
360
- Search from a query:
234
+ Request parsed Google results:
361
235
 
362
236
  ```json
363
237
  {
364
- "query_or_url": "iphone 17",
238
+ "url": "MrScraper web scraping",
239
+ "raw": false,
365
240
  "region": "id",
366
241
  "language": "id",
367
- "page": 2,
242
+ "page": 1,
368
243
  "format": "json"
369
244
  }
370
245
  ```
371
246
 
372
- A Google search URL can supply the query, locale, and page:
247
+ `raw=true` is the Python SDK-compatible default and requests HTML regardless
248
+ of `format`. Set `raw=false` and `format="json"` for parsed results.
249
+
250
+ A Google search URL supplies the query, locale, and page:
373
251
 
374
252
  ```json
375
253
  {
376
- "query_or_url": "https://www.google.com/search?q=iphone+17&gl=us&hl=en&start=20"
254
+ "url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
255
+ "raw": false
377
256
  }
378
257
  ```
379
258
 
380
- The server derives `query` from `q`, `region` from `gl`, `language`
381
- from `hl`, and a one-based page number from `start). Explicit tool inputs
382
- take priority over URL-derived values.
383
-
384
- | Input | Required | Default | Request mapping | Purpose |
385
- | ---------------- | -------- | -------------------- | ---------------------- | -------------------------------------------- |
386
- | `query_or_url` | Yes | - | Body `query` | Google query or Google search URL. |
387
- | `region` | No | URL value or omitted | Body `region` | Result country. |
388
- | `language` | No | URL value or omitted | Body `language` | Result language. |
389
- | `page` | No | URL value or omitted | Body `page` | One-based result page. |
390
- | `format` | No | `json` | Body `format` | Selects parsed JSON or result-page HTML. |
391
- | `render_js` | No | `false` | Body `renderJs` | Waits for JavaScript-rendered SERP features. |
392
- | `raw` | No | `false` | Body `format=html` | Compatibility alias for HTML output. |
393
- | `client_timeout` | No | `120` | Local request deadline | Sets the upstream HTTP timeout in seconds. |
259
+ This derives `query="web scraping"`, `region="us"`, `language="en"`, and
260
+ `page=3`. Explicit options override URL-derived values.
394
261
 
395
- ### `status`
262
+ | Input | Default | Request mapping / purpose |
263
+ | ----------- | -------------------- | ----------------------------------------------------------- |
264
+ | `url` | required | Search text or Google URL; normalized to body `query`. |
265
+ | `raw` | `true` | Forces body `format="html"`; false allows `format`. |
266
+ | `format` | `"json"` | Body `format`, either `json` or `html`, when raw is false. |
267
+ | `region` | URL value or omitted | Body `region`; overrides `gl`. |
268
+ | `language` | URL value or omitted | Body `language`; overrides `hl`. |
269
+ | `page` | URL value or omitted | Body `page`; positive one-based integer, overrides `start`. |
270
+ | `render_js` | `false` | Body `renderJs`; render dynamic search features. |
271
+ | `timeout` | `600` | Local HTTP deadline in seconds; omitted from the body. |
396
272
 
397
- `status` always reads account data from:
273
+ ### `status`
398
274
 
399
275
  ```text
400
276
  GET https://api.app.mrscraper.com/api/v1/subscription-accounts
401
277
  ```
402
278
 
403
- With `domain`, it also reads request-outcome analytics from:
404
-
405
- ```text
406
- GET https://api.app.mrscraper.com/api/v1/analytic/statuses
407
- ```
408
-
409
- ```json
410
- {
411
- "domain": "https://www.scrapethissite.com/pages/",
412
- "from": "7d",
413
- "to": "now",
414
- "action": "fetch",
415
- "api_token_name": "production"
416
- }
417
- ```
418
-
419
- | Input | Required | Default | Purpose |
420
- | ---------------- | -------- | ------------ | ------------------------------------------------------------------------------------------- |
421
- | `domain` | No | omitted | Adds request-outcome analytics for a hostname or URL. |
422
- | `from` | No | `24h` | Range start as ISO 8601, `now`, or a relative duration such as `30m`, `24h`, `7d`, or `2w`. |
423
- | `to` | No | `now` | Range end using the same date syntax. |
424
- | `action` | No | empty filter | Filters analytics by exact action. |
425
- | `api_token_name` | No | empty filter | Filters analytics by API-token name. |
426
-
427
- Successful output is a normalized account and analytics summary:
279
+ `status` takes no input. It returns the connected account's identity (name,
280
+ email, and verification state), subscription status, token limit, token usage,
281
+ remaining tokens, usage percentage, rate limit, auto-renewal, and renewal date.
282
+ It copies only those fields, so the API key and Stripe identifiers in the raw
283
+ account record never reach the client.
428
284
 
429
285
  ```json
430
286
  {
431
287
  "kind": "mrscraper-cli-status-summary",
432
- "source_endpoints": ["/subscription-accounts", "/analytic/statuses"],
288
+ "source_endpoints": ["/subscription-accounts"],
433
289
  "status_code": 200,
434
290
  "data": {
435
291
  "account": {
436
292
  "subscription_status": "active",
437
293
  "enterprise": false,
438
- "token_usage": 250,
439
- "token_limit": 1000,
440
- "token_remaining": 750,
441
- "usage_percent": 25,
442
- "rate_limit": 10,
443
- "rate_ttl": 60,
294
+ "token_usage": 61240,
295
+ "token_limit": 100000,
296
+ "token_remaining": 38760,
297
+ "usage_percent": 61.24,
298
+ "rate_limit": 0,
299
+ "rate_ttl": 0,
444
300
  "auto_renew": true,
445
- "ends_at": null,
301
+ "ends_at": "2026-12-01T00:00:00.000Z",
446
302
  "user": {
447
303
  "name": "Ada",
448
- "email": "ada@example.test",
304
+ "email": "ada@example.com",
449
305
  "verified": true
450
306
  }
451
- },
452
- "analytics": {
453
- "domain": "www.scrapethissite.com",
454
- "from": "2026-08-11 00:00:00 UTC",
455
- "to": "2026-08-18 00:00:00 UTC"
456
307
  }
457
308
  }
458
309
  }
459
310
  ```
460
311
 
461
- The summary calculates `token_remaining` and `usage_percent`, normalizes a
462
- URL to its hostname, and records the source endpoints used for the response.
312
+ An OAuth connection needs the `account:read` scope for `status`.
463
313
 
464
- ### `rerun`
314
+ ### Response envelope
465
315
 
466
- `type` and `bulk` describe different parts of the request:
316
+ `scraperAPI` and `webSearch` return `status_code`, `data`, and `headers` in
317
+ `structuredContent` and as JSON text. JSON bodies are parsed; other response
318
+ bodies remain text. `status` returns the summary shown above.
467
319
 
468
- - Set `type` to `ai` for a saved AI scraper created by the `scrape` tool.
469
- - Set `type` to `manual` for a saved step-based workflow created in the
470
- MrScraper dashboard. MCP reruns that workflow but does not create it.
471
- - Leave `bulk` as `false` for one URL. Set it to `true` to apply the same saved
472
- configuration to a comma- or newline-separated URL list in one bulk request.
320
+ A `scraperAPI` response carries the requested representations in `data`;
321
+ `screenshot` is a base64-encoded JPEG and `x-status-code` is the target page's
322
+ status:
473
323
 
474
- Manual reruns can be single or bulk, and bulk reruns can use either scraper
475
- type. Bulk describes the number of targets; manual describes how the saved
476
- scraper was built.
324
+ ```json
325
+ {
326
+ "status_code": 200,
327
+ "data": {
328
+ "html": "<!DOCTYPE html>...",
329
+ "markdown": null,
330
+ "screenshot": null,
331
+ "token_usage": 3,
332
+ "runtime": 18.02,
333
+ "retry_count": 0,
334
+ "error": null
335
+ },
336
+ "headers": {
337
+ "content-type": "application/json",
338
+ "x-status-code": "200",
339
+ "token_usage": "3"
340
+ }
341
+ }
342
+ ```
477
343
 
478
- Bulk mode submits one asynchronous backend job; MCP does not repeat the
479
- single-URL tool call locally. Save `data.data.bulkResultId` from the response
480
- and pass it to `result` as `result_id` until the stored result is finished.
344
+ A parsed `webSearch` response (`raw=false`) has organic results in
345
+ `data.data.organic`.
481
346
 
482
- `rerun` routes one tool contract to four saved-scraper endpoints:
347
+ Failures include `error` and set MCP `isError=true`. Network failures have
348
+ `status_code=null`. Credential-bearing response headers and JSON metadata
349
+ are filtered; network errors redact signed query credentials.
483
350
 
484
- | Mode | Endpoint | ID input | Target format | Crawl controls |
485
- | ------------- | ---------------------------------- | ------------ | ---------------------------- | ------------------- |
486
- | Single AI | `POST /scrapers-ai-rerun` | `scraper_id` | One URL | Available |
487
- | Bulk AI | `POST /scrapers-ai-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
488
- | Single manual | `POST /scrapers-manual-rerun` | `scraper_id` | One URL | Saved configuration |
489
- | Bulk manual | `POST /scrapers-manual-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
351
+ ## Manual testing: one tool at a time
490
352
 
491
- Single AI rerun:
353
+ [`scripts/test-tools.ts`](scripts/test-tools.ts) starts the compiled local MCP
354
+ server over stdio, verifies that exactly the three tools are exposed, calls one
355
+ tool, prints its response, and exits. No separate server terminal is needed.
356
+ The runner loads `.env`, or reads `MRSCRAPER_API_KEY` / `MRSCRAPER_API_TOKEN`
357
+ from your shell. Each tool call uses the live MrScraper service.
492
358
 
493
- ```json
494
- {
495
- "target": "https://www.scrapethissite.com/pages/forms/",
496
- "type": "ai",
497
- "scraper_id": "scraper-uuid",
498
- "max_depth": 2,
499
- "max_pages": 50,
500
- "limit": 1000,
501
- "include_patterns": "/pages/forms/",
502
- "exclude_patterns": "/login/",
503
- "proxy_country": "ID",
504
- "max_retry": 4,
505
- "timeout": 120
506
- }
507
- ```
359
+ Run these commands individually from the repository directory:
508
360
 
509
- Bulk manual rerun:
361
+ 1. Install dependencies:
510
362
 
511
- ```json
512
- {
513
- "target": "https://www.scrapethissite.com/pages/simple/,https://www.scrapethissite.com/pages/forms/\nhttps://www.scrapethissite.com/pages/ajax-javascript/",
514
- "type": "manual",
515
- "bulk": true,
516
- "id": "scraper-uuid"
517
- }
518
- ```
363
+ ```sh
364
+ npm ci
365
+ ```
519
366
 
520
- | Input | Required | Default | Purpose |
521
- | ------------------ | ----------- | ------- | -------------------------------------------------------------------- |
522
- | `target` | Yes | - | One URL, or a comma/newline-separated URL string for bulk mode. |
523
- | `type` | Yes | - | Selects `ai` or `manual`. |
524
- | `bulk` | No | `false` | Selects a bulk endpoint. |
525
- | `scraper_id` | Single mode | - | Saved scraper UUID for one URL. |
526
- | `id` | Bulk mode | - | Saved scraper UUID for the bulk URL list. |
527
- | `max_depth` | Single AI | omitted | Crawl depth; omission preserves the saved scraper/backend default. |
528
- | `max_pages` | Single AI | omitted | Page bound; omission preserves the saved scraper/backend default. |
529
- | `limit` | Single AI | omitted | Result bound; omission preserves the saved scraper/backend default. |
530
- | `include_patterns` | Single AI | omitted | URL inclusion expression; omission preserves saved/backend defaults. |
531
- | `exclude_patterns` | Single AI | omitted | URL exclusion expression; omission preserves saved/backend defaults. |
532
- | `proxy_country` | Single AI | omitted | Proxy country code. |
533
- | `max_retry` | Single AI | omitted | Retry limit; zero is accepted. |
534
- | `timeout` | Single AI | omitted | Timeout in seconds, used by listing reruns. |
535
-
536
- The MCP server sends single-AI controls only when supplied. Manual and bulk
537
- reruns reject them.
538
-
539
- Manual reruns carry a compliance acknowledgment in the MCP server
540
- instructions. MCP clients should present that acknowledgment before executing
541
- the manual mode.
542
-
543
- ### `results`
544
-
545
- `results` reads the stored result collection:
367
+ 2. Build the current server:
546
368
 
547
- ```text
548
- GET https://api.app.mrscraper.com/api/v1/results
549
- ```
369
+ ```sh
370
+ npm run build
371
+ ```
550
372
 
551
- ```json
552
- {
553
- "sort_field": "updatedAt",
554
- "sort_order": "desc",
555
- "page_size": 25,
556
- "page": 1,
557
- "search": "scrapethissite.com",
558
- "scraper_id": "scraper-uuid",
559
- "status": "Finished",
560
- "type": "Rerun-AI",
561
- "url": "https://www.scrapethissite.com/pages/forms/",
562
- "date_range_column": "updatedAt",
563
- "start_at": "2026-08-01T00:00:00Z",
564
- "end_at": "2026-08-18T23:59:59Z"
565
- }
566
- ```
373
+ 3. Set your API key:
567
374
 
568
- | Input | Required | Default | Query mapping | Purpose |
569
- | ------------------- | -------- | ----------- | -------------------- | ------------------------------------------------------------- |
570
- | `sort_field` | No | `updatedAt` | `sortField` | Field used by the results API for sorting. |
571
- | `sort_order` | No | `desc` | `sortOrder` | Case-insensitive `asc` or `desc`; sent upstream in uppercase. |
572
- | `page_size` | No | `10` | `pageSize` | Number of rows per page. |
573
- | `page` | No | `1` | `page` | One-based page index. |
574
- | `search` | No | omitted | `search` | Free-text result filter. |
575
- | `scraper_id` | No | omitted | `filters[scraperId]` | Exact saved scraper UUID filter. |
576
- | `status` | No | omitted | `filters[status]` | Exact Draft, Finished, Running, Failed, or Cancelled filter. |
577
- | `type` | No | omitted | `filters[type]` | Exact result type filter, such as AI or Rerun-AI. |
578
- | `url` | No | omitted | `filters[url]` | Exact stored target URL filter. |
579
- | `date_range_column` | No | omitted | `dateRangeColumn` | Column used by the date range. |
580
- | `start_at` | No | omitted | `startAt` | Inclusive range start. |
581
- | `end_at` | No | omitted | `endAt` | Inclusive range end. |
582
-
583
- ### `result`
584
-
585
- `result` reads one stored result:
375
+ ```sh
376
+ export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY"
377
+ ```
586
378
 
587
- ```text
588
- GET https://api.app.mrscraper.com/api/v1/results/{result_id}
379
+ 4. Test `scraperAPI`:
380
+
381
+ ```sh
382
+ npm run test:manual -- scraperAPI --url "https://www.scrapethissite.com/pages/simple/"
383
+ ```
384
+
385
+ 5. Test `webSearch`:
386
+
387
+ ```sh
388
+ npm run test:manual -- webSearch --query "MrScraper web scraping"
389
+ ```
390
+
391
+ 6. Test `status`:
392
+
393
+ ```sh
394
+ npm run test:manual -- status
395
+ ```
396
+
397
+ The webSearch sample explicitly requests `raw=false`, `format="json"`, US
398
+ results, and English. Each command prints `Tools: scraperAPI, webSearch, status`
399
+ and `PASS <tool>` on success. Exit codes are 0 for success, 1 for a tool or
400
+ connection failure, and 2 for invalid command arguments.
401
+
402
+ Pass a complete JSON object to test any supported options:
403
+
404
+ ```sh
405
+ npm run test:manual -- scraperAPI --args '{"url":"https://example.com","markdown":true,"screenshot":true,"super":true}'
589
406
  ```
590
407
 
591
- ```json
592
- {
593
- "result_id": "result-uuid",
594
- "include_html": false
595
- }
408
+ ```sh
409
+ npm run test:manual -- webSearch --args '{"url":"web scraping","raw":false,"region":"id","language":"id","page":1,"render_js":true}'
596
410
  ```
597
411
 
598
- | Input | Required | Default | Purpose |
599
- | -------------- | -------- | ------- | ------------------------------------------------------------------------ |
600
- | `result_id` | Yes | - | Stored MrScraper result UUID. |
601
- | `include_html` | No | `true` | Includes stored HTML; set false for smaller polling or data-only output. |
412
+ To test an already running HTTP MCP server, append
413
+ `--target http://127.0.0.1:8000/mcp`. Rebuild after source changes when testing
414
+ local stdio. `npm run test:manual -- --help` lists runner options.
602
415
 
603
416
  ## Run the server locally
604
417
 
@@ -675,7 +488,7 @@ server environment credentials are never used on the HTTP transport.
675
488
 
676
489
  The server filters credential-bearing response headers. Parsed JSON credential
677
490
  metadata and credentials embedded in generated curl commands are redacted.
678
- Scraper run extraction values remain available in `data`.
491
+ Page content remains available in `data`.
679
492
 
680
493
  Browser-origin requests are accepted from trusted local origins, from
681
494
  `claude.ai` and `chatgpt.com`, and from exact origins configured through
@@ -685,10 +498,10 @@ checks, since any client must be able to read them before it has credentials.
685
498
 
686
499
  ## Interactive widgets
687
500
 
688
- Tool results that suit a visual answer carry a UI resource: `serp` renders a
689
- result list, `scrape`/`results`/`result` render a record table, and `status`
690
- renders a quota card. Hosts that do not support MCP Apps ignore them and show
691
- the JSON as before.
501
+ `webSearch` carries a self-contained Google results widget for parsed JSON
502
+ results. HTML responses show a prompt to request parsed results. `status`
503
+ carries an account widget with the plan, token usage, and a usage meter. Hosts
504
+ without MCP Apps support display the normal JSON tool response.
692
505
 
693
506
  Each widget is a self-contained document with its script and styles inlined,
694
507
  because strict host CSP blocks anything external. Each one declares an empty
@@ -714,7 +527,7 @@ deployment and keep bearer authentication enabled.
714
527
  | `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. |
715
528
  | `PORT` | `8000` | HTTP listen port. |
716
529
  | `MRSCRAPER_API_KEY` | - | Primary stdio credential. |
717
- | `MRSCRAPER_API_TOKEN` | - | Legacy stdio credential alias. |
530
+ | `MRSCRAPER_API_TOKEN` | - | Alternative stdio credential (also used by the Python SDK). |
718
531
  | `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. |
719
532
  | `MRSCRAPER_OAUTH` | `1` | Accepts OAuth 2.1 access tokens alongside API keys. |
720
533
  | `MRSCRAPER_MCP_PUBLIC_URL` | `https://mcp.mrscraper.com` | Public origin; `<origin>/mcp` is the required token audience. |
@@ -722,8 +535,8 @@ deployment and keep bearer authentication enabled.
722
535
  | `MRSCRAPER_OAUTH_JWKS_URL` | `<issuer>/.well-known/jwks.json` | Key set used to verify access tokens. |
723
536
  | `MRSCRAPER_ALLOWED_ORIGINS` | - | Comma-separated browser origins allowed to call the HTTP server. |
724
537
  | `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. |
725
- | `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Fetch endpoint override. |
726
- | `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | SERP endpoint override. |
538
+ | `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Scraper API endpoint override. |
539
+ | `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | Web search endpoint override. |
727
540
  | `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. |
728
541
  | `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
729
542
 
@@ -741,11 +554,11 @@ deployment and keep bearer authentication enabled.
741
554
  - **403 Forbidden Origin:** Add the exact browser origin to
742
555
  `MRSCRAPER_ALLOWED_ORIGINS`.
743
556
  - **Tool input error:** Compare the call with the tool's parameter table and
744
- the selected scrape or rerun mode.
745
- - **Upstream timeout:** Increase `fetch.timeout` or `serp.client_timeout`
557
+ the documented SDK-compatible defaults.
558
+ - **Upstream timeout:** Increase `scraperAPI.timeout` (up to 300 seconds) or `webSearch.timeout`
746
559
  when the target requires a longer request window.
747
560
  - **New tools are missing:** Start a new client session and inspect
748
- `tools/list` for all seven names.
561
+ `tools/list` for exactly `scraperAPI` and `webSearch`.
749
562
 
750
563
  ## Development
751
564
 
@@ -766,8 +579,21 @@ npm run test:mcp -- \
766
579
  --token "$MRSCRAPER_API_KEY"
767
580
  ```
768
581
 
769
- `npm run test:e2e` exercises all seven tools against the live MrScraper
770
- service and creates stored scraper results.
582
+ `npm run test:e2e` runs both new tools sequentially against the live MrScraper
583
+ service using the manual runner. Use `-- --url URL --query QUERY` to customize
584
+ the targets, or `-- --target http://127.0.0.1:8000/mcp` to use HTTP.
585
+
586
+ ## MCP Registry
587
+
588
+ The hosted server's MCP Registry manifest is defined in
589
+ [`server.json`](server.json). It describes the Streamable HTTP endpoint and
590
+ links to this repository and the setup documentation.
591
+
592
+ See the [official registry publishing guide](https://modelcontextprotocol.io/registry/github-actions)
593
+ for GitHub Actions authentication and publication. Registry versions are
594
+ immutable, so use a new version in `server.json` when publishing changed
595
+ metadata. Publishing this manifest registers the hosted server; it does not
596
+ publish a new npm package.
771
597
 
772
598
  ## License
773
599