@mrscraper/mcp 0.1.2 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/README.md +230 -353
  2. package/dist/api.d.ts +26 -75
  3. package/dist/api.d.ts.map +1 -1
  4. package/dist/api.js +72 -171
  5. package/dist/api.js.map +1 -1
  6. package/dist/app.js +1 -1
  7. package/dist/app.js.map +1 -1
  8. package/dist/config.d.ts +5 -12
  9. package/dist/config.d.ts.map +1 -1
  10. package/dist/config.js +6 -25
  11. package/dist/config.js.map +1 -1
  12. package/dist/http.d.ts.map +1 -1
  13. package/dist/http.js +5 -1
  14. package/dist/http.js.map +1 -1
  15. package/dist/index.d.ts +0 -2
  16. package/dist/index.d.ts.map +1 -1
  17. package/dist/index.js +0 -2
  18. package/dist/index.js.map +1 -1
  19. package/dist/scopes.d.ts.map +1 -1
  20. package/dist/scopes.js +2 -6
  21. package/dist/scopes.js.map +1 -1
  22. package/dist/server.d.ts.map +1 -1
  23. package/dist/server.js +3 -16
  24. package/dist/server.js.map +1 -1
  25. package/dist/status.d.ts +0 -2
  26. package/dist/status.d.ts.map +1 -1
  27. package/dist/status.js +2 -30
  28. package/dist/status.js.map +1 -1
  29. package/dist/tools.d.ts +30 -80
  30. package/dist/tools.d.ts.map +1 -1
  31. package/dist/tools.js +184 -569
  32. package/dist/tools.js.map +1 -1
  33. package/dist/widgets/bundles.generated.d.ts.map +1 -1
  34. package/dist/widgets/bundles.generated.js +1 -2
  35. package/dist/widgets/bundles.generated.js.map +1 -1
  36. package/dist/widgets/index.d.ts +1 -7
  37. package/dist/widgets/index.d.ts.map +1 -1
  38. package/dist/widgets/index.js +1 -7
  39. package/dist/widgets/index.js.map +1 -1
  40. package/package.json +4 -4
  41. package/dist/compliance.d.ts +0 -4
  42. package/dist/compliance.d.ts.map +0 -1
  43. package/dist/compliance.js +0 -10
  44. package/dist/compliance.js.map +0 -1
package/README.md CHANGED
@@ -1,14 +1,21 @@
1
- # MrScraper MCP
1
+ # MrScraper MCP Server for Web Scraping and Google Search
2
2
 
3
- MrScraper MCP exposes the MrScraper web-data service through the
4
- [Model Context Protocol](https://modelcontextprotocol.io). Agents can fetch a
5
- known page, extract structured records, search Google, inspect account usage,
6
- rerun saved scrapers, and read stored results.
3
+ MrScraper MCP is the official [MrScraper](https://mrscraper.com) server for the
4
+ [Model Context Protocol](https://modelcontextprotocol.io). It connects AI
5
+ assistants to page content and Google search through two SDK-compatible tools,
6
+ plus an account check. Retrieve HTML, Markdown, or screenshots with
7
+ `scraperAPI`, discover pages with `webSearch`, and check your plan and token
8
+ usage with `status`.
7
9
 
8
- The server exposes seven web-data tools:
10
+ Use the hosted server with OAuth sign-in, or run the open-source server locally
11
+ with the [`@mrscraper/mcp` npm package](https://www.npmjs.com/package/@mrscraper/mcp).
12
+ See the [MrScraper MCP documentation](https://docs.mrscraper.com/docs/getting-started/mcp-server)
13
+ for setup instructions and client configurations.
14
+
15
+ The server exposes three tools:
9
16
 
10
17
  ```text
11
- fetch scrape serp status rerun results result
18
+ scraperAPI webSearch status
12
19
  ```
13
20
 
14
21
  ## Deployment options
@@ -54,11 +61,10 @@ the address will be refused.
54
61
 
55
62
  ### What you are approving
56
63
 
57
- | Scope | Grants |
58
- | -------------- | --------------------------------------------------------- |
59
- | `scrape:read` | `fetch`, `serp`, `results`, `result` |
60
- | `scrape:write` | `scrape`, `rerun`: creating scrapers and running them |
61
- | `account:read` | `status`: subscription plan, quota, and request analytics |
64
+ | Scope | Grants |
65
+ | -------------- | ---------------------------- |
66
+ | `scrape:read` | `scraperAPI` and `webSearch` |
67
+ | `account:read` | `status` |
62
68
 
63
69
  A tool called without its scope is refused with `403 insufficient_scope`, and
64
70
  your client will offer to grant the missing permission.
@@ -118,436 +124,294 @@ matches how broadly the connection should be available.
118
124
  ## Copyable agent setup prompt
119
125
 
120
126
  ```text
121
- Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that fetch, scrape, serp, status, rerun, results, and result are available.
127
+ Connect MrScraper MCP to this agent. Detect the current MCP client and configure a Streamable HTTP server named mrscraper at https://mcp.mrscraper.com/mcp with no credential. The server uses OAuth 2.1, so the client will open a browser to sign in. If the client cannot do OAuth, fall back to a MrScraper API key from https://app.mrscraper.com/api-tokens stored through the client's environment-variable or secret-storage mechanism and sent as Authorization: Bearer <key>. Reload the MCP client when required, then list the tools and confirm that scraperAPI, webSearch, and status are the only available tools.
122
128
  ```
123
129
 
124
130
  ## Choosing a tool
125
131
 
126
- | Tool | Choose it when |
127
- | --------- | --------------------------------------------------------------------------------- |
128
- | `fetch` | You already have a URL and need the page response from Web Unblocker. |
129
- | `scrape` | You already have a URL and need defined fields, records, listings, or a site map. |
130
- | `serp` | You are starting from a Google query or Google search URL. |
131
- | `status` | You need subscription, quota, token usage, or domain request outcomes. |
132
- | `rerun` | You have a saved AI or manual scraper ID and want to run it again. |
133
- | `results` | You need a paginated or filtered list of stored results. |
134
- | `result` | You know one result ID and need its complete stored record. |
132
+ | Tool | Choose it when |
133
+ | ------------ | ---------------------------------------------------------------------------- |
134
+ | `scraperAPI` | You have a page URL and need HTML, Markdown, a screenshot, or a combination. |
135
+ | `webSearch` | You have a Google query or Google search URL and need search results. |
136
+ | `status` | You need the account's plan, token usage, or remaining tokens. |
135
137
 
136
- A common discovery workflow is `serp` → `fetch` or `scrape`. A saved
137
- scraper workflow is `scrape` → `rerun` → `result`.
138
+ A discovery workflow is `webSearch` → `scraperAPI`. Both web tools advertise
139
+ `readOnlyHint: true`, `openWorldHint: true`, and `destructiveHint: false`;
140
+ `status` is read-only and not open-world. Authentication comes from the MCP
141
+ connection; no tool takes an API token argument.
138
142
 
139
- ## Response contract
143
+ ### Migration from the previous tool set
140
144
 
141
- API-backed tools return a response envelope in both MCP `structuredContent`
142
- and a formatted JSON text block:
145
+ Only `scraperAPI`, `webSearch`, and `status` are registered. The other old tool
146
+ names are removed.
143
147
 
144
- ```json
145
- {
146
- "status_code": 200,
147
- "data": {},
148
- "headers": {
149
- "content-type": "application/json"
150
- }
151
- }
152
- ```
153
-
154
- | Field | Type | Meaning |
155
- | ------------- | ----------------------------- | ------------------------------------------------------------------------------------ |
156
- | `status_code` | number or `null` | Upstream HTTP status, or `null` when a request ends before an HTTP response arrives. |
157
- | `data` | JSON value, string, or `null` | Parsed JSON or response text supplied by the MrScraper service. |
158
- | `headers` | object | Response headers with credential-bearing headers filtered out. |
159
- | `error` | string, when present | Request failure summary. |
148
+ | Previous call | Replacement |
149
+ | -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
150
+ | `fetch` | `scraperAPI`; uses POST, SDK options, and automatic browser fallback by default. |
151
+ | `serp` | `webSearch`; rename `query_or_url` to `url` and `client_timeout` to `timeout`. Set `raw=false` for JSON. |
152
+ | `status` | `status` for account usage; the `domain`, `from`, `to`, `action`, and `api_token_name` inputs are gone. |
153
+ | `scrape`, `rerun`, `results`, `result` | Removed from this MCP server. |
160
154
 
161
- An API failure keeps this envelope in `structuredContent` and sets the MCP
162
- result's `isError` flag. Input-contract errors are returned as MCP tool
163
- errors. Parsed credential metadata and generated curl credentials are
164
- sanitized while extracted scraper data remains available to the caller.
155
+ `super_mode` becomes `super`. `home_page` now maps to `warmStart`, and
156
+ `token_cap=0` is accepted. Restart the server and reconnect the MCP client after
157
+ updating so it refreshes `tools/list`.
165
158
 
166
159
  ## Tool reference
167
160
 
168
- ### `fetch`
161
+ The options and defaults of `scraperAPI` and `webSearch` follow the Python SDK's
162
+ `scraper_api` and `web_search` methods. MCP tool names use camel case; their
163
+ argument names use the SDK's snake case.
169
164
 
170
- `fetch` calls the Web Unblocker endpoint once:
165
+ ### `scraperAPI`
171
166
 
172
167
  ```text
173
- GET https://api.mrscraper.com/
168
+ POST https://api.mrscraper.com/
174
169
  ```
175
170
 
176
- Start with the URL alone:
171
+ Start with a known URL:
177
172
 
178
173
  ```json
179
174
  {
180
- "url": "https://example.com/products"
175
+ "url": "https://www.scrapethissite.com/pages/simple/"
181
176
  }
182
177
  ```
183
178
 
184
- Use browser rendering for JavaScript-driven content:
179
+ Request HTML, Markdown, and a screenshot together:
185
180
 
186
181
  ```json
187
182
  {
188
- "url": "https://example.com/products",
189
- "browser_rendering": true,
190
- "wait_for_selector": ".product-card",
183
+ "url": "https://example.com",
184
+ "html": true,
185
+ "markdown": true,
186
+ "screenshot": true,
187
+ "super": true,
188
+ "home_page": true,
191
189
  "geo_code": "ID",
192
- "timeout": 45
190
+ "timeout": 120
193
191
  }
194
192
  ```
195
193
 
196
- | Input | Required | Default | API mapping | Purpose |
197
- | ------------------- | -------- | ------- | ------------------------ | ------------------------------------------------------------------------------------ |
198
- | `url` | Yes | - | Query `url` | Target page URL. |
199
- | `browser_rendering` | No | `false` | Query `browserRendering` | Executes page JavaScript in a browser. |
200
- | `geo_code` | No | omitted | Query `geoCode` | Selects proxy-country routing. |
201
- | `wait_for_selector` | No | omitted | Query `waitForSelector` | Waits for a CSS selector together with `browser_rendering: true`. |
202
- | `home_page` | No | `false` | Query `homePage` | Visits the site root before the target URL. |
203
- | `block_resources` | No | `false` | Query `blockResources` | Applies resource blocking during page loading. |
204
- | `max_retries` | No | `3` | Query `maxRetries` | Sets the Web Unblocker retry limit; zero is accepted. |
205
- | `token_cap` | No | omitted | Query `tokenCap` | Sets the retry token budget. |
206
- | `timeout` | No | `30` | Query `timeout` | Sets the page-load deadline in seconds; transport receives an additional 30 seconds. |
207
-
208
- The response body's value is returned in `data`, commonly as HTML.
209
-
210
- ### `scrape`
211
-
212
- `scrape` creates an AI scraper through:
213
-
214
- ```text
215
- POST https://api.app.mrscraper.com/api/v1/scrapers-ai
216
- ```
217
-
218
- Choose an agent based on the extraction shape:
219
-
220
- | Agent | Designed for | Available inputs |
221
- | --------- | ------------------------------------- | ------------------------------------------------------------------------- |
222
- | `general` | Defined fields from a page | `prompt`, `schema_prompt`, `proxy_country` |
223
- | `listing` | Repeated records across listing pages | `prompt`, `schema_prompt`, `proxy_country`, `max_pages` |
224
- | `map` | URL discovery across a site | `max_depth`, `max_pages`, `limit`, `include_patterns`, `exclude_patterns` |
225
-
226
- General extraction:
227
-
228
- ```json
229
- {
230
- "url": "https://example.com/product",
231
- "agent": "general",
232
- "prompt": "Extract the product name, price, availability, and image URLs"
233
- }
234
- ```
235
-
236
- Listing extraction:
237
-
238
- ```json
239
- {
240
- "url": "https://example.com/products",
241
- "agent": "listing",
242
- "prompt": "Extract every product name, price, and detail URL",
243
- "max_pages": 5
244
- }
245
- ```
246
-
247
- Site map:
248
-
249
- ```json
250
- {
251
- "url": "https://example.com",
252
- "agent": "map",
253
- "max_depth": 2,
254
- "max_pages": 50,
255
- "limit": 1000,
256
- "include_patterns": "/products/",
257
- "exclude_patterns": "/cart/"
258
- }
259
- ```
260
-
261
- Best-effort schema guidance:
262
-
263
- ```json
264
- {
265
- "url": "https://example.com/product",
266
- "prompt": "Extract the product",
267
- "schema_prompt": {
268
- "type": "object",
269
- "properties": {
270
- "name": { "type": "string" },
271
- "price": { "type": "number" }
272
- },
273
- "required": ["name", "price"]
274
- }
275
- }
276
- ```
277
-
278
- `schema_prompt` is appended to the natural-language instruction. Treat it as
279
- shape guidance and validate returned data in the consuming application when
280
- strict conformance is required.
281
-
282
- | Input | Required | Default | Request mapping | Purpose |
283
- | ------------------ | --------------- | --------------- | ---------------------- | ----------------------------------------------------------- |
284
- | `url` | Yes | - | Body `url` | Target URL for every agent. |
285
- | `prompt` | General/listing | - | Body `message` | Natural-language extraction instructions. |
286
- | `schema_prompt` | No | omitted | Appended to `message` | Best-effort JSON Schema shape guidance for general/listing. |
287
- | `agent` | No | `general` | Body `agent` | Selects `general`, `listing`, or `map`. |
288
- | `proxy_country` | No | omitted | Body `proxyCountry` | Proxy country for general/listing. |
289
- | `max_pages` | No | service default | Body `maxPages` | Page bound for listing/map. |
290
- | `max_depth` | No | service default | Body `maxDepth` | Link-depth bound for map. |
291
- | `limit` | No | service default | Body `limit` | URL-result bound for map. |
292
- | `include_patterns` | No | service default | Body `includePatterns` | URL inclusion expression for map. |
293
- | `exclude_patterns` | No | service default | Body `excludePatterns` | URL exclusion expression for map. |
294
-
295
- General and listing send `url`, `message`, and `agent`, plus the supplied
296
- agent inputs. Map sends `url`, `agent`, and only the supplied crawl inputs.
297
-
298
- #### Reproduce a scrape with `rerun`
299
-
300
- Every successful `scrape` creates a saved AI scraper configuration by default.
301
- Its response run object contains `scraperId`. Pass that UUID to `rerun` as
302
- `scraper_id` to apply the same saved extraction configuration to the original
303
- URL or another URL without rebuilding the prompt and agent settings:
304
-
305
- ```json
306
- {
307
- "target": "https://example.com/product-2",
308
- "type": "ai",
309
- "scraper_id": "scraper-uuid"
310
- }
311
- ```
312
-
313
- This makes the scraper configuration reproducible, but it does not guarantee
314
- identical extracted values when the page or model behavior changes.
315
-
316
- `rerun` also supports dashboard-built manual workflows and asynchronous bulk
317
- jobs across multiple target URLs. See the full [`rerun`](#rerun) section below
318
- for the available modes and result-tracking workflow.
319
-
320
- ### `serp`
321
-
322
- `serp` calls the synchronous Google endpoint:
194
+ Omit `browser_rendering` or use `null` to let the service try without a browser
195
+ and fall back to browser loading. `true` forces browser loading and `false`
196
+ disables it. A screenshot always forces browser loading, even when
197
+ `browser_rendering=false`. `super` selects real-device routing independently.
198
+
199
+ The JSON body contains `url`, `maxRetries`, and optional `tokenCap`. Page
200
+ options are query parameters. For API keys, the request also includes the
201
+ SDK-compatible `token` query parameter and `Authorization`/`x-api-token`
202
+ headers. OAuth JWTs are sent only through `Authorization`.
203
+
204
+ | Input | Default | API mapping / purpose |
205
+ | ---------------------- | -------- | --------------------------------------------------------------------------------------- |
206
+ | `url` | required | Body `url`; absolute HTTP(S) page URL. |
207
+ | `html` | `true` | Query `html`; request HTML. |
208
+ | `markdown` | `false` | Query `markdown`; request Markdown. |
209
+ | `timeout` | `120` | Query `timeout`; integer from 1 to 300 seconds. Transport allows 30 additional seconds. |
210
+ | `geo_code` | `"US"` | Query `geoCode`; proxy country code. |
211
+ | `proxy_country` | omitted | Query `proxyCountry`; compatibility country field. |
212
+ | `browser_rendering` | omitted | Query `browserRendering`; true/false, or omit/null for automatic fallback. |
213
+ | `super` | `false` | Query `super=true` when enabled. |
214
+ | `use_proxy` | `true` | Query `useProxy=false` when disabled. |
215
+ | `proxy` | omitted | Query `proxy`; custom `username:password@host:port` proxy. |
216
+ | `wait_for_selector` | omitted | Query `waitForSelector`; CSS selector to wait for. |
217
+ | `home_page` | `false` | Query `warmStart=true` when enabled. |
218
+ | `screenshot` | `false` | Query `screenshot=true`; also sets `browserRendering=true`. |
219
+ | `block_resources` | `false` | Query `blockResources=true`; block images, CSS, and fonts. |
220
+ | `bypass_proxy` | `false` | Query `bypassProxy=true`; proxy-bypass/resource-blocking option. |
221
+ | `cookie_jar` | omitted | Query `cookieJar`; cookie configuration string. |
222
+ | `headers` | omitted | Query `headers`; target request headers as an object or JSON string. |
223
+ | `max_retries` | `3` | Body `maxRetries`; nonnegative integer, 0 disables retries. |
224
+ | `token_cap` | omitted | Body `tokenCap`; nonnegative integer retry-token budget. |
225
+ | `stream` | omitted | Query `stream`; response is buffered into the normal MCP envelope. |
226
+ | `discard_token_secret` | `""` | Optional `x-discard-token-secret` header. |
227
+
228
+ ### `webSearch`
323
229
 
324
230
  ```text
325
231
  POST https://sync.scraper.mrscraper.com/api/google/serp/v2/sync
326
232
  ```
327
233
 
328
- Search from a query:
234
+ Request parsed Google results:
329
235
 
330
236
  ```json
331
237
  {
332
- "query_or_url": "iphone 17",
238
+ "url": "MrScraper web scraping",
239
+ "raw": false,
333
240
  "region": "id",
334
241
  "language": "id",
335
- "page": 2,
242
+ "page": 1,
336
243
  "format": "json"
337
244
  }
338
245
  ```
339
246
 
340
- A Google search URL can supply the query, locale, and page:
247
+ `raw=true` is the Python SDK-compatible default and requests HTML regardless
248
+ of `format`. Set `raw=false` and `format="json"` for parsed results.
249
+
250
+ A Google search URL supplies the query, locale, and page:
341
251
 
342
252
  ```json
343
253
  {
344
- "query_or_url": "https://www.google.com/search?q=iphone+17&gl=us&hl=en&start=20"
254
+ "url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
255
+ "raw": false
345
256
  }
346
257
  ```
347
258
 
348
- The server derives `query` from `q`, `region` from `gl`, `language`
349
- from `hl`, and a one-based page number from `start). Explicit tool inputs
350
- take priority over URL-derived values.
351
-
352
- | Input | Required | Default | Request mapping | Purpose |
353
- | ---------------- | -------- | -------------------- | ---------------------- | -------------------------------------------- |
354
- | `query_or_url` | Yes | - | Body `query` | Google query or Google search URL. |
355
- | `region` | No | URL value or omitted | Body `region` | Result country. |
356
- | `language` | No | URL value or omitted | Body `language` | Result language. |
357
- | `page` | No | URL value or omitted | Body `page` | One-based result page. |
358
- | `format` | No | `json` | Body `format` | Selects parsed JSON or result-page HTML. |
359
- | `render_js` | No | `false` | Body `renderJs` | Waits for JavaScript-rendered SERP features. |
360
- | `raw` | No | `false` | Body `format=html` | Compatibility alias for HTML output. |
361
- | `client_timeout` | No | `120` | Local request deadline | Sets the upstream HTTP timeout in seconds. |
259
+ This derives `query="web scraping"`, `region="us"`, `language="en"`, and
260
+ `page=3`. Explicit options override URL-derived values.
362
261
 
363
- ### `status`
262
+ | Input | Default | Request mapping / purpose |
263
+ | ----------- | -------------------- | ----------------------------------------------------------- |
264
+ | `url` | required | Search text or Google URL; normalized to body `query`. |
265
+ | `raw` | `true` | Forces body `format="html"`; false allows `format`. |
266
+ | `format` | `"json"` | Body `format`, either `json` or `html`, when raw is false. |
267
+ | `region` | URL value or omitted | Body `region`; overrides `gl`. |
268
+ | `language` | URL value or omitted | Body `language`; overrides `hl`. |
269
+ | `page` | URL value or omitted | Body `page`; positive one-based integer, overrides `start`. |
270
+ | `render_js` | `false` | Body `renderJs`; render dynamic search features. |
271
+ | `timeout` | `600` | Local HTTP deadline in seconds; omitted from the body. |
364
272
 
365
- `status` always reads account data from:
273
+ ### `status`
366
274
 
367
275
  ```text
368
276
  GET https://api.app.mrscraper.com/api/v1/subscription-accounts
369
277
  ```
370
278
 
371
- With `domain`, it also reads request-outcome analytics from:
372
-
373
- ```text
374
- GET https://api.app.mrscraper.com/api/v1/analytic/statuses
375
- ```
376
-
377
- ```json
378
- {
379
- "domain": "https://www.example.com/products",
380
- "from": "7d",
381
- "to": "now",
382
- "action": "fetch",
383
- "api_token_name": "production"
384
- }
385
- ```
386
-
387
- | Input | Required | Default | Purpose |
388
- | ---------------- | -------- | ------------ | ------------------------------------------------------------------------------------------- |
389
- | `domain` | No | omitted | Adds request-outcome analytics for a hostname or URL. |
390
- | `from` | No | `24h` | Range start as ISO 8601, `now`, or a relative duration such as `30m`, `24h`, `7d`, or `2w`. |
391
- | `to` | No | `now` | Range end using the same date syntax. |
392
- | `action` | No | empty filter | Filters analytics by exact action. |
393
- | `api_token_name` | No | empty filter | Filters analytics by API-token name. |
394
-
395
- Successful output is a normalized account and analytics summary:
279
+ `status` takes no input. It returns the connected account's identity (name,
280
+ email, and verification state), subscription status, token limit, token usage,
281
+ remaining tokens, usage percentage, rate limit, auto-renewal, and renewal date.
282
+ It copies only those fields, so the API key and Stripe identifiers in the raw
283
+ account record never reach the client.
396
284
 
397
285
  ```json
398
286
  {
399
287
  "kind": "mrscraper-cli-status-summary",
400
- "source_endpoints": ["/subscription-accounts", "/analytic/statuses"],
288
+ "source_endpoints": ["/subscription-accounts"],
401
289
  "status_code": 200,
402
290
  "data": {
403
291
  "account": {
404
292
  "subscription_status": "active",
405
293
  "enterprise": false,
406
- "token_usage": 250,
407
- "token_limit": 1000,
408
- "token_remaining": 750,
409
- "usage_percent": 25,
410
- "rate_limit": 10,
411
- "rate_ttl": 60,
294
+ "token_usage": 61240,
295
+ "token_limit": 100000,
296
+ "token_remaining": 38760,
297
+ "usage_percent": 61.24,
298
+ "rate_limit": 0,
299
+ "rate_ttl": 0,
412
300
  "auto_renew": true,
413
- "ends_at": null,
301
+ "ends_at": "2026-12-01T00:00:00.000Z",
414
302
  "user": {
415
303
  "name": "Ada",
416
304
  "email": "ada@example.com",
417
305
  "verified": true
418
306
  }
419
- },
420
- "analytics": {
421
- "domain": "www.example.com",
422
- "from": "2026-08-11 00:00:00 UTC",
423
- "to": "2026-08-18 00:00:00 UTC"
424
307
  }
425
308
  }
426
309
  }
427
310
  ```
428
311
 
429
- The summary calculates `token_remaining` and `usage_percent`, normalizes a
430
- URL to its hostname, and records the source endpoints used for the response.
312
+ An OAuth connection needs the `account:read` scope for `status`.
431
313
 
432
- ### `rerun`
314
+ ### Response envelope
433
315
 
434
- `type` and `bulk` describe different parts of the request:
316
+ `scraperAPI` and `webSearch` return `status_code`, `data`, and `headers` in
317
+ `structuredContent` and as JSON text. JSON bodies are parsed; other response
318
+ bodies remain text. `status` returns the summary shown above.
435
319
 
436
- - Set `type` to `ai` for a saved AI scraper created by the `scrape` tool.
437
- - Set `type` to `manual` for a saved step-based workflow created in the
438
- MrScraper dashboard. MCP reruns that workflow but does not create it.
439
- - Leave `bulk` as `false` for one URL. Set it to `true` to apply the same saved
440
- configuration to a comma- or newline-separated URL list in one bulk request.
320
+ A `scraperAPI` response carries the requested representations in `data`;
321
+ `screenshot` is a base64-encoded JPEG and `x-status-code` is the target page's
322
+ status:
441
323
 
442
- Manual reruns can be single or bulk, and bulk reruns can use either scraper
443
- type. Bulk describes the number of targets; manual describes how the saved
444
- scraper was built.
324
+ ```json
325
+ {
326
+ "status_code": 200,
327
+ "data": {
328
+ "html": "<!DOCTYPE html>...",
329
+ "markdown": null,
330
+ "screenshot": null,
331
+ "token_usage": 3,
332
+ "runtime": 18.02,
333
+ "retry_count": 0,
334
+ "error": null
335
+ },
336
+ "headers": {
337
+ "content-type": "application/json",
338
+ "x-status-code": "200",
339
+ "token_usage": "3"
340
+ }
341
+ }
342
+ ```
445
343
 
446
- Bulk mode submits one asynchronous backend job; MCP does not repeat the
447
- single-URL tool call locally. Save `data.data.bulkResultId` from the response
448
- and pass it to `result` as `result_id` until the stored result is finished.
344
+ A parsed `webSearch` response (`raw=false`) has organic results in
345
+ `data.data.organic`.
449
346
 
450
- `rerun` routes one tool contract to four saved-scraper endpoints:
347
+ Failures include `error` and set MCP `isError=true`. Network failures have
348
+ `status_code=null`. Credential-bearing response headers and JSON metadata
349
+ are filtered; network errors redact signed query credentials.
451
350
 
452
- | Mode | Endpoint | ID input | Target format | Crawl controls |
453
- | ------------- | ---------------------------------- | ------------ | ---------------------------- | ------------------- |
454
- | Single AI | `POST /scrapers-ai-rerun` | `scraper_id` | One URL | Available |
455
- | Bulk AI | `POST /scrapers-ai-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
456
- | Single manual | `POST /scrapers-manual-rerun` | `scraper_id` | One URL | Saved configuration |
457
- | Bulk manual | `POST /scrapers-manual-rerun/bulk` | `id` | Comma/newline-separated URLs | Saved configuration |
351
+ ## Manual testing: one tool at a time
458
352
 
459
- Single AI rerun:
353
+ [`scripts/test-tools.ts`](scripts/test-tools.ts) starts the compiled local MCP
354
+ server over stdio, verifies that exactly the three tools are exposed, calls one
355
+ tool, prints its response, and exits. No separate server terminal is needed.
356
+ The runner loads `.env`, or reads `MRSCRAPER_API_KEY` / `MRSCRAPER_API_TOKEN`
357
+ from your shell. Each tool call uses the live MrScraper service.
460
358
 
461
- ```json
462
- {
463
- "target": "https://example.com/products",
464
- "type": "ai",
465
- "scraper_id": "scraper-uuid",
466
- "max_depth": 2,
467
- "max_pages": 50,
468
- "limit": 1000,
469
- "include_patterns": "/products/",
470
- "exclude_patterns": "/cart/"
471
- }
472
- ```
359
+ Run these commands individually from the repository directory:
473
360
 
474
- Bulk manual rerun:
361
+ 1. Install dependencies:
475
362
 
476
- ```json
477
- {
478
- "target": "https://example.com/a,https://example.com/b\nhttps://example.com/c",
479
- "type": "manual",
480
- "bulk": true,
481
- "id": "scraper-uuid"
482
- }
483
- ```
363
+ ```sh
364
+ npm ci
365
+ ```
484
366
 
485
- | Input | Required | Default | Purpose |
486
- | ------------------ | ----------- | ------------ | --------------------------------------------------------------- |
487
- | `target` | Yes | - | One URL, or a comma/newline-separated URL string for bulk mode. |
488
- | `type` | Yes | - | Selects `ai` or `manual`. |
489
- | `bulk` | No | `false` | Selects a bulk endpoint. |
490
- | `scraper_id` | Single mode | - | Saved scraper UUID for one URL. |
491
- | `id` | Bulk mode | - | Saved scraper UUID for the bulk URL list. |
492
- | `max_depth` | Single AI | `2` | Crawl depth. |
493
- | `max_pages` | Single AI | `50` | Page bound. |
494
- | `limit` | Single AI | `1000` | Result bound. |
495
- | `include_patterns` | Single AI | empty string | URL inclusion expression. |
496
- | `exclude_patterns` | Single AI | empty string | URL exclusion expression. |
367
+ 2. Build the current server:
497
368
 
498
- Manual reruns carry a compliance acknowledgment in the MCP server
499
- instructions. MCP clients should present that acknowledgment before executing
500
- the manual mode.
369
+ ```sh
370
+ npm run build
371
+ ```
501
372
 
502
- ### `results`
373
+ 3. Set your API key:
503
374
 
504
- `results` reads the stored result collection:
375
+ ```sh
376
+ export MRSCRAPER_API_KEY="YOUR_MRSCRAPER_API_KEY"
377
+ ```
505
378
 
506
- ```text
507
- GET https://api.app.mrscraper.com/api/v1/results
508
- ```
379
+ 4. Test `scraperAPI`:
509
380
 
510
- ```json
511
- {
512
- "sort_field": "updatedAt",
513
- "sort_order": "desc",
514
- "page_size": 25,
515
- "page": 1,
516
- "search": "example.com",
517
- "date_range_column": "updatedAt",
518
- "start_at": "2026-08-01T00:00:00Z",
519
- "end_at": "2026-08-18T23:59:59Z"
520
- }
521
- ```
381
+ ```sh
382
+ npm run test:manual -- scraperAPI --url "https://www.scrapethissite.com/pages/simple/"
383
+ ```
522
384
 
523
- | Input | Required | Default | Query mapping | Purpose |
524
- | ------------------- | -------- | ----------- | ----------------- | ------------------------------------------------------------- |
525
- | `sort_field` | No | `updatedAt` | `sortField` | Field used by the results API for sorting. |
526
- | `sort_order` | No | `desc` | `sortOrder` | Case-insensitive `asc` or `desc`; sent upstream in uppercase. |
527
- | `page_size` | No | `10` | `pageSize` | Number of rows per page. |
528
- | `page` | No | `1` | `page` | One-based page index. |
529
- | `search` | No | omitted | `search` | Free-text result filter. |
530
- | `date_range_column` | No | omitted | `dateRangeColumn` | Column used by the date range. |
531
- | `start_at` | No | omitted | `startAt` | Inclusive range start. |
532
- | `end_at` | No | omitted | `endAt` | Inclusive range end. |
385
+ 5. Test `webSearch`:
533
386
 
534
- ### `result`
387
+ ```sh
388
+ npm run test:manual -- webSearch --query "MrScraper web scraping"
389
+ ```
535
390
 
536
- `result` reads one stored result:
391
+ 6. Test `status`:
537
392
 
538
- ```text
539
- GET https://api.app.mrscraper.com/api/v1/results/{result_id}
393
+ ```sh
394
+ npm run test:manual -- status
395
+ ```
396
+
397
+ The webSearch sample explicitly requests `raw=false`, `format="json"`, US
398
+ results, and English. Each command prints `Tools: scraperAPI, webSearch, status`
399
+ and `PASS <tool>` on success. Exit codes are 0 for success, 1 for a tool or
400
+ connection failure, and 2 for invalid command arguments.
401
+
402
+ Pass a complete JSON object to test any supported options:
403
+
404
+ ```sh
405
+ npm run test:manual -- scraperAPI --args '{"url":"https://example.com","markdown":true,"screenshot":true,"super":true}'
540
406
  ```
541
407
 
542
- ```json
543
- {
544
- "result_id": "result-uuid"
545
- }
408
+ ```sh
409
+ npm run test:manual -- webSearch --args '{"url":"web scraping","raw":false,"region":"id","language":"id","page":1,"render_js":true}'
546
410
  ```
547
411
 
548
- | Input | Required | Purpose |
549
- | ----------- | -------- | ----------------------------- |
550
- | `result_id` | Yes | Stored MrScraper result UUID. |
412
+ To test an already running HTTP MCP server, append
413
+ `--target http://127.0.0.1:8000/mcp`. Rebuild after source changes when testing
414
+ local stdio. `npm run test:manual -- --help` lists runner options.
551
415
 
552
416
  ## Run the server locally
553
417
 
@@ -624,7 +488,7 @@ server environment credentials are never used on the HTTP transport.
624
488
 
625
489
  The server filters credential-bearing response headers. Parsed JSON credential
626
490
  metadata and credentials embedded in generated curl commands are redacted.
627
- Scraper run extraction values remain available in `data`.
491
+ Page content remains available in `data`.
628
492
 
629
493
  Browser-origin requests are accepted from trusted local origins, from
630
494
  `claude.ai` and `chatgpt.com`, and from exact origins configured through
@@ -634,10 +498,10 @@ checks, since any client must be able to read them before it has credentials.
634
498
 
635
499
  ## Interactive widgets
636
500
 
637
- Tool results that suit a visual answer carry a UI resource: `serp` renders a
638
- result list, `scrape`/`results`/`result` render a record table, and `status`
639
- renders a quota card. Hosts that do not support MCP Apps ignore them and show
640
- the JSON as before.
501
+ `webSearch` carries a self-contained Google results widget for parsed JSON
502
+ results. HTML responses show a prompt to request parsed results. `status`
503
+ carries an account widget with the plan, token usage, and a usage meter. Hosts
504
+ without MCP Apps support display the normal JSON tool response.
641
505
 
642
506
  Each widget is a self-contained document with its script and styles inlined,
643
507
  because strict host CSP blocks anything external. Each one declares an empty
@@ -663,7 +527,7 @@ deployment and keep bearer authentication enabled.
663
527
  | `HOST` | `127.0.0.1` | HTTP bind address; the Docker image uses `0.0.0.0`. |
664
528
  | `PORT` | `8000` | HTTP listen port. |
665
529
  | `MRSCRAPER_API_KEY` | - | Primary stdio credential. |
666
- | `MRSCRAPER_API_TOKEN` | - | Legacy stdio credential alias. |
530
+ | `MRSCRAPER_API_TOKEN` | - | Alternative stdio credential (also used by the Python SDK). |
667
531
  | `MRSCRAPER_HTTP_AUTH` | `1` | Enables HTTP bearer verification. |
668
532
  | `MRSCRAPER_OAUTH` | `1` | Accepts OAuth 2.1 access tokens alongside API keys. |
669
533
  | `MRSCRAPER_MCP_PUBLIC_URL` | `https://mcp.mrscraper.com` | Public origin; `<origin>/mcp` is the required token audience. |
@@ -671,8 +535,8 @@ deployment and keep bearer authentication enabled.
671
535
  | `MRSCRAPER_OAUTH_JWKS_URL` | `<issuer>/.well-known/jwks.json` | Key set used to verify access tokens. |
672
536
  | `MRSCRAPER_ALLOWED_ORIGINS` | - | Comma-separated browser origins allowed to call the HTTP server. |
673
537
  | `MRSCRAPER_API_BASE_URL` | MrScraper platform API | Platform endpoint override for development and testing. |
674
- | `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Fetch endpoint override. |
675
- | `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | SERP endpoint override. |
538
+ | `MRSCRAPER_FETCH_BASE_URL` | MrScraper Web Unblocker | Scraper API endpoint override. |
539
+ | `MRSCRAPER_SYNC_BASE_URL` | MrScraper synchronous scraper API | Web search endpoint override. |
676
540
  | `MRSCRAPER_LOG_HTTP_PAYLOAD` | off | Enables trusted-environment request-body diagnostics. |
677
541
  | `MRSCRAPER_LOG_HTTP_PAYLOAD_MAX` | `8192` | Maximum diagnostic payload length. |
678
542
 
@@ -690,11 +554,11 @@ deployment and keep bearer authentication enabled.
690
554
  - **403 Forbidden Origin:** Add the exact browser origin to
691
555
  `MRSCRAPER_ALLOWED_ORIGINS`.
692
556
  - **Tool input error:** Compare the call with the tool's parameter table and
693
- the selected scrape or rerun mode.
694
- - **Upstream timeout:** Increase `fetch.timeout` or `serp.client_timeout`
557
+ the documented SDK-compatible defaults.
558
+ - **Upstream timeout:** Increase `scraperAPI.timeout` (up to 300 seconds) or `webSearch.timeout`
695
559
  when the target requires a longer request window.
696
560
  - **New tools are missing:** Start a new client session and inspect
697
- `tools/list` for all seven names.
561
+ `tools/list` for exactly `scraperAPI` and `webSearch`.
698
562
 
699
563
  ## Development
700
564
 
@@ -715,8 +579,21 @@ npm run test:mcp -- \
715
579
  --token "$MRSCRAPER_API_KEY"
716
580
  ```
717
581
 
718
- `npm run test:e2e` exercises all seven tools against the live MrScraper
719
- service and creates stored scraper results.
582
+ `npm run test:e2e` runs both new tools sequentially against the live MrScraper
583
+ service using the manual runner. Use `-- --url URL --query QUERY` to customize
584
+ the targets, or `-- --target http://127.0.0.1:8000/mcp` to use HTTP.
585
+
586
+ ## MCP Registry
587
+
588
+ The hosted server's MCP Registry manifest is defined in
589
+ [`server.json`](server.json). It describes the Streamable HTTP endpoint and
590
+ links to this repository and the setup documentation.
591
+
592
+ See the [official registry publishing guide](https://modelcontextprotocol.io/registry/github-actions)
593
+ for GitHub Actions authentication and publication. Registry versions are
594
+ immutable, so use a new version in `server.json` when publishing changed
595
+ metadata. Publishing this manifest registers the hosted server; it does not
596
+ publish a new npm package.
720
597
 
721
598
  ## License
722
599