@mrscraper/opencode 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +26 -36
- package/package.json +1 -1
- package/skills/mrscraper/SKILL.md +64 -219
- package/skills/mrscraper-fetch/SKILL.md +97 -115
- package/skills/mrscraper-serp/SKILL.md +33 -40
- package/skills/mrscraper-scrape/SKILL.md +0 -206
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,15 @@
|
|
|
3
3
|
All notable changes to the MrScraper OpenCode plugin are documented here. The
|
|
4
4
|
project follows [Semantic Versioning](https://semver.org/).
|
|
5
5
|
|
|
6
|
+
## 0.3.0 - 2026-10-14
|
|
7
|
+
|
|
8
|
+
- Updated the skills for the hosted MCP server's new tools: `scraperAPI`
|
|
9
|
+
(pages as HTML, Markdown, or screenshots), `webSearch` (Google), and `status`
|
|
10
|
+
(account usage). The `fetch`, `scrape`, `serp`, `rerun`, `results`, and
|
|
11
|
+
`result` tools no longer exist.
|
|
12
|
+
- Removed the `mrscraper-scrape` skill. Skills are identical to the MrScraper
|
|
13
|
+
Claude plugin 0.2.0.
|
|
14
|
+
|
|
6
15
|
## 0.2.0 - 2026-10-12
|
|
7
16
|
|
|
8
17
|
- Added OpenCode 2 support. OpenCode 2 does not run OpenCode 1 plugins; the
|
package/README.md
CHANGED
|
@@ -1,10 +1,9 @@
|
|
|
1
1
|
# MrScraper for OpenCode
|
|
2
2
|
|
|
3
3
|
MrScraper connects [OpenCode](https://opencode.ai) to a hosted MCP server for
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
preserve raw page content.
|
|
4
|
+
retrieving public web pages as HTML, Markdown, or screenshots, Google
|
|
5
|
+
discovery, and account usage. The plugin also includes three focused skills
|
|
6
|
+
that help OpenCode choose the right workflow and preserve raw page content.
|
|
8
7
|
|
|
9
8
|
The same package works in OpenCode 2 and OpenCode 1.
|
|
10
9
|
|
|
@@ -14,10 +13,9 @@ The same package works in OpenCode 2 and OpenCode 1.
|
|
|
14
13
|
endpoint `https://mcp.mrscraper.com/mcp`.
|
|
15
14
|
- OAuth 2.1 sign-in handled by OpenCode; no credential is stored in this
|
|
16
15
|
package.
|
|
17
|
-
- The `mrscraper`, `mrscraper-fetch`,
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
verification, and follow-up transformations.
|
|
16
|
+
- The `mrscraper`, `mrscraper-fetch`, and `mrscraper-serp` skills.
|
|
17
|
+
- A retrieve-first workflow that keeps raw page responses available for
|
|
18
|
+
analysis, verification, and follow-up transformations.
|
|
21
19
|
|
|
22
20
|
If your configuration already defines a `mrscraper` MCP server, the plugin
|
|
23
21
|
keeps your entry and only adds the skills. It also leaves any skill with the
|
|
@@ -153,36 +151,28 @@ OpenCode 2 reaches MCP tools through Code Mode by default.
|
|
|
153
151
|
|
|
154
152
|
| Tool | Purpose |
|
|
155
153
|
| --- | --- |
|
|
156
|
-
| `
|
|
157
|
-
| `
|
|
158
|
-
| `
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
For roughly 100 known pages with a shared layout, the agent can fetch pages
|
|
172
|
-
concurrently, retain every raw response, and apply one reusable local extractor
|
|
173
|
-
instead of requesting 100 separate backend-LLM extractions. Use `scrape` when
|
|
174
|
-
managed extraction is explicitly requested or has a clear benefit after the page
|
|
175
|
-
structure and desired schema are understood.
|
|
154
|
+
| `scraperAPI` | Retrieve a known page as HTML, Markdown, a screenshot, or a combination. |
|
|
155
|
+
| `webSearch` | Discover public pages through Google. |
|
|
156
|
+
| `status` | Inspect subscription status, token usage, and remaining tokens. |
|
|
157
|
+
|
|
158
|
+
## Retrieve-first routing
|
|
159
|
+
|
|
160
|
+
When a public URL is already known, the skills direct OpenCode to retrieve it
|
|
161
|
+
with `scraperAPI` first and treat the raw response as the source of truth. The
|
|
162
|
+
agent can read, summarize, compare, or derive structured output locally without
|
|
163
|
+
losing details.
|
|
164
|
+
|
|
165
|
+
For roughly 100 known pages with a shared layout, the agent can retrieve pages
|
|
166
|
+
concurrently, retain every raw response, and apply one reusable local
|
|
167
|
+
extractor.
|
|
176
168
|
|
|
177
169
|
## Data and permissions
|
|
178
170
|
|
|
179
|
-
The plugin sends MCP tool inputs, including target URLs and
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
reruns create new stored results. Use it only with public or otherwise
|
|
185
|
-
authorized content, and follow the target site's requirements.
|
|
171
|
+
The plugin sends MCP tool inputs, including target URLs and search queries, to
|
|
172
|
+
MrScraper's hosted service. Page responses and tool results are then available
|
|
173
|
+
to OpenCode and its model provider for the requested task. Use it only with
|
|
174
|
+
public or otherwise authorized content, and follow the target site's
|
|
175
|
+
requirements.
|
|
186
176
|
|
|
187
177
|
Review MrScraper's [MCP documentation](https://docs.mrscraper.com/docs/getting-started/mcp-server),
|
|
188
178
|
[Privacy Policy](https://mrscraper.com/privacy-policy),
|
|
@@ -204,7 +194,7 @@ To control updates, pin a version and change it when you upgrade:
|
|
|
204
194
|
|
|
205
195
|
```json
|
|
206
196
|
{
|
|
207
|
-
"plugin": ["@mrscraper/opencode@0.
|
|
197
|
+
"plugin": ["@mrscraper/opencode@0.3.0"]
|
|
208
198
|
}
|
|
209
199
|
```
|
|
210
200
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mrscraper/opencode",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "MrScraper for OpenCode: the hosted MrScraper MCP server plus fetch-first skills for web scraping, structured data extraction, and Google search.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.ts",
|
|
@@ -1,22 +1,21 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mrscraper
|
|
3
3
|
description: |
|
|
4
|
-
Connect, authenticate, route, and troubleshoot MrScraper MCP, and
|
|
4
|
+
Connect, authenticate, route, and troubleshoot MrScraper MCP, and check account usage with the status tool. For known public URLs, retrieve and preserve page content with mrscraper-fetch first; use mrscraper-serp for query-first discovery through Google.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# MrScraper MCP
|
|
8
8
|
|
|
9
9
|
Use this skill for connection and authentication checks, tool selection, shared
|
|
10
|
-
response handling,
|
|
11
|
-
|
|
12
|
-
workflows.
|
|
10
|
+
response handling, account status, and troubleshooting. Load the focused fetch
|
|
11
|
+
or SERP skill for page retrieval and Google search.
|
|
13
12
|
|
|
14
13
|
## Step 1 — Verify or Connect MrScraper MCP
|
|
15
14
|
|
|
16
|
-
Check whether the current client exposes these
|
|
15
|
+
Check whether the current client exposes these three tools from the MrScraper
|
|
17
16
|
MCP provider:
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
scraperAPI webSearch status
|
|
20
19
|
|
|
21
20
|
The client may display server-qualified names. Match the provider and
|
|
22
21
|
unqualified tool name; do not assume a fixed namespace.
|
|
@@ -29,7 +28,7 @@ the managed Streamable HTTP endpoint:
|
|
|
29
28
|
|
|
30
29
|
A self-managed deployment may expose the same tools over Streamable HTTP or
|
|
31
30
|
stdio. Connection setup belongs to the MCP client. After setup or reconnection,
|
|
32
|
-
list the available tools and confirm all
|
|
31
|
+
list the available tools and confirm all three names before claiming the
|
|
33
32
|
integration is ready.
|
|
34
33
|
|
|
35
34
|
## Step 2 — Authenticate
|
|
@@ -48,66 +47,56 @@ Never place credentials in tool arguments, print them, write them into project
|
|
|
48
47
|
artifacts, or manually reproduce the OAuth exchange. Do not configure an
|
|
49
48
|
API-key or bearer-token fallback from these skills.
|
|
50
49
|
|
|
51
|
-
|
|
50
|
+
The connection uses two scopes: scrape:read for scraperAPI and webSearch, and
|
|
51
|
+
account:read for status. A tool called without its scope returns 403
|
|
52
|
+
insufficient_scope; reauthorize the connection so the client requests it.
|
|
52
53
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
54
|
+
Verify end-to-end access with status, which reads the account without
|
|
55
|
+
retrieving a page:
|
|
56
|
+
|
|
57
|
+
{}
|
|
56
58
|
|
|
57
|
-
A successful
|
|
58
|
-
reach
|
|
59
|
-
|
|
59
|
+
A successful account summary confirms that the connection can authenticate and
|
|
60
|
+
reach MrScraper. If it fails, report the error and do not claim setup is
|
|
61
|
+
complete.
|
|
60
62
|
|
|
61
63
|
## Step 3 — Route the Request
|
|
62
64
|
|
|
63
65
|
| User outcome | MCP tool or skill |
|
|
64
66
|
| --- | --- |
|
|
65
|
-
|
|
|
66
|
-
|
|
|
67
|
-
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
URL. The raw response preserves complete page context for later questions,
|
|
76
|
-
validation, and custom processing. Requesting JSON, a table, or named fields is
|
|
77
|
-
not by itself a reason to use scrape; derive those outputs locally from the raw
|
|
78
|
-
content when practical.
|
|
79
|
-
|
|
80
|
-
The general and listing scrape modes send page content through a backend LLM.
|
|
81
|
-
Their output is shaped by the extraction prompt and may omit information.
|
|
82
|
-
Applying them across many pages also repeats model work that the agent can often
|
|
83
|
-
replace with one reusable local extractor.
|
|
67
|
+
| Retrieve and preserve a known public page as HTML, Markdown, or a screenshot | [mrscraper-fetch](../mrscraper-fetch/SKILL.md) with scraperAPI |
|
|
68
|
+
| Discover relevant pages from a Google query | [mrscraper-serp](../mrscraper-serp/SKILL.md) with webSearch, then retrieve selected pages |
|
|
69
|
+
| Check the plan, token usage, or remaining tokens | status |
|
|
70
|
+
|
|
71
|
+
### Retrieve first, extract locally
|
|
72
|
+
|
|
73
|
+
scraperAPI returns raw page content. Produce summaries, tables, JSON, and other
|
|
74
|
+
structured output from that content with local analysis or extraction logic.
|
|
75
|
+
The raw response preserves complete page context for later questions,
|
|
76
|
+
validation, and changes to the requested schema.
|
|
84
77
|
|
|
85
78
|
Prefer this workflow for pages with a shared layout:
|
|
86
79
|
|
|
87
|
-
1.
|
|
80
|
+
1. Retrieve representative pages and inspect their content.
|
|
88
81
|
2. Understand the shared page structure and required fields.
|
|
89
82
|
3. Define a stable output schema and reusable local extractor.
|
|
90
|
-
4.
|
|
83
|
+
4. Retrieve the remaining pages, in parallel when safe.
|
|
91
84
|
5. Apply the same extractor to every saved response.
|
|
92
85
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
faster, preserves every raw input, and allows fields to be recovered later if
|
|
96
|
-
the schema changes.
|
|
86
|
+
Request Markdown when readable text is enough and HTML when an extractor needs
|
|
87
|
+
the page structure.
|
|
97
88
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
fetched source and verify important values against it. The map agent is separate
|
|
101
|
-
URL-discovery functionality; use it for bounded site mapping, then fetch the
|
|
102
|
-
pages whose content matters.
|
|
89
|
+
For discovery-first work, call webSearch, select relevant URLs, and retrieve
|
|
90
|
+
the pages whose content will inform the answer.
|
|
103
91
|
|
|
104
|
-
|
|
105
|
-
|
|
92
|
+
MrScraper MCP does not create AI scrapers, rerun saved scrapers, or list stored
|
|
93
|
+
results. Point the user to the MrScraper dashboard at https://app.mrscraper.com
|
|
94
|
+
for those.
|
|
106
95
|
|
|
107
96
|
## Step 4 — Handle Output and Artifacts
|
|
108
97
|
|
|
109
|
-
|
|
110
|
-
in a formatted JSON text block:
|
|
98
|
+
scraperAPI and webSearch return the same response envelope in MCP
|
|
99
|
+
structuredContent and in a formatted JSON text block:
|
|
111
100
|
|
|
112
101
|
{
|
|
113
102
|
"status_code": 200,
|
|
@@ -117,202 +106,58 @@ in a formatted JSON text block:
|
|
|
117
106
|
|
|
118
107
|
Prefer structuredContent when available. An API failure adds error and sets the
|
|
119
108
|
MCP result's isError flag. An input-contract failure is returned as a tool
|
|
120
|
-
error. Check those signals before trusting data.
|
|
121
|
-
|
|
122
|
-
Credentials, cookies, and generated curl credentials are sanitized. Extracted
|
|
123
|
-
scraper data is preserved even when it contains fields named token or password
|
|
124
|
-
because those fields may be legitimate source values.
|
|
109
|
+
error. Check those signals before trusting data. Credential-bearing headers and
|
|
110
|
+
credential metadata are removed from responses.
|
|
125
111
|
|
|
126
112
|
MrScraper tools do not accept output paths. When the user requests an artifact,
|
|
127
113
|
wait for a successful tool result, select the requested value, and save it with
|
|
128
|
-
the environment's local file-writing capability.
|
|
129
|
-
|
|
130
|
-
separately only when run metadata, headers, or diagnostics matter.
|
|
114
|
+
the environment's local file-writing capability. A screenshot is a base64 JPEG
|
|
115
|
+
in data.screenshot; decode it before saving an image.
|
|
131
116
|
|
|
132
117
|
Save substantial intermediate artifacts under the current project's
|
|
133
118
|
.mrscraper directory unless the user specifies another location. This project
|
|
134
119
|
artifact directory is unrelated to MCP connection credentials. Keep artifacts
|
|
135
120
|
out of version control unless the user asks to commit them.
|
|
136
121
|
|
|
137
|
-
## Step 5 —
|
|
138
|
-
|
|
139
|
-
A successful scrape creates a saved AI scraper configuration. Read its UUID
|
|
140
|
-
from structuredContent.data.data.scraperId and use rerun to apply the same
|
|
141
|
-
configuration to the same or another URL. Prefer rerun over rebuilding a prompt
|
|
142
|
-
when the user wants a repeatable version of an earlier extraction.
|
|
143
|
-
|
|
144
|
-
The scraperId makes the configuration reusable but does not guarantee
|
|
145
|
-
identical extracted values when the source page or model behavior changes.
|
|
146
|
-
|
|
147
|
-
Choose scraper type and target count independently:
|
|
148
|
-
|
|
149
|
-
1. Use type=ai for a saved AI scraper created by scrape.
|
|
150
|
-
2. Use type=manual for a saved step-based workflow created in the MrScraper
|
|
151
|
-
dashboard. MCP can rerun but cannot create manual workflows.
|
|
152
|
-
3. Leave bulk=false for one URL. Set bulk=true for a comma- or
|
|
153
|
-
newline-separated URL list submitted as one asynchronous job.
|
|
154
|
-
|
|
155
|
-
| Mode | Required ID input | Target | Crawl controls |
|
|
156
|
-
| --- | --- | --- | --- |
|
|
157
|
-
| Single AI | scraper_id | One URL | Supported |
|
|
158
|
-
| Single manual | scraper_id | One URL | Not accepted |
|
|
159
|
-
| Bulk AI | bulk=true and id | Comma/newline-separated URLs | Not accepted |
|
|
160
|
-
| Bulk manual | bulk=true and id | Comma/newline-separated URLs | Not accepted |
|
|
161
|
-
|
|
162
|
-
Single AI example:
|
|
163
|
-
|
|
164
|
-
{
|
|
165
|
-
"target": "https://www.scrapethissite.com/pages/simple/",
|
|
166
|
-
"type": "ai",
|
|
167
|
-
"scraper_id": "SCRAPER_UUID"
|
|
168
|
-
}
|
|
169
|
-
|
|
170
|
-
Single AI optionally accepts max_depth, max_pages, limit, include_patterns,
|
|
171
|
-
exclude_patterns, proxy_country, max_retry, and timeout. Omit any control that
|
|
172
|
-
the user did not request so the saved scraper or backend default remains in
|
|
173
|
-
effect; the MCP server does not inject replacement defaults.
|
|
174
|
-
|
|
175
|
-
Explicit-control example:
|
|
176
|
-
|
|
177
|
-
{
|
|
178
|
-
"target": "https://www.scrapethissite.com/pages/forms/?page_num=1",
|
|
179
|
-
"type": "ai",
|
|
180
|
-
"scraper_id": "SCRAPER_UUID",
|
|
181
|
-
"proxy_country": "ID",
|
|
182
|
-
"max_retry": 4,
|
|
183
|
-
"timeout": 120
|
|
184
|
-
}
|
|
185
|
-
|
|
186
|
-
These controls are rejected by manual and bulk reruns.
|
|
187
|
-
|
|
188
|
-
Bulk example:
|
|
122
|
+
## Step 5 — Review Account Usage
|
|
189
123
|
|
|
190
|
-
|
|
191
|
-
"target": "https://www.scrapethissite.com/pages/simple/,https://www.scrapethissite.com/pages/forms/?page_num=1",
|
|
192
|
-
"type": "ai",
|
|
193
|
-
"bulk": true,
|
|
194
|
-
"id": "SCRAPER_UUID"
|
|
195
|
-
}
|
|
196
|
-
|
|
197
|
-
Manual example, after the user accepts the warning below:
|
|
198
|
-
|
|
199
|
-
{
|
|
200
|
-
"target": "https://www.scrapethissite.com/pages/simple/",
|
|
201
|
-
"type": "manual",
|
|
202
|
-
"scraper_id": "SCRAPER_UUID",
|
|
203
|
-
"acknowledged": true
|
|
204
|
-
}
|
|
205
|
-
|
|
206
|
-
Bulk rerun submits one asynchronous backend job. Retain
|
|
207
|
-
structuredContent.data.data.bulkResultId and pass it to result as result_id.
|
|
208
|
-
Inspect that record until it reaches a terminal state when the user asked to
|
|
209
|
-
wait for completion. Do not resubmit the bulk rerun because its result is still
|
|
210
|
-
running.
|
|
211
|
-
|
|
212
|
-
Before the first rerun call with type=manual in a conversation, show the
|
|
213
|
-
following warning exactly once and wait for the user's acknowledgment. Do not
|
|
214
|
-
call the tool until the user accepts it, and do not repeat the warning on later
|
|
215
|
-
manual reruns in the same conversation. After the user accepts, pass
|
|
216
|
-
acknowledged=true on every manual rerun in that conversation; rerun refuses a
|
|
217
|
-
manual rerun without it and returns this warning instead.
|
|
218
|
-
|
|
219
|
-
> ### Compliance & Legal Risk
|
|
220
|
-
>
|
|
221
|
-
> WARNING
|
|
222
|
-
>
|
|
223
|
-
> **Scraping login-protected pages carries serious legal and compliance
|
|
224
|
-
> risks.** Many websites explicitly prohibit automated access in their Terms
|
|
225
|
-
> of Service, and bypassing authentication to scrape content may expose you to
|
|
226
|
-
> legal action including lawsuits, account termination, and financial
|
|
227
|
-
> penalties. By proceeding on scraping login-protected pages, you confirm that
|
|
228
|
-
> you have read and understood the target website's Terms of Service, and you
|
|
229
|
-
> **fully accept all legal, financial, and ethical responsibility** for your
|
|
230
|
-
> actions.
|
|
231
|
-
|
|
232
|
-
## Step 6 — Inspect Stored Results
|
|
233
|
-
|
|
234
|
-
Use results when the exact result UUID is unknown:
|
|
235
|
-
|
|
236
|
-
{
|
|
237
|
-
"sort_field": "updatedAt",
|
|
238
|
-
"sort_order": "desc",
|
|
239
|
-
"page_size": 20,
|
|
240
|
-
"page": 1
|
|
241
|
-
}
|
|
242
|
-
|
|
243
|
-
Apply exact backend filters when the user knows identifying result fields:
|
|
244
|
-
|
|
245
|
-
{
|
|
246
|
-
"scraper_id": "SCRAPER_UUID",
|
|
247
|
-
"status": "Finished",
|
|
248
|
-
"type": "Rerun-AI",
|
|
249
|
-
"url": "https://www.scrapethissite.com/pages/simple/"
|
|
250
|
-
}
|
|
251
|
-
|
|
252
|
-
| Input | Default | Use |
|
|
253
|
-
| --- | --- | --- |
|
|
254
|
-
| sort_field | updatedAt | Stored-result sort key. |
|
|
255
|
-
| sort_order | desc | Sort direction: asc or desc. |
|
|
256
|
-
| page_size | 10 | Positive number of records per page. |
|
|
257
|
-
| page | 1 | One-based page number. |
|
|
258
|
-
| search | omitted | Free-text result filter. |
|
|
259
|
-
| date_range_column | omitted | Date column used by start_at and end_at. |
|
|
260
|
-
| start_at | omitted | Inclusive ISO 8601 range start. |
|
|
261
|
-
| end_at | omitted | Inclusive ISO 8601 range end. |
|
|
262
|
-
| scraper_id | omitted | Exact saved scraper UUID filter. |
|
|
263
|
-
| status | omitted | Exact Draft, Finished, Running, Failed, or Cancelled filter. |
|
|
264
|
-
| type | omitted | Exact result type filter, such as AI or Rerun-AI. |
|
|
265
|
-
| url | omitted | Exact stored target URL filter. |
|
|
266
|
-
|
|
267
|
-
Use result when the UUID is known:
|
|
268
|
-
|
|
269
|
-
{
|
|
270
|
-
"result_id": "RESULT_UUID",
|
|
271
|
-
"include_html": false
|
|
272
|
-
}
|
|
273
|
-
|
|
274
|
-
A result_id may also be the bulkResultId returned by an asynchronous rerun.
|
|
275
|
-
include_html defaults to true. Set it to false for a smaller response while
|
|
276
|
-
polling status or when extracted data is sufficient.
|
|
277
|
-
|
|
278
|
-
## Step 7 — Review Account Usage
|
|
279
|
-
|
|
280
|
-
Call status with an empty object for subscription and token usage:
|
|
124
|
+
Call status with an empty object:
|
|
281
125
|
|
|
282
126
|
{}
|
|
283
127
|
|
|
284
|
-
|
|
128
|
+
The result summarizes the subscription status, token limit, token usage,
|
|
129
|
+
remaining tokens, usage percentage, rate limit, auto-renewal, renewal date,
|
|
130
|
+
and the account's name, email, and verification state:
|
|
285
131
|
|
|
286
132
|
{
|
|
287
|
-
"
|
|
288
|
-
"
|
|
289
|
-
"
|
|
133
|
+
"kind": "mrscraper-cli-status-summary",
|
|
134
|
+
"status_code": 200,
|
|
135
|
+
"data": {
|
|
136
|
+
"account": {
|
|
137
|
+
"subscription_status": "active",
|
|
138
|
+
"token_usage": 250,
|
|
139
|
+
"token_limit": 1000,
|
|
140
|
+
"token_remaining": 750,
|
|
141
|
+
"usage_percent": 25
|
|
142
|
+
}
|
|
143
|
+
}
|
|
290
144
|
}
|
|
291
145
|
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
| domain | omitted | Add request outcomes for a normalized hostname. |
|
|
295
|
-
| from | 24h | ISO start or relative duration such as 30m, 24h, or 7d. |
|
|
296
|
-
| to | now | ISO end, now, or a relative duration. |
|
|
297
|
-
| action | omitted | Exact request-action filter, used only with domain. |
|
|
298
|
-
| api_token_name | omitted | API-token-name filter, used only with domain. |
|
|
299
|
-
|
|
300
|
-
Domain outcomes describe MrScraper requests. They are not traffic, audience,
|
|
301
|
-
SEO, or market analytics. status reports account and request health, not
|
|
302
|
-
scrape-job progress; use result for a known asynchronous result ID.
|
|
146
|
+
Check remaining tokens before a large batch of page retrievals. status reports
|
|
147
|
+
account usage, not the progress of a request.
|
|
303
148
|
|
|
304
|
-
## Step
|
|
149
|
+
## Step 6 — Troubleshoot
|
|
305
150
|
|
|
306
151
|
- Missing tools: enable or reconnect the MrScraper MCP connection, then confirm
|
|
307
|
-
|
|
152
|
+
scraperAPI, webSearch, and status are exposed.
|
|
308
153
|
- Unauthorized or expired authorization: reauthorize the OAuth 2.1 connection
|
|
309
154
|
through the MCP client.
|
|
155
|
+
- 403 insufficient_scope: reauthorize so the client requests the missing scope.
|
|
310
156
|
- Input-contract error: inspect the tool schema and omit inputs that do not
|
|
311
|
-
apply
|
|
157
|
+
apply. The previous fetch, scrape, serp, rerun, results, and result tools were
|
|
158
|
+
replaced by scraperAPI, webSearch, and status.
|
|
312
159
|
- Page incomplete or missing dynamic content: load mrscraper-fetch and revise
|
|
313
160
|
page-loading inputs deliberately.
|
|
314
|
-
- Extraction incomplete: load mrscraper-scrape and improve the prompt, agent,
|
|
315
|
-
or limits.
|
|
316
161
|
- Search results weak: load mrscraper-serp and refine the query, locale, or
|
|
317
162
|
page.
|
|
318
163
|
|
|
@@ -1,35 +1,33 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mrscraper-fetch
|
|
3
3
|
description: |
|
|
4
|
-
Retrieve
|
|
4
|
+
Retrieve a known public URL with MrScraper's scraperAPI tool as HTML, Markdown, a screenshot, or a combination. Use for reading, summarizing, archiving, or analyzing a page and before local extraction; supports automatic browser fallback, real-device routing, proxies, and retries. Use mrscraper-serp when no URL is known.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
#
|
|
7
|
+
# Retrieve Page Content with MrScraper MCP
|
|
8
8
|
|
|
9
|
-
Use the
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
handing the page to another LLM by default. Use
|
|
9
|
+
Use the scraperAPI tool supplied by the MrScraper MCP server whenever the user
|
|
10
|
+
already has a public URL. Keep using the retrieved content for summarization,
|
|
11
|
+
comparison, transformation, and structured output. Use
|
|
13
12
|
[mrscraper](../mrscraper/SKILL.md) for connection and authentication checks,
|
|
14
|
-
|
|
13
|
+
account status, or broader routing, and
|
|
14
|
+
[mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen first.
|
|
15
15
|
|
|
16
16
|
## Page Retrieval Context
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
scraperAPI calls MrScraper's
|
|
19
19
|
[page-fetching service](https://docs.mrscraper.com/docs/features/unblocker) once.
|
|
20
|
-
Browser rendering executes page JavaScript,
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
rendering.
|
|
20
|
+
Browser rendering executes page JavaScript, geographic routing selects the
|
|
21
|
+
proxy country, selector waits let delayed content appear, and homepage
|
|
22
|
+
navigation visits the site root before the target. Real-device routing (super)
|
|
23
|
+
is independent of browser rendering.
|
|
25
24
|
|
|
26
|
-
Use
|
|
27
|
-
with the site's requirements.
|
|
28
|
-
does not work with a basic request.
|
|
25
|
+
Use scraperAPI only for content the user is authorized to access and in
|
|
26
|
+
accordance with the site's requirements.
|
|
29
27
|
|
|
30
28
|
Start with the URL alone and add only the controls the target needs. Plan-token
|
|
31
29
|
usage is based on runtime and bandwidth: one token per 30 seconds and one token
|
|
32
|
-
per 0.
|
|
30
|
+
per 0.25 MB, rounded up per component. Resource blocking can reduce bandwidth
|
|
33
31
|
for text-focused pages. Retries stop when the request succeeds, max_retries is
|
|
34
32
|
reached, or the running token total reaches token_cap. The initial request
|
|
35
33
|
always runs even when it exceeds that cap. See the
|
|
@@ -40,96 +38,79 @@ maintained calculation.
|
|
|
40
38
|
|
|
41
39
|
Confirm the target URL and what the user wants:
|
|
42
40
|
|
|
43
|
-
- Preserve a raw source before extraction, transformation, or comparison.
|
|
44
41
|
- Read, summarize, cite, or inspect the page.
|
|
45
42
|
- Check whether specific text appears.
|
|
46
|
-
- Archive the
|
|
43
|
+
- Archive the page or capture a screenshot.
|
|
47
44
|
- Produce fields, JSON, tables, or other structured output with local logic.
|
|
48
|
-
- Verify or supplement a managed scrape or saved result.
|
|
49
45
|
- Load JavaScript-rendered or geo-sensitive content.
|
|
50
46
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
requests managed extraction, or after fetch-led exploration has produced a
|
|
55
|
-
stable output schema and managed extraction still offers a concrete benefit.
|
|
56
|
-
Use [mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen
|
|
57
|
-
first.
|
|
47
|
+
When many pages share a layout, retrieve representative pages, define one local
|
|
48
|
+
extractor, retrieve the remaining pages in parallel when safe and proportional,
|
|
49
|
+
and apply the same extractor to every saved response.
|
|
58
50
|
|
|
59
|
-
|
|
51
|
+
## Step 2 — Retrieve the Page
|
|
60
52
|
|
|
61
|
-
|
|
53
|
+
The client may namespace the tool name; select scraperAPI from the MrScraper
|
|
54
|
+
MCP provider. Start with one call containing only the required URL, which
|
|
55
|
+
requests HTML:
|
|
62
56
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
4. Apply the same local extractor to every saved response.
|
|
67
|
-
|
|
68
|
-
For roughly 100 same-layout pages, concurrent fetches followed by one local
|
|
69
|
-
batch extraction are often faster than 100 separate backend-LLM extractions.
|
|
70
|
-
This also preserves every raw input for later recovery or schema changes.
|
|
71
|
-
|
|
72
|
-
## Step 2 — Run the Fetch
|
|
57
|
+
{
|
|
58
|
+
"url": "https://www.scrapethissite.com/pages/simple/"
|
|
59
|
+
}
|
|
73
60
|
|
|
74
|
-
|
|
75
|
-
provider. Start with one call containing only the required URL:
|
|
61
|
+
Request Markdown instead of HTML for readable text:
|
|
76
62
|
|
|
77
63
|
{
|
|
78
|
-
"url": "https://www.scrapethissite.com/pages/simple/"
|
|
64
|
+
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
65
|
+
"html": false,
|
|
66
|
+
"markdown": true
|
|
79
67
|
}
|
|
80
68
|
|
|
81
|
-
The result is available in MCP structuredContent and as formatted JSON text
|
|
69
|
+
The result is available in MCP structuredContent and as formatted JSON text.
|
|
70
|
+
The requested representations are in data:
|
|
82
71
|
|
|
83
72
|
{
|
|
84
73
|
"status_code": 200,
|
|
85
|
-
"data":
|
|
74
|
+
"data": {
|
|
75
|
+
"html": "<!DOCTYPE html>...",
|
|
76
|
+
"markdown": null,
|
|
77
|
+
"screenshot": null,
|
|
78
|
+
"token_usage": 3,
|
|
79
|
+
"runtime": 18.02,
|
|
80
|
+
"retry_count": 0,
|
|
81
|
+
"error": null
|
|
82
|
+
},
|
|
86
83
|
"headers": {
|
|
87
|
-
"
|
|
84
|
+
"x-status-code": "200"
|
|
88
85
|
}
|
|
89
86
|
}
|
|
90
87
|
|
|
91
|
-
Prefer structuredContent. Check isError, error,
|
|
92
|
-
|
|
88
|
+
Prefer structuredContent. Check isError, error, status_code, and
|
|
89
|
+
headers["x-status-code"] (the target page's status) before using data.
|
|
93
90
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
can return different content or failures for the same URL:
|
|
98
|
-
|
|
99
|
-
| browser_rendering | super_mode | Loading path |
|
|
100
|
-
| --- | --- | --- |
|
|
101
|
-
| false | false | Standard routing with the non-browser loader. |
|
|
102
|
-
| true | false | Standard routing with browser loading and JavaScript. |
|
|
103
|
-
| false | true | Real-device routing with the non-browser loader. |
|
|
104
|
-
| true | true | Real-device routing with browser loading and JavaScript. |
|
|
105
|
-
|
|
106
|
-
Use browser rendering when the page depends on JavaScript:
|
|
91
|
+
A screenshot is a base64 JPEG that can be large. Request it only when the user
|
|
92
|
+
needs an image, and save it to a file rather than reading it into the
|
|
93
|
+
conversation:
|
|
107
94
|
|
|
108
95
|
{
|
|
109
|
-
"url": "https://www.scrapethissite.com/pages/
|
|
110
|
-
"
|
|
96
|
+
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
97
|
+
"html": false,
|
|
98
|
+
"screenshot": true
|
|
111
99
|
}
|
|
112
100
|
|
|
113
|
-
|
|
114
|
-
browser loading is unnecessary or returns a worse response:
|
|
101
|
+
## Step 3 — Choose Page-Loading Options
|
|
115
102
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
}
|
|
103
|
+
Omit browser_rendering to let the service choose: it falls back to a browser
|
|
104
|
+
automatically when needed. true forces a browser and false disables it.
|
|
105
|
+
screenshot=true always uses a browser.
|
|
120
106
|
|
|
121
|
-
Use
|
|
107
|
+
Use real-device routing when routing may be the problem:
|
|
122
108
|
|
|
123
109
|
{
|
|
124
|
-
"url": "https://www.scrapethissite.com/pages/
|
|
125
|
-
"
|
|
126
|
-
"super_mode": true
|
|
110
|
+
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
111
|
+
"super": true
|
|
127
112
|
}
|
|
128
113
|
|
|
129
|
-
Browser rendering is not a strictly stronger mode. Some sites fail or return
|
|
130
|
-
worse content with browser_rendering=true but load successfully when it is
|
|
131
|
-
false. Super Mode does not enable browser rendering.
|
|
132
|
-
|
|
133
114
|
Wait for delayed content with a CSS selector:
|
|
134
115
|
|
|
135
116
|
{
|
|
@@ -142,21 +123,17 @@ Use geographic routing or homepage navigation when the target requires it:
|
|
|
142
123
|
|
|
143
124
|
{
|
|
144
125
|
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
145
|
-
"browser_rendering": true,
|
|
146
126
|
"geo_code": "ID",
|
|
147
127
|
"home_page": true
|
|
148
128
|
}
|
|
149
129
|
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
Bound resource use for a browser-rendered page:
|
|
130
|
+
Bound resource use:
|
|
153
131
|
|
|
154
132
|
{
|
|
155
133
|
"url": "https://www.scrapethissite.com/pages/ajax-javascript/#2015",
|
|
156
|
-
"browser_rendering": true,
|
|
157
134
|
"block_resources": true,
|
|
158
|
-
"max_retries":
|
|
159
|
-
"token_cap":
|
|
135
|
+
"max_retries": 1,
|
|
136
|
+
"token_cap": 10,
|
|
160
137
|
"timeout": 60
|
|
161
138
|
}
|
|
162
139
|
|
|
@@ -164,50 +141,55 @@ Bound resource use for a browser-rendered page:
|
|
|
164
141
|
|
|
165
142
|
| MCP input | Default | Request mapping | Use |
|
|
166
143
|
| --- | --- | --- | --- |
|
|
167
|
-
| url | required |
|
|
168
|
-
|
|
|
169
|
-
|
|
|
170
|
-
|
|
|
171
|
-
|
|
|
172
|
-
|
|
|
173
|
-
|
|
|
174
|
-
|
|
|
175
|
-
|
|
|
176
|
-
|
|
|
144
|
+
| url | required | Body url | Absolute HTTP or HTTPS target URL. |
|
|
145
|
+
| html | true | Query html | Request HTML content. |
|
|
146
|
+
| markdown | false | Query markdown | Request Markdown content. |
|
|
147
|
+
| screenshot | false | Query screenshot | Capture a base64 JPEG; forces browser rendering. |
|
|
148
|
+
| browser_rendering | automatic | Query browserRendering | true forces and false disables browser rendering. |
|
|
149
|
+
| super | false | Query super | Real-device routing, independent of browser rendering. |
|
|
150
|
+
| geo_code | US | Query geoCode | Proxy country, such as US, ID, or SG. |
|
|
151
|
+
| proxy_country | omitted | Query proxyCountry | Compatibility country option. |
|
|
152
|
+
| use_proxy | true | Query useProxy | Set false to skip the service's configured proxy. |
|
|
153
|
+
| proxy | omitted | Query proxy | Custom proxy as username:password@host:port. |
|
|
154
|
+
| wait_for_selector | omitted | Query waitForSelector | CSS selector to wait for. |
|
|
155
|
+
| home_page | false | Query warmStart | Visit the site root before the target page. |
|
|
156
|
+
| block_resources | false | Query blockResources | Block images, CSS, and fonts. |
|
|
157
|
+
| bypass_proxy | false | Query bypassProxy | Enable the proxy-bypass behavior. |
|
|
158
|
+
| cookie_jar | omitted | Query cookieJar | Cookie configuration string. |
|
|
159
|
+
| headers | omitted | Query headers | Target page request headers as an object or JSON string. |
|
|
160
|
+
| max_retries | 3 | Body maxRetries | Maximum retries; zero disables retries. |
|
|
161
|
+
| token_cap | omitted | Body tokenCap | Running token total that stops further retries. |
|
|
162
|
+
| timeout | 120 | Query timeout | Page timeout from 1 to 300 seconds. |
|
|
163
|
+
| stream | omitted | Query stream | Ask the API to stream; the response is still returned once. |
|
|
164
|
+
| discard_token_secret | empty | Header x-discard-token-secret | Optional upstream header. |
|
|
177
165
|
|
|
178
166
|
Authentication is handled by the MCP client's OAuth 2.1 connection and is
|
|
179
167
|
never a tool input.
|
|
180
168
|
|
|
181
169
|
## Step 4 — Inspect and Retry Deliberately
|
|
182
170
|
|
|
183
|
-
Start with
|
|
184
|
-
|
|
185
|
-
content:
|
|
171
|
+
Start with the URL alone unless the task already establishes a requirement. If
|
|
172
|
+
the response fails, is blocked, incomplete, or missing dynamic content:
|
|
186
173
|
|
|
187
|
-
1. Inspect the
|
|
188
|
-
2. Change one axis at a time:
|
|
174
|
+
1. Inspect status_code, data.error, headers["x-status-code"], and the content.
|
|
175
|
+
2. Change one axis at a time: force or disable browser_rendering, or set super
|
|
189
176
|
when routing may be the problem.
|
|
190
|
-
3.
|
|
191
|
-
value with browser_rendering=false.
|
|
192
|
-
4. Try the remaining untested combinations when the response is still unusable.
|
|
193
|
-
5. Add wait_for_selector, geo_code, or home_page only when evidence shows that
|
|
177
|
+
3. Add wait_for_selector, geo_code, or home_page only when evidence shows that
|
|
194
178
|
the target requires it.
|
|
195
|
-
|
|
179
|
+
4. Stop after a usable response unless the user requests a comparison.
|
|
196
180
|
|
|
197
|
-
Do not repeat an identical
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
session.
|
|
181
|
+
Do not repeat an identical request. Treat wait_for_selector as a CSS selector,
|
|
182
|
+
not a duration. Browser rendering loads a page; it does not click controls,
|
|
183
|
+
submit forms, or provide an authenticated interactive browser session.
|
|
201
184
|
|
|
202
185
|
## Step 5 — Deliver the Result
|
|
203
186
|
|
|
204
187
|
Answer the user's request from data. Keep the full envelope when headers or
|
|
205
188
|
diagnostics matter. When the user requests an archive, save the successful
|
|
206
|
-
response or
|
|
207
|
-
capability;
|
|
208
|
-
|
|
209
|
-
Keep
|
|
210
|
-
tables, JSON transformations, and extraction scripts from
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
as exhaustive source content.
|
|
189
|
+
response or the requested field with the environment's local file-writing
|
|
190
|
+
capability; scraperAPI itself has no output-path input.
|
|
191
|
+
|
|
192
|
+
Keep retrieved content as the source of truth for later steps. Build
|
|
193
|
+
summaries, tables, JSON transformations, and extraction scripts from it, and
|
|
194
|
+
retrieve the page again when a follow-up needs details that an earlier
|
|
195
|
+
transformation did not preserve.
|
|
@@ -1,28 +1,31 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mrscraper-serp
|
|
3
3
|
description: |
|
|
4
|
-
Discover public pages through Google with MrScraper using a search query or Google search URL. Use when no target URL is known or the user asks to inspect Google results. After discovery,
|
|
4
|
+
Discover public pages through Google with MrScraper's webSearch tool using a search query or Google search URL. Use when no target URL is known or the user asks to inspect Google results. After discovery, retrieve selected pages with mrscraper-fetch.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
# Discover Pages with MrScraper MCP
|
|
7
|
+
# Discover Pages with MrScraper MCP Web Search
|
|
8
8
|
|
|
9
|
-
Use the
|
|
10
|
-
a search query rather than a known target URL. Use
|
|
11
|
-
[mrscraper](../mrscraper/SKILL.md) for connection troubleshooting,
|
|
12
|
-
|
|
9
|
+
Use the webSearch tool supplied by the MrScraper MCP server when a task begins
|
|
10
|
+
with a search query rather than a known target URL. Use
|
|
11
|
+
[mrscraper](../mrscraper/SKILL.md) for connection troubleshooting, account
|
|
12
|
+
status, or broader routing.
|
|
13
13
|
|
|
14
14
|
## Step 1 — Build the Query
|
|
15
15
|
|
|
16
|
-
Include the important entity, attribute, location, or date in
|
|
16
|
+
Include the important entity, attribute, location, or date in url, and set
|
|
17
|
+
raw=false to receive parsed results:
|
|
17
18
|
|
|
18
19
|
{
|
|
19
|
-
"
|
|
20
|
+
"url": "web scraping best practices",
|
|
21
|
+
"raw": false
|
|
20
22
|
}
|
|
21
23
|
|
|
22
24
|
The input can also be a complete Google search URL:
|
|
23
25
|
|
|
24
26
|
{
|
|
25
|
-
"
|
|
27
|
+
"url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
|
|
28
|
+
"raw": false
|
|
26
29
|
}
|
|
27
30
|
|
|
28
31
|
For a Google URL, the tool reads q, gl, hl, and start. Explicit region,
|
|
@@ -33,7 +36,8 @@ language, and page inputs take precedence.
|
|
|
33
36
|
Set both country and language when results must match a locale:
|
|
34
37
|
|
|
35
38
|
{
|
|
36
|
-
"
|
|
39
|
+
"url": "running shoes",
|
|
40
|
+
"raw": false,
|
|
37
41
|
"region": "id",
|
|
38
42
|
"language": "id",
|
|
39
43
|
"page": 2
|
|
@@ -44,25 +48,21 @@ the current page before requesting another.
|
|
|
44
48
|
|
|
45
49
|
## Step 3 — Choose the Output
|
|
46
50
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
{
|
|
51
|
-
"query_or_url": "iphone 17",
|
|
52
|
-
"format": "json"
|
|
53
|
-
}
|
|
51
|
+
raw defaults to true, which returns result-page HTML as the Python SDK does.
|
|
52
|
+
Set raw=false for parsed JSON; organic results are in data.data.organic, next
|
|
53
|
+
to sections such as people_also_ask and pagination.
|
|
54
54
|
|
|
55
55
|
Request result-page HTML when the page itself is needed:
|
|
56
56
|
|
|
57
57
|
{
|
|
58
|
-
"
|
|
59
|
-
"format": "html"
|
|
58
|
+
"url": "iphone 17"
|
|
60
59
|
}
|
|
61
60
|
|
|
62
61
|
Use JavaScript rendering for dynamic result features such as AI Overview:
|
|
63
62
|
|
|
64
63
|
{
|
|
65
|
-
"
|
|
64
|
+
"url": "what is web scraping",
|
|
65
|
+
"raw": false,
|
|
66
66
|
"render_js": true
|
|
67
67
|
}
|
|
68
68
|
|
|
@@ -70,14 +70,14 @@ Use JavaScript rendering for dynamic result features such as AI Overview:
|
|
|
70
70
|
|
|
71
71
|
| MCP input | Default | Request mapping | Use |
|
|
72
72
|
| --- | --- | --- | --- |
|
|
73
|
-
|
|
|
73
|
+
| url | required | Body query | Search query or complete Google search URL. |
|
|
74
|
+
| raw | true | Body format=html | Return result-page HTML; set false to use format. |
|
|
75
|
+
| format | json | Body format | json or html when raw=false. |
|
|
74
76
|
| region | URL value or omitted | Body region | Google result country, such as us or id. |
|
|
75
77
|
| language | URL value or omitted | Body language | Google result language, such as en or id. |
|
|
76
78
|
| page | URL value or omitted | Body page | Positive, one-based results page. |
|
|
77
|
-
|
|
|
78
|
-
|
|
|
79
|
-
| raw | false | Body format=html | Deprecated compatibility alias; prefer format=html. |
|
|
80
|
-
| client_timeout | 120 | MCP upstream deadline | Maximum wait in seconds; not sent in the SERP body. |
|
|
79
|
+
| render_js | false | Body renderJs | Render JavaScript-dependent result features. |
|
|
80
|
+
| timeout | 600 | MCP upstream deadline | Maximum wait in seconds; not sent in the search body. |
|
|
81
81
|
|
|
82
82
|
Authentication is handled by the MCP client's OAuth 2.1 connection and is
|
|
83
83
|
never a tool input.
|
|
@@ -88,25 +88,19 @@ The complete response envelope is available in MCP structuredContent and as
|
|
|
88
88
|
formatted JSON text. Prefer structuredContent. Check isError, error, and
|
|
89
89
|
status_code before using data.
|
|
90
90
|
|
|
91
|
-
For format=json, inspect data for the available titles, URLs, snippets, and
|
|
92
|
-
other result fields. For format=html, data contains result-page HTML.
|
|
93
|
-
|
|
94
91
|
When the user requests a search artifact, save the successful data or full
|
|
95
|
-
envelope with local file-writing capability;
|
|
92
|
+
envelope with local file-writing capability; webSearch has no output-path
|
|
93
|
+
input.
|
|
96
94
|
|
|
97
95
|
## Step 5 — Continue with Relevant URLs
|
|
98
96
|
|
|
99
|
-
Select only URLs relevant to the user's goal, then
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
each selected page before deriving answers, comparisons, or structured output.
|
|
103
|
-
- Add [mrscraper-scrape](../mrscraper-scrape/SKILL.md) only when the user
|
|
104
|
-
explicitly requests managed extraction or it still offers a clear benefit
|
|
105
|
-
after the page structure and output schema are understood.
|
|
97
|
+
Select only URLs relevant to the user's goal, then load
|
|
98
|
+
[mrscraper-fetch](../mrscraper-fetch/SKILL.md) to retrieve and inspect each
|
|
99
|
+
selected page before deriving answers, comparisons, or structured output.
|
|
106
100
|
|
|
107
101
|
Run independent follow-up URLs in parallel when the environment supports safe
|
|
108
102
|
parallel execution. Keep the number of pages proportional to the requested
|
|
109
|
-
coverage.
|
|
103
|
+
coverage. If the user only requested URL discovery, stop after the search.
|
|
110
104
|
|
|
111
105
|
## Step 6 — Handle Weak or Missing Results
|
|
112
106
|
|
|
@@ -114,10 +108,9 @@ coverage.
|
|
|
114
108
|
- Rewrite noisy queries around the important entity, attribute, and location.
|
|
115
109
|
- Request another result page only after checking the current page.
|
|
116
110
|
- Retry once with render_js=true when a dynamic result feature is required.
|
|
117
|
-
- Increase
|
|
111
|
+
- Increase timeout when the MCP request deadline is too short.
|
|
118
112
|
- Use [mrscraper](../mrscraper/SKILL.md) when the connection or authentication
|
|
119
113
|
fails.
|
|
120
114
|
|
|
121
|
-
|
|
122
|
-
[mrscraper-scrape](../mrscraper-scrape/SKILL.md) to discover URLs within one
|
|
115
|
+
webSearch is for public Google discovery, not for discovering URLs within one
|
|
123
116
|
known site.
|
|
@@ -1,206 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: mrscraper-scrape
|
|
3
|
-
description: |
|
|
4
|
-
Run MrScraper's general, listing, or map agents for managed structured extraction or bounded URL discovery within a known site. Use when managed output is explicitly requested or justified after source-page inspection; use mrscraper-fetch for initial acquisition of known pages and mrscraper-serp when no starting URL is known.
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
# Extract Structured Data with MrScraper MCP
|
|
8
|
-
|
|
9
|
-
Do not use scrape for the first exploration of a known page. Start with
|
|
10
|
-
[mrscraper-fetch](../mrscraper-fetch/SKILL.md), inspect the complete raw
|
|
11
|
-
response, and prefer local analysis or a reusable extractor. Use scrape when
|
|
12
|
-
the user explicitly wants MrScraper-managed extraction, or after the agent
|
|
13
|
-
understands the website and has defined a stable output schema. Use
|
|
14
|
-
[mrscraper](../mrscraper/SKILL.md) for connection troubleshooting, saved runs,
|
|
15
|
-
account status, or broader routing.
|
|
16
|
-
|
|
17
|
-
For general and listing, MrScraper retrieves page HTML and asks a backend LLM to
|
|
18
|
-
interpret it according to prompt. This adds model-processing time, can narrow
|
|
19
|
-
the result to the requested fields, and repeats model work across many pages.
|
|
20
|
-
|
|
21
|
-
Before using general or listing, confirm all of the following:
|
|
22
|
-
|
|
23
|
-
1. The target or representative pages have already been fetched and inspected.
|
|
24
|
-
2. The site structure and required output fields are understood.
|
|
25
|
-
3. A stable output schema has been defined.
|
|
26
|
-
4. Managed extraction still offers a concrete benefit over local code, or the
|
|
27
|
-
user explicitly requested it.
|
|
28
|
-
|
|
29
|
-
If these conditions are not met, return to fetch. Requiring JSON, a table, or
|
|
30
|
-
named fields is not by itself a reason to call scrape. For large same-layout
|
|
31
|
-
sets—even roughly 100 pages—safe concurrent fetches plus one reusable local
|
|
32
|
-
extractor are often faster and preserve every raw response. The map agent is
|
|
33
|
-
separate URL discovery functionality and does not require an extraction schema.
|
|
34
|
-
|
|
35
|
-
## Step 1 — Choose an Agent
|
|
36
|
-
|
|
37
|
-
| Agent | Use it for | Required input | Available controls |
|
|
38
|
-
| --- | --- | --- | --- |
|
|
39
|
-
| general | One detail page or one extraction task | prompt | proxy_country, schema_prompt |
|
|
40
|
-
| listing | Repeated records or paginated listings | prompt | proxy_country, max_pages, schema_prompt |
|
|
41
|
-
| map | Discovering URLs across a site | URL only | max_depth, max_pages, limit, include_patterns, exclude_patterns |
|
|
42
|
-
|
|
43
|
-
The default agent is general. For map, omit prompt, schema_prompt, and
|
|
44
|
-
proxy_country. For general and listing, omit map-only controls. max_pages is
|
|
45
|
-
accepted by listing and map but not general.
|
|
46
|
-
|
|
47
|
-
agent selects the extraction workflow. mode independently selects the backend
|
|
48
|
-
execution tier: Cheap or Super. Omit mode to preserve the backend default, and
|
|
49
|
-
select Super only when the extraction requires the stronger mode.
|
|
50
|
-
|
|
51
|
-
## Step 2 — Define the Extraction
|
|
52
|
-
|
|
53
|
-
The examples below assume the decision gate above has been satisfied. Write a
|
|
54
|
-
prompt from the already-understood page structure and output schema, preserve
|
|
55
|
-
source values, and do not ask the model to infer unavailable values.
|
|
56
|
-
|
|
57
|
-
Detail-page example:
|
|
58
|
-
|
|
59
|
-
{
|
|
60
|
-
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
61
|
-
"agent": "general",
|
|
62
|
-
"mode": "Super",
|
|
63
|
-
"prompt": "Extract each country's name, capital, population, and area. Preserve source values and omit unavailable fields."
|
|
64
|
-
}
|
|
65
|
-
|
|
66
|
-
Repeated-listing example:
|
|
67
|
-
|
|
68
|
-
{
|
|
69
|
-
"url": "https://www.scrapethissite.com/pages/forms/?page_num=1",
|
|
70
|
-
"agent": "listing",
|
|
71
|
-
"prompt": "Extract each hockey team's name, year, wins, losses, and win percentage.",
|
|
72
|
-
"max_pages": 5
|
|
73
|
-
}
|
|
74
|
-
|
|
75
|
-
Site-map example:
|
|
76
|
-
|
|
77
|
-
{
|
|
78
|
-
"url": "https://www.scrapethissite.com/",
|
|
79
|
-
"agent": "map",
|
|
80
|
-
"max_depth": 2,
|
|
81
|
-
"max_pages": 50,
|
|
82
|
-
"limit": 1000,
|
|
83
|
-
"include_patterns": "/pages/"
|
|
84
|
-
}
|
|
85
|
-
|
|
86
|
-
## Step 3 — Add Shape Guidance When Useful
|
|
87
|
-
|
|
88
|
-
schema_prompt accepts a JSON object, not a file path. If the user supplies a
|
|
89
|
-
local JSON Schema file, read and parse it with local file tools, confirm its
|
|
90
|
-
root is an object, and pass that object:
|
|
91
|
-
|
|
92
|
-
{
|
|
93
|
-
"url": "https://www.scrapethissite.com/pages/simple/",
|
|
94
|
-
"agent": "general",
|
|
95
|
-
"prompt": "Extract every country's name, capital, population, and area.",
|
|
96
|
-
"schema_prompt": {
|
|
97
|
-
"type": "object",
|
|
98
|
-
"properties": {
|
|
99
|
-
"countries": {
|
|
100
|
-
"type": "array",
|
|
101
|
-
"items": {
|
|
102
|
-
"type": "object",
|
|
103
|
-
"properties": {
|
|
104
|
-
"name": { "type": "string" },
|
|
105
|
-
"capital": { "type": "string" },
|
|
106
|
-
"population": { "type": "string" },
|
|
107
|
-
"area": { "type": "string" }
|
|
108
|
-
}
|
|
109
|
-
}
|
|
110
|
-
}
|
|
111
|
-
}
|
|
112
|
-
}
|
|
113
|
-
}
|
|
114
|
-
|
|
115
|
-
The MCP server appends this object to the extraction prompt as best-effort
|
|
116
|
-
shape guidance. It does not validate the returned data against the schema.
|
|
117
|
-
Validate locally when strict compliance is required.
|
|
118
|
-
|
|
119
|
-
## Step 4 — Set Parameters
|
|
120
|
-
|
|
121
|
-
| MCP input | Default | Request mapping | Use |
|
|
122
|
-
| --- | --- | --- | --- |
|
|
123
|
-
| url | required | Body url | Absolute HTTP or HTTPS starting URL. |
|
|
124
|
-
| agent | general | Body agent | Select general, listing, or map. |
|
|
125
|
-
| mode | service default | Body mode | Select Cheap or Super execution without changing the agent. |
|
|
126
|
-
| prompt | required for general/listing | Body message | Describe the fields or records to extract. |
|
|
127
|
-
| proxy_country | omitted | Body proxyCountry | Route general/listing through a country. |
|
|
128
|
-
| max_pages | service default | Body maxPages | Bound listing or map pages. |
|
|
129
|
-
| max_depth | service default | Body maxDepth | Bound map crawl depth. |
|
|
130
|
-
| limit | service default | Body limit | Bound map URL results. |
|
|
131
|
-
| include_patterns | omitted | Body includePatterns | Restrict map results to matching URLs. |
|
|
132
|
-
| exclude_patterns | omitted | Body excludePatterns | Exclude matching map URLs. |
|
|
133
|
-
| schema_prompt | omitted | Appended to message | JSON Schema object for general/listing guidance. |
|
|
134
|
-
|
|
135
|
-
Authentication is handled by the MCP client's OAuth 2.1 connection. The scrape
|
|
136
|
-
tool does not accept a credential or output-path input. Choose the smallest
|
|
137
|
-
practical limits and omit optional values when the service default is
|
|
138
|
-
appropriate.
|
|
139
|
-
|
|
140
|
-
## Step 5 — Wait for Completion
|
|
141
|
-
|
|
142
|
-
General and map usually finish quickly. Listing is synchronous and can take
|
|
143
|
-
several minutes even for one page.
|
|
144
|
-
|
|
145
|
-
Before starting a listing call, tell the user it may take several minutes. Keep
|
|
146
|
-
waiting on the original MCP call within the environment's supported tool
|
|
147
|
-
timeout. Do not submit a duplicate scrape because the first call is quiet or
|
|
148
|
-
still running.
|
|
149
|
-
|
|
150
|
-
## Step 6 — Use the Output
|
|
151
|
-
|
|
152
|
-
The complete API envelope is returned in MCP structuredContent and as formatted
|
|
153
|
-
JSON text. Prefer structuredContent and check isError, error, and status_code
|
|
154
|
-
before trusting the response.
|
|
155
|
-
|
|
156
|
-
A successful run normally stores its extracted value at:
|
|
157
|
-
|
|
158
|
-
structuredContent.data.data.data
|
|
159
|
-
|
|
160
|
-
The saved scraper UUID is normally at:
|
|
161
|
-
|
|
162
|
-
structuredContent.data.data.scraperId
|
|
163
|
-
|
|
164
|
-
When the user asks for a file, save the extracted value only after the tool
|
|
165
|
-
succeeds. Use local file-writing capability and report the path. Preserve the
|
|
166
|
-
full envelope separately when the scraper ID, response headers, or diagnostics
|
|
167
|
-
also matter.
|
|
168
|
-
|
|
169
|
-
Post-process only when the user requests filtering, merging, normalization,
|
|
170
|
-
CSV, a table, or another deliverable. Extracted fields named token or password
|
|
171
|
-
may be legitimate source data and are intentionally preserved.
|
|
172
|
-
|
|
173
|
-
## Step 7 — Preserve the Reproducible Scraper
|
|
174
|
-
|
|
175
|
-
Every successful scrape creates a saved AI scraper configuration by default.
|
|
176
|
-
Report scraperId when it is available and the extraction may need to run again.
|
|
177
|
-
|
|
178
|
-
Use the MrScraper rerun tool to apply the saved configuration to the same or
|
|
179
|
-
another URL:
|
|
180
|
-
|
|
181
|
-
{
|
|
182
|
-
"target": "https://www.scrapethissite.com/pages/forms/?page_num=2",
|
|
183
|
-
"type": "ai",
|
|
184
|
-
"scraper_id": "SCRAPER_UUID"
|
|
185
|
-
}
|
|
186
|
-
|
|
187
|
-
Prefer rerun over recreating the scrape definition when the user wants a
|
|
188
|
-
repeatable extraction. Explain that this reproduces the saved configuration,
|
|
189
|
-
not necessarily identical values when the page or model behavior changes.
|
|
190
|
-
|
|
191
|
-
The rerun tool also handles dashboard-built manual workflows and asynchronous
|
|
192
|
-
bulk jobs. Follow the full rerun workflow in
|
|
193
|
-
[mrscraper](../mrscraper/SKILL.md), including the required acknowledgment
|
|
194
|
-
before any manual rerun.
|
|
195
|
-
|
|
196
|
-
## Step 8 — Handle Failures
|
|
197
|
-
|
|
198
|
-
- Check isError, error, and status_code before using or saving output.
|
|
199
|
-
- Tighten the prompt when fields are missing or grouped incorrectly.
|
|
200
|
-
- Confirm the selected agent accepts every supplied input.
|
|
201
|
-
- Validate locally when downstream code requires a strict schema.
|
|
202
|
-
- Use [mrscraper-fetch](../mrscraper-fetch/SKILL.md) for page reading.
|
|
203
|
-
- Use [mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen
|
|
204
|
-
first.
|
|
205
|
-
- Never invent replacement fields or fill missing values with unsupported
|
|
206
|
-
assumptions.
|