@mrscraper/opencode 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +110 -68
- package/index.ts +73 -5
- package/package.json +3 -2
- package/skills/mrscraper/SKILL.md +64 -219
- package/skills/mrscraper-fetch/SKILL.md +97 -115
- package/skills/mrscraper-serp/SKILL.md +33 -40
- package/skills/mrscraper-scrape/SKILL.md +0 -206
|
@@ -1,22 +1,21 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: mrscraper
|
|
3
3
|
description: |
|
|
4
|
-
Connect, authenticate, route, and troubleshoot MrScraper MCP, and
|
|
4
|
+
Connect, authenticate, route, and troubleshoot MrScraper MCP, and check account usage with the status tool. For known public URLs, retrieve and preserve page content with mrscraper-fetch first; use mrscraper-serp for query-first discovery through Google.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# MrScraper MCP
|
|
8
8
|
|
|
9
9
|
Use this skill for connection and authentication checks, tool selection, shared
|
|
10
|
-
response handling,
|
|
11
|
-
|
|
12
|
-
workflows.
|
|
10
|
+
response handling, account status, and troubleshooting. Load the focused fetch
|
|
11
|
+
or SERP skill for page retrieval and Google search.
|
|
13
12
|
|
|
14
13
|
## Step 1 — Verify or Connect MrScraper MCP
|
|
15
14
|
|
|
16
|
-
Check whether the current client exposes these
|
|
15
|
+
Check whether the current client exposes these three tools from the MrScraper
|
|
17
16
|
MCP provider:
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
scraperAPI webSearch status
|
|
20
19
|
|
|
21
20
|
The client may display server-qualified names. Match the provider and
|
|
22
21
|
unqualified tool name; do not assume a fixed namespace.
|
|
@@ -29,7 +28,7 @@ the managed Streamable HTTP endpoint:
|
|
|
29
28
|
|
|
30
29
|
A self-managed deployment may expose the same tools over Streamable HTTP or
|
|
31
30
|
stdio. Connection setup belongs to the MCP client. After setup or reconnection,
|
|
32
|
-
list the available tools and confirm all
|
|
31
|
+
list the available tools and confirm all three names before claiming the
|
|
33
32
|
integration is ready.
|
|
34
33
|
|
|
35
34
|
## Step 2 — Authenticate
|
|
@@ -48,66 +47,56 @@ Never place credentials in tool arguments, print them, write them into project
|
|
|
48
47
|
artifacts, or manually reproduce the OAuth exchange. Do not configure an
|
|
49
48
|
API-key or bearer-token fallback from these skills.
|
|
50
49
|
|
|
51
|
-
|
|
50
|
+
The connection uses two scopes: scrape:read for scraperAPI and webSearch, and
|
|
51
|
+
account:read for status. A tool called without its scope returns 403
|
|
52
|
+
insufficient_scope; reauthorize the connection so the client requests it.
|
|
52
53
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
54
|
+
Verify end-to-end access with status, which reads the account without
|
|
55
|
+
retrieving a page:
|
|
56
|
+
|
|
57
|
+
{}
|
|
56
58
|
|
|
57
|
-
A successful
|
|
58
|
-
reach
|
|
59
|
-
|
|
59
|
+
A successful account summary confirms that the connection can authenticate and
|
|
60
|
+
reach MrScraper. If it fails, report the error and do not claim setup is
|
|
61
|
+
complete.
|
|
60
62
|
|
|
61
63
|
## Step 3 — Route the Request
|
|
62
64
|
|
|
63
65
|
| User outcome | MCP tool or skill |
|
|
64
66
|
| --- | --- |
|
|
65
|
-
|
|
|
66
|
-
|
|
|
67
|
-
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
URL. The raw response preserves complete page context for later questions,
|
|
76
|
-
validation, and custom processing. Requesting JSON, a table, or named fields is
|
|
77
|
-
not by itself a reason to use scrape; derive those outputs locally from the raw
|
|
78
|
-
content when practical.
|
|
79
|
-
|
|
80
|
-
The general and listing scrape modes send page content through a backend LLM.
|
|
81
|
-
Their output is shaped by the extraction prompt and may omit information.
|
|
82
|
-
Applying them across many pages also repeats model work that the agent can often
|
|
83
|
-
replace with one reusable local extractor.
|
|
67
|
+
| Retrieve and preserve a known public page as HTML, Markdown, or a screenshot | [mrscraper-fetch](../mrscraper-fetch/SKILL.md) with scraperAPI |
|
|
68
|
+
| Discover relevant pages from a Google query | [mrscraper-serp](../mrscraper-serp/SKILL.md) with webSearch, then retrieve selected pages |
|
|
69
|
+
| Check the plan, token usage, or remaining tokens | status |
|
|
70
|
+
|
|
71
|
+
### Retrieve first, extract locally
|
|
72
|
+
|
|
73
|
+
scraperAPI returns raw page content. Produce summaries, tables, JSON, and other
|
|
74
|
+
structured output from that content with local analysis or extraction logic.
|
|
75
|
+
The raw response preserves complete page context for later questions,
|
|
76
|
+
validation, and changes to the requested schema.
|
|
84
77
|
|
|
85
78
|
Prefer this workflow for pages with a shared layout:
|
|
86
79
|
|
|
87
|
-
1.
|
|
80
|
+
1. Retrieve representative pages and inspect their content.
|
|
88
81
|
2. Understand the shared page structure and required fields.
|
|
89
82
|
3. Define a stable output schema and reusable local extractor.
|
|
90
|
-
4.
|
|
83
|
+
4. Retrieve the remaining pages, in parallel when safe.
|
|
91
84
|
5. Apply the same extractor to every saved response.
|
|
92
85
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
faster, preserves every raw input, and allows fields to be recovered later if
|
|
96
|
-
the schema changes.
|
|
86
|
+
Request Markdown when readable text is enough and HTML when an extractor needs
|
|
87
|
+
the page structure.
|
|
97
88
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
fetched source and verify important values against it. The map agent is separate
|
|
101
|
-
URL-discovery functionality; use it for bounded site mapping, then fetch the
|
|
102
|
-
pages whose content matters.
|
|
89
|
+
For discovery-first work, call webSearch, select relevant URLs, and retrieve
|
|
90
|
+
the pages whose content will inform the answer.
|
|
103
91
|
|
|
104
|
-
|
|
105
|
-
|
|
92
|
+
MrScraper MCP does not create AI scrapers, rerun saved scrapers, or list stored
|
|
93
|
+
results. Point the user to the MrScraper dashboard at https://app.mrscraper.com
|
|
94
|
+
for those.
|
|
106
95
|
|
|
107
96
|
## Step 4 — Handle Output and Artifacts
|
|
108
97
|
|
|
109
|
-
|
|
110
|
-
in a formatted JSON text block:
|
|
98
|
+
scraperAPI and webSearch return the same response envelope in MCP
|
|
99
|
+
structuredContent and in a formatted JSON text block:
|
|
111
100
|
|
|
112
101
|
{
|
|
113
102
|
"status_code": 200,
|
|
@@ -117,202 +106,58 @@ in a formatted JSON text block:
|
|
|
117
106
|
|
|
118
107
|
Prefer structuredContent when available. An API failure adds error and sets the
|
|
119
108
|
MCP result's isError flag. An input-contract failure is returned as a tool
|
|
120
|
-
error. Check those signals before trusting data.
|
|
121
|
-
|
|
122
|
-
Credentials, cookies, and generated curl credentials are sanitized. Extracted
|
|
123
|
-
scraper data is preserved even when it contains fields named token or password
|
|
124
|
-
because those fields may be legitimate source values.
|
|
109
|
+
error. Check those signals before trusting data. Credential-bearing headers and
|
|
110
|
+
credential metadata are removed from responses.
|
|
125
111
|
|
|
126
112
|
MrScraper tools do not accept output paths. When the user requests an artifact,
|
|
127
113
|
wait for a successful tool result, select the requested value, and save it with
|
|
128
|
-
the environment's local file-writing capability.
|
|
129
|
-
|
|
130
|
-
separately only when run metadata, headers, or diagnostics matter.
|
|
114
|
+
the environment's local file-writing capability. A screenshot is a base64 JPEG
|
|
115
|
+
in data.screenshot; decode it before saving an image.
|
|
131
116
|
|
|
132
117
|
Save substantial intermediate artifacts under the current project's
|
|
133
118
|
.mrscraper directory unless the user specifies another location. This project
|
|
134
119
|
artifact directory is unrelated to MCP connection credentials. Keep artifacts
|
|
135
120
|
out of version control unless the user asks to commit them.
|
|
136
121
|
|
|
137
|
-
## Step 5 —
|
|
138
|
-
|
|
139
|
-
A successful scrape creates a saved AI scraper configuration. Read its UUID
|
|
140
|
-
from structuredContent.data.data.scraperId and use rerun to apply the same
|
|
141
|
-
configuration to the same or another URL. Prefer rerun over rebuilding a prompt
|
|
142
|
-
when the user wants a repeatable version of an earlier extraction.
|
|
143
|
-
|
|
144
|
-
The scraperId makes the configuration reusable but does not guarantee
|
|
145
|
-
identical extracted values when the source page or model behavior changes.
|
|
146
|
-
|
|
147
|
-
Choose scraper type and target count independently:
|
|
148
|
-
|
|
149
|
-
1. Use type=ai for a saved AI scraper created by scrape.
|
|
150
|
-
2. Use type=manual for a saved step-based workflow created in the MrScraper
|
|
151
|
-
dashboard. MCP can rerun but cannot create manual workflows.
|
|
152
|
-
3. Leave bulk=false for one URL. Set bulk=true for a comma- or
|
|
153
|
-
newline-separated URL list submitted as one asynchronous job.
|
|
154
|
-
|
|
155
|
-
| Mode | Required ID input | Target | Crawl controls |
|
|
156
|
-
| --- | --- | --- | --- |
|
|
157
|
-
| Single AI | scraper_id | One URL | Supported |
|
|
158
|
-
| Single manual | scraper_id | One URL | Not accepted |
|
|
159
|
-
| Bulk AI | bulk=true and id | Comma/newline-separated URLs | Not accepted |
|
|
160
|
-
| Bulk manual | bulk=true and id | Comma/newline-separated URLs | Not accepted |
|
|
161
|
-
|
|
162
|
-
Single AI example:
|
|
163
|
-
|
|
164
|
-
{
|
|
165
|
-
"target": "https://www.scrapethissite.com/pages/simple/",
|
|
166
|
-
"type": "ai",
|
|
167
|
-
"scraper_id": "SCRAPER_UUID"
|
|
168
|
-
}
|
|
169
|
-
|
|
170
|
-
Single AI optionally accepts max_depth, max_pages, limit, include_patterns,
|
|
171
|
-
exclude_patterns, proxy_country, max_retry, and timeout. Omit any control that
|
|
172
|
-
the user did not request so the saved scraper or backend default remains in
|
|
173
|
-
effect; the MCP server does not inject replacement defaults.
|
|
174
|
-
|
|
175
|
-
Explicit-control example:
|
|
176
|
-
|
|
177
|
-
{
|
|
178
|
-
"target": "https://www.scrapethissite.com/pages/forms/?page_num=1",
|
|
179
|
-
"type": "ai",
|
|
180
|
-
"scraper_id": "SCRAPER_UUID",
|
|
181
|
-
"proxy_country": "ID",
|
|
182
|
-
"max_retry": 4,
|
|
183
|
-
"timeout": 120
|
|
184
|
-
}
|
|
185
|
-
|
|
186
|
-
These controls are rejected by manual and bulk reruns.
|
|
187
|
-
|
|
188
|
-
Bulk example:
|
|
122
|
+
## Step 5 — Review Account Usage
|
|
189
123
|
|
|
190
|
-
|
|
191
|
-
"target": "https://www.scrapethissite.com/pages/simple/,https://www.scrapethissite.com/pages/forms/?page_num=1",
|
|
192
|
-
"type": "ai",
|
|
193
|
-
"bulk": true,
|
|
194
|
-
"id": "SCRAPER_UUID"
|
|
195
|
-
}
|
|
196
|
-
|
|
197
|
-
Manual example, after the user accepts the warning below:
|
|
198
|
-
|
|
199
|
-
{
|
|
200
|
-
"target": "https://www.scrapethissite.com/pages/simple/",
|
|
201
|
-
"type": "manual",
|
|
202
|
-
"scraper_id": "SCRAPER_UUID",
|
|
203
|
-
"acknowledged": true
|
|
204
|
-
}
|
|
205
|
-
|
|
206
|
-
Bulk rerun submits one asynchronous backend job. Retain
|
|
207
|
-
structuredContent.data.data.bulkResultId and pass it to result as result_id.
|
|
208
|
-
Inspect that record until it reaches a terminal state when the user asked to
|
|
209
|
-
wait for completion. Do not resubmit the bulk rerun because its result is still
|
|
210
|
-
running.
|
|
211
|
-
|
|
212
|
-
Before the first rerun call with type=manual in a conversation, show the
|
|
213
|
-
following warning exactly once and wait for the user's acknowledgment. Do not
|
|
214
|
-
call the tool until the user accepts it, and do not repeat the warning on later
|
|
215
|
-
manual reruns in the same conversation. After the user accepts, pass
|
|
216
|
-
acknowledged=true on every manual rerun in that conversation; rerun refuses a
|
|
217
|
-
manual rerun without it and returns this warning instead.
|
|
218
|
-
|
|
219
|
-
> ### Compliance & Legal Risk
|
|
220
|
-
>
|
|
221
|
-
> WARNING
|
|
222
|
-
>
|
|
223
|
-
> **Scraping login-protected pages carries serious legal and compliance
|
|
224
|
-
> risks.** Many websites explicitly prohibit automated access in their Terms
|
|
225
|
-
> of Service, and bypassing authentication to scrape content may expose you to
|
|
226
|
-
> legal action including lawsuits, account termination, and financial
|
|
227
|
-
> penalties. By proceeding on scraping login-protected pages, you confirm that
|
|
228
|
-
> you have read and understood the target website's Terms of Service, and you
|
|
229
|
-
> **fully accept all legal, financial, and ethical responsibility** for your
|
|
230
|
-
> actions.
|
|
231
|
-
|
|
232
|
-
## Step 6 — Inspect Stored Results
|
|
233
|
-
|
|
234
|
-
Use results when the exact result UUID is unknown:
|
|
235
|
-
|
|
236
|
-
{
|
|
237
|
-
"sort_field": "updatedAt",
|
|
238
|
-
"sort_order": "desc",
|
|
239
|
-
"page_size": 20,
|
|
240
|
-
"page": 1
|
|
241
|
-
}
|
|
242
|
-
|
|
243
|
-
Apply exact backend filters when the user knows identifying result fields:
|
|
244
|
-
|
|
245
|
-
{
|
|
246
|
-
"scraper_id": "SCRAPER_UUID",
|
|
247
|
-
"status": "Finished",
|
|
248
|
-
"type": "Rerun-AI",
|
|
249
|
-
"url": "https://www.scrapethissite.com/pages/simple/"
|
|
250
|
-
}
|
|
251
|
-
|
|
252
|
-
| Input | Default | Use |
|
|
253
|
-
| --- | --- | --- |
|
|
254
|
-
| sort_field | updatedAt | Stored-result sort key. |
|
|
255
|
-
| sort_order | desc | Sort direction: asc or desc. |
|
|
256
|
-
| page_size | 10 | Positive number of records per page. |
|
|
257
|
-
| page | 1 | One-based page number. |
|
|
258
|
-
| search | omitted | Free-text result filter. |
|
|
259
|
-
| date_range_column | omitted | Date column used by start_at and end_at. |
|
|
260
|
-
| start_at | omitted | Inclusive ISO 8601 range start. |
|
|
261
|
-
| end_at | omitted | Inclusive ISO 8601 range end. |
|
|
262
|
-
| scraper_id | omitted | Exact saved scraper UUID filter. |
|
|
263
|
-
| status | omitted | Exact Draft, Finished, Running, Failed, or Cancelled filter. |
|
|
264
|
-
| type | omitted | Exact result type filter, such as AI or Rerun-AI. |
|
|
265
|
-
| url | omitted | Exact stored target URL filter. |
|
|
266
|
-
|
|
267
|
-
Use result when the UUID is known:
|
|
268
|
-
|
|
269
|
-
{
|
|
270
|
-
"result_id": "RESULT_UUID",
|
|
271
|
-
"include_html": false
|
|
272
|
-
}
|
|
273
|
-
|
|
274
|
-
A result_id may also be the bulkResultId returned by an asynchronous rerun.
|
|
275
|
-
include_html defaults to true. Set it to false for a smaller response while
|
|
276
|
-
polling status or when extracted data is sufficient.
|
|
277
|
-
|
|
278
|
-
## Step 7 — Review Account Usage
|
|
279
|
-
|
|
280
|
-
Call status with an empty object for subscription and token usage:
|
|
124
|
+
Call status with an empty object:
|
|
281
125
|
|
|
282
126
|
{}
|
|
283
127
|
|
|
284
|
-
|
|
128
|
+
The result summarizes the subscription status, token limit, token usage,
|
|
129
|
+
remaining tokens, usage percentage, rate limit, auto-renewal, renewal date,
|
|
130
|
+
and the account's name, email, and verification state:
|
|
285
131
|
|
|
286
132
|
{
|
|
287
|
-
"
|
|
288
|
-
"
|
|
289
|
-
"
|
|
133
|
+
"kind": "mrscraper-cli-status-summary",
|
|
134
|
+
"status_code": 200,
|
|
135
|
+
"data": {
|
|
136
|
+
"account": {
|
|
137
|
+
"subscription_status": "active",
|
|
138
|
+
"token_usage": 250,
|
|
139
|
+
"token_limit": 1000,
|
|
140
|
+
"token_remaining": 750,
|
|
141
|
+
"usage_percent": 25
|
|
142
|
+
}
|
|
143
|
+
}
|
|
290
144
|
}
|
|
291
145
|
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
| domain | omitted | Add request outcomes for a normalized hostname. |
|
|
295
|
-
| from | 24h | ISO start or relative duration such as 30m, 24h, or 7d. |
|
|
296
|
-
| to | now | ISO end, now, or a relative duration. |
|
|
297
|
-
| action | omitted | Exact request-action filter, used only with domain. |
|
|
298
|
-
| api_token_name | omitted | API-token-name filter, used only with domain. |
|
|
299
|
-
|
|
300
|
-
Domain outcomes describe MrScraper requests. They are not traffic, audience,
|
|
301
|
-
SEO, or market analytics. status reports account and request health, not
|
|
302
|
-
scrape-job progress; use result for a known asynchronous result ID.
|
|
146
|
+
Check remaining tokens before a large batch of page retrievals. status reports
|
|
147
|
+
account usage, not the progress of a request.
|
|
303
148
|
|
|
304
|
-
## Step
|
|
149
|
+
## Step 6 — Troubleshoot
|
|
305
150
|
|
|
306
151
|
- Missing tools: enable or reconnect the MrScraper MCP connection, then confirm
|
|
307
|
-
|
|
152
|
+
scraperAPI, webSearch, and status are exposed.
|
|
308
153
|
- Unauthorized or expired authorization: reauthorize the OAuth 2.1 connection
|
|
309
154
|
through the MCP client.
|
|
155
|
+
- 403 insufficient_scope: reauthorize so the client requests the missing scope.
|
|
310
156
|
- Input-contract error: inspect the tool schema and omit inputs that do not
|
|
311
|
-
apply
|
|
157
|
+
apply. The previous fetch, scrape, serp, rerun, results, and result tools were
|
|
158
|
+
replaced by scraperAPI, webSearch, and status.
|
|
312
159
|
- Page incomplete or missing dynamic content: load mrscraper-fetch and revise
|
|
313
160
|
page-loading inputs deliberately.
|
|
314
|
-
- Extraction incomplete: load mrscraper-scrape and improve the prompt, agent,
|
|
315
|
-
or limits.
|
|
316
161
|
- Search results weak: load mrscraper-serp and refine the query, locale, or
|
|
317
162
|
page.
|
|
318
163
|
|