@mrscraper/opencode 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,22 +1,21 @@
1
1
  ---
2
2
  name: mrscraper
3
3
  description: |
4
- Connect, authenticate, route, and troubleshoot MrScraper MCP, and use saved scrapers, stored results, and account tools. For known public URLs, acquire and preserve raw content with mrscraper-fetch first; use mrscraper-scrape only for explicitly requested or clearly beneficial managed extraction, and mrscraper-serp for query-first discovery.
4
+ Connect, authenticate, route, and troubleshoot MrScraper MCP, and check account usage with the status tool. For known public URLs, retrieve and preserve page content with mrscraper-fetch first; use mrscraper-serp for query-first discovery through Google.
5
5
  ---
6
6
 
7
7
  # MrScraper MCP
8
8
 
9
9
  Use this skill for connection and authentication checks, tool selection, shared
10
- response handling, saved reruns, stored results, account status, and
11
- troubleshooting. Load the focused fetch, scrape, or SERP skill for those
12
- workflows.
10
+ response handling, account status, and troubleshooting. Load the focused fetch
11
+ or SERP skill for page retrieval and Google search.
13
12
 
14
13
  ## Step 1 — Verify or Connect MrScraper MCP
15
14
 
16
- Check whether the current client exposes these seven tools from the MrScraper
15
+ Check whether the current client exposes these three tools from the MrScraper
17
16
  MCP provider:
18
17
 
19
- fetch scrape serp status rerun results result
18
+ scraperAPI webSearch status
20
19
 
21
20
  The client may display server-qualified names. Match the provider and
22
21
  unqualified tool name; do not assume a fixed namespace.
@@ -29,7 +28,7 @@ the managed Streamable HTTP endpoint:
29
28
 
30
29
  A self-managed deployment may expose the same tools over Streamable HTTP or
31
30
  stdio. Connection setup belongs to the MCP client. After setup or reconnection,
32
- list the available tools and confirm all seven names before claiming the
31
+ list the available tools and confirm all three names before claiming the
33
32
  integration is ready.
34
33
 
35
34
  ## Step 2 — Authenticate
@@ -48,66 +47,56 @@ Never place credentials in tool arguments, print them, write them into project
48
47
  artifacts, or manually reproduce the OAuth exchange. Do not configure an
49
48
  API-key or bearer-token fallback from these skills.
50
49
 
51
- Verify end-to-end access with one small fetch:
50
+ The connection uses two scopes: scrape:read for scraperAPI and webSearch, and
51
+ account:read for status. A tool called without its scope returns 403
52
+ insufficient_scope; reauthorize the connection so the client requests it.
52
53
 
53
- {
54
- "url": "https://www.scrapethissite.com/pages/simple/"
55
- }
54
+ Verify end-to-end access with status, which reads the account without
55
+ retrieving a page:
56
+
57
+ {}
56
58
 
57
- A successful page response confirms that the connection can authenticate and
58
- reach the web-data service. If it fails, report the error and do not claim
59
- setup is complete.
59
+ A successful account summary confirms that the connection can authenticate and
60
+ reach MrScraper. If it fails, report the error and do not claim setup is
61
+ complete.
60
62
 
61
63
  ## Step 3 — Route the Request
62
64
 
63
65
  | User outcome | MCP tool or skill |
64
66
  | --- | --- |
65
- | Acquire and preserve raw content from known public URLs for any downstream task | [mrscraper-fetch](../mrscraper-fetch/SKILL.md), preferably first |
66
- | Add managed extraction, repeated records, site mapping, schema guidance, or a reusable scraper | [mrscraper-scrape](../mrscraper-scrape/SKILL.md), normally after or alongside fetch |
67
- | Discover relevant pages from a Google query | [mrscraper-serp](../mrscraper-serp/SKILL.md), then fetch selected pages |
68
- | Reproduce a prior scrape or run an existing AI/manual scraper on new URLs | rerun |
69
- | List or retrieve stored results | results or result |
70
- | Check account usage or domain request outcomes | status |
71
-
72
- ### Fetch-first principle
73
-
74
- For agent-led work, always use fetch for the first exploration of a known public
75
- URL. The raw response preserves complete page context for later questions,
76
- validation, and custom processing. Requesting JSON, a table, or named fields is
77
- not by itself a reason to use scrape; derive those outputs locally from the raw
78
- content when practical.
79
-
80
- The general and listing scrape modes send page content through a backend LLM.
81
- Their output is shaped by the extraction prompt and may omit information.
82
- Applying them across many pages also repeats model work that the agent can often
83
- replace with one reusable local extractor.
67
+ | Retrieve and preserve a known public page as HTML, Markdown, or a screenshot | [mrscraper-fetch](../mrscraper-fetch/SKILL.md) with scraperAPI |
68
+ | Discover relevant pages from a Google query | [mrscraper-serp](../mrscraper-serp/SKILL.md) with webSearch, then retrieve selected pages |
69
+ | Check the plan, token usage, or remaining tokens | status |
70
+
71
+ ### Retrieve first, extract locally
72
+
73
+ scraperAPI returns raw page content. Produce summaries, tables, JSON, and other
74
+ structured output from that content with local analysis or extraction logic.
75
+ The raw response preserves complete page context for later questions,
76
+ validation, and changes to the requested schema.
84
77
 
85
78
  Prefer this workflow for pages with a shared layout:
86
79
 
87
- 1. Fetch representative pages and inspect their raw content.
80
+ 1. Retrieve representative pages and inspect their content.
88
81
  2. Understand the shared page structure and required fields.
89
82
  3. Define a stable output schema and reusable local extractor.
90
- 4. Fetch the remaining pages, in parallel when safe.
83
+ 4. Retrieve the remaining pages, in parallel when safe.
91
84
  5. Apply the same extractor to every saved response.
92
85
 
93
- For roughly 100 same-layout pages, that means 100 fetches followed by one local
94
- batch extraction instead of 100 separate backend-LLM extractions. This is often
95
- faster, preserves every raw input, and allows fields to be recovered later if
96
- the schema changes.
86
+ Request Markdown when readable text is enough and HTML when an extractor needs
87
+ the page structure.
97
88
 
98
- Use scrape only when the user explicitly requests managed extraction or when it
99
- still offers a clear benefit after fetch-led exploration. Even then, retain the
100
- fetched source and verify important values against it. The map agent is separate
101
- URL-discovery functionality; use it for bounded site mapping, then fetch the
102
- pages whose content matters.
89
+ For discovery-first work, call webSearch, select relevant URLs, and retrieve
90
+ the pages whose content will inform the answer.
103
91
 
104
- For discovery-first work, call serp, select relevant URLs, fetch those pages,
105
- and add managed extraction only when a derived structured view is useful.
92
+ MrScraper MCP does not create AI scrapers, rerun saved scrapers, or list stored
93
+ results. Point the user to the MrScraper dashboard at https://app.mrscraper.com
94
+ for those.
106
95
 
107
96
  ## Step 4 — Handle Output and Artifacts
108
97
 
109
- API-backed tools return the same response envelope in MCP structuredContent and
110
- in a formatted JSON text block:
98
+ scraperAPI and webSearch return the same response envelope in MCP
99
+ structuredContent and in a formatted JSON text block:
111
100
 
112
101
  {
113
102
  "status_code": 200,
@@ -117,202 +106,58 @@ in a formatted JSON text block:
117
106
 
118
107
  Prefer structuredContent when available. An API failure adds error and sets the
119
108
  MCP result's isError flag. An input-contract failure is returned as a tool
120
- error. Check those signals before trusting data.
121
-
122
- Credentials, cookies, and generated curl credentials are sanitized. Extracted
123
- scraper data is preserved even when it contains fields named token or password
124
- because those fields may be legitimate source values.
109
+ error. Check those signals before trusting data. Credential-bearing headers and
110
+ credential metadata are removed from responses.
125
111
 
126
112
  MrScraper tools do not accept output paths. When the user requests an artifact,
127
113
  wait for a successful tool result, select the requested value, and save it with
128
- the environment's local file-writing capability. For scrape, the extracted
129
- value is normally at structuredContent.data.data.data. Keep the full envelope
130
- separately only when run metadata, headers, or diagnostics matter.
114
+ the environment's local file-writing capability. A screenshot is a base64 JPEG
115
+ in data.screenshot; decode it before saving an image.
131
116
 
132
117
  Save substantial intermediate artifacts under the current project's
133
118
  .mrscraper directory unless the user specifies another location. This project
134
119
  artifact directory is unrelated to MCP connection credentials. Keep artifacts
135
120
  out of version control unless the user asks to commit them.
136
121
 
137
- ## Step 5 — Rerun Saved Scrapers
138
-
139
- A successful scrape creates a saved AI scraper configuration. Read its UUID
140
- from structuredContent.data.data.scraperId and use rerun to apply the same
141
- configuration to the same or another URL. Prefer rerun over rebuilding a prompt
142
- when the user wants a repeatable version of an earlier extraction.
143
-
144
- The scraperId makes the configuration reusable but does not guarantee
145
- identical extracted values when the source page or model behavior changes.
146
-
147
- Choose scraper type and target count independently:
148
-
149
- 1. Use type=ai for a saved AI scraper created by scrape.
150
- 2. Use type=manual for a saved step-based workflow created in the MrScraper
151
- dashboard. MCP can rerun but cannot create manual workflows.
152
- 3. Leave bulk=false for one URL. Set bulk=true for a comma- or
153
- newline-separated URL list submitted as one asynchronous job.
154
-
155
- | Mode | Required ID input | Target | Crawl controls |
156
- | --- | --- | --- | --- |
157
- | Single AI | scraper_id | One URL | Supported |
158
- | Single manual | scraper_id | One URL | Not accepted |
159
- | Bulk AI | bulk=true and id | Comma/newline-separated URLs | Not accepted |
160
- | Bulk manual | bulk=true and id | Comma/newline-separated URLs | Not accepted |
161
-
162
- Single AI example:
163
-
164
- {
165
- "target": "https://www.scrapethissite.com/pages/simple/",
166
- "type": "ai",
167
- "scraper_id": "SCRAPER_UUID"
168
- }
169
-
170
- Single AI optionally accepts max_depth, max_pages, limit, include_patterns,
171
- exclude_patterns, proxy_country, max_retry, and timeout. Omit any control that
172
- the user did not request so the saved scraper or backend default remains in
173
- effect; the MCP server does not inject replacement defaults.
174
-
175
- Explicit-control example:
176
-
177
- {
178
- "target": "https://www.scrapethissite.com/pages/forms/?page_num=1",
179
- "type": "ai",
180
- "scraper_id": "SCRAPER_UUID",
181
- "proxy_country": "ID",
182
- "max_retry": 4,
183
- "timeout": 120
184
- }
185
-
186
- These controls are rejected by manual and bulk reruns.
187
-
188
- Bulk example:
122
+ ## Step 5 — Review Account Usage
189
123
 
190
- {
191
- "target": "https://www.scrapethissite.com/pages/simple/,https://www.scrapethissite.com/pages/forms/?page_num=1",
192
- "type": "ai",
193
- "bulk": true,
194
- "id": "SCRAPER_UUID"
195
- }
196
-
197
- Manual example, after the user accepts the warning below:
198
-
199
- {
200
- "target": "https://www.scrapethissite.com/pages/simple/",
201
- "type": "manual",
202
- "scraper_id": "SCRAPER_UUID",
203
- "acknowledged": true
204
- }
205
-
206
- Bulk rerun submits one asynchronous backend job. Retain
207
- structuredContent.data.data.bulkResultId and pass it to result as result_id.
208
- Inspect that record until it reaches a terminal state when the user asked to
209
- wait for completion. Do not resubmit the bulk rerun because its result is still
210
- running.
211
-
212
- Before the first rerun call with type=manual in a conversation, show the
213
- following warning exactly once and wait for the user's acknowledgment. Do not
214
- call the tool until the user accepts it, and do not repeat the warning on later
215
- manual reruns in the same conversation. After the user accepts, pass
216
- acknowledged=true on every manual rerun in that conversation; rerun refuses a
217
- manual rerun without it and returns this warning instead.
218
-
219
- > ### Compliance & Legal Risk
220
- >
221
- > WARNING
222
- >
223
- > **Scraping login-protected pages carries serious legal and compliance
224
- > risks.** Many websites explicitly prohibit automated access in their Terms
225
- > of Service, and bypassing authentication to scrape content may expose you to
226
- > legal action including lawsuits, account termination, and financial
227
- > penalties. By proceeding on scraping login-protected pages, you confirm that
228
- > you have read and understood the target website's Terms of Service, and you
229
- > **fully accept all legal, financial, and ethical responsibility** for your
230
- > actions.
231
-
232
- ## Step 6 — Inspect Stored Results
233
-
234
- Use results when the exact result UUID is unknown:
235
-
236
- {
237
- "sort_field": "updatedAt",
238
- "sort_order": "desc",
239
- "page_size": 20,
240
- "page": 1
241
- }
242
-
243
- Apply exact backend filters when the user knows identifying result fields:
244
-
245
- {
246
- "scraper_id": "SCRAPER_UUID",
247
- "status": "Finished",
248
- "type": "Rerun-AI",
249
- "url": "https://www.scrapethissite.com/pages/simple/"
250
- }
251
-
252
- | Input | Default | Use |
253
- | --- | --- | --- |
254
- | sort_field | updatedAt | Stored-result sort key. |
255
- | sort_order | desc | Sort direction: asc or desc. |
256
- | page_size | 10 | Positive number of records per page. |
257
- | page | 1 | One-based page number. |
258
- | search | omitted | Free-text result filter. |
259
- | date_range_column | omitted | Date column used by start_at and end_at. |
260
- | start_at | omitted | Inclusive ISO 8601 range start. |
261
- | end_at | omitted | Inclusive ISO 8601 range end. |
262
- | scraper_id | omitted | Exact saved scraper UUID filter. |
263
- | status | omitted | Exact Draft, Finished, Running, Failed, or Cancelled filter. |
264
- | type | omitted | Exact result type filter, such as AI or Rerun-AI. |
265
- | url | omitted | Exact stored target URL filter. |
266
-
267
- Use result when the UUID is known:
268
-
269
- {
270
- "result_id": "RESULT_UUID",
271
- "include_html": false
272
- }
273
-
274
- A result_id may also be the bulkResultId returned by an asynchronous rerun.
275
- include_html defaults to true. Set it to false for a smaller response while
276
- polling status or when extracted data is sufficient.
277
-
278
- ## Step 7 — Review Account Usage
279
-
280
- Call status with an empty object for subscription and token usage:
124
+ Call status with an empty object:
281
125
 
282
126
  {}
283
127
 
284
- Add domain request outcomes and a time range only when needed:
128
+ The result summarizes the subscription status, token limit, token usage,
129
+ remaining tokens, usage percentage, rate limit, auto-renewal, renewal date,
130
+ and the account's name, email, and verification state:
285
131
 
286
132
  {
287
- "domain": "www.scrapethissite.com",
288
- "from": "7d",
289
- "to": "now"
133
+ "kind": "mrscraper-cli-status-summary",
134
+ "status_code": 200,
135
+ "data": {
136
+ "account": {
137
+ "subscription_status": "active",
138
+ "token_usage": 250,
139
+ "token_limit": 1000,
140
+ "token_remaining": 750,
141
+ "usage_percent": 25
142
+ }
143
+ }
290
144
  }
291
145
 
292
- | Input | Default | Use |
293
- | --- | --- | --- |
294
- | domain | omitted | Add request outcomes for a normalized hostname. |
295
- | from | 24h | ISO start or relative duration such as 30m, 24h, or 7d. |
296
- | to | now | ISO end, now, or a relative duration. |
297
- | action | omitted | Exact request-action filter, used only with domain. |
298
- | api_token_name | omitted | API-token-name filter, used only with domain. |
299
-
300
- Domain outcomes describe MrScraper requests. They are not traffic, audience,
301
- SEO, or market analytics. status reports account and request health, not
302
- scrape-job progress; use result for a known asynchronous result ID.
146
+ Check remaining tokens before a large batch of page retrievals. status reports
147
+ account usage, not the progress of a request.
303
148
 
304
- ## Step 8 — Troubleshoot
149
+ ## Step 6 — Troubleshoot
305
150
 
306
151
  - Missing tools: enable or reconnect the MrScraper MCP connection, then confirm
307
- all seven tools are exposed.
152
+ scraperAPI, webSearch, and status are exposed.
308
153
  - Unauthorized or expired authorization: reauthorize the OAuth 2.1 connection
309
154
  through the MCP client.
155
+ - 403 insufficient_scope: reauthorize so the client requests the missing scope.
310
156
  - Input-contract error: inspect the tool schema and omit inputs that do not
311
- apply to the selected mode.
157
+ apply. The previous fetch, scrape, serp, rerun, results, and result tools were
158
+ replaced by scraperAPI, webSearch, and status.
312
159
  - Page incomplete or missing dynamic content: load mrscraper-fetch and revise
313
160
  page-loading inputs deliberately.
314
- - Extraction incomplete: load mrscraper-scrape and improve the prompt, agent,
315
- or limits.
316
161
  - Search results weak: load mrscraper-serp and refine the query, locale, or
317
162
  page.
318
163