@mrscraper/opencode 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,35 +1,33 @@
1
1
  ---
2
2
  name: mrscraper-fetch
3
3
  description: |
4
- Retrieve raw HTML from a known public URL with MrScraper. Use for reading, inspecting, summarizing, archiving, or analyzing a page, and as the default content-acquisition step before agent-led or local extraction. Supports browser rendering, real-device Super Mode, and page-load controls; use mrscraper-serp when no target URL is known.
4
+ Retrieve a known public URL with MrScraper's scraperAPI tool as HTML, Markdown, a screenshot, or a combination. Use for reading, summarizing, archiving, or analyzing a page and before local extraction; supports automatic browser fallback, real-device routing, proxies, and retries. Use mrscraper-serp when no URL is known.
5
5
  ---
6
6
 
7
- # Fetch Page Content with MrScraper MCP
7
+ # Retrieve Page Content with MrScraper MCP
8
8
 
9
- Use the fetch tool supplied by the MrScraper MCP server for the first exploration
10
- whenever the user already has a public URL. Keep using the raw response for
11
- summarization, comparison, transformation, and structured output instead of
12
- handing the page to another LLM by default. Use
9
+ Use the scraperAPI tool supplied by the MrScraper MCP server whenever the user
10
+ already has a public URL. Keep using the retrieved content for summarization,
11
+ comparison, transformation, and structured output. Use
13
12
  [mrscraper](../mrscraper/SKILL.md) for connection and authentication checks,
14
- saved runs, account status, or broader routing.
13
+ account status, or broader routing, and
14
+ [mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen first.
15
15
 
16
16
  ## Page Retrieval Context
17
17
 
18
- fetch calls MrScraper's
18
+ scraperAPI calls MrScraper's
19
19
  [page-fetching service](https://docs.mrscraper.com/docs/features/unblocker) once.
20
- Browser rendering executes page JavaScript, locale routing selects a
21
- geo-specific country, selector waits allow delayed content to appear, and
22
- homepage navigation establishes a normal navigation path before loading the
23
- target. Super Mode selects real-device routing independently of browser
24
- rendering.
20
+ Browser rendering executes page JavaScript, geographic routing selects the
21
+ proxy country, selector waits let delayed content appear, and homepage
22
+ navigation visits the site root before the target. Real-device routing (super)
23
+ is independent of browser rendering.
25
24
 
26
- Use fetch only for content the user is authorized to access and in accordance
27
- with the site's requirements. Page-loading controls can help render a site that
28
- does not work with a basic request.
25
+ Use scraperAPI only for content the user is authorized to access and in
26
+ accordance with the site's requirements.
29
27
 
30
28
  Start with the URL alone and add only the controls the target needs. Plan-token
31
29
  usage is based on runtime and bandwidth: one token per 30 seconds and one token
32
- per 0.2 MB, rounded up per component. Resource blocking can reduce bandwidth
30
+ per 0.25 MB, rounded up per component. Resource blocking can reduce bandwidth
33
31
  for text-focused pages. Retries stop when the request succeeds, max_retries is
34
32
  reached, or the running token total reaches token_cap. The initial request
35
33
  always runs even when it exceeds that cap. See the
@@ -40,96 +38,79 @@ maintained calculation.
40
38
 
41
39
  Confirm the target URL and what the user wants:
42
40
 
43
- - Preserve a raw source before extraction, transformation, or comparison.
44
41
  - Read, summarize, cite, or inspect the page.
45
42
  - Check whether specific text appears.
46
- - Archive the response.
43
+ - Archive the page or capture a screenshot.
47
44
  - Produce fields, JSON, tables, or other structured output with local logic.
48
- - Verify or supplement a managed scrape or saved result.
49
45
  - Load JavaScript-rendered or geo-sensitive content.
50
46
 
51
- Do not call scrape merely because the requested output is structured. Fetch the
52
- page, understand its layout, and transform the saved raw content locally. Use
53
- [mrscraper-scrape](../mrscraper-scrape/SKILL.md) only when the user explicitly
54
- requests managed extraction, or after fetch-led exploration has produced a
55
- stable output schema and managed extraction still offers a concrete benefit.
56
- Use [mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen
57
- first.
47
+ When many pages share a layout, retrieve representative pages, define one local
48
+ extractor, retrieve the remaining pages in parallel when safe and proportional,
49
+ and apply the same extractor to every saved response.
58
50
 
59
- ### Prefer reusable local extraction
51
+ ## Step 2 — Retrieve the Page
60
52
 
61
- When many pages share a layout:
53
+ The client may namespace the tool name; select scraperAPI from the MrScraper
54
+ MCP provider. Start with one call containing only the required URL, which
55
+ requests HTML:
62
56
 
63
- 1. Fetch representative pages and inspect their raw content.
64
- 2. Define one local extraction schema and implementation.
65
- 3. Fetch the remaining pages, in parallel when safe and proportional.
66
- 4. Apply the same local extractor to every saved response.
67
-
68
- For roughly 100 same-layout pages, concurrent fetches followed by one local
69
- batch extraction are often faster than 100 separate backend-LLM extractions.
70
- This also preserves every raw input for later recovery or schema changes.
71
-
72
- ## Step 2 — Run the Fetch
57
+ {
58
+ "url": "https://www.scrapethissite.com/pages/simple/"
59
+ }
73
60
 
74
- The client may namespace the tool name; select fetch from the MrScraper MCP
75
- provider. Start with one call containing only the required URL:
61
+ Request Markdown instead of HTML for readable text:
76
62
 
77
63
  {
78
- "url": "https://www.scrapethissite.com/pages/simple/"
64
+ "url": "https://www.scrapethissite.com/pages/simple/",
65
+ "html": false,
66
+ "markdown": true
79
67
  }
80
68
 
81
- The result is available in MCP structuredContent and as formatted JSON text:
69
+ The result is available in MCP structuredContent and as formatted JSON text.
70
+ The requested representations are in data:
82
71
 
83
72
  {
84
73
  "status_code": 200,
85
- "data": "<html>...</html>",
74
+ "data": {
75
+ "html": "<!DOCTYPE html>...",
76
+ "markdown": null,
77
+ "screenshot": null,
78
+ "token_usage": 3,
79
+ "runtime": 18.02,
80
+ "retry_count": 0,
81
+ "error": null
82
+ },
86
83
  "headers": {
87
- "content-type": "text/html"
84
+ "x-status-code": "200"
88
85
  }
89
86
  }
90
87
 
91
- Prefer structuredContent. Check isError, error, and status_code before using
92
- data. Non-JSON page bodies are preserved exactly.
88
+ Prefer structuredContent. Check isError, error, status_code, and
89
+ headers["x-status-code"] (the target page's status) before using data.
93
90
 
94
- ## Step 3 — Choose Page-Loading Options
95
-
96
- browser_rendering and super_mode are independent axes. All four combinations
97
- can return different content or failures for the same URL:
98
-
99
- | browser_rendering | super_mode | Loading path |
100
- | --- | --- | --- |
101
- | false | false | Standard routing with the non-browser loader. |
102
- | true | false | Standard routing with browser loading and JavaScript. |
103
- | false | true | Real-device routing with the non-browser loader. |
104
- | true | true | Real-device routing with browser loading and JavaScript. |
105
-
106
- Use browser rendering when the page depends on JavaScript:
91
+ A screenshot is a base64 JPEG that can be large. Request it only when the user
92
+ needs an image, and save it to a file rather than reading it into the
93
+ conversation:
107
94
 
108
95
  {
109
- "url": "https://www.scrapethissite.com/pages/ajax-javascript/#2015",
110
- "browser_rendering": true
96
+ "url": "https://www.scrapethissite.com/pages/simple/",
97
+ "html": false,
98
+ "screenshot": true
111
99
  }
112
100
 
113
- Use Super Mode with the non-browser loader when routing may be the problem but
114
- browser loading is unnecessary or returns a worse response:
101
+ ## Step 3 — Choose Page-Loading Options
115
102
 
116
- {
117
- "url": "https://www.scrapethissite.com/pages/simple/",
118
- "super_mode": true
119
- }
103
+ Omit browser_rendering to let the service choose: it falls back to a browser
104
+ automatically when needed. true forces a browser and false disables it.
105
+ screenshot=true always uses a browser.
120
106
 
121
- Use both controls for real-device browser loading:
107
+ Use real-device routing when routing may be the problem:
122
108
 
123
109
  {
124
- "url": "https://www.scrapethissite.com/pages/ajax-javascript/#2015",
125
- "browser_rendering": true,
126
- "super_mode": true
110
+ "url": "https://www.scrapethissite.com/pages/simple/",
111
+ "super": true
127
112
  }
128
113
 
129
- Browser rendering is not a strictly stronger mode. Some sites fail or return
130
- worse content with browser_rendering=true but load successfully when it is
131
- false. Super Mode does not enable browser rendering.
132
-
133
114
  Wait for delayed content with a CSS selector:
134
115
 
135
116
  {
@@ -142,21 +123,17 @@ Use geographic routing or homepage navigation when the target requires it:
142
123
 
143
124
  {
144
125
  "url": "https://www.scrapethissite.com/pages/simple/",
145
- "browser_rendering": true,
146
126
  "geo_code": "ID",
147
127
  "home_page": true
148
128
  }
149
129
 
150
- Use geographic routing for geo-specific content.
151
-
152
- Bound resource use for a browser-rendered page:
130
+ Bound resource use:
153
131
 
154
132
  {
155
133
  "url": "https://www.scrapethissite.com/pages/ajax-javascript/#2015",
156
- "browser_rendering": true,
157
134
  "block_resources": true,
158
- "max_retries": 3,
159
- "token_cap": 10000,
135
+ "max_retries": 1,
136
+ "token_cap": 10,
160
137
  "timeout": 60
161
138
  }
162
139
 
@@ -164,50 +141,55 @@ Bound resource use for a browser-rendered page:
164
141
 
165
142
  | MCP input | Default | Request mapping | Use |
166
143
  | --- | --- | --- | --- |
167
- | url | required | Query url | Absolute HTTP or HTTPS target URL. |
168
- | browser_rendering | false | Query browserRendering | Execute page JavaScript. |
169
- | super_mode | false | Query super | Select real-device routing independently of browser rendering. |
170
- | geo_code | omitted | Query geoCode | Route through an ISO 3166-1 alpha-2 country. |
171
- | wait_for_selector | omitted | Query waitForSelector | Wait for a CSS selector; requires browser_rendering=true. |
172
- | home_page | false | Query homePage | Visit the site root before the target page. |
173
- | block_resources | false | Query blockResources | Block nonessential resources during loading. |
174
- | max_retries | 3 | Query maxRetries | Maximum retries after failure; zero disables retries. |
175
- | token_cap | omitted | Query tokenCap | Retry token budget; the initial request still runs. |
176
- | timeout | 30 | Query timeout | Page-load timeout in seconds; transport receives another 30 seconds. |
144
+ | url | required | Body url | Absolute HTTP or HTTPS target URL. |
145
+ | html | true | Query html | Request HTML content. |
146
+ | markdown | false | Query markdown | Request Markdown content. |
147
+ | screenshot | false | Query screenshot | Capture a base64 JPEG; forces browser rendering. |
148
+ | browser_rendering | automatic | Query browserRendering | true forces and false disables browser rendering. |
149
+ | super | false | Query super | Real-device routing, independent of browser rendering. |
150
+ | geo_code | US | Query geoCode | Proxy country, such as US, ID, or SG. |
151
+ | proxy_country | omitted | Query proxyCountry | Compatibility country option. |
152
+ | use_proxy | true | Query useProxy | Set false to skip the service's configured proxy. |
153
+ | proxy | omitted | Query proxy | Custom proxy as username:password@host:port. |
154
+ | wait_for_selector | omitted | Query waitForSelector | CSS selector to wait for. |
155
+ | home_page | false | Query warmStart | Visit the site root before the target page. |
156
+ | block_resources | false | Query blockResources | Block images, CSS, and fonts. |
157
+ | bypass_proxy | false | Query bypassProxy | Enable the proxy-bypass behavior. |
158
+ | cookie_jar | omitted | Query cookieJar | Cookie configuration string. |
159
+ | headers | omitted | Query headers | Target page request headers as an object or JSON string. |
160
+ | max_retries | 3 | Body maxRetries | Maximum retries; zero disables retries. |
161
+ | token_cap | omitted | Body tokenCap | Running token total that stops further retries. |
162
+ | timeout | 120 | Query timeout | Page timeout from 1 to 300 seconds. |
163
+ | stream | omitted | Query stream | Ask the API to stream; the response is still returned once. |
164
+ | discard_token_secret | empty | Header x-discard-token-secret | Optional upstream header. |
177
165
 
178
166
  Authentication is handled by the MCP client's OAuth 2.1 connection and is
179
167
  never a tool input.
180
168
 
181
169
  ## Step 4 — Inspect and Retry Deliberately
182
170
 
183
- Start with both controls false unless the task already establishes a
184
- requirement. If the response fails, is blocked, incomplete, or missing dynamic
185
- content:
171
+ Start with the URL alone unless the task already establishes a requirement. If
172
+ the response fails, is blocked, incomplete, or missing dynamic content:
186
173
 
187
- 1. Inspect the initial response.
188
- 2. Change one axis at a time: browser_rendering for JavaScript, or super_mode
174
+ 1. Inspect status_code, data.error, headers["x-status-code"], and the content.
175
+ 2. Change one axis at a time: force or disable browser_rendering, or set super
189
176
  when routing may be the problem.
190
- 3. If browser loading fails or returns worse content, retry the same super_mode
191
- value with browser_rendering=false.
192
- 4. Try the remaining untested combinations when the response is still unusable.
193
- 5. Add wait_for_selector, geo_code, or home_page only when evidence shows that
177
+ 3. Add wait_for_selector, geo_code, or home_page only when evidence shows that
194
178
  the target requires it.
195
- 6. Stop after a usable response unless the user requests a comparison.
179
+ 4. Stop after a usable response unless the user requests a comparison.
196
180
 
197
- Do not repeat an identical combination. Treat wait_for_selector as a CSS
198
- selector, not a duration. Browser rendering loads a page; it does not click
199
- controls, submit forms, or provide an authenticated interactive browser
200
- session.
181
+ Do not repeat an identical request. Treat wait_for_selector as a CSS selector,
182
+ not a duration. Browser rendering loads a page; it does not click controls,
183
+ submit forms, or provide an authenticated interactive browser session.
201
184
 
202
185
  ## Step 5 — Deliver the Result
203
186
 
204
187
  Answer the user's request from data. Keep the full envelope when headers or
205
188
  diagnostics matter. When the user requests an archive, save the successful
206
- response or its data value with the environment's local file-writing
207
- capability; fetch itself has no output-path input.
208
-
209
- Keep fetched content as the source of truth for later steps. Build summaries,
210
- tables, JSON transformations, and extraction scripts from that raw content. If
211
- fetch fails and another MrScraper workflow can still complete the task, disclose
212
- that the raw response was not preserved and do not present the narrower result
213
- as exhaustive source content.
189
+ response or the requested field with the environment's local file-writing
190
+ capability; scraperAPI itself has no output-path input.
191
+
192
+ Keep retrieved content as the source of truth for later steps. Build
193
+ summaries, tables, JSON transformations, and extraction scripts from it, and
194
+ retrieve the page again when a follow-up needs details that an earlier
195
+ transformation did not preserve.
@@ -1,28 +1,31 @@
1
1
  ---
2
2
  name: mrscraper-serp
3
3
  description: |
4
- Discover public pages through Google with MrScraper using a search query or Google search URL. Use when no target URL is known or the user asks to inspect Google results. After discovery, preserve selected pages with mrscraper-fetch before adding managed extraction through mrscraper-scrape.
4
+ Discover public pages through Google with MrScraper's webSearch tool using a search query or Google search URL. Use when no target URL is known or the user asks to inspect Google results. After discovery, retrieve selected pages with mrscraper-fetch.
5
5
  ---
6
6
 
7
- # Discover Pages with MrScraper MCP SERP
7
+ # Discover Pages with MrScraper MCP Web Search
8
8
 
9
- Use the serp tool supplied by the MrScraper MCP server when a task begins with
10
- a search query rather than a known target URL. Use
11
- [mrscraper](../mrscraper/SKILL.md) for connection troubleshooting, saved runs,
12
- account status, or broader routing.
9
+ Use the webSearch tool supplied by the MrScraper MCP server when a task begins
10
+ with a search query rather than a known target URL. Use
11
+ [mrscraper](../mrscraper/SKILL.md) for connection troubleshooting, account
12
+ status, or broader routing.
13
13
 
14
14
  ## Step 1 — Build the Query
15
15
 
16
- Include the important entity, attribute, location, or date in query_or_url:
16
+ Include the important entity, attribute, location, or date in url, and set
17
+ raw=false to receive parsed results:
17
18
 
18
19
  {
19
- "query_or_url": "web scraping best practices"
20
+ "url": "web scraping best practices",
21
+ "raw": false
20
22
  }
21
23
 
22
24
  The input can also be a complete Google search URL:
23
25
 
24
26
  {
25
- "query_or_url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20"
27
+ "url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
28
+ "raw": false
26
29
  }
27
30
 
28
31
  For a Google URL, the tool reads q, gl, hl, and start. Explicit region,
@@ -33,7 +36,8 @@ language, and page inputs take precedence.
33
36
  Set both country and language when results must match a locale:
34
37
 
35
38
  {
36
- "query_or_url": "running shoes",
39
+ "url": "running shoes",
40
+ "raw": false,
37
41
  "region": "id",
38
42
  "language": "id",
39
43
  "page": 2
@@ -44,25 +48,21 @@ the current page before requesting another.
44
48
 
45
49
  ## Step 3 — Choose the Output
46
50
 
47
- Parsed JSON is the default and is suitable for selecting titles, URLs, and
48
- snippets:
49
-
50
- {
51
- "query_or_url": "iphone 17",
52
- "format": "json"
53
- }
51
+ raw defaults to true, which returns result-page HTML as the Python SDK does.
52
+ Set raw=false for parsed JSON; organic results are in data.data.organic, next
53
+ to sections such as people_also_ask and pagination.
54
54
 
55
55
  Request result-page HTML when the page itself is needed:
56
56
 
57
57
  {
58
- "query_or_url": "iphone 17",
59
- "format": "html"
58
+ "url": "iphone 17"
60
59
  }
61
60
 
62
61
  Use JavaScript rendering for dynamic result features such as AI Overview:
63
62
 
64
63
  {
65
- "query_or_url": "what is web scraping",
64
+ "url": "what is web scraping",
65
+ "raw": false,
66
66
  "render_js": true
67
67
  }
68
68
 
@@ -70,14 +70,14 @@ Use JavaScript rendering for dynamic result features such as AI Overview:
70
70
 
71
71
  | MCP input | Default | Request mapping | Use |
72
72
  | --- | --- | --- | --- |
73
- | query_or_url | required | Body query | Search query or complete Google search URL. |
73
+ | url | required | Body query | Search query or complete Google search URL. |
74
+ | raw | true | Body format=html | Return result-page HTML; set false to use format. |
75
+ | format | json | Body format | json or html when raw=false. |
74
76
  | region | URL value or omitted | Body region | Google result country, such as us or id. |
75
77
  | language | URL value or omitted | Body language | Google result language, such as en or id. |
76
78
  | page | URL value or omitted | Body page | Positive, one-based results page. |
77
- | format | json | Body format | Return parsed JSON or result-page HTML. |
78
- | render_js | false | Body renderJs | Wait for JavaScript-rendered result features. |
79
- | raw | false | Body format=html | Deprecated compatibility alias; prefer format=html. |
80
- | client_timeout | 120 | MCP upstream deadline | Maximum wait in seconds; not sent in the SERP body. |
79
+ | render_js | false | Body renderJs | Render JavaScript-dependent result features. |
80
+ | timeout | 600 | MCP upstream deadline | Maximum wait in seconds; not sent in the search body. |
81
81
 
82
82
  Authentication is handled by the MCP client's OAuth 2.1 connection and is
83
83
  never a tool input.
@@ -88,25 +88,19 @@ The complete response envelope is available in MCP structuredContent and as
88
88
  formatted JSON text. Prefer structuredContent. Check isError, error, and
89
89
  status_code before using data.
90
90
 
91
- For format=json, inspect data for the available titles, URLs, snippets, and
92
- other result fields. For format=html, data contains result-page HTML.
93
-
94
91
  When the user requests a search artifact, save the successful data or full
95
- envelope with local file-writing capability; serp has no output-path input.
92
+ envelope with local file-writing capability; webSearch has no output-path
93
+ input.
96
94
 
97
95
  ## Step 5 — Continue with Relevant URLs
98
96
 
99
- Select only URLs relevant to the user's goal, then:
100
-
101
- - Load [mrscraper-fetch](../mrscraper-fetch/SKILL.md) to preserve and inspect
102
- each selected page before deriving answers, comparisons, or structured output.
103
- - Add [mrscraper-scrape](../mrscraper-scrape/SKILL.md) only when the user
104
- explicitly requests managed extraction or it still offers a clear benefit
105
- after the page structure and output schema are understood.
97
+ Select only URLs relevant to the user's goal, then load
98
+ [mrscraper-fetch](../mrscraper-fetch/SKILL.md) to retrieve and inspect each
99
+ selected page before deriving answers, comparisons, or structured output.
106
100
 
107
101
  Run independent follow-up URLs in parallel when the environment supports safe
108
102
  parallel execution. Keep the number of pages proportional to the requested
109
- coverage.
103
+ coverage. If the user only requested URL discovery, stop after the search.
110
104
 
111
105
  ## Step 6 — Handle Weak or Missing Results
112
106
 
@@ -114,10 +108,9 @@ coverage.
114
108
  - Rewrite noisy queries around the important entity, attribute, and location.
115
109
  - Request another result page only after checking the current page.
116
110
  - Retry once with render_js=true when a dynamic result feature is required.
117
- - Increase client_timeout when the MCP request deadline is too short.
111
+ - Increase timeout when the MCP request deadline is too short.
118
112
  - Use [mrscraper](../mrscraper/SKILL.md) when the connection or authentication
119
113
  fails.
120
114
 
121
- SERP is for public Google discovery. Use the map agent in
122
- [mrscraper-scrape](../mrscraper-scrape/SKILL.md) to discover URLs within one
115
+ webSearch is for public Google discovery, not for discovering URLs within one
123
116
  known site.
@@ -1,206 +0,0 @@
1
- ---
2
- name: mrscraper-scrape
3
- description: |
4
- Run MrScraper's general, listing, or map agents for managed structured extraction or bounded URL discovery within a known site. Use when managed output is explicitly requested or justified after source-page inspection; use mrscraper-fetch for initial acquisition of known pages and mrscraper-serp when no starting URL is known.
5
- ---
6
-
7
- # Extract Structured Data with MrScraper MCP
8
-
9
- Do not use scrape for the first exploration of a known page. Start with
10
- [mrscraper-fetch](../mrscraper-fetch/SKILL.md), inspect the complete raw
11
- response, and prefer local analysis or a reusable extractor. Use scrape when
12
- the user explicitly wants MrScraper-managed extraction, or after the agent
13
- understands the website and has defined a stable output schema. Use
14
- [mrscraper](../mrscraper/SKILL.md) for connection troubleshooting, saved runs,
15
- account status, or broader routing.
16
-
17
- For general and listing, MrScraper retrieves page HTML and asks a backend LLM to
18
- interpret it according to prompt. This adds model-processing time, can narrow
19
- the result to the requested fields, and repeats model work across many pages.
20
-
21
- Before using general or listing, confirm all of the following:
22
-
23
- 1. The target or representative pages have already been fetched and inspected.
24
- 2. The site structure and required output fields are understood.
25
- 3. A stable output schema has been defined.
26
- 4. Managed extraction still offers a concrete benefit over local code, or the
27
- user explicitly requested it.
28
-
29
- If these conditions are not met, return to fetch. Requiring JSON, a table, or
30
- named fields is not by itself a reason to call scrape. For large same-layout
31
- sets—even roughly 100 pages—safe concurrent fetches plus one reusable local
32
- extractor are often faster and preserve every raw response. The map agent is
33
- separate URL discovery functionality and does not require an extraction schema.
34
-
35
- ## Step 1 — Choose an Agent
36
-
37
- | Agent | Use it for | Required input | Available controls |
38
- | --- | --- | --- | --- |
39
- | general | One detail page or one extraction task | prompt | proxy_country, schema_prompt |
40
- | listing | Repeated records or paginated listings | prompt | proxy_country, max_pages, schema_prompt |
41
- | map | Discovering URLs across a site | URL only | max_depth, max_pages, limit, include_patterns, exclude_patterns |
42
-
43
- The default agent is general. For map, omit prompt, schema_prompt, and
44
- proxy_country. For general and listing, omit map-only controls. max_pages is
45
- accepted by listing and map but not general.
46
-
47
- agent selects the extraction workflow. mode independently selects the backend
48
- execution tier: Cheap or Super. Omit mode to preserve the backend default, and
49
- select Super only when the extraction requires the stronger mode.
50
-
51
- ## Step 2 — Define the Extraction
52
-
53
- The examples below assume the decision gate above has been satisfied. Write a
54
- prompt from the already-understood page structure and output schema, preserve
55
- source values, and do not ask the model to infer unavailable values.
56
-
57
- Detail-page example:
58
-
59
- {
60
- "url": "https://www.scrapethissite.com/pages/simple/",
61
- "agent": "general",
62
- "mode": "Super",
63
- "prompt": "Extract each country's name, capital, population, and area. Preserve source values and omit unavailable fields."
64
- }
65
-
66
- Repeated-listing example:
67
-
68
- {
69
- "url": "https://www.scrapethissite.com/pages/forms/?page_num=1",
70
- "agent": "listing",
71
- "prompt": "Extract each hockey team's name, year, wins, losses, and win percentage.",
72
- "max_pages": 5
73
- }
74
-
75
- Site-map example:
76
-
77
- {
78
- "url": "https://www.scrapethissite.com/",
79
- "agent": "map",
80
- "max_depth": 2,
81
- "max_pages": 50,
82
- "limit": 1000,
83
- "include_patterns": "/pages/"
84
- }
85
-
86
- ## Step 3 — Add Shape Guidance When Useful
87
-
88
- schema_prompt accepts a JSON object, not a file path. If the user supplies a
89
- local JSON Schema file, read and parse it with local file tools, confirm its
90
- root is an object, and pass that object:
91
-
92
- {
93
- "url": "https://www.scrapethissite.com/pages/simple/",
94
- "agent": "general",
95
- "prompt": "Extract every country's name, capital, population, and area.",
96
- "schema_prompt": {
97
- "type": "object",
98
- "properties": {
99
- "countries": {
100
- "type": "array",
101
- "items": {
102
- "type": "object",
103
- "properties": {
104
- "name": { "type": "string" },
105
- "capital": { "type": "string" },
106
- "population": { "type": "string" },
107
- "area": { "type": "string" }
108
- }
109
- }
110
- }
111
- }
112
- }
113
- }
114
-
115
- The MCP server appends this object to the extraction prompt as best-effort
116
- shape guidance. It does not validate the returned data against the schema.
117
- Validate locally when strict compliance is required.
118
-
119
- ## Step 4 — Set Parameters
120
-
121
- | MCP input | Default | Request mapping | Use |
122
- | --- | --- | --- | --- |
123
- | url | required | Body url | Absolute HTTP or HTTPS starting URL. |
124
- | agent | general | Body agent | Select general, listing, or map. |
125
- | mode | service default | Body mode | Select Cheap or Super execution without changing the agent. |
126
- | prompt | required for general/listing | Body message | Describe the fields or records to extract. |
127
- | proxy_country | omitted | Body proxyCountry | Route general/listing through a country. |
128
- | max_pages | service default | Body maxPages | Bound listing or map pages. |
129
- | max_depth | service default | Body maxDepth | Bound map crawl depth. |
130
- | limit | service default | Body limit | Bound map URL results. |
131
- | include_patterns | omitted | Body includePatterns | Restrict map results to matching URLs. |
132
- | exclude_patterns | omitted | Body excludePatterns | Exclude matching map URLs. |
133
- | schema_prompt | omitted | Appended to message | JSON Schema object for general/listing guidance. |
134
-
135
- Authentication is handled by the MCP client's OAuth 2.1 connection. The scrape
136
- tool does not accept a credential or output-path input. Choose the smallest
137
- practical limits and omit optional values when the service default is
138
- appropriate.
139
-
140
- ## Step 5 — Wait for Completion
141
-
142
- General and map usually finish quickly. Listing is synchronous and can take
143
- several minutes even for one page.
144
-
145
- Before starting a listing call, tell the user it may take several minutes. Keep
146
- waiting on the original MCP call within the environment's supported tool
147
- timeout. Do not submit a duplicate scrape because the first call is quiet or
148
- still running.
149
-
150
- ## Step 6 — Use the Output
151
-
152
- The complete API envelope is returned in MCP structuredContent and as formatted
153
- JSON text. Prefer structuredContent and check isError, error, and status_code
154
- before trusting the response.
155
-
156
- A successful run normally stores its extracted value at:
157
-
158
- structuredContent.data.data.data
159
-
160
- The saved scraper UUID is normally at:
161
-
162
- structuredContent.data.data.scraperId
163
-
164
- When the user asks for a file, save the extracted value only after the tool
165
- succeeds. Use local file-writing capability and report the path. Preserve the
166
- full envelope separately when the scraper ID, response headers, or diagnostics
167
- also matter.
168
-
169
- Post-process only when the user requests filtering, merging, normalization,
170
- CSV, a table, or another deliverable. Extracted fields named token or password
171
- may be legitimate source data and are intentionally preserved.
172
-
173
- ## Step 7 — Preserve the Reproducible Scraper
174
-
175
- Every successful scrape creates a saved AI scraper configuration by default.
176
- Report scraperId when it is available and the extraction may need to run again.
177
-
178
- Use the MrScraper rerun tool to apply the saved configuration to the same or
179
- another URL:
180
-
181
- {
182
- "target": "https://www.scrapethissite.com/pages/forms/?page_num=2",
183
- "type": "ai",
184
- "scraper_id": "SCRAPER_UUID"
185
- }
186
-
187
- Prefer rerun over recreating the scrape definition when the user wants a
188
- repeatable extraction. Explain that this reproduces the saved configuration,
189
- not necessarily identical values when the page or model behavior changes.
190
-
191
- The rerun tool also handles dashboard-built manual workflows and asynchronous
192
- bulk jobs. Follow the full rerun workflow in
193
- [mrscraper](../mrscraper/SKILL.md), including the required acknowledgment
194
- before any manual rerun.
195
-
196
- ## Step 8 — Handle Failures
197
-
198
- - Check isError, error, and status_code before using or saving output.
199
- - Tighten the prompt when fields are missing or grouped incorrectly.
200
- - Confirm the selected agent accepts every supplied input.
201
- - Validate locally when downstream code requires a strict schema.
202
- - Use [mrscraper-fetch](../mrscraper-fetch/SKILL.md) for page reading.
203
- - Use [mrscraper-serp](../mrscraper-serp/SKILL.md) when discovery must happen
204
- first.
205
- - Never invent replacement fields or fill missing values with unsupported
206
- assumptions.