@hasdata/web-scraping-mcp 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +372 -0
- package/index.mjs +24 -0
- package/package.json +40 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 HasData
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,372 @@
|
|
|
1
|
+
# Web Scraping MCP Server
|
|
2
|
+
|
|
3
|
+
<!-- mcp-name: com.hasdata/web-scraping -->
|
|
4
|
+
|
|
5
|
+
A hosted Model Context Protocol (MCP) server that gives Claude, Cursor, Windsurf and any other MCP client one read-only tool for fetching any public web page. It goes out through managed proxies, renders JavaScript when a page needs it, and returns clean markdown, plain text, raw HTML or structured JSON, with nothing to host and no browser in your stack.
|
|
6
|
+
|
|
7
|
+
This is the fallback for sites with no dedicated API. When a site does have one in the [HasData catalogue](https://docs.hasdata.com/mcp-server?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp), that tool returns parsed fields and this one returns a page.
|
|
8
|
+
|
|
9
|
+
**1,000 free credits every month, no card required.** A plain fetch costs 1 credit, so the free tier covers 1,000 of them.
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
https://mcp.hasdata.com/api/mcp?apis=web_scraping
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
[](https://glama.ai/mcp/servers/HasData/web-scraping-mcp)
|
|
16
|
+
[](https://github.com/HasData/web-scraping-mcp/actions/workflows/contract.yml)
|
|
17
|
+
[](https://mcp.hasdata.com/api/mcp?apis=web_scraping)
|
|
18
|
+
[](#tools)
|
|
19
|
+
[](https://www.npmjs.com/package/@hasdata/web-scraping-mcp)
|
|
20
|
+
[](https://pypi.org/project/hasdata-web-scraping-mcp/)
|
|
21
|
+
[](LICENSE)
|
|
22
|
+
|
|
23
|
+
## Contents
|
|
24
|
+
|
|
25
|
+
- [What you need](#what-you-need)
|
|
26
|
+
- [Quick start](#quick-start)
|
|
27
|
+
- [Example prompts](#example-prompts)
|
|
28
|
+
- [Tools](#tools)
|
|
29
|
+
- [Output formats](#output-formats)
|
|
30
|
+
- [Errors and failure paths](#errors-and-failure-paths)
|
|
31
|
+
- [Pricing, free tier and limits](#pricing-free-tier-and-limits)
|
|
32
|
+
- [How it compares](#how-it-compares)
|
|
33
|
+
- [FAQ](#faq)
|
|
34
|
+
- [HasData links](#hasdata-links)
|
|
35
|
+
- [Development](#development)
|
|
36
|
+
- [Contributing](#contributing)
|
|
37
|
+
- [License](#license)
|
|
38
|
+
|
|
39
|
+
## What you need
|
|
40
|
+
|
|
41
|
+
An MCP client and a HasData API key from the [dashboard](https://app.hasdata.com/sign-up?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp), free to create with no card. This is a remote server, so the simplest path is a URL and an `x-api-key` header, with no container to run. A client that only speaks stdio reaches it through a thin launcher, published as `@hasdata/web-scraping-mcp` on npm and `hasdata-web-scraping-mcp` on PyPI, shown below.
|
|
42
|
+
|
|
43
|
+
## Quick start
|
|
44
|
+
|
|
45
|
+
The server URL is the same for every client. We run it hands-on in Claude Code and Claude Desktop. The other blocks follow each client's own documented format for a remote server.
|
|
46
|
+
|
|
47
|
+
| Field | Value |
|
|
48
|
+
| :--- | :--- |
|
|
49
|
+
| URL | `https://mcp.hasdata.com/api/mcp?apis=web_scraping` |
|
|
50
|
+
| Transport | HTTP, streamable |
|
|
51
|
+
| Auth header | `x-api-key: HASDATA_API_KEY` |
|
|
52
|
+
|
|
53
|
+
Clients with OAuth support can add the same URL as a connector and sign in without putting a key in a config file.
|
|
54
|
+
|
|
55
|
+
<details>
|
|
56
|
+
<summary><b>Claude Code</b></summary>
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
claude mcp add --transport http web-scraping "https://mcp.hasdata.com/api/mcp?apis=web_scraping" \
|
|
60
|
+
--header "x-api-key: HASDATA_API_KEY"
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
</details>
|
|
64
|
+
|
|
65
|
+
<details>
|
|
66
|
+
<summary><b>Claude Desktop</b></summary>
|
|
67
|
+
|
|
68
|
+
Settings, then Connectors, then Add custom connector, then paste `https://mcp.hasdata.com/api/mcp?apis=web_scraping` and sign in.
|
|
69
|
+
|
|
70
|
+
For the config-file route, Claude Desktop loads only local (stdio) servers, so it reaches a remote server through a stdio launcher. The `@hasdata/web-scraping-mcp` package is that launcher, and it reads the key from the environment. Add this to `claude_desktop_config.json`:
|
|
71
|
+
|
|
72
|
+
```json
|
|
73
|
+
{
|
|
74
|
+
"mcpServers": {
|
|
75
|
+
"web-scraping": {
|
|
76
|
+
"command": "npx",
|
|
77
|
+
"args": ["-y", "@hasdata/web-scraping-mcp"],
|
|
78
|
+
"env": { "HASDATA_API_KEY": "YOUR_KEY" }
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
For Python instead of Node, swap the launcher for the PyPI package, which `uvx` runs without a manual install:
|
|
85
|
+
|
|
86
|
+
```json
|
|
87
|
+
{
|
|
88
|
+
"mcpServers": {
|
|
89
|
+
"web-scraping": {
|
|
90
|
+
"command": "uvx",
|
|
91
|
+
"args": ["hasdata-web-scraping-mcp"],
|
|
92
|
+
"env": { "HASDATA_API_KEY": "YOUR_KEY" }
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
</details>
|
|
99
|
+
|
|
100
|
+
<details>
|
|
101
|
+
<summary><b>Cursor</b></summary>
|
|
102
|
+
|
|
103
|
+
`~/.cursor/mcp.json` for every project, or `.cursor/mcp.json` for one:
|
|
104
|
+
|
|
105
|
+
```json
|
|
106
|
+
{
|
|
107
|
+
"mcpServers": {
|
|
108
|
+
"web-scraping": {
|
|
109
|
+
"url": "https://mcp.hasdata.com/api/mcp?apis=web_scraping",
|
|
110
|
+
"headers": { "x-api-key": "HASDATA_API_KEY" }
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
</details>
|
|
117
|
+
|
|
118
|
+
<details>
|
|
119
|
+
<summary><b>Windsurf</b></summary>
|
|
120
|
+
|
|
121
|
+
`~/.codeium/windsurf/mcp_config.json`. Windsurf calls the field `serverUrl`, not `url`:
|
|
122
|
+
|
|
123
|
+
```json
|
|
124
|
+
{
|
|
125
|
+
"mcpServers": {
|
|
126
|
+
"web-scraping": {
|
|
127
|
+
"serverUrl": "https://mcp.hasdata.com/api/mcp?apis=web_scraping",
|
|
128
|
+
"headers": { "x-api-key": "HASDATA_API_KEY" }
|
|
129
|
+
}
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
</details>
|
|
135
|
+
|
|
136
|
+
<details>
|
|
137
|
+
<summary><b>VS Code</b></summary>
|
|
138
|
+
|
|
139
|
+
`.vscode/mcp.json` in the workspace:
|
|
140
|
+
|
|
141
|
+
```json
|
|
142
|
+
{
|
|
143
|
+
"servers": {
|
|
144
|
+
"web-scraping": {
|
|
145
|
+
"type": "http",
|
|
146
|
+
"url": "https://mcp.hasdata.com/api/mcp?apis=web_scraping",
|
|
147
|
+
"headers": { "x-api-key": "HASDATA_API_KEY" }
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
</details>
|
|
154
|
+
|
|
155
|
+
## Example prompts
|
|
156
|
+
|
|
157
|
+
- Fetch this page as markdown and summarise it.
|
|
158
|
+
- Read this documentation page and pull out every code block.
|
|
159
|
+
- Get the titles and links from the front page of this site as JSON.
|
|
160
|
+
- This page loads its content with JavaScript, so render it and wait for the results list before reading.
|
|
161
|
+
- Fetch this page through a German residential proxy and tell me whether the pricing differs.
|
|
162
|
+
- Take a screenshot of this page.
|
|
163
|
+
|
|
164
|
+
One call answers each of these. What changes between them is how much of the browser you asked for, and that is what the call costs.
|
|
165
|
+
|
|
166
|
+
## Tools
|
|
167
|
+
|
|
168
|
+
One tool. The cost depends on what you turn on, and the table is in [Pricing](#pricing-free-tier-and-limits).
|
|
169
|
+
|
|
170
|
+
### Scrape web page
|
|
171
|
+
|
|
172
|
+
[`hasdata_web_scraping_web_scraping_scrapeWebPage`](https://docs.hasdata.com/apis/web-scraping-api/api-params?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
173
|
+
|
|
174
|
+
Fetch one URL.
|
|
175
|
+
|
|
176
|
+
| Parameter | Type | Required | Notes |
|
|
177
|
+
| :--- | :--- | :--- | :--- |
|
|
178
|
+
| `url` | string | yes | The page to fetch |
|
|
179
|
+
| `outputFormat` | array | | Any of `markdown`, `text`, `html`, `json`. See [Output formats](#output-formats) |
|
|
180
|
+
| `jsRendering` | boolean | | Render the page in a browser. On by default, and the main cost lever |
|
|
181
|
+
| `proxyType` | string | | `datacenter` or `residential` |
|
|
182
|
+
| `proxyCountry` | string | | `US`, `UK`, `DE`, `IE`, `FR`, `IT`, `SE`, `BR`, `CA`, `JP`, `SG`, `IN` or `ID` |
|
|
183
|
+
| `headers` | object | | Custom request headers |
|
|
184
|
+
| `wait` | number | | Milliseconds to wait after load |
|
|
185
|
+
| `waitFor` | string | | CSS selector to wait for before reading |
|
|
186
|
+
| `jsScenario` | array | | Actions to run on the page. See below |
|
|
187
|
+
| `extractRules` | object | | CSS selectors to pull named fields |
|
|
188
|
+
| `aiExtractRules` | object | | A typed schema an LLM fills from the page |
|
|
189
|
+
| `extractLinks` | boolean | | Collect the page's links |
|
|
190
|
+
| `extractEmails` | boolean | | Collect email addresses on the page |
|
|
191
|
+
| `screenshot` | boolean | | Capture the rendered page |
|
|
192
|
+
| `blockResources` | boolean | | Skip images and stylesheets |
|
|
193
|
+
| `blockAds` | boolean | | Skip ad requests |
|
|
194
|
+
| `blockUrls` | array | | Skip these URLs |
|
|
195
|
+
| `includeOnlyTags` | array | | Keep only elements matching these selectors |
|
|
196
|
+
| `excludeTags` | array | | Drop elements matching these selectors |
|
|
197
|
+
| `removeBase64Images` | boolean | | Strip inline base64 images from the output |
|
|
198
|
+
|
|
199
|
+
`extractRules` maps a field name to a CSS selector, with `@attr` to read an attribute rather than text.
|
|
200
|
+
|
|
201
|
+
```json
|
|
202
|
+
{ "title": "h1", "link_href": "a#link @href", "page_text": "body" }
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
`jsScenario` is an array of actions run in order, covering `click`, `wait`, `waitFor`, `waitForAndClick`, `scrollX`, `scrollY`, `fill` and `evaluate` for arbitrary JavaScript. It needs `jsRendering` on.
|
|
206
|
+
|
|
207
|
+
`aiExtractRules` describes the shape you want and lets a model fill it from the HTML. Each key is an output field, typed as `string`, `number`, `boolean`, `list` or `item` for a nested object.
|
|
208
|
+
|
|
209
|
+
## Output formats
|
|
210
|
+
|
|
211
|
+
This is the part worth reading before your first call, because the response shape moves with `outputFormat`.
|
|
212
|
+
|
|
213
|
+
Ask for exactly one of `markdown`, `text` or `html`, and the content arrives as a plain string in `text` at the top level.
|
|
214
|
+
|
|
215
|
+
```json
|
|
216
|
+
{
|
|
217
|
+
"url": "https://api.hasdata.com/scrape/web/",
|
|
218
|
+
"status": 200,
|
|
219
|
+
"json": null,
|
|
220
|
+
"text": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)\n"
|
|
221
|
+
}
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
Include `json`, alone or alongside another format, and everything moves inside `json`, the top-level `text` becomes null, and the requested formats become keys in there next to the page metadata.
|
|
225
|
+
|
|
226
|
+
```json
|
|
227
|
+
{
|
|
228
|
+
"json": {
|
|
229
|
+
"requestMetadata": { "id": "7764031a-43f7-4102-8561-a7b7a6f1cbf5", "status": "ok" },
|
|
230
|
+
"statusCode": 200,
|
|
231
|
+
"statusText": "OK",
|
|
232
|
+
"headers": { "server": "nginx", "content-type": "text/html; charset=utf-8" },
|
|
233
|
+
"extractedData": { "title": "Hacker News", "firstStory": ["iPhone Duo", "apple.com", "Show HN: What if the speed of light was 5 km/h?"] }
|
|
234
|
+
},
|
|
235
|
+
"text": null
|
|
236
|
+
}
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
`extractedData` holds the `extractRules` results. A selector matching several elements returns all of them as an array, so `.titleline a` on a listing page returns every match rather than the first.
|
|
240
|
+
|
|
241
|
+
## Errors and failure paths
|
|
242
|
+
|
|
243
|
+
Plan for these rather than assuming a happy path.
|
|
244
|
+
|
|
245
|
+
**`extractLinks` and `extractEmails` do nothing unless `outputFormat` includes `json`.** They put `links` and `emails` inside the `json` object, and there is nowhere for them to go in a plain markdown response. Asking for them with `outputFormat: ["text"]` returns the text and silently no links.
|
|
246
|
+
|
|
247
|
+
**A 404 or a 403 on the target page is a successful call.** The page's own status comes back as `statusCode` inside `json`, or as `status` at the top level, and the request is billed either way. Check the status before you parse the body.
|
|
248
|
+
|
|
249
|
+
**`jsRendering` is on by default, and it is what the call costs.** Turning it off takes a fetch from 10 credits to 1. Most static pages, documentation, articles and anything server-rendered do not need it. Turn it on when the content arrives empty without it.
|
|
250
|
+
|
|
251
|
+
**`waitFor` beats `wait`.** A fixed delay is a guess that is either too short on a slow load or wasted on a fast one. A CSS selector waits for the thing you actually need and returns as soon as it appears.
|
|
252
|
+
|
|
253
|
+
**A residential proxy is five to fifteen times the price of a datacenter one.** Reach for it when a datacenter fetch comes back blocked, rather than as the default.
|
|
254
|
+
|
|
255
|
+
**`includeOnlyTags` and `excludeTags` take `querySelectorAll` selectors.** An invalid selector narrows nothing rather than erroring, so a suspiciously complete response is the symptom of a typo.
|
|
256
|
+
|
|
257
|
+
**`aiExtractRules` runs a model over the HTML, so it is neither free nor deterministic.** Two calls on the same page can differ in wording. When a CSS selector can do the job, `extractRules` is cheaper and repeatable.
|
|
258
|
+
|
|
259
|
+
Results that carry data also carry a `requestMetadata.id` worth quoting in support.
|
|
260
|
+
|
|
261
|
+
## Pricing, free tier and limits
|
|
262
|
+
|
|
263
|
+
The cost depends on two switches, and nothing else changes it.
|
|
264
|
+
|
|
265
|
+
| | Datacenter proxy | Residential proxy |
|
|
266
|
+
| :--- | :--- | :--- |
|
|
267
|
+
| `jsRendering: false` | **1 credit** | 5 credits |
|
|
268
|
+
| `jsRendering: true` | 10 credits | 15 credits |
|
|
269
|
+
|
|
270
|
+
Rendering is on by default, so an unconfigured call costs 10. A static page fetched with `jsRendering: false` costs 1, which makes this the cheapest tool in the catalogue when you do not need a browser.
|
|
271
|
+
|
|
272
|
+
The free tier is **1,000 credits every month with no card**. That is 1,000 plain fetches, or 100 rendered ones. It renews with the billing cycle.
|
|
273
|
+
|
|
274
|
+
Paid plans start at **$49 a month** for 200,000 credits, which is 200,000 plain fetches or 20,000 rendered ones. The unit price falls with volume across the [high-volume plans](https://hasdata.com/prices?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp).
|
|
275
|
+
|
|
276
|
+
Your plan also sets concurrency. The free tier allows 1 request at a time, Startup 15, Business 30, Growth 50, and the high-volume plans run from 200 to 1,500. Retry on the 429 with a backoff in anything unattended, because an agent crawling a list of URLs will reach the ceiling before you do.
|
|
277
|
+
|
|
278
|
+
Credits come off successful requests only. A page that answers 404 is still a successful fetch of a 404.
|
|
279
|
+
|
|
280
|
+
## How it compares
|
|
281
|
+
|
|
282
|
+
The comparison worth making is against fetching the page yourself, and against the other tools in this catalogue.
|
|
283
|
+
|
|
284
|
+
| | `fetch` in your own code | A dedicated HasData tool | This server |
|
|
285
|
+
| :--- | :--- | :--- | :--- |
|
|
286
|
+
| Blocked by bot protection | Often | Handled | Handled |
|
|
287
|
+
| JavaScript-heavy pages | Needs a browser you run | Handled | A parameter |
|
|
288
|
+
| Geo-targeting | Your own proxies | Built in | A parameter |
|
|
289
|
+
| Output | Raw HTML | Parsed fields for that site | Markdown, text, HTML or JSON |
|
|
290
|
+
| Coverage | Anything | The sites with a tool | Anything public |
|
|
291
|
+
| Cost | Your infrastructure | Per call | From 1 credit a call |
|
|
292
|
+
|
|
293
|
+
The row that decides it against a dedicated tool is coverage. Amazon, Zillow, Yelp and the rest return typed fields because someone maintains a parser for that site. This one returns a page from any site and leaves the parsing to you, which is the right trade only when no dedicated tool exists.
|
|
294
|
+
|
|
295
|
+
Against your own `fetch`, the question is whether the target fights back. For a friendly page, your own code is free and this is not worth a credit.
|
|
296
|
+
|
|
297
|
+
## FAQ
|
|
298
|
+
|
|
299
|
+
### What is a web scraping MCP server?
|
|
300
|
+
|
|
301
|
+
An MCP server exposes tools an AI client can call. This one lets an agent fetch any public URL through managed proxies and get it back as markdown, text, HTML or structured JSON, without a browser or a proxy pool in your stack.
|
|
302
|
+
|
|
303
|
+
### Do I need my own proxies or a headless browser?
|
|
304
|
+
|
|
305
|
+
No. Both are on the server side. The only credential is your HasData key.
|
|
306
|
+
|
|
307
|
+
### How do I make calls cheaper?
|
|
308
|
+
|
|
309
|
+
Set `jsRendering: false`. That is the difference between 10 credits and 1. Add `blockResources` when you do need rendering, so the browser skips images and stylesheets.
|
|
310
|
+
|
|
311
|
+
### How do I know whether a page needs rendering?
|
|
312
|
+
|
|
313
|
+
Fetch it once without rendering, for 1 credit. If the content you want is there, you are done. If the body comes back as an empty shell, render it.
|
|
314
|
+
|
|
315
|
+
### Can it fill in a form or click through to the next page?
|
|
316
|
+
|
|
317
|
+
Yes, with `jsScenario`, which runs `click`, `fill`, `waitFor`, `scrollY` and `evaluate` steps in order on the rendered page.
|
|
318
|
+
|
|
319
|
+
### What is the difference between `extractRules` and `aiExtractRules`?
|
|
320
|
+
|
|
321
|
+
`extractRules` takes CSS selectors, and it is cheap, fast and repeatable. `aiExtractRules` describes the fields you want and lets a model read the page, which handles pages whose structure you cannot pin down but costs more and can vary between runs.
|
|
322
|
+
|
|
323
|
+
### Can I get a screenshot?
|
|
324
|
+
|
|
325
|
+
Yes, with `screenshot: true` on a rendered call.
|
|
326
|
+
|
|
327
|
+
### Can I use this together with other HasData APIs?
|
|
328
|
+
|
|
329
|
+
Yes. One key covers everything, and one endpoint serves them all through the `apis` parameter. Point a client at `?apis=web_scraping,google_serp` to get both tool sets in one connection, or at [`mcp.hasdata.com/api/mcp`](https://docs.hasdata.com/mcp-server?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp) for the full catalogue.
|
|
330
|
+
|
|
331
|
+
### Is HasData affiliated with the sites I fetch?
|
|
332
|
+
|
|
333
|
+
No. HasData is an independent service. This tool fetches pages you name, so what comes back is whatever that site publishes, and you are responsible for using it in line with that site's terms and the law that applies to you.
|
|
334
|
+
|
|
335
|
+
### Compliance and personal data
|
|
336
|
+
|
|
337
|
+
This tool points wherever you point it, which puts more on you than a site-specific one does. Two things deserve a decision before you build. `extractEmails` collects addresses, and an address is personal data in the GDPR sense and regulated separately again for marketing under the CAN-SPAM Act, the ePrivacy rules and their equivalents. And a page behind a login, a paywall or a robots exclusion is not made public by the fact that a proxy can reach it. Fetch what is genuinely public, keep only what your purpose needs, and check your own obligations.
|
|
338
|
+
|
|
339
|
+
## HasData links
|
|
340
|
+
|
|
341
|
+
- [Web Scraping API](https://hasdata.com/web-scraping-api?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp), the REST endpoint behind this tool
|
|
342
|
+
- [Parameter reference](https://docs.hasdata.com/apis/web-scraping-api/api-params?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
343
|
+
- [Request cost](https://docs.hasdata.com/apis/web-scraping-api/request-cost?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
344
|
+
- [Structured data extraction](https://docs.hasdata.com/apis/web-scraping-api/features/structured-data-extraction?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
345
|
+
- [MCP server documentation](https://docs.hasdata.com/mcp-server?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
346
|
+
- [Pricing](https://hasdata.com/prices?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
347
|
+
- [Dashboard](https://app.hasdata.com/sign-up?utm_source=github&utm_medium=syndication&utm_campaign=web-scraping-mcp)
|
|
348
|
+
|
|
349
|
+
Other HasData MCP servers: [Google Search](https://github.com/HasData/google-search-mcp), [Google Images](https://github.com/HasData/google-images-mcp), [Google Scholar](https://github.com/HasData/google-scholar-mcp), [Google Maps](https://github.com/HasData/google-maps-mcp), [Google Trends](https://github.com/HasData/google-trends-mcp), [Bing](https://github.com/HasData/bing-mcp), [DuckDuckGo](https://github.com/HasData/duckduckgo-mcp), [YouTube](https://github.com/HasData/youtube-mcp), [TikTok](https://github.com/HasData/tiktok-mcp), [Instagram](https://github.com/HasData/instagram-mcp), [Amazon](https://github.com/HasData/amazon-mcp), [Walmart](https://github.com/HasData/walmart-mcp), [Shopify](https://github.com/HasData/shopify-mcp), [Yelp](https://github.com/HasData/yelp-mcp), [Yellow Pages](https://github.com/HasData/yellowpages-mcp), [Zillow](https://github.com/HasData/zillow-mcp), [Redfin](https://github.com/HasData/redfin-mcp), [Airbnb](https://github.com/HasData/airbnb-mcp), [Booking.com](https://github.com/HasData/booking-mcp), [Indeed](https://github.com/HasData/indeed-mcp), [Glassdoor](https://github.com/HasData/glassdoor-mcp).
|
|
350
|
+
|
|
351
|
+
## Development
|
|
352
|
+
|
|
353
|
+
The launcher is a thin stdio bridge to the remote server, so there is nothing to build.
|
|
354
|
+
|
|
355
|
+
```bash
|
|
356
|
+
npm install
|
|
357
|
+
HASDATA_API_KEY=your_key_here npm test
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
The tests in `test/` assert the tool contract, the part that can break without a commit here. They check that `?apis=web_scraping` returns the one expected tool, that its name has not changed, that it still requires `url` and carries a description, that the parameters this README documents are still in the schema, and that the key in use is actually accepted.
|
|
361
|
+
|
|
362
|
+
Two tests pin the output-format behaviour, because it is the part of this README a reader is most likely to be caught by and the part a refactor is most likely to change. One asks for markdown alone and asserts the content arrives as a string at the top level. The other asks for `json` with `extractRules` and asserts the extraction lands in `extractedData` inside `json`. Both run without rendering, so the pair costs 2 credits.
|
|
363
|
+
|
|
364
|
+
The contract suite also runs weekly on a schedule, because the upstream tool list can change without anyone touching this repository.
|
|
365
|
+
|
|
366
|
+
## Contributing
|
|
367
|
+
|
|
368
|
+
A tool table, a response sample or a documented behaviour that does not match reality is worth an issue. There is a template for exactly that. Pull requests are welcome for the same, and for anything in the launcher.
|
|
369
|
+
|
|
370
|
+
## License
|
|
371
|
+
|
|
372
|
+
MIT, see [LICENSE](LICENSE).
|
package/index.mjs
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Thin launcher: connects an MCP client to HasData's hosted Web Scraping MCP server
|
|
3
|
+
// (streamable HTTP) through the mcp-remote stdio bridge. The server runs remotely.
|
|
4
|
+
// This package only proxies, so nothing here scrapes anything.
|
|
5
|
+
import { spawn } from 'node:child_process';
|
|
6
|
+
import { createRequire } from 'node:module';
|
|
7
|
+
import { dirname, join } from 'node:path';
|
|
8
|
+
|
|
9
|
+
const URL = 'https://mcp.hasdata.com/api/mcp?apis=web_scraping';
|
|
10
|
+
const key = process.env.HASDATA_API_KEY;
|
|
11
|
+
if (!key) {
|
|
12
|
+
process.stderr.write('HASDATA_API_KEY is not set. Create a free key at https://app.hasdata.com and set HASDATA_API_KEY.\n');
|
|
13
|
+
process.exit(1);
|
|
14
|
+
}
|
|
15
|
+
// Resolve mcp-remote's CLI from its own package.json bin, so a future layout change
|
|
16
|
+
// or an exports map does not break a hardcoded deep path.
|
|
17
|
+
const require = createRequire(import.meta.url);
|
|
18
|
+
const pkg = require('mcp-remote/package.json');
|
|
19
|
+
const proxy = join(dirname(require.resolve('mcp-remote/package.json')), pkg.bin['mcp-remote']);
|
|
20
|
+
const child = spawn(process.execPath, [proxy, URL, '--header', `x-api-key:${key}`], { stdio: 'inherit' });
|
|
21
|
+
child.on('exit', (code, signal) => {
|
|
22
|
+
if (signal) process.kill(process.pid, signal);
|
|
23
|
+
else process.exit(code ?? 0);
|
|
24
|
+
});
|
package/package.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@hasdata/web-scraping-mcp",
|
|
3
|
+
"mcpName": "com.hasdata/web-scraping",
|
|
4
|
+
"version": "1.0.0",
|
|
5
|
+
"description": "MCP server for HasData's Web Scraping API: fetch any public URL through managed proxies with optional JS rendering, geo targeting, wait conditions and extraction rules. 1,000 free credits every month.",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"bin": {
|
|
8
|
+
"hasdata-web-scraping-mcp": "index.mjs"
|
|
9
|
+
},
|
|
10
|
+
"files": [
|
|
11
|
+
"index.mjs",
|
|
12
|
+
"README.md",
|
|
13
|
+
"LICENSE"
|
|
14
|
+
],
|
|
15
|
+
"scripts": {
|
|
16
|
+
"test": "node --test"
|
|
17
|
+
},
|
|
18
|
+
"dependencies": {
|
|
19
|
+
"mcp-remote": "^0.1.43"
|
|
20
|
+
},
|
|
21
|
+
"engines": {
|
|
22
|
+
"node": ">=18"
|
|
23
|
+
},
|
|
24
|
+
"license": "MIT",
|
|
25
|
+
"repository": {
|
|
26
|
+
"type": "git",
|
|
27
|
+
"url": "git+https://github.com/HasData/web-scraping-mcp.git"
|
|
28
|
+
},
|
|
29
|
+
"homepage": "https://hasdata.com/web-scraping-api",
|
|
30
|
+
"keywords": [
|
|
31
|
+
"mcp",
|
|
32
|
+
"web-scraping",
|
|
33
|
+
"scraper",
|
|
34
|
+
"proxy",
|
|
35
|
+
"headless-browser",
|
|
36
|
+
"html-to-markdown",
|
|
37
|
+
"model-context-protocol",
|
|
38
|
+
"hasdata"
|
|
39
|
+
]
|
|
40
|
+
}
|