scrapeless-mcp-server 0.4.2 → 0.4.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +117 -147
- package/build/tools/browser/browser.js +39 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,105 +1,72 @@
|
|
|
1
|
-
|
|
2
2
|
|
|
3
3
|
# Scrapeless MCP Server
|
|
4
4
|
|
|
5
|
-
Model Context Protocol (MCP)
|
|
5
|
+
**Welcome to the official Scrapeless Model Context Protocol (MCP) Server** — a powerful integration layer that empowers LLMs, AI Agents, and AI applications to interact with the web in real time.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Built on the open MCP standard, Scrapeless MCP Server seamlessly connects models like **ChatGPT**, **Claude**, and tools like **Cursor** and **Windsurf** to a wide range of external capabilities, including:
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
- **Google services integration** (Search, Flights, Trends, Scholar, etc.)
|
|
10
|
+
- **Browser automation** for page-level navigation and interaction
|
|
11
|
+
- **Scrape** dynamic, JS-heavy sites—export as HTML, Markdown, or screenshots
|
|
10
12
|
|
|
11
|
-
|
|
12
|
-
- [glama.ai](https://glama.ai/mcp/servers/@scrapeless-ai/scrapeless-mcp-server)
|
|
13
|
+
Whether you're building an AI research assistant, a coding copilot, or autonomous web agents, this server provides the dynamic context and real-world data your workflows need—**without getting blocked**.
|
|
13
14
|
|
|
14
|
-
|
|
15
|
+
## Usage Examples
|
|
15
16
|
|
|
16
|
-
|
|
17
|
+
1. Automated Web Interaction and Data Extraction with Claude
|
|
17
18
|
|
|
18
|
-
|
|
19
|
+
Using Scrapeless MCP Browser, Claude can perform complex tasks such as web navigation, clicking, scrolling, and scraping through conversational commands, with real-time preview of web interaction results via `live sessions`.
|
|
19
20
|
|
|
20
|
-
-
|
|
21
|
+
|
|
21
22
|
|
|
22
|
-
|
|
23
|
+
2. Bypassing Cloudflare to Retrieve Target Page Content
|
|
23
24
|
|
|
24
|
-
|
|
25
|
-
- **Streamable HTTP** `--mode=streamable_http`: The new MCP transport protocol that simplifies client-server communication through a single HTTP endpoint
|
|
25
|
+
Using the Scrapeless MCP Browser service, the Cloudflare page is automatically accessed, and after the process is completed, the page content is extracted and returned in Markdown format.
|
|
26
26
|
|
|
27
|
-
|
|
27
|
+
|
|
28
28
|
|
|
29
|
-
|
|
29
|
+
3. Extracting Dynamically Rendered Page Content and Writing to File
|
|
30
30
|
|
|
31
|
-
-
|
|
32
|
-
Navigate browser to a specified URL.
|
|
31
|
+
Using the Scrapeless MCP Universal API, the JavaScript-rendered content of the target page above is scraped, exported in Markdown format, and finally written to a local file named **`text.md`**.
|
|
33
32
|
|
|
34
|
-
-
|
|
35
|
-
Go back one step in browser history.
|
|
33
|
+
|
|
36
34
|
|
|
37
|
-
|
|
38
|
-
Go forward one step in browser history.
|
|
35
|
+
4. Automated SERP Scraping
|
|
39
36
|
|
|
40
|
-
|
|
41
|
-
Click a specific element on the page.
|
|
37
|
+
Using the Scrapeless MCP Server, query the keyword “web scraping” on Google Search, retrieve the first 10 search results (including title, link, and summary), and write the content to the file named `serp.text`.
|
|
42
38
|
|
|
43
|
-
-
|
|
44
|
-
Type text into a specified input field.
|
|
39
|
+
|
|
45
40
|
|
|
46
|
-
|
|
47
|
-
Wait for a specific page element to appear.
|
|
41
|
+
Here are some additional examples of how to use these servers:
|
|
48
42
|
|
|
49
|
-
|
|
50
|
-
|
|
43
|
+
| Example |
|
|
44
|
+
| --------------------------------------------------------------------------------------------------------------------------------- |
|
|
45
|
+
| Please help me with my flight ticket from Chicago to New York on November 20, 2025. |
|
|
46
|
+
| Find papers by "Yoshua Bengio" on deep learning. |
|
|
47
|
+
| Search scrapeless by Google search. |
|
|
48
|
+
| Find the search interest for "AI" over the last year. |
|
|
49
|
+
| Use a browser to visit [chatgpt.com](http://chatgpt.com), search for "What's the weather like today?", and summarize the results. |
|
|
50
|
+
| Scrape the HTML content of [scrapeless.com](http://scrapeless.com) page. |
|
|
51
|
+
| Scrape the Markdown content of [scrapeless.com](http://scrapeless.com) page. |
|
|
52
|
+
| Get screenshots of [scrapeless.com](http://scrapeless.com). |
|
|
51
53
|
|
|
52
|
-
|
|
53
|
-
Capture a screenshot of the current page.
|
|
54
|
-
|
|
55
|
-
- `browser_get_html`
|
|
56
|
-
Get the full HTML of the current page.
|
|
57
|
-
|
|
58
|
-
- `browser_get_text`
|
|
59
|
-
Get all visible text from the current page.
|
|
60
|
-
|
|
61
|
-
- `browser_scroll`
|
|
62
|
-
Scroll to the bottom of the page.
|
|
63
|
-
|
|
64
|
-
- `browser_scroll_to`
|
|
65
|
-
Scroll a specific element into view.
|
|
66
|
-
|
|
67
|
-
### Universal Tools
|
|
68
|
-
|
|
69
|
-
- `scrape_html`
|
|
70
|
-
Scrape a URL and return its full HTML content.
|
|
71
|
-
- `scrape_markdown`
|
|
72
|
-
Scrape a URL and return its content as Markdown.
|
|
73
|
-
|
|
74
|
-
- `scrape_screenshot`
|
|
75
|
-
Capture a high-quality screenshot of any webpage.
|
|
76
|
-
|
|
77
|
-
### Deep SerpAPI Tools
|
|
78
|
-
|
|
79
|
-
- `google_search`
|
|
80
|
-
universal information search engine.retrieves any data information.explanatory queries (why, how).comparative analysis requests.
|
|
81
|
-
- `google_scholar`
|
|
82
|
-
Search for academic papers on Google Scholar.
|
|
83
|
-
|
|
84
|
-
- `google_flights`
|
|
85
|
-
Exclusive flight information query tool.
|
|
54
|
+
## Setup Guide
|
|
86
55
|
|
|
87
|
-
|
|
88
|
-
Get trending search data from Google Trends.
|
|
56
|
+
1. Get Scrapeless Key
|
|
89
57
|
|
|
90
|
-
|
|
58
|
+
- [Log in](https://app.scrapeless.com/passport/login?utm_source=github&utm_medium=github-mcp&utm_campaign=mcp) to the Scrapeless Dashboard（Free trial available）
|
|
59
|
+
- Then click "**Setting**" on the left -> select "**API Key Management**" -> click "**Create API Key**". Finally, click the API Key you created to **copy** it.
|
|
91
60
|
|
|
92
|
-
|
|
61
|
+
|
|
93
62
|
|
|
94
|
-
|
|
95
|
-
2. [Get your free trial](https://app.scrapeless.com/landing/guide?utm_source=github&utm_medium=mcp)
|
|
96
|
-
3. [Generate API Key](https://app.scrapeless.com/dashboard/settings/api-key?utm_source=github&utm_medium=mcp)
|
|
63
|
+
2. Configure Your MCP Client
|
|
97
64
|
|
|
98
|
-
|
|
65
|
+
Scrapeless MCP Server supports both **Stdio** and **Streamable HTTP** transport modes.
|
|
99
66
|
|
|
100
|
-
|
|
67
|
+
🖥️ Stdio (Local Execution)
|
|
101
68
|
|
|
102
|
-
```
|
|
69
|
+
```JSON
|
|
103
70
|
{
|
|
104
71
|
"mcpServers": {
|
|
105
72
|
"Scrapeless MCP Server": {
|
|
@@ -113,92 +80,95 @@ This project provides several MCP servers that enable AI assistants like Claude
|
|
|
113
80
|
}
|
|
114
81
|
```
|
|
115
82
|
|
|
116
|
-
|
|
83
|
+
🌐 Streamable HTTP (Hosted API Mode)
|
|
117
84
|
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
```shell
|
|
121
|
-
npx scrapeless-mcp-server --mode=streamable_http --SCRAPELESS_KEY=YOUR_SCRAPELESS_KEY
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
2. Add mcp configuration
|
|
125
|
-
|
|
126
|
-
```json
|
|
85
|
+
```JSON
|
|
127
86
|
{
|
|
128
87
|
"mcpServers": {
|
|
129
88
|
"Scrapeless MCP Server": {
|
|
130
89
|
"type": "streamable-http",
|
|
131
|
-
"url": "
|
|
132
|
-
"
|
|
90
|
+
"url": "https://api.scrapeless.com/mcp",
|
|
91
|
+
"headers": {
|
|
92
|
+
"x-api-token": "YOUR_SCRAPELESS_KEY"
|
|
93
|
+
},
|
|
94
|
+
"disabled": false,
|
|
95
|
+
"alwaysAllow": []
|
|
133
96
|
}
|
|
134
97
|
}
|
|
135
98
|
}
|
|
136
99
|
```
|
|
137
100
|
|
|
138
|
-
##
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
101
|
+
## Integration with Claude Desktop
|
|
102
|
+
|
|
103
|
+
1. Open **Claude Desktop**
|
|
104
|
+
2. Navigate to: `Settings` → `Tools` → `MCP Servers`
|
|
105
|
+
3. Click **"Add MCP Server"**
|
|
106
|
+
4. Paste either the `Stdio` or `Streamable HTTP` config above
|
|
107
|
+
5. Save and enable the server
|
|
108
|
+
6. Claude will now be able to issue web queries, extract content, and interact with pages using Scrapeless
|
|
109
|
+
|
|
110
|
+
## Integration with Cursor IDE
|
|
111
|
+
|
|
112
|
+
1. Open **Cursor**
|
|
113
|
+
2. Press `Cmd + Shift + P` and search for: `Configure MCP Servers`
|
|
114
|
+
3. Add the Scrapeless MCP config using the format above
|
|
115
|
+
4. Save the file and restart Cursor (if needed)
|
|
116
|
+
5. Now you can ask Cursor things like:
|
|
117
|
+
1. `"Search StackOverflow for a solution to this error"`
|
|
118
|
+
2. `"Scrape the HTML from this page"`
|
|
119
|
+
6. And it will use Scrapeless in the background.
|
|
120
|
+
|
|
121
|
+
## Supported MCP Tools
|
|
122
|
+
|
|
123
|
+
| Name | Description |
|
|
124
|
+
| ------------------ | ------------------------------------------------- |
|
|
125
|
+
| google_search | Universal information search engine. |
|
|
126
|
+
| google_flights | Exclusive flight information query tool. |
|
|
127
|
+
| google_trends | Get trending search data from Google Trends. |
|
|
128
|
+
| google_scholar | Search for academic papers on Google Scholar. |
|
|
129
|
+
| browser_goto | Navigate browser to a specified URL. |
|
|
130
|
+
| browser_go_back | Go back one step in browser history. |
|
|
131
|
+
| browser_go_forward | Go forward one step in browser history. |
|
|
132
|
+
| browser_click | Click a specific element on the page. |
|
|
133
|
+
| browser_type | Type text into a specified input field. |
|
|
134
|
+
| browser_press_key | Simulate a key press. |
|
|
135
|
+
| browser_wait_for | Wait for a specific page element to appear. |
|
|
136
|
+
| browser_wait | Pause execution for a fixed duration. |
|
|
137
|
+
| browser_screenshot | Capture a screenshot of the current page. |
|
|
138
|
+
| browser_get_html | Get the full HTML of the current page. |
|
|
139
|
+
| browser_get_text | Get all visible text from the current page. |
|
|
140
|
+
| browser_scroll | Scroll to the bottom of the page. |
|
|
141
|
+
| browser_scroll_to | Scroll a specific element into view. |
|
|
142
|
+
| scrape_html | Scrape a URL and return its full HTML content. |
|
|
143
|
+
| scrape_markdown | Scrape a URL and return its content as Markdown. |
|
|
144
|
+
| scrape_screenshot | Capture a high-quality screenshot of any webpage. |
|
|
145
|
+
|
|
146
|
+
## Security Best Practices
|
|
147
|
+
|
|
148
|
+
When using Scrapeless MCP Server with LLMs (like ChatGPT, Claude, or Cursor), it's critical to handle all scraped or extracted web content with care. **Web data is untrusted by default**, and improper handling may expose your application to prompt injection or other security vulnerabilities.
|
|
149
|
+
|
|
150
|
+
#### ✅ Recommended Practices
|
|
151
|
+
|
|
152
|
+
- **Never pass raw scraped content directly into LLM prompts.** Raw HTML, JavaScript, or user-generated text may contain hidden injection payloads.
|
|
153
|
+
- **Sanitize and validate all extracted content.** Strip or escape potentially harmful tags and scripts before using content in downstream logic or AI models.
|
|
154
|
+
- **Prefer structured extraction over free-form text.** Use tools like `scrape_html`, `scrape_markdown`, or targeted `browser_get_text` with known-safe selectors to extract only the content you trust.
|
|
155
|
+
- **Apply domain or selector whitelisting** when scraping dynamically generated pages, to restrict data flow to known and trusted sources.
|
|
156
|
+
- **Log and monitor all outbound requests** made via browser or scraping tools, especially if you're handling sensitive data, tokens, or internal network access.
|
|
157
|
+
|
|
158
|
+
#### 🚫 Avoid
|
|
159
|
+
|
|
160
|
+
- Injecting scraped HTML directly into prompts
|
|
161
|
+
- Letting users specify arbitrary URLs or CSS selectors without validation
|
|
162
|
+
- Storing unfiltered scraped content for future prompt usage
|
|
155
163
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
### google_flights
|
|
159
|
-
|
|
160
|
-
Please help me with my flight ticket from Chicago to New York on November 20, 2025
|
|
161
|
-
|
|
162
|
-
### google_scholar
|
|
163
|
-
|
|
164
|
-
Find papers by "Yoshua Bengio" on deep learning
|
|
165
|
-
|
|
166
|
-
### google_search
|
|
167
|
-
|
|
168
|
-
Search scrapeless by google search
|
|
169
|
-
|
|
170
|
-
### google_trends
|
|
171
|
-
|
|
172
|
-
Find the search interest for "AI" over the last year
|
|
173
|
-
|
|
174
|
-
## Installation
|
|
175
|
-
|
|
176
|
-
### Prerequisites
|
|
177
|
-
|
|
178
|
-
- Node.js 22 or higher
|
|
179
|
-
- NPM or Yarn
|
|
180
|
-
|
|
181
|
-
### Install from Source
|
|
182
|
-
|
|
183
|
-
1. Clone the repository:
|
|
184
|
-
|
|
185
|
-
```bash
|
|
186
|
-
git clone https://github.com/scrapeless-ai/scrapeless-mcp-server.git
|
|
187
|
-
cd scrapeless-mcp-server
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
2. Install dependencies:
|
|
191
|
-
|
|
192
|
-
```bash
|
|
193
|
-
npm install
|
|
194
|
-
```
|
|
164
|
+
## Community
|
|
195
165
|
|
|
196
|
-
|
|
166
|
+
- [MCP Server Discord](https://backend.scrapeless.com/app/api/v1/public/links/discord)
|
|
197
167
|
|
|
198
|
-
|
|
199
|
-
npm run build
|
|
200
|
-
```
|
|
168
|
+
## Contact Us
|
|
201
169
|
|
|
202
|
-
|
|
170
|
+
For questions, suggestions, or collaboration inquiries, feel free to contact us via:
|
|
203
171
|
|
|
204
|
-
- [
|
|
172
|
+
- Email: [market@scrapeless.com](mailto:market@scrapeless.com)
|
|
173
|
+
- Official Website: [https://www.scrapeless.com](https://www.scrapeless.com/)
|
|
174
|
+
- Community Forum: https://discord.gg/Np4CAHxB9a
|
|
@@ -442,3 +442,42 @@ export const browserScrollTo = defineTool({
|
|
|
442
442
|
}
|
|
443
443
|
},
|
|
444
444
|
});
|
|
445
|
+
export const browserPressKey = defineTool({
|
|
446
|
+
name: "browser_press_key",
|
|
447
|
+
description: `Simulate a key press.
|
|
448
|
+
Restrictions: Must specify a valid key name; optional target selector.
|
|
449
|
+
Valid: Press Enter in #search.
|
|
450
|
+
Invalid: Press a key without specifying the key name.`,
|
|
451
|
+
inputSchema: {
|
|
452
|
+
selector: z.string().describe("The CSS selector of the element to focus.").optional(),
|
|
453
|
+
key: z.string().describe('Name of the key to press or a character to generate, such as `ArrowLeft` or `a`'),
|
|
454
|
+
},
|
|
455
|
+
handle: async (context, params) => {
|
|
456
|
+
const session = context.getSession(params.sessionId);
|
|
457
|
+
if (!session?.page) {
|
|
458
|
+
return wrapMcpBrowserResponse("No active browser session found. Please create a browser session first.");
|
|
459
|
+
}
|
|
460
|
+
try {
|
|
461
|
+
if (params.selector) {
|
|
462
|
+
await session.page.focus(params.selector);
|
|
463
|
+
}
|
|
464
|
+
await session.page.keyboard.press(params.key);
|
|
465
|
+
const accessibilitySnapshot = await snapshot(session.page);
|
|
466
|
+
return {
|
|
467
|
+
content: [
|
|
468
|
+
{
|
|
469
|
+
type: "text",
|
|
470
|
+
text: params.selector ? `Focus on the ${params.selector} and press ${params.key}` : `press ${params.key}`,
|
|
471
|
+
},
|
|
472
|
+
{
|
|
473
|
+
type: "text",
|
|
474
|
+
text: accessibilitySnapshot,
|
|
475
|
+
},
|
|
476
|
+
],
|
|
477
|
+
};
|
|
478
|
+
}
|
|
479
|
+
catch (error) {
|
|
480
|
+
return wrapMcpBrowserResponse(`Failed to press key: ${error.message}`);
|
|
481
|
+
}
|
|
482
|
+
},
|
|
483
|
+
});
|