@spicrawl/mcp 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/CHANGELOG.md +7 -0
  2. package/README.md +293 -133
  3. package/package.json +17 -9
package/CHANGELOG.md CHANGED
@@ -3,6 +3,13 @@
3
3
  All notable changes to `@spicrawl/mcp` are documented here. This project follows
4
4
  [Semantic Versioning](https://semver.org/).
5
5
 
6
+ ## 0.1.1 — 2026-10-01
7
+
8
+ Documentation and package metadata only; the server and its 25 tools are unchanged.
9
+
10
+ - README rewritten for npmjs.com: hosted endpoint first, one-click install links for Cursor and VS Code, per-client setup for Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex, tool tables by group, a "which tool to use" guide, environment variables, self-hosting, FAQ and troubleshooting. Every link is absolute so it works on npm.
11
+ - `package.json`: `mcpName` (`io.github.Spicrawl/mcp`) for the official MCP registry, a new `description` and `keywords`, and `homepage` now points at https://docs.spicrawl.com/agents/mcp.
12
+
6
13
  ## 0.1.0 — 2026-10-01
7
14
 
8
15
  First public release of the Spicrawl MCP server, published to npm as `@spicrawl/mcp` from https://github.com/Spicrawl/mcp with an npm provenance attestation.
package/README.md CHANGED
@@ -1,37 +1,145 @@
1
- # Spicrawl MCP Server: Web Scraping Tools for AI Agents
1
+ # Spicrawl MCP Server
2
2
 
3
- The official [Model Context Protocol](https://modelcontextprotocol.io) server for [Spicrawl](https://spicrawl.com), the web scraping API for AI agents and LLMs. Connect it to Claude Code, Claude Desktop, Cursor, VS Code, Codex or any MCP client, and your agent can scrape pages to Markdown, extract structured data, run batch jobs over thousands of URLs, keep browser sessions and read the Spicrawl docs, all as native tools.
3
+ The Spicrawl MCP server is a Model Context Protocol server that gives AI agents in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex web scraping to Markdown, structured data extraction, screenshots, browser sessions and batch scraping as 25 native tools.
4
4
 
5
5
  [![npm version](https://img.shields.io/npm/v/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
6
- [![license](https://img.shields.io/npm/l/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
6
+ [![npm downloads](https://img.shields.io/npm/dm/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
7
+ [![license](https://img.shields.io/npm/l/@spicrawl/mcp.svg)](https://github.com/Spicrawl/mcp/blob/main/LICENSE)
7
8
  [![CI](https://github.com/Spicrawl/mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/Spicrawl/mcp/actions/workflows/ci.yml)
8
- [![docs](https://img.shields.io/badge/docs-docs.spicrawl.com-blue.svg)](https://docs.spicrawl.com/agents/mcp)
9
9
 
10
- **[Docs](https://docs.spicrawl.com/agents/mcp)** · **[Get an API key](https://app.spicrawl.com/signup)** · **[GitHub](https://github.com/Spicrawl/mcp)** · **[SDK](https://github.com/Spicrawl/sdk)** · **[CLI](https://github.com/Spicrawl/cli)** · **[Changelog](https://github.com/Spicrawl/mcp/blob/main/CHANGELOG.md)**
10
+ **[Docs](https://docs.spicrawl.com/agents/mcp)** · **[Get an API key](https://app.spicrawl.com/signup)** · **[TypeScript SDK](https://github.com/Spicrawl/sdk)** · **[CLI](https://github.com/Spicrawl/cli)** · **[Agent plugins](https://github.com/Spicrawl/agent-plugins)** · **[Changelog](https://github.com/Spicrawl/mcp/blob/main/CHANGELOG.md)**
11
11
 
12
- ## Two ways to connect
12
+ <p>
13
+ <a href="https://cursor.com/en/install-mcp?name=spicrawl&config=eyJ1cmwiOiJodHRwczovL21jcC5zcGljcmF3bC5jb20vbWNwIiwiaGVhZGVycyI6eyJBdXRob3JpemF0aW9uIjoiQmVhcmVyICR7ZW52OlNQSUNSQVdMX0FQSV9LRVl9In19"><img src="https://cursor.com/deeplink/mcp-install-dark.svg" alt="Add Spicrawl MCP server to Cursor" height="32" /></a>
14
+ <a href="https://vscode.dev/redirect/mcp/install?name=spicrawl&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22spicrawl-api-key%22%2C%22description%22%3A%22Spicrawl%20API%20key%22%2C%22password%22%3Atrue%7D%5D&config=%7B%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fmcp.spicrawl.com%2Fmcp%22%2C%22headers%22%3A%7B%22Authorization%22%3A%22Bearer%20%24%7Binput%3Aspicrawl-api-key%7D%22%7D%7D"><img src="https://img.shields.io/badge/VS_Code-Install_Spicrawl_MCP-0098FF?style=for-the-badge&logo=visualstudiocode&logoColor=white" alt="Install Spicrawl MCP server in VS Code" height="32" /></a>
15
+ </p>
13
16
 
14
- | | Hosted endpoint | Local (stdio) |
17
+ ## Quickstart
18
+
19
+ 1. Create an API key at [app.spicrawl.com](https://app.spicrawl.com/signup). Keys start with `spicrawl_live_` or `spicrawl_test_`.
20
+ 2. Connect the **hosted endpoint**, `https://mcp.spicrawl.com/mcp` (MCP Streamable HTTP, nothing to install):
21
+
22
+ ```bash
23
+ export SPICRAWL_API_KEY=spicrawl_live_...
24
+ claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
25
+ --header "Authorization: Bearer $SPICRAWL_API_KEY"
26
+ ```
27
+
28
+ Codex: `codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY`
29
+
30
+ 3. Or run it **locally over stdio** (Node.js 20+), for clients that only launch local servers:
31
+
32
+ ```bash
33
+ claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
34
+ ```
35
+
36
+ 4. Ask your agent: *"Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price."* It calls `spicrawl_scrape` and gets the page back as Markdown.
37
+
38
+ Both transports serve the same 25 `spicrawl_*` tools, and every call runs under your key with the same scopes, limits and credits as a direct API call.
39
+
40
+ ## What it does
41
+
42
+ - **Scrape any URL to Markdown**, text, HTML, a JSON envelope or a printed PDF with one tool, `spicrawl_scrape`. Markdown is the default and strips the page to its main content.
43
+ - **Render JavaScript** with `render: true` (the `obscura` browser engine, 3 credits) or pin real Chromium with `engine: "chromium"` (8 credits). A plain fetch costs 1 credit, and `mode: "auto"` escalates fetch to browser and bills only the rung that worked ([price table](https://docs.spicrawl.com/credits)).
44
+ - **Extract structured data** with a CSS/XPath selector map (`extract`) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (`autoparse`).
45
+ - **Drive the page** before capture with 8 typed browser actions: `click`, `fill`, `wait_for`, `wait_for_navigation`, `scroll` (including infinite scroll), `select`, `evaluate` and `screenshot`.
46
+ - **Return screenshots and PDFs the model can see**: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as `application/pdf` resource blocks (up to 10 MB).
47
+ - **Batch scrape thousands of URLs** as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open for crawling.
48
+ - **Keep a logged-in browser** with persistent sessions: cookies, storage and a pinned engine reused across scrapes.
49
+ - **Answer "what did that cost?"** from usage and request-log tools, and **read the Spicrawl docs** from inside the agent.
50
+
51
+ ## Which tool to use
52
+
53
+ - If you need one page's content or data now, use `spicrawl_scrape`.
54
+ - If the page is blank or a skeleton without JavaScript, call `spicrawl_scrape` again with `render: true`.
55
+ - If you know the page's structure, use `extract` with CSS selectors; if you don't, try `autoparse`.
56
+ - If you have more than about 20 URLs, use `spicrawl_batch_submit`, then `spicrawl_batch_status` and `spicrawl_batch_results`.
57
+ - If you discover URLs as you go (crawling), submit with `open: true`, append with `spicrawl_batch_add_items` and finish with `spicrawl_batch_close`.
58
+ - If you must log in first, use `spicrawl_session_create` and pass its `id` as `session_id` to `spicrawl_scrape`.
59
+ - If a call failed and you need to know why, use `spicrawl_request_get` with the request id, or `spicrawl_requests_list` with `only_errors`.
60
+ - If you are unsure of a parameter or an error code, use `spicrawl_docs_search` before guessing.
61
+
62
+ ## Tools
63
+
64
+ 25 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Full argument reference: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp#tools).
65
+
66
+ ### Scrape a website to Markdown
67
+
68
+ | Tool | What it does | Key parameters |
69
+ |---|---|---|
70
+ | `spicrawl_scrape` | Retrieve one URL as `markdown` (default), `text`, `html`, `json` or `pdf`, with optional rendering, extraction, links, screenshots and browser actions. | `url`, `format`, `render`, `engine`, `mode`, `main_content_only`, `include_tags`, `exclude_tags`, `extract`, `autoparse`, `links`, `screenshot`, `actions`, `wait_for`, `session_id`, `proxy`, `cache`, `max_cost` |
71
+
72
+ ### Batch scrape many URLs
73
+
74
+ | Tool | What it does | Key parameters |
75
+ |---|---|---|
76
+ | `spicrawl_batch_submit` | Queue many URLs as one async job with shared settings; returns the job with `estimated_credits`. | `urls` or `items`, `render`, `format`, `open`, `max_attempts`, `credit_budget`, `max_cost` |
77
+ | `spicrawl_batch_list` | List the project's batch jobs, newest first. | `status`, `limit` (max 200), `cursor` |
78
+ | `spicrawl_batch_status` | One job's status and progress. | `job_id` |
79
+ | `spicrawl_batch_results` | A page of finished items; works while the job runs. | `job_id`, `status`, `limit` (max 5000), `cursor` |
80
+ | `spicrawl_batch_task_content` | The full payload of one item. | `job_id`, `seq` |
81
+ | `spicrawl_batch_cancel` | Cancel a job; finished items keep their results. | `job_id` |
82
+ | `spicrawl_batch_retry` | Re-queue every failed item. | `job_id` |
83
+ | `spicrawl_batch_add_items` | Append URLs to a job submitted with `open: true`. | `job_id`, `urls` or `items` |
84
+ | `spicrawl_batch_close` | Stop an open job accepting items so it can complete. | `job_id` |
85
+
86
+ ### Browser sessions
87
+
88
+ | Tool | What it does | Key parameters |
89
+ |---|---|---|
90
+ | `spicrawl_session_create` | Create a persistent browser identity (cookies, storage, pinned engine). | `engine`, `ttl_seconds`, `session_context` |
91
+ | `spicrawl_session_list` | List sessions, newest first. | `status`, `engine`, `limit`, `cursor` |
92
+ | `spicrawl_session_get` | One session's status, engine, exit, usage and expiry. | `session_id` |
93
+ | `spicrawl_session_context` | Dump a live session's cookies and storage (contains secrets). | `session_id` |
94
+ | `spicrawl_session_release` | End a session; its record stays as `released`. | `session_id` |
95
+ | `spicrawl_session_delete` | Permanently delete a session and its state. | `session_id` |
96
+
97
+ ### Usage and request history
98
+
99
+ | Tool | What it does | Key parameters |
15
100
  |---|---|---|
16
- | What | `https://mcp.spicrawl.com/mcp`, MCP Streamable HTTP | `npx -y @spicrawl/mcp`, a subprocess of your MCP client |
17
- | Install | nothing | Node.js 20 or later |
18
- | Auth | `Authorization: Bearer <API key>` header | `SPICRAWL_API_KEY` environment variable |
19
- | Use when | your client supports remote MCP servers (most do) | your client only launches local servers, or you want to pin a version |
101
+ | `spicrawl_usage` | Billing-grade usage over a date window (`read` scope). | `from`, `to`, `group_by`, `metrics` |
102
+ | `spicrawl_usage_summary` | Current-period totals (`read` scope). | none |
103
+ | `spicrawl_usage_reconciliation` | One day's drift between the request log and the billed rollup (`read` scope). | `day` |
104
+ | `spicrawl_requests_list` | Recent API calls with status, error code, engine, latency and credits. | `only_errors`, `status`, `limit`, `all_projects` |
105
+ | `spicrawl_request_get` | The full log record of one call. | `id` |
106
+
107
+ ### Docs
108
+
109
+ | Tool | What it does | Key parameters |
110
+ |---|---|---|
111
+ | `spicrawl_docs_search` | Search the Spicrawl docs; an error code also links to its entry. | `query`, `limit` (1-20, default 8) |
112
+ | `spicrawl_docs_read` | Read one docs page as Markdown (truncated over 60,000 characters). | `path` |
113
+ | `spicrawl_docs_index` | The docs' `llms.txt`: every page with its title and `.md` link. | none |
114
+
115
+ ### Coming soon
116
+
117
+ `spicrawl_browser_connect_url` will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from Puppeteer or Playwright; remote browsers are not available yet. The schemas also accept `ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country` and `sticky_key` ahead of launch. Agents should not send them yet.
118
+
119
+ ## Example prompts
20
120
 
21
- Both serve the same 25 `spicrawl_*` tools. [Create an API key](https://app.spicrawl.com/signup) in the dashboard first; keys start with `spicrawl_live_` or `spicrawl_test_`.
121
+ - *"Scrape https://example.com/pricing as markdown and summarise the plans."*
122
+ - *"Get the product name, price and stock status from this URL."* (`spicrawl_scrape` with `extract` or `autoparse`)
123
+ - *"Take a full-page screenshot of https://example.com after the cookie banner is gone."* (`render`, `actions`, `screenshot_fullpage`)
124
+ - *"Scrape these 500 URLs and give me each page title."* (`spicrawl_batch_submit`, then `spicrawl_batch_results`)
125
+ - *"Why did my last scrape fail?"* (`spicrawl_requests_list` with `only_errors`, then `spicrawl_request_get`)
126
+ - *"How many credits have I used this month?"* (`spicrawl_usage_summary`)
127
+
128
+ ## Set up your MCP client
22
129
 
23
- ## Hosted endpoint
130
+ Put your key in `SPICRAWL_API_KEY` and reference it from the config where the client allows, so the key never lands in a committed file.
24
131
 
25
- Each tool call runs under your key, exactly as if you had called the API yourself, and a session is bound to the key that opened it. The key is checked when a session opens: an unknown or revoked key gets `401`, and if the API cannot be reached to check it the endpoint answers `503` with `Retry-After` (retry rather than rotating the key).
132
+ <details>
133
+ <summary><b>Claude Code</b></summary>
26
134
 
27
- **Claude Code:**
135
+ Hosted endpoint, stored for you only (add `--scope user` for every project):
28
136
 
29
137
  ```bash
30
138
  claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
31
- --header "Authorization: Bearer spicrawl_live_..."
139
+ --header "Authorization: Bearer $SPICRAWL_API_KEY"
32
140
  ```
33
141
 
34
- or in a project's `.mcp.json`:
142
+ To share it with a team, commit `.mcp.json` at the project root. Claude Code expands `${VAR}`, so each teammate's own key is used:
35
143
 
36
144
  ```json
37
145
  {
@@ -39,26 +147,68 @@ or in a project's `.mcp.json`:
39
147
  "spicrawl": {
40
148
  "type": "http",
41
149
  "url": "https://mcp.spicrawl.com/mcp",
42
- "headers": { "Authorization": "Bearer spicrawl_live_..." }
150
+ "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
151
+ }
152
+ }
153
+ }
154
+ ```
155
+
156
+ Or install the plugin, which bundles the MCP server and the Spicrawl agent skill:
157
+
158
+ ```bash
159
+ claude plugin marketplace add Spicrawl/agent-plugins
160
+ claude plugin install spicrawl@spicrawl-plugins
161
+ ```
162
+
163
+ Check with `claude mcp list`, or `/mcp` in a session. More: [docs.spicrawl.com/agents/claude-code](https://docs.spicrawl.com/agents/claude-code).
164
+
165
+ </details>
166
+
167
+ <details>
168
+ <summary><b>Claude Desktop</b></summary>
169
+
170
+ Claude Desktop's `claude_desktop_config.json` starts local (stdio) servers only, and custom connectors cannot send an `Authorization` header, so run the package locally. The file is at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS and `%APPDATA%\Claude\claude_desktop_config.json` on Windows:
171
+
172
+ ```json
173
+ {
174
+ "mcpServers": {
175
+ "spicrawl": {
176
+ "command": "npx",
177
+ "args": ["-y", "@spicrawl/mcp"],
178
+ "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
43
179
  }
44
180
  }
45
181
  }
46
182
  ```
47
183
 
48
- **Cursor** (`.cursor/mcp.json` or `~/.cursor/mcp.json`):
184
+ Restart Claude Desktop after saving.
185
+
186
+ </details>
187
+
188
+ <details>
189
+ <summary><b>Cursor</b></summary>
190
+
191
+ Use the **Add to Cursor** button above, or add this to `.cursor/mcp.json` in the project (or `~/.cursor/mcp.json` for every project). Cursor resolves `${env:NAME}` in `url` and `headers`, so start Cursor from an environment where `SPICRAWL_API_KEY` is set:
49
192
 
50
193
  ```json
51
194
  {
52
195
  "mcpServers": {
53
196
  "spicrawl": {
54
197
  "url": "https://mcp.spicrawl.com/mcp",
55
- "headers": { "Authorization": "Bearer spicrawl_live_..." }
198
+ "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
56
199
  }
57
200
  }
58
201
  }
59
202
  ```
60
203
 
61
- **VS Code** (`.vscode/mcp.json`; VS Code uses the `servers` key). The `inputs` entry makes VS Code ask for the key once and store it, so the file is safe to commit:
204
+ More: [docs.spicrawl.com/agents/cursor](https://docs.spicrawl.com/agents/cursor).
205
+
206
+ </details>
207
+
208
+ <details>
209
+ <summary><b>VS Code (GitHub Copilot)</b></summary>
210
+
211
+ Use the **Install in VS Code** button above, or add this to `.vscode/mcp.json`. VS Code uses the `servers` key, and the input variable makes it ask for the key once and store it securely, so the file is safe to commit:
62
212
 
63
213
  ```json
64
214
  {
@@ -75,66 +225,80 @@ or in a project's `.mcp.json`:
75
225
  }
76
226
  ```
77
227
 
78
- **Codex** reads the key from the environment each time it connects, so the key never lands in a file:
228
+ VS Code does not expand `${env:...}` inside `headers`, so use the input variable. More: [docs.spicrawl.com/agents/github-copilot](https://docs.spicrawl.com/agents/github-copilot).
229
+
230
+ </details>
231
+
232
+ <details>
233
+ <summary><b>Windsurf (Devin Desktop)</b></summary>
234
+
235
+ Windsurf is now named Devin Desktop. For the default **Devin Local** agent, add the server in local scope (saved to the gitignored `.devin/mcp_config.local.json`):
79
236
 
80
237
  ```bash
81
- export SPICRAWL_API_KEY=spicrawl_live_...
82
- codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY
238
+ devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"
83
239
  ```
84
240
 
85
- The equivalent `~/.codex/config.toml`:
241
+ For the legacy **Cascade** agent, edit `~/.config/devin/mcp_config.json` (`%APPDATA%\devin\mcp_config.json` on Windows). Cascade reads the URL from `serverUrl` and replaces `${env:...}`:
86
242
 
87
- ```toml
88
- [mcp_servers.spicrawl]
89
- url = "https://mcp.spicrawl.com/mcp"
90
- bearer_token_env_var = "SPICRAWL_API_KEY"
243
+ ```json
244
+ {
245
+ "mcpServers": {
246
+ "spicrawl": {
247
+ "serverUrl": "https://mcp.spicrawl.com/mcp",
248
+ "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
249
+ }
250
+ }
251
+ }
91
252
  ```
92
253
 
93
- **Claude Desktop** only launches local servers from `claude_desktop_config.json`, and custom connectors cannot send an `Authorization` header, so use the [local server](#local-server-stdio) below.
254
+ More: [docs.spicrawl.com/agents/windsurf](https://docs.spicrawl.com/agents/windsurf).
94
255
 
95
- ## Local server (stdio)
256
+ </details>
96
257
 
97
- `npx -y @spicrawl/mcp` downloads the package and runs the stdio server. The client starts it and passes your key in its environment.
258
+ <details>
259
+ <summary><b>Gemini CLI</b></summary>
98
260
 
99
- **Claude Code:**
261
+ Gemini CLI strips variables whose names contain `KEY`, `TOKEN` or `AUTH` before expanding headers, so export the key under another name:
100
262
 
101
263
  ```bash
102
- claude mcp add spicrawl --env SPICRAWL_API_KEY=spicrawl_live_... -- npx -y @spicrawl/mcp
264
+ export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"
103
265
  ```
104
266
 
105
- **Claude Desktop** (`claude_desktop_config.json`) and **Cursor** (`.cursor/mcp.json`):
267
+ Then add to `~/.gemini/settings.json` (or `.gemini/settings.json` for one project). Streamable HTTP uses the `httpUrl` key:
106
268
 
107
269
  ```json
108
270
  {
109
271
  "mcpServers": {
110
272
  "spicrawl": {
111
- "command": "npx",
112
- "args": ["-y", "@spicrawl/mcp"],
113
- "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
273
+ "httpUrl": "https://mcp.spicrawl.com/mcp",
274
+ "headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
114
275
  }
115
276
  }
116
277
  }
117
278
  ```
118
279
 
119
- **VS Code** (`.vscode/mcp.json`):
280
+ Or: `gemini mcp add --scope user --transport http --header 'Authorization: Bearer $SPICRAWL_MCP_BEARER' spicrawl https://mcp.spicrawl.com/mcp`. Run `/mcp` to check. More: [docs.spicrawl.com/agents/gemini-cli](https://docs.spicrawl.com/agents/gemini-cli).
120
281
 
121
- ```json
122
- {
123
- "inputs": [
124
- { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
125
- ],
126
- "servers": {
127
- "spicrawl": {
128
- "type": "stdio",
129
- "command": "npx",
130
- "args": ["-y", "@spicrawl/mcp"],
131
- "env": { "SPICRAWL_API_KEY": "${input:spicrawl-api-key}" }
132
- }
133
- }
134
- }
282
+ </details>
283
+
284
+ <details>
285
+ <summary><b>Codex</b></summary>
286
+
287
+ Codex reads the key from the environment each time it connects, so the key never lands in a file:
288
+
289
+ ```bash
290
+ codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY
135
291
  ```
136
292
 
137
- **Codex** (`~/.codex/config.toml`):
293
+ The equivalent `~/.codex/config.toml` (or `.codex/config.toml` in a trusted project):
294
+
295
+ ```toml
296
+ [mcp_servers.spicrawl]
297
+ url = "https://mcp.spicrawl.com/mcp"
298
+ bearer_token_env_var = "SPICRAWL_API_KEY"
299
+ ```
300
+
301
+ Local stdio instead:
138
302
 
139
303
  ```toml
140
304
  [mcp_servers.spicrawl]
@@ -143,79 +307,40 @@ args = ["-y", "@spicrawl/mcp"]
143
307
  env = { SPICRAWL_API_KEY = "spicrawl_live_..." }
144
308
  ```
145
309
 
146
- Restart the client; the `spicrawl_*` tools appear. To install it once instead of fetching it on each start, run `npm install -g @spicrawl/mcp` and use the command `spicrawl-mcp` (`spicrawl-mcp-server` is an alias).
310
+ More: [docs.spicrawl.com/agents/codex](https://docs.spicrawl.com/agents/codex).
147
311
 
148
- ### Environment variables
149
-
150
- | Variable | Required | Default | Notes |
151
- |---|---|---|---|
152
- | `SPICRAWL_API_KEY` | yes (stdio) | — | Your Spicrawl API key. The HTTP server ignores it and uses each caller's bearer token instead. |
153
- | `SPICRAWL_BASE_URL` | no | `https://api.spicrawl.com` (stdio), `http://127.0.0.1:8080` (HTTP server) | The Spicrawl API base URL. Set it for a self-hosted or local stack, and always when you run the HTTP server against the public API. |
154
- | `SPICRAWL_PUBLIC_BASE_URL` | no | `SPICRAWL_BASE_URL` | The API as the end user reaches it, used only for URLs handed back (`spicrawl_browser_connect_url`). Set it when the server talks to the API over a private address. |
155
- | `SPICRAWL_DOCS_URL` | no | see below | The docs base URL, no trailing slash. The docs tools read `<base>/api/search`, `<base>/<page>.md` and `<base>/llms.txt`. The docs are public: your API key is never sent there. |
156
- | `SPICRAWL_DOCS_HOST` | no | — | Legacy: an origin whose docs are at `/docs`. Read only when `SPICRAWL_DOCS_URL` is unset. |
157
- | `SPICRAWL_MCP_ADDR` | no | `127.0.0.1:8090` | HTTP server only: the listen address, `host:port`. |
158
- | `SPICRAWL_MCP_SESSION_IDLE_MS` | no | `1800000` (30 min) | HTTP server only: idle sessions are dropped after this long. |
312
+ </details>
159
313
 
160
- Without `SPICRAWL_DOCS_URL` or `SPICRAWL_DOCS_HOST`, the docs tools read a self-hosted API's own docs (`SPICRAWL_PUBLIC_BASE_URL`, else `SPICRAWL_BASE_URL`, plus `/docs`), and otherwise `https://docs.spicrawl.com`.
314
+ <details>
315
+ <summary><b>Any other MCP client</b></summary>
161
316
 
162
- ## Tools
163
-
164
- | Tool | What it does |
317
+ | Setting | Value |
165
318
  |---|---|
166
- | **Scrape** | |
167
- | `spicrawl_scrape` | Retrieve a URL as markdown/text/html/json, or print it to a PDF. Optional browser render, selector or AI extraction, links, screenshot, DOM scoping, proxy control, caching, browser actions, network capture. |
168
- | **Batch** | |
169
- | `spicrawl_batch_submit` | Scrape many URLs as one async job with shared settings. Output is markdown unless `format` says otherwise, the same default as `spicrawl_scrape`. Returns the job object. |
170
- | `spicrawl_batch_list` | List the project's batch jobs, newest first, with status and progress. |
171
- | `spicrawl_batch_status` | Poll one batch job's status and progress. |
172
- | `spicrawl_batch_results` | Read a page of a job's finished items (content, or a `result_url` API content path per item). |
173
- | `spicrawl_batch_task_content` | Read the full payload of one item by its `seq`. |
174
- | `spicrawl_batch_cancel` | Cancel a job: unstarted items never run, finished ones keep their results. |
175
- | `spicrawl_batch_retry` | Re-queue every failed item of a job. |
176
- | `spicrawl_batch_add_items` | Append URLs to a job submitted with `open: true`. |
177
- | `spicrawl_batch_close` | Stop an open job accepting items so it can complete. |
178
- | **Sessions** | |
179
- | `spicrawl_session_create` | Create a persistent browser identity (cookies, storage, pinned engine, sticky exit IP) to reuse across scrapes. |
180
- | `spicrawl_session_list` | List the project's sessions, newest first. |
181
- | `spicrawl_session_get` | One session's metadata (status, engine, exit, usage, expiry). |
182
- | `spicrawl_session_context` | Dump a live session's stored state (cookies, storage). It contains secrets, and can clone a session. |
183
- | `spicrawl_session_release` | End a session; its record stays visible as `released`. |
184
- | `spicrawl_session_delete` | Permanently delete a session and its stored state. |
185
- | **Usage and request history** | |
186
- | `spicrawl_usage` | Billing-grade usage over a date window, grouped by day, project, engine or feature (needs the `read` scope). |
187
- | `spicrawl_usage_summary` | Current-period usage totals (needs the `read` scope). |
188
- | `spicrawl_usage_reconciliation` | One day's drift between the request log and the billed rollup (needs the `read` scope). |
189
- | `spicrawl_requests_list` | Recent API calls in the key's project with status, errors, engine, proxy, latency and credits; `all_projects: true` lists every project in the organization (needs the `read` scope). |
190
- | `spicrawl_request_get` | The full log record of one call by its request id, for debugging a failed or slow scrape. |
191
- | **Browser** | |
192
- | `spicrawl_browser_connect_url` | Coming soon: remote browsers are not available yet. Mints a single-use CDP WebSocket URL for driving Spicrawl's remote browser from your own Puppeteer or Playwright. |
193
- | **Docs** | |
194
- | `spicrawl_docs_search` | Search the Spicrawl docs (`query`, optional `limit` 1-20, default 8). Results are grouped by page, with matching sections, snippets and the `.md` URL to read next. An error code (`ERR::FAMILY::NAME`) also gets a link to its entry on the errors page. |
195
- | `spicrawl_docs_read` | Read one docs page as Markdown by path (`guides/anti-bot`) or docs URL, optionally with `#anchor`. Pages over 60k characters are truncated and say so; other hosts and `..` are refused. |
196
- | `spicrawl_docs_index` | The docs' `llms.txt`: every page with its title, description and `.md` link. |
197
-
198
- See the [MCP server docs](https://docs.spicrawl.com/agents/mcp) for each tool's arguments.
319
+ | Transport | Streamable HTTP |
320
+ | URL | `https://mcp.spicrawl.com/mcp` |
321
+ | Header | `Authorization: Bearer <your key>` |
199
322
 
200
- ## Example prompts
323
+ A stdio-only client runs `npx -y @spicrawl/mcp` with `SPICRAWL_API_KEY` in its environment. The server accepts API keys only, not OAuth. Setup pages for more clients (OpenCode, Zed, JetBrains, Cline, Goose and others): [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp).
201
324
 
202
- Once connected, an agent can act on requests like:
325
+ </details>
203
326
 
204
- - *"Scrape https://example.com/pricing as markdown and summarise the plans."*
205
- - *"Get the product name, price and stock status from this URL."* The agent calls `spicrawl_scrape` with `autoparse` or an `extract` map.
206
- - *"Scrape these 50 URLs and give me the titles."* The agent calls `spicrawl_batch_submit`, then `spicrawl_batch_status` and `spicrawl_batch_results`.
327
+ ## Environment variables
207
328
 
208
- ## Behaviour worth knowing
329
+ | Variable | Default | Used by | Purpose |
330
+ |---|---|---|---|
331
+ | `SPICRAWL_API_KEY` | none (required) | stdio | Your API key. The HTTP server ignores it and uses each caller's bearer token. |
332
+ | `SPICRAWL_BASE_URL` | `https://api.spicrawl.com` (stdio), `http://127.0.0.1:8080` (HTTP) | both | The Spicrawl API the tools call. |
333
+ | `SPICRAWL_PUBLIC_BASE_URL` | `SPICRAWL_BASE_URL` | both | The API as the end user reaches it, used only in URLs handed back. |
334
+ | `SPICRAWL_DOCS_URL` | `https://docs.spicrawl.com` | both | Docs base URL for the docs tools. Your key is never sent there. |
335
+ | `SPICRAWL_DOCS_HOST` | none | both | Legacy: an origin whose docs are at `/docs`. Read only when `SPICRAWL_DOCS_URL` is unset. |
336
+ | `SPICRAWL_MCP_ADDR` | `127.0.0.1:8090` | HTTP | Listen address, `host:port`. |
337
+ | `SPICRAWL_MCP_SESSION_IDLE_MS` | `1800000` (30 min) | HTTP | Idle MCP sessions are dropped after this long. |
209
338
 
210
- - Tools reject unknown arguments rather than ignoring them. A misspelt or unsupported field (e.g. `js_render` on `spicrawl_scrape`, whose argument is `render`) fails the call with an error naming it, so a setting the agent believes it applied is never dropped.
211
- - Errors carry the platform's own code and guidance (e.g. `ERR::UPSTREAM::CHALLENGE` when a site served a bot wall), so the agent can decide whether to retry or change parameters.
212
- - `spicrawl_scrape` screenshots come back as MCP image content blocks the model can see. In the JSON, each `screenshots` entry keeps its metadata but its base64 is replaced by a pointer to its block. An image over 5 MB of base64 is not attached, and its entry says how to get a smaller one.
213
- - `spicrawl_scrape` with `format: "pdf"` (plus `render` or `engine: "chromium"`) returns the printed page as an MCP resource block (`application/pdf`). A PDF over 10 MB of base64 is not attached; `pdf.not_attached` says so.
214
- - Coming soon, and marked so in the tool descriptions: `ai_extract`, stealth mode (`stealth`, engine `camoufox`), `extract_preset`, the managed proxy pool (`premium_proxy`, `proxy_country`, `sticky_key`, session `rotate_ip`/`region_pool`) and `spicrawl_browser_connect_url`. The schemas already accept them so nothing changes the day they ship; until then agents should not send them.
339
+ With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at `<SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs`.
215
340
 
216
- ## Self-hosting the HTTP server
341
+ ## Self-host the MCP server over HTTP
217
342
 
218
- The hosted endpoint is this package's Streamable HTTP server. To run your own, for example beside a self-hosted Spicrawl deployment:
343
+ The hosted endpoint runs this package's Streamable HTTP server. To run your own:
219
344
 
220
345
  ```bash
221
346
  SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
@@ -223,30 +348,65 @@ SPICRAWL_BASE_URL=https://api.spicrawl.com \
223
348
  npx -y @spicrawl/mcp --http
224
349
  ```
225
350
 
226
- or, from an installed copy, `node node_modules/@spicrawl/mcp/dist/http.js` with the same environment.
351
+ - `SPICRAWL_BASE_URL` **defaults to `http://127.0.0.1:8080`** in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it to `https://api.spicrawl.com` to front the public API.
352
+ - MCP is served at `POST`/`GET`/`DELETE /mcp`; liveness at `GET /healthz` (no auth).
353
+ - Each caller sends their own key as `Authorization: Bearer spicrawl_live_...`. The key is checked against the API before a session opens (`401` if refused, `503` with `Retry-After` if the API cannot be reached), and a session is pinned to the key that opened it.
354
+ - The server speaks plain HTTP. Put it behind TLS before sending keys across a network.
227
355
 
228
- - It serves MCP at `POST/GET/DELETE /mcp` and liveness at `GET /healthz` (no auth): `curl http://127.0.0.1:8090/healthz`.
229
- - **`SPICRAWL_BASE_URL` defaults to `http://127.0.0.1:8080`** for the HTTP server (a Spicrawl API on the same host), not to the public API. Set it to `https://api.spicrawl.com` to front the public API.
230
- - Each caller authenticates with their own key as `Authorization: Bearer spicrawl_live_...`. The key is validated against the API before a session opens, and a session is pinned to the key that created it.
231
- - The server speaks plain HTTP. Put it behind TLS (a reverse proxy) before sending keys across a network.
356
+ Installed globally (`npm install -g @spicrawl/mcp`), the command is `spicrawl-mcp` (alias `spicrawl-mcp-server`), with `--http`, `--version` and `--help`.
232
357
 
233
- ## Development
358
+ ## FAQ
234
359
 
235
- ```bash
236
- git clone https://github.com/Spicrawl/mcp.git && cd mcp
237
- npm ci
238
- npm run typecheck
239
- npm test
240
- ```
360
+ ### Is there a hosted Spicrawl MCP server?
361
+ Yes: `https://mcp.spicrawl.com/mcp`, MCP Streamable HTTP, authenticated with `Authorization: Bearer <API key>`, so there is nothing to install.
362
+
363
+ ### Is Spicrawl free?
364
+ Spicrawl bills credits per successful request, and during the beta every organization gets one plan of 1,000 credits per month. Failed requests cost 0 credits; a cache hit is billed at the same price as the fetch that stored it. See [Credits](https://docs.spicrawl.com/credits).
365
+
366
+ ### Does it render JavaScript?
367
+ Yes. Set `render: true` on `spicrawl_scrape` to run the page in a browser, or `mode: "auto"` to let Spicrawl escalate from a plain fetch only when needed.
368
+
369
+ ### Does it work with Claude Code, Cursor and VS Code?
370
+ Yes, and with Claude Desktop, Windsurf, Gemini CLI, Codex and any client that speaks MCP Streamable HTTP with custom headers or launches stdio servers.
371
+
372
+ ### Can it crawl a whole website?
373
+ There is no single crawl tool. Scrape a page with `links: true` to get its links, then feed them to an open batch job (`open: true` and `spicrawl_batch_add_items`).
374
+
375
+ ### Does it support OAuth?
376
+ No. The server accepts Spicrawl API keys only; turn OAuth off for this server in clients that try it on a `401`.
377
+
378
+ ### Should I use the MCP server, the SDK or the CLI?
379
+ Use the MCP server when an AI agent should call Spicrawl as tools. Use [`@spicrawl/sdk`](https://github.com/Spicrawl/sdk) in your own TypeScript or JavaScript code, [`@spicrawl/cli`](https://github.com/Spicrawl/cli) in a terminal or shell script, and the [REST API](https://docs.spicrawl.com/quickstart) from any other language.
380
+
381
+ ## Troubleshooting
382
+
383
+ | Symptom | Fix |
384
+ |---|---|
385
+ | HTTP `401` or "Unauthorized: send your Spicrawl API key" | The header is missing, or the key is unknown, revoked or expired. Check that the variable is set in the environment the client started from. |
386
+ | The header contains a literal `${SPICRAWL_API_KEY}` | The client did not expand it. Use `${VAR}` in Claude Code, `${env:VAR}` in Cursor, `${input:id}` in VS Code, `bearer_token_env_var` in Codex. |
387
+ | HTTP `503` with `Retry-After` on connect | The server could not reach the API to check your key. Retry; your key is fine. |
388
+ | `ERR::AUTH::INSUFFICIENT_SCOPE` from a `spicrawl_usage*` tool | Grant the key the `read` scope in the dashboard. |
389
+ | `ERR::LIMIT::QUOTA_EXCEEDED` (HTTP 402) | The monthly credit allowance is used up. Not retryable until it resets. |
390
+ | "Unknown argument" on `spicrawl_scrape` | Use the tool's names: `render`, not `js_render`; `format`, not `response_format`. |
391
+ | Tools do not appear | Restart the client, and check the config key (`servers` in VS Code, `mcpServers` elsewhere). |
392
+ | `npx` not found (stdio) | Install Node.js 20 or later. |
393
+
394
+ `curl -i https://mcp.spicrawl.com/mcp` without a key answers `401`: the endpoint is reachable.
241
395
 
242
- `npm test` builds the package, then runs the `node:test` suite in `test/` against the real `dist/http.js`, pointed at a fake Spicrawl API on loopback. It needs no deployment, no network and no real API key, and finishes in a few seconds.
396
+ ## Related packages
243
397
 
244
- `npm run build` writes `dist/index.js` (the `spicrawl-mcp` command: stdio, or HTTP with `--http`) and `dist/http.js` (the HTTP server). Run the stdio server from a checkout with `SPICRAWL_API_KEY=... node dist/index.js`.
398
+ - [`@spicrawl/sdk`](https://github.com/Spicrawl/sdk): the official TypeScript SDK for the Spicrawl API.
399
+ - [`@spicrawl/cli`](https://github.com/Spicrawl/cli): the Spicrawl command-line interface.
400
+ - [Spicrawl/agent-plugins](https://github.com/Spicrawl/agent-plugins): plugins that bundle this server and the Spicrawl skill for Claude Code, Codex, Cursor, Gemini CLI and more.
245
401
 
246
- ## Security
402
+ ## Links
247
403
 
248
- See [SECURITY.md](SECURITY.md). Never commit an API key; if one leaks, revoke it at https://app.spicrawl.com.
404
+ - Documentation: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp)
405
+ - Docs for LLMs: [docs.spicrawl.com/llms.txt](https://docs.spicrawl.com/llms.txt)
406
+ - Dashboard and API keys: [app.spicrawl.com](https://app.spicrawl.com)
407
+ - Issues: [github.com/Spicrawl/mcp/issues](https://github.com/Spicrawl/mcp/issues)
408
+ - Security: [SECURITY.md](https://github.com/Spicrawl/mcp/blob/main/SECURITY.md). Never commit an API key; revoke a leaked one at [app.spicrawl.com](https://app.spicrawl.com).
249
409
 
250
410
  ## License
251
411
 
252
- [Apache-2.0](LICENSE)
412
+ [Apache-2.0](https://github.com/Spicrawl/mcp/blob/main/LICENSE)
package/package.json CHANGED
@@ -1,10 +1,11 @@
1
1
  {
2
2
  "name": "@spicrawl/mcp",
3
- "version": "0.1.0",
4
- "description": "Spicrawl MCP server: web scraping, structured data extraction and batch crawling as native tools for AI agents in Claude Code, Claude Desktop, Cursor, VS Code, Codex and any MCP client.",
3
+ "version": "0.1.1",
4
+ "mcpName": "io.github.Spicrawl/mcp",
5
+ "description": "The Spicrawl MCP server is a Model Context Protocol server that gives AI agents in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex web scraping to Markdown, structured data extraction, screenshots, browser sessions and batch scraping as 25 native tools.",
5
6
  "license": "Apache-2.0",
6
7
  "author": "Spicrawl",
7
- "homepage": "https://docs.spicrawl.com/docs/agents/mcp",
8
+ "homepage": "https://docs.spicrawl.com/agents/mcp",
8
9
  "repository": {
9
10
  "type": "git",
10
11
  "url": "git+https://github.com/Spicrawl/mcp.git"
@@ -18,19 +19,26 @@
18
19
  "mcp-server",
19
20
  "model-context-protocol",
20
21
  "web-scraping",
21
- "web-scraping-api",
22
- "scraper",
23
- "crawler",
22
+ "web-scraper",
23
+ "web-crawler",
24
24
  "html-to-markdown",
25
25
  "markdown",
26
+ "data-extraction",
27
+ "structured-data",
28
+ "screenshot",
29
+ "headless-browser",
30
+ "batch-scraping",
26
31
  "llm",
27
32
  "ai-agents",
33
+ "rag",
28
34
  "claude",
29
35
  "claude-code",
36
+ "claude-desktop",
30
37
  "cursor",
31
- "data-extraction",
32
- "headless-browser",
33
- "batch-scraping"
38
+ "vscode",
39
+ "codex",
40
+ "gemini-cli",
41
+ "windsurf"
34
42
  ],
35
43
  "type": "module",
36
44
  "bin": {