@spicrawl/mcp 0.1.0 → 0.1.2

package/CHANGELOG.md

@@ -3,6 +3,20 @@
 All notable changes to `@spicrawl/mcp` are documented here. This project follows
 [Semantic Versioning](https://semver.org/).
 
+## 0.1.2 — 2026-10-01
+
+- Every tool's MCP annotations are now accurate, and a tool without a `title` and explicit `readOnlyHint`, `destructiveHint` and `openWorldHint` no longer compiles. `spicrawl_scrape` is no longer read-only, because `method` and `actions` can submit forms. `spicrawl_session_release` is destructive, because it purges cookies. Tools confined to your own account are no longer open-world. `test/annotations.test.mjs` pins each value.
+- `SPICRAWL_MCP_HIDE_UNAVAILABLE` (off by default) leaves out what is announced but not available yet: `spicrawl_browser_connect_url` is not registered and every "Coming soon" argument is dropped from the schemas.
+- Tool descriptions no longer name another product. The scrape description states billing and side effects. A printed PDF's JSON no longer carries the request id.
+- A ChatGPT endpoint on the HTTP server, off unless `SPICRAWL_CHATGPT_ENABLED` is on: stateless `POST /chatgpt/mcp` that takes OAuth access tokens only (introspected at `<SPICRAWL_OAUTH_ISSUER>/api/oauth/introspect`, positive answers cached at most 30 s), protected-resource metadata at `/.well-known/oauth-protected-resource/chatgpt/mcp` (and at the root with `SPICRAWL_CHATGPT_ROOT_PRM`), the OpenAI domain challenge at `/.well-known/openai-apps-challenge`, and a restricted six-tool profile with OAuth `securitySchemes` and sanitized results. `/mcp` is unchanged.
+
+## 0.1.1 — 2026-10-01
+
+Documentation and package metadata only; the server and its 25 tools are unchanged.
+
+- README rewritten for npmjs.com: hosted endpoint first, one-click install links for Cursor and VS Code, per-client setup for Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex, tool tables by group, a "which tool to use" guide, environment variables, self-hosting, FAQ and troubleshooting. Every link is absolute so it works on npm.
+- `package.json`: `mcpName` (`io.github.Spicrawl/mcp`) for the official MCP registry, a new `description` and `keywords`, and `homepage` now points at https://docs.spicrawl.com/agents/mcp.
+
 ## 0.1.0 — 2026-10-01
 
 First public release of the Spicrawl MCP server, published to npm as `@spicrawl/mcp` from https://github.com/Spicrawl/mcp with an npm provenance attestation.

package/README.md

@@ -1,37 +1,145 @@
-# Spicrawl MCP Server: Web Scraping Tools for AI Agents
+# Spicrawl MCP Server
 
-The official [Model Context Protocol](https://modelcontextprotocol.io) server for [Spicrawl](https://spicrawl.com), the web scraping API for AI agents and LLMs. Connect it to Claude Code, Claude Desktop, Cursor, VS Code, Codex or any MCP client, and your agent can scrape pages to Markdown, extract structured data, run batch jobs over thousands of URLs, keep browser sessions and read the Spicrawl docs, all as native tools.
+The Spicrawl MCP server is a Model Context Protocol server that gives AI agents in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex web scraping to Markdown, structured data extraction, screenshots, browser sessions and batch scraping as 25 native tools.
 
 [![npm version](https://img.shields.io/npm/v/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
-[![license](https://img.shields.io/npm/l/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
+[![npm downloads](https://img.shields.io/npm/dm/@spicrawl/mcp.svg)](https://www.npmjs.com/package/@spicrawl/mcp)
+[![license](https://img.shields.io/npm/l/@spicrawl/mcp.svg)](https://github.com/Spicrawl/mcp/blob/main/LICENSE)
 [![CI](https://github.com/Spicrawl/mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/Spicrawl/mcp/actions/workflows/ci.yml)
-[![docs](https://img.shields.io/badge/docs-docs.spicrawl.com-blue.svg)](https://docs.spicrawl.com/agents/mcp)
 
-**[Docs](https://docs.spicrawl.com/agents/mcp)** · **[Get an API key](https://app.spicrawl.com/signup)** · **[GitHub](https://github.com/Spicrawl/mcp)** · **[SDK](https://github.com/Spicrawl/sdk)** · **[CLI](https://github.com/Spicrawl/cli)** · **[Changelog](https://github.com/Spicrawl/mcp/blob/main/CHANGELOG.md)**
+**[Docs](https://docs.spicrawl.com/agents/mcp)** · **[Get an API key](https://app.spicrawl.com/signup)** · **[TypeScript SDK](https://github.com/Spicrawl/sdk)** · **[CLI](https://github.com/Spicrawl/cli)** · **[Agent plugins](https://github.com/Spicrawl/agent-plugins)** · **[Changelog](https://github.com/Spicrawl/mcp/blob/main/CHANGELOG.md)**
 
-## Two ways to connect
+<p>
+  <a href="https://cursor.com/en/install-mcp?name=spicrawl&config=eyJ1cmwiOiJodHRwczovL21jcC5zcGljcmF3bC5jb20vbWNwIiwiaGVhZGVycyI6eyJBdXRob3JpemF0aW9uIjoiQmVhcmVyICR7ZW52OlNQSUNSQVdMX0FQSV9LRVl9In19"><img src="https://cursor.com/deeplink/mcp-install-dark.svg" alt="Add Spicrawl MCP server to Cursor" height="32" /></a>
+  <a href="https://vscode.dev/redirect/mcp/install?name=spicrawl&inputs=%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22spicrawl-api-key%22%2C%22description%22%3A%22Spicrawl%20API%20key%22%2C%22password%22%3Atrue%7D%5D&config=%7B%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fmcp.spicrawl.com%2Fmcp%22%2C%22headers%22%3A%7B%22Authorization%22%3A%22Bearer%20%24%7Binput%3Aspicrawl-api-key%7D%22%7D%7D"><img src="https://img.shields.io/badge/VS_Code-Install_Spicrawl_MCP-0098FF?style=for-the-badge&logo=visualstudiocode&logoColor=white" alt="Install Spicrawl MCP server in VS Code" height="32" /></a>
+</p>
 
-| | Hosted endpoint | Local (stdio) |
+## Quickstart
+
+1. Create an API key at [app.spicrawl.com](https://app.spicrawl.com/signup). Keys start with `spicrawl_live_` or `spicrawl_test_`.
+2. Connect the **hosted endpoint**, `https://mcp.spicrawl.com/mcp` (MCP Streamable HTTP, nothing to install):
+
+   ```bash
+   export SPICRAWL_API_KEY=spicrawl_live_...
+   claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
+     --header "Authorization: Bearer $SPICRAWL_API_KEY"
+   ```
+
+   Codex: `codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY`
+
+3. Or run it **locally over stdio** (Node.js 20+), for clients that only launch local servers:
+
+   ```bash
+   claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
+   ```
+
+4. Ask your agent: *"Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price."* It calls `spicrawl_scrape` and gets the page back as Markdown.
+
+Both transports serve the same 25 `spicrawl_*` tools, and every call runs under your key with the same scopes, limits and credits as a direct API call.
+
+## What it does
+
+- **Scrape any URL to Markdown**, text, HTML, a JSON envelope or a printed PDF with one tool, `spicrawl_scrape`. Markdown is the default and strips the page to its main content.
+- **Render JavaScript** with `render: true` (the `obscura` browser engine, 3 credits) or pin real Chromium with `engine: "chromium"` (8 credits). A plain fetch costs 1 credit, and `mode: "auto"` escalates fetch to browser and bills only the rung that worked ([price table](https://docs.spicrawl.com/credits)).
+- **Extract structured data** with a CSS/XPath selector map (`extract`) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (`autoparse`).
+- **Drive the page** before capture with 8 typed browser actions: `click`, `fill`, `wait_for`, `wait_for_navigation`, `scroll` (including infinite scroll), `select`, `evaluate` and `screenshot`.
+- **Return screenshots and PDFs the model can see**: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as `application/pdf` resource blocks (up to 10 MB).
+- **Batch scrape thousands of URLs** as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open for crawling.
+- **Keep a logged-in browser** with persistent sessions: cookies, storage and a pinned engine reused across scrapes.
+- **Answer "what did that cost?"** from usage and request-log tools, and **read the Spicrawl docs** from inside the agent.
+
+## Which tool to use
+
+- If you need one page's content or data now, use `spicrawl_scrape`.
+- If the page is blank or a skeleton without JavaScript, call `spicrawl_scrape` again with `render: true`.
+- If you know the page's structure, use `extract` with CSS selectors; if you don't, try `autoparse`.
+- If you have more than about 20 URLs, use `spicrawl_batch_submit`, then `spicrawl_batch_status` and `spicrawl_batch_results`.
+- If you discover URLs as you go (crawling), submit with `open: true`, append with `spicrawl_batch_add_items` and finish with `spicrawl_batch_close`.
+- If you must log in first, use `spicrawl_session_create` and pass its `id` as `session_id` to `spicrawl_scrape`.
+- If a call failed and you need to know why, use `spicrawl_request_get` with the request id, or `spicrawl_requests_list` with `only_errors`.
+- If you are unsure of a parameter or an error code, use `spicrawl_docs_search` before guessing.
+
+## Tools
+
+25 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a `title` and explicit `readOnlyHint`, `destructiveHint` and `openWorldHint` annotations, so a client knows which calls to confirm: `spicrawl_scrape` is not read-only (its `method` and `actions` can submit forms), and `spicrawl_batch_cancel`, `spicrawl_session_release` and `spicrawl_session_delete` are destructive. Full argument reference: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp#tools).
+
+### Scrape a website to Markdown
+
+| Tool | What it does | Key parameters |
 |---|---|---|
-| What | `https://mcp.spicrawl.com/mcp`, MCP Streamable HTTP | `npx -y @spicrawl/mcp`, a subprocess of your MCP client |
-| Install | nothing | Node.js 20 or later |
-| Auth | `Authorization: Bearer <API key>` header | `SPICRAWL_API_KEY` environment variable |
-| Use when | your client supports remote MCP servers (most do) | your client only launches local servers, or you want to pin a version |
+| `spicrawl_scrape` | Retrieve one URL as `markdown` (default), `text`, `html`, `json` or `pdf`, with optional rendering, extraction, links, screenshots and browser actions. | `url`, `format`, `render`, `engine`, `mode`, `main_content_only`, `include_tags`, `exclude_tags`, `extract`, `autoparse`, `links`, `screenshot`, `actions`, `wait_for`, `session_id`, `proxy`, `cache`, `max_cost` |
 
-Both serve the same 25 `spicrawl_*` tools. [Create an API key](https://app.spicrawl.com/signup) in the dashboard first; keys start with `spicrawl_live_` or `spicrawl_test_`.
+### Batch scrape many URLs
 
-## Hosted endpoint
+| Tool | What it does | Key parameters |
+|---|---|---|
+| `spicrawl_batch_submit` | Queue many URLs as one async job with shared settings; returns the job with `estimated_credits`. | `urls` or `items`, `render`, `format`, `open`, `max_attempts`, `credit_budget`, `max_cost` |
+| `spicrawl_batch_list` | List the project's batch jobs, newest first. | `status`, `limit` (max 200), `cursor` |
+| `spicrawl_batch_status` | One job's status and progress. | `job_id` |
+| `spicrawl_batch_results` | A page of finished items; works while the job runs. | `job_id`, `status`, `limit` (max 5000), `cursor` |
+| `spicrawl_batch_task_content` | The full payload of one item. | `job_id`, `seq` |
+| `spicrawl_batch_cancel` | Cancel a job; finished items keep their results. | `job_id` |
+| `spicrawl_batch_retry` | Re-queue every failed item. | `job_id` |
+| `spicrawl_batch_add_items` | Append URLs to a job submitted with `open: true`. | `job_id`, `urls` or `items` |
+| `spicrawl_batch_close` | Stop an open job accepting items so it can complete. | `job_id` |
+
+### Browser sessions
+
+| Tool | What it does | Key parameters |
+|---|---|---|
+| `spicrawl_session_create` | Create a persistent browser identity (cookies, storage, pinned engine). | `engine`, `ttl_seconds`, `session_context` |
+| `spicrawl_session_list` | List sessions, newest first. | `status`, `engine`, `limit`, `cursor` |
+| `spicrawl_session_get` | One session's status, engine, exit, usage and expiry. | `session_id` |
+| `spicrawl_session_context` | Dump a live session's cookies and storage (contains secrets). | `session_id` |
+| `spicrawl_session_release` | End a session; its record stays as `released`. | `session_id` |
+| `spicrawl_session_delete` | Permanently delete a session and its state. | `session_id` |
+
+### Usage and request history
+
+| Tool | What it does | Key parameters |
+|---|---|---|
+| `spicrawl_usage` | Billing-grade usage over a date window (`read` scope). | `from`, `to`, `group_by`, `metrics` |
+| `spicrawl_usage_summary` | Current-period totals (`read` scope). | none |
+| `spicrawl_usage_reconciliation` | One day's drift between the request log and the billed rollup (`read` scope). | `day` |
+| `spicrawl_requests_list` | Recent API calls with status, error code, engine, latency and credits. | `only_errors`, `status`, `limit`, `all_projects` |
+| `spicrawl_request_get` | The full log record of one call. | `id` |
+
+### Docs
+
+| Tool | What it does | Key parameters |
+|---|---|---|
+| `spicrawl_docs_search` | Search the Spicrawl docs; an error code also links to its entry. | `query`, `limit` (1-20, default 8) |
+| `spicrawl_docs_read` | Read one docs page as Markdown (truncated over 60,000 characters). | `path` |
+| `spicrawl_docs_index` | The docs' `llms.txt`: every page with its title and `.md` link. | none |
+
+### Coming soon
 
-Each tool call runs under your key, exactly as if you had called the API yourself, and a session is bound to the key that opened it. The key is checked when a session opens: an unknown or revoked key gets `401`, and if the API cannot be reached to check it the endpoint answers `503` with `Retry-After` (retry rather than rotating the key).
+`spicrawl_browser_connect_url` will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from Puppeteer or Playwright; remote browsers are not available yet. The schemas also accept `ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country` and `sticky_key` ahead of launch. Agents should not send them yet. To list only what works today, set `SPICRAWL_MCP_HIDE_UNAVAILABLE=1` (see [Environment variables](https://github.com/Spicrawl/mcp#environment-variables)).
 
-**Claude Code:**
+## Example prompts
+
+- *"Scrape https://example.com/pricing as markdown and summarise the plans."*
+- *"Get the product name, price and stock status from this URL."* (`spicrawl_scrape` with `extract` or `autoparse`)
+- *"Take a full-page screenshot of https://example.com after the cookie banner is gone."* (`render`, `actions`, `screenshot_fullpage`)
+- *"Scrape these 500 URLs and give me each page title."* (`spicrawl_batch_submit`, then `spicrawl_batch_results`)
+- *"Why did my last scrape fail?"* (`spicrawl_requests_list` with `only_errors`, then `spicrawl_request_get`)
+- *"How many credits have I used this month?"* (`spicrawl_usage_summary`)
+
+## Set up your MCP client
+
+Put your key in `SPICRAWL_API_KEY` and reference it from the config where the client allows, so the key never lands in a committed file.
+
+<details>
+<summary><b>Claude Code</b></summary>
+
+Hosted endpoint, stored for you only (add `--scope user` for every project):
 
 ```bash
 claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
-  --header "Authorization: Bearer spicrawl_live_..."
+  --header "Authorization: Bearer $SPICRAWL_API_KEY"
 ```
 
-or in a project's `.mcp.json`:
+To share it with a team, commit `.mcp.json` at the project root. Claude Code expands `${VAR}`, so each teammate's own key is used:
 
 ```json
 {
@@ -39,26 +147,68 @@ or in a project's `.mcp.json`:
     "spicrawl": {
       "type": "http",
       "url": "https://mcp.spicrawl.com/mcp",
-      "headers": { "Authorization": "Bearer spicrawl_live_..." }
+      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
+    }
+  }
+}
+```
+
+Or install the plugin, which bundles the MCP server and the Spicrawl agent skill:
+
+```bash
+claude plugin marketplace add Spicrawl/agent-plugins
+claude plugin install spicrawl@spicrawl-plugins
+```
+
+Check with `claude mcp list`, or `/mcp` in a session. More: [docs.spicrawl.com/agents/claude-code](https://docs.spicrawl.com/agents/claude-code).
+
+</details>
+
+<details>
+<summary><b>Claude Desktop</b></summary>
+
+Claude Desktop's `claude_desktop_config.json` starts local (stdio) servers only, and custom connectors cannot send an `Authorization` header, so run the package locally. The file is at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS and `%APPDATA%\Claude\claude_desktop_config.json` on Windows:
+
+```json
+{
+  "mcpServers": {
+    "spicrawl": {
+      "command": "npx",
+      "args": ["-y", "@spicrawl/mcp"],
+      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
     }
   }
 }
 ```
 
-**Cursor** (`.cursor/mcp.json` or `~/.cursor/mcp.json`):
+Restart Claude Desktop after saving.
+
+</details>
+
+<details>
+<summary><b>Cursor</b></summary>
+
+Use the **Add to Cursor** button above, or add this to `.cursor/mcp.json` in the project (or `~/.cursor/mcp.json` for every project). Cursor resolves `${env:NAME}` in `url` and `headers`, so start Cursor from an environment where `SPICRAWL_API_KEY` is set:
 
 ```json
 {
   "mcpServers": {
     "spicrawl": {
       "url": "https://mcp.spicrawl.com/mcp",
-      "headers": { "Authorization": "Bearer spicrawl_live_..." }
+      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
     }
   }
 }
 ```
 
-**VS Code** (`.vscode/mcp.json`; VS Code uses the `servers` key). The `inputs` entry makes VS Code ask for the key once and store it, so the file is safe to commit:
+More: [docs.spicrawl.com/agents/cursor](https://docs.spicrawl.com/agents/cursor).
+
+</details>
+
+<details>
+<summary><b>VS Code (GitHub Copilot)</b></summary>
+
+Use the **Install in VS Code** button above, or add this to `.vscode/mcp.json`. VS Code uses the `servers` key, and the input variable makes it ask for the key once and store it securely, so the file is safe to commit:
 
 ```json
 {
@@ -75,66 +225,80 @@ or in a project's `.mcp.json`:
 }
 ```
 
-**Codex** reads the key from the environment each time it connects, so the key never lands in a file:
+VS Code does not expand `${env:...}` inside `headers`, so use the input variable. More: [docs.spicrawl.com/agents/github-copilot](https://docs.spicrawl.com/agents/github-copilot).
+
+</details>
+
+<details>
+<summary><b>Windsurf (Devin Desktop)</b></summary>
+
+Windsurf is now named Devin Desktop. For the default **Devin Local** agent, add the server in local scope (saved to the gitignored `.devin/mcp_config.local.json`):
 
 ```bash
-export SPICRAWL_API_KEY=spicrawl_live_...
-codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY
+devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"
 ```
 
-The equivalent `~/.codex/config.toml`:
+For the legacy **Cascade** agent, edit `~/.config/devin/mcp_config.json` (`%APPDATA%\devin\mcp_config.json` on Windows). Cascade reads the URL from `serverUrl` and replaces `${env:...}`:
 
-```toml
-[mcp_servers.spicrawl]
-url = "https://mcp.spicrawl.com/mcp"
-bearer_token_env_var = "SPICRAWL_API_KEY"
+```json
+{
+  "mcpServers": {
+    "spicrawl": {
+      "serverUrl": "https://mcp.spicrawl.com/mcp",
+      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
+    }
+  }
+}
 ```
 
-**Claude Desktop** only launches local servers from `claude_desktop_config.json`, and custom connectors cannot send an `Authorization` header, so use the [local server](#local-server-stdio) below.
+More: [docs.spicrawl.com/agents/windsurf](https://docs.spicrawl.com/agents/windsurf).
 
-## Local server (stdio)
+</details>
 
-`npx -y @spicrawl/mcp` downloads the package and runs the stdio server. The client starts it and passes your key in its environment.
+<details>
+<summary><b>Gemini CLI</b></summary>
 
-**Claude Code:**
+Gemini CLI strips variables whose names contain `KEY`, `TOKEN` or `AUTH` before expanding headers, so export the key under another name:
 
 ```bash
-claude mcp add spicrawl --env SPICRAWL_API_KEY=spicrawl_live_... -- npx -y @spicrawl/mcp
+export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"
 ```
 
-**Claude Desktop** (`claude_desktop_config.json`) and **Cursor** (`.cursor/mcp.json`):
+Then add to `~/.gemini/settings.json` (or `.gemini/settings.json` for one project). Streamable HTTP uses the `httpUrl` key:
 
 ```json
 {
   "mcpServers": {
     "spicrawl": {
-      "command": "npx",
-      "args": ["-y", "@spicrawl/mcp"],
-      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
+      "httpUrl": "https://mcp.spicrawl.com/mcp",
+      "headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
     }
   }
 }
 ```
 
-**VS Code** (`.vscode/mcp.json`):
+Or: `gemini mcp add --scope user --transport http --header 'Authorization: Bearer $SPICRAWL_MCP_BEARER' spicrawl https://mcp.spicrawl.com/mcp`. Run `/mcp` to check. More: [docs.spicrawl.com/agents/gemini-cli](https://docs.spicrawl.com/agents/gemini-cli).
 
-```json
-{
-  "inputs": [
-    { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
-  ],
-  "servers": {
-    "spicrawl": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@spicrawl/mcp"],
-      "env": { "SPICRAWL_API_KEY": "${input:spicrawl-api-key}" }
-    }
-  }
-}
+</details>
+
+<details>
+<summary><b>Codex</b></summary>
+
+Codex reads the key from the environment each time it connects, so the key never lands in a file:
+
+```bash
+codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY
 ```
 
-**Codex** (`~/.codex/config.toml`):
+The equivalent `~/.codex/config.toml` (or `.codex/config.toml` in a trusted project):
+
+```toml
+[mcp_servers.spicrawl]
+url = "https://mcp.spicrawl.com/mcp"
+bearer_token_env_var = "SPICRAWL_API_KEY"
+```
+
+Local stdio instead:
 
 ```toml
 [mcp_servers.spicrawl]
@@ -143,110 +307,146 @@ args = ["-y", "@spicrawl/mcp"]
 env = { SPICRAWL_API_KEY = "spicrawl_live_..." }
 ```
 
-Restart the client; the `spicrawl_*` tools appear. To install it once instead of fetching it on each start, run `npm install -g @spicrawl/mcp` and use the command `spicrawl-mcp` (`spicrawl-mcp-server` is an alias).
+More: [docs.spicrawl.com/agents/codex](https://docs.spicrawl.com/agents/codex).
+
+</details>
+
+<details>
+<summary><b>Any other MCP client</b></summary>
+
+| Setting | Value |
+|---|---|
+| Transport | Streamable HTTP |
+| URL | `https://mcp.spicrawl.com/mcp` |
+| Header | `Authorization: Bearer <your key>` |
+
+A stdio-only client runs `npx -y @spicrawl/mcp` with `SPICRAWL_API_KEY` in its environment. The server accepts API keys only, not OAuth. Setup pages for more clients (OpenCode, Zed, JetBrains, Cline, Goose and others): [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp).
 
-### Environment variables
+</details>
 
-| Variable | Required | Default | Notes |
+## Environment variables
+
+| Variable | Default | Used by | Purpose |
 |---|---|---|---|
-| `SPICRAWL_API_KEY` | yes (stdio) | — | Your Spicrawl API key. The HTTP server ignores it and uses each caller's bearer token instead. |
-| `SPICRAWL_BASE_URL` | no | `https://api.spicrawl.com` (stdio), `http://127.0.0.1:8080` (HTTP server) | The Spicrawl API base URL. Set it for a self-hosted or local stack, and always when you run the HTTP server against the public API. |
-| `SPICRAWL_PUBLIC_BASE_URL` | no | `SPICRAWL_BASE_URL` | The API as the end user reaches it, used only for URLs handed back (`spicrawl_browser_connect_url`). Set it when the server talks to the API over a private address. |
-| `SPICRAWL_DOCS_URL` | no | see below | The docs base URL, no trailing slash. The docs tools read `<base>/api/search`, `<base>/<page>.md` and `<base>/llms.txt`. The docs are public: your API key is never sent there. |
-| `SPICRAWL_DOCS_HOST` | no | — | Legacy: an origin whose docs are at `/docs`. Read only when `SPICRAWL_DOCS_URL` is unset. |
-| `SPICRAWL_MCP_ADDR` | no | `127.0.0.1:8090` | HTTP server only: the listen address, `host:port`. |
-| `SPICRAWL_MCP_SESSION_IDLE_MS` | no | `1800000` (30 min) | HTTP server only: idle sessions are dropped after this long. |
+| `SPICRAWL_API_KEY` | none (required) | stdio | Your API key. The HTTP server ignores it and uses each caller's bearer token. |
+| `SPICRAWL_BASE_URL` | `https://api.spicrawl.com` (stdio), `http://127.0.0.1:8080` (HTTP) | both | The Spicrawl API the tools call. |
+| `SPICRAWL_PUBLIC_BASE_URL` | `SPICRAWL_BASE_URL` | both | The API as the end user reaches it, used only in URLs handed back. |
+| `SPICRAWL_DOCS_URL` | `https://docs.spicrawl.com` | both | Docs base URL for the docs tools. Your key is never sent there. |
+| `SPICRAWL_DOCS_HOST` | none | both | Legacy: an origin whose docs are at `/docs`. Read only when `SPICRAWL_DOCS_URL` is unset. |
+| `SPICRAWL_MCP_ADDR` | `127.0.0.1:8090` | HTTP | Listen address, `host:port`. |
+| `SPICRAWL_MCP_SESSION_IDLE_MS` | `1800000` (30 min) | HTTP | Idle MCP sessions are dropped after this long. |
+| `SPICRAWL_MCP_HIDE_UNAVAILABLE` | off | both | `1`, `true`, `yes` or `on` leaves out what is announced but not available yet: `spicrawl_browser_connect_url` is not registered (24 tools) and every argument marked "Coming soon" is dropped from the schemas (`ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country`, `sticky_key`, and the session `rotate_ip`, `region_pool` and `fingerprint`). The `camoufox` engine value stays in the `engine` enums. Read when the server starts a session, so restart the process after changing it. |
+| `SPICRAWL_CHATGPT_ENABLED` | off | HTTP | `1`, `true`, `yes` or `on` turns on the [ChatGPT endpoint](#chatgpt-endpoint-oauth). Off, `/chatgpt/mcp`, its metadata and the OpenAI challenge all answer `404`. |
+| `SPICRAWL_CHATGPT_RESOURCE` | `https://mcp.spicrawl.com/chatgpt/mcp` | HTTP | The OAuth resource identifier. Tokens must be issued for exactly this `aud`; the metadata is published at `/.well-known/oauth-protected-resource` + its path. |
+| `SPICRAWL_OAUTH_ISSUER` | `https://app.spicrawl.com` | HTTP | The authorization server, listed in the metadata. Tokens are introspected at `<issuer>/api/oauth/introspect`. |
+| `SPICRAWL_OAUTH_INTROSPECT_SECRET` | none | HTTP | This server's bearer secret at the introspection endpoint. Unset, every token is answered `503`. |
+| `SPICRAWL_CHATGPT_ROOT_PRM` | off | HTTP | Also serve the metadata at the root `/.well-known/oauth-protected-resource`. Off by default because `/mcp` takes API keys and must not advertise OAuth. |
+| `SPICRAWL_OPENAI_APPS_CHALLENGE` | none | HTTP | The OpenAI domain-verification token served at `/.well-known/openai-apps-challenge`. |
+| `SPICRAWL_OPENAI_APPS_CHALLENGE_FILE` | none | HTTP | A file holding that token (surrounding whitespace and the trailing newline are dropped). Read when `SPICRAWL_OPENAI_APPS_CHALLENGE` is unset. |
+
+With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at `<SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs`.
+
+## Self-host the MCP server over HTTP
+
+The hosted endpoint runs this package's Streamable HTTP server. To run your own:
+
+```bash
+SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
+SPICRAWL_BASE_URL=https://api.spicrawl.com \
+npx -y @spicrawl/mcp --http
+```
 
-Without `SPICRAWL_DOCS_URL` or `SPICRAWL_DOCS_HOST`, the docs tools read a self-hosted API's own docs (`SPICRAWL_PUBLIC_BASE_URL`, else `SPICRAWL_BASE_URL`, plus `/docs`), and otherwise `https://docs.spicrawl.com`.
+- `SPICRAWL_BASE_URL` **defaults to `http://127.0.0.1:8080`** in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it to `https://api.spicrawl.com` to front the public API.
+- MCP is served at `POST`/`GET`/`DELETE /mcp`; liveness at `GET /healthz` (no auth).
+- Each caller sends their own key as `Authorization: Bearer spicrawl_live_...`. The key is checked against the API before a session opens (`401` if refused, `503` with `Retry-After` if the API cannot be reached), and a session is pinned to the key that opened it.
+- The server speaks plain HTTP. Put it behind TLS before sending keys across a network.
 
-## Tools
+### ChatGPT endpoint (OAuth)
 
-| Tool | What it does |
+The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless `SPICRAWL_CHATGPT_ENABLED` is on, and `/mcp` is unchanged either way.
+
+| Route | What it does |
 |---|---|
-| **Scrape** | |
-| `spicrawl_scrape` | Retrieve a URL as markdown/text/html/json, or print it to a PDF. Optional browser render, selector or AI extraction, links, screenshot, DOM scoping, proxy control, caching, browser actions, network capture. |
-| **Batch** | |
-| `spicrawl_batch_submit` | Scrape many URLs as one async job with shared settings. Output is markdown unless `format` says otherwise, the same default as `spicrawl_scrape`. Returns the job object. |
-| `spicrawl_batch_list` | List the project's batch jobs, newest first, with status and progress. |
-| `spicrawl_batch_status` | Poll one batch job's status and progress. |
-| `spicrawl_batch_results` | Read a page of a job's finished items (content, or a `result_url` API content path per item). |
-| `spicrawl_batch_task_content` | Read the full payload of one item by its `seq`. |
-| `spicrawl_batch_cancel` | Cancel a job: unstarted items never run, finished ones keep their results. |
-| `spicrawl_batch_retry` | Re-queue every failed item of a job. |
-| `spicrawl_batch_add_items` | Append URLs to a job submitted with `open: true`. |
-| `spicrawl_batch_close` | Stop an open job accepting items so it can complete. |
-| **Sessions** | |
-| `spicrawl_session_create` | Create a persistent browser identity (cookies, storage, pinned engine, sticky exit IP) to reuse across scrapes. |
-| `spicrawl_session_list` | List the project's sessions, newest first. |
-| `spicrawl_session_get` | One session's metadata (status, engine, exit, usage, expiry). |
-| `spicrawl_session_context` | Dump a live session's stored state (cookies, storage). It contains secrets, and can clone a session. |
-| `spicrawl_session_release` | End a session; its record stays visible as `released`. |
-| `spicrawl_session_delete` | Permanently delete a session and its stored state. |
-| **Usage and request history** | |
-| `spicrawl_usage` | Billing-grade usage over a date window, grouped by day, project, engine or feature (needs the `read` scope). |
-| `spicrawl_usage_summary` | Current-period usage totals (needs the `read` scope). |
-| `spicrawl_usage_reconciliation` | One day's drift between the request log and the billed rollup (needs the `read` scope). |
-| `spicrawl_requests_list` | Recent API calls in the key's project with status, errors, engine, proxy, latency and credits; `all_projects: true` lists every project in the organization (needs the `read` scope). |
-| `spicrawl_request_get` | The full log record of one call by its request id, for debugging a failed or slow scrape. |
-| **Browser** | |
-| `spicrawl_browser_connect_url` | Coming soon: remote browsers are not available yet. Mints a single-use CDP WebSocket URL for driving Spicrawl's remote browser from your own Puppeteer or Playwright. |
-| **Docs** | |
-| `spicrawl_docs_search` | Search the Spicrawl docs (`query`, optional `limit` 1-20, default 8). Results are grouped by page, with matching sections, snippets and the `.md` URL to read next. An error code (`ERR::FAMILY::NAME`) also gets a link to its entry on the errors page. |
-| `spicrawl_docs_read` | Read one docs page as Markdown by path (`guides/anti-bot`) or docs URL, optionally with `#anchor`. Pages over 60k characters are truncated and say so; other hosts and `..` are refused. |
-| `spicrawl_docs_index` | The docs' `llms.txt`: every page with its title, description and `.md` link. |
-
-See the [MCP server docs](https://docs.spicrawl.com/agents/mcp) for each tool's arguments.
+| `POST /chatgpt/mcp` | MCP Streamable HTTP, **stateless**: no `Mcp-Session-Id`, a fresh server per request, JSON responses. Other methods answer `405`. |
+| `GET /.well-known/oauth-protected-resource/chatgpt/mcp` | Protected-resource metadata (RFC 9728): `resource`, `authorization_servers` (the issuer), `scopes_supported: ["scrape","batch"]`, `resource_documentation`. |
+| `GET /.well-known/oauth-protected-resource` | The same document, only with `SPICRAWL_CHATGPT_ROOT_PRM`. |
+| `GET /.well-known/openai-apps-challenge` | The verification token as `text/plain`, exact bytes, no trailing newline; `404` when none is configured. |
 
-## Example prompts
+Authentication takes OAuth access tokens (`spicrawl_oat_…`) only:
 
-Once connected, an agent can act on requests like:
+- No token: `401` with `WWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch"`.
+- A bad, inactive, expired or wrong-audience token, or an API key (`spicrawl_live_…`/`spicrawl_test_…`, never accepted here): the same `401` plus `error="invalid_token", error_description="…"`.
+- A tool call the token's scopes do not cover: `403` with `error="insufficient_scope"`. `spicrawl_scrape` needs `scrape`, the batch tools need `batch`, the docs tools need only a valid token.
+- The authorization server unreachable or refusing this server's secret: `503` with `Retry-After`.
 
-- *"Scrape https://example.com/pricing as markdown and summarise the plans."*
-- *"Get the product name, price and stock status from this URL."* The agent calls `spicrawl_scrape` with `autoparse` or an `extract` map.
-- *"Scrape these 50 URLs and give me the titles."* The agent calls `spicrawl_batch_submit`, then `spicrawl_batch_status` and `spicrawl_batch_results`.
+Each token is checked with `POST <issuer>/api/oauth/introspect` (`Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>`, form body `token=…`). Only `active: true` with `aud` equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its `exp`. Tools run upstream under the `api_key` the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose `_meta["mcp/www_authenticate"]` carries a fresh `invalid_token` challenge, and the token is dropped from the cache.
 
-## Behaviour worth knowing
+The endpoint lists six tools, each with explicit `readOnlyHint`/`destructiveHint`/`openWorldHint`, a title, and `securitySchemes: [{type: "oauth2", scopes: [...]}]` (also in `_meta.securitySchemes`):
 
-- Tools reject unknown arguments rather than ignoring them. A misspelt or unsupported field (e.g. `js_render` on `spicrawl_scrape`, whose argument is `render`) fails the call with an error naming it, so a setting the agent believes it applied is never dropped.
-- Errors carry the platform's own code and guidance (e.g. `ERR::UPSTREAM::CHALLENGE` when a site served a bot wall), so the agent can decide whether to retry or change parameters.
-- `spicrawl_scrape` screenshots come back as MCP image content blocks the model can see. In the JSON, each `screenshots` entry keeps its metadata but its base64 is replaced by a pointer to its block. An image over 5 MB of base64 is not attached, and its entry says how to get a smaller one.
-- `spicrawl_scrape` with `format: "pdf"` (plus `render` or `engine: "chromium"`) returns the printed page as an MCP resource block (`application/pdf`). A PDF over 10 MB of base64 is not attached; `pdf.not_attached` says so.
-- Coming soon, and marked so in the tool descriptions: `ai_extract`, stealth mode (`stealth`, engine `camoufox`), `extract_preset`, the managed proxy pool (`premium_proxy`, `proxy_country`, `sticky_key`, session `rotate_ip`/`region_pool`) and `spicrawl_browser_connect_url`. The schemas already accept them so nothing changes the day they ship; until then agents should not send them.
+| Tool | Scopes | Arguments |
+|---|---|---|
+| `spicrawl_scrape` | `scrape` | `url` (http/https only), `format` (`markdown`/`text`/`html`/`json`), `render`, `main_content_only`, `include_tags`, `exclude_tags`, `links`, `extract` (CSS selectors), `autoparse`, `wait_for`, `cache`, `cache_ttl`, `max_cost` (default 5). Always an HTTP GET. |
+| `spicrawl_batch_submit` | `batch` | `urls` (1 to 25), `format`, `render`, `main_content_only`, `max_cost`, `credit_budget` (default 50). |
+| `spicrawl_batch_status` | `batch` | `job_id`. |
+| `spicrawl_batch_results` | `batch` | `job_id`, `status`, `limit` (default 25, max 100), `cursor`. |
+| `spicrawl_docs_search`, `spicrawl_docs_read` | none | As on `/mcp`. |
 
-## Self-hosting the HTTP server
+Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.
 
-The hosted endpoint is this package's Streamable HTTP server. To run your own, for example beside a self-hosted Spicrawl deployment:
+Installed globally (`npm install -g @spicrawl/mcp`), the command is `spicrawl-mcp` (alias `spicrawl-mcp-server`), with `--http`, `--version` and `--help`.
 
-```bash
-SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
-SPICRAWL_BASE_URL=https://api.spicrawl.com \
-npx -y @spicrawl/mcp --http
-```
+## FAQ
 
-or, from an installed copy, `node node_modules/@spicrawl/mcp/dist/http.js` with the same environment.
+### Is there a hosted Spicrawl MCP server?
+Yes: `https://mcp.spicrawl.com/mcp`, MCP Streamable HTTP, authenticated with `Authorization: Bearer <API key>`, so there is nothing to install.
 
-- It serves MCP at `POST/GET/DELETE /mcp` and liveness at `GET /healthz` (no auth): `curl http://127.0.0.1:8090/healthz`.
-- **`SPICRAWL_BASE_URL` defaults to `http://127.0.0.1:8080`** for the HTTP server (a Spicrawl API on the same host), not to the public API. Set it to `https://api.spicrawl.com` to front the public API.
-- Each caller authenticates with their own key as `Authorization: Bearer spicrawl_live_...`. The key is validated against the API before a session opens, and a session is pinned to the key that created it.
-- The server speaks plain HTTP. Put it behind TLS (a reverse proxy) before sending keys across a network.
+### Is Spicrawl free?
+Spicrawl bills credits per successful request, and during the beta every organization gets one plan of 1,000 credits per month. Failed requests cost 0 credits; a cache hit is billed at the same price as the fetch that stored it. See [Credits](https://docs.spicrawl.com/credits).
 
-## Development
+### Does it render JavaScript?
+Yes. Set `render: true` on `spicrawl_scrape` to run the page in a browser, or `mode: "auto"` to let Spicrawl escalate from a plain fetch only when needed.
 
-```bash
-git clone https://github.com/Spicrawl/mcp.git && cd mcp
-npm ci
-npm run typecheck
-npm test
-```
+### Does it work with Claude Code, Cursor and VS Code?
+Yes, and with Claude Desktop, Windsurf, Gemini CLI, Codex and any client that speaks MCP Streamable HTTP with custom headers or launches stdio servers.
+
+### Can it crawl a whole website?
+There is no single crawl tool. Scrape a page with `links: true` to get its links, then feed them to an open batch job (`open: true` and `spicrawl_batch_add_items`).
+
+### Does it support OAuth?
+Not on `/mcp`: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a `401`. The separate [ChatGPT endpoint](#chatgpt-endpoint-oauth) at `/chatgpt/mcp` takes OAuth access tokens only, when the operator turns it on.
+
+### Should I use the MCP server, the SDK or the CLI?
+Use the MCP server when an AI agent should call Spicrawl as tools. Use [`@spicrawl/sdk`](https://github.com/Spicrawl/sdk) in your own TypeScript or JavaScript code, [`@spicrawl/cli`](https://github.com/Spicrawl/cli) in a terminal or shell script, and the [REST API](https://docs.spicrawl.com/quickstart) from any other language.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| HTTP `401` or "Unauthorized: send your Spicrawl API key" | The header is missing, or the key is unknown, revoked or expired. Check that the variable is set in the environment the client started from. |
+| The header contains a literal `${SPICRAWL_API_KEY}` | The client did not expand it. Use `${VAR}` in Claude Code, `${env:VAR}` in Cursor, `${input:id}` in VS Code, `bearer_token_env_var` in Codex. |
+| HTTP `503` with `Retry-After` on connect | The server could not reach the API to check your key. Retry; your key is fine. |
+| `ERR::AUTH::INSUFFICIENT_SCOPE` from a `spicrawl_usage*` tool | Grant the key the `read` scope in the dashboard. |
+| `ERR::LIMIT::QUOTA_EXCEEDED` (HTTP 402) | The monthly credit allowance is used up. Not retryable until it resets. |
+| "Unknown argument" on `spicrawl_scrape` | Use the tool's names: `render`, not `js_render`; `format`, not `response_format`. |
+| Tools do not appear | Restart the client, and check the config key (`servers` in VS Code, `mcpServers` elsewhere). |
+| `npx` not found (stdio) | Install Node.js 20 or later. |
+
+`curl -i https://mcp.spicrawl.com/mcp` without a key answers `401`: the endpoint is reachable.
 
-`npm test` builds the package, then runs the `node:test` suite in `test/` against the real `dist/http.js`, pointed at a fake Spicrawl API on loopback. It needs no deployment, no network and no real API key, and finishes in a few seconds.
+## Related packages
 
-`npm run build` writes `dist/index.js` (the `spicrawl-mcp` command: stdio, or HTTP with `--http`) and `dist/http.js` (the HTTP server). Run the stdio server from a checkout with `SPICRAWL_API_KEY=... node dist/index.js`.
+- [`@spicrawl/sdk`](https://github.com/Spicrawl/sdk): the official TypeScript SDK for the Spicrawl API.
+- [`@spicrawl/cli`](https://github.com/Spicrawl/cli): the Spicrawl command-line interface.
+- [Spicrawl/agent-plugins](https://github.com/Spicrawl/agent-plugins): plugins that bundle this server and the Spicrawl skill for Claude Code, Codex, Cursor, Gemini CLI and more.
 
-## Security
+## Links
 
-See [SECURITY.md](SECURITY.md). Never commit an API key; if one leaks, revoke it at https://app.spicrawl.com.
+- Documentation: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp)
+- Docs for LLMs: [docs.spicrawl.com/llms.txt](https://docs.spicrawl.com/llms.txt)
+- Dashboard and API keys: [app.spicrawl.com](https://app.spicrawl.com)
+- Issues: [github.com/Spicrawl/mcp/issues](https://github.com/Spicrawl/mcp/issues)
+- Security: [SECURITY.md](https://github.com/Spicrawl/mcp/blob/main/SECURITY.md). Never commit an API key; revoke a leaked one at [app.spicrawl.com](https://app.spicrawl.com).
 
 ## License
 
-[Apache-2.0](LICENSE)
+[Apache-2.0](https://github.com/Spicrawl/mcp/blob/main/LICENSE)