@ariangibson/firecrawl-lite-mcp-server 1.1.1 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (9) hide show
  1. package/README.md +362 -181
  2. package/dist/browser.js +145 -0
  3. package/dist/config.js +67 -0
  4. package/dist/firecrawlApi.js +86 -0
  5. package/dist/htmlToMarkdown.js +263 -0
  6. package/dist/index.js +362 -359
  7. package/dist/scraper.js +138 -0
  8. package/dist/utils.js +124 -0
  9. package/package.json +13 -7
package/README.md CHANGED
@@ -1,91 +1,101 @@
1
+ <div align="center">
2
+
3
+ <img src="docs/banner.jpg" alt="Firecrawl Lite MCP Server" width="100%" />
4
+
1
5
  # Firecrawl Lite MCP Server
2
6
 
3
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
4
-
5
- A **privacy-first, standalone** MCP server that provides web scraping and data extraction tools using local browser automation and your own LLM API key. **No external dependencies or API keys required** - completely decoupled from Firecrawl's cloud service.
6
-
7
- ## � **What Makes Firecrawl Lite Special**
8
-
9
- ### **🔒 Privacy-First Architecture**
10
- - **Local Processing** - All web scraping and data extraction happens on your machine
11
- - **Your Data Stays Local** - Content is processed locally, not sent to third parties
12
- - **No External Service Lock-in** - Doesn't require Firecrawl's cloud API
13
- - **Complete Control** - You own your data and infrastructure
14
-
15
- ### **💰 Cost-Effective & Transparent**
16
- - **Pay Only for LLM Usage** - No additional subscription or API fees
17
- - **Your LLM Provider** - Compatible with OpenAI, xAI, Anthropic, Ollama, etc.
18
- - **Predictable Costs** - Transparent pricing based on your chosen LLM rates
19
-
20
- ### **⚡ Performance & Simplicity**
21
- - **Lightning-Fast Startup** - Lightweight design means quick initialization
22
- - **Single Container** - Simple deployment with Docker support
23
- - **Minimal Resource Usage** - Optimized for efficiency and low memory footprint
24
-
25
- ## 📊 **Feature Comparison**
26
-
27
- | Feature | Firecrawl Lite ✅ | Original Firecrawl ❌ |
28
- |---------|-------------------|----------------------|
29
- | **🏠 Deployment** | **Standalone/Local** | Cloud Service |
30
- | **🔑 API Keys Required** | **Your LLM key only** | Firecrawl API + LLM keys |
31
- | **🔒 Data Privacy** | **100% local processing** | Cloud processing |
32
- | **💰 Cost Model** | **LLM usage only** | Subscription + LLM costs |
33
- | **⚙️ Setup Complexity** | **Single container** | Multi-service deployment |
34
- | **📦 Bundle Size** | **~50MB lightweight** | Heavy multi-service |
35
- | **🏠 Local LLM Support** | **✅ Ollama/Local LLMs** | Limited local options |
36
- | **🎛️ Customization** | **Full control** | Limited customization |
37
- | **🚀 Startup Time** | **< 5 seconds** | Variable (cloud dependent) |
38
- | **🔧 Maintenance** | **Self-managed** | Managed service |
39
-
40
- ## �️ **Available Tools**
41
-
42
- This standalone version provides local web scraping and data extraction using Puppeteer and your own LLM:
43
-
44
- ### ✅ **`scrape_page`** - Extract content from a single webpage
45
- - **Implementation**: Local browser automation with Puppeteer
46
- - **Use case**: Get webpage content for LLMs to read
47
- - **Parameters**: `url`, `onlyMainContent`
48
- - **Privacy**: All data processed locally
49
-
50
- ### ✅ **`batch_scrape`** - Scrape multiple URLs in a single request
51
- - **Implementation**: Sequential local scraping with rate limiting
52
- - **Use case**: Process multiple pages efficiently
53
- - **Parameters**: `urls[]`, `onlyMainContent`
54
- - **Privacy**: All data processed locally
55
-
56
- ### ✅ **`extract_data`** - Extract structured data using LLM
57
- - **Implementation**: Local scraping + your LLM for data extraction
58
- - **Use case**: Pull specific data from pages using natural language prompts
59
- - **Parameters**: `urls[]`, `prompt`, `enableWebSearch`
60
- - **Privacy**: Content scraped locally, sent to your LLM only
61
-
62
- ### ✅ **`extract_with_schema`** - Extract data using JSON schema
63
- - **Implementation**: Local scraping + schema-guided LLM extraction
64
- - **Use case**: Extract structured data with predefined schema
65
- - **Parameters**: `urls[]`, `schema`, `prompt`, `enableWebSearch`
66
- - **Privacy**: Content scraped locally, sent to your LLM only
67
-
68
- ## 🚀 **Quick Start**
69
-
70
- ### **1. Install the package:**
7
+ **Privacy-first web scraping for AI agents — an MCP server _and_ a Firecrawl-compatible API in one small process, powered by local browser automation and your own LLM key.**
8
+
9
+ [![npm version](https://img.shields.io/npm/v/@ariangibson/firecrawl-lite-mcp-server?logo=npm&color=cb3837)](https://www.npmjs.com/package/@ariangibson/firecrawl-lite-mcp-server)
10
+ [![Docker Pulls](https://img.shields.io/docker/pulls/ariangibson/firecrawl-lite-mcp-server?logo=docker&logoColor=white)](https://hub.docker.com/r/ariangibson/firecrawl-lite-mcp-server)
11
+ [![Image Size](https://img.shields.io/docker/image-size/ariangibson/firecrawl-lite-mcp-server/latest?logo=docker&logoColor=white&label=image%20size)](https://hub.docker.com/r/ariangibson/firecrawl-lite-mcp-server)
12
+ [![Build](https://img.shields.io/github/actions/workflow/status/ariangibson/firecrawl-lite-mcp-server/docker-build.yml?branch=main&logo=github&label=build)](https://github.com/ariangibson/firecrawl-lite-mcp-server/actions/workflows/docker-build.yml)
13
+ [![Tests](https://img.shields.io/github/actions/workflow/status/ariangibson/firecrawl-lite-mcp-server/test.yml?branch=main&logo=github&label=tests)](https://github.com/ariangibson/firecrawl-lite-mcp-server/actions/workflows/test.yml)
14
+ [![Node](https://img.shields.io/node/v/@ariangibson/firecrawl-lite-mcp-server?logo=node.js&logoColor=white)](https://nodejs.org)
15
+ [![MCP](https://img.shields.io/badge/MCP-compatible-6E56CF)](https://modelcontextprotocol.io)
16
+ [![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
17
+
18
+ </div>
19
+
20
+ ---
21
+
22
+ Firecrawl Lite gives AI agents and MCP clients the ability to fetch, render, and extract web pages **without a Firecrawl account and without Firecrawl's multi-service self-hosted stack**. Pages are rendered by a local, stealth-enabled headless browser and converted to clean Markdown. It speaks two protocols from a single Node.js process:
23
+
24
+ - **[Model Context Protocol](https://modelcontextprotocol.io)** — for Claude Desktop, Claude Code, Cursor, and any other MCP client.
25
+ - **Firecrawl-compatible REST API** — for anything that speaks the Firecrawl SDK, most notably [Hermes Agent](https://hermes-agent.nousresearch.com). Point `FIRECRAWL_API_URL` at it and it behaves like a self-hosted Firecrawl instance.
26
+
27
+ Structured extraction (`extract_data`, `extract_with_schema`) uses **your own LLM provider** via any OpenAI-compatible endpoint. The LLM is optional — scraping and the Firecrawl-compatible API work with no API keys at all.
28
+
29
+ ## Contents
30
+
31
+ - [Why Firecrawl Lite](#why-firecrawl-lite)
32
+ - [Use with Hermes Agent](#use-with-hermes-agent)
33
+ - [Use as an MCP Server](#use-as-an-mcp-server)
34
+ - [Available Tools](#available-tools)
35
+ - [Configuration](#configuration)
36
+ - [Remote Deployment](#remote-deployment)
37
+ - [Firecrawl-compatible API](#firecrawl-compatible-api)
38
+ - [Advanced Configuration](#advanced-configuration)
39
+ - [Usage Examples](#usage-examples)
40
+ - [Troubleshooting](#troubleshooting)
41
+ - [Container Images](#container-images)
42
+ - [Development](#development)
43
+ - [Credits](#credits)
44
+ - [License](#license)
45
+
46
+ ## Why Firecrawl Lite
47
+
48
+ **One process, no infrastructure.** Self-hosting Firecrawl means Redis, a Playwright service, API and worker containers, and more. Firecrawl Lite is a single Node.js process with a bundled headless browser. Run it with `npx`, or as one container.
49
+
50
+ **Privacy-first.** Scraping and rendering happen on your own machine or server. Page content is only ever sent to the LLM provider you explicitly configure — nothing is routed through a third-party scraping cloud.
51
+
52
+ **Bring your own model (or none).** Extraction tools work with any OpenAI-compatible `chat/completions` endpoint: OpenAI, xAI (Grok), Anthropic, OpenRouter, or a local model via Ollama. If your agent already has a model in the loop, skip the LLM entirely and just use the scraping tools.
53
+
54
+ **Built for real scraping.** Stealth browser automation, rotating user agents, configurable delays, DOM-settle detection for JS-heavy pages, optional upstream proxies (including port-range rotation), and tunable retry/backoff.
55
+
56
+ ## Use with Hermes Agent
57
+
58
+ [Hermes Agent](https://hermes-agent.nousresearch.com) uses Firecrawl by default for its `web_extract` tool, and supports self-hosted Firecrawl via `FIRECRAWL_API_URL`. Firecrawl Lite implements the scrape endpoint Hermes needs (and returns a clear `501` for search), so you get local, private page extraction without installing Firecrawl.
59
+
60
+ **1. Run Firecrawl Lite with the Firecrawl-compatible API enabled:**
61
+
71
62
  ```bash
72
- npm install -g @ariangibson/firecrawl-lite-mcp-server
63
+ docker run -d -p 3000:3000 --name firecrawl-lite \
64
+ ariangibson/firecrawl-lite-mcp-server:latest
73
65
  ```
74
66
 
75
- Or use npx to run without global installation (recommended).
67
+ The container image enables the API by default. To run it without Docker (this stays in the foreground; set `PORT` to change the listen port):
76
68
 
77
- ### **2. Configure your LLM:**
78
69
  ```bash
79
- # Create a .env file or set environment variables
80
- LLM_API_KEY=your_llm_api_key_here
81
- LLM_PROVIDER_BASE_URL=https://api.x.ai/v1
82
- LLM_MODEL=grok-code-fast-1
70
+ ENABLE_FIRECRAWL_API=true npx -y @ariangibson/firecrawl-lite-mcp-server
71
+ ```
72
+
73
+ **2. Point Hermes at it.** In your Hermes environment (e.g. `~/.hermes/.env`):
74
+
75
+ ```bash
76
+ FIRECRAWL_API_URL=http://localhost:3000
77
+ ```
78
+
79
+ **3. Pick a search backend.** Firecrawl Lite renders pages; it is not a search engine. Tell Hermes to use a free search provider for `web_search` and Firecrawl Lite for `web_extract`, in `~/.hermes/config.yaml`:
80
+
81
+ ```yaml
82
+ web:
83
+ search_backend: ddgs # DuckDuckGo — no API key. Or: searxng, brave-free
84
+ extract_backend: firecrawl # → your Firecrawl Lite instance
83
85
  ```
84
86
 
85
- ### **3. Configure your MCP client:**
87
+ That's it. The native Hermes `web_extract` tool now renders through your local stealth browser. If you set `FIRECRAWL_API_KEY` on the server, set the same `FIRECRAWL_API_KEY` in Hermes and the SDK will send it as a bearer token.
88
+
89
+ > **Tip:** Hermes also supports MCP servers, so you can additionally add Firecrawl Lite over MCP to get `screenshot` and the LLM-backed `extract_with_schema` tool. For most agent use, the Firecrawl-compatible API alone is the cleaner setup.
90
+
91
+ ## Use as an MCP Server
92
+
93
+ The fastest way to use Firecrawl Lite locally is over stdio via `npx` — no install or container required. LLM credentials are only needed for the `extract_*` tools (see [LLM provider examples](#llm-provider-examples)); omit them if you only need scraping and screenshots.
94
+
95
+ ### Claude Desktop
96
+
97
+ Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
86
98
 
87
- #### **Claude Desktop**
88
- Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
89
99
  ```json
90
100
  {
91
101
  "mcpServers": {
@@ -94,125 +104,246 @@ Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
94
104
  "args": ["-y", "@ariangibson/firecrawl-lite-mcp-server"],
95
105
  "env": {
96
106
  "LLM_API_KEY": "your_llm_api_key_here",
97
- "LLM_PROVIDER_BASE_URL": "https://api.x.ai/v1",
98
- "LLM_MODEL": "grok-code-fast-1"
107
+ "LLM_PROVIDER_BASE_URL": "https://api.openai.com/v1",
108
+ "LLM_MODEL": "gpt-5.5"
99
109
  }
100
110
  }
101
111
  }
102
112
  }
103
113
  ```
104
114
 
105
- #### **Claude Code (CLI)**
115
+ ### Claude Code (CLI)
116
+
106
117
  ```bash
107
- claude config mcp add firecrawl-lite \
108
- --command "npx" \
109
- --args "-y" --args "@ariangibson/firecrawl-lite-mcp-server" \
110
- --env LLM_API_KEY=your_llm_api_key_here \
111
- --env LLM_PROVIDER_BASE_URL=https://api.x.ai/v1 \
112
- --env LLM_MODEL=grok-code-fast-1
118
+ claude mcp add firecrawl-lite npx -- -y @ariangibson/firecrawl-lite-mcp-server \
119
+ --env LLM_API_KEY=your_key \
120
+ --env LLM_PROVIDER_BASE_URL=https://api.openai.com/v1 \
121
+ --env LLM_MODEL=gpt-5.5
113
122
  ```
114
123
 
115
- ### **4. Restart your MCP client and start scraping!**
124
+ ### Cursor
116
125
 
117
- ## ⚙️ **Configuration Guide**
126
+ Add to your Cursor MCP configuration (`~/.cursor/mcp.json`):
118
127
 
119
- ### **Required Environment Variables**
120
- ```bash
121
- # Your LLM API key (xAI, OpenAI, Anthropic, etc.)
122
- LLM_API_KEY=your_api_key_here
128
+ ```json
129
+ {
130
+ "mcpServers": {
131
+ "firecrawl-lite": {
132
+ "command": "npx",
133
+ "args": ["-y", "@ariangibson/firecrawl-lite-mcp-server"],
134
+ "env": {
135
+ "LLM_API_KEY": "your_llm_api_key_here",
136
+ "LLM_PROVIDER_BASE_URL": "https://api.openai.com/v1",
137
+ "LLM_MODEL": "gpt-5.5"
138
+ }
139
+ }
140
+ }
141
+ }
142
+ ```
123
143
 
124
- # LLM provider base URL
125
- LLM_PROVIDER_BASE_URL=https://api.x.ai/v1
144
+ ## Available Tools
126
145
 
127
- # LLM model name
128
- LLM_MODEL=grok-code-fast-1
129
- ```
146
+ | Tool | Description | Required params | Optional params | Needs LLM |
147
+ | --- | --- | --- | --- | --- |
148
+ | `scrape_page` | Fetch and render a single page, returning clean Markdown. | `url` | `onlyMainContent` | No |
149
+ | `batch_scrape` | Scrape multiple URLs in one request (up to 10). | `urls[]` | `onlyMainContent` | No |
150
+ | `screenshot` | Capture a screenshot of a page via the stealth browser. | `url` | `width` (1920), `height` (1080), `fullPage` (false) | No |
151
+ | `extract_data` | Extract structured data from pages using a natural-language prompt. | `urls[]`, `prompt` | — | Yes |
152
+ | `extract_with_schema` | Extract data conforming to a supplied JSON Schema. | `urls[]`, `schema` | `prompt` | Yes |
130
153
 
131
- ### **LLM Provider Examples**
132
- ```bash
133
- # xAI (Grok)
134
- LLM_PROVIDER_BASE_URL=https://api.x.ai/v1
135
- LLM_API_KEY=xai-your-key-here
136
- LLM_MODEL=grok-code-fast-1
154
+ ## Configuration
155
+
156
+ All configuration is via environment variables. Everything has a sensible default; the LLM variables are only required if you use the `extract_*` tools.
157
+
158
+ ### LLM (required for `extract_data` / `extract_with_schema`)
137
159
 
160
+ | Variable | Description |
161
+ | --- | --- |
162
+ | `LLM_API_KEY` | API key for your LLM provider. |
163
+ | `LLM_PROVIDER_BASE_URL` | Base URL of an OpenAI-compatible API (the server calls `{base_url}/chat/completions`). |
164
+ | `LLM_MODEL` | Model name to use for extraction. |
165
+
166
+ ### Optional LLM tuning
167
+
168
+ These are passed straight through to the provider's `chat/completions` request. Leave any of them unset to use the default; the optional sampling parameters are omitted from the request entirely when unset.
169
+
170
+ | Variable | Default | Notes |
171
+ | --- | --- | --- |
172
+ | `LLM_TEMPERATURE` | `0.1` | Sampling temperature. |
173
+ | `LLM_MAX_TOKENS` | `2000` | Maximum tokens in the response. Raise this if extractions are being truncated. |
174
+ | `LLM_TOP_P` | _unset_ | Nucleus sampling; omitted from the request unless set. |
175
+ | `LLM_REASONING_EFFORT` | _unset_ | `reasoning_effort` for reasoning-capable models; omitted unless set. |
176
+
177
+ ### LLM provider examples
178
+
179
+ ```bash
138
180
  # OpenAI
139
181
  LLM_PROVIDER_BASE_URL=https://api.openai.com/v1
140
- LLM_API_KEY=sk-your-key-here
141
- LLM_MODEL=gpt-4o-mini
182
+ LLM_MODEL=gpt-5.5
183
+
184
+ # xAI (Grok)
185
+ LLM_PROVIDER_BASE_URL=https://api.x.ai/v1
186
+ LLM_MODEL=grok-4
142
187
 
143
188
  # Anthropic
144
- LLM_PROVIDER_BASE_URL=https://api.anthropic.com
145
- LLM_API_KEY=sk-ant-your-key-here
146
- LLM_MODEL=claude-3-haiku-20240307
189
+ LLM_PROVIDER_BASE_URL=https://api.anthropic.com/v1
190
+ LLM_MODEL=claude-haiku-4-5
147
191
 
148
- # Local LLM (Ollama)
192
+ # OpenRouter
193
+ LLM_PROVIDER_BASE_URL=https://openrouter.ai/api/v1
194
+ LLM_MODEL=openai/gpt-5.5
195
+
196
+ # Local (Ollama)
149
197
  LLM_PROVIDER_BASE_URL=http://localhost:11434/v1
150
- LLM_API_KEY=your-local-key
151
- LLM_MODEL=llama2
198
+ LLM_MODEL=llama3.3
152
199
  ```
153
200
 
154
- ### **Optional Configuration**
155
- ```bash
156
- # Proxy configuration (for web scraping and LLM API calls)
157
- PROXY_SERVER_URL=http://your-proxy.com:8080
158
- PROXY_SERVER_USERNAME=your_proxy_username
159
- PROXY_SERVER_PASSWORD=your_proxy_password
201
+ ### HTTP endpoints
160
202
 
161
- # Scraping configuration (anti-detection and rate limiting)
162
- SCRAPE_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
163
- SCRAPE_VIEWPORT_WIDTH=1920
164
- SCRAPE_VIEWPORT_HEIGHT=1080
165
- SCRAPE_DELAY_MIN=1000
166
- SCRAPE_DELAY_MAX=3000
167
- ```
203
+ All HTTP endpoints are **disabled by default** when running from `npx`, and the server speaks MCP over stdio. Enabling any of them switches the process to an HTTP server listening on `PORT` (stdio is not served); `/health` is always available in that mode. The Docker image enables `/mcp` and the Firecrawl-compatible API by default.
168
204
 
169
- ## 🛡️ **Anti-Detection Features**
205
+ | Variable | Default | Enables |
206
+ | --- | --- | --- |
207
+ | `ENABLE_FIRECRAWL_API` | `false` (`true` in Docker) | `POST /v2/scrape` (and `/v1/scrape`) — Firecrawl-compatible API for Hermes Agent and Firecrawl SDK clients. |
208
+ | `FIRECRAWL_API_KEY` | _unset_ | If set, the Firecrawl-compatible API requires `Authorization: Bearer <key>`. |
209
+ | `ENABLE_HTTP_STREAMABLE_ENDPOINT` | `false` (`true` in Docker) | `/mcp` — Streamable HTTP transport for Claude Code and other remote MCP clients. |
210
+ | `ENABLE_SSE_ENDPOINT` | `false` | `/sse` — deprecated SSE transport (Claude Desktop via `mcp-proxy`). |
211
+ | `PORT` | `3000` | HTTP listen port. |
170
212
 
171
- Firecrawl Lite includes sophisticated anti-detection measures to handle modern websites with bot protection:
213
+ See [`.env.example`](.env.example) for the full, annotated list of variables.
172
214
 
173
- ### ✅ **Built-in Anti-Detection**
174
- - **Realistic Browser Fingerprinting**: Spoofs navigator properties, plugins, and browser APIs
175
- - **Random Delays**: Adds human-like delays between requests (configurable)
176
- - **Modern User Agent**: Uses up-to-date Chrome user agent strings
177
- - **Viewport Simulation**: Sets realistic desktop viewport sizes
178
- - **Headless Optimization**: Configured for maximum stealth in headless mode
215
+ ## Remote Deployment
216
+
217
+ ### Docker
179
218
 
180
- ### ✅ **Configurable Settings**
181
219
  ```bash
182
- # Control delays (in milliseconds)
183
- SCRAPE_DELAY_MIN=1000 # Minimum delay before navigation
184
- SCRAPE_DELAY_MAX=3000 # Maximum delay before navigation
185
- SCRAPE_BATCH_DELAY_MIN=2000 # Minimum delay between batch requests
186
- SCRAPE_BATCH_DELAY_MAX=5000 # Maximum delay between batch requests
220
+ docker run -d \
221
+ -p 3000:3000 \
222
+ -e LLM_API_KEY=your_key_here \
223
+ -e LLM_PROVIDER_BASE_URL=https://api.openai.com/v1 \
224
+ -e LLM_MODEL=gpt-5.5 \
225
+ ariangibson/firecrawl-lite-mcp-server:latest
187
226
  ```
188
227
 
189
- ## � **Docker Deployment (Optional)**
228
+ The LLM variables are optional — drop them if you only need the Firecrawl-compatible API or the scraping tools.
229
+
230
+ ### Docker Compose / Portainer / Swarm
190
231
 
191
- If you prefer Docker deployment:
232
+ A ready-to-use [`docker-compose.yml`](docker-compose.yml) is included. Set your variables in a `.env` file and deploy:
192
233
 
193
234
  ```bash
194
- # Build and run with Docker
195
- docker-compose up --build
235
+ docker compose up -d
236
+ ```
237
+
238
+ > **Note for Docker Swarm / Portainer:** the published image is Alpine-based and does **not** include `curl`. The bundled compose file uses a `wget`-based health check for this reason — see [Troubleshooting](#container-keeps-restarting-or-is-killed-with-sigterm) if you have replaced it with a `curl`-based check.
196
239
 
197
- # Run in background
198
- docker-compose up -d --build
240
+ ### Remote MCP client configuration
241
+
242
+ **Claude Code (Streamable HTTP):**
243
+
244
+ ```bash
245
+ claude mcp add firecrawl-lite-remote http://your-server:3000/mcp -t http
199
246
  ```
200
247
 
201
- The server will be available at `http://localhost:3000` with a health endpoint at `http://localhost:3000/health`.
248
+ **Claude Desktop — Connectors (recommended, HTTPS only):**
202
249
 
203
- ## 📊 **Usage Examples**
250
+ Settings → Connectors → add `https://your-server.com:3000/mcp`. Requires a valid TLS certificate.
251
+
252
+ **Claude Desktop — `mcp-proxy` (HTTP fallback, no certificate):**
253
+
254
+ ```bash
255
+ pip install mcp-proxy
256
+ ```
204
257
 
205
- ### Scrape a webpage
206
258
  ```json
207
259
  {
208
- "name": "scrape_page",
209
- "arguments": {
210
- "url": "https://example.com"
260
+ "mcpServers": {
261
+ "firecrawl-lite": {
262
+ "command": "mcp-proxy",
263
+ "args": ["http://your-server:3000/sse"]
264
+ }
211
265
  }
212
266
  }
213
267
  ```
214
268
 
215
- ### Batch scrape multiple URLs
269
+ Requires `ENABLE_SSE_ENDPOINT=true` on the server.
270
+
271
+ ## Firecrawl-compatible API
272
+
273
+ When `ENABLE_FIRECRAWL_API=true`, the server implements the subset of the [Firecrawl v2 API](https://docs.firecrawl.dev/api-reference/endpoint/scrape) that SDK clients use for page extraction. It works with the official `firecrawl-py` / `@mendable/firecrawl-js` SDKs by setting `api_url` / `apiUrl`.
274
+
275
+ | Endpoint | Behaviour |
276
+ | --- | --- |
277
+ | `POST /v2/scrape` (and `/v1/scrape`) | Renders the page and returns `{ success, data }` where `data` is a Firecrawl Document: `markdown`, `html`, `rawHtml`, `links` (per requested `formats`) plus `metadata` (`title`, `description`, `language`, `sourceURL`, `url`, `statusCode` — always `200` for a successful render). The `screenshot` format is accepted but ignored. |
278
+ | `POST /v2/search` (and `/v1/search`) | Returns `501` with a clear error. Firecrawl Lite is a renderer, not a search engine — pair it with a dedicated search backend. |
279
+
280
+ Request body: `{ "url": "https://…", "formats": ["markdown", "html"], "onlyMainContent": true }`. `formats` defaults to `["markdown"]`; both string and `{ "type": "markdown" }` entries are accepted.
281
+
282
+ ```bash
283
+ curl -X POST http://localhost:3000/v2/scrape \
284
+ -H 'Content-Type: application/json' \
285
+ -d '{"url":"https://example.com","formats":["markdown"]}'
286
+ ```
287
+
288
+ ```python
289
+ from firecrawl import Firecrawl
290
+
291
+ fc = Firecrawl(api_key="unused", api_url="http://localhost:3000")
292
+ doc = fc.scrape("https://example.com", formats=["markdown"])
293
+ print(doc.markdown)
294
+ ```
295
+
296
+ Not implemented: crawl, map, batch jobs, extract, and other async job-based endpoints.
297
+
298
+ ## Advanced Configuration
299
+
300
+ ### Proxy
301
+
302
+ Route the scraping browser through an upstream proxy. A port range (e.g. `:10001-10010`) enables automatic rotation across ports.
303
+
304
+ ```bash
305
+ PROXY_SERVER_URL=http://proxy.example.com:10001-10010
306
+ PROXY_SERVER_USERNAME=your-username
307
+ PROXY_SERVER_PASSWORD=your-password
308
+ ```
309
+
310
+ By default the proxy is used **only for scraping target sites** — LLM provider API calls go out directly. Routing your own LLM calls through a rotating (often residential) proxy is slower, can trip provider abuse detection, and may fail TLS. If you specifically need the LLM call proxied as well, opt in with `PROXY_LLM_API=true` (default: `false`).
311
+
312
+ ### Anti-detection and rate limiting
313
+
314
+ `SCRAPE_USER_AGENT` accepts either a single string or a JSON array of strings to rotate through. When using a JSON array, keep it on a single line.
315
+
316
+ ```bash
317
+ SCRAPE_USER_AGENT=["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... Safari/537.36","Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Safari/537.36"]
318
+ SCRAPE_VIEWPORT_WIDTH=1920
319
+ SCRAPE_VIEWPORT_HEIGHT=1080
320
+ SCRAPE_DELAY_MIN=1000 # min delay before navigation (ms)
321
+ SCRAPE_DELAY_MAX=3000 # max delay before navigation (ms)
322
+ SCRAPE_BATCH_DELAY_MIN=2000 # min delay between batch requests (ms)
323
+ SCRAPE_BATCH_DELAY_MAX=5000 # max delay between batch requests (ms)
324
+ SCRAPE_SETTLE_MAX_MS=3000 # max wait for the DOM to stop changing after load (ms); raise for slow JS sites
325
+ ```
326
+
327
+ After load the scraper scrolls to trigger lazy/AJAX content and waits for the DOM to settle (exiting early once stable). Pages that inject content via a long `setTimeout` may need a higher `SCRAPE_SETTLE_MAX_MS`.
328
+
329
+ ### Retries
330
+
331
+ Each scrape or screenshot is attempted up to `FIRECRAWL_RETRY_MAX_ATTEMPTS` times (default `3`), advancing through the proxy and user-agent rotation on each attempt.
332
+
333
+ ```bash
334
+ FIRECRAWL_RETRY_MAX_ATTEMPTS=3
335
+ ```
336
+
337
+ ## Usage Examples
338
+
339
+ **Scrape a page**
340
+
341
+ ```json
342
+ { "name": "scrape_page", "arguments": { "url": "https://example.com" } }
343
+ ```
344
+
345
+ **Batch scrape**
346
+
216
347
  ```json
217
348
  {
218
349
  "name": "batch_scrape",
@@ -223,18 +354,20 @@ The server will be available at `http://localhost:3000` with a health endpoint a
223
354
  }
224
355
  ```
225
356
 
226
- ### Extract data with prompt
357
+ **Extract with a prompt**
358
+
227
359
  ```json
228
360
  {
229
361
  "name": "extract_data",
230
362
  "arguments": {
231
363
  "urls": ["https://example.com"],
232
- "prompt": "Extract the main article title and summary"
364
+ "prompt": "Extract the main article title and a one-sentence summary."
233
365
  }
234
366
  }
235
367
  ```
236
368
 
237
- ### Extract with schema
369
+ **Extract with a JSON Schema**
370
+
238
371
  ```json
239
372
  {
240
373
  "name": "extract_with_schema",
@@ -243,45 +376,93 @@ The server will be available at `http://localhost:3000` with a health endpoint a
243
376
  "schema": {
244
377
  "type": "object",
245
378
  "properties": {
246
- "title": {"type": "string"},
247
- "description": {"type": "string"}
379
+ "title": { "type": "string" },
380
+ "description": { "type": "string" }
248
381
  }
249
382
  }
250
383
  }
251
384
  }
252
385
  ```
253
386
 
254
- ## ❓ **Important Notes**
387
+ ## Troubleshooting
388
+
389
+ ### Chrome / Chromium issues
390
+
391
+ The container image bundles Chromium. When run via `npx`, Chrome is downloaded automatically on first install. If it's missing (scrapes fail with `Could not find Chrome`):
392
+
393
+ ```bash
394
+ npx puppeteer browsers install chrome
395
+ # or reset a corrupted install
396
+ rm -rf ~/.cache/puppeteer && npx puppeteer browsers install chrome
397
+ ```
398
+
399
+ ### Extraction returns an error
400
+
401
+ If `scrape_page` works but `extract_data` fails, the problem is the LLM call, not scraping. The server logs the upstream status, error code, and response body to stderr (`LLM extract_data request failed: ...`) and surfaces the HTTP status in the tool result. Common causes:
402
+
403
+ - **HTTP 401** — invalid `LLM_API_KEY`.
404
+ - **HTTP 400** — wrong `LLM_MODEL`, or a tuning parameter the model rejects (e.g. `LLM_MAX_TOKENS` above the model's limit, or `LLM_REASONING_EFFORT` on a non-reasoning model).
405
+ - **HTTP 429** — provider rate limit.
406
+
407
+ ### Hermes says "Firecrawl backend selected but not configured" or extract fails
408
+
409
+ - Confirm `FIRECRAWL_API_URL` is set in the Hermes environment and reachable: `curl http://your-server:3000/health` should report `"endpoints": { ..., "firecrawlApi": "enabled" }`.
410
+ - If you set `FIRECRAWL_API_KEY` on the server, Hermes needs the same value in `FIRECRAWL_API_KEY`.
411
+ - `web_search` failing is expected unless `web.search_backend` points at a real search provider (see [Use with Hermes Agent](#use-with-hermes-agent)).
412
+
413
+ ### Container keeps restarting or is killed with `SIGTERM`
414
+
415
+ If the logs show the server start (`listening on port 3000`) and then exit with `npm error signal SIGTERM`, the container is being killed by a **failing health check**, not by the app. The Alpine image does not include `curl`, so a `curl`-based health check always fails and Swarm restarts the task in a loop. Use a `wget`-based check (busybox provides `wget`):
416
+
417
+ ```yaml
418
+ healthcheck:
419
+ test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1"]
420
+ interval: 30s
421
+ timeout: 10s
422
+ retries: 3
423
+ start_period: 40s
424
+ ```
425
+
426
+ The bundled `docker-compose.yml` already uses this form, and the image's built-in `HEALTHCHECK` uses Node, so neither needs `curl`.
427
+
428
+ ## Container Images
429
+
430
+ Pre-built, multi-architecture (`amd64`, `arm64`) images are published automatically on every push to `main` and on release:
255
431
 
256
- ### **🌐 Internet Requirements**
257
- - **Requires Internet Access** - Still needs to access target websites
258
- - **LLM API Access** - Requires connection to your chosen LLM provider
259
- - **No Offline Operation** - Cannot work completely offline
432
+ - **Docker Hub:** `ariangibson/firecrawl-lite-mcp-server:latest`
433
+ - **GitHub Container Registry:** `ghcr.io/ariangibson/firecrawl-lite-mcp-server:latest`
260
434
 
261
- ### **� Intentionally Excluded Features**
262
- By design, this lite version excludes advanced features to maintain simplicity:
263
- - Web search functionality
264
- - Website URL discovery/mapping
265
- - Multi-page website crawling
266
- - LLMs.txt file generation
267
- - Advanced research capabilities
268
- - Crawl job status checking
435
+ ## Development
436
+
437
+ Requires Node.js 20 or newer (the container image uses Node 22).
438
+
439
+ ```bash
440
+ npm install # install dependencies (downloads Chrome for Puppeteer)
441
+ npm run build # compile TypeScript to dist/
442
+ npm run lint # type-check without emitting
443
+ npm test # run the unit test suite
444
+ npm start # run the built server
445
+ ```
269
446
 
270
- ## 🙏 **Credits & Acknowledgments**
447
+ The code is organised so that the interesting behaviour is testable without a browser or network:
271
448
 
272
- This project is inspired by and builds upon the excellent work of the original Firecrawl projects:
449
+ | Module | Responsibility |
450
+ | --- | --- |
451
+ | `src/config.ts` | `loadConfig(env)` — every environment variable, parsed once into a typed object. |
452
+ | `src/browser.ts` | One stealth browser session: launch flags, proxy/user-agent, navigation, cleanup, retries. The browser is injected. |
453
+ | `src/scraper.ts` | `createScraper(config, deps)` → `{ scrape, screenshot }`: rotation, retries, scroll/settle heuristics, HTML → Markdown. |
454
+ | `src/htmlToMarkdown.ts` | HTML cleaning, Markdown/text conversion, page metadata. |
455
+ | `src/firecrawlApi.ts` | Firecrawl-compatible request parsing, Document shaping, bearer auth. |
456
+ | `src/index.ts` | MCP tool definitions, LLM extraction, and the stdio/HTTP transports. |
273
457
 
274
- ### 🔥 **[Firecrawl](https://firecrawl.com)**
275
- The original Firecrawl project by **Mendable.ai** - a comprehensive web scraping and crawling platform with advanced features like website mapping, multi-page crawling, and deep research capabilities.
458
+ Unit tests run in well under a second: the scraper is exercised end to end through a stub browser and a fake clock (`tests/scraper.test.ts`), alongside the pure helpers. CI runs them against Node 20, 22, and 24. The scripts in `dev-scripts/` exercise the tools against live sites.
276
459
 
277
- ### 🔥 **[Firecrawl MCP Server](https://github.com/firecrawl/firecrawl-mcp-server)**
278
- The official MCP server implementation by the Firecrawl team, providing MCP integration for their cloud-based scraping service.
460
+ ## Credits
279
461
 
280
- **We give huge thanks to the Firecrawl team for their pioneering work in web scraping and MCP integration!** 🚀
462
+ Inspired by the excellent work of the [Firecrawl](https://firecrawl.com) team at Mendable.ai and their official [Firecrawl MCP Server](https://github.com/firecrawl/firecrawl-mcp-server). Firecrawl Lite is an independent, self-hosted take on the same idea — huge thanks to them for pioneering web scraping for the MCP ecosystem.
281
463
 
282
- > **💡 Looking for a very generous free tier and dead-simple cloud-hosted solution?**
283
- > Visit **[firecrawl.com](https://firecrawl.com)** and sign up for a Firecrawl account! Their cloud service offers enterprise-grade web scraping with zero setup complexity.
464
+ Looking for a fully managed, enterprise-grade scraping platform? Check out [firecrawl.com](https://firecrawl.com).
284
465
 
285
- ## �📝 **License**
466
+ ## License
286
467
 
287
- MIT License - see [LICENSE](LICENSE) for details.
468
+ MIT — see [LICENSE](LICENSE).