firecrawl-mcp 3.22.0 → 3.22.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -197
- package/dist/index.js +58 -14
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -213,37 +213,12 @@ Hosted Firecrawl can issue OAuth **access tokens** (`fco_…`) via the authoriza
|
|
|
213
213
|
|
|
214
214
|
Use **access** tokens (`fco_…`) only. Refresh tokens (`fcr_…`) must be exchanged at the token endpoint, not passed to the scrape/search API.
|
|
215
215
|
|
|
216
|
-
#### Optional Configuration
|
|
217
|
-
|
|
218
|
-
##### Retry Configuration
|
|
219
|
-
|
|
220
|
-
- `FIRECRAWL_RETRY_MAX_ATTEMPTS`: Maximum number of retry attempts (default: 3)
|
|
221
|
-
- `FIRECRAWL_RETRY_INITIAL_DELAY`: Initial delay in milliseconds before first retry (default: 1000)
|
|
222
|
-
- `FIRECRAWL_RETRY_MAX_DELAY`: Maximum delay in milliseconds between retries (default: 10000)
|
|
223
|
-
- `FIRECRAWL_RETRY_BACKOFF_FACTOR`: Exponential backoff multiplier (default: 2)
|
|
224
|
-
|
|
225
|
-
##### Credit Usage Monitoring
|
|
226
|
-
|
|
227
|
-
- `FIRECRAWL_CREDIT_WARNING_THRESHOLD`: Credit usage warning threshold (default: 1000)
|
|
228
|
-
- `FIRECRAWL_CREDIT_CRITICAL_THRESHOLD`: Credit usage critical threshold (default: 100)
|
|
229
|
-
|
|
230
216
|
### Configuration Examples
|
|
231
217
|
|
|
232
|
-
For cloud API usage
|
|
218
|
+
For cloud API usage:
|
|
233
219
|
|
|
234
220
|
```bash
|
|
235
|
-
# Required for cloud API
|
|
236
221
|
export FIRECRAWL_API_KEY=your-api-key
|
|
237
|
-
|
|
238
|
-
# Optional retry configuration
|
|
239
|
-
export FIRECRAWL_RETRY_MAX_ATTEMPTS=5 # Increase max retry attempts
|
|
240
|
-
export FIRECRAWL_RETRY_INITIAL_DELAY=2000 # Start with 2s delay
|
|
241
|
-
export FIRECRAWL_RETRY_MAX_DELAY=30000 # Maximum 30s delay
|
|
242
|
-
export FIRECRAWL_RETRY_BACKOFF_FACTOR=3 # More aggressive backoff
|
|
243
|
-
|
|
244
|
-
# Optional credit monitoring
|
|
245
|
-
export FIRECRAWL_CREDIT_WARNING_THRESHOLD=2000 # Warning at 2000 credits
|
|
246
|
-
export FIRECRAWL_CREDIT_CRITICAL_THRESHOLD=500 # Critical at 500 credits
|
|
247
222
|
```
|
|
248
223
|
|
|
249
224
|
For self-hosted instance:
|
|
@@ -254,10 +229,6 @@ export FIRECRAWL_API_URL=https://firecrawl.your-domain.com
|
|
|
254
229
|
|
|
255
230
|
# Optional authentication for self-hosted
|
|
256
231
|
export FIRECRAWL_API_KEY=your-api-key # If your instance requires auth
|
|
257
|
-
|
|
258
|
-
# Custom retry configuration
|
|
259
|
-
export FIRECRAWL_RETRY_MAX_ATTEMPTS=10
|
|
260
|
-
export FIRECRAWL_RETRY_INITIAL_DELAY=500 # Start with faster retries
|
|
261
232
|
```
|
|
262
233
|
|
|
263
234
|
### Usage with Claude Desktop
|
|
@@ -271,75 +242,19 @@ Add this to your `claude_desktop_config.json`:
|
|
|
271
242
|
"command": "npx",
|
|
272
243
|
"args": ["-y", "firecrawl-mcp"],
|
|
273
244
|
"env": {
|
|
274
|
-
"FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE"
|
|
275
|
-
|
|
276
|
-
"FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
|
|
277
|
-
"FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
|
|
278
|
-
"FIRECRAWL_RETRY_MAX_DELAY": "30000",
|
|
279
|
-
"FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",
|
|
280
|
-
|
|
281
|
-
"FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
|
|
282
|
-
"FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
|
|
245
|
+
"FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE"
|
|
283
246
|
}
|
|
284
247
|
}
|
|
285
248
|
}
|
|
286
249
|
}
|
|
287
250
|
```
|
|
288
251
|
|
|
289
|
-
### System Configuration
|
|
290
|
-
|
|
291
|
-
The server includes several configurable parameters that can be set via environment variables. Here are the default values if not configured:
|
|
292
|
-
|
|
293
|
-
```typescript
|
|
294
|
-
const CONFIG = {
|
|
295
|
-
retry: {
|
|
296
|
-
maxAttempts: 3, // Number of retry attempts for rate-limited requests
|
|
297
|
-
initialDelay: 1000, // Initial delay before first retry (in milliseconds)
|
|
298
|
-
maxDelay: 10000, // Maximum delay between retries (in milliseconds)
|
|
299
|
-
backoffFactor: 2, // Multiplier for exponential backoff
|
|
300
|
-
},
|
|
301
|
-
credit: {
|
|
302
|
-
warningThreshold: 1000, // Warn when credit usage reaches this level
|
|
303
|
-
criticalThreshold: 100, // Critical alert when credit usage reaches this level
|
|
304
|
-
},
|
|
305
|
-
};
|
|
306
|
-
```
|
|
307
|
-
|
|
308
|
-
These configurations control:
|
|
309
|
-
|
|
310
|
-
1. **Retry Behavior**
|
|
311
|
-
|
|
312
|
-
- Automatically retries failed requests due to rate limits
|
|
313
|
-
- Uses exponential backoff to avoid overwhelming the API
|
|
314
|
-
- Example: With default settings, retries will be attempted at:
|
|
315
|
-
- 1st retry: 1 second delay
|
|
316
|
-
- 2nd retry: 2 seconds delay
|
|
317
|
-
- 3rd retry: 4 seconds delay (capped at maxDelay)
|
|
318
|
-
|
|
319
|
-
2. **Credit Usage Monitoring**
|
|
320
|
-
- Tracks API credit consumption for cloud API usage
|
|
321
|
-
- Provides warnings at specified thresholds
|
|
322
|
-
- Helps prevent unexpected service interruption
|
|
323
|
-
- Example: With default settings:
|
|
324
|
-
- Warning at 1000 credits remaining
|
|
325
|
-
- Critical alert at 100 credits remaining
|
|
326
|
-
|
|
327
|
-
### Rate Limiting and Batch Processing
|
|
328
|
-
|
|
329
|
-
The server utilizes Firecrawl's built-in rate limiting and batch processing capabilities:
|
|
330
|
-
|
|
331
|
-
- Automatic rate limit handling with exponential backoff
|
|
332
|
-
- Efficient parallel processing for batch operations
|
|
333
|
-
- Smart request queuing and throttling
|
|
334
|
-
- Automatic retries for transient errors
|
|
335
|
-
|
|
336
252
|
## How to Choose a Tool
|
|
337
253
|
|
|
338
254
|
Use this guide to select the right tool for your task:
|
|
339
255
|
|
|
340
|
-
- **If you know the exact URL
|
|
341
|
-
|
|
342
|
-
- For many: use **batch_scrape**
|
|
256
|
+
- **If you know the exact URL you want:** use **scrape** (with JSON format for structured data)
|
|
257
|
+
- **If you have multiple known URLs:** call **scrape** for each URL. If you specifically need one bulk API operation, use the Firecrawl API batch endpoint outside MCP.
|
|
343
258
|
- **If you need to discover URLs on a site:** use **map**
|
|
344
259
|
- **If you want to search the web for info:** use **search**
|
|
345
260
|
- **If you need complex research across multiple unknown sources:** use **agent**
|
|
@@ -352,15 +267,18 @@ Use this guide to select the right tool for your task:
|
|
|
352
267
|
| ------------ | ---------------------------------------------- | ------------------------------ |
|
|
353
268
|
| scrape | Single page content | JSON (preferred) or markdown |
|
|
354
269
|
| interact | Interact with a URL or scraped page | Execution result + scrapeId for URL mode |
|
|
355
|
-
| batch_scrape | Multiple known URLs | JSON (preferred) or markdown[] |
|
|
356
270
|
| map | Discovering URLs on a site | URL[] |
|
|
357
|
-
| crawl | Multi-page extraction (with limits) |
|
|
271
|
+
| crawl | Multi-page extraction (with limits) | final crawl status/data after internal polling |
|
|
272
|
+
| parse | Files and hosted upload refs | markdown, JSON, or document output |
|
|
273
|
+
| extract | Structured extraction from URLs | JSON structured data |
|
|
358
274
|
| search | Web search for info | results[] |
|
|
359
275
|
| agent | Complex multi-source research | JSON (structured data) |
|
|
276
|
+
| monitor | Recurring page checks | monitor/check metadata and diffs |
|
|
277
|
+
| research | Paper and GitHub repository research | research results and repo matches |
|
|
360
278
|
|
|
361
279
|
### Format Selection Guide
|
|
362
280
|
|
|
363
|
-
When using `scrape
|
|
281
|
+
When using `scrape`, choose the right format:
|
|
364
282
|
|
|
365
283
|
- **JSON format (recommended for most cases):** Use when you need specific data from a page. Define a schema based on what you need to extract. This keeps responses small and avoids context window overflow.
|
|
366
284
|
- **Markdown format (use sparingly):** Only when you genuinely need the full page content, such as reading an entire article for summarization or analyzing page structure.
|
|
@@ -377,12 +295,12 @@ Scrape content from a single URL with advanced options.
|
|
|
377
295
|
|
|
378
296
|
**Not recommended for:**
|
|
379
297
|
|
|
380
|
-
- Extracting content from multiple pages (use
|
|
298
|
+
- Extracting content from multiple pages (use repeated scrape calls for known URLs, or map + scrape to discover URLs first, or crawl for full page content)
|
|
381
299
|
- When you're unsure which page contains the information (use search)
|
|
382
300
|
|
|
383
301
|
**Common mistakes:**
|
|
384
302
|
|
|
385
|
-
-
|
|
303
|
+
- Passing a list of URLs to one scrape call. Call scrape once per URL in MCP. If you specifically need one bulk API operation, use the Firecrawl API batch endpoint outside MCP.
|
|
386
304
|
- Using markdown format by default (use JSON format to extract only what you need).
|
|
387
305
|
|
|
388
306
|
**Choosing the right format:**
|
|
@@ -452,72 +370,7 @@ Scrape content from a single URL with advanced options.
|
|
|
452
370
|
|
|
453
371
|
- JSON structured data, markdown, branding profile, or other formats as specified.
|
|
454
372
|
|
|
455
|
-
### 2.
|
|
456
|
-
|
|
457
|
-
Scrape multiple URLs efficiently with built-in rate limiting and parallel processing.
|
|
458
|
-
|
|
459
|
-
**Best for:**
|
|
460
|
-
|
|
461
|
-
- Retrieving content from multiple pages, when you know exactly which pages to scrape.
|
|
462
|
-
|
|
463
|
-
**Not recommended for:**
|
|
464
|
-
|
|
465
|
-
- Discovering URLs (use map first if you don't know the URLs)
|
|
466
|
-
- Scraping a single page (use scrape)
|
|
467
|
-
|
|
468
|
-
**Common mistakes:**
|
|
469
|
-
|
|
470
|
-
- Using batch_scrape with too many URLs at once (may hit rate limits or token overflow)
|
|
471
|
-
|
|
472
|
-
**Prompt Example:**
|
|
473
|
-
|
|
474
|
-
> "Get the content of these three blog posts: [url1, url2, url3]."
|
|
475
|
-
|
|
476
|
-
**Usage Example:**
|
|
477
|
-
|
|
478
|
-
```json
|
|
479
|
-
{
|
|
480
|
-
"name": "firecrawl_batch_scrape",
|
|
481
|
-
"arguments": {
|
|
482
|
-
"urls": ["https://example1.com", "https://example2.com"],
|
|
483
|
-
"options": {
|
|
484
|
-
"formats": ["markdown"],
|
|
485
|
-
"onlyMainContent": true
|
|
486
|
-
}
|
|
487
|
-
}
|
|
488
|
-
}
|
|
489
|
-
```
|
|
490
|
-
|
|
491
|
-
**Returns:**
|
|
492
|
-
|
|
493
|
-
- Response includes operation ID for status checking:
|
|
494
|
-
|
|
495
|
-
```json
|
|
496
|
-
{
|
|
497
|
-
"content": [
|
|
498
|
-
{
|
|
499
|
-
"type": "text",
|
|
500
|
-
"text": "Batch operation queued with ID: batch_1. Use firecrawl_check_batch_status to check progress."
|
|
501
|
-
}
|
|
502
|
-
],
|
|
503
|
-
"isError": false
|
|
504
|
-
}
|
|
505
|
-
```
|
|
506
|
-
|
|
507
|
-
### 3. Check Batch Status (`firecrawl_check_batch_status`)
|
|
508
|
-
|
|
509
|
-
Check the status of a batch operation.
|
|
510
|
-
|
|
511
|
-
```json
|
|
512
|
-
{
|
|
513
|
-
"name": "firecrawl_check_batch_status",
|
|
514
|
-
"arguments": {
|
|
515
|
-
"id": "batch_1"
|
|
516
|
-
}
|
|
517
|
-
}
|
|
518
|
-
```
|
|
519
|
-
|
|
520
|
-
### 4. Map Tool (`firecrawl_map`)
|
|
373
|
+
### 2. Map Tool (`firecrawl_map`)
|
|
521
374
|
|
|
522
375
|
Map a website to discover all indexed URLs on the site.
|
|
523
376
|
|
|
@@ -528,7 +381,7 @@ Map a website to discover all indexed URLs on the site.
|
|
|
528
381
|
|
|
529
382
|
**Not recommended for:**
|
|
530
383
|
|
|
531
|
-
- When you already know which specific URL you need (use scrape
|
|
384
|
+
- When you already know which specific URL you need (use scrape)
|
|
532
385
|
- When you need the content of the pages (use scrape after mapping)
|
|
533
386
|
|
|
534
387
|
**Common mistakes:**
|
|
@@ -554,7 +407,7 @@ Map a website to discover all indexed URLs on the site.
|
|
|
554
407
|
|
|
555
408
|
- Array of URLs found on the site
|
|
556
409
|
|
|
557
|
-
###
|
|
410
|
+
### 3. Search Tool (`firecrawl_search`)
|
|
558
411
|
|
|
559
412
|
Search the web and optionally extract content from search results.
|
|
560
413
|
|
|
@@ -599,7 +452,7 @@ Search the web and optionally extract content from search results.
|
|
|
599
452
|
|
|
600
453
|
> "Find the latest research papers on AI published in 2023."
|
|
601
454
|
|
|
602
|
-
###
|
|
455
|
+
### 3b. Search Feedback Tool (`firecrawl_search_feedback`)
|
|
603
456
|
|
|
604
457
|
Sends structured feedback on a previous `firecrawl_search` result. The first feedback per search id refunds 1 credit and improves Firecrawl's search quality. Idempotent per search id.
|
|
605
458
|
|
|
@@ -641,7 +494,7 @@ Sends structured feedback on a previous `firecrawl_search` result. The first fee
|
|
|
641
494
|
|
|
642
495
|
- `{ success, feedbackId, creditsRefunded, alreadySubmitted? }` JSON.
|
|
643
496
|
|
|
644
|
-
###
|
|
497
|
+
### 3c. Generic Feedback Tool (`firecrawl_feedback`)
|
|
645
498
|
|
|
646
499
|
Sends structured feedback for a completed v2 endpoint job through `/v2/feedback`.
|
|
647
500
|
Use this for endpoint-level feedback on `scrape`, `parse`, `map`, or `search`
|
|
@@ -678,9 +531,9 @@ and small metadata objects. Do not include raw scrape/parse outputs.
|
|
|
678
531
|
|
|
679
532
|
- `{ success, feedbackId, creditsRefunded, creditsRefundedToday?, dailyRefundCap?, dailyCapReached?, alreadySubmitted?, warning? }` JSON.
|
|
680
533
|
|
|
681
|
-
###
|
|
534
|
+
### 4. Crawl Tool (`firecrawl_crawl`)
|
|
682
535
|
|
|
683
|
-
Starts
|
|
536
|
+
Starts a crawl job, polls until it reaches a terminal state, and returns the final crawl status/data.
|
|
684
537
|
|
|
685
538
|
**Best for:**
|
|
686
539
|
|
|
@@ -689,14 +542,14 @@ Starts an asynchronous crawl job on a website and extract content from all pages
|
|
|
689
542
|
**Not recommended for:**
|
|
690
543
|
|
|
691
544
|
- Extracting content from a single page (use scrape)
|
|
692
|
-
- When token limits are a concern (use map +
|
|
545
|
+
- When token limits are a concern (use map + scrape for tighter control)
|
|
693
546
|
- When you need fast results (crawling can be slow)
|
|
694
547
|
|
|
695
|
-
**Warning:** Crawl responses can be very large and may exceed token limits. Limit the crawl depth and number of pages, or use map +
|
|
548
|
+
**Warning:** Crawl responses can be very large and may exceed token limits. Limit the crawl depth and number of pages, or use map + scrape for tighter control.
|
|
696
549
|
|
|
697
550
|
**Common mistakes:**
|
|
698
551
|
|
|
699
|
-
- Setting limit or
|
|
552
|
+
- Setting limit or maxDiscoveryDepth too high (causes token overflow)
|
|
700
553
|
- Using crawl for a single page (use scrape instead)
|
|
701
554
|
|
|
702
555
|
**Prompt Example:**
|
|
@@ -710,7 +563,7 @@ Starts an asynchronous crawl job on a website and extract content from all pages
|
|
|
710
563
|
"name": "firecrawl_crawl",
|
|
711
564
|
"arguments": {
|
|
712
565
|
"url": "https://example.com/blog/*",
|
|
713
|
-
"
|
|
566
|
+
"maxDiscoveryDepth": 2,
|
|
714
567
|
"limit": 100,
|
|
715
568
|
"allowExternalLinks": false,
|
|
716
569
|
"deduplicateSimilarURLs": true
|
|
@@ -718,25 +571,14 @@ Starts an asynchronous crawl job on a website and extract content from all pages
|
|
|
718
571
|
}
|
|
719
572
|
```
|
|
720
573
|
|
|
721
|
-
**Returns:**
|
|
722
574
|
|
|
723
|
-
|
|
575
|
+
**Returns:**
|
|
724
576
|
|
|
725
|
-
|
|
726
|
-
{
|
|
727
|
-
"content": [
|
|
728
|
-
{
|
|
729
|
-
"type": "text",
|
|
730
|
-
"text": "Started crawl for: https://example.com/* with job ID: 550e8400-e29b-41d4-a716-446655440000. Use firecrawl_check_crawl_status to check progress."
|
|
731
|
-
}
|
|
732
|
-
],
|
|
733
|
-
"isError": false
|
|
734
|
-
}
|
|
735
|
-
```
|
|
577
|
+
- Final crawl status and data after internal polling, including `id`, `status`, `completed`, `total`, `creditsUsed`, `expiresAt`, `next`, and `data`. Use the returned `id` with `firecrawl_check_crawl_status` if you need to re-check the job later.
|
|
736
578
|
|
|
737
|
-
###
|
|
579
|
+
### 5. Check Crawl Status (`firecrawl_check_crawl_status`)
|
|
738
580
|
|
|
739
|
-
Check the status of
|
|
581
|
+
Check the status and results of an existing crawl job by ID.
|
|
740
582
|
|
|
741
583
|
```json
|
|
742
584
|
{
|
|
@@ -751,7 +593,33 @@ Check the status of a crawl job.
|
|
|
751
593
|
|
|
752
594
|
- Response includes the status of the crawl job:
|
|
753
595
|
|
|
754
|
-
###
|
|
596
|
+
### 6. Parse Tool (`firecrawl_parse`)
|
|
597
|
+
|
|
598
|
+
Parse local files or hosted upload references with Firecrawl's `/v2/parse` endpoint.
|
|
599
|
+
|
|
600
|
+
**Best for:** PDFs, Word documents, spreadsheets, HTML files, and other documents that need markdown or structured JSON output. Hosted MCP supports a two-step upload-ref flow; local direct file reads require a self-hosted `FIRECRAWL_API_URL`.
|
|
601
|
+
|
|
602
|
+
**Not recommended for:** Remote URLs (use scrape), multiple files in one call (call parse once per file), or browser-only actions such as screenshots and clicks.
|
|
603
|
+
|
|
604
|
+
**Hosted MCP flow:** Hosted MCP cannot read the caller's filesystem directly. Call `firecrawl_parse` with `filePath` to receive a short-lived upload command and `nextToolCall`, upload the file locally, then call `firecrawl_parse` again with the returned `uploadRef`. Minting the hosted upload URL requires Firecrawl auth or keyless eligibility. In local `npx firecrawl-mcp` mode, direct file parsing currently requires `FIRECRAWL_API_URL` pointing to a self-hosted Firecrawl API; a plain cloud API-key-only local server cannot read and upload files through this tool.
|
|
605
|
+
|
|
606
|
+
**Usage Example:**
|
|
607
|
+
|
|
608
|
+
```json
|
|
609
|
+
{
|
|
610
|
+
"name": "firecrawl_parse",
|
|
611
|
+
"arguments": {
|
|
612
|
+
"filePath": "/absolute/path/to/document.pdf",
|
|
613
|
+
"formats": ["markdown"],
|
|
614
|
+
"parsers": ["pdf"],
|
|
615
|
+
"zeroDataRetention": true
|
|
616
|
+
}
|
|
617
|
+
}
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
**Returns:** Parsed document content or hosted upload instructions with a `nextToolCall`.
|
|
621
|
+
|
|
622
|
+
### 7. Extract Tool (`firecrawl_extract`)
|
|
755
623
|
|
|
756
624
|
Extract structured information from web pages using LLM capabilities. Supports both cloud AI and self-hosted LLM extraction.
|
|
757
625
|
|
|
@@ -824,7 +692,7 @@ When using a self-hosted instance, the extraction will use your configured LLM.
|
|
|
824
692
|
}
|
|
825
693
|
```
|
|
826
694
|
|
|
827
|
-
###
|
|
695
|
+
### 8. Agent Tool (`firecrawl_agent`)
|
|
828
696
|
|
|
829
697
|
Autonomous web research agent. This is a separate AI agent layer that independently browses the internet, searches for information, navigates through pages, and extracts structured data based on your query.
|
|
830
698
|
|
|
@@ -905,7 +773,7 @@ Then poll with `firecrawl_agent_status` using the returned job ID.
|
|
|
905
773
|
|
|
906
774
|
- Job ID for status checking. Use `firecrawl_agent_status` to poll for results.
|
|
907
775
|
|
|
908
|
-
###
|
|
776
|
+
### 9. Check Agent Status (`firecrawl_agent_status`)
|
|
909
777
|
|
|
910
778
|
Check the status of an agent job and retrieve results when complete. Use this to poll for results after starting an agent.
|
|
911
779
|
|
|
@@ -926,7 +794,60 @@ Check the status of an agent job and retrieve results when complete. Use this to
|
|
|
926
794
|
- `completed`: Research finished - response includes the extracted data
|
|
927
795
|
- `failed`: An error occurred
|
|
928
796
|
|
|
929
|
-
###
|
|
797
|
+
### 10. Interact Tool (`firecrawl_interact`)
|
|
798
|
+
|
|
799
|
+
Interact with a fresh URL or with a page that was already opened by `firecrawl_scrape`.
|
|
800
|
+
|
|
801
|
+
**Best for:** Clicking, typing, navigating, and extracting state from dynamic pages without restoring the deprecated browser tools.
|
|
802
|
+
|
|
803
|
+
**Usage options:**
|
|
804
|
+
|
|
805
|
+
- Pass `url` to scrape and open a page for interaction in one MCP call.
|
|
806
|
+
- Pass `scrapeId` to continue interacting with an existing scraped page.
|
|
807
|
+
- Pass exactly one of `url` or `scrapeId`, plus either `prompt` or `code`.
|
|
808
|
+
|
|
809
|
+
**Usage Example:**
|
|
810
|
+
|
|
811
|
+
```json
|
|
812
|
+
{
|
|
813
|
+
"name": "firecrawl_interact",
|
|
814
|
+
"arguments": {
|
|
815
|
+
"url": "https://example.com",
|
|
816
|
+
"prompt": "Click the pricing link and summarize the visible plans"
|
|
817
|
+
}
|
|
818
|
+
}
|
|
819
|
+
```
|
|
820
|
+
|
|
821
|
+
**Returns:** Interaction result and, for URL mode, the derived `scrapeId` for follow-up or cleanup.
|
|
822
|
+
|
|
823
|
+
### 11. Stop Interact Tool (`firecrawl_interact_stop`)
|
|
824
|
+
|
|
825
|
+
Stop an interact session for a scraped page when you are done interacting.
|
|
826
|
+
|
|
827
|
+
```json
|
|
828
|
+
{
|
|
829
|
+
"name": "firecrawl_interact_stop",
|
|
830
|
+
"arguments": {
|
|
831
|
+
"scrapeId": "scrape-id-here"
|
|
832
|
+
}
|
|
833
|
+
}
|
|
834
|
+
```
|
|
835
|
+
|
|
836
|
+
### 12. Research Tools (`firecrawl_research_*`)
|
|
837
|
+
|
|
838
|
+
Search and inspect papers and GitHub repositories through the research MCP tools.
|
|
839
|
+
|
|
840
|
+
**Available research tools:**
|
|
841
|
+
|
|
842
|
+
- `firecrawl_research_search_papers`: search research papers.
|
|
843
|
+
- `firecrawl_research_inspect_paper`: inspect one paper.
|
|
844
|
+
- `firecrawl_research_related_papers`: find related papers.
|
|
845
|
+
- `firecrawl_research_read_paper`: read paper content.
|
|
846
|
+
- `firecrawl_research_search_github`: search GitHub repositories.
|
|
847
|
+
|
|
848
|
+
**Best for:** Literature review, paper lookup, and repository discovery workflows where the agent needs a focused research surface instead of general web scraping.
|
|
849
|
+
|
|
850
|
+
### 13. Monitor Tools (`firecrawl_monitor_*`)
|
|
930
851
|
|
|
931
852
|
Create and manage recurring page monitors. Monitors run scheduled scrapes or crawls, diff each result against the last retained snapshot, and can notify by webhook or email.
|
|
932
853
|
|
|
@@ -991,6 +912,7 @@ Pass `body` when you need crawl targets, JSON change tracking, custom retention,
|
|
|
991
912
|
- `firecrawl_monitor_get`: get one monitor.
|
|
992
913
|
- `firecrawl_monitor_update`: update fields including `goal`, `judgeEnabled`, `webhook`, and `notification`.
|
|
993
914
|
- `firecrawl_monitor_run`: trigger a check now.
|
|
915
|
+
- `firecrawl_monitor_delete`: delete a monitor (destructive; only call when the user intends to remove it).
|
|
994
916
|
- `firecrawl_monitor_checks`: list checks, optionally filtered by status.
|
|
995
917
|
- `firecrawl_monitor_check`: get page-level results, including `diff`, `snapshot`, `judgment.meaningful`, and `judgment.meaningfulChanges`.
|
|
996
918
|
|
|
@@ -1000,7 +922,6 @@ The server includes comprehensive logging:
|
|
|
1000
922
|
|
|
1001
923
|
- Operation status and progress
|
|
1002
924
|
- Performance metrics
|
|
1003
|
-
- Credit usage monitoring
|
|
1004
925
|
- Rate limit tracking
|
|
1005
926
|
- Error conditions
|
|
1006
927
|
|
|
@@ -1009,19 +930,15 @@ Example log messages:
|
|
|
1009
930
|
```
|
|
1010
931
|
[INFO] Firecrawl MCP Server initialized successfully
|
|
1011
932
|
[INFO] Starting scrape for URL: https://example.com
|
|
1012
|
-
[
|
|
1013
|
-
[WARNING] Credit usage has reached warning threshold
|
|
1014
|
-
[ERROR] Rate limit exceeded, retrying in 2s...
|
|
933
|
+
[ERROR] Rate limit exceeded
|
|
1015
934
|
```
|
|
1016
935
|
|
|
1017
936
|
## Error Handling
|
|
1018
937
|
|
|
1019
938
|
The server provides robust error handling:
|
|
1020
939
|
|
|
1021
|
-
-
|
|
1022
|
-
- Rate limit handling with backoff
|
|
940
|
+
- API rate-limit errors surfaced to the MCP client
|
|
1023
941
|
- Detailed error messages
|
|
1024
|
-
- Credit usage warnings
|
|
1025
942
|
- Network resilience
|
|
1026
943
|
|
|
1027
944
|
Example error response:
|
|
@@ -1031,7 +948,7 @@ Example error response:
|
|
|
1031
948
|
"content": [
|
|
1032
949
|
{
|
|
1033
950
|
"type": "text",
|
|
1034
|
-
"text": "Error: Rate limit exceeded
|
|
951
|
+
"text": "Error: Rate limit exceeded"
|
|
1035
952
|
}
|
|
1036
953
|
],
|
|
1037
954
|
"isError": true
|
package/dist/index.js
CHANGED
|
@@ -1385,9 +1385,11 @@ function buildMonitorCreateBody(args2) {
|
|
|
1385
1385
|
args2.page,
|
|
1386
1386
|
args2.pages
|
|
1387
1387
|
);
|
|
1388
|
-
|
|
1388
|
+
const queries = Array.isArray(args2.queries) ? args2.queries.filter((q) => typeof q === "string").map((q) => q.trim()).filter(Boolean) : [];
|
|
1389
|
+
const isSearch = queries.length > 0;
|
|
1390
|
+
if (urls.length === 0 && !isSearch) {
|
|
1389
1391
|
throw new Error(
|
|
1390
|
-
"firecrawl_monitor_create requires either `body`, `page`, or `
|
|
1392
|
+
"firecrawl_monitor_create requires either `body`, `page`/`pages`, or `queries`."
|
|
1391
1393
|
);
|
|
1392
1394
|
}
|
|
1393
1395
|
const goal = typeof args2.goal === "string" ? args2.goal.trim() : "";
|
|
@@ -1396,6 +1398,25 @@ function buildMonitorCreateBody(args2) {
|
|
|
1396
1398
|
"firecrawl_monitor_create shorthand requires `goal`. Use `body` for advanced requests without a goal."
|
|
1397
1399
|
);
|
|
1398
1400
|
}
|
|
1401
|
+
let target;
|
|
1402
|
+
if (isSearch) {
|
|
1403
|
+
const includeDomains = Array.isArray(args2.includeDomains) ? args2.includeDomains.filter(
|
|
1404
|
+
(d) => typeof d === "string"
|
|
1405
|
+
) : void 0;
|
|
1406
|
+
const excludeDomains = Array.isArray(args2.excludeDomains) ? args2.excludeDomains.filter(
|
|
1407
|
+
(d) => typeof d === "string"
|
|
1408
|
+
) : void 0;
|
|
1409
|
+
target = {
|
|
1410
|
+
type: "search",
|
|
1411
|
+
queries,
|
|
1412
|
+
...typeof args2.searchWindow === "string" && args2.searchWindow.trim() ? { searchWindow: args2.searchWindow.trim() } : {},
|
|
1413
|
+
...typeof args2.maxResults === "number" ? { maxResults: args2.maxResults } : {},
|
|
1414
|
+
...includeDomains && includeDomains.length > 0 ? { includeDomains } : {},
|
|
1415
|
+
...excludeDomains && excludeDomains.length > 0 ? { excludeDomains } : {}
|
|
1416
|
+
};
|
|
1417
|
+
} else {
|
|
1418
|
+
target = { type: "scrape", urls };
|
|
1419
|
+
}
|
|
1399
1420
|
const webhookUrl = typeof args2.webhookUrl === "string" ? args2.webhookUrl.trim() : "";
|
|
1400
1421
|
const email = typeof args2.email === "string" && args2.email.trim() ? {
|
|
1401
1422
|
email: {
|
|
@@ -1405,13 +1426,13 @@ function buildMonitorCreateBody(args2) {
|
|
|
1405
1426
|
}
|
|
1406
1427
|
} : void 0;
|
|
1407
1428
|
return {
|
|
1408
|
-
name: typeof args2.name === "string" && args2.name.trim() ? args2.name.trim() : `Monitor ${urls[0]}`,
|
|
1429
|
+
name: typeof args2.name === "string" && args2.name.trim() ? args2.name.trim() : isSearch ? `Monitor ${queries[0]}` : `Monitor ${urls[0]}`,
|
|
1409
1430
|
schedule: {
|
|
1410
1431
|
text: typeof args2.scheduleText === "string" && args2.scheduleText.trim() ? args2.scheduleText.trim() : "every 30 minutes",
|
|
1411
1432
|
timezone: typeof args2.timezone === "string" && args2.timezone.trim() ? args2.timezone.trim() : "UTC"
|
|
1412
1433
|
},
|
|
1413
1434
|
goal,
|
|
1414
|
-
targets: [
|
|
1435
|
+
targets: [target],
|
|
1415
1436
|
...email ? { notification: email } : {},
|
|
1416
1437
|
...webhookUrl ? {
|
|
1417
1438
|
webhook: {
|
|
@@ -1434,20 +1455,38 @@ function registerMonitorTools(server2) {
|
|
|
1434
1455
|
// Additive; creates a new monitor without deleting existing monitors or external content.
|
|
1435
1456
|
},
|
|
1436
1457
|
description: `
|
|
1437
|
-
Create a Firecrawl monitor \u2014 a recurring scrape or
|
|
1458
|
+
Create a Firecrawl monitor \u2014 a recurring scrape, crawl, or search that diffs each result against the last retained snapshot.
|
|
1438
1459
|
|
|
1439
|
-
Prefer the simple path: pass \`page\` or \`pages\` plus \`goal
|
|
1460
|
+
Prefer the simple path: pass \`page\` or \`pages\` plus \`goal\` to monitor specific URLs, OR pass \`queries\` plus \`goal\` to monitor web search results for new/changed hits. The tool will create the monitor with a 30-minute schedule and meaningful-change judging enabled by the API. Use \`body\` only for advanced requests such as crawl targets, JSON change tracking, custom retention, or manual \`judgeEnabled\` control.
|
|
1440
1461
|
|
|
1441
1462
|
Meaningful-change judge: set \`goal\` to a plain-language description of what the user actually cares about. \`judgeEnabled\` defaults to true when \`goal\` is set, so providing \`goal\` is enough. Page webhooks expose \`isMeaningful\` and \`judgment\` on \`monitor.page\` events.
|
|
1442
1463
|
|
|
1443
1464
|
Simple fields:
|
|
1444
1465
|
- \`page\`: one page URL to monitor.
|
|
1445
1466
|
- \`pages\`: multiple page URLs to monitor.
|
|
1446
|
-
- \`
|
|
1467
|
+
- \`queries\`: one or more search queries (1-12) to monitor instead of fixed URLs. Each check runs the searches and diffs the result set, so you get alerted when new or changed results appear. Mutually exclusive with \`page\`/\`pages\` in the simple path.
|
|
1468
|
+
- \`searchWindow\`: optional recency window for search targets \u2014 one of \`5m\`, \`15m\`, \`1h\`, \`6h\`, \`24h\`, \`7d\` (default \`24h\`).
|
|
1469
|
+
- \`maxResults\`: optional max results per search, 1-50 (default 10).
|
|
1470
|
+
- \`includeDomains\` / \`excludeDomains\`: optional domain allow/deny lists for search targets.
|
|
1471
|
+
- \`goal\`: plain-English instruction for what changes matter. Required for the simple path (and always required when \`queries\` are set \u2014 web monitors must have a goal).
|
|
1447
1472
|
- \`scheduleText\`: optional natural-language schedule, default \`every 30 minutes\`.
|
|
1448
1473
|
- \`email\`: optional email recipient for summaries.
|
|
1449
1474
|
- \`webhookUrl\`: optional webhook URL. Configures \`monitor.page\` and \`monitor.check.completed\`.
|
|
1450
1475
|
|
|
1476
|
+
**Search-mode example:**
|
|
1477
|
+
|
|
1478
|
+
\`\`\`json
|
|
1479
|
+
{
|
|
1480
|
+
"name": "firecrawl_monitor_create",
|
|
1481
|
+
"arguments": {
|
|
1482
|
+
"queries": ["new LLM release", "frontier model launch"],
|
|
1483
|
+
"goal": "Notify me about major new LLM model releases.",
|
|
1484
|
+
"searchWindow": "24h",
|
|
1485
|
+
"maxResults": 10
|
|
1486
|
+
}
|
|
1487
|
+
}
|
|
1488
|
+
\`\`\`
|
|
1489
|
+
|
|
1451
1490
|
Goal guidance:
|
|
1452
1491
|
- Expand the user's one-line monitoring intent into a concise 2-3 sentence monitor goal.
|
|
1453
1492
|
- State what should trigger an alert, restate any scope the user gave, and include intent-specific exclusions only when obvious from the user's request.
|
|
@@ -1456,7 +1495,7 @@ Goal guidance:
|
|
|
1456
1495
|
- If the user says they do not care about something, include that explicitly. It is okay to ask whether they want to ignore specific noise when it is likely to matter.
|
|
1457
1496
|
- Do not invent page-specific sections, thresholds, entities, or business rules unless the user mentioned them.
|
|
1458
1497
|
|
|
1459
|
-
Full \`body\` requests require: \`name\`, \`schedule\` (with \`cron\` or \`text\`), and \`targets\` (one or more \`{ type: 'scrape', urls: [...] }
|
|
1498
|
+
Full \`body\` requests require: \`name\`, \`schedule\` (with \`cron\` or \`text\`), and \`targets\` (one or more \`{ type: 'scrape', urls: [...] }\`, \`{ type: 'crawl', url: '...' }\`, or \`{ type: 'search', queries: [...], searchWindow?, maxResults?, includeDomains?, excludeDomains? }\`). Optional: \`goal\` (required when any search target is present), \`judgeEnabled\`, \`webhook\`, \`notification\`, \`retentionDays\`.
|
|
1460
1499
|
|
|
1461
1500
|
**Markdown-mode (default):** Each check produces a unified text diff of the page's markdown. No extra configuration needed.
|
|
1462
1501
|
|
|
@@ -1532,6 +1571,11 @@ Full \`body\` requests require: \`name\`, \`schedule\` (with \`cron\` or \`text\
|
|
|
1532
1571
|
body: z2.record(z2.string(), z2.any()).optional(),
|
|
1533
1572
|
page: z2.string().optional(),
|
|
1534
1573
|
pages: z2.array(z2.string()).optional(),
|
|
1574
|
+
queries: z2.array(z2.string()).optional(),
|
|
1575
|
+
searchWindow: z2.enum(["5m", "15m", "1h", "6h", "24h", "7d"]).optional(),
|
|
1576
|
+
maxResults: z2.number().int().min(1).max(50).optional(),
|
|
1577
|
+
includeDomains: z2.array(z2.string()).optional(),
|
|
1578
|
+
excludeDomains: z2.array(z2.string()).optional(),
|
|
1535
1579
|
goal: z2.string().optional(),
|
|
1536
1580
|
name: z2.string().optional(),
|
|
1537
1581
|
scheduleText: z2.string().optional(),
|
|
@@ -3484,20 +3528,20 @@ Do not store multi-MB outputs in feedback. Use concise notes, issue codes, URLs,
|
|
|
3484
3528
|
server.addTool({
|
|
3485
3529
|
name: "firecrawl_crawl",
|
|
3486
3530
|
annotations: {
|
|
3487
|
-
title: "
|
|
3531
|
+
title: "Run a site crawl",
|
|
3488
3532
|
readOnlyHint: false,
|
|
3489
|
-
// Starts
|
|
3533
|
+
// Starts a server-side crawl job and polls until the job reaches a terminal state.
|
|
3490
3534
|
openWorldHint: true,
|
|
3491
3535
|
// Crawls user-specified URLs across the public web.
|
|
3492
3536
|
destructiveHint: false
|
|
3493
3537
|
// Reads pages from target sites; does not delete or alter external websites.
|
|
3494
3538
|
},
|
|
3495
3539
|
description: `
|
|
3496
|
-
Starts a crawl job on a website and
|
|
3540
|
+
Starts a crawl job on a website, polls until it reaches a terminal state, and returns the final crawl status/data.
|
|
3497
3541
|
|
|
3498
3542
|
**Best for:** Extracting content from multiple related pages, when you need comprehensive coverage.
|
|
3499
|
-
**Not recommended for:** Extracting content from a single page (use scrape); when token limits are a concern (use map +
|
|
3500
|
-
**Warning:** Crawl responses can be very large and may exceed token limits. Limit the crawl depth and number of pages, or use map +
|
|
3543
|
+
**Not recommended for:** Extracting content from a single page (use scrape); when token limits are a concern (use map + scrape for tighter control); when you need fast results (crawling can be slow).
|
|
3544
|
+
**Warning:** Crawl responses can be very large and may exceed token limits. Limit the crawl depth and number of pages, or use map + scrape for tighter control.
|
|
3501
3545
|
**Common mistakes:** Setting limit or maxDiscoveryDepth too high (causes token overflow) or too low (causes missing pages); using crawl for a single page (use scrape instead). Using a /* wildcard is not recommended.
|
|
3502
3546
|
**Prompt Example:** "Get all blog posts from the first two levels of example.com/blog."
|
|
3503
3547
|
**Usage Example:**
|
|
@@ -3514,7 +3558,7 @@ server.addTool({
|
|
|
3514
3558
|
}
|
|
3515
3559
|
}
|
|
3516
3560
|
\`\`\`
|
|
3517
|
-
**Returns:**
|
|
3561
|
+
**Returns:** Final crawl status and data after internal polling, including the crawl id. Use firecrawl_check_crawl_status only when you need to re-check an existing crawl ID later.
|
|
3518
3562
|
${SAFE_MODE ? "**Safe Mode:** Read-only crawling. Webhooks and interactive actions are disabled for security." : ""}
|
|
3519
3563
|
`,
|
|
3520
3564
|
parameters: z4.object({
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "firecrawl-mcp",
|
|
3
|
-
"version": "3.22.
|
|
3
|
+
"version": "3.22.1",
|
|
4
4
|
"description": "MCP server for Firecrawl — search, scrape, and interact with the web. Supports both cloud and self-hosted instances. Features include web search, scraping, page interaction, batch processing, and LLM-powered content analysis.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"mcpName": "io.github.firecrawl/firecrawl-mcp-server",
|