firecrawl 2.16.5__tar.gz → 3.0.3__tar.gz

{firecrawl-2.16.5 → firecrawl-3.0.3}/PKG-INFO

@@ -1,15 +1,15 @@
 Metadata-Version: 2.1
 Name: firecrawl
-Version: 2.16.5
+Version: 3.0.3
 Summary: Python SDK for Firecrawl API
-Home-page: https://github.com/mendableai/firecrawl
+Home-page: https://github.com/firecrawl/firecrawl
 Author: Mendable.ai
 Author-email: "Mendable.ai" <nick@mendable.ai>
 Maintainer-email: "Mendable.ai" <nick@mendable.ai>
 License: MIT License
 Project-URL: Documentation, https://docs.firecrawl.dev
-Project-URL: Source, https://github.com/mendableai/firecrawl
-Project-URL: Tracker, https://github.com/mendableai/firecrawl/issues
+Project-URL: Source, https://github.com/firecrawl/firecrawl
+Project-URL: Tracker, https://github.com/firecrawl/firecrawl/issues
 Keywords: SDK,API,firecrawl
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Web Environment
@@ -34,6 +34,7 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: requests
+Requires-Dist: httpx
 Requires-Dist: python-dotenv
 Requires-Dist: websockets
 Requires-Dist: nest-asyncio
@@ -55,24 +56,25 @@ pip install firecrawl-py
 ## Usage
 
 1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
-2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.
 
 Here's an example of how to use the SDK:
 
 ```python 
-from firecrawl import FirecrawlApp, ScrapeOptions
+from firecrawl import Firecrawl
+from firecrawl.types import ScrapeOptions
 
-app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
 
-# Scrape a website:
-data = app.scrape_url(
+# Scrape a website (v2):
+data = firecrawl.scrape(
   'https://firecrawl.dev', 
   formats=['markdown', 'html']
 )
 print(data)
 
-# Crawl a website:
-crawl_status = app.crawl_url(
+# Crawl a website (v2 waiter):
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html'])
@@ -82,20 +84,20 @@ print(crawl_status)
 
 ### Scraping a URL
 
-To scrape a single URL, use the `scrape_url` method. It takes the URL as a parameter and returns the scraped data as a dictionary.
+To scrape a single URL, use the `scrape` method. It takes the URL as a parameter and returns a document with the requested formats.
 
 ```python 
-# Scrape a website:
-scrape_result = app.scrape_url('firecrawl.dev', formats=['markdown', 'html'])
+# Scrape a website (v2):
+scrape_result = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
 print(scrape_result)
 ```
 
 ### Crawling a Website
 
-To crawl a website, use the `crawl_url` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To crawl a website, use the `crawl` method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.
 
 ```python 
-crawl_status = app.crawl_url(
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
@@ -108,23 +110,23 @@ print(crawl_status)
 
 <Tip>Looking for async operations? Check out the [Async Class](#async-class) section below.</Tip>
 
-To crawl a website asynchronously, use the `crawl_url_async` method. It returns the crawl `ID` which you can use to check the status of the crawl job. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To enqueue a crawl asynchronously, use `start_crawl`. It returns the crawl `ID` which you can use to check the status of the crawl job.
 
 ```python 
-crawl_status = app.async_crawl_url(
+crawl_job = firecrawl.start_crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
 )
-print(crawl_status)
+print(crawl_job)
 ```
 
 ### Checking Crawl Status
 
-To check the status of a crawl job, use the `check_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
+To check the status of a crawl job, use the `get_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
 
 ```python 
-crawl_status = app.check_crawl_status("<crawl_id>")
+crawl_status = firecrawl.get_crawl_status("<crawl_id>")
 print(crawl_status)
 ```
 
@@ -133,17 +135,17 @@ print(crawl_status)
 To cancel an asynchronous crawl job, use the `cancel_crawl` method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.
 
 ```python 
-cancel_crawl = app.cancel_crawl(id)
+cancel_crawl = firecrawl.cancel_crawl(id)
 print(cancel_crawl)
 ```
 
 ### Map a Website
 
-Use `map_url` to generate a list of URLs from a website. The `params` argument let you customize the mapping process, including options to exclude subdomains or to utilize the sitemap.
+Use `map` to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.
 
 ```python 
-# Map a website:
-map_result = app.map_url('https://firecrawl.dev')
+# Map a website (v2):
+map_result = firecrawl.map('https://firecrawl.dev')
 print(map_result)
 ```
 
@@ -194,20 +196,35 @@ The SDK handles errors returned by the Firecrawl API and raises appropriate exce
 
 ## Async Class
 
-For async operations, you can use the `AsyncFirecrawlApp` class. Its methods are the same as the `FirecrawlApp` class, but they don't block the main thread.
+For async operations, you can use the `AsyncFirecrawl` class. Its methods mirror the `Firecrawl` class, but you `await` them.
 
 ```python 
-from firecrawl import AsyncFirecrawlApp
+from firecrawl import AsyncFirecrawl
 
-app = AsyncFirecrawlApp(api_key="YOUR_API_KEY")
+firecrawl = AsyncFirecrawl(api_key="YOUR_API_KEY")
 
-# Async Scrape
+# Async Scrape (v2)
 async def example_scrape():
-  scrape_result = await app.scrape_url(url="https://example.com")
+  scrape_result = await firecrawl.scrape(url="https://example.com")
   print(scrape_result)
 
-# Async Crawl
+# Async Crawl (v2)
 async def example_crawl():
-  crawl_result = await app.crawl_url(url="https://example.com")
+  crawl_result = await firecrawl.crawl(url="https://example.com")
   print(crawl_result)
 ```
+
+## v1 compatibility
+
+For legacy code paths, v1 remains available under `firecrawl.v1` with the original method names.
+
+```python
+from firecrawl import Firecrawl
+
+firecrawl = Firecrawl(api_key="YOUR_API_KEY")
+
+# v1 methods (feature‑frozen)
+doc_v1 = firecrawl.v1.scrape_url('https://firecrawl.dev', formats=['markdown', 'html'])
+crawl_v1 = firecrawl.v1.crawl_url('https://firecrawl.dev', limit=100)
+map_v1 = firecrawl.v1.map_url('https://firecrawl.dev')
+```

{firecrawl-2.16.5 → firecrawl-3.0.3}/README.md

@@ -13,24 +13,25 @@ pip install firecrawl-py
 ## Usage
 
 1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
-2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.
 
 Here's an example of how to use the SDK:
 
 ```python 
-from firecrawl import FirecrawlApp, ScrapeOptions
+from firecrawl import Firecrawl
+from firecrawl.types import ScrapeOptions
 
-app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
 
-# Scrape a website:
-data = app.scrape_url(
+# Scrape a website (v2):
+data = firecrawl.scrape(
   'https://firecrawl.dev', 
   formats=['markdown', 'html']
 )
 print(data)
 
-# Crawl a website:
-crawl_status = app.crawl_url(
+# Crawl a website (v2 waiter):
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html'])
@@ -40,20 +41,20 @@ print(crawl_status)
 
 ### Scraping a URL
 
-To scrape a single URL, use the `scrape_url` method. It takes the URL as a parameter and returns the scraped data as a dictionary.
+To scrape a single URL, use the `scrape` method. It takes the URL as a parameter and returns a document with the requested formats.
 
 ```python 
-# Scrape a website:
-scrape_result = app.scrape_url('firecrawl.dev', formats=['markdown', 'html'])
+# Scrape a website (v2):
+scrape_result = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
 print(scrape_result)
 ```
 
 ### Crawling a Website
 
-To crawl a website, use the `crawl_url` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To crawl a website, use the `crawl` method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.
 
 ```python 
-crawl_status = app.crawl_url(
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
@@ -66,23 +67,23 @@ print(crawl_status)
 
 <Tip>Looking for async operations? Check out the [Async Class](#async-class) section below.</Tip>
 
-To crawl a website asynchronously, use the `crawl_url_async` method. It returns the crawl `ID` which you can use to check the status of the crawl job. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To enqueue a crawl asynchronously, use `start_crawl`. It returns the crawl `ID` which you can use to check the status of the crawl job.
 
 ```python 
-crawl_status = app.async_crawl_url(
+crawl_job = firecrawl.start_crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
 )
-print(crawl_status)
+print(crawl_job)
 ```
 
 ### Checking Crawl Status
 
-To check the status of a crawl job, use the `check_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
+To check the status of a crawl job, use the `get_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
 
 ```python 
-crawl_status = app.check_crawl_status("<crawl_id>")
+crawl_status = firecrawl.get_crawl_status("<crawl_id>")
 print(crawl_status)
 ```
 
@@ -91,17 +92,17 @@ print(crawl_status)
 To cancel an asynchronous crawl job, use the `cancel_crawl` method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.
 
 ```python 
-cancel_crawl = app.cancel_crawl(id)
+cancel_crawl = firecrawl.cancel_crawl(id)
 print(cancel_crawl)
 ```
 
 ### Map a Website
 
-Use `map_url` to generate a list of URLs from a website. The `params` argument let you customize the mapping process, including options to exclude subdomains or to utilize the sitemap.
+Use `map` to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.
 
 ```python 
-# Map a website:
-map_result = app.map_url('https://firecrawl.dev')
+# Map a website (v2):
+map_result = firecrawl.map('https://firecrawl.dev')
 print(map_result)
 ```
 
@@ -152,20 +153,35 @@ The SDK handles errors returned by the Firecrawl API and raises appropriate exce
 
 ## Async Class
 
-For async operations, you can use the `AsyncFirecrawlApp` class. Its methods are the same as the `FirecrawlApp` class, but they don't block the main thread.
+For async operations, you can use the `AsyncFirecrawl` class. Its methods mirror the `Firecrawl` class, but you `await` them.
 
 ```python 
-from firecrawl import AsyncFirecrawlApp
+from firecrawl import AsyncFirecrawl
 
-app = AsyncFirecrawlApp(api_key="YOUR_API_KEY")
+firecrawl = AsyncFirecrawl(api_key="YOUR_API_KEY")
 
-# Async Scrape
+# Async Scrape (v2)
 async def example_scrape():
-  scrape_result = await app.scrape_url(url="https://example.com")
+  scrape_result = await firecrawl.scrape(url="https://example.com")
   print(scrape_result)
 
-# Async Crawl
+# Async Crawl (v2)
 async def example_crawl():
-  crawl_result = await app.crawl_url(url="https://example.com")
+  crawl_result = await firecrawl.crawl(url="https://example.com")
   print(crawl_result)
+```
+
+## v1 compatibility
+
+For legacy code paths, v1 remains available under `firecrawl.v1` with the original method names.
+
+```python
+from firecrawl import Firecrawl
+
+firecrawl = Firecrawl(api_key="YOUR_API_KEY")
+
+# v1 methods (feature‑frozen)
+doc_v1 = firecrawl.v1.scrape_url('https://firecrawl.dev', formats=['markdown', 'html'])
+crawl_v1 = firecrawl.v1.crawl_url('https://firecrawl.dev', limit=100)
+map_v1 = firecrawl.v1.map_url('https://firecrawl.dev')
 ```

{firecrawl-2.16.5 → firecrawl-3.0.3}/firecrawl/__init__.py

@@ -1,19 +1,23 @@
 """
-This is the Firecrawl package.
+Firecrawl Python SDK
 
-This package provides a Python SDK for interacting with the Firecrawl API.
-It includes methods to scrape URLs, perform searches, initiate and monitor crawl jobs,
-and check the status of these jobs.
-
-For more information visit https://github.com/firecrawl/
 """
 
 import logging
 import os
 
-from .firecrawl import FirecrawlApp, AsyncFirecrawlApp, JsonConfig, ScrapeOptions, ChangeTrackingOptions # noqa
+from .client import Firecrawl, AsyncFirecrawl, FirecrawlApp, AsyncFirecrawlApp
+from .v2.watcher import Watcher
+from .v2.watcher_async import AsyncWatcher
+from .v1 import (
+    V1FirecrawlApp,
+    AsyncV1FirecrawlApp,
+    V1JsonConfig,
+    V1ScrapeOptions,
+    V1ChangeTrackingOptions,
+)
 
-__version__ = "2.16.5"
+__version__ = "3.0.3"
 
 # Define the logger for the Firecrawl project
 logger: logging.Logger = logging.getLogger("firecrawl")
@@ -27,17 +31,14 @@ def _configure_logger() -> None:
     format to the firecrawl logger.
     """
     try:
-        # Create the formatter
         formatter = logging.Formatter(
             "[%(asctime)s - %(name)s:%(lineno)d - %(levelname)s] %(message)s",
             datefmt="%Y-%m-%d %H:%M:%S",
         )
 
-        # Create the console handler and set the formatter
         console_handler = logging.StreamHandler()
         console_handler.setFormatter(formatter)
 
-        # Add the console handler to the firecrawl logger
         logger.addHandler(console_handler)
     except Exception as e:
         logger.error("Failed to configure logging: %s", e)
@@ -45,20 +46,15 @@ def _configure_logger() -> None:
 
 def setup_logging() -> None:
     """Set up logging based on the FIRECRAWL_LOGGING_LEVEL environment variable."""
-    # Check if the firecrawl logger already has a handler
     if logger.hasHandlers():
-        return # To prevent duplicate logging
+        return
 
-    # Check if the FIRECRAWL_LOGGING_LEVEL environment variable is set
     if not (env := os.getenv("FIRECRAWL_LOGGING_LEVEL", "").upper()):
-        # Attach a no-op handler to prevent warnings about no handlers
         logger.addHandler(logging.NullHandler()) 
         return
 
-    # Attach the console handler to the firecrawl logger
     _configure_logger()
 
-    # Set the logging level based on the FIRECRAWL_LOGGING_LEVEL environment variable
     if env == "DEBUG":
         logger.setLevel(logging.DEBUG)
     elif env == "INFO":
@@ -73,7 +69,19 @@ def setup_logging() -> None:
         logger.setLevel(logging.INFO)
         logger.warning("Unknown logging level: %s, defaulting to INFO", env)
 
-
-# Initialize logging configuration when the module is imported
 setup_logging()
 logger.debug("Debugging logger setup")
+
+__all__ = [
+    'Firecrawl',
+    'AsyncFirecrawl',
+    'FirecrawlApp',
+    'AsyncFirecrawlApp',
+    'Watcher',
+    'AsyncWatcher',
+    'V1FirecrawlApp',
+    'AsyncV1FirecrawlApp',
+    'V1JsonConfig',
+    'V1ScrapeOptions',
+    'V1ChangeTrackingOptions',
+]

firecrawl-3.0.3/firecrawl/__tests__/e2e/v2/aio/test_aio_batch_scrape.py

@@ -0,0 +1,79 @@
+import os
+import asyncio
+import pytest
+from dotenv import load_dotenv
+from firecrawl import AsyncFirecrawl
+
+
+load_dotenv()
+
+if not os.getenv("API_KEY"):
+    raise ValueError("API_KEY is not set")
+
+if not os.getenv("API_URL"):
+    raise ValueError("API_URL is not set")
+
+
+@pytest.mark.asyncio
+async def test_async_batch_start_and_status():
+    client = AsyncFirecrawl(api_key=os.getenv("API_KEY"), api_url=os.getenv("API_URL"))
+    start = await client.start_batch_scrape([
+        "https://docs.firecrawl.dev",
+        "https://firecrawl.dev",
+    ], formats=["markdown"], max_concurrency=1)
+    job_id = start.id
+
+    deadline = asyncio.get_event_loop().time() + 240
+    status = await client.get_batch_scrape_status(job_id)
+    while status.status not in ("completed", "failed", "cancelled") and asyncio.get_event_loop().time() < deadline:
+        await asyncio.sleep(2)
+        status = await client.get_batch_scrape_status(job_id)
+
+    assert status.status in ("completed", "failed", "cancelled")
+
+
+@pytest.mark.asyncio
+async def test_async_batch_wait_minimal():
+    client = AsyncFirecrawl(api_key=os.getenv("API_KEY"), api_url=os.getenv("API_URL"))
+    job = await client.batch_scrape([
+        "https://docs.firecrawl.dev",
+        "https://firecrawl.dev",
+    ], formats=["markdown"], poll_interval=1, timeout=120)
+    assert job.status in ("completed", "failed")
+
+
+@pytest.mark.asyncio
+async def test_async_batch_wait_with_all_params():
+    client = AsyncFirecrawl(api_key=os.getenv("API_KEY"), api_url=os.getenv("API_URL"))
+    json_schema = {"type": "object", "properties": {"title": {"type": "string"}}, "required": ["title"]}
+    job = await client.batch_scrape(
+        [
+            "https://docs.firecrawl.dev",
+            "https://firecrawl.dev",
+        ],
+        formats=[
+            "markdown",
+            {"type": "json", "prompt": "Extract page title", "schema": json_schema},
+            {"type": "changeTracking", "prompt": "Track changes", "modes": ["json"]},
+        ],
+        only_main_content=True,
+        mobile=False,
+        ignore_invalid_urls=True,
+        max_concurrency=2,
+        zero_data_retention=False,
+        poll_interval=1,
+        timeout=180,
+    )
+    assert job.status in ("completed", "failed")
+
+
+@pytest.mark.asyncio
+async def test_async_cancel_batch():
+    client = AsyncFirecrawl(api_key=os.getenv("API_KEY"), api_url=os.getenv("API_URL"))
+    start = await client.start_batch_scrape([
+        "https://docs.firecrawl.dev",
+        "https://firecrawl.dev",
+    ], formats=["markdown"], max_concurrency=1)
+    ok = await client.cancel_batch_scrape(start.id)
+    assert ok is True
+