firecrawl 2.16.5__py3-none-any.whl → 3.0.3__py3-none-any.whl

{firecrawl-2.16.5.dist-info → firecrawl-3.0.3.dist-info}/METADATA

@@ -1,15 +1,15 @@
 Metadata-Version: 2.1
 Name: firecrawl
-Version: 2.16.5
+Version: 3.0.3
 Summary: Python SDK for Firecrawl API
-Home-page: https://github.com/mendableai/firecrawl
+Home-page: https://github.com/firecrawl/firecrawl
 Author: Mendable.ai
 Author-email: "Mendable.ai" <nick@mendable.ai>
 Maintainer-email: "Mendable.ai" <nick@mendable.ai>
 License: MIT License
 Project-URL: Documentation, https://docs.firecrawl.dev
-Project-URL: Source, https://github.com/mendableai/firecrawl
-Project-URL: Tracker, https://github.com/mendableai/firecrawl/issues
+Project-URL: Source, https://github.com/firecrawl/firecrawl
+Project-URL: Tracker, https://github.com/firecrawl/firecrawl/issues
 Keywords: SDK,API,firecrawl
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Web Environment
@@ -34,6 +34,7 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: requests
+Requires-Dist: httpx
 Requires-Dist: python-dotenv
 Requires-Dist: websockets
 Requires-Dist: nest-asyncio
@@ -55,24 +56,25 @@ pip install firecrawl-py
 ## Usage
 
 1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
-2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.
 
 Here's an example of how to use the SDK:
 
 ```python 
-from firecrawl import FirecrawlApp, ScrapeOptions
+from firecrawl import Firecrawl
+from firecrawl.types import ScrapeOptions
 
-app = FirecrawlApp(api_key="fc-YOUR_API_KEY")
+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
 
-# Scrape a website:
-data = app.scrape_url(
+# Scrape a website (v2):
+data = firecrawl.scrape(
   'https://firecrawl.dev', 
   formats=['markdown', 'html']
 )
 print(data)
 
-# Crawl a website:
-crawl_status = app.crawl_url(
+# Crawl a website (v2 waiter):
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html'])
@@ -82,20 +84,20 @@ print(crawl_status)
 
 ### Scraping a URL
 
-To scrape a single URL, use the `scrape_url` method. It takes the URL as a parameter and returns the scraped data as a dictionary.
+To scrape a single URL, use the `scrape` method. It takes the URL as a parameter and returns a document with the requested formats.
 
 ```python 
-# Scrape a website:
-scrape_result = app.scrape_url('firecrawl.dev', formats=['markdown', 'html'])
+# Scrape a website (v2):
+scrape_result = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
 print(scrape_result)
 ```
 
 ### Crawling a Website
 
-To crawl a website, use the `crawl_url` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To crawl a website, use the `crawl` method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.
 
 ```python 
-crawl_status = app.crawl_url(
+crawl_status = firecrawl.crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
@@ -108,23 +110,23 @@ print(crawl_status)
 
 <Tip>Looking for async operations? Check out the [Async Class](#async-class) section below.</Tip>
 
-To crawl a website asynchronously, use the `crawl_url_async` method. It returns the crawl `ID` which you can use to check the status of the crawl job. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+To enqueue a crawl asynchronously, use `start_crawl`. It returns the crawl `ID` which you can use to check the status of the crawl job.
 
 ```python 
-crawl_status = app.async_crawl_url(
+crawl_job = firecrawl.start_crawl(
   'https://firecrawl.dev', 
   limit=100, 
   scrape_options=ScrapeOptions(formats=['markdown', 'html']),
 )
-print(crawl_status)
+print(crawl_job)
 ```
 
 ### Checking Crawl Status
 
-To check the status of a crawl job, use the `check_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
+To check the status of a crawl job, use the `get_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
 
 ```python 
-crawl_status = app.check_crawl_status("<crawl_id>")
+crawl_status = firecrawl.get_crawl_status("<crawl_id>")
 print(crawl_status)
 ```
 
@@ -133,17 +135,17 @@ print(crawl_status)
 To cancel an asynchronous crawl job, use the `cancel_crawl` method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.
 
 ```python 
-cancel_crawl = app.cancel_crawl(id)
+cancel_crawl = firecrawl.cancel_crawl(id)
 print(cancel_crawl)
 ```
 
 ### Map a Website
 
-Use `map_url` to generate a list of URLs from a website. The `params` argument let you customize the mapping process, including options to exclude subdomains or to utilize the sitemap.
+Use `map` to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.
 
 ```python 
-# Map a website:
-map_result = app.map_url('https://firecrawl.dev')
+# Map a website (v2):
+map_result = firecrawl.map('https://firecrawl.dev')
 print(map_result)
 ```
 
@@ -194,20 +196,35 @@ The SDK handles errors returned by the Firecrawl API and raises appropriate exce
 
 ## Async Class
 
-For async operations, you can use the `AsyncFirecrawlApp` class. Its methods are the same as the `FirecrawlApp` class, but they don't block the main thread.
+For async operations, you can use the `AsyncFirecrawl` class. Its methods mirror the `Firecrawl` class, but you `await` them.
 
 ```python 
-from firecrawl import AsyncFirecrawlApp
+from firecrawl import AsyncFirecrawl
 
-app = AsyncFirecrawlApp(api_key="YOUR_API_KEY")
+firecrawl = AsyncFirecrawl(api_key="YOUR_API_KEY")
 
-# Async Scrape
+# Async Scrape (v2)
 async def example_scrape():
-  scrape_result = await app.scrape_url(url="https://example.com")
+  scrape_result = await firecrawl.scrape(url="https://example.com")
   print(scrape_result)
 
-# Async Crawl
+# Async Crawl (v2)
 async def example_crawl():
-  crawl_result = await app.crawl_url(url="https://example.com")
+  crawl_result = await firecrawl.crawl(url="https://example.com")
   print(crawl_result)
 ```
+
+## v1 compatibility
+
+For legacy code paths, v1 remains available under `firecrawl.v1` with the original method names.
+
+```python
+from firecrawl import Firecrawl
+
+firecrawl = Firecrawl(api_key="YOUR_API_KEY")
+
+# v1 methods (feature‑frozen)
+doc_v1 = firecrawl.v1.scrape_url('https://firecrawl.dev', formats=['markdown', 'html'])
+crawl_v1 = firecrawl.v1.crawl_url('https://firecrawl.dev', limit=100)
+map_v1 = firecrawl.v1.map_url('https://firecrawl.dev')
+```

firecrawl-3.0.3.dist-info/RECORD

@@ -0,0 +1,78 @@
+firecrawl/__init__.py,sha256=5w9g4kvMhD7vpl37kKrLpgrUEQ1OWfXyj4tzsvAiQyE,2191
+firecrawl/client.py,sha256=VxlMrvoq288KiIMKFk7Fq22KG0DGLZQQm56vilT71pQ,11058
+firecrawl/firecrawl.backup.py,sha256=v1FEN3jR4g5Aupg4xp6SLkuFvYMQuUKND2YELbYjE6c,200430
+firecrawl/types.py,sha256=yZ4iza0M1T2kxNbt-tLEOKH7o6mFKZZ11VAZGodHSq4,2734
+firecrawl/__tests__/e2e/v2/conftest.py,sha256=I28TUpN5j0-9gM79NlbrDS8Jlsheao657od2f-2xK0Y,2587
+firecrawl/__tests__/e2e/v2/test_async.py,sha256=ZXpf1FVOJgNclITglrxIyFwP4cOiqzWLicGaxIm70BQ,2526
+firecrawl/__tests__/e2e/v2/test_batch_scrape.py,sha256=H9GtuwHIFdOQ958SOVThi_kvDDxcXAK_ECRh95ogonQ,3265
+firecrawl/__tests__/e2e/v2/test_crawl.py,sha256=cOssZvIwtghAtLiM1QdNLhPEwAxZ9j9umTrBUPtJjpU,9951
+firecrawl/__tests__/e2e/v2/test_extract.py,sha256=HgvGiDlyWtFygiPo5EP44Dem1oWrwgRF-hfc1LfeVSU,1670
+firecrawl/__tests__/e2e/v2/test_map.py,sha256=9sT-Yq8V_8c9esl_bv5hnTA9WXb2Dg81kj6M-s0484c,1618
+firecrawl/__tests__/e2e/v2/test_scrape.py,sha256=psW2nfcA_hMFpZ4msL_VJWJTMa3Sidp11ubhftbm52g,5759
+firecrawl/__tests__/e2e/v2/test_search.py,sha256=MN-q82gHlm5DT2HsnAQgW1NwVbgowlFYmKW1KGJd1ig,8811
+firecrawl/__tests__/e2e/v2/test_usage.py,sha256=JlBkYblhThua5qF2crRjsPpq4Ja0cBsdzxZ5zxXnQ_Y,805
+firecrawl/__tests__/e2e/v2/test_watcher.py,sha256=OPTKLhVAKWqXl2Tieo6zCN1xpEwZDsz-B977CVJgLMA,1932
+firecrawl/__tests__/e2e/v2/aio/test_aio_batch_scrape.py,sha256=gJv_mLzzoAYftETB2TLkrpSfB5c04kaYgkD4hQTYsIg,2639
+firecrawl/__tests__/e2e/v2/aio/test_aio_crawl.py,sha256=AefCZA1he1UkGv80UXtyQru-zQbESd1L4yIAdycW-Y0,7317
+firecrawl/__tests__/e2e/v2/aio/test_aio_extract.py,sha256=3CNRIFzgBMcOYOLhnKcK1k5a3Gy--u08EGDkL31uieM,1199
+firecrawl/__tests__/e2e/v2/aio/test_aio_map.py,sha256=nckl1kbiEaaTdu5lm__tOoTDG-txTYwwSH3KZEvyKzc,1199
+firecrawl/__tests__/e2e/v2/aio/test_aio_scrape.py,sha256=b17A7advBEjxrjdait2w8GHztZeKy_P3zZ3ixm5H7xw,4453
+firecrawl/__tests__/e2e/v2/aio/test_aio_search.py,sha256=dnrRyTIzivlwe5wt5Wa0hdghZcJmNjC1l-XrAA_JZUU,7308
+firecrawl/__tests__/e2e/v2/aio/test_aio_usage.py,sha256=Dh9BVo48NKSZOKgLbO7n8fpMjvYmeMXDFzbIhnCTMhE,1014
+firecrawl/__tests__/e2e/v2/aio/test_aio_watcher.py,sha256=hwES4Nu5c0hniZ9heIPDfvh_2JmJ2wPoX9ULTZ0Asjs,1471
+firecrawl/__tests__/unit/v2/methods/test_batch_request_preparation.py,sha256=HeOxN-sPYSssytcIRAEicJSZsFt_Oa5qGXAtdumR54c,4040
+firecrawl/__tests__/unit/v2/methods/test_crawl_params.py,sha256=p9hzg14uAs1iHKXPDSXhGU6hEzPBF_Ae34RAf5XYa10,2387
+firecrawl/__tests__/unit/v2/methods/test_crawl_request_preparation.py,sha256=9DbLkBg6tuMyg7ASGX_oaQmAy0VCV4oITCOrfeR2UkY,8806
+firecrawl/__tests__/unit/v2/methods/test_crawl_validation.py,sha256=kErOmHSD01eMjXiMd4rgsMVGd_aU2G9uVymBjbAFoGw,3918
+firecrawl/__tests__/unit/v2/methods/test_map_request_preparation.py,sha256=toVcgnMp_cFeYsIUuyKGEWZGp0nAAkzaeFGUbY0zY0o,1868
+firecrawl/__tests__/unit/v2/methods/test_scrape_request_preparation.py,sha256=wDOslsA5BN4kyezlaT5GeMv_Ifn8f461EaA7i5ujnaQ,3482
+firecrawl/__tests__/unit/v2/methods/test_search_request_preparation.py,sha256=14lUgFpQsiosgMKjDustBRVE0zXnHujBI76F8BC5PZ4,6072
+firecrawl/__tests__/unit/v2/methods/test_search_validation.py,sha256=PaV_kSgzjW8A3eFBCCn1-y4WFZBR2nf84NZk4UEBPX8,8275
+firecrawl/__tests__/unit/v2/methods/test_usage_types.py,sha256=cCHHfa6agSjD0brQ9rcAcw2kaI9riUH5C0dXV-fqktg,591
+firecrawl/__tests__/unit/v2/methods/test_webhook.py,sha256=AvvW-bKpUA--Lvtif2bmUIp-AxiaMJ29ie1i9dk8WbI,4586
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_params.py,sha256=9azJxVvDOBqUevLp-wBF9gF7Ptj-7nN6LOkPQncFX2M,456
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_request_preparation.py,sha256=vFbitMhH92JS5AtU78KQPi6kbT2fv68i9-rBrY5hVss,2574
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_validation.py,sha256=WMgltdrrT2HOflqGyahC4v-Wb29_8sypN0hwS9lYXe8,403
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_map_request_preparation.py,sha256=PdUJrR0JLWqrithAnRXwuRrnsIN2h_DTu6-xvTOn_UU,725
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_scrape_request_preparation.py,sha256=A5DT4wpH4vrIPvFxKVHrtDH5A3bgJ_ad4fmVQ8LN1t0,1993
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_search_request_preparation.py,sha256=hFk4XgqF3aFPGFJe0ikB1uwf_0FsppNGA088OrWUXvg,2091
+firecrawl/__tests__/unit/v2/methods/aio/test_batch_request_preparation_async.py,sha256=E26UnUhpbjG-EG0ab4WRD94AxA5IBWmIHq8ZLBOWoAA,1202
+firecrawl/__tests__/unit/v2/methods/aio/test_ensure_async.py,sha256=pUwuWhRbVUTbgsZn4hgZesMkTMesTv_NPmvFW--ls-Y,3815
+firecrawl/__tests__/unit/v2/utils/test_validation.py,sha256=E4n4jpBhH_W7E0ikI5r8KMAKiOhbfGD3i_B8-dv3PlI,10803
+firecrawl/__tests__/unit/v2/watcher/test_ws_watcher.py,sha256=87w47n0iOihtu4jTR4-4rw1-xVKWmLg2BOBGxjQPnUk,9517
+firecrawl/v1/__init__.py,sha256=aP1oisPeZVGGZynvENc07JySMOZfv_4zAlxQ0ecMJXA,481
+firecrawl/v1/client.py,sha256=sydurfEFTsXyowyaGryA1lkPxN_r9Nf6iQpM43OwJyM,201672
+firecrawl/v2/__init__.py,sha256=Jc6a8tBjYG5OPkjDM5pl-notyys-7DEj7PLEfepv3fc,137
+firecrawl/v2/client.py,sha256=P6WAzwYGLLIANTrqAM-K4EUdGWQoFsi-zCjBibbxKQw,30507
+firecrawl/v2/client_async.py,sha256=zwxHis1bSh0tSF1480ze-4XDQEDJ5yDur1ZqtL94dwc,10127
+firecrawl/v2/types.py,sha256=Qj4kZ05rKKUQRzzcn4xZbcUZHgt6LXdG29T0NyZTPns,17866
+firecrawl/v2/watcher.py,sha256=tUPyYEGfQq93sAPDzxEbOmDTaUqW6pltKGHz2QdSGks,15063
+firecrawl/v2/watcher_async.py,sha256=yw3Jp_tNvTgR697AyNPYhAIb0vL5KixUwv2oVkHaQEA,10456
+firecrawl/v2/methods/batch.py,sha256=bTn9uMslVOfQdU3645kmt151t1j0suPPyNRgYM9zXHU,12165
+firecrawl/v2/methods/crawl.py,sha256=xrUe2y_T7ZYmAVll45Gag7BdS-Mmd1XxbJ88hXAMSuI,15404
+firecrawl/v2/methods/extract.py,sha256=-Jr4BtraU3b7hd3JIY73V-S69rUclxyXyUpoQb6DCQk,4274
+firecrawl/v2/methods/map.py,sha256=4SADb0-lkbdOWDmO6k8_TzK0yRti5xsN40N45nUl9uA,2592
+firecrawl/v2/methods/scrape.py,sha256=Sd3KNNCmSXvR17yLB72OEPeVPKk5DGM-chkm0WpYtSk,2178
+firecrawl/v2/methods/search.py,sha256=Ou0R_3rO0co7BJx3XBhiTX4bXPFlFIuU8b68bzaFMes,6488
+firecrawl/v2/methods/usage.py,sha256=OJlkxwaB-AAtgO3WLr9QiqBRmjdh6GVhroCgleegupQ,1460
+firecrawl/v2/methods/aio/__init__.py,sha256=RocMJnGwnLIvGu3G8ZvY8INkipC7WHZiu2bE31eSyJs,35
+firecrawl/v2/methods/aio/batch.py,sha256=GS_xsd_Uib1fxFITBK1sH88VGzFMrIcqJVQqOvMQ540,3735
+firecrawl/v2/methods/aio/crawl.py,sha256=PEFIqZ7UtTBZSbs3fQuoxWMN68WbsTcj2AnZZvnfggk,6936
+firecrawl/v2/methods/aio/extract.py,sha256=IfNr2ETqt4dR73JFzrEYI4kk5vpKnJOG0BmPEjGEoO4,4217
+firecrawl/v2/methods/aio/map.py,sha256=EuT-5A0cQr_e5SBfEZ6pnl8u0JUwEEvSwhyT2N-QoKU,2326
+firecrawl/v2/methods/aio/scrape.py,sha256=-VLFlE7Ma9TS0yXN4esvC3CPCic1Kq6d5Hc4j8CudGc,1586
+firecrawl/v2/methods/aio/search.py,sha256=UA5KJlzOvSqc3TYEm1wnZLNVoP4SsrRwdQbXGe7UcDk,2781
+firecrawl/v2/methods/aio/usage.py,sha256=OtBi6X-aT09MMR2dpm3vBCm9JrJZIJLCQ8jJ3L7vie4,1606
+firecrawl/v2/utils/__init__.py,sha256=i1GgxySmqEXpWSBQCu3iZBPIJG7fXj0QXCDWGwerWNs,338
+firecrawl/v2/utils/error_handler.py,sha256=Iuf916dHphDY8ObNNlWy75628DFeJ0Rv8ljRp4LttLE,4199
+firecrawl/v2/utils/get_version.py,sha256=0CxW_41q2hlzIxEWOivUCaYw3GFiSIH32RPUMcIgwAY,492
+firecrawl/v2/utils/http_client.py,sha256=_n8mp4xi6GGihg662Lsv6TSlvw9zykyADwEk0fg8mYA,4873
+firecrawl/v2/utils/http_client_async.py,sha256=P4XG6nTz6kKH3vCPTz6i7DRhbpK4IImRGaFvQFGBFRc,1874
+firecrawl/v2/utils/validation.py,sha256=L8by7z-t6GuMGIYkK7il1BM8d-4_-sAdG9hDMF_LeG4,14518
+tests/test_change_tracking.py,sha256=_IJ5ShLcoj2fHDBaw-nE4I4lHdmDB617ocK_XMHhXps,4177
+tests/test_timeout_conversion.py,sha256=PWlIEMASQNhu4cp1OW_ebklnE9NCiigPnEFCtI5N3w0,3996
+firecrawl-3.0.3.dist-info/LICENSE,sha256=nPCunEDwjRGHlmjvsiDUyIWbkqqyj3Ej84ntnh0g0zA,1084
+firecrawl-3.0.3.dist-info/METADATA,sha256=Z2T2KXXKCvSTsCgbtgkGmD-guzilPnWncCO9eZT2s50,7305
+firecrawl-3.0.3.dist-info/WHEEL,sha256=2wepM1nk4DS4eFpYrW1TTqPcoGNfHhhO_i5m4cOimbo,92
+firecrawl-3.0.3.dist-info/top_level.txt,sha256=8T3jOaSN5mtLghO-R3MQ8KO290gIX8hmfxQmglBPdLE,16
+firecrawl-3.0.3.dist-info/RECORD,,

tests/test_timeout_conversion.py

@@ -0,0 +1,117 @@
+import unittest
+from unittest.mock import patch, MagicMock
+import os
+from firecrawl import FirecrawlApp
+
+
+class TestTimeoutConversion(unittest.TestCase):
+    
+    @patch('requests.post')
+    def test_scrape_url_timeout_conversion(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            'success': True,
+            'data': {
+                'markdown': 'Test content'
+            }
+        }
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        app.scrape_url('https://example.com', timeout=60000)
+
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 65.0)
+
+    @patch('requests.post')
+    def test_scrape_url_default_timeout(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            'success': True,
+            'data': {
+                'markdown': 'Test content'
+            }
+        }
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        app.scrape_url('https://example.com')
+
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 35.0)
+
+    @patch('requests.post')
+    def test_post_request_timeout_conversion(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        
+        data = {'timeout': 30000}
+        headers = {'Content-Type': 'application/json'}
+        
+        app._post_request('https://example.com/api', data, headers)
+
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 35.0)
+
+    @patch('requests.post')
+    def test_post_request_default_timeout(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        
+        data = {'timeout': 30000, 'url': 'https://example.com'}
+        headers = {'Content-Type': 'application/json'}
+        
+        app._post_request('https://example.com/api', data, headers)
+
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 35.0)
+
+    @patch('requests.post')
+    def test_timeout_edge_cases(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            'success': True,
+            'data': {
+                'markdown': 'Test content'
+            }
+        }
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        
+        app.scrape_url('https://example.com', timeout=1000)
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 6.0)
+        
+        app.scrape_url('https://example.com', timeout=0)
+        args, kwargs = mock_post.call_args
+        self.assertEqual(kwargs['timeout'], 5.0)
+
+    @patch('requests.post')
+    def test_post_request_no_timeout_key(self, mock_post):
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        mock_post.return_value = mock_response
+
+        app = FirecrawlApp(api_key=os.environ.get('TEST_API_KEY', 'dummy-api-key-for-testing'))
+        
+        data = {'url': 'https://example.com'}
+        headers = {'Content-Type': 'application/json'}
+        
+        app._post_request('https://example.com/api', data, headers)
+
+        args, kwargs = mock_post.call_args
+        self.assertIsNone(kwargs['timeout'])
+
+
+if __name__ == '__main__':
+    unittest.main()

firecrawl/__tests__/e2e_withAuth/test.py

@@ -1,170 +0,0 @@
-import importlib.util
-import pytest
-import time
-import os
-from uuid import uuid4
-from dotenv import load_dotenv
-
-load_dotenv()
-
-API_URL = "http://127.0.0.1:3002"
-ABSOLUTE_FIRECRAWL_PATH = "firecrawl/firecrawl.py"
-TEST_API_KEY = os.getenv('TEST_API_KEY')
-
-print(f"ABSOLUTE_FIRECRAWL_PATH: {ABSOLUTE_FIRECRAWL_PATH}")
-
-spec = importlib.util.spec_from_file_location("FirecrawlApp", ABSOLUTE_FIRECRAWL_PATH)
-firecrawl = importlib.util.module_from_spec(spec)
-spec.loader.exec_module(firecrawl)
-FirecrawlApp = firecrawl.FirecrawlApp
-
-def test_no_api_key():
-    with pytest.raises(Exception) as excinfo:
-      invalid_app = FirecrawlApp(api_url=API_URL, version='v0')
-    assert "No API key provided" in str(excinfo.value)
-
-def test_scrape_url_invalid_api_key():
-    invalid_app = FirecrawlApp(api_url=API_URL, api_key="invalid_api_key", version='v0')
-    with pytest.raises(Exception) as excinfo:
-        invalid_app.scrape_url('https://firecrawl.dev')
-    assert "Unexpected error during scrape URL: Status code 401. Unauthorized: Invalid token" in str(excinfo.value)
-
-# def test_blocklisted_url():
-#     blocklisted_url = "https://facebook.com/fake-test"
-#     app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-#     with pytest.raises(Exception) as excinfo:
-#         app.scrape_url(blocklisted_url)
-#     assert "Unexpected error during scrape URL: Status code 403. Firecrawl currently does not support social media scraping due to policy restrictions. We're actively working on building support for it." in str(excinfo.value)
-
-def test_successful_response_with_valid_preview_token():
-    app = FirecrawlApp(api_url=API_URL, api_key=os.getenv('PREVIEW_TOKEN'), version='v0')
-    response = app.scrape_url('https://roastmywebsite.ai')
-    assert response is not None
-    assert 'content' in response
-    assert "_Roast_" in response['content']
-
-def test_scrape_url_e2e():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.scrape_url('https://roastmywebsite.ai')
-    print(response)
-
-    assert response is not None
-    assert 'content' in response
-    assert 'markdown' in response
-    assert 'metadata' in response
-    assert 'html' not in response
-    assert "_Roast_" in response['content']
-
-def test_successful_response_with_valid_api_key_and_include_html():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.scrape_url('https://roastmywebsite.ai', {'pageOptions': {'includeHtml': True}})
-    assert response is not None
-    assert 'content' in response
-    assert 'markdown' in response
-    assert 'html' in response
-    assert 'metadata' in response
-    assert "_Roast_" in response['content']
-    assert "_Roast_" in response['markdown']
-    assert "<h1" in response['html']
-
-def test_successful_response_for_valid_scrape_with_pdf_file():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.scrape_url('https://arxiv.org/pdf/astro-ph/9301001.pdf')
-    assert response is not None
-    assert 'content' in response
-    assert 'metadata' in response
-    assert 'We present spectrophotometric observations of the Broad Line Radio Galaxy' in response['content']
-
-def test_successful_response_for_valid_scrape_with_pdf_file_without_explicit_extension():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.scrape_url('https://arxiv.org/pdf/astro-ph/9301001')
-    time.sleep(6)  # wait for 6 seconds
-    assert response is not None
-    assert 'content' in response
-    assert 'metadata' in response
-    assert 'We present spectrophotometric observations of the Broad Line Radio Galaxy' in response['content']
-
-def test_crawl_url_invalid_api_key():
-    invalid_app = FirecrawlApp(api_url=API_URL, api_key="invalid_api_key", version='v0')
-    with pytest.raises(Exception) as excinfo:
-        invalid_app.crawl_url('https://firecrawl.dev')
-    assert "Unexpected error during start crawl job: Status code 401. Unauthorized: Invalid token" in str(excinfo.value)
-
-# def test_should_return_error_for_blocklisted_url():
-#     app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-#     blocklisted_url = "https://twitter.com/fake-test"
-#     with pytest.raises(Exception) as excinfo:
-#         app.crawl_url(blocklisted_url)
-#     assert "Unexpected error during start crawl job: Status code 403. Firecrawl currently does not support social media scraping due to policy restrictions. We're actively working on building support for it." in str(excinfo.value)
-
-def test_crawl_url_wait_for_completion_e2e():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.crawl_url('https://roastmywebsite.ai', {'crawlerOptions': {'excludes': ['blog/*']}}, True)
-    assert response is not None
-    assert len(response) > 0
-    assert 'content' in response[0]
-    assert "_Roast_" in response[0]['content']
-
-def test_crawl_url_with_idempotency_key_e2e():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    uniqueIdempotencyKey = str(uuid4())
-    response = app.crawl_url('https://roastmywebsite.ai', {'crawlerOptions': {'excludes': ['blog/*']}}, True, 2, uniqueIdempotencyKey)
-    assert response is not None
-    assert len(response) > 0
-    assert 'content' in response[0]
-    assert "_Roast_" in response[0]['content']
-
-    with pytest.raises(Exception) as excinfo:
-        app.crawl_url('https://firecrawl.dev', {'crawlerOptions': {'excludes': ['blog/*']}}, True, 2, uniqueIdempotencyKey)
-    assert "Conflict: Failed to start crawl job due to a conflict. Idempotency key already used" in str(excinfo.value) 
-
-def test_check_crawl_status_e2e():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.crawl_url('https://firecrawl.dev', {'crawlerOptions': {'excludes': ['blog/*']}}, False)
-    assert response is not None
-    assert 'jobId' in response
-    
-    time.sleep(30)  # wait for 30 seconds
-    status_response = app.check_crawl_status(response['jobId'])
-    assert status_response is not None
-    assert 'status' in status_response
-    assert status_response['status'] == 'completed'
-    assert 'data' in status_response
-    assert len(status_response['data']) > 0
-
-def test_search_e2e():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.search("test query")
-    assert response is not None
-    assert 'content' in response[0]
-    assert len(response) > 2
-
-def test_search_invalid_api_key():
-    invalid_app = FirecrawlApp(api_url=API_URL, api_key="invalid_api_key", version='v0')
-    with pytest.raises(Exception) as excinfo:
-        invalid_app.search("test query")
-    assert "Unexpected error during search: Status code 401. Unauthorized: Invalid token" in str(excinfo.value)
-
-def test_llm_extraction():
-    app = FirecrawlApp(api_url=API_URL, api_key=TEST_API_KEY, version='v0')
-    response = app.scrape_url("https://firecrawl.dev", {
-        'extractorOptions': {
-            'mode': 'llm-extraction',
-            'extractionPrompt': "Based on the information on the page, find what the company's mission is and whether it supports SSO, and whether it is open source",
-            'extractionSchema': {
-                'type': 'object',
-                'properties': {
-                    'company_mission': {'type': 'string'},
-                    'supports_sso': {'type': 'boolean'},
-                    'is_open_source': {'type': 'boolean'}
-                },
-                'required': ['company_mission', 'supports_sso', 'is_open_source']
-            }
-        }
-    })
-    assert response is not None
-    assert 'llm_extraction' in response
-    llm_extraction = response['llm_extraction']
-    assert 'company_mission' in llm_extraction
-    assert isinstance(llm_extraction['supports_sso'], bool)
-    assert isinstance(llm_extraction['is_open_source'], bool)