firecrawl 4.12.0__py3-none-any.whl

firecrawl-4.12.0.dist-info/METADATA

@@ -0,0 +1,234 @@
+Metadata-Version: 2.4
+Name: firecrawl
+Version: 4.12.0
+Summary: Python SDK for Firecrawl API
+Home-page: https://github.com/firecrawl/firecrawl
+Author: Mendable.ai
+Author-email: "Mendable.ai" <nick@mendable.ai>
+Maintainer-email: "Mendable.ai" <nick@mendable.ai>
+License: MIT License
+Project-URL: Documentation, https://docs.firecrawl.dev
+Project-URL: Source, https://github.com/firecrawl/firecrawl
+Project-URL: Tracker, https://github.com/firecrawl/firecrawl/issues
+Keywords: SDK,API,firecrawl
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Topic :: Internet
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: httpx
+Requires-Dist: python-dotenv
+Requires-Dist: websockets
+Requires-Dist: nest-asyncio
+Requires-Dist: pydantic>=2.0
+Requires-Dist: aiohttp
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# Firecrawl Python SDK
+
+The Firecrawl Python SDK is a library that allows you to easily scrape and crawl websites, and output the data in a format ready for use with language models (LLMs). It provides a simple and intuitive interface for interacting with the Firecrawl API.
+
+## Installation
+
+To install the Firecrawl Python SDK, you can use pip:
+
+```bash 
+pip install firecrawl-py
+```
+
+## Usage
+
+1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.
+
+Here's an example of how to use the SDK:
+
+```python 
+from firecrawl import Firecrawl
+from firecrawl.types import ScrapeOptions
+
+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
+
+# Scrape a website (v2):
+data = firecrawl.scrape(
+  'https://firecrawl.dev', 
+  formats=['markdown', 'html']
+)
+print(data)
+
+# Crawl a website (v2 waiter):
+crawl_status = firecrawl.crawl(
+  'https://firecrawl.dev', 
+  limit=100, 
+  scrape_options=ScrapeOptions(formats=['markdown', 'html'])
+)
+print(crawl_status)
+```
+
+### Scraping a URL
+
+To scrape a single URL, use the `scrape` method. It takes the URL as a parameter and returns a document with the requested formats.
+
+```python 
+# Scrape a website (v2):
+scrape_result = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
+print(scrape_result)
+```
+
+### Crawling a Website
+
+To crawl a website, use the `crawl` method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.
+
+```python 
+crawl_status = firecrawl.crawl(
+  'https://firecrawl.dev', 
+  limit=100, 
+  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
+  poll_interval=30
+)
+print(crawl_status)
+```
+
+### Asynchronous Crawling
+
+<Tip>Looking for async operations? Check out the [Async Class](#async-class) section below.</Tip>
+
+To enqueue a crawl asynchronously, use `start_crawl`. It returns the crawl `ID` which you can use to check the status of the crawl job.
+
+```python 
+crawl_job = firecrawl.start_crawl(
+  'https://firecrawl.dev', 
+  limit=100, 
+  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
+)
+print(crawl_job)
+```
+
+### Checking Crawl Status
+
+To check the status of a crawl job, use the `get_crawl_status` method. It takes the job ID as a parameter and returns the current status of the crawl job.
+
+```python 
+crawl_status = firecrawl.get_crawl_status("<crawl_id>")
+print(crawl_status)
+```
+
+### Cancelling a Crawl
+
+To cancel an asynchronous crawl job, use the `cancel_crawl` method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.
+
+```python 
+cancel_crawl = firecrawl.cancel_crawl(id)
+print(cancel_crawl)
+```
+
+### Map a Website
+
+Use `map` to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.
+
+```python 
+# Map a website (v2):
+map_result = firecrawl.map('https://firecrawl.dev')
+print(map_result)
+```
+
+{/* ### Extracting Structured Data from Websites
+
+  To extract structured data from websites, use the `extract` method. It takes the URLs to extract data from, a prompt, and a schema as arguments. The schema is a Pydantic model that defines the structure of the extracted data.
+
+  <ExtractPythonShort /> */}
+
+### Crawling a Website with WebSockets
+
+To crawl a website with WebSockets, use the `crawl_url_and_watch` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+
+```python 
+# inside an async function...
+nest_asyncio.apply()
+
+# Define event handlers
+def on_document(detail):
+    print("DOC", detail)
+
+def on_error(detail):
+    print("ERR", detail['error'])
+
+def on_done(detail):
+    print("DONE", detail['status'])
+
+    # Function to start the crawl and watch process
+async def start_crawl_and_watch():
+    # Initiate the crawl job and get the watcher
+    watcher = app.crawl_url_and_watch('firecrawl.dev', exclude_paths=['blog/*'], limit=5)
+
+    # Add event listeners
+    watcher.add_event_listener("document", on_document)
+    watcher.add_event_listener("error", on_error)
+    watcher.add_event_listener("done", on_done)
+
+    # Start the watcher
+    await watcher.connect()
+
+# Run the event loop
+await start_crawl_and_watch()
+```
+
+## Error Handling
+
+The SDK handles errors returned by the Firecrawl API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.
+
+## Async Class
+
+For async operations, you can use the `AsyncFirecrawl` class. Its methods mirror the `Firecrawl` class, but you `await` them.
+
+```python 
+from firecrawl import AsyncFirecrawl
+
+firecrawl = AsyncFirecrawl(api_key="YOUR_API_KEY")
+
+# Async Scrape (v2)
+async def example_scrape():
+  scrape_result = await firecrawl.scrape(url="https://example.com")
+  print(scrape_result)
+
+# Async Crawl (v2)
+async def example_crawl():
+  crawl_result = await firecrawl.crawl(url="https://example.com")
+  print(crawl_result)
+```
+
+## v1 compatibility
+
+For legacy code paths, v1 remains available under `firecrawl.v1` with the original method names.
+
+```python
+from firecrawl import Firecrawl
+
+firecrawl = Firecrawl(api_key="YOUR_API_KEY")
+
+# v1 methods (feature‑frozen)
+doc_v1 = firecrawl.v1.scrape_url('https://firecrawl.dev', formats=['markdown', 'html'])
+crawl_v1 = firecrawl.v1.crawl_url('https://firecrawl.dev', limit=100)
+map_v1 = firecrawl.v1.map_url('https://firecrawl.dev')
+```

firecrawl-4.12.0.dist-info/RECORD

@@ -0,0 +1,92 @@
+firecrawl/__init__.py,sha256=q1bLmuUqk74q-9DlgP_46FYnQJEOpUqInNB06fvj8W8,2193
+firecrawl/client.py,sha256=u0bT1vkR0TOzW-2ZnJ88Tj6llbruOFvLjM9EPF7Cm4Q,12756
+firecrawl/firecrawl.backup.py,sha256=v1FEN3jR4g5Aupg4xp6SLkuFvYMQuUKND2YELbYjE6c,200430
+firecrawl/types.py,sha256=RmLTq14Z-Nf883wgZxubrtn2HDu9mecsCEdcIdBCu14,2923
+firecrawl/__tests__/e2e/v2/conftest.py,sha256=I28TUpN5j0-9gM79NlbrDS8Jlsheao657od2f-2xK0Y,2587
+firecrawl/__tests__/e2e/v2/test_async.py,sha256=ZXpf1FVOJgNclITglrxIyFwP4cOiqzWLicGaxIm70BQ,2526
+firecrawl/__tests__/e2e/v2/test_batch_scrape.py,sha256=tbuJ9y10ec9TtOnq97zmaEpOgZr9VzplRtZ_b6jkhq4,3302
+firecrawl/__tests__/e2e/v2/test_crawl.py,sha256=HTMx4cFzLjzWyBVgnzfIlNJ0CRCBx5q2u5LSlnFqtg8,10014
+firecrawl/__tests__/e2e/v2/test_extract.py,sha256=b3WL4xPtINrPAn7oKKyYWyPIMSl0fr_DGVUU5NjJe-Y,1707
+firecrawl/__tests__/e2e/v2/test_map.py,sha256=K7abzGcmQp4FLchZytQv4Kwkm9AAivPYyAC5kCb8ecE,1655
+firecrawl/__tests__/e2e/v2/test_scrape.py,sha256=4ElTgZqPmoQCC5tfjDnbw7W75oWa7PJ9WPXWaHVMRMs,7235
+firecrawl/__tests__/e2e/v2/test_search.py,sha256=xlWuBqcwfWGkLpQidcsG3kdbyqHFjLQTMsJzCE_CFyY,9112
+firecrawl/__tests__/e2e/v2/test_usage.py,sha256=JlBkYblhThua5qF2crRjsPpq4Ja0cBsdzxZ5zxXnQ_Y,805
+firecrawl/__tests__/e2e/v2/test_watcher.py,sha256=OPTKLhVAKWqXl2Tieo6zCN1xpEwZDsz-B977CVJgLMA,1932
+firecrawl/__tests__/e2e/v2/aio/conftest.py,sha256=hgLGGrNDSdpVx9KvxMw-a4HwNW4WVHWkwIncHJDxG04,1812
+firecrawl/__tests__/e2e/v2/aio/test_aio_batch_scrape.py,sha256=SGmQ8nAcbzs86TJCBzYJrF1XJ25Q96mLb0JnEkBSuto,2429
+firecrawl/__tests__/e2e/v2/aio/test_aio_crawl.py,sha256=oytaAS7GCMlVv-EMLoSVkK3PiRt19FmBdCv8W4kstXc,7315
+firecrawl/__tests__/e2e/v2/aio/test_aio_extract.py,sha256=RGIBtnvAKebwpPy1ZWWT-AqFvdf4bn_Td6y55F5l5As,1232
+firecrawl/__tests__/e2e/v2/aio/test_aio_map.py,sha256=uaSIk68BeeA9Z-2NL-HCYLcRMAEWyKX7oplpfbKUf20,1232
+firecrawl/__tests__/e2e/v2/aio/test_aio_scrape.py,sha256=xjCXkmoyp4AVa2LiAJKlMYOzDnEAnenkWMK2jhHkD7U,4486
+firecrawl/__tests__/e2e/v2/aio/test_aio_search.py,sha256=_IkHkIuvWY6vH99EsqrCZuKcfAX8qkG4NVG4KJNYu-0,8279
+firecrawl/__tests__/e2e/v2/aio/test_aio_usage.py,sha256=lVGfwR79eaZamUZXgKStUJcpclCnnlpwHGo2pMOUhCY,1255
+firecrawl/__tests__/e2e/v2/aio/test_aio_watcher.py,sha256=hwES4Nu5c0hniZ9heIPDfvh_2JmJ2wPoX9ULTZ0Asjs,1471
+firecrawl/__tests__/unit/test_recursive_schema_v1.py,sha256=raMbKRnJQT3BBue67twwDBt64hwfgSAmX4xMF6XtIFk,41706
+firecrawl/__tests__/unit/v2/methods/test_agent.py,sha256=gl6ZSXwhFbOyyigL2rAKxNz7LNKvxV7n9BiMdQUAAno,11863
+firecrawl/__tests__/unit/v2/methods/test_agent_request_preparation.py,sha256=38_KZE6QJyV3mzXIJv7ecQ_Ugp0p6FFer6TeZxVh6OQ,7699
+firecrawl/__tests__/unit/v2/methods/test_batch_request_preparation.py,sha256=xAx-aH4bD6uCWavg1cw_8-9FnLIFJNkvVPyOCVJ7r2E,4052
+firecrawl/__tests__/unit/v2/methods/test_branding.py,sha256=f5DkvCMSQKcLEvzByOO0Ae3FuCSj8YzJjDvey6svJJM,8281
+firecrawl/__tests__/unit/v2/methods/test_crawl_params.py,sha256=p9hzg14uAs1iHKXPDSXhGU6hEzPBF_Ae34RAf5XYa10,2387
+firecrawl/__tests__/unit/v2/methods/test_crawl_request_preparation.py,sha256=PEKbooNXfQwPpvcPHXABJnveztgAA-RFBhtlSs8uPro,8780
+firecrawl/__tests__/unit/v2/methods/test_crawl_validation.py,sha256=kErOmHSD01eMjXiMd4rgsMVGd_aU2G9uVymBjbAFoGw,3918
+firecrawl/__tests__/unit/v2/methods/test_map_request_preparation.py,sha256=w4FZrDqk9XGOuCHw3SV5CkbRuyb_F4Kc8C5eJ7zVcFs,1959
+firecrawl/__tests__/unit/v2/methods/test_pagination.py,sha256=-L4MLt6P_UVaQZQP9GVajxktABzqJHz7CojxuZnGjwI,24967
+firecrawl/__tests__/unit/v2/methods/test_scrape_request_preparation.py,sha256=mxx4B7v4cC42ivLUCosFB2cBIaBI7m9uOUsbE8pyyGU,4077
+firecrawl/__tests__/unit/v2/methods/test_search_request_preparation.py,sha256=tIImolryFlltn-zKoGFYodhc8eCInvxcmSu5n9ao1F0,6164
+firecrawl/__tests__/unit/v2/methods/test_search_validation.py,sha256=Ilj-57ibqKpm84CZs8eBP3SKjWONvE5yWDr-mcuwzWw,9499
+firecrawl/__tests__/unit/v2/methods/test_usage_types.py,sha256=cCHHfa6agSjD0brQ9rcAcw2kaI9riUH5C0dXV-fqktg,591
+firecrawl/__tests__/unit/v2/methods/test_webhook.py,sha256=AvvW-bKpUA--Lvtif2bmUIp-AxiaMJ29ie1i9dk8WbI,4586
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_params.py,sha256=9azJxVvDOBqUevLp-wBF9gF7Ptj-7nN6LOkPQncFX2M,456
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_request_preparation.py,sha256=fOSPJcCVsjk2WSDViwTqTnAPsUvsb6yT9lVG_q7iQfk,3208
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_crawl_validation.py,sha256=WMgltdrrT2HOflqGyahC4v-Wb29_8sypN0hwS9lYXe8,403
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_map_request_preparation.py,sha256=DFjgi4wdcesuD7SQzRSzqjbpN2YSSMkMY7oJ-q_wyrA,809
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_scrape_request_preparation.py,sha256=A5DT4wpH4vrIPvFxKVHrtDH5A3bgJ_ad4fmVQ8LN1t0,1993
+firecrawl/__tests__/unit/v2/methods/aio/test_aio_search_request_preparation.py,sha256=UAubD9xPX7H5oI6gttTxR3opvc3D-5ZDVCBHdpNFtYU,2182
+firecrawl/__tests__/unit/v2/methods/aio/test_batch_request_preparation_async.py,sha256=E26UnUhpbjG-EG0ab4WRD94AxA5IBWmIHq8ZLBOWoAA,1202
+firecrawl/__tests__/unit/v2/methods/aio/test_ensure_async.py,sha256=pUwuWhRbVUTbgsZn4hgZesMkTMesTv_NPmvFW--ls-Y,3815
+firecrawl/__tests__/unit/v2/utils/test_metadata_extras.py,sha256=2XqJ0UAWun9s-fnj8CvAvLMNIdafcsX4uub8wfqh5Eg,3468
+firecrawl/__tests__/unit/v2/utils/test_metadata_extras_multivalue.py,sha256=E3sbMRhHHWND3BbNW6i3onRbRZRNHK_cEd7KCTKWAkw,702
+firecrawl/__tests__/unit/v2/utils/test_recursive_schema.py,sha256=u27COoDIJySQJ9NOAuxy6U4lpuyUhIGoaEQMfUwD73I,37930
+firecrawl/__tests__/unit/v2/utils/test_validation.py,sha256=wiCgvIrXAilEZ94clCYbbiygtj5OUyctWcaYB6pyzm0,11535
+firecrawl/__tests__/unit/v2/watcher/test_ws_watcher.py,sha256=87w47n0iOihtu4jTR4-4rw1-xVKWmLg2BOBGxjQPnUk,9517
+firecrawl/v1/__init__.py,sha256=aP1oisPeZVGGZynvENc07JySMOZfv_4zAlxQ0ecMJXA,481
+firecrawl/v1/client.py,sha256=lkbgeZolvqf1aAv58Vb38cPuuj2nMmbkESzPXS_lhCw,221663
+firecrawl/v2/__init__.py,sha256=Jc6a8tBjYG5OPkjDM5pl-notyys-7DEj7PLEfepv3fc,137
+firecrawl/v2/client.py,sha256=4j65L6yvf6lOb0GkD9QDnqx_x3WwhRR6Apn0a_n9pg0,36890
+firecrawl/v2/client_async.py,sha256=qFoh4w-r08tOc6GLFl5DzRxdbvibnZm0SBqIcsJCjiU,15819
+firecrawl/v2/types.py,sha256=1A8W_81Lv5gXDBrzLurb6qKJXFw3wO-L8i-CF_sbViA,31059
+firecrawl/v2/watcher.py,sha256=FOU71tqSKxgeuGycu4ye0SLc2dw7clIcoQjPsi-4Csc,14229
+firecrawl/v2/watcher_async.py,sha256=dMMACMgeKrne_xSYeRvPu0m8nXqdNkDEsaiNBiD5ilw,10370
+firecrawl/v2/methods/agent.py,sha256=Qd0-Qj7g4W63WSZ-njFsPepppgbHJC6MVe4UhT7mhDs,4425
+firecrawl/v2/methods/batch.py,sha256=k7IoCUGpIirgk0Z-EqJJVSBRU09HfzRta9IxxLimbz4,14756
+firecrawl/v2/methods/crawl.py,sha256=XyR88MPfF11HUOhZR3JJTQIv477ZyJ2uty286H-p5K4,20049
+firecrawl/v2/methods/extract.py,sha256=ml0IEc7ckeTtapOYzgETjJjmHSc3JcJT5FK9_DL4S5Y,5538
+firecrawl/v2/methods/map.py,sha256=5LdPTmGu2u1dvgojIdYaUgcWNQ6otVdiFWOsqdR0iHc,2981
+firecrawl/v2/methods/scrape.py,sha256=CSHBwC-P91UfrW3zHirjNAs2h899FKcWvd1DY_4fJdo,1921
+firecrawl/v2/methods/search.py,sha256=VD619vGZECZ8ESFAbFNWvSUCaEV3WK96VIxVyQb6_z4,8537
+firecrawl/v2/methods/usage.py,sha256=NqkmFd-ziw8ijbZxwaxjxZHl85u0LTe_TYqr_NGWFwE,3693
+firecrawl/v2/methods/aio/__init__.py,sha256=RocMJnGwnLIvGu3G8ZvY8INkipC7WHZiu2bE31eSyJs,35
+firecrawl/v2/methods/aio/agent.py,sha256=rcGlFkaDgfzglPl_aCQ7k0kNHqV8A6dBW3E4b3DauLc,4323
+firecrawl/v2/methods/aio/batch.py,sha256=_jhoM4w9zVCQ9qb3sIKp9BTvkV_8IT_PnbhJcaiVcso,6975
+firecrawl/v2/methods/aio/crawl.py,sha256=X5P4X_kEI2-Fcm4p46I-qdUW-RomlreAQNBxztX2pfo,12244
+firecrawl/v2/methods/aio/extract.py,sha256=oc7LcjJ3g3nGYJeedEn2YWOg8X0NqgQpd0DrlI0SyiU,4516
+firecrawl/v2/methods/aio/map.py,sha256=ZFGthx3BuTZ1elewssceHtmyWdlAMK2B97vSnHF-ELA,2683
+firecrawl/v2/methods/aio/scrape.py,sha256=ilA9qco8YGwCFpE0PN1XBQUyuHPQwH2QioZ-xsfxhgU,1386
+firecrawl/v2/methods/aio/search.py,sha256=d1SFbK1HtBKvR8qCvJQlJQCkN-3pbire80Fbbn7CnSw,6431
+firecrawl/v2/methods/aio/usage.py,sha256=iUzTkdAWRheq-V5rRXcW0bc3MSODaVS1AqroRF0fO9M,3964
+firecrawl/v2/utils/__init__.py,sha256=i1GgxySmqEXpWSBQCu3iZBPIJG7fXj0QXCDWGwerWNs,338
+firecrawl/v2/utils/error_handler.py,sha256=Iuf916dHphDY8ObNNlWy75628DFeJ0Rv8ljRp4LttLE,4199
+firecrawl/v2/utils/get_version.py,sha256=0CxW_41q2hlzIxEWOivUCaYw3GFiSIH32RPUMcIgwAY,492
+firecrawl/v2/utils/http_client.py,sha256=0hII3mnF_1Vd1nElu-hC9PipTUABGamUKb27q92_m5E,6068
+firecrawl/v2/utils/http_client_async.py,sha256=Mt6Dw_i2R_W81ONXnl9N_AlPiggfylOPfbD5Rpgi7tA,1991
+firecrawl/v2/utils/normalize.py,sha256=vjZFT0q-Nl5nGyRd21cWbJhzvnu0FtIysNI7tbxp1zk,4173
+firecrawl/v2/utils/validation.py,sha256=LQo7iNuC-stjDBfUml8h6NoWpnxdgJ2kfzD-63iJbuM,27330
+firecrawl-4.12.0.dist-info/licenses/LICENSE,sha256=nPCunEDwjRGHlmjvsiDUyIWbkqqyj3Ej84ntnh0g0zA,1084
+tests/test_agent_integration.py,sha256=8Uo0K4pOjFQV6F_2Wu1_2sYYlCdci9gbvUHpiAb9TxY,10217
+tests/test_api_key_handling.py,sha256=iNaHp6zc9bIwpN3DdiWB2Rzk0j7HCP7VgpRE_1byNYc,1303
+tests/test_change_tracking.py,sha256=_IJ5ShLcoj2fHDBaw-nE4I4lHdmDB617ocK_XMHhXps,4177
+tests/test_timeout_conversion.py,sha256=PWlIEMASQNhu4cp1OW_ebklnE9NCiigPnEFCtI5N3w0,3996
+firecrawl-4.12.0.dist-info/METADATA,sha256=7qxysEMUMo2zFMRSk-_F1dZ1XmY1rDFlKbHn9q0oAGs,7393
+firecrawl-4.12.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+firecrawl-4.12.0.dist-info/top_level.txt,sha256=8T3jOaSN5mtLghO-R3MQ8KO290gIX8hmfxQmglBPdLE,16
+firecrawl-4.12.0.dist-info/RECORD,,

firecrawl-4.12.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

firecrawl-4.12.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Sideguide Technologies Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

firecrawl-4.12.0.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+firecrawl
+tests

tests/test_agent_integration.py

@@ -0,0 +1,277 @@
+"""
+Integration tests for agent method with mocked requests.
+"""
+
+import unittest
+from unittest.mock import patch, MagicMock
+from pydantic import BaseModel, Field
+from typing import List, Optional
+
+from firecrawl import FirecrawlApp
+
+
+class Founder(BaseModel):
+    name: str = Field(description="Full name of the founder")
+    role: Optional[str] = Field(None, description="Role or position")
+    background: Optional[str] = Field(None, description="Professional background")
+
+
+class FoundersSchema(BaseModel):
+    founders: List[Founder] = Field(description="List of founders")
+
+
+class TestAgent(unittest.TestCase):
+    """Integration tests for agent method."""
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    @patch('firecrawl.v2.utils.http_client.requests.get')
+    def test_agent_basic(self, mock_get, mock_post):
+        """Test basic agent call."""
+        # Mock start agent response
+        mock_start_response = MagicMock()
+        mock_start_response.ok = True
+        mock_start_response.status_code = 200
+        mock_start_response.json.return_value = {
+            "success": True,
+            "id": "test-agent-123",
+            "status": "processing"
+        }
+        mock_post.return_value = mock_start_response
+        
+        # Mock get status response (completed)
+        mock_status_response = MagicMock()
+        mock_status_response.ok = True
+        mock_status_response.status_code = 200
+        mock_status_response.json.return_value = {
+            "success": True,
+            "id": "test-agent-123",
+            "status": "completed",
+            "data": {
+                "founders": [
+                    {"name": "John Doe", "role": "CEO", "background": "Tech entrepreneur"},
+                    {"name": "Jane Smith", "role": "CTO", "background": "Software engineer"}
+                ]
+            },
+            "creditsUsed": 10,
+            "expiresAt": "2024-01-01T00:00:00Z"
+        }
+        mock_get.return_value = mock_status_response
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(
+            prompt="Find the founders of Firecrawl",
+            schema=FoundersSchema
+        )
+        
+        # Verify post was called with correct URL and data
+        mock_post.assert_called_once()
+        post_call_args = mock_post.call_args
+        post_url = post_call_args[1]["url"] if "url" in post_call_args[1] else post_call_args[0][0]
+        assert "/v2/agent" in str(post_url)
+        
+        # Check request body
+        request_body = post_call_args[1]["json"]
+        assert request_body["prompt"] == "Find the founders of Firecrawl"
+        assert "schema" in request_body
+        assert request_body["schema"]["type"] == "object"
+        assert "founders" in request_body["schema"]["properties"]
+        
+        # Verify get was called to check status
+        mock_get.assert_called()
+        
+        # Check result
+        assert result.status == "completed"
+        assert result.data is not None
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    def test_agent_with_urls(self, mock_post):
+        """Test agent call with URLs."""
+        mock_response = MagicMock()
+        mock_response.ok = True
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            "success": True,
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_post.return_value = mock_response
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(
+            urls=["https://example.com", "https://test.com"],
+            prompt="Extract information",
+            schema={"type": "object", "properties": {"info": {"type": "string"}}}
+        )
+        
+        # Check request body includes URLs
+        post_call_args = mock_post.call_args
+        request_body = post_call_args[1]["json"]
+        assert request_body["urls"] == ["https://example.com", "https://test.com"]
+        assert request_body["prompt"] == "Extract information"
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    def test_agent_with_dict_schema(self, mock_post):
+        """Test agent call with dict schema."""
+        mock_response = MagicMock()
+        mock_response.ok = True
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            "success": True,
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_post.return_value = mock_response
+        
+        schema = {
+            "type": "object",
+            "properties": {
+                "name": {"type": "string"},
+                "age": {"type": "integer"}
+            }
+        }
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(
+            prompt="Extract person data",
+            schema=schema
+        )
+        
+        # Check request body includes schema
+        post_call_args = mock_post.call_args
+        request_body = post_call_args[1]["json"]
+        assert request_body["schema"] == schema
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    def test_agent_with_all_params(self, mock_post):
+        """Test agent call with all parameters."""
+        mock_response = MagicMock()
+        mock_response.ok = True
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            "success": True,
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_post.return_value = mock_response
+        
+        schema = {"type": "object"}
+        urls = ["https://example.com"]
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(
+            urls=urls,
+            prompt="Complete test",
+            schema=schema,
+            integration="test-integration",
+            max_credits=50,
+            strict_constrain_to_urls=True,
+            poll_interval=1,
+            timeout=30
+        )
+        
+        # Check all parameters are in request body
+        post_call_args = mock_post.call_args
+        request_body = post_call_args[1]["json"]
+        assert request_body["prompt"] == "Complete test"
+        assert request_body["urls"] == urls
+        assert request_body["schema"] == schema
+        assert request_body["integration"] == "test-integration"
+        assert request_body["maxCredits"] == 50
+        assert request_body["strictConstrainToURLs"] is True
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    def test_agent_pydantic_schema_normalization(self, mock_post):
+        """Test that Pydantic schemas are properly normalized."""
+        mock_response = MagicMock()
+        mock_response.ok = True
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            "success": True,
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_post.return_value = mock_response
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(
+            prompt="Find founders",
+            schema=FoundersSchema
+        )
+        
+        # Check that schema was normalized to JSON schema format
+        post_call_args = mock_post.call_args
+        request_body = post_call_args[1]["json"]
+        assert "schema" in request_body
+        schema = request_body["schema"]
+        assert schema["type"] == "object"
+        assert "properties" in schema
+        assert "founders" in schema["properties"]
+        assert schema["properties"]["founders"]["type"] == "array"
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    @patch('firecrawl.v2.utils.http_client.requests.get')
+    def test_agent_url_construction(self, mock_get, mock_post):
+        """Test that agent requests are sent to correct URL."""
+        # Mock start agent response
+        mock_start_response = MagicMock()
+        mock_start_response.ok = True
+        mock_start_response.status_code = 200
+        mock_start_response.json.return_value = {
+            "success": True,
+            "id": "test-agent-123",
+            "status": "processing"
+        }
+        mock_post.return_value = mock_start_response
+        
+        # Mock get status response
+        mock_status_response = MagicMock()
+        mock_status_response.ok = True
+        mock_status_response.status_code = 200
+        mock_status_response.json.return_value = {
+            "success": True,
+            "id": "test-agent-123",
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_get.return_value = mock_status_response
+        
+        app = FirecrawlApp(api_key="test-api-key", api_url="https://api.firecrawl.dev")
+        result = app.agent(prompt="Test prompt")
+        
+        # Check POST URL - requests.post is called with url as keyword arg
+        post_call_args = mock_post.call_args
+        post_url = post_call_args[1].get("url") if "url" in post_call_args[1] else post_call_args[0][0]
+        assert "/v2/agent" in str(post_url)
+        
+        # Check GET URL
+        get_call_args = mock_get.call_args
+        get_url = get_call_args[1].get("url") if "url" in get_call_args[1] else get_call_args[0][0]
+        assert "/v2/agent/test-agent-123" in str(get_url)
+
+    @patch('firecrawl.v2.utils.http_client.requests.post')
+    def test_agent_headers(self, mock_post):
+        """Test that agent requests include correct headers."""
+        mock_response = MagicMock()
+        mock_response.ok = True
+        mock_response.status_code = 200
+        mock_response.json.return_value = {
+            "success": True,
+            "status": "completed",
+            "data": {"result": "done"}
+        }
+        mock_post.return_value = mock_response
+        
+        app = FirecrawlApp(api_key="test-api-key")
+        result = app.agent(prompt="Test prompt")
+        
+        # Check headers
+        post_call_args = mock_post.call_args
+        headers = post_call_args[1]["headers"]
+        assert "Authorization" in headers
+        assert headers["Authorization"] == "Bearer test-api-key"
+        assert headers["Content-Type"] == "application/json"
+
+
+if __name__ == '__main__':
+    unittest.main()
+