webcrawlerapi-langchain 0.1.0__tar.gz → 0.2.0__tar.gz

{webcrawlerapi_langchain-0.1.0 → webcrawlerapi_langchain-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webcrawlerapi-langchain
-Version: 0.1.0
+Version: 0.2.0
 Summary: LangChain integration for WebCrawlerAPI
 Home-page: https://github.com/webcrawlerapi/webcrawlerapi-langchain
 Author: WebCrawlerAPI
@@ -8,10 +8,10 @@ Author-email: support@webcrawlerapi.com
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.8
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
-Requires-Dist: webcrawlerapi==1.0.6
-Requires-Dist: langchain-core>=0.1.0
+Requires-Dist: webcrawlerapi>=2.1.5
+Requires-Dist: langchain-core>=0.3.0
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -26,7 +26,7 @@ Dynamic: summary
 
 [WebcrawlerAPI](https://webcrawlerapi.com/) - is a website to LLM data API. It allows to convert websites and webpages markdown or cleaned content. 
 
-**No subscription required**.
+**Pay as you go (No subscription required).**
 
 This package provides LangChain integration for [WebCrawlerAPI](https://webcrawlerapi.com/), allowing you to easily use web crawling capabilities with [LangChain](https://www.langchain.com/) document processing pipeline.
 
@@ -40,6 +40,8 @@ pip install webcrawlerapi-langchain
 
 ## Usage
 
+Requires Python 3.10 or newer. `api_key` and `base_url` fall back to the `WEBCRAWLERAPI_API_KEY` and `WEBCRAWLERAPI_BASE_URL` environment variables when not passed.
+
 ### Basic Loading
 ```python
 from webcrawlerapi_langchain import WebCrawlerAPILoader
@@ -48,7 +50,7 @@ from webcrawlerapi_langchain import WebCrawlerAPILoader
 loader = WebCrawlerAPILoader(
     url="https://example.com",
     api_key="your-api-key",
-    scrape_type="markdown",
+    output_format="markdown",
     items_limit=10
 )
 
@@ -68,6 +70,7 @@ documents = await loader.aload()
 ```
 
 ### Lazy Loading
+Yields each page as soon as it finishes crawling:
 ```python
 # Lazy loading
 for doc in loader.lazy_load():
@@ -81,16 +84,37 @@ async for doc in loader.alazy_load():
     print(doc.page_content[:100])
 ```
 
+### Error Handling
+Crawling and API failures raise `WebCrawlerAPILoaderError`, which is importable from `webcrawlerapi_langchain`. Invalid parameters raise `ValueError`.
+```python
+from webcrawlerapi_langchain import WebCrawlerAPILoaderError
+
+try:
+    documents = loader.load()
+except WebCrawlerAPILoaderError as e:
+    print(f"Crawl failed: {e}")
+```
+
 ## Configuration
 
 The loader accepts the following parameters:
 
-- `url`: The URL to crawl
-- `api_key`: Your WebCrawlerAPI API key
-- `scrape_type`: Type of scraping (html, cleaned, markdown)
-- `items_limit`: Maximum number of pages to crawl
+- `url`: The URL to crawl (keyword arguments follow it)
+- `api_key`: Your WebCrawlerAPI API key. Defaults to `WEBCRAWLERAPI_API_KEY`
+- `base_url`: API base URL. Defaults to `WEBCRAWLERAPI_BASE_URL`
+- `output_format`: Content format of each Document: `markdown` (default), `cleaned`, or `html`
+- `scrape_type`: Deprecated, use `output_format` instead
+- `items_limit`: Maximum number of pages to crawl (default `10`)
 - `whitelist_regexp`: Regex pattern for URL whitelist
 - `blacklist_regexp`: Regex pattern for URL blacklist
+- `main_content_only`: Extract only the main content of each page (default `False`)
+- `max_depth`: Maximum crawl depth. `0` is the seed URL only, `1` is the seed URL plus direct links
+- `max_age`: Max age in seconds for cached content. `0` always fetches fresh content
+- `respect_robots_txt`: Whether to respect robots.txt (default `False`)
+- `keep_query_params`: Keep URL query params when deduplicating links
+- `max_polls`: Maximum number of job status checks before giving up (default `100`)
+
+Document metadata keys: `url`, `title`, `status_code`, `created_at`, `referred_url`, `depth`, `cost`, `job_id`, `item_id`.
 
 ### Links
 - [WebCrawlerAPI Python SDK](https://github.com/WebCrawlerAPI/webcrawlerapi-python-sdk)

{webcrawlerapi_langchain-0.1.0 → webcrawlerapi_langchain-0.2.0}/README.md

@@ -2,7 +2,7 @@
 
 [WebcrawlerAPI](https://webcrawlerapi.com/) - is a website to LLM data API. It allows to convert websites and webpages markdown or cleaned content. 
 
-**No subscription required**.
+**Pay as you go (No subscription required).**
 
 This package provides LangChain integration for [WebCrawlerAPI](https://webcrawlerapi.com/), allowing you to easily use web crawling capabilities with [LangChain](https://www.langchain.com/) document processing pipeline.
 
@@ -16,6 +16,8 @@ pip install webcrawlerapi-langchain
 
 ## Usage
 
+Requires Python 3.10 or newer. `api_key` and `base_url` fall back to the `WEBCRAWLERAPI_API_KEY` and `WEBCRAWLERAPI_BASE_URL` environment variables when not passed.
+
 ### Basic Loading
 ```python
 from webcrawlerapi_langchain import WebCrawlerAPILoader
@@ -24,7 +26,7 @@ from webcrawlerapi_langchain import WebCrawlerAPILoader
 loader = WebCrawlerAPILoader(
     url="https://example.com",
     api_key="your-api-key",
-    scrape_type="markdown",
+    output_format="markdown",
     items_limit=10
 )
 
@@ -44,6 +46,7 @@ documents = await loader.aload()
 ```
 
 ### Lazy Loading
+Yields each page as soon as it finishes crawling:
 ```python
 # Lazy loading
 for doc in loader.lazy_load():
@@ -57,16 +60,37 @@ async for doc in loader.alazy_load():
     print(doc.page_content[:100])
 ```
 
+### Error Handling
+Crawling and API failures raise `WebCrawlerAPILoaderError`, which is importable from `webcrawlerapi_langchain`. Invalid parameters raise `ValueError`.
+```python
+from webcrawlerapi_langchain import WebCrawlerAPILoaderError
+
+try:
+    documents = loader.load()
+except WebCrawlerAPILoaderError as e:
+    print(f"Crawl failed: {e}")
+```
+
 ## Configuration
 
 The loader accepts the following parameters:
 
-- `url`: The URL to crawl
-- `api_key`: Your WebCrawlerAPI API key
-- `scrape_type`: Type of scraping (html, cleaned, markdown)
-- `items_limit`: Maximum number of pages to crawl
+- `url`: The URL to crawl (keyword arguments follow it)
+- `api_key`: Your WebCrawlerAPI API key. Defaults to `WEBCRAWLERAPI_API_KEY`
+- `base_url`: API base URL. Defaults to `WEBCRAWLERAPI_BASE_URL`
+- `output_format`: Content format of each Document: `markdown` (default), `cleaned`, or `html`
+- `scrape_type`: Deprecated, use `output_format` instead
+- `items_limit`: Maximum number of pages to crawl (default `10`)
 - `whitelist_regexp`: Regex pattern for URL whitelist
 - `blacklist_regexp`: Regex pattern for URL blacklist
+- `main_content_only`: Extract only the main content of each page (default `False`)
+- `max_depth`: Maximum crawl depth. `0` is the seed URL only, `1` is the seed URL plus direct links
+- `max_age`: Max age in seconds for cached content. `0` always fetches fresh content
+- `respect_robots_txt`: Whether to respect robots.txt (default `False`)
+- `keep_query_params`: Keep URL query params when deduplicating links
+- `max_polls`: Maximum number of job status checks before giving up (default `100`)
+
+Document metadata keys: `url`, `title`, `status_code`, `created_at`, `referred_url`, `depth`, `cost`, `job_id`, `item_id`.
 
 ### Links
 - [WebCrawlerAPI Python SDK](https://github.com/WebCrawlerAPI/webcrawlerapi-python-sdk)

{webcrawlerapi_langchain-0.1.0 → webcrawlerapi_langchain-0.2.0}/setup.py

@@ -2,11 +2,11 @@ from setuptools import setup, find_packages
 
 setup(
     name="webcrawlerapi-langchain",
-    version="0.1.0",
+    version="0.2.0",
     packages=find_packages(),
     install_requires=[
-        "webcrawlerapi==1.0.6",
-        "langchain-core>=0.1.0",
+        "webcrawlerapi>=2.1.5",
+        "langchain-core>=0.3.0",
     ],
     author="WebCrawlerAPI",
     author_email="support@webcrawlerapi.com",
@@ -19,5 +19,5 @@ setup(
         "License :: OSI Approved :: MIT License",
         "Operating System :: OS Independent",
     ],
-    python_requires=">=3.8",
+    python_requires=">=3.10",
 ) 

webcrawlerapi_langchain-0.2.0/webcrawlerapi_langchain/__init__.py

@@ -0,0 +1,4 @@
+from .loader import WebCrawlerAPILoader, WebCrawlerAPILoaderError
+
+__version__ = "0.2.0"
+__all__ = ["WebCrawlerAPILoader", "WebCrawlerAPILoaderError"]

{webcrawlerapi_langchain-0.1.0 → webcrawlerapi_langchain-0.2.0}/webcrawlerapi_langchain/loader.py

@@ -1,7 +1,9 @@
-from typing import Dict, Iterator, List, Optional, AsyncIterator, Any, Literal
-import os
+from typing import Iterator, List, Optional, AsyncIterator, Any, Literal
+import asyncio
 import json
 import logging
+import time
+import warnings
 from langchain_core.documents import Document
 from langchain_core.utils import get_from_env
 from langchain_core.document_loaders import BaseLoader
@@ -9,12 +11,16 @@ from webcrawlerapi import WebCrawlerAPI
 
 # Configure logging
 logger = logging.getLogger(__name__)
-logger.setLevel(logging.DEBUG)
+
+OutputFormat = Literal["markdown", "cleaned", "html"]
+ALLOWED_OUTPUT_FORMATS = ("markdown", "cleaned", "html")
+
 
 class WebCrawlerAPILoaderError(Exception):
     """Custom exception class for WebCrawlerAPILoader errors."""
     pass
 
+
 class WebCrawlerAPILoader(BaseLoader):
     """WebCrawlerAPI document loader integration.
 
@@ -32,12 +38,16 @@ class WebCrawlerAPILoader(BaseLoader):
         *,
         api_key: Optional[str] = None,
         base_url: Optional[str] = None,
-        version: str = "v1",
-        scrape_type: Literal["html", "cleaned", "markdown"] = "markdown",
+        output_format: Optional[OutputFormat] = None,
+        scrape_type: Optional[OutputFormat] = None,
         items_limit: int = 10,
-        allow_subdomains: bool = False,
         whitelist_regexp: Optional[str] = None,
         blacklist_regexp: Optional[str] = None,
+        main_content_only: bool = False,
+        max_depth: Optional[int] = None,
+        max_age: Optional[int] = None,
+        respect_robots_txt: bool = False,
+        keep_query_params: Optional[bool] = None,
         max_polls: int = 100
     ):
         """Initialize the WebCrawlerAPI document loader.
@@ -46,13 +56,17 @@ class WebCrawlerAPILoader(BaseLoader):
             url: The URL to crawl
             api_key: Your WebCrawlerAPI API key. If not provided, will try to get from WEBCRAWLERAPI_API_KEY env var
             base_url: The base URL of the API. If not provided, will try to get from WEBCRAWLERAPI_BASE_URL env var
-            version: API version to use (optional)
-            scrape_type: Type of scraping (html, cleaned, markdown)
+            output_format: Content format of each Document (markdown, cleaned, html). Defaults to markdown
+            scrape_type: Deprecated. Use output_format instead
             items_limit: Maximum number of pages to crawl
-            allow_subdomains: Whether to crawl subdomains
             whitelist_regexp: Regex pattern for URL whitelist
             blacklist_regexp: Regex pattern for URL blacklist
-            max_polls: Maximum number of status checks before returning
+            main_content_only: Extract only the main content of each page
+            max_depth: Maximum crawl depth (0 = seed only, 1 = seed + direct links)
+            max_age: Max age in seconds for cached content. 0 = always fresh
+            respect_robots_txt: Whether to respect robots.txt
+            keep_query_params: Keep URL query params when deduplicating links
+            max_polls: Maximum number of status checks before giving up
         """
         if not url:
             raise ValueError("URL must be provided")
@@ -66,10 +80,17 @@ class WebCrawlerAPILoader(BaseLoader):
         if max_polls < 1:
             raise ValueError("max_polls must be greater than 0")
 
-        if scrape_type not in ("html", "cleaned", "markdown"):
+        if scrape_type is not None:
+            warnings.warn(
+                "scrape_type is deprecated, use output_format instead",
+                DeprecationWarning,
+                stacklevel=2,
+            )
+        output_format = output_format or scrape_type or "markdown"
+        if output_format not in ALLOWED_OUTPUT_FORMATS:
             raise ValueError(
-                f"Invalid scrape_type '{scrape_type}'. "
-                "Allowed: 'html', 'cleaned', 'markdown'."
+                f"Invalid output_format '{output_format}'. "
+                "Allowed: 'markdown', 'cleaned', 'html'."
             )
 
         # Get API key and base URL from env vars if not provided
@@ -78,26 +99,46 @@ class WebCrawlerAPILoader(BaseLoader):
             raise ValueError("API key must be provided either through api_key parameter or WEBCRAWLERAPI_API_KEY environment variable")
 
         self.base_url = base_url or get_from_env(
-            "base_url", 
-            "WEBCRAWLERAPI_BASE_URL", 
+            "base_url",
+            "WEBCRAWLERAPI_BASE_URL",
             default="https://api.webcrawlerapi.com"
         )
 
         self.url = url
-        self.version = version
-        self.scrape_type = scrape_type
+        self.output_format = output_format
         self.items_limit = items_limit
-        self.allow_subdomains = allow_subdomains
         self.whitelist_regexp = whitelist_regexp
         self.blacklist_regexp = blacklist_regexp
+        self.main_content_only = main_content_only
+        self.max_depth = max_depth
+        self.max_age = max_age
+        self.respect_robots_txt = respect_robots_txt
+        self.keep_query_params = keep_query_params
         self.max_polls = max_polls
 
-        logger.debug(f"Initializing WebCrawlerAPILoader with URL: {url}, base_url: {self.base_url}")
-        self.client = WebCrawlerAPI(
-            api_key=self.api_key,
-            base_url=self.base_url,
-            version=version
-        )
+        self.client = WebCrawlerAPI(api_key=self.api_key, base_url=self.base_url)
+
+    def _crawl_params(self) -> dict:
+        return {
+            "url": self.url,
+            "output_formats": [self.output_format],
+            "items_limit": self.items_limit,
+            "whitelist_regexp": self.whitelist_regexp,
+            "blacklist_regexp": self.blacklist_regexp,
+            "main_content_only": self.main_content_only,
+            "max_depth": self.max_depth,
+            "max_age": self.max_age,
+            "respect_robots_txt": self.respect_robots_txt,
+            "keep_query_params": self.keep_query_params,
+        }
+
+    def _fetch_item_content(self, item: Any) -> Optional[str]:
+        """Fetch item content in the configured output format."""
+        if self.output_format == "markdown":
+            return item.get_markdown()
+        if self.output_format == "cleaned":
+            return item.get_cleaned()
+        return item.get_html()
 
     def _create_document(self, item: Any) -> Optional[Document]:
         """Create a Document from a job item if it's valid.
@@ -106,30 +147,46 @@ class WebCrawlerAPILoader(BaseLoader):
             item: Job item from the API response
 
         Returns:
-            Document if item is valid and has content, None otherwise
+            Document if item is done and has content, None otherwise
         """
         try:
-            logger.debug(f"Processing job item: {item}")
-            if not (item.status == "done" and item.content):
-                logger.debug(f"Skipping item - status: {item.status}, has_content: {bool(item.content)}")
+            if item.status != "done":
+                return None
+
+            content = self._fetch_item_content(item)
+            if not content:
                 return None
 
-            doc = Document(
-                page_content=item.content,
+            return Document(
+                page_content=content,
                 metadata={
                     "url": item.original_url,
                     "title": item.title,
                     "status_code": item.page_status_code,
                     "created_at": item.created_at,
                     "referred_url": item.referred_url,
-                    "cost": item.cost
+                    "depth": item.depth,
+                    "cost": item.cost,
+                    "job_id": item.job_id,
+                    "item_id": item.id,
                 }
             )
-            logger.debug(f"Created document from item: {item.original_url}")
-            return doc
         except AttributeError as e:
             logger.error(f"Failed to create document from item: {e}")
             raise WebCrawlerAPILoaderError(f"Invalid job item format: {str(e)}") from e
+        except Exception as e:
+            logger.error(f"Failed to fetch content for {getattr(item, 'original_url', '?')}: {e}")
+            raise WebCrawlerAPILoaderError(f"Failed to fetch item content: {str(e)}") from e
+
+    @staticmethod
+    def _job_error_message(job: Any) -> str:
+        """Build an error message from failed job items, as the job itself carries no error field."""
+        errors = [
+            f"{item.original_url}: {item.error_code or 'error'} {item.last_error or ''}".strip()
+            for item in job.job_items
+            if item.status == "error"
+        ]
+        return "; ".join(errors) if errors else "Unknown error"
 
     def load(self) -> List[Document]:
         """Load data into Document objects.
@@ -139,27 +196,10 @@ class WebCrawlerAPILoader(BaseLoader):
 
         Raises:
             WebCrawlerAPILoaderError: If there are issues with the crawling job or API response
-            ValueError: If the response format is invalid
-            json.JSONDecodeError: If the API response contains invalid JSON
         """
         logger.info(f"Starting crawl for URL: {self.url}")
         try:
-            logger.debug("Making crawl request with params: "
-                        f"scrape_type={self.scrape_type}, "
-                        f"items_limit={self.items_limit}, "
-                        f"allow_subdomains={self.allow_subdomains}")
-            
-            job = self.client.crawl(
-                url=self.url,
-                scrape_type=self.scrape_type,
-                items_limit=self.items_limit,
-                allow_subdomains=self.allow_subdomains,
-                whitelist_regexp=self.whitelist_regexp,
-                blacklist_regexp=self.blacklist_regexp,
-                max_polls=self.max_polls
-            )
-            logger.debug(f"Received crawl response: {job}")
-            
+            job = self.client.crawl(**self._crawl_params(), max_polls=self.max_polls)
         except json.JSONDecodeError as e:
             logger.error(f"JSON decode error in crawl response: {e}")
             raise WebCrawlerAPILoaderError(
@@ -170,22 +210,21 @@ class WebCrawlerAPILoader(BaseLoader):
             raise WebCrawlerAPILoaderError(f"API request failed: {str(e)}") from e
 
         if job.status == "error":
-            error_msg = getattr(job, 'error', 'Unknown error')
+            error_msg = self._job_error_message(job)
             logger.error(f"Crawl job failed with error: {error_msg}")
             raise WebCrawlerAPILoaderError(f"Crawling job failed: {error_msg}")
 
-        documents = []
+        if not job.is_terminal:
+            raise WebCrawlerAPILoaderError(
+                f"Maximum number of polls ({self.max_polls}) reached without job completion"
+            )
+
         logger.info(f"Processing {len(job.job_items)} job items")
+        documents = []
         for item in job.job_items:
-            try:
-                doc = self._create_document(item)
-                if doc:
-                    documents.append(doc)
-            except AttributeError as e:
-                logger.error(f"Failed to process job item: {e}")
-                raise WebCrawlerAPILoaderError(
-                    f"Invalid job item format - missing required field: {str(e)}"
-                ) from e
+            doc = self._create_document(item)
+            if doc:
+                documents.append(doc)
 
         logger.info(f"Successfully created {len(documents)} documents")
         return documents
@@ -194,26 +233,14 @@ class WebCrawlerAPILoader(BaseLoader):
         """A lazy loader for Documents.
 
         Yields:
-            Document objects one at a time as they are crawled.
+            Document objects one at a time as pages finish crawling.
 
         Raises:
             WebCrawlerAPILoaderError: If there are issues with the crawling job or API response
-            ValueError: If the response format is invalid
-            json.JSONDecodeError: If the API response contains invalid JSON
         """
         logger.info(f"Starting async crawl for URL: {self.url}")
         try:
-            logger.debug("Making async crawl request")
-            response = self.client.crawl_async(
-                url=self.url,
-                scrape_type=self.scrape_type,
-                items_limit=self.items_limit,
-                allow_subdomains=self.allow_subdomains,
-                whitelist_regexp=self.whitelist_regexp,
-                blacklist_regexp=self.blacklist_regexp
-            )
-            logger.debug(f"Received async crawl response: {response}")
-            
+            response = self.client.crawl_async(**self._crawl_params())
         except json.JSONDecodeError as e:
             logger.error(f"JSON decode error in async crawl response: {e}")
             raise WebCrawlerAPILoaderError(
@@ -224,16 +251,12 @@ class WebCrawlerAPILoader(BaseLoader):
             raise WebCrawlerAPILoaderError(f"API request failed: {str(e)}") from e
 
         job_id = response.id
-        polls = 0
         processed_items = set()
         logger.info(f"Starting to poll job {job_id}")
 
-        while polls < self.max_polls:
+        for _ in range(self.max_polls):
             try:
-                logger.debug(f"Polling job {job_id} (attempt {polls + 1}/{self.max_polls})")
                 job = self.client.get_job(job_id)
-                logger.debug(f"Received job status: {job.status}")
-                
             except json.JSONDecodeError as e:
                 logger.error(f"JSON decode error in job status response: {e}")
                 raise WebCrawlerAPILoaderError(
@@ -244,44 +267,34 @@ class WebCrawlerAPILoader(BaseLoader):
                 raise WebCrawlerAPILoaderError(f"Failed to fetch job status: {str(e)}") from e
 
             if job.status == "error":
-                error_msg = getattr(job, 'error', 'Unknown error')
+                error_msg = self._job_error_message(job)
                 logger.error(f"Job failed with error: {error_msg}")
                 raise WebCrawlerAPILoaderError(f"Crawling job failed: {error_msg}")
 
+            # Yield pages as soon as each item is done, without waiting for the whole job
             for item in job.job_items:
-                if item.id not in processed_items:
-                    try:
-                        doc = self._create_document(item)
-                        if doc:
-                            processed_items.add(item.id)
-                            logger.debug(f"Yielding document for URL: {item.original_url}")
-                            yield doc
-                    except AttributeError as e:
-                        logger.error(f"Failed to process job item: {e}")
-                        raise WebCrawlerAPILoaderError(
-                            f"Invalid job item format - missing required field: {str(e)}"
-                        ) from e
+                if item.id in processed_items or item.status not in ("done", "error"):
+                    continue
+                processed_items.add(item.id)
+                doc = self._create_document(item)
+                if doc:
+                    yield doc
 
             if job.is_terminal:
-                logger.info("Job completed successfully")
-                break
+                logger.info("Job completed")
+                return
 
             delay_seconds = (
                 job.recommended_pull_delay_ms / 1000
                 if job.recommended_pull_delay_ms
                 else self.client.DEFAULT_POLL_DELAY_SECONDS
             )
-
-            import time
-            logger.debug(f"Waiting {delay_seconds} seconds before next poll")
             time.sleep(delay_seconds)
-            polls += 1
 
-        if not job.is_terminal and polls >= self.max_polls:
-            logger.error(f"Job timed out after {self.max_polls} polls")
-            raise WebCrawlerAPILoaderError(
-                f"Maximum number of polls ({self.max_polls}) reached without job completion"
-            )
+        logger.error(f"Job timed out after {self.max_polls} polls")
+        raise WebCrawlerAPILoaderError(
+            f"Maximum number of polls ({self.max_polls}) reached without job completion"
+        )
 
     async def aload(self) -> List[Document]:
         """Asynchronously load data into Document objects.
@@ -290,21 +303,23 @@ class WebCrawlerAPILoader(BaseLoader):
             List of Document objects, one for each crawled page.
 
         Raises:
-            RuntimeError: If the crawling job fails
+            WebCrawlerAPILoaderError: If the crawling job fails
         """
-        import asyncio
         return await asyncio.to_thread(self.load)
 
     async def alazy_load(self) -> AsyncIterator[Document]:
         """An async lazy loader for Documents.
 
         Yields:
-            Document objects one at a time as they are crawled.
+            Document objects one at a time as pages finish crawling.
 
         Raises:
-            RuntimeError: If the crawling job fails
+            WebCrawlerAPILoaderError: If the crawling job fails
         """
-        import asyncio
-        for doc in self.lazy_load():
+        iterator = self.lazy_load()
+        sentinel = object()
+        while True:
+            doc = await asyncio.to_thread(next, iterator, sentinel)
+            if doc is sentinel:
+                break
             yield doc
-            await asyncio.sleep(0) 

{webcrawlerapi_langchain-0.1.0 → webcrawlerapi_langchain-0.2.0}/webcrawlerapi_langchain.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webcrawlerapi-langchain
-Version: 0.1.0
+Version: 0.2.0
 Summary: LangChain integration for WebCrawlerAPI
 Home-page: https://github.com/webcrawlerapi/webcrawlerapi-langchain
 Author: WebCrawlerAPI
@@ -8,10 +8,10 @@ Author-email: support@webcrawlerapi.com
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.8
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
-Requires-Dist: webcrawlerapi==1.0.6
-Requires-Dist: langchain-core>=0.1.0
+Requires-Dist: webcrawlerapi>=2.1.5
+Requires-Dist: langchain-core>=0.3.0
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -26,7 +26,7 @@ Dynamic: summary
 
 [WebcrawlerAPI](https://webcrawlerapi.com/) - is a website to LLM data API. It allows to convert websites and webpages markdown or cleaned content. 
 
-**No subscription required**.
+**Pay as you go (No subscription required).**
 
 This package provides LangChain integration for [WebCrawlerAPI](https://webcrawlerapi.com/), allowing you to easily use web crawling capabilities with [LangChain](https://www.langchain.com/) document processing pipeline.
 
@@ -40,6 +40,8 @@ pip install webcrawlerapi-langchain
 
 ## Usage
 
+Requires Python 3.10 or newer. `api_key` and `base_url` fall back to the `WEBCRAWLERAPI_API_KEY` and `WEBCRAWLERAPI_BASE_URL` environment variables when not passed.
+
 ### Basic Loading
 ```python
 from webcrawlerapi_langchain import WebCrawlerAPILoader
@@ -48,7 +50,7 @@ from webcrawlerapi_langchain import WebCrawlerAPILoader
 loader = WebCrawlerAPILoader(
     url="https://example.com",
     api_key="your-api-key",
-    scrape_type="markdown",
+    output_format="markdown",
     items_limit=10
 )
 
@@ -68,6 +70,7 @@ documents = await loader.aload()
 ```
 
 ### Lazy Loading
+Yields each page as soon as it finishes crawling:
 ```python
 # Lazy loading
 for doc in loader.lazy_load():
@@ -81,16 +84,37 @@ async for doc in loader.alazy_load():
     print(doc.page_content[:100])
 ```
 
+### Error Handling
+Crawling and API failures raise `WebCrawlerAPILoaderError`, which is importable from `webcrawlerapi_langchain`. Invalid parameters raise `ValueError`.
+```python
+from webcrawlerapi_langchain import WebCrawlerAPILoaderError
+
+try:
+    documents = loader.load()
+except WebCrawlerAPILoaderError as e:
+    print(f"Crawl failed: {e}")
+```
+
 ## Configuration
 
 The loader accepts the following parameters:
 
-- `url`: The URL to crawl
-- `api_key`: Your WebCrawlerAPI API key
-- `scrape_type`: Type of scraping (html, cleaned, markdown)
-- `items_limit`: Maximum number of pages to crawl
+- `url`: The URL to crawl (keyword arguments follow it)
+- `api_key`: Your WebCrawlerAPI API key. Defaults to `WEBCRAWLERAPI_API_KEY`
+- `base_url`: API base URL. Defaults to `WEBCRAWLERAPI_BASE_URL`
+- `output_format`: Content format of each Document: `markdown` (default), `cleaned`, or `html`
+- `scrape_type`: Deprecated, use `output_format` instead
+- `items_limit`: Maximum number of pages to crawl (default `10`)
 - `whitelist_regexp`: Regex pattern for URL whitelist
 - `blacklist_regexp`: Regex pattern for URL blacklist
+- `main_content_only`: Extract only the main content of each page (default `False`)
+- `max_depth`: Maximum crawl depth. `0` is the seed URL only, `1` is the seed URL plus direct links
+- `max_age`: Max age in seconds for cached content. `0` always fetches fresh content
+- `respect_robots_txt`: Whether to respect robots.txt (default `False`)
+- `keep_query_params`: Keep URL query params when deduplicating links
+- `max_polls`: Maximum number of job status checks before giving up (default `100`)
+
+Document metadata keys: `url`, `title`, `status_code`, `created_at`, `referred_url`, `depth`, `cost`, `job_id`, `item_id`.
 
 ### Links
 - [WebCrawlerAPI Python SDK](https://github.com/WebCrawlerAPI/webcrawlerapi-python-sdk)

webcrawlerapi_langchain-0.2.0/webcrawlerapi_langchain.egg-info/requires.txt

@@ -0,0 +1,2 @@
+webcrawlerapi>=2.1.5
+langchain-core>=0.3.0

webcrawlerapi_langchain-0.1.0/webcrawlerapi_langchain/__init__.py

@@ -1,4 +0,0 @@
-from .loader import WebCrawlerAPILoader
-
-__version__ = "0.1.0"
-__all__ = ["WebCrawlerAPILoader"] 

webcrawlerapi_langchain-0.1.0/webcrawlerapi_langchain.egg-info/requires.txt

@@ -1,2 +0,0 @@
-webcrawlerapi==1.0.6
-langchain-core>=0.1.0