webcrawlerapi 1.0.1__tar.gz

webcrawlerapi-1.0.1/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.1
+Name: webcrawlerapi
+Version: 1.0.1
+Summary: Python SDK for WebCrawler API
+Home-page: https://github.com/yourusername/webcrawlerapi-python
+Author: Andrew
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.25.0
+
+# WebCrawler API Python SDK
+
+A Python SDK for interacting with the WebCrawler API.
+
+## Installation
+
+```bash
+pip install webcrawlerapi
+```
+
+## Usage
+
+```python
+from webcrawlerapi import WebCrawlerAPI
+
+# Initialize the client
+crawler = WebCrawlerAPI(api_key="your_api_key")
+
+# Synchronous crawling (blocks until completion)
+job = crawler.crawl(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False,
+    max_polls=100  # Optional: maximum number of status checks
+)
+print(f"Job completed with status: {job.status}")
+
+# Or use asynchronous crawling
+response = crawler.crawl_async(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False
+)
+
+# Get the job ID from the response
+job_id = response.id
+print(f"Crawling job started with ID: {job_id}")
+
+# Check job status and get results
+job = crawler.get_job(job_id)
+print(f"Job status: {job.status}")
+
+# Access job details
+print(f"Crawled URL: {job.url}")
+print(f"Created at: {job.created_at}")
+print(f"Number of items: {len(job.job_items)}")
+
+# Access individual crawled items
+for item in job.job_items:
+    print(f"Page title: {item.title}")
+    print(f"Original URL: {item.original_url}")
+    print(f"Content URL: {item.markdown_content_url}")
+
+# Cancel a running job if needed
+cancel_response = crawler.cancel_job(job_id)
+print(f"Cancellation response: {cancel_response['message']}")
+```
+
+## API Methods
+
+### crawl()
+Starts a new crawling job and waits for its completion. This method will continuously poll the job status until:
+- The job reaches a terminal state (done, error, or cancelled)
+- The maximum number of polls is reached (default: 100)
+- The polling interval is determined by the server's `recommended_pull_delay_ms` or defaults to 5 seconds
+
+### crawl_async()
+Starts a new crawling job and returns immediately with a job ID. Use this when you want to handle polling and status checks yourself, or when using webhooks.
+
+### get_job()
+Retrieves the current status and details of a specific job.
+
+### cancel_job()
+Cancels a running job. Any items that are not in progress or already completed will be marked as canceled and will not be charged.
+
+## Parameters
+
+### Crawl Methods (crawl and crawl_async)
+- `url` (required): The seed URL where the crawler starts. Can be any valid URL.
+- `scrape_type` (default: "html"): The type of scraping you want to perform. Can be "html", "cleaned", or "markdown".
+- `items_limit` (default: 10): Crawler will stop when it reaches this limit of pages for this job.
+- `webhook_url` (optional): The URL where the server will send a POST request once the task is completed.
+- `allow_subdomains` (default: False): If True, the crawler will also crawl subdomains.
+- `whitelist_regexp` (optional): A regular expression to whitelist URLs. Only URLs that match the pattern will be crawled.
+- `blacklist_regexp` (optional): A regular expression to blacklist URLs. URLs that match the pattern will be skipped.
+- `max_polls` (optional, crawl only): Maximum number of status checks before returning (default: 100)
+
+### Responses
+
+#### CrawlAsync Response
+The `crawl_async()` method returns a `CrawlResponse` object with:
+- `id`: The unique identifier of the created job
+
+#### Job Response
+The Job object contains detailed information about the crawling job:
+
+- `id`: The unique identifier of the job
+- `org_id`: Your organization identifier
+- `url`: The seed URL where the crawler started
+- `status`: The status of the job (new, in_progress, done, error)
+- `scrape_type`: The type of scraping performed
+- `created_at`: The date when the job was created
+- `finished_at`: The date when the job was finished (if completed)
+- `webhook_url`: The webhook URL for notifications
+- `webhook_status`: The status of the webhook request
+- `webhook_error`: Any error message if the webhook request failed
+- `job_items`: List of JobItem objects representing crawled pages
+- `recommended_pull_delay_ms`: Server-recommended delay between status checks
+
+### JobItem Properties
+
+Each JobItem object represents a crawled page and contains:
+
+- `id`: The unique identifier of the item
+- `job_id`: The parent job identifier
+- `original_url`: The URL of the page
+- `page_status_code`: The HTTP status code of the page request
+- `status`: The status of the item (new, in_progress, done, error)
+- `title`: The page title
+- `created_at`: The date when the item was created
+- `cost`: The cost of the item in $
+- `referred_url`: The URL where the page was referred from
+- `last_error`: Any error message if the item failed
+- `raw_content_url`: URL to the raw content (if available)
+- `cleaned_content_url`: URL to the cleaned content (if scrape_type is "cleaned")
+- `markdown_content_url`: URL to the markdown content (if scrape_type is "markdown")
+
+## Requirements
+
+- Python 3.6+
+- requests>=2.25.0
+
+## License
+
+MIT License 

webcrawlerapi-1.0.1/README.md

@@ -0,0 +1,139 @@
+# WebCrawler API Python SDK
+
+A Python SDK for interacting with the WebCrawler API.
+
+## Installation
+
+```bash
+pip install webcrawlerapi
+```
+
+## Usage
+
+```python
+from webcrawlerapi import WebCrawlerAPI
+
+# Initialize the client
+crawler = WebCrawlerAPI(api_key="your_api_key")
+
+# Synchronous crawling (blocks until completion)
+job = crawler.crawl(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False,
+    max_polls=100  # Optional: maximum number of status checks
+)
+print(f"Job completed with status: {job.status}")
+
+# Or use asynchronous crawling
+response = crawler.crawl_async(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False
+)
+
+# Get the job ID from the response
+job_id = response.id
+print(f"Crawling job started with ID: {job_id}")
+
+# Check job status and get results
+job = crawler.get_job(job_id)
+print(f"Job status: {job.status}")
+
+# Access job details
+print(f"Crawled URL: {job.url}")
+print(f"Created at: {job.created_at}")
+print(f"Number of items: {len(job.job_items)}")
+
+# Access individual crawled items
+for item in job.job_items:
+    print(f"Page title: {item.title}")
+    print(f"Original URL: {item.original_url}")
+    print(f"Content URL: {item.markdown_content_url}")
+
+# Cancel a running job if needed
+cancel_response = crawler.cancel_job(job_id)
+print(f"Cancellation response: {cancel_response['message']}")
+```
+
+## API Methods
+
+### crawl()
+Starts a new crawling job and waits for its completion. This method will continuously poll the job status until:
+- The job reaches a terminal state (done, error, or cancelled)
+- The maximum number of polls is reached (default: 100)
+- The polling interval is determined by the server's `recommended_pull_delay_ms` or defaults to 5 seconds
+
+### crawl_async()
+Starts a new crawling job and returns immediately with a job ID. Use this when you want to handle polling and status checks yourself, or when using webhooks.
+
+### get_job()
+Retrieves the current status and details of a specific job.
+
+### cancel_job()
+Cancels a running job. Any items that are not in progress or already completed will be marked as canceled and will not be charged.
+
+## Parameters
+
+### Crawl Methods (crawl and crawl_async)
+- `url` (required): The seed URL where the crawler starts. Can be any valid URL.
+- `scrape_type` (default: "html"): The type of scraping you want to perform. Can be "html", "cleaned", or "markdown".
+- `items_limit` (default: 10): Crawler will stop when it reaches this limit of pages for this job.
+- `webhook_url` (optional): The URL where the server will send a POST request once the task is completed.
+- `allow_subdomains` (default: False): If True, the crawler will also crawl subdomains.
+- `whitelist_regexp` (optional): A regular expression to whitelist URLs. Only URLs that match the pattern will be crawled.
+- `blacklist_regexp` (optional): A regular expression to blacklist URLs. URLs that match the pattern will be skipped.
+- `max_polls` (optional, crawl only): Maximum number of status checks before returning (default: 100)
+
+### Responses
+
+#### CrawlAsync Response
+The `crawl_async()` method returns a `CrawlResponse` object with:
+- `id`: The unique identifier of the created job
+
+#### Job Response
+The Job object contains detailed information about the crawling job:
+
+- `id`: The unique identifier of the job
+- `org_id`: Your organization identifier
+- `url`: The seed URL where the crawler started
+- `status`: The status of the job (new, in_progress, done, error)
+- `scrape_type`: The type of scraping performed
+- `created_at`: The date when the job was created
+- `finished_at`: The date when the job was finished (if completed)
+- `webhook_url`: The webhook URL for notifications
+- `webhook_status`: The status of the webhook request
+- `webhook_error`: Any error message if the webhook request failed
+- `job_items`: List of JobItem objects representing crawled pages
+- `recommended_pull_delay_ms`: Server-recommended delay between status checks
+
+### JobItem Properties
+
+Each JobItem object represents a crawled page and contains:
+
+- `id`: The unique identifier of the item
+- `job_id`: The parent job identifier
+- `original_url`: The URL of the page
+- `page_status_code`: The HTTP status code of the page request
+- `status`: The status of the item (new, in_progress, done, error)
+- `title`: The page title
+- `created_at`: The date when the item was created
+- `cost`: The cost of the item in $
+- `referred_url`: The URL where the page was referred from
+- `last_error`: Any error message if the item failed
+- `raw_content_url`: URL to the raw content (if available)
+- `cleaned_content_url`: URL to the cleaned content (if scrape_type is "cleaned")
+- `markdown_content_url`: URL to the markdown content (if scrape_type is "markdown")
+
+## Requirements
+
+- Python 3.6+
+- requests>=2.25.0
+
+## License
+
+MIT License 

webcrawlerapi-1.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

webcrawlerapi-1.0.1/setup.py

@@ -0,0 +1,21 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="webcrawlerapi",
+    version="1.0.1",
+    packages=find_packages(),
+    install_requires=[
+        "requests>=2.25.0",
+    ],
+    author="Andrew",
+    description="Python SDK for WebCrawler API",
+    long_description=open("README.md").read(),
+    long_description_content_type="text/markdown",
+    url="https://github.com/yourusername/webcrawlerapi-python",
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+    python_requires=">=3.6",
+) 

webcrawlerapi-1.0.1/webcrawlerapi/__init__.py

@@ -0,0 +1,19 @@
+"""
+WebCrawler API Python SDK
+~~~~~~~~~~~~~~~~~~~~~
+
+A Python SDK for interacting with the WebCrawler API.
+
+Basic usage:
+
+    >>> from webcrawlerapi import WebCrawlerAPI
+    >>> crawler = WebCrawlerAPI(api_key="your_api_key")
+    >>> response = crawler.crawl(url="https://example.com")
+    >>> job_id = response["job_id"]
+    >>> job = crawler.get_job(job_id)
+"""
+
+from .client import WebCrawlerAPI, Job, JobItem
+
+__version__ = "1.0.0"
+__all__ = ["WebCrawlerAPI", "Job", "JobItem"] 

webcrawlerapi-1.0.1/webcrawlerapi/client.py

@@ -0,0 +1,251 @@
+import requests
+from typing import Optional, Dict, Any, List
+from urllib.parse import urljoin
+from datetime import datetime
+import time
+from dataclasses import dataclass
+
+
+@dataclass
+class CrawlResponse:
+    """Response from an asynchronous crawl request."""
+    id: str
+
+
+class JobItem:
+    """Represents a single crawled page item in a job."""
+    
+    def __init__(self, data: Dict[str, Any]):
+        self.id: str = data["id"]
+        self.job_id: str = data["job_id"]
+        self.original_url: str = data["original_url"]
+        self.page_status_code: int = data["page_status_code"]
+        self.status: str = data["status"]
+        self.title: str = data["title"]
+        self.created_at: datetime = datetime.fromisoformat(data["created_at"].replace('Z', '+00:00'))
+        self.updated_at: datetime = datetime.fromisoformat(data["updated_at"].replace('Z', '+00:00'))
+        self.cost: int = data["cost"]
+        self.referred_url: str = data["referred_url"]
+        self.last_error: str = data["last_error"]
+        
+        # Optional content URLs based on scrape_type
+        self.raw_content_url: Optional[str] = data.get("raw_content_url")
+        self.cleaned_content_url: Optional[str] = data.get("cleaned_content_url")
+        self.markdown_content_url: Optional[str] = data.get("markdown_content_url")
+
+
+class Job:
+    """Represents a crawling job."""
+    
+    TERMINAL_STATUSES = {"done", "error", "cancelled"}
+    
+    def __init__(self, data: Dict[str, Any]):
+        self.id: str = data["id"]
+        self.org_id: str = data["org_id"]
+        self.url: str = data["url"]
+        self.status: str = data["status"]
+        self.scrape_type: str = data["scrape_type"]
+        self.whitelist_regexp: str = data["whitelist_regexp"]
+        self.blacklist_regexp: str = data["blacklist_regexp"]
+        self.allow_subdomains: bool = data["allow_subdomains"]
+        self.items_limit: int = data["items_limit"]
+        self.created_at: datetime = datetime.fromisoformat(data["created_at"].replace('Z', '+00:00'))
+        self.updated_at: datetime = datetime.fromisoformat(data["updated_at"].replace('Z', '+00:00'))
+        self.webhook_url: str = data["webhook_url"]
+        self.recommended_pull_delay_ms: int = data.get("recommended_pull_delay_ms", 0)
+        
+        # Optional fields
+        self.finished_at: Optional[datetime] = None
+        if data.get("finished_at"):
+            self.finished_at = datetime.fromisoformat(data["finished_at"].replace('Z', '+00:00'))
+        
+        self.webhook_status: Optional[str] = data.get("webhook_status")
+        self.webhook_error: Optional[str] = data.get("webhook_error")
+        
+        # Parse job items
+        self.job_items: List[JobItem] = [JobItem(item) for item in data.get("job_items", [])]
+
+    @property
+    def is_terminal(self) -> bool:
+        """Check if the job is in a terminal state (done, error, or cancelled)."""
+        return self.status in self.TERMINAL_STATUSES
+
+
+class WebCrawlerAPI:
+    """Python SDK for WebCrawler API."""
+    
+    DEFAULT_POLL_DELAY_SECONDS = 5
+    
+    def __init__(self, api_key: str, base_url: str = "https://api.webcrawlerapi.com", version: str = "v1"):
+        """
+        Initialize the WebCrawler API client.
+        
+        Args:
+            api_key (str): Your API key for authentication
+            base_url (str): The base URL of the API (optional)
+            version (str): API version to use (optional, defaults to 'v1')
+        """
+        self.api_key = api_key
+        self.base_url = base_url.rstrip('/')
+        self.version = version
+        self.session = requests.Session()
+        self.session.headers.update({
+            'Authorization': f'Bearer {api_key}',
+            'Content-Type': 'application/json'
+        })
+
+    def crawl_async(
+        self,
+        url: str,
+        scrape_type: str = "html",
+        items_limit: int = 10,
+        webhook_url: Optional[str] = None,
+        allow_subdomains: bool = False,
+        whitelist_regexp: Optional[str] = None,
+        blacklist_regexp: Optional[str] = None
+    ) -> CrawlResponse:
+        """
+        Start a new crawling job asynchronously.
+        
+        Args:
+            url (str): The seed URL where the crawler starts
+            scrape_type (str): Type of scraping (html, cleaned, markdown)
+            items_limit (int): Maximum number of pages to crawl
+            webhook_url (str, optional): URL for webhook notifications
+            allow_subdomains (bool): Whether to crawl subdomains
+            whitelist_regexp (str, optional): Regex pattern for URL whitelist
+            blacklist_regexp (str, optional): Regex pattern for URL blacklist
+        
+        Returns:
+            CrawlResponse: Response containing the job ID
+        
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        payload = {
+            "url": url,
+            "scrape_type": scrape_type,
+            "items_limit": items_limit,
+            "allow_subdomains": allow_subdomains
+        }
+
+        if webhook_url:
+            payload["webhook_url"] = webhook_url
+        if whitelist_regexp:
+            payload["whitelist_regexp"] = whitelist_regexp
+        if blacklist_regexp:
+            payload["blacklist_regexp"] = blacklist_regexp
+
+        response = self.session.post(
+            urljoin(self.base_url, f"/{self.version}/crawl"),
+            json=payload
+        )
+        response.raise_for_status()
+        return CrawlResponse(id=response.json()["job_id"])
+
+    def get_job(self, job_id: str) -> Job:
+        """
+        Get the status and details of a specific job.
+        
+        Args:
+            job_id (str): The unique identifier of the job
+            
+        Returns:
+            Job: A Job object containing all job details and items
+            
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        response = self.session.get(
+            urljoin(self.base_url, f"/{self.version}/job/{job_id}")
+        )
+        response.raise_for_status()
+        return Job(response.json())
+
+    def cancel_job(self, job_id: str) -> Dict[str, str]:
+        """
+        Cancel a running job. All items that are not in progress and not done
+        will be marked as canceled and will not be charged.
+        
+        Args:
+            job_id (str): The unique identifier of the job to cancel
+            
+        Returns:
+            dict: Response containing confirmation message
+            
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        response = self.session.put(
+            urljoin(self.base_url, f"/{self.version}/job/{job_id}/cancel")
+        )
+        response.raise_for_status()
+        return response.json()
+
+    def crawl(
+        self,
+        url: str,
+        scrape_type: str = "html",
+        items_limit: int = 10,
+        webhook_url: Optional[str] = None,
+        allow_subdomains: bool = False,
+        whitelist_regexp: Optional[str] = None,
+        blacklist_regexp: Optional[str] = None,
+        max_polls: int = 100
+    ) -> Job:
+        """
+        Start a new crawling job and wait for its completion.
+        
+        This method will start a crawling job and continuously poll its status
+        until it reaches a terminal state (done, error, or cancelled) or until
+        the maximum number of polls is reached.
+        
+        Args:
+            url (str): The seed URL where the crawler starts
+            scrape_type (str): Type of scraping (html, cleaned, markdown)
+            items_limit (int): Maximum number of pages to crawl
+            webhook_url (str, optional): URL for webhook notifications
+            allow_subdomains (bool): Whether to crawl subdomains
+            whitelist_regexp (str, optional): Regex pattern for URL whitelist
+            blacklist_regexp (str, optional): Regex pattern for URL blacklist
+            max_polls (int): Maximum number of status checks before returning (default: 100)
+        
+        Returns:
+            Job: The final job state after completion or max polls
+            
+        Raises:
+            requests.exceptions.RequestException: If any API request fails
+        """
+        # Start the crawling job
+        response = self.crawl_async(
+            url=url,
+            scrape_type=scrape_type,
+            items_limit=items_limit,
+            webhook_url=webhook_url,
+            allow_subdomains=allow_subdomains,
+            whitelist_regexp=whitelist_regexp,
+            blacklist_regexp=blacklist_regexp
+        )
+        
+        job_id = response.id
+        polls = 0
+        
+        while polls < max_polls:
+            job = self.get_job(job_id)
+            
+            # Return immediately if job is in a terminal state
+            if job.is_terminal:
+                return job
+            
+            # Calculate delay for next poll
+            delay_seconds = (
+                job.recommended_pull_delay_ms / 1000
+                if job.recommended_pull_delay_ms
+                else self.DEFAULT_POLL_DELAY_SECONDS
+            )
+            
+            time.sleep(delay_seconds)
+            polls += 1
+        
+        # Return the last known state if max_polls is reached
+        return job 

webcrawlerapi-1.0.1/webcrawlerapi.egg-info/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.1
+Name: webcrawlerapi
+Version: 1.0.1
+Summary: Python SDK for WebCrawler API
+Home-page: https://github.com/yourusername/webcrawlerapi-python
+Author: Andrew
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.25.0
+
+# WebCrawler API Python SDK
+
+A Python SDK for interacting with the WebCrawler API.
+
+## Installation
+
+```bash
+pip install webcrawlerapi
+```
+
+## Usage
+
+```python
+from webcrawlerapi import WebCrawlerAPI
+
+# Initialize the client
+crawler = WebCrawlerAPI(api_key="your_api_key")
+
+# Synchronous crawling (blocks until completion)
+job = crawler.crawl(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False,
+    max_polls=100  # Optional: maximum number of status checks
+)
+print(f"Job completed with status: {job.status}")
+
+# Or use asynchronous crawling
+response = crawler.crawl_async(
+    url="https://example.com",
+    scrape_type="markdown",
+    items_limit=10,
+    webhook_url="https://yourserver.com/webhook",
+    allow_subdomains=False
+)
+
+# Get the job ID from the response
+job_id = response.id
+print(f"Crawling job started with ID: {job_id}")
+
+# Check job status and get results
+job = crawler.get_job(job_id)
+print(f"Job status: {job.status}")
+
+# Access job details
+print(f"Crawled URL: {job.url}")
+print(f"Created at: {job.created_at}")
+print(f"Number of items: {len(job.job_items)}")
+
+# Access individual crawled items
+for item in job.job_items:
+    print(f"Page title: {item.title}")
+    print(f"Original URL: {item.original_url}")
+    print(f"Content URL: {item.markdown_content_url}")
+
+# Cancel a running job if needed
+cancel_response = crawler.cancel_job(job_id)
+print(f"Cancellation response: {cancel_response['message']}")
+```
+
+## API Methods
+
+### crawl()
+Starts a new crawling job and waits for its completion. This method will continuously poll the job status until:
+- The job reaches a terminal state (done, error, or cancelled)
+- The maximum number of polls is reached (default: 100)
+- The polling interval is determined by the server's `recommended_pull_delay_ms` or defaults to 5 seconds
+
+### crawl_async()
+Starts a new crawling job and returns immediately with a job ID. Use this when you want to handle polling and status checks yourself, or when using webhooks.
+
+### get_job()
+Retrieves the current status and details of a specific job.
+
+### cancel_job()
+Cancels a running job. Any items that are not in progress or already completed will be marked as canceled and will not be charged.
+
+## Parameters
+
+### Crawl Methods (crawl and crawl_async)
+- `url` (required): The seed URL where the crawler starts. Can be any valid URL.
+- `scrape_type` (default: "html"): The type of scraping you want to perform. Can be "html", "cleaned", or "markdown".
+- `items_limit` (default: 10): Crawler will stop when it reaches this limit of pages for this job.
+- `webhook_url` (optional): The URL where the server will send a POST request once the task is completed.
+- `allow_subdomains` (default: False): If True, the crawler will also crawl subdomains.
+- `whitelist_regexp` (optional): A regular expression to whitelist URLs. Only URLs that match the pattern will be crawled.
+- `blacklist_regexp` (optional): A regular expression to blacklist URLs. URLs that match the pattern will be skipped.
+- `max_polls` (optional, crawl only): Maximum number of status checks before returning (default: 100)
+
+### Responses
+
+#### CrawlAsync Response
+The `crawl_async()` method returns a `CrawlResponse` object with:
+- `id`: The unique identifier of the created job
+
+#### Job Response
+The Job object contains detailed information about the crawling job:
+
+- `id`: The unique identifier of the job
+- `org_id`: Your organization identifier
+- `url`: The seed URL where the crawler started
+- `status`: The status of the job (new, in_progress, done, error)
+- `scrape_type`: The type of scraping performed
+- `created_at`: The date when the job was created
+- `finished_at`: The date when the job was finished (if completed)
+- `webhook_url`: The webhook URL for notifications
+- `webhook_status`: The status of the webhook request
+- `webhook_error`: Any error message if the webhook request failed
+- `job_items`: List of JobItem objects representing crawled pages
+- `recommended_pull_delay_ms`: Server-recommended delay between status checks
+
+### JobItem Properties
+
+Each JobItem object represents a crawled page and contains:
+
+- `id`: The unique identifier of the item
+- `job_id`: The parent job identifier
+- `original_url`: The URL of the page
+- `page_status_code`: The HTTP status code of the page request
+- `status`: The status of the item (new, in_progress, done, error)
+- `title`: The page title
+- `created_at`: The date when the item was created
+- `cost`: The cost of the item in $
+- `referred_url`: The URL where the page was referred from
+- `last_error`: Any error message if the item failed
+- `raw_content_url`: URL to the raw content (if available)
+- `cleaned_content_url`: URL to the cleaned content (if scrape_type is "cleaned")
+- `markdown_content_url`: URL to the markdown content (if scrape_type is "markdown")
+
+## Requirements
+
+- Python 3.6+
+- requests>=2.25.0
+
+## License
+
+MIT License 

webcrawlerapi-1.0.1/webcrawlerapi.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+README.md
+setup.py
+webcrawlerapi/__init__.py
+webcrawlerapi/client.py
+webcrawlerapi.egg-info/PKG-INFO
+webcrawlerapi.egg-info/SOURCES.txt
+webcrawlerapi.egg-info/dependency_links.txt
+webcrawlerapi.egg-info/requires.txt
+webcrawlerapi.egg-info/top_level.txt

webcrawlerapi-1.0.1/webcrawlerapi.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

webcrawlerapi-1.0.1/webcrawlerapi.egg-info/requires.txt

@@ -0,0 +1 @@
+requests>=2.25.0

webcrawlerapi-1.0.1/webcrawlerapi.egg-info/top_level.txt

@@ -0,0 +1 @@
+webcrawlerapi