scrapeunblocker-haystack 0.1.0__tar.gz → 0.2.1__tar.gz

scrapeunblocker_haystack-0.2.1/CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+## 0.2.1
+
+- A URL whose page does not exist (the site answers HTTP 404 or 410) no longer
+  shows up as a generic request failure, and an older API response that carried
+  the not-found page as a 200 no longer becomes a Document. The fetcher skips it
+  with a clear warning, or raises the new `TargetNotFoundError` (`url`,
+  `origin_status`, `body`) when `raise_on_failure=True`. The call is billed by
+  ScrapeUnblocker and retrying returns the same answer.
+
+## 0.2.0
+
+- `ScrapeUnblockerFetcher` now supports **browser steps** via the `steps`
+  parameter - a list of browser actions (`wait_for`, `wait_for_text`, `wait`,
+  `click`, `type`, `select`, `press_key`, `scroll`) run in a real browser after
+  the page loads, before the HTML is captured. A failed step raises
+  `StepExecutionError` (skipped or raised per `raise_on_failure`).
+- `ScrapeUnblockerFetcher` now supports **listing elements** via the
+  `list_elements` parameter - returns the parsed elements JSON
+  (`{url, count, elements: [...]}`) instead of HTML.
+- New `StepExecutionError` exception exposing `step_index`, `action`, `selector`
+  and `reason`.
+
+## 0.1.0
+
+Initial release.
+
+- `ScrapeUnblockerFetcher` - fetch pages behind anti-bot protections as Documents
+- `ScrapeUnblockerWebSearch` - Google organic results as Documents

{scrapeunblocker_haystack-0.1.0 → scrapeunblocker_haystack-0.2.1}/PKG-INFO

@@ -1,9 +1,9 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: scrapeunblocker-haystack
-Version: 0.1.0
+Version: 0.2.1
 Summary: Haystack integration for ScrapeUnblocker - scrape pages behind anti-bot protections
 Project-URL: Homepage, https://www.scrapeunblocker.com
-Project-URL: Documentation, https://developers.scrapeunblocker.com
+Project-URL: Documentation, https://docs.scrapeunblocker.com
 Project-URL: Source, https://github.com/ScrapeUnblocker/scrapeunblocker-haystack
 Project-URL: Issues, https://github.com/ScrapeUnblocker/scrapeunblocker-haystack/issues
 Author-email: ScrapeUnblocker <info@scrapeunblocker.com>
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 # scrapeunblocker-haystack
 
 [Haystack](https://haystack.deepset.ai/) integration for
-[ScrapeUnblocker](https://www.scrapeunblocker.com) - fetch pages that block
+[ScrapeUnblocker](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration) - fetch pages that block
 ordinary HTTP requests.
 
 ScrapeUnblocker renders web pages in a real browser behind anti-bot protections
@@ -41,7 +41,7 @@ pip install scrapeunblocker-haystack
 
 ## Setup
 
-Get an API key at [scrapeunblocker.com](https://www.scrapeunblocker.com) and
+Get an API key at [scrapeunblocker.com](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration) and
 export it:
 
 ```bash
@@ -71,6 +71,8 @@ print(result["documents"][0].content[:200])
 | `parsed_data` | `False` | Return AI-parsed structured JSON instead of raw HTML |
 | `proxy_country` | `None` | Two-letter country code for the exit IP |
 | `time_sleep` | `None` | Seconds to wait after load before capturing |
+| `steps` | `None` | Browser actions to run before the HTML is captured (see below) |
+| `list_elements` | `False` | Return the parsed elements JSON instead of HTML (see below) |
 | `base_url` | `https://api.scrapeunblocker.com` | API base URL |
 | `timeout` | `180` | HTTP timeout in seconds |
 | `raise_on_failure` | `False` | Raise instead of skipping a URL that fails |
@@ -78,6 +80,69 @@ print(result["documents"][0].content[:200])
 By default a URL that cannot be fetched is logged and skipped, so one bad URL
 does not discard the rest of the batch.
 
+#### Browser steps
+
+Pass `steps` to drive the page in a real browser before its HTML is captured -
+wait for content to appear, click, type, scroll, and so on. Each step is a dict
+with an `action` key plus its parameters:
+
+```python
+from scrapeunblocker_haystack import ScrapeUnblockerFetcher
+
+fetcher = ScrapeUnblockerFetcher(
+    steps=[
+        {"action": "wait_for", "selector": "#results"},
+        {"action": "type", "selector": "input#q", "value": "laptops"},
+        {"action": "press_key", "value": "Enter"},
+        {"action": "wait_for_text", "value": "in stock"},
+        {"action": "click", "selector": "button.load-more"},
+        {"action": "scroll", "value": "bottom"},
+    ]
+)
+result = fetcher.run(urls=["https://example.com/search"])
+```
+
+Available actions:
+
+| Action | Parameters |
+| --- | --- |
+| `wait_for` | `selector`, `selector_type?` (`css`\|`xPath`\|`className`\|`tagName`), `timeout_ms?` |
+| `wait_for_text` | `value`, `timeout_ms?` |
+| `wait` | `value` (milliseconds) |
+| `click` | `selector`, `selector_type?`, `timeout_ms?` |
+| `type` | `selector`, `selector_type?`, `value`, `clear?`, `timeout_ms?` (human-like typing) |
+| `select` | `selector`, `selector_type?`, `value`, `timeout_ms?` |
+| `press_key` | `value` (`Enter`\|`Tab`\|`Escape`\|`Backspace`\|`Delete`\|`Space`\|`Arrow*`\|`Home`\|`End`\|`PageUp`\|`PageDown`) |
+| `scroll` | `value` (`"bottom"` or an integer pixel offset) |
+
+Steps are **non-idempotent**. If a step fails, the fetch raises
+`StepExecutionError` (with `step_index`, `action`, `selector` and `reason`),
+which is logged and skipped by default, or re-raised when
+`raise_on_failure=True`.
+
+#### Pages that do not exist
+
+If the site answers HTTP 404 or 410, there is no page to index: the URL is
+skipped with a warning, or `TargetNotFoundError` (with `url`, `origin_status`
+and the site's own `body`) is raised when `raise_on_failure=True`. That is the
+site's answer, not a block - the call is billed and a retry returns the same
+result.
+
+#### List elements
+
+Set `list_elements=True` to return the parsed elements JSON
+(`{url, count, elements: [...]}`) instead of the page HTML. The JSON is stored
+as the Document content, and the element count is available in `meta["count"]`:
+
+```python
+fetcher = ScrapeUnblockerFetcher(list_elements=True)
+result = fetcher.run(urls=["https://example.com"])
+
+doc = result["documents"][0]
+print(doc.meta["count"])          # number of elements
+print(doc.content)                # {"url": ..., "count": ..., "elements": [...]}
+```
+
 ### ScrapeUnblockerWebSearch
 
 Searches Google and returns the organic results as `Document` objects, with the
@@ -148,8 +213,8 @@ reference, not as its value.
 
 ## Links
 
-- ScrapeUnblocker: https://www.scrapeunblocker.com
-- API documentation: https://developers.scrapeunblocker.com
+- ScrapeUnblocker: [www.scrapeunblocker.com](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration)
+- API documentation: [docs.scrapeunblocker.com](https://docs.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration)
 - Haystack: https://haystack.deepset.ai
 
 ## License

{scrapeunblocker_haystack-0.1.0 → scrapeunblocker_haystack-0.2.1}/README.md

@@ -1,7 +1,7 @@
 # scrapeunblocker-haystack
 
 [Haystack](https://haystack.deepset.ai/) integration for
-[ScrapeUnblocker](https://www.scrapeunblocker.com) - fetch pages that block
+[ScrapeUnblocker](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration) - fetch pages that block
 ordinary HTTP requests.
 
 ScrapeUnblocker renders web pages in a real browser behind anti-bot protections
@@ -16,7 +16,7 @@ pip install scrapeunblocker-haystack
 
 ## Setup
 
-Get an API key at [scrapeunblocker.com](https://www.scrapeunblocker.com) and
+Get an API key at [scrapeunblocker.com](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration) and
 export it:
 
 ```bash
@@ -46,6 +46,8 @@ print(result["documents"][0].content[:200])
 | `parsed_data` | `False` | Return AI-parsed structured JSON instead of raw HTML |
 | `proxy_country` | `None` | Two-letter country code for the exit IP |
 | `time_sleep` | `None` | Seconds to wait after load before capturing |
+| `steps` | `None` | Browser actions to run before the HTML is captured (see below) |
+| `list_elements` | `False` | Return the parsed elements JSON instead of HTML (see below) |
 | `base_url` | `https://api.scrapeunblocker.com` | API base URL |
 | `timeout` | `180` | HTTP timeout in seconds |
 | `raise_on_failure` | `False` | Raise instead of skipping a URL that fails |
@@ -53,6 +55,69 @@ print(result["documents"][0].content[:200])
 By default a URL that cannot be fetched is logged and skipped, so one bad URL
 does not discard the rest of the batch.
 
+#### Browser steps
+
+Pass `steps` to drive the page in a real browser before its HTML is captured -
+wait for content to appear, click, type, scroll, and so on. Each step is a dict
+with an `action` key plus its parameters:
+
+```python
+from scrapeunblocker_haystack import ScrapeUnblockerFetcher
+
+fetcher = ScrapeUnblockerFetcher(
+    steps=[
+        {"action": "wait_for", "selector": "#results"},
+        {"action": "type", "selector": "input#q", "value": "laptops"},
+        {"action": "press_key", "value": "Enter"},
+        {"action": "wait_for_text", "value": "in stock"},
+        {"action": "click", "selector": "button.load-more"},
+        {"action": "scroll", "value": "bottom"},
+    ]
+)
+result = fetcher.run(urls=["https://example.com/search"])
+```
+
+Available actions:
+
+| Action | Parameters |
+| --- | --- |
+| `wait_for` | `selector`, `selector_type?` (`css`\|`xPath`\|`className`\|`tagName`), `timeout_ms?` |
+| `wait_for_text` | `value`, `timeout_ms?` |
+| `wait` | `value` (milliseconds) |
+| `click` | `selector`, `selector_type?`, `timeout_ms?` |
+| `type` | `selector`, `selector_type?`, `value`, `clear?`, `timeout_ms?` (human-like typing) |
+| `select` | `selector`, `selector_type?`, `value`, `timeout_ms?` |
+| `press_key` | `value` (`Enter`\|`Tab`\|`Escape`\|`Backspace`\|`Delete`\|`Space`\|`Arrow*`\|`Home`\|`End`\|`PageUp`\|`PageDown`) |
+| `scroll` | `value` (`"bottom"` or an integer pixel offset) |
+
+Steps are **non-idempotent**. If a step fails, the fetch raises
+`StepExecutionError` (with `step_index`, `action`, `selector` and `reason`),
+which is logged and skipped by default, or re-raised when
+`raise_on_failure=True`.
+
+#### Pages that do not exist
+
+If the site answers HTTP 404 or 410, there is no page to index: the URL is
+skipped with a warning, or `TargetNotFoundError` (with `url`, `origin_status`
+and the site's own `body`) is raised when `raise_on_failure=True`. That is the
+site's answer, not a block - the call is billed and a retry returns the same
+result.
+
+#### List elements
+
+Set `list_elements=True` to return the parsed elements JSON
+(`{url, count, elements: [...]}`) instead of the page HTML. The JSON is stored
+as the Document content, and the element count is available in `meta["count"]`:
+
+```python
+fetcher = ScrapeUnblockerFetcher(list_elements=True)
+result = fetcher.run(urls=["https://example.com"])
+
+doc = result["documents"][0]
+print(doc.meta["count"])          # number of elements
+print(doc.content)                # {"url": ..., "count": ..., "elements": [...]}
+```
+
 ### ScrapeUnblockerWebSearch
 
 Searches Google and returns the organic results as `Document` objects, with the
@@ -123,8 +188,8 @@ reference, not as its value.
 
 ## Links
 
-- ScrapeUnblocker: https://www.scrapeunblocker.com
-- API documentation: https://developers.scrapeunblocker.com
+- ScrapeUnblocker: [www.scrapeunblocker.com](https://www.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration)
+- API documentation: [docs.scrapeunblocker.com](https://docs.scrapeunblocker.com?utm_source=haystack&utm_medium=integration&utm_campaign=haystack-integration)
 - Haystack: https://haystack.deepset.ai
 
 ## License

{scrapeunblocker_haystack-0.1.0 → scrapeunblocker_haystack-0.2.1}/pyproject.toml

@@ -25,7 +25,7 @@ dependencies = ["haystack-ai>=2.0.0", "requests>=2.31.0"]
 
 [project.urls]
 Homepage = "https://www.scrapeunblocker.com"
-Documentation = "https://developers.scrapeunblocker.com"
+Documentation = "https://docs.scrapeunblocker.com"
 Source = "https://github.com/ScrapeUnblocker/scrapeunblocker-haystack"
 Issues = "https://github.com/ScrapeUnblocker/scrapeunblocker-haystack/issues"
 

scrapeunblocker_haystack-0.2.1/src/scrapeunblocker_haystack/__init__.py

@@ -0,0 +1,13 @@
+from scrapeunblocker_haystack.fetcher import (
+    ScrapeUnblockerFetcher,
+    StepExecutionError,
+    TargetNotFoundError,
+)
+from scrapeunblocker_haystack.search import ScrapeUnblockerWebSearch
+
+__all__ = [
+    "ScrapeUnblockerFetcher",
+    "ScrapeUnblockerWebSearch",
+    "StepExecutionError",
+    "TargetNotFoundError",
+]

scrapeunblocker_haystack-0.2.1/src/scrapeunblocker_haystack/fetcher.py

@@ -0,0 +1,275 @@
+"""Haystack component that fetches pages through the ScrapeUnblocker API."""
+
+from __future__ import annotations
+
+import json
+from typing import Any, Dict, List, Optional
+
+import requests
+from haystack import Document, component, default_from_dict, default_to_dict, logging
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
+DEFAULT_TIMEOUT = 180
+_TARGET_GONE = frozenset({404, 410})
+
+
+class TargetNotFoundError(Exception):
+    """Raised when the target page does not exist (the site answered 404/410).
+
+    This is the website's own answer, not a block or an API failure: the call
+    is billed, and retrying returns the same result. ``body`` is the target's
+    own not-found page (can be empty).
+    """
+
+    def __init__(self, url: str, origin_status: int, body: str = "") -> None:
+        self.url = url
+        self.origin_status = origin_status
+        self.body = body
+        super().__init__(
+            f"Target page does not exist: {url} answered HTTP {origin_status}. "
+            "This is the site's own answer, not a block or an API failure; the "
+            "call was billed and retrying returns the same result."
+        )
+
+
+def _target_gone_status(response: requests.Response) -> Optional[int]:
+    """The target's own 404/410 answer, or None.
+
+    ``/getPageSource`` always exists, so a 404/410 from it is the target site's
+    answer, sent with ``X-Origin-Status``. Older API versions returned that
+    same answer as a 200 carrying the header, so the header is checked first.
+    """
+    try:
+        origin = int(response.headers.get("X-Origin-Status", ""))
+    except (TypeError, ValueError):
+        origin = None
+    if origin in _TARGET_GONE:
+        return origin
+    return response.status_code if response.status_code in _TARGET_GONE else None
+
+
+class StepExecutionError(Exception):
+    """Raised when a browser step fails (getPageSource returns HTTP 422)."""
+
+    def __init__(
+        self,
+        reason: str,
+        step_index: Optional[int] = None,
+        action: Optional[str] = None,
+        selector: Optional[str] = None,
+    ) -> None:
+        self.reason = reason
+        self.step_index = step_index
+        self.action = action
+        self.selector = selector
+        detail = f"step {step_index} ({action})" if action is not None else "browser step"
+        if selector:
+            detail += f" on {selector!r}"
+        super().__init__(f"ScrapeUnblocker {detail} failed: {reason}")
+
+
+@component
+class ScrapeUnblockerFetcher:
+    """
+    Fetches web pages through the ScrapeUnblocker API and returns them as Documents.
+
+    ScrapeUnblocker renders pages in a real browser behind anti-bot protections
+    (Cloudflare, DataDome, PerimeterX, Akamai), so it reaches pages that a plain
+    HTTP fetch cannot.
+
+    ### Usage example
+
+    ```python
+    from scrapeunblocker_haystack import ScrapeUnblockerFetcher
+
+    fetcher = ScrapeUnblockerFetcher()
+    result = fetcher.run(urls=["https://example.com"])
+    print(result["documents"][0].content[:200])
+    ```
+
+    ### Browser steps
+
+    Drive the page in a real browser before the HTML is captured - click, type,
+    wait for a selector, scroll, and so on:
+
+    ```python
+    fetcher = ScrapeUnblockerFetcher(
+        steps=[
+            {"action": "wait_for", "selector": "#results"},
+            {"action": "click", "selector": "button.load-more"},
+            {"action": "scroll", "value": "bottom"},
+        ]
+    )
+    ```
+
+    ### List elements
+
+    Return the parsed elements JSON (`{url, count, elements: [...]}`) instead of
+    HTML by setting `list_elements=True`.
+    """
+
+    def __init__(
+        self,
+        api_key: Secret = Secret.from_env_var("SCRAPEUNBLOCKER_API_KEY"),
+        parsed_data: bool = False,
+        proxy_country: Optional[str] = None,
+        time_sleep: Optional[int] = None,
+        steps: Optional[List[Dict[str, Any]]] = None,
+        list_elements: bool = False,
+        base_url: str = DEFAULT_BASE_URL,
+        timeout: int = DEFAULT_TIMEOUT,
+        raise_on_failure: bool = False,
+    ) -> None:
+        """
+        :param api_key: ScrapeUnblocker API key. Read from `SCRAPEUNBLOCKER_API_KEY` by default.
+        :param parsed_data: Return AI-parsed structured JSON instead of raw HTML.
+        :param proxy_country: Two-letter country code for the exit IP, for geo-restricted content.
+        :param time_sleep: Seconds to wait after page load before capturing.
+        :param steps: Browser actions to run after the page loads, before the HTML is
+            captured. A list of dicts, each with an `action` key (for example
+            `wait_for`, `wait_for_text`, `wait`, `click`, `type`, `select`,
+            `press_key`, `scroll`) plus its parameters. These actions are
+            non-idempotent. Sent JSON-encoded in the `steps` query parameter.
+        :param list_elements: Return the parsed elements JSON
+            (`{url, count, elements: [...]}`) instead of the page HTML.
+        :param base_url: API base URL. Override to target a different environment.
+        :param timeout: HTTP timeout in seconds.
+        :param raise_on_failure: Raise instead of skipping when a URL cannot be fetched
+            (including when a browser step fails or the page does not exist).
+        """
+        self.api_key = api_key
+        self.parsed_data = parsed_data
+        self.proxy_country = proxy_country
+        self.time_sleep = time_sleep
+        self.steps = steps
+        self.list_elements = list_elements
+        self.base_url = base_url.rstrip("/")
+        self.timeout = timeout
+        self.raise_on_failure = raise_on_failure
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize this component to a dictionary."""
+        return default_to_dict(
+            self,
+            api_key=self.api_key.to_dict(),
+            parsed_data=self.parsed_data,
+            proxy_country=self.proxy_country,
+            time_sleep=self.time_sleep,
+            steps=self.steps,
+            list_elements=self.list_elements,
+            base_url=self.base_url,
+            timeout=self.timeout,
+            raise_on_failure=self.raise_on_failure,
+        )
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ScrapeUnblockerFetcher":
+        """Deserialize this component from a dictionary."""
+        deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
+        return default_from_dict(cls, data)
+
+    def _fetch(self, url: str) -> requests.Response:
+        params: Dict[str, Any] = {"url": url}
+        if self.parsed_data:
+            params["parsed_data"] = True
+        if self.proxy_country:
+            params["proxy_country"] = self.proxy_country
+        if self.time_sleep is not None:
+            params["time_sleep"] = self.time_sleep
+        if self.steps:
+            params["steps"] = json.dumps(self.steps)
+        if self.list_elements:
+            params["list_elements"] = True
+
+        response = requests.post(
+            f"{self.base_url}/getPageSource",
+            params=params,
+            headers={"X-ScrapeUnblocker-Key": self.api_key.resolve_value()},
+            timeout=self.timeout,
+        )
+
+        gone = _target_gone_status(response)
+        if gone is not None:
+            raise TargetNotFoundError(url, gone, response.text)
+
+        # A failed browser step returns 422 with a structured JSON error.
+        if response.status_code == 422:
+            error = self._parse_step_error(response)
+            if error is not None:
+                raise error
+
+        response.raise_for_status()
+        return response
+
+    @staticmethod
+    def _parse_step_error(response: requests.Response) -> Optional[StepExecutionError]:
+        try:
+            payload = response.json()
+        except ValueError:
+            return None
+        if not isinstance(payload, dict) or payload.get("error") != "step_failed":
+            return None
+        return StepExecutionError(
+            reason=payload.get("reason", "unknown"),
+            step_index=payload.get("step_index"),
+            action=payload.get("action"),
+            selector=payload.get("selector"),
+        )
+
+    @component.output_types(documents=List[Document])
+    def run(self, urls: List[str]) -> Dict[str, List[Document]]:
+        """
+        Fetch each URL and return one Document per successfully fetched page.
+
+        A URL whose page does not exist (the site answers 404/410) yields no
+        Document; with `raise_on_failure` it raises `TargetNotFoundError`.
+
+        :param urls: URLs to fetch.
+        :returns: A dictionary with a `documents` key holding the fetched pages.
+        """
+        documents: List[Document] = []
+        # `parsed_data` and `list_elements` both return a JSON body rather than HTML.
+        json_mode = self.parsed_data or self.list_elements
+
+        for url in urls:
+            try:
+                response = self._fetch(url)
+            except TargetNotFoundError as exc:
+                if self.raise_on_failure:
+                    raise
+                # A page that does not exist has no content to index.
+                logger.warning("ScrapeUnblocker skipped {url}: {error}", url=url, error=str(exc))
+                continue
+            except Exception as exc:
+                if self.raise_on_failure:
+                    raise
+                # One unreachable URL should not discard the rest of the batch.
+                logger.warning(
+                    "ScrapeUnblocker could not fetch {url}: {error}", url=url, error=str(exc)
+                )
+                continue
+
+            meta: Dict[str, Any] = {
+                "url": url,
+                "content_type": response.headers.get("Content-Type", ""),
+                "parsed_data": self.parsed_data,
+                "list_elements": self.list_elements,
+            }
+
+            if json_mode:
+                try:
+                    payload = response.json()
+                    content = json.dumps(payload, ensure_ascii=False)
+                    if self.list_elements and isinstance(payload, dict):
+                        meta["count"] = payload.get("count")
+                except ValueError:
+                    content = response.text
+            else:
+                content = response.text
+
+            documents.append(Document(content=content, meta=meta))
+
+        return {"documents": documents}

scrapeunblocker_haystack-0.2.1/src/scrapeunblocker_haystack/version.py

@@ -0,0 +1 @@
+__version__ = "0.2.1"

scrapeunblocker_haystack-0.2.1/tests/test_components.py

@@ -0,0 +1,299 @@
+"""Unit tests for the ScrapeUnblocker Haystack components."""
+
+import json
+from unittest.mock import MagicMock, patch
+
+import pytest
+from haystack import Document
+from haystack.utils import Secret
+
+from scrapeunblocker_haystack import (
+    ScrapeUnblockerFetcher,
+    ScrapeUnblockerWebSearch,
+    StepExecutionError,
+    TargetNotFoundError,
+)
+
+API_KEY = Secret.from_token("test_key")
+# Haystack refuses to serialize token secrets, so to_dict tests use an env-var secret.
+ENV_KEY = Secret.from_env_var("SCRAPEUNBLOCKER_API_KEY")
+
+
+def _response(
+    text: str = "<html><title>Test</title></html>", json_data=None, status_code: int = 200
+) -> MagicMock:
+    response = MagicMock()
+    response.text = text
+    response.status_code = status_code
+    response.headers = {"Content-Type": "text/html"}
+    response.raise_for_status.return_value = None
+    if json_data is not None:
+        response.json.return_value = json_data
+    return response
+
+
+class TestFetcher:
+    def test_defaults(self):
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+        assert fetcher.parsed_data is False
+        assert fetcher.base_url == "https://api.scrapeunblocker.com"
+
+    def test_base_url_trailing_slash_stripped(self):
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, base_url="https://example.com/")
+        assert fetcher.base_url == "https://example.com"
+
+    def test_to_dict_and_back(self):
+        fetcher = ScrapeUnblockerFetcher(api_key=ENV_KEY, parsed_data=True, proxy_country="de")
+        data = fetcher.to_dict()
+        assert data["init_parameters"]["parsed_data"] is True
+        assert data["init_parameters"]["proxy_country"] == "de"
+
+        restored = ScrapeUnblockerFetcher.from_dict(data)
+        assert restored.parsed_data is True
+        assert restored.proxy_country == "de"
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_run_single_url(self, mock_post):
+        mock_post.return_value = _response()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        result = fetcher.run(urls=["https://example.com"])
+
+        assert len(result["documents"]) == 1
+        doc = result["documents"][0]
+        assert isinstance(doc, Document)
+        assert "Test" in doc.content
+        assert doc.meta["url"] == "https://example.com"
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["headers"]["X-ScrapeUnblocker-Key"] == "test_key"
+        assert kwargs["params"] == {"url": "https://example.com"}
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_run_forwards_options(self, mock_post):
+        mock_post.return_value = _response(json_data={"ok": True})
+        fetcher = ScrapeUnblockerFetcher(
+            api_key=API_KEY, parsed_data=True, proxy_country="de", time_sleep=5
+        )
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["parsed_data"] is True
+        assert kwargs["params"]["proxy_country"] == "de"
+        assert kwargs["params"]["time_sleep"] == 5
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_parsed_data_serialised_as_json(self, mock_post):
+        mock_post.return_value = _response(json_data={"title": "Test", "price": "9.99"})
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, parsed_data=True)
+
+        result = fetcher.run(urls=["https://example.com"])
+
+        assert json.loads(result["documents"][0].content) == {"title": "Test", "price": "9.99"}
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_failure_is_skipped_by_default(self, mock_post):
+        mock_post.side_effect = [Exception("boom"), _response()]
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        result = fetcher.run(urls=["https://broken.com", "https://ok.com"])
+
+        assert len(result["documents"]) == 1
+        assert result["documents"][0].meta["url"] == "https://ok.com"
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_failure_raises_when_requested(self, mock_post):
+        mock_post.side_effect = Exception("boom")
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, raise_on_failure=True)
+
+        with pytest.raises(Exception, match="boom"):
+            fetcher.run(urls=["https://broken.com"])
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_steps_are_json_encoded(self, mock_post):
+        mock_post.return_value = _response()
+        steps = [
+            {"action": "wait_for", "selector": "#results"},
+            {"action": "click", "selector": "button.load-more"},
+        ]
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, steps=steps)
+
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["steps"] == json.dumps(steps)
+        assert json.loads(kwargs["params"]["steps"]) == steps
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_list_elements_returns_json(self, mock_post):
+        payload = {
+            "url": "https://example.com",
+            "count": 2,
+            "elements": [{"tag": "a"}, {"tag": "div"}],
+        }
+        mock_post.return_value = _response(json_data=payload)
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, list_elements=True)
+
+        result = fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["list_elements"] is True
+        doc = result["documents"][0]
+        assert json.loads(doc.content) == payload
+        assert doc.meta["count"] == 2
+        assert doc.meta["list_elements"] is True
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_step_failure_is_skipped_by_default(self, mock_post):
+        mock_post.return_value = _response(
+            json_data={
+                "error": "step_failed",
+                "step_index": 1,
+                "action": "click",
+                "reason": "selector not found",
+                "selector": "button.load-more",
+            },
+            status_code=422,
+        )
+        fetcher = ScrapeUnblockerFetcher(
+            api_key=API_KEY, steps=[{"action": "click", "selector": "button.load-more"}]
+        )
+
+        result = fetcher.run(urls=["https://example.com"])
+
+        assert result["documents"] == []
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_step_failure_raises_when_requested(self, mock_post):
+        mock_post.return_value = _response(
+            json_data={
+                "error": "step_failed",
+                "step_index": 1,
+                "action": "click",
+                "reason": "selector not found",
+                "selector": "button.load-more",
+            },
+            status_code=422,
+        )
+        fetcher = ScrapeUnblockerFetcher(
+            api_key=API_KEY,
+            steps=[{"action": "click", "selector": "button.load-more"}],
+            raise_on_failure=True,
+        )
+
+        with pytest.raises(StepExecutionError) as exc_info:
+            fetcher.run(urls=["https://example.com"])
+
+        error = exc_info.value
+        assert error.step_index == 1
+        assert error.action == "click"
+        assert error.selector == "button.load-more"
+        assert error.reason == "selector not found"
+
+    def test_to_dict_includes_steps_and_list_elements(self):
+        steps = [{"action": "wait_for", "selector": "#results"}]
+        fetcher = ScrapeUnblockerFetcher(
+            api_key=ENV_KEY, steps=steps, list_elements=True
+        )
+        data = fetcher.to_dict()
+        assert data["init_parameters"]["steps"] == steps
+        assert data["init_parameters"]["list_elements"] is True
+
+        restored = ScrapeUnblockerFetcher.from_dict(data)
+        assert restored.steps == steps
+        assert restored.list_elements is True
+
+
+class TestWebSearch:
+    @patch("scrapeunblocker_haystack.search.requests.post")
+    def test_run(self, mock_post):
+        mock_post.return_value = _response(
+            json_data={
+                "organic": [
+                    {"title": "First", "url": "https://a.com", "description": "snippet a", "position": "1"},
+                    {"title": "Second", "url": "https://b.com", "description": "snippet b", "position": "2"},
+                ]
+            }
+        )
+        search = ScrapeUnblockerWebSearch(api_key=API_KEY)
+
+        result = search.run(query="test query")
+
+        assert len(result["documents"]) == 2
+        assert result["documents"][0].content == "snippet a"
+        assert result["documents"][0].meta["title"] == "First"
+        assert result["documents"][0].meta["link"] == "https://a.com"
+        assert result["documents"][0].meta["query"] == "test query"
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["keyword"] == "test query"
+
+    @patch("scrapeunblocker_haystack.search.requests.post")
+    def test_top_k(self, mock_post):
+        mock_post.return_value = _response(
+            json_data={"organic": [{"title": str(i), "url": "", "description": str(i)} for i in range(10)]}
+        )
+        search = ScrapeUnblockerWebSearch(api_key=API_KEY, top_k=3)
+
+        assert len(search.run(query="q")["documents"]) == 3
+
+    @patch("scrapeunblocker_haystack.search.requests.post")
+    def test_missing_organic_returns_empty(self, mock_post):
+        mock_post.return_value = _response(json_data={"totalResults": None})
+        search = ScrapeUnblockerWebSearch(api_key=API_KEY)
+
+        assert search.run(query="q")["documents"] == []
+
+    def test_to_dict_and_back(self):
+        search = ScrapeUnblockerWebSearch(api_key=ENV_KEY, top_k=5, proxy_country="us")
+        data = search.to_dict()
+        restored = ScrapeUnblockerWebSearch.from_dict(data)
+        assert restored.top_k == 5
+        assert restored.proxy_country == "us"
+
+
+class TestTargetNotFound:
+    NOT_FOUND = "<html><title>404 Not Found</title></html>"
+
+    def _gone(self, status_code: int, origin: str) -> MagicMock:
+        response = _response(text=self.NOT_FOUND, status_code=status_code)
+        response.headers = {"Content-Type": "text/html", "X-Origin-Status": origin}
+        return response
+
+    @pytest.mark.parametrize("status", [404, 410])
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_missing_page_is_skipped(self, mock_post, status):
+        mock_post.side_effect = [self._gone(status, str(status)), _response()]
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        result = fetcher.run(urls=["https://example.com/gone", "https://example.com"])
+
+        assert [d.meta["url"] for d in result["documents"]] == ["https://example.com"]
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_legacy_200_with_origin_404_is_skipped(self, mock_post):
+        mock_post.return_value = self._gone(200, "404")
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        assert fetcher.run(urls=["https://example.com/gone"])["documents"] == []
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_missing_page_raises_when_asked(self, mock_post):
+        mock_post.return_value = self._gone(410, "410")
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, raise_on_failure=True)
+
+        with pytest.raises(TargetNotFoundError) as exc:
+            fetcher.run(urls=["https://example.com/gone"])
+        assert exc.value.origin_status == 410
+        assert exc.value.url == "https://example.com/gone"
+        assert "404 Not Found" in exc.value.body
+        assert "billed" in str(exc.value)
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_dead_origin_5xx_page_is_still_a_document(self, mock_post):
+        response = _response(text="<html>502 Bad Gateway</html>")
+        response.headers = {"Content-Type": "text/html", "X-Origin-Status": "502"}
+        mock_post.return_value = response
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        assert len(fetcher.run(urls=["https://example.com"])["documents"]) == 1

scrapeunblocker_haystack-0.1.0/CHANGELOG.md

@@ -1,8 +0,0 @@
-# Changelog
-
-## 0.1.0
-
-Initial release.
-
-- `ScrapeUnblockerFetcher` - fetch pages behind anti-bot protections as Documents
-- `ScrapeUnblockerWebSearch` - Google organic results as Documents

scrapeunblocker_haystack-0.1.0/src/scrapeunblocker_haystack/__init__.py

@@ -1,4 +0,0 @@
-from scrapeunblocker_haystack.fetcher import ScrapeUnblockerFetcher
-from scrapeunblocker_haystack.search import ScrapeUnblockerWebSearch
-
-__all__ = ["ScrapeUnblockerFetcher", "ScrapeUnblockerWebSearch"]

scrapeunblocker_haystack-0.1.0/src/scrapeunblocker_haystack/fetcher.py

@@ -1,143 +0,0 @@
-"""Haystack component that fetches pages through the ScrapeUnblocker API."""
-
-from __future__ import annotations
-
-import json
-from typing import Any, Dict, List, Optional
-
-import requests
-from haystack import Document, component, default_from_dict, default_to_dict, logging
-from haystack.utils import Secret, deserialize_secrets_inplace
-
-logger = logging.getLogger(__name__)
-
-DEFAULT_BASE_URL = "https://api.scrapeunblocker.com"
-DEFAULT_TIMEOUT = 180
-
-
-@component
-class ScrapeUnblockerFetcher:
-    """
-    Fetches web pages through the ScrapeUnblocker API and returns them as Documents.
-
-    ScrapeUnblocker renders pages in a real browser behind anti-bot protections
-    (Cloudflare, DataDome, PerimeterX, Akamai), so it reaches pages that a plain
-    HTTP fetch cannot.
-
-    ### Usage example
-
-    ```python
-    from scrapeunblocker_haystack import ScrapeUnblockerFetcher
-
-    fetcher = ScrapeUnblockerFetcher()
-    result = fetcher.run(urls=["https://example.com"])
-    print(result["documents"][0].content[:200])
-    ```
-    """
-
-    def __init__(
-        self,
-        api_key: Secret = Secret.from_env_var("SCRAPEUNBLOCKER_API_KEY"),
-        parsed_data: bool = False,
-        proxy_country: Optional[str] = None,
-        time_sleep: Optional[int] = None,
-        base_url: str = DEFAULT_BASE_URL,
-        timeout: int = DEFAULT_TIMEOUT,
-        raise_on_failure: bool = False,
-    ) -> None:
-        """
-        :param api_key: ScrapeUnblocker API key. Read from `SCRAPEUNBLOCKER_API_KEY` by default.
-        :param parsed_data: Return AI-parsed structured JSON instead of raw HTML.
-        :param proxy_country: Two-letter country code for the exit IP, for geo-restricted content.
-        :param time_sleep: Seconds to wait after page load before capturing.
-        :param base_url: API base URL. Override to target a different environment.
-        :param timeout: HTTP timeout in seconds.
-        :param raise_on_failure: Raise instead of skipping when a URL cannot be fetched.
-        """
-        self.api_key = api_key
-        self.parsed_data = parsed_data
-        self.proxy_country = proxy_country
-        self.time_sleep = time_sleep
-        self.base_url = base_url.rstrip("/")
-        self.timeout = timeout
-        self.raise_on_failure = raise_on_failure
-
-    def to_dict(self) -> Dict[str, Any]:
-        """Serialize this component to a dictionary."""
-        return default_to_dict(
-            self,
-            api_key=self.api_key.to_dict(),
-            parsed_data=self.parsed_data,
-            proxy_country=self.proxy_country,
-            time_sleep=self.time_sleep,
-            base_url=self.base_url,
-            timeout=self.timeout,
-            raise_on_failure=self.raise_on_failure,
-        )
-
-    @classmethod
-    def from_dict(cls, data: Dict[str, Any]) -> "ScrapeUnblockerFetcher":
-        """Deserialize this component from a dictionary."""
-        deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
-        return default_from_dict(cls, data)
-
-    def _fetch(self, url: str) -> requests.Response:
-        params: Dict[str, Any] = {"url": url}
-        if self.parsed_data:
-            params["parsed_data"] = True
-        if self.proxy_country:
-            params["proxy_country"] = self.proxy_country
-        if self.time_sleep is not None:
-            params["time_sleep"] = self.time_sleep
-
-        response = requests.post(
-            f"{self.base_url}/getPageSource",
-            params=params,
-            headers={"X-ScrapeUnblocker-Key": self.api_key.resolve_value()},
-            timeout=self.timeout,
-        )
-        response.raise_for_status()
-        return response
-
-    @component.output_types(documents=List[Document])
-    def run(self, urls: List[str]) -> Dict[str, List[Document]]:
-        """
-        Fetch each URL and return one Document per successfully fetched page.
-
-        :param urls: URLs to fetch.
-        :returns: A dictionary with a `documents` key holding the fetched pages.
-        """
-        documents: List[Document] = []
-
-        for url in urls:
-            try:
-                response = self._fetch(url)
-            except Exception as exc:
-                if self.raise_on_failure:
-                    raise
-                # One unreachable URL should not discard the rest of the batch.
-                logger.warning(
-                    "ScrapeUnblocker could not fetch {url}: {error}", url=url, error=str(exc)
-                )
-                continue
-
-            if self.parsed_data:
-                try:
-                    content = json.dumps(response.json(), ensure_ascii=False)
-                except ValueError:
-                    content = response.text
-            else:
-                content = response.text
-
-            documents.append(
-                Document(
-                    content=content,
-                    meta={
-                        "url": url,
-                        "content_type": response.headers.get("Content-Type", ""),
-                        "parsed_data": self.parsed_data,
-                    },
-                )
-            )
-
-        return {"documents": documents}

scrapeunblocker_haystack-0.1.0/src/scrapeunblocker_haystack/version.py

@@ -1 +0,0 @@
-__version__ = "0.1.0"

scrapeunblocker_haystack-0.1.0/tests/test_components.py

@@ -1,150 +0,0 @@
-"""Unit tests for the ScrapeUnblocker Haystack components."""
-
-import json
-from unittest.mock import MagicMock, patch
-
-import pytest
-from haystack import Document
-from haystack.utils import Secret
-
-from scrapeunblocker_haystack import ScrapeUnblockerFetcher, ScrapeUnblockerWebSearch
-
-API_KEY = Secret.from_token("test_key")
-# Haystack refuses to serialize token secrets, so to_dict tests use an env-var secret.
-ENV_KEY = Secret.from_env_var("SCRAPEUNBLOCKER_API_KEY")
-
-
-def _response(text: str = "<html><title>Test</title></html>", json_data=None) -> MagicMock:
-    response = MagicMock()
-    response.text = text
-    response.headers = {"Content-Type": "text/html"}
-    response.raise_for_status.return_value = None
-    if json_data is not None:
-        response.json.return_value = json_data
-    return response
-
-
-class TestFetcher:
-    def test_defaults(self):
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
-        assert fetcher.parsed_data is False
-        assert fetcher.base_url == "https://api.scrapeunblocker.com"
-
-    def test_base_url_trailing_slash_stripped(self):
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, base_url="https://example.com/")
-        assert fetcher.base_url == "https://example.com"
-
-    def test_to_dict_and_back(self):
-        fetcher = ScrapeUnblockerFetcher(api_key=ENV_KEY, parsed_data=True, proxy_country="de")
-        data = fetcher.to_dict()
-        assert data["init_parameters"]["parsed_data"] is True
-        assert data["init_parameters"]["proxy_country"] == "de"
-
-        restored = ScrapeUnblockerFetcher.from_dict(data)
-        assert restored.parsed_data is True
-        assert restored.proxy_country == "de"
-
-    @patch("scrapeunblocker_haystack.fetcher.requests.post")
-    def test_run_single_url(self, mock_post):
-        mock_post.return_value = _response()
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
-
-        result = fetcher.run(urls=["https://example.com"])
-
-        assert len(result["documents"]) == 1
-        doc = result["documents"][0]
-        assert isinstance(doc, Document)
-        assert "Test" in doc.content
-        assert doc.meta["url"] == "https://example.com"
-
-        _, kwargs = mock_post.call_args
-        assert kwargs["headers"]["X-ScrapeUnblocker-Key"] == "test_key"
-        assert kwargs["params"] == {"url": "https://example.com"}
-
-    @patch("scrapeunblocker_haystack.fetcher.requests.post")
-    def test_run_forwards_options(self, mock_post):
-        mock_post.return_value = _response(json_data={"ok": True})
-        fetcher = ScrapeUnblockerFetcher(
-            api_key=API_KEY, parsed_data=True, proxy_country="de", time_sleep=5
-        )
-        fetcher.run(urls=["https://example.com"])
-
-        _, kwargs = mock_post.call_args
-        assert kwargs["params"]["parsed_data"] is True
-        assert kwargs["params"]["proxy_country"] == "de"
-        assert kwargs["params"]["time_sleep"] == 5
-
-    @patch("scrapeunblocker_haystack.fetcher.requests.post")
-    def test_parsed_data_serialised_as_json(self, mock_post):
-        mock_post.return_value = _response(json_data={"title": "Test", "price": "9.99"})
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, parsed_data=True)
-
-        result = fetcher.run(urls=["https://example.com"])
-
-        assert json.loads(result["documents"][0].content) == {"title": "Test", "price": "9.99"}
-
-    @patch("scrapeunblocker_haystack.fetcher.requests.post")
-    def test_failure_is_skipped_by_default(self, mock_post):
-        mock_post.side_effect = [Exception("boom"), _response()]
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
-
-        result = fetcher.run(urls=["https://broken.com", "https://ok.com"])
-
-        assert len(result["documents"]) == 1
-        assert result["documents"][0].meta["url"] == "https://ok.com"
-
-    @patch("scrapeunblocker_haystack.fetcher.requests.post")
-    def test_failure_raises_when_requested(self, mock_post):
-        mock_post.side_effect = Exception("boom")
-        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, raise_on_failure=True)
-
-        with pytest.raises(Exception, match="boom"):
-            fetcher.run(urls=["https://broken.com"])
-
-
-class TestWebSearch:
-    @patch("scrapeunblocker_haystack.search.requests.post")
-    def test_run(self, mock_post):
-        mock_post.return_value = _response(
-            json_data={
-                "organic": [
-                    {"title": "First", "url": "https://a.com", "description": "snippet a", "position": "1"},
-                    {"title": "Second", "url": "https://b.com", "description": "snippet b", "position": "2"},
-                ]
-            }
-        )
-        search = ScrapeUnblockerWebSearch(api_key=API_KEY)
-
-        result = search.run(query="test query")
-
-        assert len(result["documents"]) == 2
-        assert result["documents"][0].content == "snippet a"
-        assert result["documents"][0].meta["title"] == "First"
-        assert result["documents"][0].meta["link"] == "https://a.com"
-        assert result["documents"][0].meta["query"] == "test query"
-
-        _, kwargs = mock_post.call_args
-        assert kwargs["params"]["keyword"] == "test query"
-
-    @patch("scrapeunblocker_haystack.search.requests.post")
-    def test_top_k(self, mock_post):
-        mock_post.return_value = _response(
-            json_data={"organic": [{"title": str(i), "url": "", "description": str(i)} for i in range(10)]}
-        )
-        search = ScrapeUnblockerWebSearch(api_key=API_KEY, top_k=3)
-
-        assert len(search.run(query="q")["documents"]) == 3
-
-    @patch("scrapeunblocker_haystack.search.requests.post")
-    def test_missing_organic_returns_empty(self, mock_post):
-        mock_post.return_value = _response(json_data={"totalResults": None})
-        search = ScrapeUnblockerWebSearch(api_key=API_KEY)
-
-        assert search.run(query="q")["documents"] == []
-
-    def test_to_dict_and_back(self):
-        search = ScrapeUnblockerWebSearch(api_key=ENV_KEY, top_k=5, proxy_country="us")
-        data = search.to_dict()
-        restored = ScrapeUnblockerWebSearch.from_dict(data)
-        assert restored.top_k == 5
-        assert restored.proxy_country == "us"