scrapeunblocker-haystack 0.3.0__tar.gz → 0.4.1__tar.gz

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.4.1
+
+- `ScrapeUnblockerWebSearch` adds `displayed_url`, `domain`, `source` (the
+  site's name) and `date` to each Document's metadata. The API now returns
+  every organic result with its real URL, also those Google shows without a
+  snippet; such a Document's content is the result's title.
+
+## 0.4.0
+
+- `ScrapeUnblockerFetcher` can return pages as clean Markdown for LLMs and RAG.
+  With `markdown=True` each Document's content is the page as Markdown
+  (headings, lists, GitHub-flavored tables, absolute links, images with alt
+  text, fenced code; no scripts, styles, navigation, footers or forms), ready
+  for a prompt or a splitter without an HTML converter. `main_content_only`
+  (default `True`) keeps only the main content; `False` converts the whole
+  page. `meta["markdown"]` is `True` for these Documents (`False` otherwise)
+  and `meta["main_content_only"]` records the setting. Billed like an HTML
+  fetch. `markdown` cannot be combined with `parsed_data` or `list_elements`
+  (the API refuses it), so that raises `ValueError`. Defaults are unchanged.
+- README: the pipeline example fetches Markdown and passes it straight to the
+  prompt. The previous example connected the fetcher's Documents to
+  `HTMLToDocument`, which only accepts sources, so the pipeline did not build.
+
 ## 0.3.0
 
 - With `parsed_data=True`, a page that loads but holds no structured data is

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: scrapeunblocker-haystack
-Version: 0.3.0
+Version: 0.4.1
 Summary: Haystack integration for ScrapeUnblocker - scrape pages behind anti-bot protections
 Project-URL: Homepage, https://www.scrapeunblocker.com
 Project-URL: Documentation, https://docs.scrapeunblocker.com
@@ -69,10 +69,12 @@ print(result["documents"][0].content[:200])
 | --- | --- | --- |
 | `api_key` | `SCRAPEUNBLOCKER_API_KEY` env var | ScrapeUnblocker API key |
 | `parsed_data` | `False` | Return AI-parsed structured JSON instead of raw HTML |
-| `proxy_country` | `None` | Two-letter country code for the exit IP |
+| `proxy_country` | `None` | Two-letter country code of the country the request is made from |
 | `time_sleep` | `None` | Seconds to wait after load before capturing |
 | `steps` | `None` | Browser actions to run before the HTML is captured (see below) |
 | `list_elements` | `False` | Return the parsed elements JSON instead of HTML (see below) |
+| `markdown` | `False` | Return the page as clean Markdown instead of HTML (see below) |
+| `main_content_only` | `True` | With `markdown`, keep only the main content (drop navigation, header, footer, sidebars) |
 | `base_url` | `https://api.scrapeunblocker.com` | API base URL |
 | `timeout` | `180` | HTTP timeout in seconds |
 | `raise_on_failure` | `False` | Raise instead of skipping a URL that fails |
@@ -80,6 +82,36 @@ print(result["documents"][0].content[:200])
 By default a URL that cannot be fetched is logged and skipped, so one bad URL
 does not discard the rest of the batch.
 
+#### Markdown for LLMs and RAG
+
+Set `markdown=True` to get each page as clean Markdown instead of HTML:
+headings, lists, tables (GitHub-flavored), absolute links, images with their
+alt text and fenced code blocks, without scripts, styles, navigation, footers,
+forms or hidden elements. The Markdown goes straight into a prompt or a
+splitter, with no HTML converter, and is usually a small fraction of the
+HTML's size. It is billed like an HTML fetch.
+
+```python
+from scrapeunblocker_haystack import ScrapeUnblockerFetcher
+
+fetcher = ScrapeUnblockerFetcher(markdown=True)
+result = fetcher.run(
+    urls=["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]
+)
+
+doc = result["documents"][0]
+print("# A Light in the Attic" in doc.content)  # True
+print(doc.meta["content_type"])                 # text/markdown; charset=utf-8
+```
+
+By default only the page's main content is kept: navigation, the site header,
+footer and sidebars are dropped. Pass `main_content_only=False` to convert the
+whole page. `markdown` works with `proxy_country`, `time_sleep` and `steps`,
+but not with `parsed_data` or `list_elements`, which return their own JSON
+formats; combining them raises `ValueError`. A page that does not exist
+(404/410) is skipped or raises `TargetNotFoundError`, as below, with the
+site's own page as Markdown in `body`.
+
 #### Browser steps
 
 Pass `steps` to drive the page in a real browser before its HTML is captured -
@@ -153,7 +185,8 @@ print(doc.content)                # {"url": ..., "count": ..., "elements": [...]
 ### ScrapeUnblockerWebSearch
 
 Searches Google and returns the organic results as `Document` objects, with the
-snippet as content and `title` / `link` / `position` in the metadata.
+snippet as content and `title` / `link` / `position` / `displayed_url` / `domain` /
+`source` (the site's name) / `date` in the metadata.
 
 ```python
 from scrapeunblocker_haystack import ScrapeUnblockerWebSearch
@@ -174,12 +207,11 @@ for doc in result["documents"]:
 
 ## In a pipeline
 
-Search the web, fetch the pages behind the results, and answer from them:
+Fetch pages as Markdown and answer a question from them:
 
 ```python
 from haystack import Pipeline
 from haystack.components.builders import ChatPromptBuilder
-from haystack.components.converters import HTMLToDocument
 from haystack.components.generators.chat import OpenAIChatGenerator
 from haystack.dataclasses import ChatMessage
 
@@ -194,13 +226,11 @@ prompt = [
 ]
 
 pipe = Pipeline()
-pipe.add_component("fetcher", ScrapeUnblockerFetcher())
-pipe.add_component("converter", HTMLToDocument())
+pipe.add_component("fetcher", ScrapeUnblockerFetcher(markdown=True))
 pipe.add_component("prompt_builder", ChatPromptBuilder(template=prompt, required_variables="*"))
 pipe.add_component("llm", OpenAIChatGenerator())
 
-pipe.connect("fetcher.documents", "converter.sources")
-pipe.connect("converter.documents", "prompt_builder.documents")
+pipe.connect("fetcher.documents", "prompt_builder.documents")
 pipe.connect("prompt_builder.prompt", "llm.messages")
 
 result = pipe.run(

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/README.md

@@ -44,10 +44,12 @@ print(result["documents"][0].content[:200])
 | --- | --- | --- |
 | `api_key` | `SCRAPEUNBLOCKER_API_KEY` env var | ScrapeUnblocker API key |
 | `parsed_data` | `False` | Return AI-parsed structured JSON instead of raw HTML |
-| `proxy_country` | `None` | Two-letter country code for the exit IP |
+| `proxy_country` | `None` | Two-letter country code of the country the request is made from |
 | `time_sleep` | `None` | Seconds to wait after load before capturing |
 | `steps` | `None` | Browser actions to run before the HTML is captured (see below) |
 | `list_elements` | `False` | Return the parsed elements JSON instead of HTML (see below) |
+| `markdown` | `False` | Return the page as clean Markdown instead of HTML (see below) |
+| `main_content_only` | `True` | With `markdown`, keep only the main content (drop navigation, header, footer, sidebars) |
 | `base_url` | `https://api.scrapeunblocker.com` | API base URL |
 | `timeout` | `180` | HTTP timeout in seconds |
 | `raise_on_failure` | `False` | Raise instead of skipping a URL that fails |
@@ -55,6 +57,36 @@ print(result["documents"][0].content[:200])
 By default a URL that cannot be fetched is logged and skipped, so one bad URL
 does not discard the rest of the batch.
 
+#### Markdown for LLMs and RAG
+
+Set `markdown=True` to get each page as clean Markdown instead of HTML:
+headings, lists, tables (GitHub-flavored), absolute links, images with their
+alt text and fenced code blocks, without scripts, styles, navigation, footers,
+forms or hidden elements. The Markdown goes straight into a prompt or a
+splitter, with no HTML converter, and is usually a small fraction of the
+HTML's size. It is billed like an HTML fetch.
+
+```python
+from scrapeunblocker_haystack import ScrapeUnblockerFetcher
+
+fetcher = ScrapeUnblockerFetcher(markdown=True)
+result = fetcher.run(
+    urls=["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]
+)
+
+doc = result["documents"][0]
+print("# A Light in the Attic" in doc.content)  # True
+print(doc.meta["content_type"])                 # text/markdown; charset=utf-8
+```
+
+By default only the page's main content is kept: navigation, the site header,
+footer and sidebars are dropped. Pass `main_content_only=False` to convert the
+whole page. `markdown` works with `proxy_country`, `time_sleep` and `steps`,
+but not with `parsed_data` or `list_elements`, which return their own JSON
+formats; combining them raises `ValueError`. A page that does not exist
+(404/410) is skipped or raises `TargetNotFoundError`, as below, with the
+site's own page as Markdown in `body`.
+
 #### Browser steps
 
 Pass `steps` to drive the page in a real browser before its HTML is captured -
@@ -128,7 +160,8 @@ print(doc.content)                # {"url": ..., "count": ..., "elements": [...]
 ### ScrapeUnblockerWebSearch
 
 Searches Google and returns the organic results as `Document` objects, with the
-snippet as content and `title` / `link` / `position` in the metadata.
+snippet as content and `title` / `link` / `position` / `displayed_url` / `domain` /
+`source` (the site's name) / `date` in the metadata.
 
 ```python
 from scrapeunblocker_haystack import ScrapeUnblockerWebSearch
@@ -149,12 +182,11 @@ for doc in result["documents"]:
 
 ## In a pipeline
 
-Search the web, fetch the pages behind the results, and answer from them:
+Fetch pages as Markdown and answer a question from them:
 
 ```python
 from haystack import Pipeline
 from haystack.components.builders import ChatPromptBuilder
-from haystack.components.converters import HTMLToDocument
 from haystack.components.generators.chat import OpenAIChatGenerator
 from haystack.dataclasses import ChatMessage
 
@@ -169,13 +201,11 @@ prompt = [
 ]
 
 pipe = Pipeline()
-pipe.add_component("fetcher", ScrapeUnblockerFetcher())
-pipe.add_component("converter", HTMLToDocument())
+pipe.add_component("fetcher", ScrapeUnblockerFetcher(markdown=True))
 pipe.add_component("prompt_builder", ChatPromptBuilder(template=prompt, required_variables="*"))
 pipe.add_component("llm", OpenAIChatGenerator())
 
-pipe.connect("fetcher.documents", "converter.sources")
-pipe.connect("converter.documents", "prompt_builder.documents")
+pipe.connect("fetcher.documents", "prompt_builder.documents")
 pipe.connect("prompt_builder.prompt", "llm.messages")
 
 result = pipe.run(

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/src/scrapeunblocker_haystack/fetcher.py

@@ -103,6 +103,18 @@ class ScrapeUnblockerFetcher:
     print(result["documents"][0].content[:200])
     ```
 
+    ### Markdown
+
+    Set `markdown=True` to get each page as clean Markdown, ready for an LLM or a
+    RAG pipeline, instead of HTML. By default only the main content is kept;
+    `main_content_only=False` converts the whole page:
+
+    ```python
+    fetcher = ScrapeUnblockerFetcher(markdown=True)
+    result = fetcher.run(urls=["https://example.com"])
+    print(result["documents"][0].content[:200])
+    ```
+
     ### Browser steps
 
     Drive the page in a real browser before the HTML is captured - click, type,
@@ -135,6 +147,8 @@ class ScrapeUnblockerFetcher:
         base_url: str = DEFAULT_BASE_URL,
         timeout: int = DEFAULT_TIMEOUT,
         raise_on_failure: bool = False,
+        markdown: bool = False,
+        main_content_only: bool = True,
     ) -> None:
         """
         :param api_key: ScrapeUnblocker API key. Read from `SCRAPEUNBLOCKER_API_KEY` by default.
@@ -152,7 +166,21 @@ class ScrapeUnblockerFetcher:
         :param timeout: HTTP timeout in seconds.
         :param raise_on_failure: Raise instead of skipping when a URL cannot be fetched
             (including when a browser step fails or the page does not exist).
+        :param markdown: Return each page as clean Markdown instead of HTML: headings,
+            lists, tables, absolute links, images with alt text and code blocks, without
+            scripts, styles, navigation, footers or forms. Billed like an HTML fetch.
+            Cannot be combined with `parsed_data` or `list_elements`.
+        :param main_content_only: With `markdown`, keep only the page's main content and
+            drop navigation, the site header, footer and sidebars. `False` converts the
+            whole page. Ignored without `markdown`.
+        :raises ValueError: If `markdown` is combined with `parsed_data` or `list_elements`.
         """
+        if markdown and (parsed_data or list_elements):
+            # The API refuses these combinations (HTTP 422): each returns its own format.
+            raise ValueError(
+                "markdown=True cannot be combined with parsed_data or list_elements: "
+                "each returns its own format."
+            )
         self.api_key = api_key
         self.parsed_data = parsed_data
         self.proxy_country = proxy_country
@@ -162,6 +190,8 @@ class ScrapeUnblockerFetcher:
         self.base_url = base_url.rstrip("/")
         self.timeout = timeout
         self.raise_on_failure = raise_on_failure
+        self.markdown = markdown
+        self.main_content_only = main_content_only
 
     def to_dict(self) -> Dict[str, Any]:
         """Serialize this component to a dictionary."""
@@ -176,6 +206,8 @@ class ScrapeUnblockerFetcher:
             base_url=self.base_url,
             timeout=self.timeout,
             raise_on_failure=self.raise_on_failure,
+            markdown=self.markdown,
+            main_content_only=self.main_content_only,
         )
 
     @classmethod
@@ -196,6 +228,9 @@ class ScrapeUnblockerFetcher:
             params["steps"] = json.dumps(self.steps)
         if self.list_elements:
             params["list_elements"] = True
+        if self.markdown:
+            params["markdown"] = True
+            params["main_content_only"] = self.main_content_only
 
         response = requests.post(
             f"{self.base_url}/getPageSource",
@@ -241,7 +276,8 @@ class ScrapeUnblockerFetcher:
         Document; with `raise_on_failure` it raises `TargetNotFoundError`. With
         `parsed_data`, a page holding no structured data still yields a
         Document: its JSON carries `data_extracted: false` and the rendered
-        `html`, and `meta["data_extracted"]` is False.
+        `html`, and `meta["data_extracted"]` is False. With `markdown`, the
+        Document's content is the page as Markdown and `meta["markdown"]` is True.
 
         :param urls: URLs to fetch.
         :returns: A dictionary with a `documents` key holding the fetched pages.
@@ -273,7 +309,10 @@ class ScrapeUnblockerFetcher:
                 "content_type": response.headers.get("Content-Type", ""),
                 "parsed_data": self.parsed_data,
                 "list_elements": self.list_elements,
+                "markdown": self.markdown,
             }
+            if self.markdown:
+                meta["main_content_only"] = self.main_content_only
 
             if json_mode:
                 try:

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/src/scrapeunblocker_haystack/search.py

@@ -17,8 +17,9 @@ class ScrapeUnblockerWebSearch:
     """
     Searches Google through the ScrapeUnblocker API and returns the organic results.
 
-    Each result becomes a Document whose content is the snippet, with the title,
-    link and position in the metadata.
+    Each result becomes a Document whose content is the snippet (the title when
+    Google shows none), with the title, link, position, displayed URL, domain,
+    site name and date in the metadata.
 
     ### Usage example
 
@@ -107,6 +108,10 @@ class ScrapeUnblockerWebSearch:
                     "title": result.get("title"),
                     "link": result.get("url"),
                     "position": result.get("position"),
+                    "displayed_url": result.get("displayedUrl"),
+                    "domain": result.get("domain"),
+                    "source": result.get("source"),
+                    "date": result.get("date"),
                     "query": query,
                 },
             )

scrapeunblocker_haystack-0.4.1/src/scrapeunblocker_haystack/version.py

@@ -0,0 +1 @@
+__version__ = "0.4.1"

{scrapeunblocker_haystack-0.3.0 → scrapeunblocker_haystack-0.4.1}/tests/test_components.py

@@ -211,7 +211,9 @@ class TestWebSearch:
         mock_post.return_value = _response(
             json_data={
                 "organic": [
-                    {"title": "First", "url": "https://a.com", "description": "snippet a", "position": "1"},
+                    {"title": "First", "url": "https://a.com", "description": "snippet a", "position": "1",
+                     "displayedUrl": "https://a.com › docs", "domain": "a.com", "source": "A",
+                     "date": "Oct 2, 2026"},
                     {"title": "Second", "url": "https://b.com", "description": "snippet b", "position": "2"},
                 ]
             }
@@ -225,6 +227,10 @@ class TestWebSearch:
         assert result["documents"][0].meta["title"] == "First"
         assert result["documents"][0].meta["link"] == "https://a.com"
         assert result["documents"][0].meta["query"] == "test query"
+        assert result["documents"][0].meta["domain"] == "a.com"
+        assert result["documents"][0].meta["source"] == "A"
+        assert result["documents"][0].meta["date"] == "Oct 2, 2026"
+        assert result["documents"][1].meta["date"] is None
 
         _, kwargs = mock_post.call_args
         assert kwargs["params"]["keyword"] == "test query"
@@ -345,3 +351,143 @@ class TestNoDataExtracted:
         fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, parsed_data=True)
 
         assert fetcher.run(urls=["https://example.com/gone"])["documents"] == []
+
+
+class TestMarkdown:
+    MARKDOWN = "# A Light in the Attic\n\n£51.77\n\n| UPC | a897fe39b1053632 |\n"
+
+    def _markdown(self, text: str = MARKDOWN, status_code: int = 200, origin=None) -> MagicMock:
+        response = _response(text=text, status_code=status_code)
+        response.headers = {"Content-Type": "text/markdown; charset=utf-8"}
+        if origin is not None:
+            response.headers["X-Origin-Status"] = origin
+        return response
+
+    def test_off_by_default(self):
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+        assert fetcher.markdown is False
+        assert fetcher.main_content_only is True
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_markdown_query(self, mock_post):
+        mock_post.return_value = self._markdown()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True)
+
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"] == {
+            "url": "https://example.com",
+            "markdown": True,
+            "main_content_only": True,
+        }
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_whole_page(self, mock_post):
+        mock_post.return_value = self._markdown()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True, main_content_only=False)
+
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["main_content_only"] is False
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_main_content_only_not_sent_without_markdown(self, mock_post):
+        mock_post.return_value = _response()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, main_content_only=False)
+
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"] == {"url": "https://example.com"}
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_markdown_document(self, mock_post):
+        mock_post.return_value = self._markdown()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True, proxy_country="gb")
+
+        doc = fetcher.run(urls=["https://example.com/book"])["documents"][0]
+
+        assert doc.content == self.MARKDOWN
+        assert doc.meta["url"] == "https://example.com/book"
+        assert doc.meta["markdown"] is True
+        assert doc.meta["main_content_only"] is True
+        assert doc.meta["content_type"].startswith("text/markdown")
+        _, kwargs = mock_post.call_args
+        assert kwargs["params"]["proxy_country"] == "gb"
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_markdown_that_looks_like_json_stays_text(self, mock_post):
+        mock_post.return_value = self._markdown(text='{"a": 1}')
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True)
+
+        assert fetcher.run(urls=["https://example.com"])["documents"][0].content == '{"a": 1}'
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_html_document_meta_says_not_markdown(self, mock_post):
+        mock_post.return_value = _response()
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY)
+
+        doc = fetcher.run(urls=["https://example.com"])["documents"][0]
+
+        assert doc.meta["markdown"] is False
+        assert "main_content_only" not in doc.meta
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_markdown_with_steps(self, mock_post):
+        mock_post.return_value = self._markdown()
+        steps = [{"action": "click", "selector": "#more"}]
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True, steps=steps)
+
+        fetcher.run(urls=["https://example.com"])
+
+        _, kwargs = mock_post.call_args
+        assert json.loads(kwargs["params"]["steps"]) == steps
+        assert kwargs["params"]["markdown"] is True
+
+    @pytest.mark.parametrize("other", [{"parsed_data": True}, {"list_elements": True}])
+    def test_rejects_json_formats(self, other):
+        with pytest.raises(ValueError, match="cannot be combined"):
+            ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True, **other)
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_missing_page_is_skipped(self, mock_post):
+        mock_post.side_effect = [
+            self._markdown(text="# 404 Not Found", status_code=404, origin="404"),
+            self._markdown(),
+        ]
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True)
+
+        result = fetcher.run(urls=["https://example.com/gone", "https://example.com"])
+
+        assert [d.meta["url"] for d in result["documents"]] == ["https://example.com"]
+
+    @patch("scrapeunblocker_haystack.fetcher.requests.post")
+    def test_missing_page_raises_with_markdown_body(self, mock_post):
+        mock_post.return_value = self._markdown(text="# 404 Not Found", status_code=404, origin="404")
+        fetcher = ScrapeUnblockerFetcher(api_key=API_KEY, markdown=True, raise_on_failure=True)
+
+        with pytest.raises(TargetNotFoundError) as exc:
+            fetcher.run(urls=["https://example.com/gone"])
+        assert exc.value.origin_status == 404
+        assert exc.value.body == "# 404 Not Found"
+
+    def test_to_dict_and_back(self):
+        fetcher = ScrapeUnblockerFetcher(api_key=ENV_KEY, markdown=True, main_content_only=False)
+        data = fetcher.to_dict()
+        assert data["init_parameters"]["markdown"] is True
+        assert data["init_parameters"]["main_content_only"] is False
+
+        restored = ScrapeUnblockerFetcher.from_dict(data)
+        assert restored.markdown is True
+        assert restored.main_content_only is False
+
+    def test_from_dict_of_older_version(self):
+        data = ScrapeUnblockerFetcher(api_key=ENV_KEY).to_dict()
+        del data["init_parameters"]["markdown"]
+        del data["init_parameters"]["main_content_only"]
+
+        restored = ScrapeUnblockerFetcher.from_dict(data)
+        assert restored.markdown is False
+        assert restored.main_content_only is True

scrapeunblocker_haystack-0.3.0/src/scrapeunblocker_haystack/version.py

@@ -1 +0,0 @@
-__version__ = "0.3.0"