scrapegraphai 2.2.1__tar.gz → 2.2.3__tar.gz

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/.github/workflows/test-suite.yml

@@ -42,6 +42,7 @@ jobs:
           tests/test_batch_api.py
           tests/test_csv_scraper_multi_graph.py
           tests/test_depth_search_graph.py
+          tests/test_error_page_detection.py
           tests/test_json_scraper_graph.py
           tests/test_minimax_models.py
           tests/test_scrape_do.py

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/CHANGELOG.md

@@ -1,3 +1,22 @@
+## [2.2.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.2...v2.2.3) (2026-09-07)
+
+
+### Bug Fixes
+
+* **proxy:** handle schemeless proxy server format and broker routing ([70dbd2d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/70dbd2d26d63bdc7473293371a275a4d856737f9))
+
+
+### Docs
+
+* add NodeMaven sponsor to all README languages ([cc8568c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cc8568cc76f765f3d2bd2db0bdf5018d347af101))
+
+## [2.2.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.1...v2.2.2) (2026-08-23)
+
+
+### Bug Fixes
+
+* **fetch:** surface HTTP errors and missing content instead of answering NA ([adc92f7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/adc92f7eff9aa39d70e3848c6867328c3ba2ba2d)), closes [#1102](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1102) [#1102](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1102)
+
 ## [2.2.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0...v2.2.1) (2026-08-21)
 
 

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: scrapegraphai
-Version: 2.2.1
+Version: 2.2.3
 Summary: A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines.
 Author-email: Marco Vinciguerra <mvincig11@gmail.com>, Lorenzo Padoan <lorenzo.padoan977@gmail.com>
 Keywords: ai,artificial intelligence,gpt,graph,langchain,llm,machine learning,natural language processing,nlp,openai,rag,scrapegraph,scrapegraphai,scraping,web scraping,web scraping library,web scraping tool,webscraping

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/chinese.md

@@ -158,6 +158,12 @@ print(json.dumps(result, indent=4))
 
 ScrapeGraphAI 的文档可以在[这里](https://docs.scrapegraphai.com/introduction)找到。
 
+## 🏆 赞助商
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+ScrapeGraphAI 用户优惠码：`SCRAPEGRAPH35`（移动和住宅代理 35% 折扣），`SCRAPEGRAPH40`（ISP / 静态代理 40% 折扣）。
+
 ## 🤝 贡献
 
 欢迎贡献并加入我们的 Discord 服务器与我们讨论改进和提出建议！

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/french.md

@@ -160,6 +160,12 @@ N'oubliez pas d'installer [Ollama](https://ollama.com/) et de télécharger les
 
 La documentation pour ScrapeGraphAI se trouve [ici](https://docs.scrapegraphai.com/introduction).
 
+## 🏆 Sponsors
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Codes pour les utilisateurs de ScrapeGraphAI : `SCRAPEGRAPH35` (35 % sur les proxies mobile et résidentiels), `SCRAPEGRAPH40` (40 % sur les proxies ISP / statiques).
+
 ## 🤝 Contribuer
 
 N'hésitez pas à contribuer et à rejoindre notre serveur Discord pour discuter avec nous des améliorations et nous donner des suggestions !

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/german.md

@@ -160,6 +160,12 @@ Denken Sie daran, [Ollama](https://ollama.com/) installiert zu haben und die Mod
 
 Die Dokumentation zu ScrapeGraphAI finden Sie [hier](https://docs.scrapegraphai.com/introduction).
 
+## 🏆 Sponsoren
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Codes für ScrapeGraphAI-Nutzer: `SCRAPEGRAPH35` (35 % auf Mobile- und Residential-Proxys), `SCRAPEGRAPH40` (40 % auf ISP- / Static-Proxys).
+
 ## 🤝 Mitwirken
 
 Fühlen Sie sich frei, beizutragen und treten Sie unserem Discord-Server bei, um mit uns über Verbesserungen zu diskutieren und uns Vorschläge zu machen!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/italian.md

@@ -162,6 +162,12 @@ Ricordati di avere [Ollama](https://ollama.com/) installato e di scaricare i mod
 La documentazione di ScrapeGraphAI è disponibile [qui](https://scrapegraph-ai.readthedocs.io/en/latest/).
 Consulta anche il Docusaurus [qui](https://docs-oss.scrapegraphai.com/).
 
+## 🏆 Sponsor
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Codici per gli utenti ScrapeGraphAI: `SCRAPEGRAPH35` (35% su proxy mobile e residenziali), `SCRAPEGRAPH40` (40% su proxy ISP / statici).
+
 ## 🤝 Vuoi contribuire?
 
 Sentiti libero di contribuire e unisciti al nostro server Discord per discutere con noi su cosa migliorare e darci suggerimenti!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/japanese.md

@@ -158,6 +158,12 @@ print(json.dumps(result, indent=4))
 
 ScrapeGraphAIのドキュメントは[こちら](https://docs.scrapegraphai.com/introduction)で見ることができます。
 
+## 🏆 スポンサー
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+ScrapeGraphAI ユーザー向けコード：`SCRAPEGRAPH35`（モバイル / 住宅用プロキシ 35% オフ）、`SCRAPEGRAPH40`（ISP / 静的プロキシ 40% オフ）。
+
 ## 🤝 貢献
 
 貢献を歓迎し、Discordサーバーで改善や提案について話し合います！

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/korean.md

@@ -158,6 +158,12 @@ OpenAI, Groq, Azure, Gemini와 같은 API를 통해 다양한 LLM을 사용할
 
 ScrapeGraphAI 관련 문서는 [여기](https://docs.scrapegraphai.com/introduction)에서 확인하실 수 있습니다.
 
+## 🏆 스폰서
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+ScrapeGraphAI 사용자 코드: `SCRAPEGRAPH35` (모바일 및 주거용 프록시 35% 할인), `SCRAPEGRAPH40` (ISP / 고정 프록시 40% 할인).
+
 ## 🤝 기여
 
 자유롭게 기여하고 Discord 서버에 참여하여 개선 사항을 논의하고 제안해 주세요!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/portuguese.md

@@ -158,6 +158,12 @@ Lembre-se de ter o [Ollama](https://ollama.com/) instalado e baixar os modelos u
 
 A documentação do ScrapeGraphAI pode ser encontrada [aqui](https://docs.scrapegraphai.com/introduction).
 
+## 🏆 Patrocinadores
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Códigos para usuários do ScrapeGraphAI: `SCRAPEGRAPH35` (35% em proxies móveis e residenciais), `SCRAPEGRAPH40` (40% em proxies ISP / estáticos).
+
 ## 🤝 Contribuindo
 
 Sinta-se à vontade para contribuir e junte-se ao nosso servidor Discord para discutir melhorias e nos dar sugestões!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/russian.md

@@ -158,6 +158,12 @@ print(json.dumps(result, indent=4))
 
 Документация для ScrapeGraphAI доступна [здесь](https://docs.scrapegraphai.com/introduction).
 
+## 🏆 Спонсоры
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Коды для пользователей ScrapeGraphAI: `SCRAPEGRAPH35` (скидка 35% на mobile и residential-прокси), `SCRAPEGRAPH40` (скидка 40% на ISP / static-прокси).
+
 ## 🤝 Участие
 
 Не стесняйтесь вносить свой вклад и присоединяйтесь к нашему серверу Discord, чтобы обсудить с нами улучшения и дать нам предложения!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/spanish.md

@@ -165,6 +165,12 @@ Recuerda tener [Ollama](https://ollama.com/) instalado y descargar los modelos c
 
 La documentación de ScrapeGraphAI está disponible [aquí](https://docs.scrapegraphai.com/introduction).
 
+## 🏆 Patrocinadores
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+Códigos para usuarios de ScrapeGraphAI: `SCRAPEGRAPH35` (35% en proxies móviles y residenciales), `SCRAPEGRAPH40` (40% en proxies ISP / estáticos).
+
 ## 🤝 Contribuciones
 
 ¡Siéntete libre de contribuir y únete a nuestro servidor de Discord para discutir mejoras y compartir sugerencias!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/docs/turkish.md

@@ -159,6 +159,12 @@ Yerel modelleri kullanmak istiyorsanız, [Ollama](https://ollama.com/) kurulu ol
 
 ScrapeGraphAI dokümantasyonuna [buradan](https://docs.scrapegraphai.com/introduction) ulaşabilirsiniz.
 
+## 🏆 Sponsorlar
+
+[![NodeMaven](assets/nodemaven-banner.png)](https://go.nodemaven.com/scrapegraphyai)
+
+ScrapeGraphAI kullanıcıları için kodlar: `SCRAPEGRAPH35` (mobil ve residential proxy'lerde %35 indirim), `SCRAPEGRAPH40` (ISP / static proxy'lerde %40 indirim).
+
 ## 🤝 Katkıda Bulunun
 
 Projeye katkıda bulunmaktan çekinmeyin ve geliştirmeleri tartışmak ve bize önerilerde bulunmak için Discord sunucumuza katılın!

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "scrapegraphai"
 
-version = "2.2.1"
+version = "2.2.3"
 
 description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
 authors = [

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/docloaders/chromium.py

@@ -10,6 +10,32 @@ from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
 logger = get_logger("web-loader")
 
 
+def _warn_on_error_status(response: Any, url: str) -> None:
+    """Log a warning when a navigation returned an HTTP error status.
+
+    Playwright's ``page.goto()`` returns the main-frame ``Response``, but the
+    scrapers only keep ``page.content()``. Without this check an error page
+    (404, 403, 500, a captcha wall, a login redirect) is indistinguishable
+    from the intended document once it reaches the LLM, which then produces a
+    confidently wrong answer with no signal that anything went wrong.
+
+    This mirrors the behaviour of the ``use_soup=True`` path in ``FetchNode``:
+    it warns rather than raising, so scraping error pages on purpose keeps
+    working.
+
+    Args:
+        response: The ``Response`` returned by ``page.goto()``; may be ``None``
+            (for example on a same-document navigation) or lack a usable status.
+        url: The URL that was requested, used in the warning message.
+    """
+    status = getattr(response, "status", None)
+    if isinstance(status, int) and status >= 400:
+        logger.warning(
+            f"Received HTTP {status} for {url}; the scraped content is likely "
+            "an error page, not the intended document."
+        )
+
+
 class ChromiumLoader:
     """Scrapes HTML pages from URLs using a (headless) instance of the
     Chromium web driver with proxy protection.
@@ -251,7 +277,8 @@ class ChromiumLoader:
                     context = await browser.new_context()
                     await Malenia.apply_stealth(context)
                     page = await context.new_page()
-                    await page.goto(url, wait_until="domcontentloaded")
+                    response = await page.goto(url, wait_until="domcontentloaded")
+                    _warn_on_error_status(response, url)
                     await page.wait_for_load_state(self.load_state)
 
                     previous_height = None
@@ -364,7 +391,8 @@ class ChromiumLoader:
                     )
                     await Malenia.apply_stealth(context)
                     page = await context.new_page()
-                    await page.goto(url, wait_until="domcontentloaded")
+                    response = await page.goto(url, wait_until="domcontentloaded")
+                    _warn_on_error_status(response, url)
                     await page.wait_for_load_state(self.load_state)
                     results = await page.content()
                     logger.info("Content scraped")
@@ -421,7 +449,8 @@ class ChromiumLoader:
                         storage_state=self.storage_state
                     )
                     page = await context.new_page()
-                    await page.goto(url, wait_until="networkidle")
+                    response = await page.goto(url, wait_until="networkidle")
+                    _warn_on_error_status(response, url)
                     results = await page.content()
                     logger.info("Content scraped after JavaScript rendering")
                     return results

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/code_generator_graph.py

@@ -93,7 +93,11 @@ class CodeGeneratorGraph(AbstractGraph):
         parse_node = ParseNode(
             input="doc",
             output=["parsed_doc"],
-            node_config={"llm_model": self.llm_model, "chunk_size": self.model_token},
+            node_config={
+                "llm_model": self.llm_model,
+                "chunk_size": self.model_token,
+                "schema": self.schema,
+            },
         )
 
         generate_validation_answer_node = GenerateAnswerNode(

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/document_scraper_graph.py

@@ -76,6 +76,7 @@ class DocumentScraperGraph(AbstractGraph):
                 "parse_html": False,
                 "chunk_size": self.model_token,
                 "llm_model": self.llm_model,
+                "schema": self.schema,
             },
         )
         generate_answer_node = GenerateAnswerNode(

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/omni_scraper_graph.py

@@ -83,6 +83,7 @@ class OmniScraperGraph(AbstractGraph):
                 "chunk_size": self.model_token,
                 "parse_urls": True,
                 "llm_model": self.llm_model,
+                "schema": self.schema,
             },
         )
 

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/script_creator_graph.py

@@ -82,6 +82,7 @@ class ScriptCreatorGraph(AbstractGraph):
                 "chunk_size": self.model_token,
                 "parse_html": False,
                 "llm_model": self.llm_model,
+                "schema": self.schema,
             },
         )
 

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/smart_scraper_graph.py

@@ -110,7 +110,11 @@ class SmartScraperGraph(AbstractGraph):
         parse_node = ParseNode(
             input="doc",
             output=["parsed_doc"],
-            node_config={"llm_model": self.llm_model, "chunk_size": self.model_token},
+            node_config={
+                "llm_model": self.llm_model,
+                "chunk_size": self.model_token,
+                "schema": self.schema,
+            },
         )
 
         generate_answer_node = GenerateAnswerNode(
@@ -152,6 +156,7 @@ class SmartScraperGraph(AbstractGraph):
                 node_config={
                     "llm_model": self.llm_model,
                     "chunk_size": self.model_token,
+                    "schema": self.schema,
                 },
             )
 

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/smart_scraper_lite_graph.py

@@ -74,7 +74,11 @@ class SmartScraperLiteGraph(AbstractGraph):
         parse_node = ParseNode(
             input="doc",
             output=["parsed_doc"],
-            node_config={"llm_model": self.llm_model, "chunk_size": self.model_token},
+            node_config={
+                "llm_model": self.llm_model,
+                "chunk_size": self.model_token,
+                "schema": self.schema,
+            },
         )
 
         return BaseGraph(

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/smart_scraper_multi_batch_graph.py

@@ -5,9 +5,8 @@ A scraping pipeline that uses the OpenAI Batch API for LLM calls,
 providing 50% cost savings compared to real-time API calls.
 """
 
-import asyncio
 from copy import deepcopy
-from typing import Dict, List, Optional, Type
+from typing import List, Optional, Type
 
 from pydantic import BaseModel
 
@@ -17,7 +16,6 @@ from ..nodes.merge_answers_node import MergeAnswersNode
 from ..utils.copy import safe_deepcopy
 from .abstract_graph import AbstractGraph
 from .base_graph import BaseGraph
-from .smart_scraper_graph import SmartScraperGraph
 
 
 class _FetchParseOnlyGraph(AbstractGraph):
@@ -57,6 +55,7 @@ class _FetchParseOnlyGraph(AbstractGraph):
             node_config={
                 "llm_model": self.llm_model,
                 "chunk_size": self.model_token,
+                "schema": self.schema,
             },
         )
 

{scrapegraphai-2.2.1 → scrapegraphai-2.2.3}/scrapegraphai/graphs/speech_graph.py

@@ -70,7 +70,11 @@ class SpeechGraph(AbstractGraph):
         parse_node = ParseNode(
             input="doc",
             output=["parsed_doc"],
-            node_config={"chunk_size": self.model_token, "llm_model": self.llm_model},
+            node_config={
+                "chunk_size": self.model_token,
+                "llm_model": self.llm_model,
+                "schema": self.schema,
+            },
         )
 
         generate_answer_node = GenerateAnswerNode(