scrapegraphai 2.1.2__tar.gz → 2.1.4__tar.gz

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/CHANGELOG.md

@@ -1,3 +1,46 @@
+## [2.1.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.3...v2.1.4) (2026-06-23)
+
+
+### Bug Fixes
+
+* lazy imports to prevent torchcodec FFmpeg DLL crash on Windows ([#1089](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1089)) ([#1092](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1092)) ([e5c2a42](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e5c2a4292174bae948731379c88aafb14197ec6c))
+
+
+### Docs
+
+* swap Integrations infographic for new API banner; fix CTA links ([71ab440](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71ab4406867ef02c22349c2dedc18c57d8bdfdec))
+
+## [2.1.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.2...v2.1.3) (2026-06-02)
+
+
+### Bug Fixes
+
+* **search:** restore SearchGraph by migrating DuckDuckGo backend to ddgs ([#1083](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1083)) ([2139e37](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2139e37c0addfb9039e3b985bb33801f20e4e05b)), closes [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082) [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082)
+
+## [2.2.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.2...v2.2.0-beta.3) (2026-06-01)
+
+
+### Bug Fixes
+
+* **nodes:** update outdated ChatOllama import path to langchain_ollama ([#1076](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1076)) ([e6054cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6054cbf19a7fe940899ea70b30706f676f86fa7))
+
+
+### Docs
+
+* 📚 Standardize and fix links across translated READMEs ([#1074](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1074)) ([458d36a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/458d36a6b83f4a412206cdbe9935a059e9d47f57))
+
+
+### CI
+
+* **release:** 2.1.2 [skip ci] ([210c992](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/210c99280048863774fa27053412185a5c18150d)), closes [#1076](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1076) [#1074](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1074)
+
+## [2.2.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.1...v2.2.0-beta.2) (2026-06-01)
+
+
+### Features
+
+* upgrade MiniMax default model to M3 ([#1080](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1080)) ([1b16c26](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1b16c268f4e9044c1386ccfaf67b38692b487e5a))
+
 ## [2.1.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.1...v2.1.2) (2026-06-01)
 
 

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: scrapegraphai
-Version: 2.1.2
+Version: 2.1.4
 Summary: A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines.
 Author-email: Marco Vinciguerra <mvincig11@gmail.com>, Lorenzo Padoan <lorenzo.padoan977@gmail.com>
 Keywords: ai,artificial intelligence,gpt,graph,langchain,llm,machine learning,natural language processing,nlp,openai,rag,scrapegraph,scrapegraphai,scraping,web scraping,web scraping library,web scraping tool,webscraping
@@ -11,7 +11,7 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: <4.0,>=3.12
 Requires-Dist: async-timeout>=4.0.0
 Requires-Dist: beautifulsoup4>=4.14.3
-Requires-Dist: duckduckgo-search>=8.1.1
+Requires-Dist: ddgs>=9.0.0
 Requires-Dist: free-proxy>=1.1.3
 Requires-Dist: html2text>=2025.4.15
 Requires-Dist: jsonschema>=4.25.1
@@ -81,7 +81,9 @@ Just say which information you want to extract and the library will do it for yo
 ScrapeGraphAI offers seamless integration with popular frameworks and tools to enhance your scraping capabilities. Whether you're building with Python or Node.js, using LLM frameworks, or working with no-code platforms, we've got you covered with our comprehensive integration options..
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
+  <a href="https://scrapegraphai.com">
+    <img src="https://raw.githubusercontent.com/ScrapeGraphAI/.github/main/profile/assets/api_banner.png" alt="Web data extraction at scale? Try ScrapeGraphAI cloud" style="width: 100%;">
+  </a>
 </p>
 
 You can find more informations at the following [link](https://scrapegraphai.com)
@@ -219,9 +221,9 @@ Please see the [contributing guidelines](https://github.com/ScrapeGraphAI/Scrape
 [![My Skills](https://skillicons.dev/icons?i=twitter)](https://twitter.com/scrapegraphai)
 
 ## 🔗 ScrapeGraph API & SDKs
-If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
+If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://scrapegraphai.com)
 
-[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://dashboard.scrapegraphai.com/login)
+[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://scrapegraphai.com)
 
 We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
 

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/README.md

@@ -36,7 +36,9 @@ Just say which information you want to extract and the library will do it for yo
 ScrapeGraphAI offers seamless integration with popular frameworks and tools to enhance your scraping capabilities. Whether you're building with Python or Node.js, using LLM frameworks, or working with no-code platforms, we've got you covered with our comprehensive integration options..
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
+  <a href="https://scrapegraphai.com">
+    <img src="https://raw.githubusercontent.com/ScrapeGraphAI/.github/main/profile/assets/api_banner.png" alt="Web data extraction at scale? Try ScrapeGraphAI cloud" style="width: 100%;">
+  </a>
 </p>
 
 You can find more informations at the following [link](https://scrapegraphai.com)
@@ -174,9 +176,9 @@ Please see the [contributing guidelines](https://github.com/ScrapeGraphAI/Scrape
 [![My Skills](https://skillicons.dev/icons?i=twitter)](https://twitter.com/scrapegraphai)
 
 ## 🔗 ScrapeGraph API & SDKs
-If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
+If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://scrapegraphai.com)
 
-[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://dashboard.scrapegraphai.com/login)
+[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://scrapegraphai.com)
 
 We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
 

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "scrapegraphai"
 
-version = "2.1.2"
+version = "2.1.4"
 
 description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
 authors = [
@@ -30,7 +30,7 @@ dependencies = [
     "async-timeout>=4.0.0",
     "simpleeval>=1.0.3",
     "jsonschema>=4.25.1",
-    "duckduckgo-search>=8.1.1",
+    "ddgs>=9.0.0",
     "pydantic>=2.12.5",
     "scrapegraph-py>=2.0.0",
 ]

scrapegraphai-2.1.4/scrapegraphai/docloaders/__init__.py

@@ -0,0 +1,30 @@
+"""
+This module handles document loading functionalities for the ScrapeGraphAI application.
+
+Note: ChromiumLoader and PlasmateLoader are lazy-imported to avoid triggering
+torchcodec/FFmpeg DLL loading at import time (sentence_transformers -> torchcodec chain).
+"""
+
+from .browser_base import browser_base_fetch
+from .scrape_do import scrape_do_fetch
+
+_LAZY_MODULES = {
+    "ChromiumLoader": ".chromium",
+    "PlasmateLoader": ".plasmate",
+}
+
+
+def __getattr__(name):
+    if name in _LAZY_MODULES:
+        import importlib
+        module = importlib.import_module(_LAZY_MODULES[name], __package__)
+        return getattr(module, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = [
+    "browser_base_fetch",
+    "ChromiumLoader",
+    "PlasmateLoader",
+    "scrape_do_fetch",
+]

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/scrapegraphai/docloaders/chromium.py

@@ -3,7 +3,6 @@ from typing import Any, AsyncIterator, Iterator, List, Optional, Union
 
 import aiohttp
 import async_timeout
-from langchain_community.document_loaders.base import BaseLoader
 from langchain_core.documents import Document
 
 from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
@@ -11,7 +10,7 @@ from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
 logger = get_logger("web-loader")
 
 
-class ChromiumLoader(BaseLoader):
+class ChromiumLoader:
     """Scrapes HTML pages from URLs using a (headless) instance of the
     Chromium web driver with proxy protection.
 
@@ -436,6 +435,14 @@ class ChromiumLoader(BaseLoader):
             finally:
                 await browser.close()
 
+    def load(self) -> List[Document]:
+        """Load all documents synchronously."""
+        return list(self.lazy_load())
+
+    async def aload(self) -> List[Document]:
+        """Load all documents asynchronously."""
+        return [doc async for doc in self.alazy_load()]
+
     def lazy_load(self) -> Iterator[Document]:
         """
         Lazily load text content from the provided URLs.

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/scrapegraphai/helpers/models_tokens.py

@@ -408,12 +408,8 @@ models_tokens = {
         "grok-beta": 128000,
     },
     "minimax": {
+        "MiniMax-M3": 524288,
         "MiniMax-M2.7": 204000,
         "MiniMax-M2.7-highspeed": 204000,
-        "MiniMax-M1": 1000000,
-        "MiniMax-M1-40k": 40000,
-        "MiniMax-M2": 204000,
-        "MiniMax-M2.5": 204000,
-        "MiniMax-M2.5-highspeed": 204000,
     },
 }

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/scrapegraphai/nodes/fetch_node.py

@@ -7,7 +7,6 @@ from typing import List, Optional
 import concurrent.futures
 
 import requests
-from langchain_community.document_loaders import PyPDFLoader
 from langchain_core.documents import Document
 from langchain_openai import AzureChatOpenAI, ChatOpenAI
 
@@ -182,6 +181,7 @@ class FetchNode(BaseNode):
         """
 
         if input_type == "pdf":
+            from langchain_community.document_loaders import PyPDFLoader
             loader = PyPDFLoader(source)
             # PyPDFLoader.load() can be blocking for large PDFs. Run it in a thread and
             # enforce the configured timeout if provided.

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/scrapegraphai/nodes/robots_node.py

@@ -7,7 +7,6 @@ from urllib.parse import urlparse
 
 from langchain_core.output_parsers import CommaSeparatedListOutputParser
 from langchain_core.prompts import PromptTemplate
-from langchain_community.document_loaders import AsyncChromiumLoader
 
 from ..helpers import robots_dictionary
 from ..prompts import TEMPLATE_ROBOT
@@ -90,6 +89,7 @@ class RobotsNode(BaseNode):
         else:
             parsed_url = urlparse(source)
             base_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
+            from langchain_community.document_loaders import AsyncChromiumLoader
             loader = AsyncChromiumLoader(f"{base_url}/robots.txt")
             document = loader.load()
             if "ollama" in self.llm_model.model:

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/scrapegraphai/utils/research_web.py

@@ -11,7 +11,6 @@ from typing import Dict, List, Optional, Union
 
 import requests
 from bs4 import BeautifulSoup
-from langchain_community.tools import DuckDuckGoSearchResults
 from pydantic import BaseModel, Field, validator
 
 
@@ -215,12 +214,9 @@ def search_on_web(
 
         results = []
         if config.search_engine == "duckduckgo":
-            # Create a DuckDuckGo search object with max_results
-            research = DuckDuckGoSearchResults(max_results=config.max_results)
-            # Run the search
-            res = research.run(config.query)
-            # Extract URLs using regex
-            results = re.findall(r"https?://[^\s,\]]+", res)
+            results = _search_duckduckgo(
+                config.query, config.max_results, formatted_proxy
+            )
 
         elif config.search_engine == "bing":
             results = _search_bing(
@@ -247,6 +243,45 @@ def search_on_web(
         raise SearchConfigError(f"Invalid search configuration: {str(e)}")
 
 
+def _search_duckduckgo(
+    query: str, max_results: int, proxy: Optional[str] = None
+) -> List[str]:
+    """
+    Helper function for DuckDuckGo search using the ``ddgs`` package.
+
+    The ``duckduckgo-search`` package was renamed to ``ddgs``; recent
+    ``langchain-community`` releases import ``from ddgs import DDGS``, which
+    silently broke the previous langchain-based implementation. This calls
+    ``ddgs`` directly so results no longer depend on parsing a formatted string.
+
+    Args:
+        query (str): Search query
+        max_results (int): Maximum number of results to return
+        proxy (str, optional): Proxy configuration
+
+    Returns:
+        List[str]: List of URLs from search results
+    """
+    try:
+        from ddgs import DDGS
+    except ImportError as e:
+        raise ImportError(
+            "Could not import the 'ddgs' package required for DuckDuckGo "
+            "search. Please install it with `pip install -U ddgs`."
+        ) from e
+
+    try:
+        with DDGS(proxy=proxy) as ddgs:
+            results = [
+                result["href"]
+                for result in ddgs.text(query, max_results=max_results)
+                if result.get("href")
+            ]
+        return results
+    except Exception as e:
+        raise SearchRequestError(f"DuckDuckGo search failed: {str(e)}")
+
+
 def _search_bing(
     query: str, max_results: int, timeout: int, proxy: Optional[str] = None
 ) -> List[str]:

{scrapegraphai-2.1.2 → scrapegraphai-2.1.4}/tests/conftest.py

@@ -14,9 +14,24 @@ from pathlib import Path
 from typing import Any, Dict
 from unittest.mock import Mock
 
+import sys
+import types
+
 import pytest
 from dotenv import load_dotenv
 
+# Mock torchcodec to prevent FFmpeg DLL crashes at import time.
+# sentence_transformers -> torchcodec -> FFmpeg native DLLs can't load on some systems.
+_tc = types.ModuleType("torchcodec")
+_tc.__version__ = "0.0.0"
+_tc.__file__ = "<mock>"
+_tc.__spec__ = types.ModuleType("spec")
+_tc.__spec__.name = "torchcodec"
+_tc.__spec__.loader = None
+_tc.__spec__.submodule_search_locations = []
+if "torchcodec" not in sys.modules:
+    sys.modules["torchcodec"] = _tc
+
 # Load environment variables
 load_dotenv()
 

scrapegraphai-2.1.4/tests/test_minimax_models.py

@@ -0,0 +1,62 @@
+"""Tests for MiniMax model configuration."""
+
+import importlib.util
+import os
+import sys
+
+import pytest
+
+
+@pytest.fixture(scope="module")
+def models_tokens():
+    """Import models_tokens directly to avoid triggering the full package init."""
+    spec = importlib.util.spec_from_file_location(
+        "models_tokens",
+        os.path.join(
+            os.path.dirname(__file__),
+            "..",
+            "scrapegraphai",
+            "helpers",
+            "models_tokens.py",
+        ),
+    )
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module.models_tokens
+
+
+def test_minimax_m3_in_model_list(models_tokens):
+    """MiniMax-M3 should be in the model list."""
+    minimax_models = models_tokens["minimax"]
+    assert "MiniMax-M3" in minimax_models
+
+
+def test_minimax_m3_listed_first(models_tokens):
+    """MiniMax-M3 should be the first (default) model in the minimax dict."""
+    minimax_models = list(models_tokens["minimax"].keys())
+    assert minimax_models[0] == "MiniMax-M3"
+
+
+def test_minimax_m27_still_available(models_tokens):
+    """MiniMax-M2.7 and its highspeed variant should remain as legacy options."""
+    minimax_models = models_tokens["minimax"]
+    assert "MiniMax-M2.7" in minimax_models
+    assert "MiniMax-M2.7-highspeed" in minimax_models
+
+
+def test_minimax_deprecated_models_removed(models_tokens):
+    """Older deprecated MiniMax models should be removed from the list."""
+    minimax_models = models_tokens["minimax"]
+    assert "MiniMax-M2.5" not in minimax_models
+    assert "MiniMax-M2.5-highspeed" not in minimax_models
+    assert "MiniMax-M2" not in minimax_models
+    assert "MiniMax-M1" not in minimax_models
+    assert "MiniMax-M1-40k" not in minimax_models
+
+
+def test_minimax_token_limits(models_tokens):
+    """MiniMax model token limits should match upstream documentation."""
+    minimax_models = models_tokens["minimax"]
+    assert minimax_models["MiniMax-M3"] == 524288
+    assert minimax_models["MiniMax-M2.7"] == 204000
+    assert minimax_models["MiniMax-M2.7-highspeed"] == 204000