scrapegraphai 2.2.0b4__tar.gz → 2.2.0b6__tar.gz

scrapegraphai-2.2.0b6/.github/workflows/test-suite.yml

@@ -0,0 +1,51 @@
+name: Test Suite
+
+on:
+  push:
+    branches: [main, pre/beta, dev]
+  pull_request:
+    branches: [main, pre/beta]
+  workflow_dispatch:
+
+jobs:
+  unit-tests:
+    name: Unit Tests
+    runs-on: ubuntu-latest
+    # Hard cap so a hanging test can never burn the full 6h runner budget again.
+    timeout-minutes: 15
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Install dependencies
+        run: uv sync
+
+      # Run only the deterministic, mock-based unit suites. The provider graph
+      # tests (tests/graphs/*), integration tests (tests/integration/*) and the
+      # browser/network suites (e.g. tests/test_chromium.py) require real API
+      # keys or live network/browsers, so in CI they either fail or hang
+      # indefinitely. They should be re-added here once mocked or gated behind
+      # markers + secrets.
+      - name: Run unit tests
+        run: >
+          uv run pytest
+          tests/test_atlascloud_model.py
+          tests/test_batch_api.py
+          tests/test_csv_scraper_multi_graph.py
+          tests/test_depth_search_graph.py
+          tests/test_json_scraper_graph.py
+          tests/test_minimax_models.py
+          tests/test_scrape_do.py
+          tests/test_search_graph.py
+          tests/utils/convert_to_md_test.py
+          tests/utils/output_parser_test.py
+          tests/utils/parse_state_keys_test.py

scrapegraphai-2.2.0b6/AGENTS.md

@@ -0,0 +1,136 @@
+# AGENTS.md
+
+Instructions for AI coding agents (Claude Code, Codex, Cursor, Copilot agents, …)
+working on **ScrapeGraphAI**. Human contributors should read
+[CONTRIBUTING.md](CONTRIBUTING.md); everything here is in addition to it.
+
+---
+
+## 1. Golden rule: everything goes to `pre/beta`
+
+**`main` is never written to directly. All work is based on and merged into `pre/beta`.**
+
+`pre/beta` is the prerelease branch: pushes to it publish a `beta` prerelease via
+semantic-release (see `.releaserc.yml`). `main` only receives releases when a
+maintainer promotes `pre/beta`.
+
+```bash
+# 1. always start from an up-to-date pre/beta
+git fetch origin
+git checkout -b feat/my-change origin/pre/beta
+
+# 2. commit your work
+git add <only the files you touched>
+git commit -m "feat(nodes): add X"
+
+# 3. push and open the PR against pre/beta
+git push -u origin feat/my-change
+gh pr create --base pre/beta --title "feat(nodes): add X" --body "..."
+```
+
+Checklist before you commit:
+
+- [ ] The branch is based on `origin/pre/beta` (`git merge-base --is-ancestor origin/pre/beta HEAD`).
+- [ ] The PR base is `pre/beta`, **not** `main`.
+- [ ] No commits directly on `main` or `pre/beta`, no force-push to either.
+- [ ] One logical change per branch/PR.
+
+If a task genuinely requires targeting `main` (e.g. a hotfix on a released
+version), stop and ask a maintainer first.
+
+## 2. Environment setup
+
+Python `>=3.12`, dependencies managed with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv sync                     # create the venv and install deps
+uv run pre-commit install   # install the git hooks
+```
+
+Never hand-edit `uv.lock`; regenerate it with `uv lock` / `uv sync` and commit
+the result only when you actually changed dependencies in `pyproject.toml`.
+
+## 3. Checks to run before pushing
+
+```bash
+make lint         # ruff + black --check + isort --check-only
+make type-check   # mypy (strict)
+make test         # pytest with coverage
+make pre-commit   # run all hooks on all files
+```
+
+Run at least `make lint` and the tests covering what you touched. Report the
+real result: if something fails or you skipped a step, say so in the PR
+description instead of implying a clean run.
+
+Style: PEP 8 + Google Python docstrings, `black` formatting, line length 88.
+Match the conventions of the surrounding file rather than introducing new ones.
+
+## 4. Commit messages
+
+Commits are parsed by semantic-release (Conventional Commits, `conventionalcommits`
+preset), so the message decides the next version number. Use:
+
+```
+feat:     ✨ new feature          -> minor bump
+fix:      🐛 bug fix              -> patch bump
+docs:     📚 documentation
+style:    💅 formatting only
+refactor: ♻️  no behaviour change
+perf:     ⚡ performance
+test:     🧪 tests
+build:    📦 build system / deps
+ci:       🤖 CI configuration
+chore:    🧹 everything else
+```
+
+Format: `type(optional-scope): imperative summary`, optional body, and
+`BREAKING CHANGE:` in the footer for incompatible changes. Reference issues with
+`Fixes #123`.
+
+## 5. Files agents must not touch
+
+- `CHANGELOG.md` and the `version` field in `pyproject.toml` — owned by
+  semantic-release; editing them by hand breaks releases.
+- Git tags and release notes on GitHub.
+- `.github/workflows/*` — only when the task is explicitly about CI.
+- Anything under `htmlcov/`, `coverage.xml`, `.pytest_cache/`, `__pycache__/`:
+  build artifacts, never commit them.
+
+Also: never commit secrets. API keys go in a local `.env` (git-ignored) and are
+read via `os.getenv`; examples and tests must use placeholders such as
+`OPENAI_APIKEY` from the environment.
+
+## 6. Repository layout
+
+```
+scrapegraphai/
+├── graphs/        # pipelines (SmartScraperGraph, SearchGraph, …)
+├── nodes/         # single graph steps (FetchNode, ParseNode, GenerateAnswerNode, …)
+├── models/        # LLM wrappers and token/model metadata
+├── docloaders/    # loaders (ChromiumLoader, …)
+├── prompts/       # prompt templates
+├── helpers/       # shared constants and schemas
+├── integrations/  # third-party / managed-API integrations
+└── utils/         # utilities (html cleanup, tokenization, …)
+examples/          # runnable usage examples, one folder per graph
+tests/             # pytest suite, mirrors the package layout
+docs/              # documentation sources
+```
+
+When adding a node or graph, register it in the corresponding `__init__.py` and
+add a test under `tests/` next to the existing ones for that layer. New
+user-facing features need an entry in `examples/` and, when they change public
+behaviour, a docs update.
+
+## 7. Working style expected from agents
+
+- Prefer small, reviewable diffs; do not reformat or "clean up" untouched files.
+- Do not add dependencies unless the task requires it — say why in the PR.
+- Write all commits, PR titles/bodies, issue comments, code comments and
+  docstrings **in English**.
+- Do not delete or rewrite existing tests to make a change pass.
+- If a test is already failing on `pre/beta`, mention it rather than silently
+  fixing unrelated things in the same PR.
+- Never commit other people's in-progress work: check `git status` and stage
+  only the files belonging to your change.

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/CHANGELOG.md

@@ -1,3 +1,113 @@
+## [2.2.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.5...v2.2.0-beta.6) (2026-08-19)
+
+
+### Bug Fixes
+
+* **graph:** expose when the 8192 token fallback was used ([470da9d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/470da9d51d6220a8020ee8a1d91b618d3d622042)), closes [#1121](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1121)
+* lazy imports to prevent torchcodec FFmpeg DLL crash on Windows ([#1089](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1089)) ([#1092](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1092)) ([e5c2a42](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e5c2a4292174bae948731379c88aafb14197ec6c))
+* pop model_tokens so it is not forwarded to the model client ([#1100](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1100)) ([d2b970c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d2b970ca77da64b6b6b91fa2353f2e41b9cfe105))
+* **search:** restore SearchGraph by migrating DuckDuckGo backend to ddgs ([#1083](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1083)) ([2139e37](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2139e37c0addfb9039e3b985bb33801f20e4e05b)), closes [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082) [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082)
+* update MiniMax model metadata and endpoints ([#1103](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1103)) ([e5f8f2b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e5f8f2bf008cd7ce7f3d5980bf6f98e6153b2264))
+
+
+### Docs
+
+* add AGENTS.md with contribution rules for AI agents ([3b2f986](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b2f986612bbc70879a0337197a1064cf5875245))
+* **readme:** add Open Source vs Managed API comparison ([#1091](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1091)) ([ef3523b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ef3523b1e1c62052b5c0e6b14cebe7933441a7e5))
+* swap Integrations infographic for new API banner; fix CTA links ([71ab440](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71ab4406867ef02c22349c2dedc18c57d8bdfdec))
+
+
+### CI
+
+* **release:** 2.1.3 [skip ci] ([cfb815f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cfb815fb0bba3dfc8262a24a7ddab28e22e13893)), closes [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082) [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082)
+* **release:** 2.1.4 [skip ci] ([7fc9c57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7fc9c57a0bb861d829f38ed672e35c5ba0e6c79c)), closes [#1089](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1089) [#1092](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1092)
+* **release:** 2.1.5 [skip ci] ([c9e0bd0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c9e0bd036a1f5def8ac92759cda90ba52e3b35e5)), closes [#1100](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1100) [#1091](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1091) [#1095](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1095)
+* **release:** 2.1.6 [skip ci] ([27d9d28](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27d9d289f964844a28006f6d7d1518e293fd49ff)), closes [#1103](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1103)
+* **release:** 2.1.7 [skip ci] ([ca112db](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ca112db4d8b1bd34961e9f1b7e3adf0c7d600c01)), closes [#1121](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1121)
+* run only deterministic unit suites in Test Suite workflow ([#1095](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1095)) ([037a42e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/037a42ed73354c8b3eabd9dcc28236152602ef9e))
+* run the two new deterministic unit suites ([23e3d06](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/23e3d064e3850de222bcfc13907695278ca3bba9)), closes [#1104](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1104) [#1085](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1085)
+
+## [2.1.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.6...v2.1.7) (2026-08-19)
+
+
+### Bug Fixes
+
+* **graph:** expose when the 8192 token fallback was used ([470da9d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/470da9d51d6220a8020ee8a1d91b618d3d622042)), closes [#1121](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1121)
+
+## [2.1.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.5...v2.1.6) (2026-07-20)
+
+
+### Bug Fixes
+
+* update MiniMax model metadata and endpoints ([#1103](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1103)) ([e5f8f2b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e5f8f2bf008cd7ce7f3d5980bf6f98e6153b2264))
+
+## [2.1.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.4...v2.1.5) (2026-07-08)
+
+
+### Bug Fixes
+
+* pop model_tokens so it is not forwarded to the model client ([#1100](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1100)) ([d2b970c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d2b970ca77da64b6b6b91fa2353f2e41b9cfe105))
+
+
+### Docs
+
+* **readme:** add Open Source vs Managed API comparison ([#1091](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1091)) ([ef3523b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ef3523b1e1c62052b5c0e6b14cebe7933441a7e5))
+
+
+### CI
+
+* run only deterministic unit suites in Test Suite workflow ([#1095](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1095)) ([037a42e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/037a42ed73354c8b3eabd9dcc28236152602ef9e))
+
+## [2.1.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.3...v2.1.4) (2026-06-23)
+
+
+### Bug Fixes
+
+* lazy imports to prevent torchcodec FFmpeg DLL crash on Windows ([#1089](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1089)) ([#1092](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1092)) ([e5c2a42](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e5c2a4292174bae948731379c88aafb14197ec6c))
+
+
+### Docs
+
+* swap Integrations infographic for new API banner; fix CTA links ([71ab440](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71ab4406867ef02c22349c2dedc18c57d8bdfdec))
+
+## [2.1.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.1.2...v2.1.3) (2026-06-02)
+
+
+### Bug Fixes
+
+* **search:** restore SearchGraph by migrating DuckDuckGo backend to ddgs ([#1083](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1083)) ([2139e37](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2139e37c0addfb9039e3b985bb33801f20e4e05b)), closes [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082) [#1082](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1082)
+
+## [2.2.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.2...v2.2.0-beta.3) (2026-06-01)
+
+
+### Bug Fixes
+
+* **nodes:** update outdated ChatOllama import path to langchain_ollama ([#1076](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1076)) ([e6054cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6054cbf19a7fe940899ea70b30706f676f86fa7))
+
+
+### Docs
+
+* 📚 Standardize and fix links across translated READMEs ([#1074](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1074)) ([458d36a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/458d36a6b83f4a412206cdbe9935a059e9d47f57))
+
+
+### CI
+
+* **release:** 2.1.2 [skip ci] ([210c992](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/210c99280048863774fa27053412185a5c18150d)), closes [#1076](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1076) [#1074](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1074)
+
+## [2.2.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.1...v2.2.0-beta.2) (2026-06-01)
+
+
+### Features
+
+* upgrade MiniMax default model to M3 ([#1080](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1080)) ([1b16c26](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1b16c268f4e9044c1386ccfaf67b38692b487e5a))
+
+## [2.2.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.4...v2.2.0-beta.5) (2026-06-23)
+
+
+### Bug Fixes
+
+* lazy imports to prevent torchcodec FFmpeg DLL crash on Windows ([#1089](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1089)) ([2f55377](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2f55377010fb16c81655aa1c8d08b88daa880797))
+
 ## [2.2.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v2.2.0-beta.3...v2.2.0-beta.4) (2026-06-11)
 
 
@@ -1565,7 +1675,6 @@ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 
 * implement ScrapeGraph class for only web scraping automation ([612c644](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/612c644623fa6f4fe77a64a5f1a6a4d6cd5f4254))
 * Implement SmartScraperMultiParseMergeFirstGraph class that scrapes a list of URLs and merge the content first and finally generates answers to a given prompt. ([3e3e1b2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e3e1b2f3ae8ed803d03b3b44b199e139baa68d4))
-=======
 ## [1.26.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.6...v1.26.7) (2024-10-19)
 
 
@@ -3639,7 +3748,6 @@ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 * **release:** 1.6.1 [skip ci] ([44fbd71](https://github.com/VinciGit00/Scrapegraph-ai/commit/44fbd71742a57a4b10f22ed33781bb67aa77e58d))
 
 ## [1.6.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0...v1.6.1) (2024-06-15)
-=======
 
 
 ### Bug Fixes

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: scrapegraphai
-Version: 2.2.0b4
+Version: 2.2.0b6
 Summary: A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines.
 Author-email: Marco Vinciguerra <mvincig11@gmail.com>, Lorenzo Padoan <lorenzo.padoan977@gmail.com>
 Keywords: ai,artificial intelligence,gpt,graph,langchain,llm,machine learning,natural language processing,nlp,openai,rag,scrapegraph,scrapegraphai,scraping,web scraping,web scraping library,web scraping tool,webscraping
@@ -11,7 +11,7 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: <4.0,>=3.12
 Requires-Dist: async-timeout>=4.0.0
 Requires-Dist: beautifulsoup4>=4.14.3
-Requires-Dist: duckduckgo-search>=8.1.1
+Requires-Dist: ddgs>=9.0.0
 Requires-Dist: free-proxy>=1.1.3
 Requires-Dist: html2text>=2025.4.15
 Requires-Dist: jsonschema>=4.25.1
@@ -81,7 +81,9 @@ Just say which information you want to extract and the library will do it for yo
 ScrapeGraphAI offers seamless integration with popular frameworks and tools to enhance your scraping capabilities. Whether you're building with Python or Node.js, using LLM frameworks, or working with no-code platforms, we've got you covered with our comprehensive integration options..
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
+  <a href="https://scrapegraphai.com">
+    <img src="https://raw.githubusercontent.com/ScrapeGraphAI/.github/main/profile/assets/api_banner.png" alt="Web data extraction at scale? Try ScrapeGraphAI cloud" style="width: 100%;">
+  </a>
 </p>
 
 You can find more informations at the following [link](https://scrapegraphai.com)
@@ -93,6 +95,7 @@ You can find more informations at the following [link](https://scrapegraphai.com
 - **Low-code Frameworks**: [Pipedream](https://pipedream.com/apps/scrapegraphai), [Bubble](https://bubble.io/plugin/scrapegraphai-1745408893195x213542371433906180), [Zapier](https://zapier.com/apps/scrapegraphai/integrations), [n8n](http://localhost:5001/dashboard), [Dify](https://dify.ai), [Toolhouse](https://app.toolhouse.ai/mcp-servers/scrapegraph_smartscraper)
 - **MCP server**:  [Link](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp)
 
+
 ## 🚀 Quick install
 
 The reference page for Scrapegraph-ai is available on the official page of PyPI: [pypi](https://pypi.org/project/scrapegraphai/).
@@ -197,7 +200,7 @@ There are other pipelines that can be used to extract information from multiple
 
 For each of these graphs there is the multi version. It allows to make calls of the LLM in parallel.
 
-It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure**, **Gemini**, **MiniMax** and more, or local models using **Ollama**.
+It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure**, **Gemini**, **[MiniMax](docs/minimax.md)** and more, or local models using **Ollama**.
 
 Remember to have [Ollama](https://ollama.com/) installed and download the models using the **ollama pull** command, if you want to use local models.
 
@@ -207,6 +210,32 @@ Remember to have [Ollama](https://ollama.com/) installed and download the models
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing)
 
 The documentation for ScrapeGraphAI can be found [here](https://docs.scrapegraphai.com/introduction).
+## 🆚 Open Source vs Managed API
+
+ScrapeGraphAI comes in two flavours: **this open-source library**, which you run yourself, and the **managed cloud API** (used via the [Python](https://github.com/ScrapeGraphAI/scrapegraph-py) and [JS/TS](https://github.com/ScrapeGraphAI/scrapegraph-js) SDKs). This table explains the difference so you can pick the right one.
+
+| | Open Source (`scrapegraphai`) | Managed API (`scrapegraph-py` / `scrapegraph-js`) |
+|---|---|---|
+| **What it is** | A Python library you run yourself | A hosted cloud service you call via SDK |
+| **Where it runs** | Your own infrastructure (self-hosted) | ScrapeGraphAI cloud |
+| **LLM** | Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) | Managed for you |
+| **Browser / JS rendering** | You configure it (Playwright) | Managed (stealth, `auto`/`fast`/`js` modes) |
+| **Proxies & anti-bot** | Your responsibility | Included |
+| **Scaling & maintenance** | Your responsibility | Fully managed |
+| **Cost model** | LLM tokens + your own infra | Pay-as-you-go credits |
+| **Auth** | Your own LLM keys | `SGAI_API_KEY` |
+| **Capabilities** | Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) | Scrape, Extract, Search, Crawl, Monitor, History |
+| **Setup effort** | More configuration | Minimal — API key + one call |
+| **License** | MIT | SDK is MIT; the API service is paid |
+
+**Choose the open-source library** if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself.
+
+**Choose the managed API** if you want zero infrastructure, managed JS rendering & anti-bot, built-in **Crawl** and scheduled **Monitor** jobs, and the fastest path to production — billed per credit.
+
+- Open-source library: https://github.com/ScrapeGraphAI/Scrapegraph-ai
+- Python SDK: https://github.com/ScrapeGraphAI/scrapegraph-py
+- JS/TS SDK: https://github.com/ScrapeGraphAI/scrapegraph-js
+- API docs: https://docs.scrapegraphai.com/introduction
 
 ## 🤝 Contributing
 
@@ -219,9 +248,9 @@ Please see the [contributing guidelines](https://github.com/ScrapeGraphAI/Scrape
 [![My Skills](https://skillicons.dev/icons?i=twitter)](https://twitter.com/scrapegraphai)
 
 ## 🔗 ScrapeGraph API & SDKs
-If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
+If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://scrapegraphai.com)
 
-[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://dashboard.scrapegraphai.com/login)
+[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://scrapegraphai.com)
 
 We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
 

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/README.md

@@ -36,7 +36,9 @@ Just say which information you want to extract and the library will do it for yo
 ScrapeGraphAI offers seamless integration with popular frameworks and tools to enhance your scraping capabilities. Whether you're building with Python or Node.js, using LLM frameworks, or working with no-code platforms, we've got you covered with our comprehensive integration options..
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
+  <a href="https://scrapegraphai.com">
+    <img src="https://raw.githubusercontent.com/ScrapeGraphAI/.github/main/profile/assets/api_banner.png" alt="Web data extraction at scale? Try ScrapeGraphAI cloud" style="width: 100%;">
+  </a>
 </p>
 
 You can find more informations at the following [link](https://scrapegraphai.com)
@@ -48,6 +50,7 @@ You can find more informations at the following [link](https://scrapegraphai.com
 - **Low-code Frameworks**: [Pipedream](https://pipedream.com/apps/scrapegraphai), [Bubble](https://bubble.io/plugin/scrapegraphai-1745408893195x213542371433906180), [Zapier](https://zapier.com/apps/scrapegraphai/integrations), [n8n](http://localhost:5001/dashboard), [Dify](https://dify.ai), [Toolhouse](https://app.toolhouse.ai/mcp-servers/scrapegraph_smartscraper)
 - **MCP server**:  [Link](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp)
 
+
 ## 🚀 Quick install
 
 The reference page for Scrapegraph-ai is available on the official page of PyPI: [pypi](https://pypi.org/project/scrapegraphai/).
@@ -152,7 +155,7 @@ There are other pipelines that can be used to extract information from multiple
 
 For each of these graphs there is the multi version. It allows to make calls of the LLM in parallel.
 
-It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure**, **Gemini**, **MiniMax** and more, or local models using **Ollama**.
+It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure**, **Gemini**, **[MiniMax](docs/minimax.md)** and more, or local models using **Ollama**.
 
 Remember to have [Ollama](https://ollama.com/) installed and download the models using the **ollama pull** command, if you want to use local models.
 
@@ -162,6 +165,32 @@ Remember to have [Ollama](https://ollama.com/) installed and download the models
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing)
 
 The documentation for ScrapeGraphAI can be found [here](https://docs.scrapegraphai.com/introduction).
+## 🆚 Open Source vs Managed API
+
+ScrapeGraphAI comes in two flavours: **this open-source library**, which you run yourself, and the **managed cloud API** (used via the [Python](https://github.com/ScrapeGraphAI/scrapegraph-py) and [JS/TS](https://github.com/ScrapeGraphAI/scrapegraph-js) SDKs). This table explains the difference so you can pick the right one.
+
+| | Open Source (`scrapegraphai`) | Managed API (`scrapegraph-py` / `scrapegraph-js`) |
+|---|---|---|
+| **What it is** | A Python library you run yourself | A hosted cloud service you call via SDK |
+| **Where it runs** | Your own infrastructure (self-hosted) | ScrapeGraphAI cloud |
+| **LLM** | Bring your own (OpenAI, Groq, Gemini, Azure, local via Ollama) | Managed for you |
+| **Browser / JS rendering** | You configure it (Playwright) | Managed (stealth, `auto`/`fast`/`js` modes) |
+| **Proxies & anti-bot** | Your responsibility | Included |
+| **Scaling & maintenance** | Your responsibility | Fully managed |
+| **Cost model** | LLM tokens + your own infra | Pay-as-you-go credits |
+| **Auth** | Your own LLM keys | `SGAI_API_KEY` |
+| **Capabilities** | Graph pipelines (SmartScraper, Search, Speech, ScriptCreator…) | Scrape, Extract, Search, Crawl, Monitor, History |
+| **Setup effort** | More configuration | Minimal — API key + one call |
+| **License** | MIT | SDK is MIT; the API service is paid |
+
+**Choose the open-source library** if you want full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning — and you're happy to manage browsers, proxies and scaling yourself.
+
+**Choose the managed API** if you want zero infrastructure, managed JS rendering & anti-bot, built-in **Crawl** and scheduled **Monitor** jobs, and the fastest path to production — billed per credit.
+
+- Open-source library: https://github.com/ScrapeGraphAI/Scrapegraph-ai
+- Python SDK: https://github.com/ScrapeGraphAI/scrapegraph-py
+- JS/TS SDK: https://github.com/ScrapeGraphAI/scrapegraph-js
+- API docs: https://docs.scrapegraphai.com/introduction
 
 ## 🤝 Contributing
 
@@ -174,9 +203,9 @@ Please see the [contributing guidelines](https://github.com/ScrapeGraphAI/Scrape
 [![My Skills](https://skillicons.dev/icons?i=twitter)](https://twitter.com/scrapegraphai)
 
 ## 🔗 ScrapeGraph API & SDKs
-If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
+If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://scrapegraphai.com)
 
-[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://dashboard.scrapegraphai.com/login)
+[![API Banner](https://raw.githubusercontent.com/ScrapeGraphAI/Scrapegraph-ai/main/docs/assets/api_banner.png)](https://scrapegraphai.com)
 
 We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
 

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "scrapegraphai"
 
-version = "2.2.0b4"
+version = "2.2.0b6"
 
 description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
 authors = [
@@ -30,7 +30,7 @@ dependencies = [
     "async-timeout>=4.0.0",
     "simpleeval>=1.0.3",
     "jsonschema>=4.25.1",
-    "duckduckgo-search>=8.1.1",
+    "ddgs>=9.0.0",
     "pydantic>=2.12.5",
     "scrapegraph-py>=2.0.0",
 ]

scrapegraphai-2.2.0b6/scrapegraphai/docloaders/__init__.py

@@ -0,0 +1,30 @@
+"""
+This module handles document loading functionalities for the ScrapeGraphAI application.
+
+Note: ChromiumLoader and PlasmateLoader are lazy-imported to avoid triggering
+torchcodec/FFmpeg DLL loading at import time (sentence_transformers -> torchcodec chain).
+"""
+
+from .browser_base import browser_base_fetch
+from .scrape_do import scrape_do_fetch
+
+_LAZY_MODULES = {
+    "ChromiumLoader": ".chromium",
+    "PlasmateLoader": ".plasmate",
+}
+
+
+def __getattr__(name):
+    if name in _LAZY_MODULES:
+        import importlib
+        module = importlib.import_module(_LAZY_MODULES[name], __package__)
+        return getattr(module, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+__all__ = [
+    "browser_base_fetch",
+    "ChromiumLoader",
+    "PlasmateLoader",
+    "scrape_do_fetch",
+]

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/scrapegraphai/docloaders/chromium.py

@@ -3,7 +3,6 @@ from typing import Any, AsyncIterator, Iterator, List, Optional, Union
 
 import aiohttp
 import async_timeout
-from langchain_community.document_loaders.base import BaseLoader
 from langchain_core.documents import Document
 
 from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
@@ -11,7 +10,7 @@ from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
 logger = get_logger("web-loader")
 
 
-class ChromiumLoader(BaseLoader):
+class ChromiumLoader:
     """Scrapes HTML pages from URLs using a (headless) instance of the
     Chromium web driver with proxy protection.
 
@@ -436,6 +435,14 @@ class ChromiumLoader(BaseLoader):
             finally:
                 await browser.close()
 
+    def load(self) -> List[Document]:
+        """Load all documents synchronously."""
+        return list(self.lazy_load())
+
+    async def aload(self) -> List[Document]:
+        """Load all documents asynchronously."""
+        return [doc async for doc in self.alazy_load()]
+
     def lazy_load(self) -> Iterator[Document]:
         """
         Lazily load text content from the provided URLs.

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/scrapegraphai/graphs/abstract_graph.py

@@ -64,6 +64,9 @@ class AbstractGraph(ABC):
         self.source = source
         self.config = config
         self.schema = schema
+        # Set to True when no token limit is known for the configured model and
+        # the 8192 fallback is used; see _create_llm.
+        self.model_tokens_defaulted = False
         self.llm_model = self._create_llm(config["llm"])
         self.verbose = False if config is None else config.get("verbose", False)
         self.headless = True if self.config is None else config.get("headless", True)
@@ -218,9 +221,18 @@ class AbstractGraph(ABC):
                     llm_params["model"],
                 )
                 self.model_token = 8192
+                # A silent 8192 window truncates long pages and changes the
+                # answer without failing. The log line alone is not reachable
+                # from the returned object, so callers that batch runs (or
+                # capture stdout only) have no way to tell a real limit from
+                # the fallback. Record it so it can be asserted on.
+                self.model_tokens_defaulted = True
         else:
             self.model_token = llm_params["model_tokens"]
 
+        # Consumed by ScrapeGraphAI; must not be forwarded to the model client.
+        llm_params.pop("model_tokens", None)
+
         try:
             if llm_params["model_provider"] not in {
                 "oneapi",

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/scrapegraphai/helpers/models_tokens.py

@@ -335,6 +335,10 @@ models_tokens = {
         "deepseek-ai/DeepSeek-V2-Chat": 131072,
         "claude-3-haiku": 200000,
     },
+    "atlascloud": {
+        "deepseek-ai/deepseek-v4-pro": 1048576,
+        "qwen/qwen3.5-flash": 1000000,
+    },
     "deepseek": {
         "deepseek-chat": 128000,
         "deepseek-coder": 128000,
@@ -408,8 +412,8 @@ models_tokens = {
         "grok-beta": 128000,
     },
     "minimax": {
-        "MiniMax-M3": 524288,
-        "MiniMax-M2.7": 204000,
-        "MiniMax-M2.7-highspeed": 204000,
+        "MiniMax-M3": 1000000,
+        "MiniMax-M2.7": 204800,
+        "MiniMax-M2.7-highspeed": 204800,
     },
 }

{scrapegraphai-2.2.0b4 → scrapegraphai-2.2.0b6}/scrapegraphai/models/__init__.py

@@ -2,6 +2,7 @@
 This module contains the model definitions used in the ScrapeGraphAI application.
 """
 
+from .atlascloud import AtlasCloud
 from .clod import CLoD
 from .deepseek import DeepSeek
 from .minimax import MiniMax
@@ -11,4 +12,4 @@ from .openai_itt import OpenAIImageToText
 from .openai_tts import OpenAITextToSpeech
 from .xai import XAI
 
-__all__ = ["DeepSeek", "MiniMax", "OneApi", "OpenAIImageToText", "OpenAITextToSpeech", "CLoD", "XAI", "Nvidia"]
+__all__ = ["AtlasCloud", "DeepSeek", "MiniMax", "OneApi", "OpenAIImageToText", "OpenAITextToSpeech", "CLoD", "XAI", "Nvidia"]