scrapeflow-py 0.3.0__tar.gz → 0.4.0__tar.gz

scrapeflow_py-0.4.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.0] - 2026-10-31
+
+### Fixed
+- LLM extraction works with current `mistralai` releases (2.x/3.x moved the client to
+  `mistralai.client.Mistral`). Previously every LLM call failed with a misleading
+  "mistralai is required" `ImportError` even though it was installed.
+- The wheel no longer installs stray top-level `tests` and `examples` packages into
+  `site-packages`.
+- `create_mcp_backend()` no longer silently returns a do-nothing backend: unknown backend names
+  raise `ValueError`, and the not-yet-implemented `scrapy_mcp` / `playwright_mcp` backends emit a
+  `UserWarning`.
+- Step timeouts chain the original `asyncio.TimeoutError` (`raise ... from`).
+- Dockerfile: browsers are installed where `PLAYWRIGHT_BROWSERS_PATH` points, and `pytest` is
+  available for the default `CMD` (the image previously could not run its own default command).
+
+### Added
+- `py.typed` marker (PEP 561) and `Typing :: Typed` classifier.
+- `__version__` is read from installed package metadata (`importlib.metadata`).
+- Tests: mocked Mistral schema generation and extraction (code-fence handling, bad JSON, async,
+  new SDK layout), MCP backend factory, and real-browser tests against a local HTTP server
+  (spec-driven extraction and robots.txt enforcement). 36 tests total.
+- CHANGELOG, SECURITY.md, CODE_OF_CONDUCT.md, issue/PR templates, Dependabot, CITATION.cff.
+
+### Changed
+- Requires Python 3.10+ (3.8 and 3.9 are end-of-life); classifiers for 3.10 to 3.13.
+- Packaging is `pyproject.toml`-only (removed `setup.py`, `pytest.ini`, `requirements*.txt`),
+  with PEP 639 `license = "MIT"` and a `dev` extra (pytest, black, ruff, mypy, build, twine).
+- CI: lint (ruff + black) is enforced instead of `|| true`, tests run on Python 3.10 to 3.13 with
+  Chromium installed, and the Docker job now runs the test suite inside the image.
+- Codebase formatted with black and cleaned with ruff (imports, modern type hints).
+
+## [0.3.0] - 2026-03-09
+
+### Added
+- Hybrid extraction, Mistral LLM extraction, session persistence, content cleaning.
+
+## [0.2.0] - 2026-03-09
+
+### Added
+- Specification-driven extraction, robots.txt compliance, ethical crawling, architecture refactor.
+
+## [0.1.2] - 2025-11-27
+
+- Initial public releases (0.1.0 to 0.1.2).

scrapeflow_py-0.4.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include scrapeflow/py.typed
+recursive-include examples *.py

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/PKG-INFO

@@ -1,20 +1,31 @@
 Metadata-Version: 2.4
 Name: scrapeflow-py
-Version: 0.3.0
-Summary: An opinionated scraping workflow engine built on Playwright
-Home-page: https://github.com/irfanalidv/scrapeflow-py
-Author: Irfan Ali
-License: MIT
-Classifier: Development Status :: 3 - Alpha
+Version: 0.4.0
+Summary: Opinionated Playwright scraping workflow engine: retries, rate limiting, anti-detection, robots.txt compliance, Pydantic spec-driven and LLM-powered extraction, sessions, and metrics.
+Author-email: Irfan Ali <irfanali29@hotmail.com>
+Maintainer-email: Irfan Ali <irfanali29@hotmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/irfanalidv/scrapeflow-py
+Project-URL: Documentation, https://github.com/irfanalidv/scrapeflow-py#readme
+Project-URL: Repository, https://github.com/irfanalidv/scrapeflow-py
+Project-URL: Issues, https://github.com/irfanalidv/scrapeflow-py/issues
+Project-URL: Changelog, https://github.com/irfanalidv/scrapeflow-py/blob/main/CHANGELOG.md
+Keywords: web-scraping,scraping,scraper,crawler,playwright,browser-automation,workflow,data-extraction,llm-extraction,mistral,pydantic,rate-limiting,retry,robots-txt,anti-detection,asyncio
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Framework :: AsyncIO
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Requires-Python: >=3.8
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: playwright>=1.40.0
@@ -23,22 +34,34 @@ Requires-Dist: tenacity>=8.2.0
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: mistralai>=1.0.0
 Requires-Dist: python-dotenv>=1.0.0
-Dynamic: author
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: black>=24.0.0; extra == "dev"
+Requires-Dist: ruff>=0.6.0; extra == "dev"
+Requires-Dist: mypy>=1.7.0; extra == "dev"
+Requires-Dist: build>=1.0.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
 Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
+
+<div align="center">
 
 # ScrapeFlow
 
-**An opinionated scraping workflow engine built on Playwright**
+**An opinionated scraping workflow engine built on Playwright.**
+
+Retries, rate limiting, anti-detection, robots.txt compliance, Pydantic spec-driven and
+Mistral LLM-powered extraction, login sessions, workflows, and metrics in one async API.
+
+[![PyPI version](https://img.shields.io/pypi/v/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![Python versions](https://img.shields.io/pypi/pyversions/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![CI](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml/badge.svg)](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml)
+[![PyPI Downloads](https://static.pepy.tech/badge/scrapeflow-py)](https://pepy.tech/projects/scrapeflow-py)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/)
 
-[![GitHub](https://img.shields.io/github/license/irfanalidv/scrapeflow-py)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) [![PyPI](https://img.shields.io/pypi/v/scrapeflow-py)](https://pypi.org/project/scrapeflow-py/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/scrapeflow-py?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/scrapeflow-py) [![Python](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/) [![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/) [![Status](https://img.shields.io/badge/status-active-success)](https://github.com/irfanalidv/scrapeflow-py)
+</div>
 
 ---
 
@@ -51,7 +74,7 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **⚖️ Ethical Crawling (GDPR/CCPA)**: Configurable data retention, anonymization, and consent options in the specification layer
 - **📦 Component Registry**: Shared, versioned selectors, pagination handlers, and login flows—platform thinking over one-off scrapers
 - **🔔 Monitoring & Alerting**: Alert callbacks on failure thresholds; rollback hooks for failed extraction runs
-- **🔌 MCP Extensibility**: Pluggable backends for Scrapy MCP Server, Playwright MCP, or LLM-based semantic extraction
+- **🔌 Pluggable Extraction Backends**: `MCPBackend` interface with a built-in Mistral LLM backend; subclass it to plug in an MCP server (Scrapy/Playwright MCP adapters are planned, not yet implemented)
 - **🤖 Mistral LLM Extraction**: Generate schemas from natural language, extract without selectors—uses `MISTRAL_API_KEY`
 - **🔄 Hybrid Extraction**: Selector-first, LLM fallback when validation fails—self-healing spiders
 - **🛡️ Selector Fallback Chain**: Try multiple selectors per field (`[".price", ".cost"]`) for resilience
@@ -67,13 +90,16 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **📝 Type Hints**: Full type support for better IDE experience
 
 <p align="center">
-  <img src="docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
+  <img src="https://raw.githubusercontent.com/irfanalidv/scrapeflow-py/main/docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
 </p>
 
 ## 📦 Installation
 
+Requires Python 3.10+.
+
 ```bash
 pip install scrapeflow-py
+playwright install chromium   # download the browser Playwright drives
 ```
 
 Or install from source:
@@ -82,17 +108,24 @@ Or install from source:
 git clone https://github.com/irfanalidv/scrapeflow-py.git
 cd scrapeflow-py
 pip install -e .
+playwright install chromium
 ```
 
-**Note**: After installation, install Playwright browsers:
+On a fresh Linux machine or container, `playwright install --with-deps chromium` also installs
+the system libraries Chromium needs. A `Dockerfile` is included.
 
-```bash
-playwright install
-```
+## ⚙️ Configuration (environment variables)
+
+| Variable | Purpose |
+| --- | --- |
+| `MISTRAL_API_KEY` | Required only for LLM features (`generate_schema_from_prompt`, `extract_with_schema`, `MistralLLMBackend`, `HybridExtractor` LLM fallback). Read from the environment or a `.env` file. |
+
+Everything else (rate limits, retries, browser, anti-detection, robots.txt, ethical crawling) is
+configured in code via `ScrapeFlowConfig`; see [Configuration](#configuration) below.
 
 ## 🎯 Real-World Use Cases
 
-ScrapeFlow is used in production for:
+Typical use cases:
 
 - **💰 E-commerce Price Monitoring** - Track competitor prices, monitor deals, and optimize pricing strategies
 - **📰 News & Content Aggregation** - Collect articles from multiple sources for content platforms
@@ -1247,11 +1280,20 @@ scrapeflow/
 └── exceptions.py       # Custom exceptions
 ```
 
-For deeper design details, see [`ARCHITECTURE.md`](ARCHITECTURE.md).
+For deeper design details, see [`ARCHITECTURE.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/ARCHITECTURE.md). Release notes are in [`CHANGELOG.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/CHANGELOG.md).
 
 ## 🤝 Contributing
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+Contributions are welcome! See [CONTRIBUTING.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/CONTRIBUTING.md)
+and the [Code of Conduct](https://github.com/irfanalidv/scrapeflow-py/blob/main/CODE_OF_CONDUCT.md).
+
+```bash
+git clone https://github.com/irfanalidv/scrapeflow-py.git && cd scrapeflow-py
+pip install -e ".[dev]" && playwright install chromium
+ruff check . && black --check . && pytest
+```
+
+Security issues: please follow [SECURITY.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/SECURITY.md).
 
 1. Fork the repository
 2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
@@ -1261,7 +1303,7 @@ Contributions are welcome! Please feel free to submit a Pull Request.
 
 ## 📝 License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License - see the [LICENSE](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) file for details.
 
 ## 🙏 Acknowledgments
 

scrapeflow_py-0.3.0/scrapeflow_py.egg-info/PKG-INFO → scrapeflow_py-0.4.0/README.md

@@ -1,44 +1,21 @@
-Metadata-Version: 2.4
-Name: scrapeflow-py
-Version: 0.3.0
-Summary: An opinionated scraping workflow engine built on Playwright
-Home-page: https://github.com/irfanalidv/scrapeflow-py
-Author: Irfan Ali
-License: MIT
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: playwright>=1.40.0
-Requires-Dist: aiohttp>=3.9.0
-Requires-Dist: tenacity>=8.2.0
-Requires-Dist: pydantic>=2.0.0
-Requires-Dist: mistralai>=1.0.0
-Requires-Dist: python-dotenv>=1.0.0
-Dynamic: author
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
-Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
+<div align="center">
 
 # ScrapeFlow
 
-**An opinionated scraping workflow engine built on Playwright**
+**An opinionated scraping workflow engine built on Playwright.**
 
-[![GitHub](https://img.shields.io/github/license/irfanalidv/scrapeflow-py)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) [![PyPI](https://img.shields.io/pypi/v/scrapeflow-py)](https://pypi.org/project/scrapeflow-py/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/scrapeflow-py?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/scrapeflow-py) [![Python](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/) [![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/) [![Status](https://img.shields.io/badge/status-active-success)](https://github.com/irfanalidv/scrapeflow-py)
+Retries, rate limiting, anti-detection, robots.txt compliance, Pydantic spec-driven and
+Mistral LLM-powered extraction, login sessions, workflows, and metrics in one async API.
+
+[![PyPI version](https://img.shields.io/pypi/v/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![Python versions](https://img.shields.io/pypi/pyversions/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![CI](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml/badge.svg)](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml)
+[![PyPI Downloads](https://static.pepy.tech/badge/scrapeflow-py)](https://pepy.tech/projects/scrapeflow-py)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/)
+
+</div>
 
 ---
 
@@ -51,7 +28,7 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **⚖️ Ethical Crawling (GDPR/CCPA)**: Configurable data retention, anonymization, and consent options in the specification layer
 - **📦 Component Registry**: Shared, versioned selectors, pagination handlers, and login flows—platform thinking over one-off scrapers
 - **🔔 Monitoring & Alerting**: Alert callbacks on failure thresholds; rollback hooks for failed extraction runs
-- **🔌 MCP Extensibility**: Pluggable backends for Scrapy MCP Server, Playwright MCP, or LLM-based semantic extraction
+- **🔌 Pluggable Extraction Backends**: `MCPBackend` interface with a built-in Mistral LLM backend; subclass it to plug in an MCP server (Scrapy/Playwright MCP adapters are planned, not yet implemented)
 - **🤖 Mistral LLM Extraction**: Generate schemas from natural language, extract without selectors—uses `MISTRAL_API_KEY`
 - **🔄 Hybrid Extraction**: Selector-first, LLM fallback when validation fails—self-healing spiders
 - **🛡️ Selector Fallback Chain**: Try multiple selectors per field (`[".price", ".cost"]`) for resilience
@@ -67,13 +44,16 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **📝 Type Hints**: Full type support for better IDE experience
 
 <p align="center">
-  <img src="docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
+  <img src="https://raw.githubusercontent.com/irfanalidv/scrapeflow-py/main/docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
 </p>
 
 ## 📦 Installation
 
+Requires Python 3.10+.
+
 ```bash
 pip install scrapeflow-py
+playwright install chromium   # download the browser Playwright drives
 ```
 
 Or install from source:
@@ -82,17 +62,24 @@ Or install from source:
 git clone https://github.com/irfanalidv/scrapeflow-py.git
 cd scrapeflow-py
 pip install -e .
+playwright install chromium
 ```
 
-**Note**: After installation, install Playwright browsers:
+On a fresh Linux machine or container, `playwright install --with-deps chromium` also installs
+the system libraries Chromium needs. A `Dockerfile` is included.
 
-```bash
-playwright install
-```
+## ⚙️ Configuration (environment variables)
+
+| Variable | Purpose |
+| --- | --- |
+| `MISTRAL_API_KEY` | Required only for LLM features (`generate_schema_from_prompt`, `extract_with_schema`, `MistralLLMBackend`, `HybridExtractor` LLM fallback). Read from the environment or a `.env` file. |
+
+Everything else (rate limits, retries, browser, anti-detection, robots.txt, ethical crawling) is
+configured in code via `ScrapeFlowConfig`; see [Configuration](#configuration) below.
 
 ## 🎯 Real-World Use Cases
 
-ScrapeFlow is used in production for:
+Typical use cases:
 
 - **💰 E-commerce Price Monitoring** - Track competitor prices, monitor deals, and optimize pricing strategies
 - **📰 News & Content Aggregation** - Collect articles from multiple sources for content platforms
@@ -1247,11 +1234,20 @@ scrapeflow/
 └── exceptions.py       # Custom exceptions
 ```
 
-For deeper design details, see [`ARCHITECTURE.md`](ARCHITECTURE.md).
+For deeper design details, see [`ARCHITECTURE.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/ARCHITECTURE.md). Release notes are in [`CHANGELOG.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/CHANGELOG.md).
 
 ## 🤝 Contributing
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+Contributions are welcome! See [CONTRIBUTING.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/CONTRIBUTING.md)
+and the [Code of Conduct](https://github.com/irfanalidv/scrapeflow-py/blob/main/CODE_OF_CONDUCT.md).
+
+```bash
+git clone https://github.com/irfanalidv/scrapeflow-py.git && cd scrapeflow-py
+pip install -e ".[dev]" && playwright install chromium
+ruff check . && black --check . && pytest
+```
+
+Security issues: please follow [SECURITY.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/SECURITY.md).
 
 1. Fork the repository
 2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
@@ -1261,7 +1257,7 @@ Contributions are welcome! Please feel free to submit a Pull Request.
 
 ## 📝 License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License - see the [LICENSE](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) file for details.
 
 ## 🙏 Acknowledgments
 

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/__init__.py

@@ -1,2 +1 @@
 """Examples for ScrapeFlow usage."""
-

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/advanced_example.py

@@ -1,14 +1,15 @@
 """Advanced example showing anti-detection, rate limiting, and structured extraction."""
 
 import asyncio
+
 from scrapeflow import ScrapeFlow
 from scrapeflow.config import (
-    ScrapeFlowConfig,
     AntiDetectionConfig,
     RateLimitConfig,
     RetryConfig,
+    ScrapeFlowConfig,
 )
-from scrapeflow.extractors import StructuredExtractor, Extractor
+from scrapeflow.extractors import Extractor, StructuredExtractor
 
 
 async def main():
@@ -65,8 +66,7 @@ async def main():
         # Extract tags separately using Extractor for each quote
         quotes_with_tags = []
         quote_elements = scraper.page.locator(".quote")
-        count = await quote_elements.count()
-        
+
         for i, quote_data in enumerate(data.get("quotes", [])[:5]):  # First 5
             quote_elem = quote_elements.nth(i)
             tags = await Extractor.extract_texts(quote_elem, ".tag")
@@ -84,7 +84,7 @@ async def main():
 
         # Get metrics
         metrics = scraper.get_metrics()
-        print(f"\nScraping completed:")
+        print("\nScraping completed:")
         print(f"  Success rate: {metrics.get_success_rate():.2f}%")
         print(f"  Total requests: {metrics.total_requests}")
         print(f"  Retries: {metrics.retry_count}")
@@ -92,4 +92,3 @@ async def main():
 
 if __name__ == "__main__":
     asyncio.run(main())
-

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/authenticated_quotes_example.py

@@ -8,6 +8,7 @@ Demonstrates:
 """
 
 import asyncio
+
 from pydantic import BaseModel
 
 from scrapeflow import (
@@ -16,7 +17,7 @@ from scrapeflow import (
     get_registry,
     register_quotes_login_handler,
 )
-from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig
+from scrapeflow.config import EthicalCrawlingConfig, ScrapeFlowConfig
 from scrapeflow.specifications import FieldSpec, ItemSpec
 
 

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/basic_usage.py

@@ -5,8 +5,9 @@ a real website designed for scraping practice.
 """
 
 import asyncio
+
 from scrapeflow import ScrapeFlow
-from scrapeflow.config import ScrapeFlowConfig, RateLimitConfig, RetryConfig
+from scrapeflow.config import RateLimitConfig, RetryConfig, ScrapeFlowConfig
 from scrapeflow.extractors import Extractor
 
 

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/hybrid_extraction_example.py

@@ -11,17 +11,18 @@ Requires MISTRAL_API_KEY in .env for LLM fallback.
 
 import asyncio
 import os
-from typing import Optional
 
 try:
     from dotenv import load_dotenv
+
     load_dotenv()
 except ImportError:
     pass
 
 from pydantic import BaseModel, Field
-from scrapeflow import ScrapeFlow, HybridExtractor, SpecificationExtractor
-from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig
+
+from scrapeflow import HybridExtractor, ScrapeFlow
+from scrapeflow.config import EthicalCrawlingConfig, ScrapeFlowConfig
 from scrapeflow.specifications import FieldSpec
 
 
@@ -30,7 +31,7 @@ class ProductSpec(BaseModel):
 
     title: str = Field(..., description="Product title")
     price: str = Field(..., description="Price string")
-    availability: Optional[str] = Field(None, description="Stock status")
+    availability: str | None = Field(None, description="Stock status")
 
 
 async def run_hybrid_example():
@@ -47,7 +48,9 @@ async def run_hybrid_example():
     }
 
     async with ScrapeFlow(config) as scraper:
-        await scraper.navigate("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
+        await scraper.navigate(
+            "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
+        )
 
         # Use HybridExtractor: selectors first, LLM fallback if validation fails
         use_llm = bool(os.environ.get("MISTRAL_API_KEY"))

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/llm_extraction_example.py

@@ -15,13 +15,14 @@ import os
 # Load .env if available
 try:
     from dotenv import load_dotenv
+
     load_dotenv()
 except ImportError:
     pass
 
 from scrapeflow import ScrapeFlow, create_mcp_backend
-from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig
-from scrapeflow.llm_extract import generate_schema_from_prompt, extract_with_schema
+from scrapeflow.config import EthicalCrawlingConfig, ScrapeFlowConfig
+from scrapeflow.llm_extract import extract_with_schema, generate_schema_from_prompt
 
 
 async def run_llm_extraction():
@@ -38,15 +39,11 @@ async def run_llm_extraction():
         await scraper.navigate("https://quotes.toscrape.com/")
 
         # Option 1: Generate schema from prompt
-        schema = generate_schema_from_prompt(
-            "Extract quote text, author name, and list of tags"
-        )
+        schema = generate_schema_from_prompt("Extract quote text, author name, and list of tags")
         print("Generated schema:", schema)
 
         # Option 2: Extract using schema + page content
-        content = await scraper.page.evaluate(
-            "() => document.body.innerText"
-        )
+        content = await scraper.page.evaluate("() => document.body.innerText")
         data = extract_with_schema(
             content,
             schema,

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/session_persistence_example.py

@@ -10,11 +10,17 @@ import asyncio
 import os
 import tempfile
 
-from scrapeflow import ScrapeFlow, SpecificationExtractor, get_registry, register_quotes_login_handler
-from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig, BrowserConfig
-from scrapeflow.specifications import FieldSpec, ItemSpec
 from pydantic import BaseModel
 
+from scrapeflow import (
+    ScrapeFlow,
+    SpecificationExtractor,
+    get_registry,
+    register_quotes_login_handler,
+)
+from scrapeflow.config import BrowserConfig, EthicalCrawlingConfig, ScrapeFlowConfig
+from scrapeflow.specifications import FieldSpec, ItemSpec
+
 
 class QuoteItem(BaseModel):
     text: str

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/specification_driven_example.py

@@ -10,16 +10,16 @@ Demonstrates:
 """
 
 import asyncio
-from pydantic import BaseModel, Field
+
+from pydantic import BaseModel
 
 from scrapeflow import ScrapeFlow, SpecificationExtractor
 from scrapeflow.config import (
-    ScrapeFlowConfig,
     EthicalCrawlingConfig,
     RateLimitConfig,
+    ScrapeFlowConfig,
 )
 from scrapeflow.specifications import FieldSpec, ItemSpec, JobListingSpec
-from scrapeflow.registry import get_registry, register_job_listing_schema
 
 
 class JobListingsPage(BaseModel):
@@ -73,7 +73,9 @@ async def scrape_job_listings():
             print(f"  {i+1}. {job.title[:50]}... | {job.company or 'N/A'}")
 
         metrics = scraper.get_metrics()
-        print(f"\nMetrics: {metrics.total_requests} requests, {metrics.get_success_rate():.1f}% success")
+        print(
+            f"\nMetrics: {metrics.total_requests} requests, {metrics.get_success_rate():.1f}% success"
+        )
 
 
 if __name__ == "__main__":

{scrapeflow_py-0.3.0 → scrapeflow_py-0.4.0}/examples/workflow_example.py

@@ -5,6 +5,7 @@ a real e-commerce site designed for scraping practice.
 """
 
 import asyncio
+
 from scrapeflow import ScrapeFlow, Workflow
 from scrapeflow.config import ScrapeFlowConfig
 from scrapeflow.extractors import StructuredExtractor
@@ -20,12 +21,12 @@ async def scrape_books_page(page, context):
                 "title": "h3 a",
                 "price": ".price_color",
                 "availability": ".instock.availability",
-            }
+            },
         }
     }
     extractor = StructuredExtractor(schema)
     data = await extractor.extract(page)
-    
+
     context["books_data"] = data
     return data
 
@@ -90,7 +91,7 @@ async def main():
 
         # Print metrics
         metrics = scraper.get_metrics()
-        print(f"\n📊 Metrics:")
+        print("\n📊 Metrics:")
         print(f"   Total requests: {metrics.total_requests}")
         print(f"   Success rate: {metrics.get_success_rate():.2f}%")
         print(f"   Average response time: {metrics.average_response_time:.2f}s")
@@ -98,4 +99,3 @@ async def main():
 
 if __name__ == "__main__":
     asyncio.run(main())
-