scrapeflow-py 0.2.0__tar.gz → 0.4.0__tar.gz

scrapeflow_py-0.4.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.0] - 2026-10-31
+
+### Fixed
+- LLM extraction works with current `mistralai` releases (2.x/3.x moved the client to
+  `mistralai.client.Mistral`). Previously every LLM call failed with a misleading
+  "mistralai is required" `ImportError` even though it was installed.
+- The wheel no longer installs stray top-level `tests` and `examples` packages into
+  `site-packages`.
+- `create_mcp_backend()` no longer silently returns a do-nothing backend: unknown backend names
+  raise `ValueError`, and the not-yet-implemented `scrapy_mcp` / `playwright_mcp` backends emit a
+  `UserWarning`.
+- Step timeouts chain the original `asyncio.TimeoutError` (`raise ... from`).
+- Dockerfile: browsers are installed where `PLAYWRIGHT_BROWSERS_PATH` points, and `pytest` is
+  available for the default `CMD` (the image previously could not run its own default command).
+
+### Added
+- `py.typed` marker (PEP 561) and `Typing :: Typed` classifier.
+- `__version__` is read from installed package metadata (`importlib.metadata`).
+- Tests: mocked Mistral schema generation and extraction (code-fence handling, bad JSON, async,
+  new SDK layout), MCP backend factory, and real-browser tests against a local HTTP server
+  (spec-driven extraction and robots.txt enforcement). 36 tests total.
+- CHANGELOG, SECURITY.md, CODE_OF_CONDUCT.md, issue/PR templates, Dependabot, CITATION.cff.
+
+### Changed
+- Requires Python 3.10+ (3.8 and 3.9 are end-of-life); classifiers for 3.10 to 3.13.
+- Packaging is `pyproject.toml`-only (removed `setup.py`, `pytest.ini`, `requirements*.txt`),
+  with PEP 639 `license = "MIT"` and a `dev` extra (pytest, black, ruff, mypy, build, twine).
+- CI: lint (ruff + black) is enforced instead of `|| true`, tests run on Python 3.10 to 3.13 with
+  Chromium installed, and the Docker job now runs the test suite inside the image.
+- Codebase formatted with black and cleaned with ruff (imports, modern type hints).
+
+## [0.3.0] - 2026-03-09
+
+### Added
+- Hybrid extraction, Mistral LLM extraction, session persistence, content cleaning.
+
+## [0.2.0] - 2026-03-09
+
+### Added
+- Specification-driven extraction, robots.txt compliance, ethical crawling, architecture refactor.
+
+## [0.1.2] - 2025-11-27
+
+- Initial public releases (0.1.0 to 0.1.2).

scrapeflow_py-0.4.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include scrapeflow/py.typed
+recursive-include examples *.py

{scrapeflow_py-0.2.0 → scrapeflow_py-0.4.0}/PKG-INFO

@@ -1,42 +1,67 @@
 Metadata-Version: 2.4
 Name: scrapeflow-py
-Version: 0.2.0
-Summary: An opinionated scraping workflow engine built on Playwright
-Home-page: https://github.com/irfanalidv/scrapeflow-py
-Author: Irfan Ali
-License: MIT
-Classifier: Development Status :: 3 - Alpha
+Version: 0.4.0
+Summary: Opinionated Playwright scraping workflow engine: retries, rate limiting, anti-detection, robots.txt compliance, Pydantic spec-driven and LLM-powered extraction, sessions, and metrics.
+Author-email: Irfan Ali <irfanali29@hotmail.com>
+Maintainer-email: Irfan Ali <irfanali29@hotmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/irfanalidv/scrapeflow-py
+Project-URL: Documentation, https://github.com/irfanalidv/scrapeflow-py#readme
+Project-URL: Repository, https://github.com/irfanalidv/scrapeflow-py
+Project-URL: Issues, https://github.com/irfanalidv/scrapeflow-py/issues
+Project-URL: Changelog, https://github.com/irfanalidv/scrapeflow-py/blob/main/CHANGELOG.md
+Keywords: web-scraping,scraping,scraper,crawler,playwright,browser-automation,workflow,data-extraction,llm-extraction,mistral,pydantic,rate-limiting,retry,robots-txt,anti-detection,asyncio
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Framework :: AsyncIO
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Requires-Python: >=3.8
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: playwright>=1.40.0
 Requires-Dist: aiohttp>=3.9.0
 Requires-Dist: tenacity>=8.2.0
 Requires-Dist: pydantic>=2.0.0
-Dynamic: author
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
+Requires-Dist: mistralai>=1.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: black>=24.0.0; extra == "dev"
+Requires-Dist: ruff>=0.6.0; extra == "dev"
+Requires-Dist: mypy>=1.7.0; extra == "dev"
+Requires-Dist: build>=1.0.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
 Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
+
+<div align="center">
 
 # ScrapeFlow
 
-**An opinionated scraping workflow engine built on Playwright**
+**An opinionated scraping workflow engine built on Playwright.**
+
+Retries, rate limiting, anti-detection, robots.txt compliance, Pydantic spec-driven and
+Mistral LLM-powered extraction, login sessions, workflows, and metrics in one async API.
+
+[![PyPI version](https://img.shields.io/pypi/v/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![Python versions](https://img.shields.io/pypi/pyversions/scrapeflow-py.svg)](https://pypi.org/project/scrapeflow-py/)
+[![CI](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml/badge.svg)](https://github.com/irfanalidv/scrapeflow-py/actions/workflows/ci.yml)
+[![PyPI Downloads](https://static.pepy.tech/badge/scrapeflow-py)](https://pepy.tech/projects/scrapeflow-py)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/)
 
-[![GitHub](https://img.shields.io/github/license/irfanalidv/scrapeflow-py)](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) [![PyPI](https://img.shields.io/pypi/v/scrapeflow-py)](https://pypi.org/project/scrapeflow-py/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/scrapeflow-py?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/scrapeflow-py) [![Python](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/) [![Playwright](https://img.shields.io/badge/Playwright-1.40%2B-green)](https://playwright.dev/) [![Status](https://img.shields.io/badge/status-active-success)](https://github.com/irfanalidv/scrapeflow-py)
+</div>
 
 ---
 
@@ -49,7 +74,12 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **⚖️ Ethical Crawling (GDPR/CCPA)**: Configurable data retention, anonymization, and consent options in the specification layer
 - **📦 Component Registry**: Shared, versioned selectors, pagination handlers, and login flows—platform thinking over one-off scrapers
 - **🔔 Monitoring & Alerting**: Alert callbacks on failure thresholds; rollback hooks for failed extraction runs
-- **🔌 MCP Extensibility**: Pluggable backends for Scrapy MCP Server, Playwright MCP, or LLM-based semantic extraction
+- **🔌 Pluggable Extraction Backends**: `MCPBackend` interface with a built-in Mistral LLM backend; subclass it to plug in an MCP server (Scrapy/Playwright MCP adapters are planned, not yet implemented)
+- **🤖 Mistral LLM Extraction**: Generate schemas from natural language, extract without selectors—uses `MISTRAL_API_KEY`
+- **🔄 Hybrid Extraction**: Selector-first, LLM fallback when validation fails—self-healing spiders
+- **🛡️ Selector Fallback Chain**: Try multiple selectors per field (`[".price", ".cost"]`) for resilience
+- **🫙 Session Persistence**: Save/load cookies and localStorage via `save_storage_state()` and `storage_state_path`
+- **🧹 Content Cleaning**: Strip scripts/styles before LLM extraction for better accuracy and token usage
 - **🔄 Intelligent Retry Logic**: Automatic retries with exponential backoff and jitter
 - **⚡ Rate Limiting**: Token bucket algorithm to respect server limits
 - **🕵️ Anti-Detection**: Stealth mode, user agent rotation, and proxy support
@@ -60,13 +90,16 @@ ScrapeFlow is a production-ready Python library that transforms Playwright into
 - **📝 Type Hints**: Full type support for better IDE experience
 
 <p align="center">
-  <img src="docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
+  <img src="https://raw.githubusercontent.com/irfanalidv/scrapeflow-py/main/docs/scrapeflow-architecture.png" alt="ScrapeFlow Architecture" width="800">
 </p>
 
 ## 📦 Installation
 
+Requires Python 3.10+.
+
 ```bash
 pip install scrapeflow-py
+playwright install chromium   # download the browser Playwright drives
 ```
 
 Or install from source:
@@ -75,17 +108,24 @@ Or install from source:
 git clone https://github.com/irfanalidv/scrapeflow-py.git
 cd scrapeflow-py
 pip install -e .
+playwright install chromium
 ```
 
-**Note**: After installation, install Playwright browsers:
+On a fresh Linux machine or container, `playwright install --with-deps chromium` also installs
+the system libraries Chromium needs. A `Dockerfile` is included.
 
-```bash
-playwright install
-```
+## ⚙️ Configuration (environment variables)
+
+| Variable | Purpose |
+| --- | --- |
+| `MISTRAL_API_KEY` | Required only for LLM features (`generate_schema_from_prompt`, `extract_with_schema`, `MistralLLMBackend`, `HybridExtractor` LLM fallback). Read from the environment or a `.env` file. |
+
+Everything else (rate limits, retries, browser, anti-detection, robots.txt, ethical crawling) is
+configured in code via `ScrapeFlowConfig`; see [Configuration](#configuration) below.
 
 ## 🎯 Real-World Use Cases
 
-ScrapeFlow is used in production for:
+Typical use cases:
 
 - **💰 E-commerce Price Monitoring** - Track competitor prices, monitor deals, and optimize pricing strategies
 - **📰 News & Content Aggregation** - Collect articles from multiple sources for content platforms
@@ -371,6 +411,132 @@ print(f"💼 Found {len(quotes)} quotes across pages")
 💼 Found 20 quotes across pages
 ```
 
+### Use Case 5: LLM-Powered Extraction (No Selectors)
+
+**Real-world scenario:** Extract structured data using Mistral AI—no CSS selectors, just natural language field descriptions. Ideal for pages with inconsistent markup or when you want semantic understanding.
+
+**Requires:** `MISTRAL_API_KEY` in `.env` or environment.
+
+```python
+import asyncio
+import os
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+from scrapeflow import ScrapeFlow, create_mcp_backend
+from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig
+from scrapeflow.llm_extract import generate_schema_from_prompt, extract_with_schema
+
+async def run_llm_extraction():
+    if not os.environ.get("MISTRAL_API_KEY"):
+        print("Set MISTRAL_API_KEY in .env to run this example")
+        return
+
+    config = ScrapeFlowConfig(
+        ethical_crawling=EthicalCrawlingConfig(respect_robots_txt=True),
+    )
+
+    async with ScrapeFlow(config) as scraper:
+        await scraper.navigate("https://quotes.toscrape.com/")
+
+        # Option 1: Generate schema from natural language
+        schema = generate_schema_from_prompt(
+            "Extract quote text, author name, and list of tags"
+        )
+
+        # Option 2: Extract using schema + page content (no selectors!)
+        content = await scraper.page.evaluate("() => document.body.innerText")
+        data = extract_with_schema(
+            content,
+            schema,
+            prompt="Extract all quotes from this page with text, author, and tags",
+        )
+        print("LLM extracted:", data)
+
+        # Option 3: MistralLLMBackend—field descriptions only
+        backend = create_mcp_backend("mistral")
+        semantic_data = await backend.extract_with_semantics(
+            scraper.page,
+            field_descriptions={
+                "quote": "The quote text",
+                "author": "Author name",
+                "tags": "Comma-separated tags",
+            },
+            context="Extract the first quote on the page",
+        )
+        print("Semantic extraction:", semantic_data)
+
+asyncio.run(run_llm_extraction())
+```
+
+**Real Output:**
+
+```
+Generated schema: {'type': 'object', 'properties': {'quote_text': {...}, 'author_name': {...}, 'tags': {...}}, 'required': ['quote_text', 'author_name', 'tags']}
+
+LLM extracted: [{'quote_text': 'The world as we have created it is a process of our thinking...', 'author_name': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}, ...]
+
+Semantic extraction: {'quote': 'The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.', 'author': 'Albert Einstein', 'tags': 'change, deep-thoughts, thinking, world'}
+```
+
+**What you get:** Schema generation from prompts, extraction without brittle selectors, and semantic understanding of page content—powerful for dynamic or inconsistent sites.
+
+### Use Case 6: Login + Session Persistence
+
+**Real-world scenario:** Log in once, save cookies, then reuse the session on future runs—no re-login needed.
+
+```python
+import asyncio
+import os
+from scrapeflow import ScrapeFlow, SpecificationExtractor, get_registry, register_quotes_login_handler
+from scrapeflow.config import ScrapeFlowConfig, EthicalCrawlingConfig, BrowserConfig
+from scrapeflow.specifications import FieldSpec, ItemSpec
+from pydantic import BaseModel
+
+class QuoteItem(BaseModel):
+    text: str
+    author: str
+
+class QuotesPage(BaseModel):
+    quotes: list[QuoteItem]
+
+async def run():
+    state_file = "/tmp/scrapeflow_session.json"
+    schema = {"quotes": ItemSpec(items_selector=".quote", fields={"text": ".text", "author": ".author"})}
+
+    # Step 1: Login and save (if first run)
+    if not os.path.exists(state_file):
+        async with ScrapeFlow(ScrapeFlowConfig(ethical_crawling=EthicalCrawlingConfig(respect_robots_txt=True))) as scraper:
+            register_quotes_login_handler(get_registry())
+            handler = get_registry().get_login("quotes_login")
+            await scraper.login_with_handler("https://quotes.toscrape.com/login", "admin", "amdin", handler)
+            await scraper.save_storage_state(state_file)
+
+    # Step 2: Use saved session (no login)
+    config = ScrapeFlowConfig(ethical_crawling=EthicalCrawlingConfig(respect_robots_txt=True),
+                             browser=BrowserConfig(storage_state_path=state_file))
+    async with ScrapeFlow(config) as scraper:
+        await scraper.navigate("https://quotes.toscrape.com/")
+        data = await SpecificationExtractor(QuotesPage, schema=schema).extract(scraper.page)
+        print(f"Extracted {len(data.quotes)} quotes (authenticated)")
+
+asyncio.run(run())
+```
+
+**Real Output:**
+
+```
+Extracted 10 quotes (authenticated)
+```
+
+*(On first run: login + save; on subsequent runs: load session and extract without re-login.)*
+
+**What you get:** `login_with_handler`, `save_storage_state`, `storage_state_path`—reusable authenticated sessions.
+
 ## 📚 Documentation
 
 ### Configuration
@@ -568,6 +734,40 @@ async with ScrapeFlow(config) as scraper:
     # - Temporary blocks
 ```
 
+### Pagination
+
+**Use Case:** Scrape multiple pages (e.g. search results, product listings) with limits.
+
+```python
+import asyncio
+from scrapeflow import ScrapeFlow
+from scrapeflow.config import ScrapeFlowConfig, PaginationConfig
+from scrapeflow.pagination import paginate
+from scrapeflow.registry import get_registry
+
+async def extract_quotes(page, context):
+    quotes = []
+    for i in range(await page.locator(".quote").count()):
+        q = page.locator(".quote").nth(i)
+        text = await q.locator(".text").text_content()
+        author = await q.locator(".author").text_content()
+        quotes.append({"text": text or "", "author": author or ""})
+    return quotes
+
+async def main():
+    get_registry().register_pagination("quotes", next_selector="li.next a", has_next="li.next")
+    handler = get_registry().get_pagination("quotes")
+    config = PaginationConfig(max_pages=3, max_results=25)
+
+    async with ScrapeFlow(ScrapeFlowConfig()) as scraper:
+        async for page_data in paginate(scraper, "https://quotes.toscrape.com/", handler, extract_quotes, config):
+            print(f"Page: {len(page_data)} quotes")
+            for q in page_data[:2]:
+                print(f"  - {q['author']}: {q['text'][:40]}...")
+
+asyncio.run(main())
+```
+
 ### Data Extraction
 
 **Use Case:** Extracting structured data from [quotes.toscrape.com](https://quotes.toscrape.com/) and [books.toscrape.com](https://books.toscrape.com/).
@@ -798,13 +998,175 @@ asyncio.run(scrape_with_error_handling())
 ✅ Successfully scraped: Quotes to Scrape
 ```
 
+## 📋 Feature Coverage—Nothing Hidden
+
+Every ScrapeFlow feature is demonstrated somewhere. Quick reference:
+
+| Feature | Where to See It |
+|---------|-----------------|
+| **Extractors** | Use Case 1, `basic_usage.py`, `advanced_example.py` |
+| **StructuredExtractor** | Use Case 2, `workflow_example.py`, `advanced_example.py` |
+| **SpecificationExtractor** | Use Case 6, Doc section, `specification_driven_example.py` |
+| **HybridExtractor** | `hybrid_extraction_example.py` |
+| **FieldSpec, ItemSpec** | Doc section, `specification_driven_example.py`, `hybrid_extraction_example.py` |
+| **Selector fallback** `[".a", ".b"]` | `hybrid_extraction_example.py` |
+| **LLM extraction** | Use Case 5, `llm_extraction_example.py` |
+| **Login** | Use Case 6, `authenticated_quotes_example.py`, `session_persistence_example.py` |
+| **Session persistence** | Use Case 6, `session_persistence_example.py` |
+| **Registry, LoginHandler** | `authenticated_quotes_example.py`, `specification_driven_example.py` |
+| **Workflow** | Use Case 2, `workflow_example.py` |
+| **Rate limit, Retry** | Use Cases 1, 4, `basic_usage.py`, `advanced_example.py` |
+| **Anti-detection** | Use Case 3, `advanced_example.py` |
+| **Ethical crawling, robots.txt** | `specification_driven_example.py`, `authenticated_quotes_example.py` |
+| **Pagination** | Doc section, `scrapeflow.pagination.paginate`, `PaginationConfig` |
+| **Content cleaning** | Used internally by MistralLLMBackend; `scrapeflow.content_utils` |
+
 ## 🎨 Complete Examples
 
-Check out the `examples/` directory for more examples:
+Check out the `examples/` directory—each example showcases specific features:
+
+| Example | Features Demonstrated |
+|---------|------------------------|
+| `basic_usage.py` | ScrapeFlow, Extractor, rate limit, retry, metrics |
+| `workflow_example.py` | Workflow, steps, on_success/on_error, StructuredExtractor |
+| `advanced_example.py` | Anti-detection, rate limit, retry, StructuredExtractor |
+| `specification_driven_example.py` | SpecificationExtractor, FieldSpec, ItemSpec, ethical crawling, registry |
+| `authenticated_quotes_example.py` | Login, `login_with_handler`, registry, LoginHandler |
+| `llm_extraction_example.py` | Mistral LLM, `generate_schema_from_prompt`, `extract_with_schema`, MistralLLMBackend *(requires `MISTRAL_API_KEY`)* |
+| `hybrid_extraction_example.py` | HybridExtractor, selector fallback chain `[".a", ".b"]`, LLM fallback |
+| `session_persistence_example.py` | `save_storage_state`, `storage_state_path`, skip login on reuse |
+
+### Real Outputs from Examples
+
+Outputs below are from running each example against live sites (quotes.toscrape.com, books.toscrape.com).
+
+**`basic_usage.py`**
+
+```
+Page title: Quotes to Scrape
+
+Extracted 5 quotes:
+
+1. "The world as we have created it is a process of our thinkin...
+   Author: Albert Einstein
+   Tags: change, deep-thoughts, thinking, world
+
+2. "It is our choices, Harry, that show what we truly are, far ...
+   Author: J.K. Rowling
+   Tags: abilities, choices
+
+3. "There are only two ways to live your life. One is as though...
+   Author: Albert Einstein
+   Tags: inspirational, life, live, miracle, miracles
+
+4. "The person, be it gentleman or lady, who has not pleasure i...
+   Author: Jane Austen
+   Tags: aliteracy, books, classic, humor
+
+5. "Imperfection is beauty, madness is genius and it's better t...
+   Author: Marilyn Monroe
+   Tags: be-yourself, inspirational
+
+📊 Metrics:
+   Success rate: 100.00%
+   Total requests: 1
+   Average response time: 1.72s
+```
+
+**`advanced_example.py`**
+
+```
+📝 Extracted quotes:
+
+1. "The world as we have created it is a process of our thinkin...
+   Author: Albert Einstein
+   Tags: change, deep-thoughts, thinking, world
+
+2. "It is our choices, Harry, that show what we truly are, far ...
+   Author: J.K. Rowling
+   Tags: abilities, choices
+
+3. "There are only two ways to live your life. One is as though...
+   Author: Albert Einstein
+   Tags: inspirational, life, live, miracle, miracles
+
+4. "The person, be it gentleman or lady, who has not pleasure i...
+   Author: Jane Austen
+   Tags: aliteracy, books, classic, humor
+
+5. "Imperfection is beauty, madness is genius and it's better t...
+   Author: Marilyn Monroe
+   Tags: be-yourself, inspirational
+
+Scraping completed:
+  Success rate: 100.00%
+  Total requests: 1
+  Retries: 0
+```
+
+**`workflow_example.py`**
 
-- `basic_usage.py` - Simple scraping example
-- `workflow_example.py` - Workflow orchestration
-- `advanced_example.py` - All features combined
+```
+💾 Saving 20 books to database...
+   - A Light in the ...... - £51.77
+   - Tipping the Velvet... - £53.74
+   - Soumission... - £50.10
+
+✅ Workflow completed successfully!
+📚 Extracted 20 books
+
+📊 Metrics:
+   Total requests: 3
+   Success rate: 100.00%
+   Average response time: 5.85s
+```
+
+**`specification_driven_example.py`**
+
+```
+Extracted 10 items (validated via Pydantic)
+  1. "The world as we have created it is a process of o... | Albert Einstein
+  2. "It is our choices, Harry, that show what we truly... | J.K. Rowling
+  3. "There are only two ways to live your life. One is... | Albert Einstein
+
+Metrics: 1 requests, 100.0% success
+```
+
+**`authenticated_quotes_example.py`**
+
+```
+Authenticated login: True
+Quotes extracted: 10
+1. Albert Einstein: "The world as we have created it is a process of our thinking. It cannot be chan...
+2. J.K. Rowling: "It is our choices, Harry, that show what we truly are, far more than our abilit...
+3. Albert Einstein: "There are only two ways to live your life. One is as though nothing is a miracl...
+```
+
+**`hybrid_extraction_example.py`**
+
+```
+Extracted (selector or LLM):
+  Title: A Light in the Attic
+  Price: £51.77
+  Availability: In stock (22 available)
+```
+
+**`llm_extraction_example.py`** *(requires MISTRAL_API_KEY)*
+
+```
+Generated schema: {'type': 'object', 'properties': {'quote_text': {'type': 'string', 'description': 'The text of the quote'}, 'author_name': {'type': 'string', 'description': 'The name of the author of the quote'}, 'tags': {'type': 'array', 'description': 'A list of tags associated with the quote'}}, 'required': ['quote_text', 'author_name', 'tags']}
+
+LLM extracted: [{'quote_text': 'The world as we have created it is a process of our thinking...', 'author_name': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}, ...]
+
+Semantic extraction: {'quote': 'The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.', 'author': 'Albert Einstein', 'tags': 'change, deep-thoughts, thinking, world'}
+```
+
+**`session_persistence_example.py`**
+
+```
+Session saved to /tmp/scrapeflow_quotes_session.json
+Loaded session: extracted 10 quotes (authenticated)
+```
 
 ### Example: Complete Book Scraper with All Features
 
@@ -896,30 +1258,42 @@ ScrapeFlow is built with a modular architecture:
 
 ```
 scrapeflow/
-├── engine.py          # Main ScrapeFlow engine
-├── ports.py           # Protocols for dependency inversion
-├── browser_runtime.py # Playwright runtime adapter
-├── workflow.py        # Workflow definition entities
+├── engine.py           # Main ScrapeFlow engine
+├── ports.py            # Protocols for dependency inversion
+├── browser_runtime.py  # Playwright runtime adapter
+├── workflow.py         # Workflow definition entities
 ├── workflow_executor.py # Workflow execution service
-├── config.py          # Configuration classes (incl. EthicalCrawlingConfig)
-├── specifications.py  # Pydantic specification-driven extraction
-├── schema_library.py  # Reusable schema definitions
-├── robots.py          # robots.txt parsing and enforcement
-├── registry.py        # Shared selector/component registry
-├── mcp_backend.py     # MCP integration extensibility
-├── anti_detection.py  # Anti-detection utilities
-├── rate_limiter.py    # Rate limiting implementation
-├── retry.py           # Retry logic and error classification
-├── monitoring.py      # Metrics, logging, alerting
-├── extractors.py      # Data extraction utilities
-└── exceptions.py      # Custom exceptions
-```
-
-For deeper design details, see [`ARCHITECTURE.md`](ARCHITECTURE.md).
+├── config.py           # Configuration (EthicalCrawling, Pagination, etc.)
+├── specifications.py   # SpecificationExtractor, HybridExtractor, FieldSpec
+├── schema_library.py   # Reusable schema definitions
+├── extractors.py       # Extractor, StructuredExtractor
+├── llm_extract.py      # Mistral LLM schema + extraction
+├── content_utils.py    # HTML cleaning for LLM
+├── mcp_backend.py      # MCPBackend, MistralLLMBackend
+├── pagination.py       # paginate() helper
+├── robots.py           # robots.txt parsing and enforcement
+├── registry.py         # Selectors, login handlers, pagination
+├── anti_detection.py   # Stealth mode, user agent rotation
+├── rate_limiter.py     # Rate limiting implementation
+├── retry.py            # Retry logic and error classification
+├── monitoring.py       # Metrics, logging, alerting
+└── exceptions.py       # Custom exceptions
+```
+
+For deeper design details, see [`ARCHITECTURE.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/ARCHITECTURE.md). Release notes are in [`CHANGELOG.md`](https://github.com/irfanalidv/scrapeflow-py/blob/main/CHANGELOG.md).
 
 ## 🤝 Contributing
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+Contributions are welcome! See [CONTRIBUTING.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/CONTRIBUTING.md)
+and the [Code of Conduct](https://github.com/irfanalidv/scrapeflow-py/blob/main/CODE_OF_CONDUCT.md).
+
+```bash
+git clone https://github.com/irfanalidv/scrapeflow-py.git && cd scrapeflow-py
+pip install -e ".[dev]" && playwright install chromium
+ruff check . && black --check . && pytest
+```
+
+Security issues: please follow [SECURITY.md](https://github.com/irfanalidv/scrapeflow-py/blob/main/SECURITY.md).
 
 1. Fork the repository
 2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
@@ -929,7 +1303,7 @@ Contributions are welcome! Please feel free to submit a Pull Request.
 
 ## 📝 License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+This project is licensed under the MIT License - see the [LICENSE](https://github.com/irfanalidv/scrapeflow-py/blob/main/LICENSE) file for details.
 
 ## 🙏 Acknowledgments