evoscrape 0.1.0__tar.gz

evoscrape-0.1.0/PKG-INFO

@@ -0,0 +1,5 @@
+Metadata-Version: 2.4
+Name: evoscrape
+Version: 0.1.0
+Summary: AI-powered self-healing web scraper
+Requires-Python: >=3.10

evoscrape-0.1.0/README.md

@@ -0,0 +1,666 @@
+\# EvoScrape 🤖
+
+
+
+AI-Assisted Self-Healing Web Scraper
+
+
+
+\## 📌 About
+
+
+
+EvoScrape is an AI-assisted web scraper that can automatically recover when a website changes its HTML structure or CSS selectors.
+
+
+
+Traditional web scrapers can break when a website changes something like:
+
+
+
+```text
+
+\[data-testid="price"]
+
+```
+
+
+
+to:
+
+
+
+```text
+
+.product-price
+
+```
+
+
+
+EvoScrape detects the broken selector, analyzes the current webpage, finds possible replacement elements, scores them, validates the best candidate, saves the new selector, and continues scraping.
+
+
+
+\## ✨ Features
+
+
+
+\* 🔎 Live webpage inspection
+
+\* 🎯 CSS selector validation
+
+\* 🧠 Intelligent candidate scoring
+
+\* 🔧 Automatic selector healing
+
+\* 🧪 Replacement selector validation
+
+\* 💾 Automatic selector updates
+
+\* 📊 Healing confidence scores
+
+\* 📝 Healing history logs
+
+\* 📦 Product data extraction
+
+\* 🚀 One-command execution
+
+
+
+\## 🧠 How EvoScrape Works
+
+
+
+```text
+
+Saved Selector
+
+      │
+
+      ▼
+
+Check Selector
+
+      │
+
+      ├── ✅ Works ───────► Extract Data
+
+      │
+
+      └── ❌ Failed
+
+            │
+
+            ▼
+
+     Discover Candidates
+
+            │
+
+            ▼
+
+      Score Candidates
+
+            │
+
+            ▼
+
+       Best Candidate
+
+            │
+
+            ▼
+
+     Validate Selector
+
+            │
+
+            ▼
+
+      Save New Selector
+
+            │
+
+            ▼
+
+       Extract Data
+
+```
+
+
+
+\## 🏗️ Project Structure
+
+
+
+```text
+
+EvoScrape/
+
+│
+
+├── run.py
+
+├── demo.py
+
+├── README.md
+
+│
+
+├── ai/
+
+│   ├── \_\_init\_\_.py
+
+│   ├── auto\_heal.py
+
+│   ├── healer.py
+
+│   ├── scorer.py
+
+│   └── validator.py
+
+│
+
+├── config/
+
+│   ├── selectors.json
+
+│   └── healing\_log.json
+
+│
+
+└── website/
+
+    ├── index.html
+
+    └── index\_backup.html
+
+```
+
+
+
+\## ⚙️ Technologies Used
+
+
+
+\* Python
+
+\* Playwright
+
+\* BeautifulSoup
+
+\* HTML
+
+\* CSS Selectors
+
+\* JSON
+
+\* PowerShell
+
+
+
+\## 📦 Requirements
+
+
+
+Python 3.10 or newer.
+
+
+
+Install the required packages:
+
+
+
+```bash
+
+pip install playwright beautifulsoup4
+
+```
+
+
+
+Install the Playwright browser:
+
+
+
+```bash
+
+playwright install chromium
+
+```
+
+
+
+\## 🚀 Running EvoScrape
+
+
+
+\### Step 1 — Start the test website
+
+
+
+Open a PowerShell terminal inside the EvoScrape folder and run:
+
+
+
+```powershell
+
+python -m http.server 8000 --directory website
+
+```
+
+
+
+Keep this terminal running.
+
+
+
+\### Step 2 — Open another PowerShell
+
+
+
+Go to the project folder:
+
+
+
+```powershell
+
+cd C:\\Users\\rick\\Desktop\\EvoScrape
+
+```
+
+
+
+Activate the virtual environment:
+
+
+
+```powershell
+
+.\\venv\\Scripts\\Activate.ps1
+
+```
+
+
+
+\### Step 3 — Run EvoScrape
+
+
+
+```powershell
+
+python .\\run.py
+
+```
+
+
+
+\## 🧪 Self-Healing Demo
+
+
+
+The project contains a simulated website redesign.
+
+
+
+\### Original selector
+
+
+
+```text
+
+\[data-testid="price"]
+
+```
+
+
+
+\### New selector
+
+
+
+```text
+
+.product-price
+
+```
+
+
+
+When the old selector stops working, EvoScrape detects the failure.
+
+
+
+It discovers candidates such as:
+
+
+
+```text
+
+.product
+
+.product-price
+
+```
+
+
+
+The candidate scorer evaluates them.
+
+
+
+Example:
+
+
+
+```text
+
+Candidate:
+
+{'tag': 'div',
+
+ 'text': '₹49,999',
+
+ 'data-testid': None,
+
+ 'id': None,
+
+ 'class': \['product-price']}
+
+
+
+Score: 85
+
+
+
+Reasons:
+
+\- exact value match
+
+\- price-like value
+
+\- price-related class
+
+```
+
+
+
+The system chooses:
+
+
+
+```text
+
+.product-price
+
+```
+
+
+
+It then validates the selector and saves it automatically.
+
+
+
+\## 📊 Example Output
+
+
+
+```text
+
+================================
+
+        EVOSCRAPE AI
+
+================================
+
+
+
+🔧 Checking scraper health...
+
+
+
+🔎 Checking saved selector:
+
+\[data-testid="price"]
+
+
+
+❌ Selector failed.
+
+Reason: Selector found no elements
+
+
+
+🔧 AUTO-HEAL ACTIVATED
+
+
+
+🔎 Candidates discovered:
+
+
+
+Candidate:
+
+{'tag': 'div', 'text': '₹49,999',
+
+ 'data-testid': None,
+
+ 'id': None,
+
+ 'class': \['product-price']}
+
+
+
+Score: 85
+
+
+
+🏆 BEST REPLACEMENT
+
+Element : div
+
+Value   : ₹49,999
+
+Selector: .product-price
+
+Score   : 85
+
+
+
+🧪 Validating replacement selector...
+
+
+
+❤️ AUTO-HEAL SUCCESSFUL
+
+Old: \[data-testid="price"]
+
+New: .product-price
+
+
+
+💾 Selector saved!
+
+File: config/selectors.json
+
+
+
+📊 Healing report saved!
+
+File: config/healing\_log.json
+
+
+
+🚀 Starting extraction...
+
+
+
+================================
+
+       EVOSCRAPE RESULT
+
+================================
+
+
+
+📦 Product : Gaming Laptop
+
+💰 Price   : ₹49,999
+
+⭐ Rating  : 4.5
+
+
+
+🔎 Selector : .product-price
+
+
+
+✅ SCRAPING COMPLETE
+
+```
+
+
+
+\## 📊 Healing Log
+
+
+
+Every successful healing operation can be recorded in:
+
+
+
+```text
+
+config/healing\_log.json
+
+```
+
+
+
+Example:
+
+
+
+```json
+
+{
+
+    "field": "price",
+
+    "old\_selector": "\[data-testid=\\"price\\"]",
+
+    "new\_selector": ".product-price",
+
+    "confidence": 0.85,
+
+    "reasons": \[
+
+        "exact value match",
+
+        "price-like value",
+
+        "price-related class"
+
+    ],
+
+    "status": "healed"
+
+}
+
+```
+
+
+
+
+
+\## 🎯 Why EvoScrape?
+
+
+
+Traditional scrapers often depend on fixed selectors.
+
+
+
+For example:
+
+
+
+```html
+
+<span data-testid="price">₹49,999</span>
+
+```
+
+
+
+If a website redesign changes the element to:
+
+
+
+```html
+
+<div class="product-price">₹49,999</div>
+
+```
+
+
+
+the original selector may stop working.
+
+
+
+EvoScrape attempts to recover automatically instead of requiring the developer to manually update the scraper.
+
+
+
+\## 🔮 Future Improvements
+
+
+
+Future versions could include:
+
+
+
+\* 🤖 LLM-powered element reasoning
+
+\* 🧠 More advanced semantic matching
+
+\* 🔄 Automatic retries
+
+\* 📄 Multiple-page scraping
+
+\* 📦 CSV and JSON exports
+
+\* 🖼️ Screenshot-based element detection
+
+\* 🎯 Multiple-field healing
+
+\* 🌐 Support for more complex websites
+
+\* 📊 Web dashboard for healing history
+
+\* 📈 Advanced confidence thresholds
+
+
+
+\## 🎓 Educational Purpose
+
+
+
+EvoScrape is an educational project exploring how AI-assisted systems can make web automation more resilient to website changes.
+
+
+
+The current version uses deterministic scoring and validation rather than relying on an external AI API.
+
+
+
+\## 📜 License
+
+
+
+This project is intended for educational and experimental purposes.
+
+
+

evoscrape-0.1.0/evoscrape/__init__.py

@@ -0,0 +1,5 @@
+"""
+EvoScrape - AI-powered self-healing web scraper.
+"""
+
+__version__ = "0.1.0"

evoscrape-0.1.0/evoscrape.egg-info/PKG-INFO

@@ -0,0 +1,5 @@
+Metadata-Version: 2.4
+Name: evoscrape
+Version: 0.1.0
+Summary: AI-powered self-healing web scraper
+Requires-Python: >=3.10

evoscrape-0.1.0/evoscrape.egg-info/SOURCES.txt

@@ -0,0 +1,7 @@
+README.md
+pyproject.toml
+evoscrape/__init__.py
+evoscrape.egg-info/PKG-INFO
+evoscrape.egg-info/SOURCES.txt
+evoscrape.egg-info/dependency_links.txt
+evoscrape.egg-info/top_level.txt

evoscrape-0.1.0/evoscrape.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

evoscrape-0.1.0/evoscrape.egg-info/top_level.txt

@@ -0,0 +1 @@
+evoscrape

evoscrape-0.1.0/pyproject.toml

@@ -0,0 +1,12 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "evoscrape"
+version = "0.1.0"
+description = "AI-powered self-healing web scraper"
+requires-python = ">=3.10"
+
+[tool.setuptools]
+packages = ["evoscrape"]

evoscrape-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+