webtools-cli 1.0.0__tar.gz → 1.0.3__tar.gz

webtools_cli-1.0.3/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: webtools-cli
+Version: 1.0.3
+Summary: Advanced Web Intelligence & Scraping Toolkit with CLI and Web UI
+Author: Abhinav Adarsh
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/abhinavgautam08/webtools-cli
+Keywords: web-scraping,osint,seo,intelligence,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: flask
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: qrcode
+Requires-Dist: opencv-python
+Requires-Dist: numpy
+Requires-Dist: textblob
+Requires-Dist: Pillow
+Requires-Dist: mtranslate
+Requires-Dist: colorama
+Requires-Dist: pyreadline3; platform_system == "Windows"
+Provides-Extra: playwright
+Requires-Dist: playwright; extra == "playwright"
+Dynamic: license-file
+
+# WebTools CLI
+
+[![PyPI Version](https://img.shields.io/pypi/v/webtools-cli)](https://pypi.org/project/webtools-cli/)
+[![License](https://img.shields.io/github/license/abhinavgautam08/webtools-cli)](https://github.com/abhinavgautam08/webtools-cli/blob/main/LICENSE)
+[![Python Version](https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white)](https://pypi.org/project/webtools-cli/)
+
+![WebTools CLI Interface](CLI.png)
+
+WebTools CLI is an advanced web intelligence suite for researchers, OSINT enthusiasts, and developers. It brings the power of deep web analysis and automated scraping directly into your terminal, bridging the gap between a high-speed **Terminal UI** and a feature-rich **Cyber-themed Dashboard**.
+
+---
+
+## 🚀 Why WebTools CLI?
+
+- **🎯 Stealth & Speed**: Smart proxy rotation and Turbo-Fetch logic for evasion and performance.
+- **🧠 AI-Powered**: Automated content summarization, sentiment analysis, and readability scoring.
+- **🔧 Security-Centric**: Built-in honeypot detection, threat leveling, and image forensic analysis.
+- **💻 Terminal-First**: Designed for power users who live in the command line.
+- **🛡️ Cross-Platform**: Works seamlessly on Windows, Linux, and macOS (with auto-download for Windows tunnels).
+- **🔌 SPA Ready**: Automatic Playwright fallback for JavaScript-heavy sites like LinkedIn/Instagram.
+
+---
+
+## 📦 Installation
+
+See the installation guide for recommended system specifications.
+
+### Quick Install
+
+Install globally via pip:
+
+```bash
+pip install webtools-cli
+```
+
+To upgrade to the latest version:
+
+```bash
+pip install webtools-cli --upgrade
+```
+
+### Optional Dependencies
+
+For Single Page Application (SPA) support:
+
+```bash
+playwright install chromium
+```
+
+---
+
+## 📋 Key Features
+
+### Advanced Scraping & Stealth
+- **Smart Proxy Rotation**: Automatically rotates User-Agents and Proxies to evade detection.
+- **Turbo-Fetch**: Parallel chunk downloads for large media (Videos/Images).
+- **Deep Crawl**: Recursive link mapping up to 3 levels deep.
+- **Headless Fallback**: Integrated Playwright support for auth-walled or SPA environments.
+
+### Intelligence & Security Analysis
+- **OSINT Toolkit**: Auto-extract emails, phones, locations, social media, and tech stacks.
+- **SEO Auditor**: Page score, heading hierarchy, link integrity, and image alt-text auditing.
+- **Image Forensics**: CLI-based Error Level Analysis (ELA) and AI-likelihood detection.
+- **Honeypot Detector**: Identifies hidden traps and anti-bot measures (Cloudflare/CAPTCHAs).
+
+### Modern Experience
+- **Matrix Background**: "Flickering Grid" animated dashboard (Canvas-based).
+- **Responsive Preview**: Live rendering scaling for desktop and mobile viewpoints.
+- **History & Stats**: Phase-by-phase performance tracking and historical session management.
+
+---
+
+## 🚀 Getting Started
+
+### Basic Usage
+
+#### Launch Interactive Menu
+```bash
+webtools
+```
+
+#### Non-Interactive Script Mode
+```bash
+python -m webtools
+```
+
+### Slash Commands Reference
+
+Navigate the suite using quick terminal commands:
+
+| Command | Alias | Description |
+|---------|-------|-------------|
+| `/web` | `/w` | Launch **Web UI** (Cloudflare Tunnel + QR) |
+| `/cli` | `/c` | Launch **CLI Intelligence** scan |
+| `/image` | `/i` | **Image Forensics** & AI Likelihood |
+| `/history`| `/hi`| View and manage scan history |
+| `/help` | `/h` | Show full command documentation |
+| `/clear` | - | Purge all locally scraped data |
+| `/quit` | `/q` | Exit the application |
+
+---
+
+## ☁️ Deployment Options
+
+- **Local Development**: Run on your machine with a generated QR code for mobile access.
+- **Cloud Tunnels**: Automatic `cloudflared` integration to expose your UI globally.
+- **Google Colab**: Compatible with Colab for cloud-based scraping (see badge above).
+
+---
+
+## 🤝 Resources & Support
+
+- **[GitHub Repository](https://github.com/abhinavgautam08/webtools-cli)** - Source code and updates.
+- **[Issue Tracker](https://github.com/abhinavgautam08/webtools-cli/issues)** - Report bugs or request features.
+- **[License](./LICENSE)** - MIT License.
+
+---
+
+## ⚖️ Legal
+
+This tool is for **educational and testing purposes only**. Always respect `robots.txt` and the Terms of Service of the websites you scrape. Neither the author nor the contributors are responsible for any misuse of this tool.
+
+---
+
+<p align="center">
+  Built with ❤️ by <strong>Abhinav Adarsh</strong> and the open source community
+</p>

webtools_cli-1.0.3/README.md

@@ -0,0 +1,127 @@
+# WebTools CLI
+
+[![PyPI Version](https://img.shields.io/pypi/v/webtools-cli)](https://pypi.org/project/webtools-cli/)
+[![License](https://img.shields.io/github/license/abhinavgautam08/webtools-cli)](https://github.com/abhinavgautam08/webtools-cli/blob/main/LICENSE)
+[![Python Version](https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white)](https://pypi.org/project/webtools-cli/)
+
+![WebTools CLI Interface](CLI.png)
+
+WebTools CLI is an advanced web intelligence suite for researchers, OSINT enthusiasts, and developers. It brings the power of deep web analysis and automated scraping directly into your terminal, bridging the gap between a high-speed **Terminal UI** and a feature-rich **Cyber-themed Dashboard**.
+
+---
+
+## 🚀 Why WebTools CLI?
+
+- **🎯 Stealth & Speed**: Smart proxy rotation and Turbo-Fetch logic for evasion and performance.
+- **🧠 AI-Powered**: Automated content summarization, sentiment analysis, and readability scoring.
+- **🔧 Security-Centric**: Built-in honeypot detection, threat leveling, and image forensic analysis.
+- **💻 Terminal-First**: Designed for power users who live in the command line.
+- **🛡️ Cross-Platform**: Works seamlessly on Windows, Linux, and macOS (with auto-download for Windows tunnels).
+- **🔌 SPA Ready**: Automatic Playwright fallback for JavaScript-heavy sites like LinkedIn/Instagram.
+
+---
+
+## 📦 Installation
+
+See the installation guide for recommended system specifications.
+
+### Quick Install
+
+Install globally via pip:
+
+```bash
+pip install webtools-cli
+```
+
+To upgrade to the latest version:
+
+```bash
+pip install webtools-cli --upgrade
+```
+
+### Optional Dependencies
+
+For Single Page Application (SPA) support:
+
+```bash
+playwright install chromium
+```
+
+---
+
+## 📋 Key Features
+
+### Advanced Scraping & Stealth
+- **Smart Proxy Rotation**: Automatically rotates User-Agents and Proxies to evade detection.
+- **Turbo-Fetch**: Parallel chunk downloads for large media (Videos/Images).
+- **Deep Crawl**: Recursive link mapping up to 3 levels deep.
+- **Headless Fallback**: Integrated Playwright support for auth-walled or SPA environments.
+
+### Intelligence & Security Analysis
+- **OSINT Toolkit**: Auto-extract emails, phones, locations, social media, and tech stacks.
+- **SEO Auditor**: Page score, heading hierarchy, link integrity, and image alt-text auditing.
+- **Image Forensics**: CLI-based Error Level Analysis (ELA) and AI-likelihood detection.
+- **Honeypot Detector**: Identifies hidden traps and anti-bot measures (Cloudflare/CAPTCHAs).
+
+### Modern Experience
+- **Matrix Background**: "Flickering Grid" animated dashboard (Canvas-based).
+- **Responsive Preview**: Live rendering scaling for desktop and mobile viewpoints.
+- **History & Stats**: Phase-by-phase performance tracking and historical session management.
+
+---
+
+## 🚀 Getting Started
+
+### Basic Usage
+
+#### Launch Interactive Menu
+```bash
+webtools
+```
+
+#### Non-Interactive Script Mode
+```bash
+python -m webtools
+```
+
+### Slash Commands Reference
+
+Navigate the suite using quick terminal commands:
+
+| Command | Alias | Description |
+|---------|-------|-------------|
+| `/web` | `/w` | Launch **Web UI** (Cloudflare Tunnel + QR) |
+| `/cli` | `/c` | Launch **CLI Intelligence** scan |
+| `/image` | `/i` | **Image Forensics** & AI Likelihood |
+| `/history`| `/hi`| View and manage scan history |
+| `/help` | `/h` | Show full command documentation |
+| `/clear` | - | Purge all locally scraped data |
+| `/quit` | `/q` | Exit the application |
+
+---
+
+## ☁️ Deployment Options
+
+- **Local Development**: Run on your machine with a generated QR code for mobile access.
+- **Cloud Tunnels**: Automatic `cloudflared` integration to expose your UI globally.
+- **Google Colab**: Compatible with Colab for cloud-based scraping (see badge above).
+
+---
+
+## 🤝 Resources & Support
+
+- **[GitHub Repository](https://github.com/abhinavgautam08/webtools-cli)** - Source code and updates.
+- **[Issue Tracker](https://github.com/abhinavgautam08/webtools-cli/issues)** - Report bugs or request features.
+- **[License](./LICENSE)** - MIT License.
+
+---
+
+## ⚖️ Legal
+
+This tool is for **educational and testing purposes only**. Always respect `robots.txt` and the Terms of Service of the websites you scrape. Neither the author nor the contributors are responsible for any misuse of this tool.
+
+---
+
+<p align="center">
+  Built with ❤️ by <strong>Abhinav Adarsh</strong> and the open source community
+</p>

{webtools_cli-1.0.0 → webtools_cli-1.0.3}/pyproject.toml

@@ -4,10 +4,10 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "webtools-cli"
-version = "1.0.0"
+version = "1.0.3"
 description = "Advanced Web Intelligence & Scraping Toolkit with CLI and Web UI"
 readme = "README.md"
-license = {text = "MIT"}
+license = "MIT"
 requires-python = ">=3.9"
 authors = [
     {name = "Abhinav Adarsh"},
@@ -17,7 +17,6 @@ classifiers = [
     "Development Status :: 4 - Beta",
     "Environment :: Console",
     "Intended Audience :: Developers",
-    "License :: OSI Approved :: MIT License",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
@@ -46,7 +45,7 @@ playwright = ["playwright"]
 webtools = "webtools.cli:main"
 
 [project.urls]
-Homepage = "https://github.com/abhinavgautam08/webtools"
+Homepage = "https://github.com/abhinavgautam08/webtools-cli"
 
 [tool.setuptools.packages.find]
 include = ["webtools*"]

{webtools_cli-1.0.0 → webtools_cli-1.0.3}/webtools/core.py

@@ -4,7 +4,8 @@ sys.dont_write_bytecode = True
 # --- PACKAGE PATHS ---
 PACKAGE_DIR = os.path.dirname(os.path.abspath(__file__))
 DATA_DIR = os.path.join(os.path.expanduser('~'), '.webtools')
-os.makedirs(DATA_DIR, exist_ok=True)
+SCRAPED_DIR = os.path.join(DATA_DIR, 'scraped')
+os.makedirs(SCRAPED_DIR, exist_ok=True)
 try:
     from colorama import init, Fore, Style
     init(autoreset=True)
@@ -97,9 +98,8 @@ log.setLevel(logging.ERROR)
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
 # Directories setup kar rahe hain
-os.makedirs('webfiles/scraped', exist_ok=True)
-os.makedirs('webfiles/scraped/images', exist_ok=True)
-os.makedirs('webfiles/scraped/videos', exist_ok=True)
+os.makedirs(os.path.join(SCRAPED_DIR, 'images'), exist_ok=True)
+os.makedirs(os.path.join(SCRAPED_DIR, 'videos'), exist_ok=True)
 
 # --- PERFORMANCE AUDITOR ---
 class PerformanceTracker:
@@ -385,7 +385,7 @@ def serve_favicon():
 
 @app.route('/download/<path:filename>')
 def serve_scraped_file(filename):
-    return send_from_directory('webfiles/scraped', filename)
+    return send_from_directory(SCRAPED_DIR, filename)
 
 def scrape_with_playwright(url, proxy=None):
     if not PLAYWRIGHT_AVAILABLE:
@@ -561,9 +561,24 @@ def detect_tech_stack(soup, response):
     
     return list(stack)
 
+def ensure_textblob_corpora():
+    """Ensure necessary NLTK corpora for TextBlob are downloaded"""
+    try:
+        import nltk
+        needed = ['punkt', 'brown', 'averaged_perceptron_tagger']
+        for corpus in needed:
+            try:
+                nltk.data.find(f'tokenizers/{corpus}' if corpus == 'punkt' else f'corpora/{corpus}')
+            except (LookupError, AttributeError):
+                print(f"Downloading required AI data: {corpus}...")
+                nltk.download(corpus, quiet=True)
+    except:
+        pass
+
 def analyze_ai_content(text):
     """Text analyze karo (sentiment, summary, readability, aur keywords)"""
     try:
+        ensure_textblob_corpora()
         from textblob import TextBlob
         import re
         
@@ -816,9 +831,9 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
                             self.headers = {}
                     response = MockResponse(pw_html)
                 else:
-                    return jsonify({'error': f'Request failed and Playwright fallback failed: {str(e)}'}), 400
+                    return {'success': False, 'error': f'Request failed and Playwright fallback failed: {str(e)}'}
             else:
-                 return jsonify({'error': f'Request failed: {str(e)}'}), 400
+                 return {'success': False, 'error': f'Request failed: {str(e)}'}
         
         soup = BeautifulSoup(response.text, 'html.parser')
         perf_tracker.record_phase("HTML Parsing")
@@ -990,7 +1005,7 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
                                 else:
                                      filename = f"{base}_{uuid.uuid4().hex[:8]}{ext}"
 
-                            filepath = f'webfiles/scraped/videos/{filename}'
+                            filepath = os.path.join(SCRAPED_DIR, 'videos', filename)
                             
                             # Pehle TURBO FETCH try karo
                             if not download_file_turbo(v_url, filepath):
@@ -1161,7 +1176,7 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
                     
                     filename = f"{uuid.uuid4().hex[:8]}_{filename}"
 
-                    filepath = f'webfiles/scraped/images/{filename}'
+                    filepath = os.path.join(SCRAPED_DIR, 'images', filename)
                     with open(filepath, 'wb') as f:
                         f.write(content)
                     return (img_src, f'images/{filename}', f'/download/images/{filename}', image_hash, filepath)
@@ -1212,7 +1227,7 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
         # Image Tasks collect karo
         image_tasks = []
         if fetch_images:
-            os.makedirs('webfiles/scraped/images', exist_ok=True)
+            os.makedirs(os.path.join(SCRAPED_DIR, 'images'), exist_ok=True)
             
             # Exclude karne ke liye Video Posters ID karo
             poster_blacklist = set()
@@ -1277,7 +1292,7 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
         # Video Tasks collect karo
         video_tasks = []
         if fetch_videos:
-            os.makedirs('webfiles/scraped/videos', exist_ok=True)
+            os.makedirs(os.path.join(SCRAPED_DIR, 'videos'), exist_ok=True)
             # -------------------------------------------------------------------------
             # RESOURCE SNIFFER (Deep Scan)
             # Scans raw HTML/JS for hidden video links (mp4, m3u8, etc)
@@ -1461,11 +1476,11 @@ def execute_scrape_logic(url, fetch_images=False, fetch_videos=False, crawl_dept
             body.append(js_script)
         # Files save karo
         html_content = str(soup)
-        with open('webfiles/scraped/index.html', 'w', encoding='utf-8') as f:
+        with open(os.path.join(SCRAPED_DIR, 'index.html'), 'w', encoding='utf-8') as f:
             f.write(html_content)
-        with open('webfiles/scraped/style.css', 'w', encoding='utf-8') as f:
+        with open(os.path.join(SCRAPED_DIR, 'style.css'), 'w', encoding='utf-8') as f:
             f.write('\n\n'.join(css_content) or '/* No CSS found */')
-        with open('webfiles/scraped/script.js', 'w', encoding='utf-8') as f:
+        with open(os.path.join(SCRAPED_DIR, 'script.js'), 'w', encoding='utf-8') as f:
             f.write('\n\n'.join(js_content) or '// No JS found */')
         # Stats calculate karo
         def get_size(content):
@@ -1767,12 +1782,12 @@ def api_save():
 @app.route('/api/download-zip')
 def download_zip():
     try:
-        zip_path = '/tmp/scraped_files.zip'
+        zip_path = os.path.join(DATA_DIR, 'scraped_files.zip')
         with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
-            for root, dirs, files in os.walk('webfiles/scraped'):
+            for root, dirs, files in os.walk(SCRAPED_DIR):
                 for file in files:
                     file_path = os.path.join(root, file)
-                    arcname = os.path.relpath(file_path, 'webfiles/scraped')
+                    arcname = os.path.relpath(file_path, SCRAPED_DIR)
                     zipf.write(file_path, arcname)
         
         return send_file(zip_path, as_attachment=True, download_name='scraped_files.zip')
@@ -1781,12 +1796,10 @@ def download_zip():
 
 def clear_scraped_data():
     try:
-        folder = 'webfiles/scraped'
-        if os.path.exists(folder):
-            shutil.rmtree(folder)
-        os.makedirs('webfiles/scraped', exist_ok=True)
-        os.makedirs('webfiles/scraped/images', exist_ok=True)
-        os.makedirs('webfiles/scraped/videos', exist_ok=True)
+        if os.path.exists(SCRAPED_DIR):
+            shutil.rmtree(SCRAPED_DIR)
+        os.makedirs(os.path.join(SCRAPED_DIR, 'images'), exist_ok=True)
+        os.makedirs(os.path.join(SCRAPED_DIR, 'videos'), exist_ok=True)
         return True
     except Exception as e:
         print(f"Cleanup Error: {e}")
@@ -2156,11 +2169,20 @@ def start_cloudflare_tunnel(port):
         # OS ke hisaab se executable choose karo
         cf_executable = os.path.join(DATA_DIR, 'cloudflared.exe') if os.name == 'nt' else os.path.join(DATA_DIR, 'cloudflared')
         
-        # Agar missing ho (Linux/Colab) toh download karo
-        if not os.path.exists(cf_executable) and os.name != 'nt':
-            print("Downloading cloudflared...")
-            subprocess.run(['wget', '-q', '-O', cf_executable, 'https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64'])
-            subprocess.run(['chmod', '+x', cf_executable])
+        # Agar missing ho toh download karo
+        if not os.path.exists(cf_executable):
+            print(f"Downloading cloudflared for {os.name}...")
+            if os.name == 'nt':
+                # Windows binary download URL
+                win_url = "https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-windows-amd64.exe"
+                resp = requests.get(win_url, stream=True)
+                with open(cf_executable, 'wb') as f:
+                    for chunk in resp.iter_content(chunk_size=8192):
+                        f.write(chunk)
+            else:
+                # Linux binary download URL
+                subprocess.run(['wget', '-q', '-O', cf_executable, 'https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64'])
+                subprocess.run(['chmod', '+x', cf_executable])
         
         process = subprocess.Popen(
             [cf_executable, 'tunnel', '--protocol', 'http2', '--url', f'http://127.0.0.1:{port}'],

webtools_cli-1.0.3/webtools_cli.egg-info/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: webtools-cli
+Version: 1.0.3
+Summary: Advanced Web Intelligence & Scraping Toolkit with CLI and Web UI
+Author: Abhinav Adarsh
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/abhinavgautam08/webtools-cli
+Keywords: web-scraping,osint,seo,intelligence,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: flask
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: qrcode
+Requires-Dist: opencv-python
+Requires-Dist: numpy
+Requires-Dist: textblob
+Requires-Dist: Pillow
+Requires-Dist: mtranslate
+Requires-Dist: colorama
+Requires-Dist: pyreadline3; platform_system == "Windows"
+Provides-Extra: playwright
+Requires-Dist: playwright; extra == "playwright"
+Dynamic: license-file
+
+# WebTools CLI
+
+[![PyPI Version](https://img.shields.io/pypi/v/webtools-cli)](https://pypi.org/project/webtools-cli/)
+[![License](https://img.shields.io/github/license/abhinavgautam08/webtools-cli)](https://github.com/abhinavgautam08/webtools-cli/blob/main/LICENSE)
+[![Python Version](https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white)](https://pypi.org/project/webtools-cli/)
+
+![WebTools CLI Interface](CLI.png)
+
+WebTools CLI is an advanced web intelligence suite for researchers, OSINT enthusiasts, and developers. It brings the power of deep web analysis and automated scraping directly into your terminal, bridging the gap between a high-speed **Terminal UI** and a feature-rich **Cyber-themed Dashboard**.
+
+---
+
+## 🚀 Why WebTools CLI?
+
+- **🎯 Stealth & Speed**: Smart proxy rotation and Turbo-Fetch logic for evasion and performance.
+- **🧠 AI-Powered**: Automated content summarization, sentiment analysis, and readability scoring.
+- **🔧 Security-Centric**: Built-in honeypot detection, threat leveling, and image forensic analysis.
+- **💻 Terminal-First**: Designed for power users who live in the command line.
+- **🛡️ Cross-Platform**: Works seamlessly on Windows, Linux, and macOS (with auto-download for Windows tunnels).
+- **🔌 SPA Ready**: Automatic Playwright fallback for JavaScript-heavy sites like LinkedIn/Instagram.
+
+---
+
+## 📦 Installation
+
+See the installation guide for recommended system specifications.
+
+### Quick Install
+
+Install globally via pip:
+
+```bash
+pip install webtools-cli
+```
+
+To upgrade to the latest version:
+
+```bash
+pip install webtools-cli --upgrade
+```
+
+### Optional Dependencies
+
+For Single Page Application (SPA) support:
+
+```bash
+playwright install chromium
+```
+
+---
+
+## 📋 Key Features
+
+### Advanced Scraping & Stealth
+- **Smart Proxy Rotation**: Automatically rotates User-Agents and Proxies to evade detection.
+- **Turbo-Fetch**: Parallel chunk downloads for large media (Videos/Images).
+- **Deep Crawl**: Recursive link mapping up to 3 levels deep.
+- **Headless Fallback**: Integrated Playwright support for auth-walled or SPA environments.
+
+### Intelligence & Security Analysis
+- **OSINT Toolkit**: Auto-extract emails, phones, locations, social media, and tech stacks.
+- **SEO Auditor**: Page score, heading hierarchy, link integrity, and image alt-text auditing.
+- **Image Forensics**: CLI-based Error Level Analysis (ELA) and AI-likelihood detection.
+- **Honeypot Detector**: Identifies hidden traps and anti-bot measures (Cloudflare/CAPTCHAs).
+
+### Modern Experience
+- **Matrix Background**: "Flickering Grid" animated dashboard (Canvas-based).
+- **Responsive Preview**: Live rendering scaling for desktop and mobile viewpoints.
+- **History & Stats**: Phase-by-phase performance tracking and historical session management.
+
+---
+
+## 🚀 Getting Started
+
+### Basic Usage
+
+#### Launch Interactive Menu
+```bash
+webtools
+```
+
+#### Non-Interactive Script Mode
+```bash
+python -m webtools
+```
+
+### Slash Commands Reference
+
+Navigate the suite using quick terminal commands:
+
+| Command | Alias | Description |
+|---------|-------|-------------|
+| `/web` | `/w` | Launch **Web UI** (Cloudflare Tunnel + QR) |
+| `/cli` | `/c` | Launch **CLI Intelligence** scan |
+| `/image` | `/i` | **Image Forensics** & AI Likelihood |
+| `/history`| `/hi`| View and manage scan history |
+| `/help` | `/h` | Show full command documentation |
+| `/clear` | - | Purge all locally scraped data |
+| `/quit` | `/q` | Exit the application |
+
+---
+
+## ☁️ Deployment Options
+
+- **Local Development**: Run on your machine with a generated QR code for mobile access.
+- **Cloud Tunnels**: Automatic `cloudflared` integration to expose your UI globally.
+- **Google Colab**: Compatible with Colab for cloud-based scraping (see badge above).
+
+---
+
+## 🤝 Resources & Support
+
+- **[GitHub Repository](https://github.com/abhinavgautam08/webtools-cli)** - Source code and updates.
+- **[Issue Tracker](https://github.com/abhinavgautam08/webtools-cli/issues)** - Report bugs or request features.
+- **[License](./LICENSE)** - MIT License.
+
+---
+
+## ⚖️ Legal
+
+This tool is for **educational and testing purposes only**. Always respect `robots.txt` and the Terms of Service of the websites you scrape. Neither the author nor the contributors are responsible for any misuse of this tool.
+
+---
+
+<p align="center">
+  Built with ❤️ by <strong>Abhinav Adarsh</strong> and the open source community
+</p>

webtools_cli-1.0.0/PKG-INFO

@@ -1,110 +0,0 @@
-Metadata-Version: 2.4
-Name: webtools-cli
-Version: 1.0.0
-Summary: Advanced Web Intelligence & Scraping Toolkit with CLI and Web UI
-Author: Abhinav Adarsh
-License: MIT
-Project-URL: Homepage, https://github.com/abhinavgautam08/webtools
-Keywords: web-scraping,osint,seo,intelligence,cli
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Topic :: Internet :: WWW/HTTP
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: flask
-Requires-Dist: requests
-Requires-Dist: beautifulsoup4
-Requires-Dist: qrcode
-Requires-Dist: opencv-python
-Requires-Dist: numpy
-Requires-Dist: textblob
-Requires-Dist: Pillow
-Requires-Dist: mtranslate
-Requires-Dist: colorama
-Requires-Dist: pyreadline3; platform_system == "Windows"
-Provides-Extra: playwright
-Requires-Dist: playwright; extra == "playwright"
-Dynamic: license-file
-
-<p align="center">
-  <img src="Web_Tools.png" alt="WebTools CLI" width="180">
-</p>
-
-<h1 align="center">WebTools CLI</h1>
-
-<p align="center">
-  <strong>Advanced Web Intelligence & Scraping Toolkit</strong><br>
-  <em>OSINT - SEO - AI Analysis - Security Scanner</em>
-</p>
-
-<p align="center">
-  <img src="https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white" alt="Python">
-  <img src="https://img.shields.io/badge/license-MIT-green" alt="License">
-  <img src="https://img.shields.io/badge/version-1.0.0-cyan" alt="Version">
-</p>
-
----
-
-## Install
-
-```bash
-pip install webtools-cli
-```
-
-## Usage
-
-```bash
-# Launch CLI
-webtools
-
-# Or via Python module
-python -m webtools
-```
-
-## Features
-
-| Feature | Description |
-|---------|-------------|
-| **Web Mode** | Full web UI with Cloudflare tunnel + QR code sharing |
-| **CLI Intelligence** | Deep scan any URL from terminal |
-| **Security Scanner** | Threat detection, honeypot traps, CSRF checks |
-| **SEO Analyzer** | Score, headings, broken links, image audit |
-| **AI Analysis** | Sentiment, readability, keywords, summarization |
-| **OSINT** | Emails, phones, locations, social media, tech stack |
-| **Smart Media** | Image quality filter + video deep-scan with sniffer |
-| **Proxy Intelligence** | Smart proxy rotation with learning algorithm |
-| **Playwright Fallback** | Handles SPAs and auth walls automatically |
-| **Performance Tracker** | Phase-by-phase timing with historical stats |
-
-## CLI Commands
-
-| Command | Description |
-|---------|-------------|
-| `/web` or `/w` | Launch Web UI mode |
-| `/cli` or `/c` | Launch CLI Intelligence mode |
-| `/image` or `/i` | Image Forensics & AI Detection |
-| `/help` or `/h` | Show all commands |
-| `/history` or `/hi` | View scan history |
-| `/clear` | Purge scraped data |
-| `/quit` or `/q` | Exit |
-
-## Requirements
-
-- Python 3.9+
-- Optional: `playwright` for SPA/auth wall bypass
-
-## Author
-
-**Abhinav Adarsh**
-
-## License
-
-MIT

webtools_cli-1.0.0/README.md

@@ -1,74 +0,0 @@
-<p align="center">
-  <img src="Web_Tools.png" alt="WebTools CLI" width="180">
-</p>
-
-<h1 align="center">WebTools CLI</h1>
-
-<p align="center">
-  <strong>Advanced Web Intelligence & Scraping Toolkit</strong><br>
-  <em>OSINT - SEO - AI Analysis - Security Scanner</em>
-</p>
-
-<p align="center">
-  <img src="https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white" alt="Python">
-  <img src="https://img.shields.io/badge/license-MIT-green" alt="License">
-  <img src="https://img.shields.io/badge/version-1.0.0-cyan" alt="Version">
-</p>
-
----
-
-## Install
-
-```bash
-pip install webtools-cli
-```
-
-## Usage
-
-```bash
-# Launch CLI
-webtools
-
-# Or via Python module
-python -m webtools
-```
-
-## Features
-
-| Feature | Description |
-|---------|-------------|
-| **Web Mode** | Full web UI with Cloudflare tunnel + QR code sharing |
-| **CLI Intelligence** | Deep scan any URL from terminal |
-| **Security Scanner** | Threat detection, honeypot traps, CSRF checks |
-| **SEO Analyzer** | Score, headings, broken links, image audit |
-| **AI Analysis** | Sentiment, readability, keywords, summarization |
-| **OSINT** | Emails, phones, locations, social media, tech stack |
-| **Smart Media** | Image quality filter + video deep-scan with sniffer |
-| **Proxy Intelligence** | Smart proxy rotation with learning algorithm |
-| **Playwright Fallback** | Handles SPAs and auth walls automatically |
-| **Performance Tracker** | Phase-by-phase timing with historical stats |
-
-## CLI Commands
-
-| Command | Description |
-|---------|-------------|
-| `/web` or `/w` | Launch Web UI mode |
-| `/cli` or `/c` | Launch CLI Intelligence mode |
-| `/image` or `/i` | Image Forensics & AI Detection |
-| `/help` or `/h` | Show all commands |
-| `/history` or `/hi` | View scan history |
-| `/clear` | Purge scraped data |
-| `/quit` or `/q` | Exit |
-
-## Requirements
-
-- Python 3.9+
-- Optional: `playwright` for SPA/auth wall bypass
-
-## Author
-
-**Abhinav Adarsh**
-
-## License
-
-MIT

webtools_cli-1.0.0/webtools_cli.egg-info/PKG-INFO

@@ -1,110 +0,0 @@
-Metadata-Version: 2.4
-Name: webtools-cli
-Version: 1.0.0
-Summary: Advanced Web Intelligence & Scraping Toolkit with CLI and Web UI
-Author: Abhinav Adarsh
-License: MIT
-Project-URL: Homepage, https://github.com/abhinavgautam08/webtools
-Keywords: web-scraping,osint,seo,intelligence,cli
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Topic :: Internet :: WWW/HTTP
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: flask
-Requires-Dist: requests
-Requires-Dist: beautifulsoup4
-Requires-Dist: qrcode
-Requires-Dist: opencv-python
-Requires-Dist: numpy
-Requires-Dist: textblob
-Requires-Dist: Pillow
-Requires-Dist: mtranslate
-Requires-Dist: colorama
-Requires-Dist: pyreadline3; platform_system == "Windows"
-Provides-Extra: playwright
-Requires-Dist: playwright; extra == "playwright"
-Dynamic: license-file
-
-<p align="center">
-  <img src="Web_Tools.png" alt="WebTools CLI" width="180">
-</p>
-
-<h1 align="center">WebTools CLI</h1>
-
-<p align="center">
-  <strong>Advanced Web Intelligence & Scraping Toolkit</strong><br>
-  <em>OSINT - SEO - AI Analysis - Security Scanner</em>
-</p>
-
-<p align="center">
-  <img src="https://img.shields.io/badge/python-3.9+-blue?logo=python&logoColor=white" alt="Python">
-  <img src="https://img.shields.io/badge/license-MIT-green" alt="License">
-  <img src="https://img.shields.io/badge/version-1.0.0-cyan" alt="Version">
-</p>
-
----
-
-## Install
-
-```bash
-pip install webtools-cli
-```
-
-## Usage
-
-```bash
-# Launch CLI
-webtools
-
-# Or via Python module
-python -m webtools
-```
-
-## Features
-
-| Feature | Description |
-|---------|-------------|
-| **Web Mode** | Full web UI with Cloudflare tunnel + QR code sharing |
-| **CLI Intelligence** | Deep scan any URL from terminal |
-| **Security Scanner** | Threat detection, honeypot traps, CSRF checks |
-| **SEO Analyzer** | Score, headings, broken links, image audit |
-| **AI Analysis** | Sentiment, readability, keywords, summarization |
-| **OSINT** | Emails, phones, locations, social media, tech stack |
-| **Smart Media** | Image quality filter + video deep-scan with sniffer |
-| **Proxy Intelligence** | Smart proxy rotation with learning algorithm |
-| **Playwright Fallback** | Handles SPAs and auth walls automatically |
-| **Performance Tracker** | Phase-by-phase timing with historical stats |
-
-## CLI Commands
-
-| Command | Description |
-|---------|-------------|
-| `/web` or `/w` | Launch Web UI mode |
-| `/cli` or `/c` | Launch CLI Intelligence mode |
-| `/image` or `/i` | Image Forensics & AI Detection |
-| `/help` or `/h` | Show all commands |
-| `/history` or `/hi` | View scan history |
-| `/clear` | Purge scraped data |
-| `/quit` or `/q` | Exit |
-
-## Requirements
-
-- Python 3.9+
-- Optional: `playwright` for SPA/auth wall bypass
-
-## Author
-
-**Abhinav Adarsh**
-
-## License
-
-MIT