webscout 6.4__tar.gz → 6.5__tar.gz

{webscout-6.4/webscout.egg-info → webscout-6.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: webscout
-Version: 6.4
+Version: 6.5
 Summary: Search for anything using Google, DuckDuckGo, phind.com, Contains AI models, can transcribe yt videos, temporary email and phone number generation, has TTS support, webai (terminal gpt and open interpreter) and offline LLMs and more
 Author: OEvortex
 Author-email: helpingai5@gmail.com
@@ -28,10 +28,7 @@ Requires-Dist: mistune
 Requires-Dist: tenacity
 Requires-Dist: curl_cffi
 Requires-Dist: nest-asyncio
-Requires-Dist: g4f>=0.2.2.3
-Requires-Dist: g4f[webdriver]
 Requires-Dist: rich
-Requires-Dist: beautifulsoup4
 Requires-Dist: markdownify
 Requires-Dist: requests
 Requires-Dist: google-generativeai
@@ -39,12 +36,10 @@ Requires-Dist: lxml>=5.2.2
 Requires-Dist: termcolor
 Requires-Dist: orjson
 Requires-Dist: PyYAML
-Requires-Dist: appdirs
 Requires-Dist: tls_client
 Requires-Dist: clipman
 Requires-Dist: playsound
 Requires-Dist: ollama
-Requires-Dist: pyfiglet
 Requires-Dist: pillow
 Requires-Dist: bson
 Requires-Dist: cloudscraper
@@ -53,10 +48,10 @@ Requires-Dist: openai
 Requires-Dist: prompt-toolkit
 Requires-Dist: primp
 Requires-Dist: pyreqwest_impersonate
-Requires-Dist: lxml_html_clean
 Requires-Dist: gradio_client
 Requires-Dist: psutil
 Requires-Dist: yaspin
+Requires-Dist: websocket
 Provides-Extra: dev
 Requires-Dist: ruff>=0.1.6; extra == "dev"
 Requires-Dist: pytest>=7.4.2; extra == "dev"
@@ -103,12 +98,11 @@ Requires-Dist: unicorn; extra == "local"
 </div>
 
 ## 🚀 Features
-* **Comprehensive Search:** Leverage Google, DuckDuckGo, and Phind.com for diverse search results.
+* **Comprehensive Search:** Leverage Google, DuckDuckGo for diverse search results.
 * **AI Powerhouse:** Access and interact with various AI models, including OpenAI, Cohere, and more.
-* **YouTube Toolkit:** Transcribe YouTube videos effortlessly and download audio/video content.
+* **[YouTube Toolkit](webscout/Extra/YTToolkit):** Advanced YouTube video and transcript management with multi-language support, versatile downloading, and intelligent data extraction
 * **Tempmail & Temp Number:** Generate temporary email addresses and phone numbers for enhanced privacy.
-* **Text-to-Speech (TTS):** Convert text into natural-sounding speech using various TTS providers.
-* **WebAI:** Experience the power of terminal-based GPT and an open interpreter for code execution and more.
+* **[Text-to-Speech (TTS)](webscout/Provider/TTS/README.md):** Convert text into natural-sounding speech using multiple AI-powered providers like ElevenLabs, StreamElements, and Voicepods.
 * **Offline LLMs:** Utilize powerful language models offline with GGUF support.
 * **Extensive Provider Ecosystem:** Explore a vast collection of providers, including BasedGPT, DeepSeek, and many others.
 * **Local LLM Execution:** Run GGUF models locally with minimal configuration.
@@ -121,6 +115,7 @@ Requires-Dist: unicorn; extra == "local"
 * **[LitLogger](webscout/litlogger/Readme.md):** Simplifies logging with customizable formats and color schemes
 * **[LitAgent](webscout/litagent/Readme.md):** Powerful and modern user agent generator that keeps your requests fresh and undetectable
 * **[Text-to-Image](webscout/Provider/TTI/README.md):** Generate high-quality images using a wide range of AI art providers
+* **[MarkdownLite](webscout/Extra/markdownlite/README.md):** Powerful HTML to Markdown conversion library with advanced parsing and structured output
 
 ## ⚙️ Installation
 ```python
@@ -226,29 +221,7 @@ python -m webscout --help
 
 [Go To TOP](#webscout-️)
 
-## ⬇️ YTdownloader 
 
-```python
-from os import rename, getcwd
-from webscout import YTdownloader
-def download_audio(video_id):
-    youtube_link = video_id 
-    handler = YTdownloader.Handler(query=youtube_link)
-    for third_query_data in handler.run(format='mp3', quality='128kbps', limit=1):
-        audio_path = handler.save(third_query_data, dir=getcwd())  
-        rename(audio_path, "audio.mp3")
-
-def download_video(video_id):
-    youtube_link = video_id 
-    handler = YTdownloader.Handler(query=youtube_link)
-    for third_query_data in handler.run(format='mp4', quality='auto', limit=1):
-        video_path = handler.save(third_query_data, dir=getcwd())  
-        rename(video_path, "video.mp4")
-        
-if __name__ == "__main__":
-    # download_audio("https://www.youtube.com/watch?v=c0tMvzB0OKw")
-    download_video("https://www.youtube.com/watch?v=c0tMvzB0OKw")
-```
 
 ## ☀️ Weather
 
@@ -304,20 +277,7 @@ if __name__ == "__main__":
     asyncio.run(main())
 ```
 
-## 📝 Transcriber
 
-The `transcriber` function in Webscout is a handy tool that transcribes YouTube videos. 
-
-**Example:**
-
-```python
-from webscout import YTTranscriber
-yt = YTTranscriber()
-from rich import print
-video_url = input("Enter the YouTube video URL: ") 
-transcript = yt.get_transcript(video_url, languages=None) 
-print(transcript)
-```
 
 ## 🔍 GoogleS (formerly DWEBS)
 
@@ -330,16 +290,6 @@ for result in results:
     print(result)
 ```
 
-### BingS
-
-```python
-from webscout import BingS
-from rich import print
-searcher = BingS()
-results = searcher.search("HelpingAI-9B", max_results=20, extract_webpage_text=True, max_extract_characters=1000)
-for result in results:
-    print(result)
-```
 
 ## 🦆 WEBS and AsyncWEBS
 
@@ -956,18 +906,12 @@ ai = BLACKBOXAI(
     model=None # You can specify a model if needed
 )
 
-# Start an infinite loop for continuous interaction
-while True:
-    # Define a prompt to send to the AI
-    prompt = input("Enter your prompt: ")
-    
-    # Check if the user wants to exit the loop
-    if prompt.lower() == "exit":
-        break
-    
-    # Use the 'chat' method to send the prompt and receive a response
-    r = ai.chat(prompt)
-    print(r)
+
+# Define a prompt to send to the AI
+prompt = "Tell me about india"
+# Use the 'chat' method to send the prompt and receive a response
+r = ai.chat(prompt)
+print(r)
 ```
 
 ###  ❓ `PERPLEXITY` - Search with PERPLEXITY

{webscout-6.4 → webscout-6.5}/README.md

@@ -34,12 +34,11 @@
 </div>
 
 ## 🚀 Features
-* **Comprehensive Search:** Leverage Google, DuckDuckGo, and Phind.com for diverse search results.
+* **Comprehensive Search:** Leverage Google, DuckDuckGo for diverse search results.
 * **AI Powerhouse:** Access and interact with various AI models, including OpenAI, Cohere, and more.
-* **YouTube Toolkit:** Transcribe YouTube videos effortlessly and download audio/video content.
+* **[YouTube Toolkit](webscout/Extra/YTToolkit):** Advanced YouTube video and transcript management with multi-language support, versatile downloading, and intelligent data extraction
 * **Tempmail & Temp Number:** Generate temporary email addresses and phone numbers for enhanced privacy.
-* **Text-to-Speech (TTS):** Convert text into natural-sounding speech using various TTS providers.
-* **WebAI:** Experience the power of terminal-based GPT and an open interpreter for code execution and more.
+* **[Text-to-Speech (TTS)](webscout/Provider/TTS/README.md):** Convert text into natural-sounding speech using multiple AI-powered providers like ElevenLabs, StreamElements, and Voicepods.
 * **Offline LLMs:** Utilize powerful language models offline with GGUF support.
 * **Extensive Provider Ecosystem:** Explore a vast collection of providers, including BasedGPT, DeepSeek, and many others.
 * **Local LLM Execution:** Run GGUF models locally with minimal configuration.
@@ -52,6 +51,7 @@
 * **[LitLogger](webscout/litlogger/Readme.md):** Simplifies logging with customizable formats and color schemes
 * **[LitAgent](webscout/litagent/Readme.md):** Powerful and modern user agent generator that keeps your requests fresh and undetectable
 * **[Text-to-Image](webscout/Provider/TTI/README.md):** Generate high-quality images using a wide range of AI art providers
+* **[MarkdownLite](webscout/Extra/markdownlite/README.md):** Powerful HTML to Markdown conversion library with advanced parsing and structured output
 
 ## ⚙️ Installation
 ```python
@@ -157,29 +157,7 @@ python -m webscout --help
 
 [Go To TOP](#webscout-️)
 
-## ⬇️ YTdownloader 
 
-```python
-from os import rename, getcwd
-from webscout import YTdownloader
-def download_audio(video_id):
-    youtube_link = video_id 
-    handler = YTdownloader.Handler(query=youtube_link)
-    for third_query_data in handler.run(format='mp3', quality='128kbps', limit=1):
-        audio_path = handler.save(third_query_data, dir=getcwd())  
-        rename(audio_path, "audio.mp3")
-
-def download_video(video_id):
-    youtube_link = video_id 
-    handler = YTdownloader.Handler(query=youtube_link)
-    for third_query_data in handler.run(format='mp4', quality='auto', limit=1):
-        video_path = handler.save(third_query_data, dir=getcwd())  
-        rename(video_path, "video.mp4")
-        
-if __name__ == "__main__":
-    # download_audio("https://www.youtube.com/watch?v=c0tMvzB0OKw")
-    download_video("https://www.youtube.com/watch?v=c0tMvzB0OKw")
-```
 
 ## ☀️ Weather
 
@@ -235,20 +213,7 @@ if __name__ == "__main__":
     asyncio.run(main())
 ```
 
-## 📝 Transcriber
 
-The `transcriber` function in Webscout is a handy tool that transcribes YouTube videos. 
-
-**Example:**
-
-```python
-from webscout import YTTranscriber
-yt = YTTranscriber()
-from rich import print
-video_url = input("Enter the YouTube video URL: ") 
-transcript = yt.get_transcript(video_url, languages=None) 
-print(transcript)
-```
 
 ## 🔍 GoogleS (formerly DWEBS)
 
@@ -261,16 +226,6 @@ for result in results:
     print(result)
 ```
 
-### BingS
-
-```python
-from webscout import BingS
-from rich import print
-searcher = BingS()
-results = searcher.search("HelpingAI-9B", max_results=20, extract_webpage_text=True, max_extract_characters=1000)
-for result in results:
-    print(result)
-```
 
 ## 🦆 WEBS and AsyncWEBS
 
@@ -887,18 +842,12 @@ ai = BLACKBOXAI(
     model=None # You can specify a model if needed
 )
 
-# Start an infinite loop for continuous interaction
-while True:
-    # Define a prompt to send to the AI
-    prompt = input("Enter your prompt: ")
-    
-    # Check if the user wants to exit the loop
-    if prompt.lower() == "exit":
-        break
-    
-    # Use the 'chat' method to send the prompt and receive a response
-    r = ai.chat(prompt)
-    print(r)
+
+# Define a prompt to send to the AI
+prompt = "Tell me about india"
+# Use the 'chat' method to send the prompt and receive a response
+r = ai.chat(prompt)
+print(r)
 ```
 
 ###  ❓ `PERPLEXITY` - Search with PERPLEXITY

{webscout-6.4 → webscout-6.5}/setup.py

@@ -34,10 +34,7 @@ setup(
         "tenacity",
         "curl_cffi",
         "nest-asyncio",
-        "g4f>=0.2.2.3",
-        "g4f[webdriver]",
         "rich",
-        "beautifulsoup4",
         "markdownify",
         "requests",
         "google-generativeai",
@@ -45,12 +42,10 @@ setup(
         "termcolor",
         "orjson",
         "PyYAML",
-        "appdirs",
         "tls_client",
         "clipman",
         "playsound",
         "ollama",
-        "pyfiglet",
         "pillow",
         "bson",
         "cloudscraper",
@@ -59,12 +54,11 @@ setup(
         "prompt-toolkit",
         "primp",
         "pyreqwest_impersonate",
-        "lxml_html_clean",
         "gradio_client",
         "psutil",
         "yaspin",
+        "websocket"
         # "cerebras_cloud_sdk",
-
     ],
     entry_points={
         "console_scripts": [

{webscout-6.4 → webscout-6.5}/webscout/AIutel.py

@@ -5,14 +5,11 @@ import subprocess
 import logging
 import threading
 import time
-import appdirs
 import datetime
 import re
 import sys
-import click
 from rich.markdown import Markdown
 from rich.console import Console
-import g4f
 from typing import List, Tuple, Union
 from typing import NoReturn
 import requests
@@ -21,49 +18,9 @@ from playsound import playsound
 from time import sleep as wait
 import pathlib
 import urllib.parse
-appdir = appdirs.AppDirs("AIWEBS", "webscout")
-
-default_path = appdir.user_cache_dir
-
-if not os.path.exists(default_path):
-    os.makedirs(default_path)
-webai = [
-   "leo",
-   "openai",
-   "opengpt",
-   "koboldai",
-   "gemini",
-   "phind",
-   "blackboxai",
-   "g4fauto",
-   "perplexity",
-   "groq",
-   "reka",
-   "cohere",
-   "yepchat",
-   "you",
-   "xjai",
-   "thinkany",
-   "berlin4h",
-   "chatgptuk",
-   "auto",
-   "poe",
-   "basedgpt",
-   "deepseek",
-   "deepinfra",
-   "vtlchat",
-   "geminiflash",
-   "geminipro",
-   "ollama",
-   "andi",
-   "llama3"
-]
-
-gpt4free_providers = [
-    provider.__name__ for provider in g4f.Provider.__providers__  # if provider.working
-]
-
-available_providers = webai + gpt4free_providers
+
+default_path = os.path.join(os.path.expanduser("~"), ".cache", "AIWEBS", "webscout")
+
 def sanitize_stream(
     chunk: str, intro_value: str = "data:", to_json: bool = True
 ) -> str | dict:
@@ -109,14 +66,10 @@ def run_system_command(
         )
         return (True, result)
     except subprocess.CalledProcessError as e:
-        # Handle error if the command returns a non-zero exit code
-        if stdout_error:
-            click.secho(f"Error Occurred: while running '{command}'", fg="yellow")
-            click.secho(e.stderr, fg="red")
-            if help is not None:
-                click.secho(help, fg="cyan")
-        sys.exit(e.returncode) if exit_on_error else None
-        return (False, e)
+        if exit_on_error:
+            raise Exception(f"Command failed with exit code {e.returncode}") from e
+        else:
+            return (False, e)  
 
 
 from .conversation import Conversation

{webscout-6.4 → webscout-6.5}/webscout/DWEBS.py

@@ -1,8 +1,9 @@
-from bs4 import BeautifulSoup
 import requests
 from typing import Dict, List, Optional, Union, Any
 from concurrent.futures import ThreadPoolExecutor, as_completed
+from webscout.scout import Scout
 from urllib.parse import quote, urljoin
+from webscout.litagent import LitAgent
 
 import time
 import random
@@ -10,14 +11,13 @@ import json
 import os
 from datetime import datetime, timedelta
 from functools import lru_cache
-from .Litlogger import LitLogger, LogFormat, ColorScheme
-from tenacity import retry, stop_after_attempt, wait_exponential
+from webscout.Litlogger import LitLogger, LogFormat, ColorScheme
 
 class GoogleS:
-    """A Python interface for Google search with advanced features 🔥
+    """A Python interface for Google search with advanced features
 
     The GoogleS class provides a powerful interface to perform web searches, image searches,
-    and advanced filtering on Google. Built with love by HAI to keep it 💯
+    and advanced filtering on Google. Built with love by HAI to keep it
 
     Basic Usage:
         >>> from webscout.DWEBS import GoogleS
@@ -122,7 +122,7 @@ class GoogleS:
         """
         self.proxy = proxy
         self.headers = headers if headers else {
-            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
+            "User-Agent": LitAgent().random()  # Use LitAgent to generate user agent
         }
         self.headers["Referer"] = "https://www.google.com/"
         self.client = requests.Session()
@@ -155,25 +155,46 @@ class GoogleS:
             time.sleep(self.rate_limit - time_since_last)
         self.last_request_time = time.time()
 
-    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
     def _get_url(self, method: str, url: str, params: Optional[Dict[str, str]] = None,
-                  data: Optional[Union[Dict[str, str], bytes]] = None) -> bytes:
+                  data: Optional[Union[Dict[str, str], bytes]] = None, max_retries: int = 3) -> bytes:
         """
-        Makes an HTTP request with retry logic and rate limiting.
+        Makes an HTTP request with manual retry logic and rate limiting.
+        
+        Args:
+            method (str): HTTP method (GET, POST, etc.)
+            url (str): Target URL
+            params (Optional[Dict[str, str]]): Query parameters
+            data (Optional[Union[Dict[str, str], bytes]]): Request payload
+            max_retries (int): Maximum number of retry attempts
+        
+        Returns:
+            bytes: Response content
         """
         self._respect_rate_limit()
-        try:
-            if self.use_litlogger:
-                self.logger.debug(f"Making {method} request to {url}")
-            resp = self.client.request(method, url, params=params, data=data, timeout=self.timeout)
-            resp.raise_for_status()
-            if self.use_litlogger:
-                self.logger.success(f"Request successful: {resp.status_code}")
-            return resp.content
-        except requests.exceptions.RequestException as ex:
-            if self.use_litlogger:
-                self.logger.error(f"Request failed: {url} - {str(ex)}")
-            raise
+        
+        for attempt in range(max_retries):
+            try:
+                if self.use_litlogger:
+                    self.logger.debug(f"Making {method} request to {url} (Attempt {attempt + 1})")
+                
+                resp = self.client.request(method, url, params=params, data=data, timeout=self.timeout)
+                resp.raise_for_status()
+                
+                if self.use_litlogger:
+                    self.logger.success(f"Request successful: {resp.status_code}")
+                
+                return resp.content
+            
+            except requests.exceptions.RequestException as ex:
+                if self.use_litlogger:
+                    self.logger.error(f"Request failed: {url} - {str(ex)}")
+                
+                # Exponential backoff
+                if attempt < max_retries - 1:
+                    wait_time = (2 ** attempt) + random.random()
+                    time.sleep(wait_time)
+                else:
+                    raise
 
     @lru_cache(maxsize=100)
     def _cache_key(self, query: str, **kwargs) -> str:
@@ -217,7 +238,7 @@ class GoogleS:
         type_filter: Optional[str] = None,
         **kwargs
     ) -> List[Dict[str, str]]:
-        """Search for images on Google with style! 🖼️
+        """Search for images on Google with style! 
 
         Args:
             query (str): What you're looking for fam
@@ -270,7 +291,7 @@ class GoogleS:
             params["tbs"] = f"itp:{type_filter}"
 
         content = self._get_url("GET", self.SEARCH_TYPES["image"], params=params)
-        soup = BeautifulSoup(content, 'lxml')
+        soup = Scout(content)  # Use Scout parser
         
         results = []
         for img in soup.find_all("img", class_="rg_i"):
@@ -365,7 +386,8 @@ class GoogleS:
             for future in as_completed(futures):
                 try:
                     resp_content = future.result()
-                    soup = BeautifulSoup(resp_content, 'lxml')  # Use lxml parser
+                    soup = Scout(resp_content)  # Use Scout parser
+                    
                     result_blocks = soup.find_all("div", class_="g")
 
                     if not result_blocks:
@@ -429,9 +451,9 @@ class GoogleS:
 
     def _extract_text_from_webpage(self, html_content: bytes, max_characters: Optional[int] = None) -> str:
         """
-        Extracts visible text from HTML content using lxml parser.
+        Extracts visible text from HTML content using Scout parser.
         """
-        soup = BeautifulSoup(html_content, 'lxml')  # Use lxml parser
+        soup = Scout(html_content)  # Use Scout parser
         for tag in soup(["script", "style", "header", "footer", "nav"]):
             tag.extract()
         visible_text = soup.get_text(strip=True)