webscout 4.8__tar.gz → 5.0__tar.gz

{webscout-4.8/webscout.egg-info → webscout-5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: webscout
-Version: 4.8
+Version: 5.0
 Summary: Search for anything using Google, DuckDuckGo, brave, qwant,  phind.com, Contains AI models, can transcribe yt videos, temporary email and phone number generation, has TTS support, webai (terminal gpt and open interpreter) and offline LLMs and more
 Author: OEvortex
 Author-email: helpingai5@gmail.com
@@ -64,6 +64,8 @@ Requires-Dist: yaspin
 Requires-Dist: pillow
 Requires-Dist: requests_html
 Requires-Dist: bson
+Requires-Dist: cloudscraper
+Requires-Dist: emoji
 Provides-Extra: dev
 Requires-Dist: ruff>=0.1.6; extra == "dev"
 Requires-Dist: pytest>=7.4.2; extra == "dev"
@@ -208,7 +210,7 @@ python -m webscout --help
 
 
 [Go To TOP](#TOP)
-## YTdownloader -webscout can now download yt videos
+## YTdownloader
 
 ```python
 from os import rename, getcwd
@@ -232,7 +234,7 @@ if __name__ == "__main__":
     download_video("https://www.youtube.com/watch?v=c0tMvzB0OKw")
 ```
 
-## Weather - webscout can now forcast weather
+## Weather
 1. weather
 ```python
 from webscout import weather as w
@@ -331,6 +333,7 @@ async def main() -> None:
 if __name__ == '__main__':
     asyncio.run(main())
 ```
+
 ## Transcriber
 The transcriber function in webscout is a handy tool that transcribes YouTube videos. Here's an example code demonstrating its usage:
 ```python
@@ -387,77 +390,25 @@ if __name__ == "__main__":
     main()
 ```
 
-## DWEBS: Advanced Web Searches
-
-`DWEBS` is a standalone feature designed to perform advanced web searches with enhanced capabilities. It is particularly powerful in extracting relevant information directly from webpages and Search engine, focusing exclusively on text (web) searches. Unlike the `WEBS` , which provides a broader range of search functionalities, `DWEBS` is specifically tailored for in-depth web searches.
-
-### Activating DWEBS
-
-To utilize the `DWEBS` feature, you must first create an instance of the `DWEBS` . This is designed to be used independently of the `WEBS` , offering a focused approach to web searches.
-
-### Point to remember before using `DWEBS`
-As `DWEBS` is designed to extract relevant information directly from webpages and Search engine, It extracts html from webpages and saves them to folder named files 
-
-### Usage Example
-
-Here's a basic example of how to use the `DWEBS` :
+## GoogleS -- formerly DWEBS
 ```python
-from webscout import DWEBS
-
-def finalextractor(extract_webpage=True):
-    print('---------------Here Running for GoogleSearch--------------------')
-    # 1. Google Search
-    google_searcher = DWEBS.GoogleSearcher()
-    query_html_path = google_searcher.search(
-        query='HelpingAI-9B',
-        result_num=10,
-        safe=False,
-        overwrite=False,
-    )
-
-    # 2. Search Result Extraction
-    query_results_extractor = DWEBS.QueryResultsExtractor()
-    query_search_results = query_results_extractor.extract(query_html_path)
-
-    if extract_webpage:
-        print('---------------Batch Webpage Fetcher--------------------')
-        # 3. Batch Webpage Fetching
-        batch_webpage_fetcher = DWEBS.BatchWebpageFetcher()
-        urls = [query_extracts['url'] for query_extracts in query_search_results['query_results']]
-        url_and_html_path_list = batch_webpage_fetcher.fetch(
-            urls,
-            overwrite=False,
-            output_parent=query_search_results["query"],
-        )
-
-        print('---------------Batch Webpage Extractor--------------------')
-        # 4. Batch Webpage Content Extraction
-        batch_webpage_content_extractor = DWEBS.BatchWebpageContentExtractor()
-        webpageurls = [url_and_html['html_path'] for url_and_html in url_and_html_path_list]
-        html_path_and_extracted_content_list = batch_webpage_content_extractor.extract(webpageurls)
-
-        # 5. Printing Extracted Content
-        for html_path_and_extracted_content in html_path_and_extracted_content_list:
-            print(html_path_and_extracted_content['extracted_content'])
-    else:
-        # Print only search results if extract_webpage is False
-        for result in query_search_results['query_results']:
-            DWEBS.logger.mesg(
-                f"{result['title']}\n"
-                f" - {result['site']}\n"
-                f" - {result['url']}\n"
-                f" - {result['abstract']}\n"
-                f"\n"
-            )
-
-        DWEBS.logger.success(f"- {len(query_search_results['query_results'])} query results")
-        DWEBS.logger.success(f"- {len(query_search_results['related_questions'])} related questions")
-
-# Example usage:
-finalextractor(extract_webpage=True)  # Extract webpage content
-finalextractor(extract_webpage=False) # Skip webpage extraction and print search results only
-
+from webscout import GoogleS
+from rich import print
+searcher = GoogleS()
+results = searcher.search("HelpingAI-9B", max_results=20, extract_webpage_text=False, max_extract_characters=100)
+for result in results:
+    print(result)
 ```
+### BingS
+```python
+from webscout import BingS
+from rich import print
+searcher = BingS()
+results = searcher.search("Python development tools", max_results=30)
+for result in results:
+    print(result)
+```
+
 ## Text-to-Speech:
 ```python
 from webscout import play_audio
@@ -1406,7 +1357,7 @@ from webscout import AndiSearch
 a = AndiSearch()
 print(a.chat("HelpingAI-9B"))
 ```
-### Function calling-bete
+### Function calling-beta
 ```python
 import json
 import logging
@@ -1519,7 +1470,7 @@ if "error" not in function_call_data:
 else:
     print(f"Error: {function_call_data['error']}")
 ```
-###  LLAMA3, pizzagpt, RUBIKSAI, Koala, Darkai, AI4Chat, Farfalle
+###  LLAMA3, pizzagpt, RUBIKSAI, Koala, Darkai, AI4Chat, Farfalle, PIAI, Felo, XDASH, Julius, YouChat, YEPCHAT, Cloudflare, TurboSeek,
 code similar to other provider
 ### `LLM` 
 ```python

{webscout-4.8 → webscout-5.0}/README.md

@@ -133,7 +133,7 @@ python -m webscout --help
 
 
 [Go To TOP](#TOP)
-## YTdownloader -webscout can now download yt videos
+## YTdownloader
 
 ```python
 from os import rename, getcwd
@@ -157,7 +157,7 @@ if __name__ == "__main__":
     download_video("https://www.youtube.com/watch?v=c0tMvzB0OKw")
 ```
 
-## Weather - webscout can now forcast weather
+## Weather
 1. weather
 ```python
 from webscout import weather as w
@@ -256,6 +256,7 @@ async def main() -> None:
 if __name__ == '__main__':
     asyncio.run(main())
 ```
+
 ## Transcriber
 The transcriber function in webscout is a handy tool that transcribes YouTube videos. Here's an example code demonstrating its usage:
 ```python
@@ -312,77 +313,25 @@ if __name__ == "__main__":
     main()
 ```
 
-## DWEBS: Advanced Web Searches
-
-`DWEBS` is a standalone feature designed to perform advanced web searches with enhanced capabilities. It is particularly powerful in extracting relevant information directly from webpages and Search engine, focusing exclusively on text (web) searches. Unlike the `WEBS` , which provides a broader range of search functionalities, `DWEBS` is specifically tailored for in-depth web searches.
-
-### Activating DWEBS
-
-To utilize the `DWEBS` feature, you must first create an instance of the `DWEBS` . This is designed to be used independently of the `WEBS` , offering a focused approach to web searches.
-
-### Point to remember before using `DWEBS`
-As `DWEBS` is designed to extract relevant information directly from webpages and Search engine, It extracts html from webpages and saves them to folder named files 
-
-### Usage Example
-
-Here's a basic example of how to use the `DWEBS` :
+## GoogleS -- formerly DWEBS
 ```python
-from webscout import DWEBS
-
-def finalextractor(extract_webpage=True):
-    print('---------------Here Running for GoogleSearch--------------------')
-    # 1. Google Search
-    google_searcher = DWEBS.GoogleSearcher()
-    query_html_path = google_searcher.search(
-        query='HelpingAI-9B',
-        result_num=10,
-        safe=False,
-        overwrite=False,
-    )
-
-    # 2. Search Result Extraction
-    query_results_extractor = DWEBS.QueryResultsExtractor()
-    query_search_results = query_results_extractor.extract(query_html_path)
-
-    if extract_webpage:
-        print('---------------Batch Webpage Fetcher--------------------')
-        # 3. Batch Webpage Fetching
-        batch_webpage_fetcher = DWEBS.BatchWebpageFetcher()
-        urls = [query_extracts['url'] for query_extracts in query_search_results['query_results']]
-        url_and_html_path_list = batch_webpage_fetcher.fetch(
-            urls,
-            overwrite=False,
-            output_parent=query_search_results["query"],
-        )
-
-        print('---------------Batch Webpage Extractor--------------------')
-        # 4. Batch Webpage Content Extraction
-        batch_webpage_content_extractor = DWEBS.BatchWebpageContentExtractor()
-        webpageurls = [url_and_html['html_path'] for url_and_html in url_and_html_path_list]
-        html_path_and_extracted_content_list = batch_webpage_content_extractor.extract(webpageurls)
-
-        # 5. Printing Extracted Content
-        for html_path_and_extracted_content in html_path_and_extracted_content_list:
-            print(html_path_and_extracted_content['extracted_content'])
-    else:
-        # Print only search results if extract_webpage is False
-        for result in query_search_results['query_results']:
-            DWEBS.logger.mesg(
-                f"{result['title']}\n"
-                f" - {result['site']}\n"
-                f" - {result['url']}\n"
-                f" - {result['abstract']}\n"
-                f"\n"
-            )
-
-        DWEBS.logger.success(f"- {len(query_search_results['query_results'])} query results")
-        DWEBS.logger.success(f"- {len(query_search_results['related_questions'])} related questions")
-
-# Example usage:
-finalextractor(extract_webpage=True)  # Extract webpage content
-finalextractor(extract_webpage=False) # Skip webpage extraction and print search results only
-
+from webscout import GoogleS
+from rich import print
+searcher = GoogleS()
+results = searcher.search("HelpingAI-9B", max_results=20, extract_webpage_text=False, max_extract_characters=100)
+for result in results:
+    print(result)
 ```
+### BingS
+```python
+from webscout import BingS
+from rich import print
+searcher = BingS()
+results = searcher.search("Python development tools", max_results=30)
+for result in results:
+    print(result)
+```
+
 ## Text-to-Speech:
 ```python
 from webscout import play_audio
@@ -1331,7 +1280,7 @@ from webscout import AndiSearch
 a = AndiSearch()
 print(a.chat("HelpingAI-9B"))
 ```
-### Function calling-bete
+### Function calling-beta
 ```python
 import json
 import logging
@@ -1444,7 +1393,7 @@ if "error" not in function_call_data:
 else:
     print(f"Error: {function_call_data['error']}")
 ```
-###  LLAMA3, pizzagpt, RUBIKSAI, Koala, Darkai, AI4Chat, Farfalle
+###  LLAMA3, pizzagpt, RUBIKSAI, Koala, Darkai, AI4Chat, Farfalle, PIAI, Felo, XDASH, Julius, YouChat, YEPCHAT, Cloudflare, TurboSeek,
 code similar to other provider
 ### `LLM` 
 ```python

{webscout-4.8 → webscout-5.0}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", encoding="utf-8") as f:
 
 setup(
     name="webscout",
-    version="4.8",
+    version="5.0",
     description="Search for anything using Google, DuckDuckGo, brave, qwant,  phind.com, Contains AI models, can transcribe yt videos, temporary email and phone number generation, has TTS support, webai (terminal gpt and open interpreter) and offline LLMs and more",
     long_description=README,
     long_description_content_type="text/markdown",
@@ -68,6 +68,8 @@ setup(
         "pillow",
         "requests_html",
         "bson",
+        "cloudscraper",
+        "emoji"
     ],
     entry_points={
         "console_scripts": [

webscout-5.0/webscout/Agents/functioncall.py

@@ -0,0 +1,142 @@
+import json
+import logging
+from webscout import LLAMA3, WEBS
+
+class FunctionCallingAgent:
+    def __init__(self, model: str = "llama3-8b", 
+                 system_prompt: str = 'You are a helpful assistant that will always answer what the user wants', 
+                 tools: list = None):
+        self.LLAMA3 = LLAMA3(model=model, system=system_prompt, timeout=300)
+        self.tools = tools if tools is not None else []
+        self.webs = WEBS()
+
+    def function_call_handler(self, message_text: str) -> dict:
+        system_message = self._generate_system_message(message_text)
+        response = self.LLAMA3.chat(system_message)
+        # logging.info(f"Raw response: {response}")
+        return self._parse_function_call(response)
+
+    def _generate_system_message(self, user_message: str) -> str:
+        tools_description = '\n'.join([f"- {tool['function']['name']}: {tool['function'].get('description', '')}" for tool in self.tools])
+        return (
+            "You are an AI assistant capable of understanding user requests and using tools to fulfill them. "
+            "Always respond using the JSON format specified below, even if you're not sure about the answer. "
+            f"Available tools:\n{tools_description}\n\n"
+            "Instructions:\n"
+            "1. Analyze the user's request.\n"
+            "2. Choose the most appropriate tool based on the request.\n"
+            "3. Respond ONLY with a JSON object in this exact format:\n"
+            "{\n"
+            '  "tool_name": "name_of_the_tool",\n'
+            '  "tool_input": {\n'
+            '    "param1": "value1",\n'
+            '    "param2": "value2"\n'
+            "  }\n"
+            "}\n\n"
+            "If you can't determine a suitable tool, use the 'general_ai' tool with the user's message as the 'question' parameter.\n\n"
+            f"User request: {user_message}\n\n"
+            "Your response (in JSON format):"
+        )
+
+    def _parse_function_call(self, response: str) -> dict:
+        try:
+            # Find the JSON-like part of the response
+            start_idx = response.find("{")
+            end_idx = response.rfind("}") + 1
+
+            if start_idx == -1 or end_idx == -1:
+                raise ValueError("No valid JSON structure found in the response.")
+
+            response_json_str = response[start_idx:end_idx]
+            
+            # Attempt to load the JSON string
+            return json.loads(response_json_str)
+
+        except (ValueError, json.JSONDecodeError) as e:
+            logging.error(f"Error parsing function call: {e}")
+            return {"error": str(e)}
+
+    def execute_function(self, function_call_data: dict) -> str:
+        function_name = function_call_data.get("tool_name")
+        arguments = function_call_data.get("tool_input", {})
+
+        if not isinstance(arguments, dict):
+            logging.error("Invalid arguments format.")
+            return "Invalid arguments format."
+
+        logging.info(f"Executing function: {function_name} with arguments: {arguments}")
+
+    #     if function_name == "web_search":
+    #         return self._handle_web_search(arguments)
+    #     elif function_name == "general_ai":
+    #         return self._handle_general_ai(arguments)
+    #     else:
+    #         return f"Function '{function_name}' is not implemented."
+
+    # def _handle_web_search(self, arguments: dict) -> str:
+    #     query = arguments.get("query")
+    #     if not query:
+    #         return "Please provide a search query."
+
+    #     search_results = self.webs.text(query, max_results=3)
+    #     formatted_results = "\n\n".join(
+    #         f"{i+1}. {result['title']}\n{result['body']}\nURL: {result['href']}"
+    #         for i, result in enumerate(search_results)
+    #     )
+    #     return f"Here's what I found:\n\n{formatted_results}"
+
+    # def _handle_general_ai(self, arguments: dict) -> str:
+    #     question = arguments.get("question")
+    #     if not question:
+    #         return "Please provide a question for the AI to answer."
+
+    #     response = self.LLAMA3.chat(question)
+    #     return response
+
+# Example usage
+if __name__ == "__main__":
+    tools = [
+        {
+            "type": "function",
+            "function": {
+                "name": "web_search",
+                "description": "Search query on Google",
+                "parameters": {
+                    "type": "object",
+                    "properties": {
+                        "query": {
+                            "type": "string",
+                            "description": "web search query"
+                        }
+                    },
+                    "required": ["query"]
+                }
+            }
+        },
+        {
+            "type": "function",
+            "function": {
+                "name": "general_ai",
+                "description": "Use AI to answer a general question",
+                "parameters": {
+                    "type": "object",
+                    "properties": {
+                        "question": {
+                            "type": "string",
+                            "description": "The question to be answered by the AI"
+                        }
+                    },
+                    "required": ["question"]
+                }
+            }
+        }
+    ]
+
+    agent = FunctionCallingAgent(tools=tools)
+    message = "open yt"
+    function_call_data = agent.function_call_handler(message)
+    print(f"Function Call Data: {function_call_data}")
+
+    if "error" not in function_call_data:
+        result = agent.execute_function(function_call_data)
+        print(f"Function Execution Result: {result}")

webscout-5.0/webscout/Bing_search.py

@@ -0,0 +1,124 @@
+from bs4 import BeautifulSoup
+import requests
+from typing import Dict, List, Optional, Union
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from urllib.parse import urlparse
+from termcolor import colored
+import time
+import random
+
+class BingS:
+    """Bing search class to get search results from bing.com."""
+
+    _executor: ThreadPoolExecutor = ThreadPoolExecutor(max_workers=10)
+
+    def __init__(
+        self,
+        headers: Optional[Dict[str, str]] = None,
+        proxy: Optional[str] = None,
+        timeout: Optional[int] = 10,
+    ) -> None:
+        """Initialize the BingS object."""
+        self.proxy: Optional[str] = proxy
+        self.headers = headers if headers else {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.62"
+        }
+        self.headers["Referer"] = "https://www.bing.com/"
+        self.client = requests.Session()
+        self.client.headers.update(self.headers)
+        self.client.proxies.update({"http": self.proxy, "https": self.proxy})
+        self.timeout = timeout
+
+    def __enter__(self) -> "BingS":
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.client.close()
+
+    def _get_url(
+        self,
+        method: str,
+        url: str,
+        params: Optional[Dict[str, str]] = None,
+        data: Optional[Union[Dict[str, str], bytes]] = None,
+    ) -> bytes:
+        try:
+            resp = self.client.request(method, url, params=params, data=data, timeout=self.timeout)
+        except Exception as ex:
+            raise Exception(f"{url} {type(ex).__name__}: {ex}") from ex
+        if resp.status_code == 200:
+            return resp.content
+        raise Exception(f"{resp.url} returned status code {resp.status_code}. {params=} {data=}")
+
+    def search(
+        self,
+        keywords: str,
+        region: str = "us-EN",  # Bing uses us-EN
+        lang: str = "en",
+        safe: str = "off",
+        timelimit: Optional[str] = None,  # Not directly supported by Bing
+        max_results: Optional[int] = None,
+    ) -> List[Dict[str, str]]:
+        """Bing text search."""
+        assert keywords, "keywords is mandatory"
+
+        results = []
+        start = 1  # Bing uses 1-based indexing for pages
+        while len(results) < (max_results or float('inf')):
+            params = {
+                "q": keywords,
+                "count": 10,  # Number of results per page
+                "mkt": region,
+                "setlang": lang,
+                "safeSearch": safe,
+                "first": start,  # Bing uses 'first' for pagination
+            }
+
+            try:
+                resp_content = self._get_url("GET", "https://www.bing.com/search", params=params)
+                soup = BeautifulSoup(resp_content, "html.parser")
+                result_block = soup.find_all("li", class_="b_algo")
+
+                if not result_block:
+                    break
+
+                for result in result_block:
+                    try:
+                        link = result.find("a", href=True)
+                        if link:
+                            initial_url = link["href"]
+
+                            title = result.find("h2").text if result.find("h2") else ""
+                            description = result.find("p").text.strip() if result.find("p") else ""  # Strip whitespace
+
+                            # Remove 'WEB' prefix if present
+                            if description.startswith("WEB"):
+                                description = description[4:]  # Skip the first 4 characters ('WEB ')
+
+                            results.append({
+                                "title": title,
+                                "href": initial_url,
+                                "abstract": description,
+                                "index": len(results),
+                                "type": "web",
+                            })
+
+                            if len(results) >= max_results:
+                                return results
+
+                    except Exception as e:
+                        print(f"Error extracting result: {e}")
+
+            except Exception as e:
+                print(f"Error fetching URL: {e}")
+
+            start += 10
+
+        return results
+
+if __name__ == "__main__":
+    from rich import print
+    searcher = BingS()
+    results = searcher.search("Python development tools", max_results=30)
+    for result in results:
+        print(result)