webscout 3.3__py3-none-any.whl → 3.5__py3-none-any.whl

{webscout-3.3.dist-info → webscout-3.5.dist-info}/METADATA

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: webscout
-Version: 3.3
-Summary: Search for anything using Google, DuckDuckGo, phind.com, Contains AI models, can transcribe yt videos, temporary email and phone number generation, has TTS support, webai (terminal gpt and open interpreter) and offline LLMs
+Version: 3.5
+Summary: Search for anything using Google, DuckDuckGo, brave, qwant,  phind.com, Contains AI models, can transcribe yt videos, temporary email and phone number generation, has TTS support, webai (terminal gpt and open interpreter) and offline LLMs
 Author: OEvortex
 Author-email: helpingai5@gmail.com
 License: HelpingAI
@@ -104,14 +104,14 @@ Search for anything using Google, DuckDuckGo, phind.com, Contains AI models, can
     - [Temp number](#temp-number)
     - [Tempmail](#tempmail)
   - [Transcriber](#transcriber)
-  - [DeepWEBS: Advanced Web Searches](#deepwebs-advanced-web-searches)
-    - [Activating DeepWEBS](#activating-deepwebs)
-    - [Point to remember before using `DeepWEBS`](#point-to-remember-before-using-deepwebs)
+  - [DWEBS: Advanced Web Searches](#dwebs-advanced-web-searches)
+    - [Activating DWEBS](#activating-dwebs)
+    - [Point to remember before using `DWEBS`](#point-to-remember-before-using-dwebs)
     - [Usage Example](#usage-example)
   - [Text-to-Speech:](#text-to-speech)
     - [Available TTS Voices:](#available-tts-voices)
   - [Exceptions](#exceptions)
-  - [usage of webscout](#usage-of-webscout)
+  - [usage of WEBS](#usage-of-webs)
     - [1. `text()` - text search by DuckDuckGo.com](#1-text---text-search-by-duckduckgocom)
     - [2. `answers()` - instant answers by DuckDuckGo.com](#2-answers---instant-answers-by-duckduckgocom)
     - [3. `images()` - image search by DuckDuckGo.com](#3-images---image-search-by-duckduckgocom)
@@ -120,6 +120,7 @@ Search for anything using Google, DuckDuckGo, phind.com, Contains AI models, can
     - [6. `maps()` - map search by DuckDuckGo.com](#6-maps---map-search-by-duckduckgocom)
     - [7. `translate()` - translation by DuckDuckGo.com](#7-translate---translation-by-duckduckgocom)
     - [8. `suggestions()` - suggestions by DuckDuckGo.com](#8-suggestions---suggestions-by-duckduckgocom)
+  - [usage of WEBSX -- Another Websearch thing](#usage-of-websx----another-websearch-thing)
   - [ALL acts](#all-acts)
   - [Webscout Supported Acts:](#webscout-supported-acts)
   - [usage of webscout AI](#usage-of-webscout-ai)
@@ -141,12 +142,12 @@ Search for anything using Google, DuckDuckGo, phind.com, Contains AI models, can
     - [15. `poe`- chat with poe](#15-poe--chat-with-poe)
     - [16. `BasedGPT` - chat with GPT](#16-basedgpt---chat-with-gpt)
     - [17. `DeepSeek` -chat with deepseek](#17-deepseek--chat-with-deepseek)
+    - [18. Deepinfra](#18-deepinfra)
+    - [19. Deepinfra - VLM](#19-deepinfra---vlm)
     - [`LLM`](#llm)
     - [`Local-LLM` webscout can now run GGUF models](#local-llm-webscout-can-now-run-gguf-models)
-    - [`Function-calling-local-llm`](#function-calling-local-llm)
     - [`Local-rawdog`](#local-rawdog)
     - [`LLM` with internet](#llm-with-internet)
-    - [LLM with deepwebs](#llm-with-deepwebs)
   - [`Webai` - terminal gpt and a open interpeter](#webai---terminal-gpt-and-a-open-interpeter)
 
 ## Install
@@ -392,68 +393,76 @@ def main():
 if __name__ == "__main__":
     main()
 ```
-## DeepWEBS: Advanced Web Searches
 
-`DeepWEBS` is a standalone feature designed to perform advanced web searches with enhanced capabilities. It is particularly powerful in extracting relevant information directly from webpages and Search engine, focusing exclusively on text (web) searches. Unlike the `WEBS` , which provides a broader range of search functionalities, `DeepWEBS` is specifically tailored for in-depth web searches.
+## DWEBS: Advanced Web Searches
 
-### Activating DeepWEBS
+`DWEBS` is a standalone feature designed to perform advanced web searches with enhanced capabilities. It is particularly powerful in extracting relevant information directly from webpages and Search engine, focusing exclusively on text (web) searches. Unlike the `WEBS` , which provides a broader range of search functionalities, `DWEBS` is specifically tailored for in-depth web searches.
 
-To utilize the `DeepWEBS` feature, you must first create an instance of the `DeepWEBS` . This is designed to be used independently of the `WEBS` , offering a focused approach to web searches.
+### Activating DWEBS
 
-### Point to remember before using `DeepWEBS`
-As `DeepWEBS` is designed to extract relevant information directly from webpages and Search engine, It extracts html from webpages and saves them to folder named files in `DeepWEBS` that can be found at `C:\Users\Username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\DeepWEBS`
+To utilize the `DWEBS` feature, you must first create an instance of the `DWEBS` . This is designed to be used independently of the `WEBS` , offering a focused approach to web searches.
+
+### Point to remember before using `DWEBS`
+As `DWEBS` is designed to extract relevant information directly from webpages and Search engine, It extracts html from webpages and saves them to folder named files 
 
 ### Usage Example
 
-Here's a basic example of how to use the `DeepWEBS` :
+Here's a basic example of how to use the `DWEBS` :
 ```python
-from webscout import DeepWEBS
-
-def perform_web_search(query):
-    # Initialize the DeepWEBS class
-    D = DeepWEBS()
-    
-    # Set up the search parameters
-    search_params = D.DeepSearch(
-        queries=[query], # Query to search
-        result_num=5, # Number of search results
-        safe=True, # Enable SafeSearch
-        types=["web"], # Search type: web
-        extract_webpage=True, # True for extracting webpages
-        overwrite_query_html=False,
-        overwrite_webpage_html=False,
+from webscout import DWEBS
+
+def finalextractor(extract_webpage=True):
+    print('---------------Here Running for GoogleSearch--------------------')
+    # 1. Google Search
+    google_searcher = DWEBS.GoogleSearcher()
+    query_html_path = google_searcher.search(
+        query='HelpingAI-9B',
+        result_num=10,
+        safe=False,
+        overwrite=False,
     )
-    
-    # Execute the search and retrieve results
-    results = D.queries_to_search_results(search_params)
-    
-    return results
 
-def print_search_results(results):
-    """
-    Print the search results.
-    
-    Args:
-    - search_results (list): List of search results to print.
-    """
-    if results:
-        for index, result in enumerate(results, start=1):
-            print(f"Result {index}: {result}")
+    # 2. Search Result Extraction
+    query_results_extractor = DWEBS.QueryResultsExtractor()
+    query_search_results = query_results_extractor.extract(query_html_path)
+
+    if extract_webpage:
+        print('---------------Batch Webpage Fetcher--------------------')
+        # 3. Batch Webpage Fetching
+        batch_webpage_fetcher = DWEBS.BatchWebpageFetcher()
+        urls = [query_extracts['url'] for query_extracts in query_search_results['query_results']]
+        url_and_html_path_list = batch_webpage_fetcher.fetch(
+            urls,
+            overwrite=False,
+            output_parent=query_search_results["query"],
+        )
+
+        print('---------------Batch Webpage Extractor--------------------')
+        # 4. Batch Webpage Content Extraction
+        batch_webpage_content_extractor = DWEBS.BatchWebpageContentExtractor()
+        webpageurls = [url_and_html['html_path'] for url_and_html in url_and_html_path_list]
+        html_path_and_extracted_content_list = batch_webpage_content_extractor.extract(webpageurls)
+
+        # 5. Printing Extracted Content
+        for html_path_and_extracted_content in html_path_and_extracted_content_list:
+            print(html_path_and_extracted_content['extracted_content'])
     else:
-        print("No search results found.")
+        # Print only search results if extract_webpage is False
+        for result in query_search_results['query_results']:
+            DWEBS.logger.mesg(
+                f"{result['title']}\n"
+                f" - {result['site']}\n"
+                f" - {result['url']}\n"
+                f" - {result['abstract']}\n"
+                f"\n"
+            )
 
-def main():
-    # Prompt the user for a search query
-    query = input("Enter your search query: ")
-    
-    # Perform the web search
-    results = perform_web_search(query)
-    
-    # Print the search results
-    print_search_results(results)
+        DWEBS.logger.success(f"- {len(query_search_results['query_results'])} query results")
+        DWEBS.logger.success(f"- {len(query_search_results['related_questions'])} related questions")
 
-if __name__ == "__main__":
-    main()
+# Example usage:
+finalextractor(extract_webpage=True)  # Extract webpage content
+finalextractor(extract_webpage=False) # Skip webpage extraction and print search results only
 
 ```
 ## Text-to-Speech:
@@ -532,7 +541,7 @@ This ensures proper resource management and cleanup, as the context manager will
 Exceptions:
 - `WebscoutE`: Raised when there is a generic exception during the API request.
 
-## usage of webscout
+## usage of WEBS
 
 ### 1. `text()` - text search by DuckDuckGo.com 
 
@@ -682,6 +691,36 @@ with WEBS() as WEBS:
     for r in WEBS.suggestions("fly"):
         print(r)
 ```
+
+
+## usage of WEBSX -- Another Websearch thing
+```python
+from webscout import WEBSX
+
+def main():
+    # Initialize the WEBSX client
+    search = WEBSX(
+        k=10,  
+    )
+
+    # Example using `run` method - Get a summary
+    query = "What is the capital of France?"
+    answer = search.run(query)
+    print(f"Answer: {answer}\n")
+
+    # Example using `results` method - Get detailed results with metadata
+    query = "What is the capital of France?"
+    results = search.results(query, num_results=3)
+    print("Search Results:")
+    for result in results:
+        print(f"Title: {result['title']}")
+        print(f"Snippet: {result['snippet']}")
+        print(f"Link: {result['link']}\n")
+        print(f'Engines: {result["engines"]}')
+
+if __name__ == "__main__":
+    main()
+```
 ## ALL acts
 <details>
   <summary>expand</summary>
@@ -943,7 +982,7 @@ ___
 ### 0. `Duckchat` - chat with LLM
 ```python
 from webscout import WEBS as w
-R = w().chat("hello", model='claude-3-haiku') # GPT-3.5 Turbo
+R = w().chat("hello", model='claude-3-haiku') # GPT-3.5 Turbo, mixtral-8x7b, llama-3-70b, claude-3-haiku
 print(R)
 ```
 ### 1. `PhindSearch` - Search using Phind.com 
@@ -1253,6 +1292,57 @@ while True:
     r = ai.chat(prompt)
     print(r)
 ```
+### 18. Deepinfra
+```python
+from webscout import DeepInfra
+
+ai = DeepInfra(
+    is_conversation=True,
+    model= "Qwen/Qwen2-72B-Instruct",
+    max_tokens=800,
+    timeout=30,
+    intro=None,
+    filepath=None,
+    update_file=True,
+    proxies={},
+    history_offset=10250,
+    act=None,
+)
+
+prompt = "what is meaning of life"
+
+response = ai.ask(prompt)
+
+# Extract and print the message from the response
+message = ai.get_message(response)
+print(message)
+```
+
+### 19. Deepinfra - VLM
+```python
+from webscout import DeepInfra
+
+ai = DeepInfra(
+    is_conversation=True,
+    model= "Qwen/Qwen2-72B-Instruct",
+    max_tokens=800,
+    timeout=30,
+    intro=None,
+    filepath=None,
+    update_file=True,
+    proxies={},
+    history_offset=10250,
+    act=None,
+)
+
+prompt = "what is meaning of life"
+
+response = ai.ask(prompt)
+
+# Extract and print the message from the response
+message = ai.get_message(response)
+print(message)
+```
 ### `LLM` 
 ```python
 from webscout.LLM import LLM
@@ -1300,78 +1390,7 @@ thread = Thread(model, formats.phi3)
 # 4. Start interacting with the model
 thread.interact()
 ```
-### `Function-calling-local-llm`
-```python
-from webscout.Local import Model, Thread, formats
-from webscout import DeepWEBS
-from webscout.Local.utils import download_model
-from webscout.Local.model import Model
-from webscout.Local.thread import Thread
-from webscout.Local import formats
-from webscout.Local.samplers import SamplerSettings
-def deepwebs_search(query, max_results=5):
-    """Performs a web search using DeepWEBS and returns results as JSON."""
-    deepwebs = DeepWEBS()
-    search_config = DeepWEBS.DeepSearch(
-        queries=[query],
-        max_results=max_results,
-        extract_webpage=False,
-        safe=False,
-        types=["web"],
-        overwrite_query_html=True,
-        overwrite_webpage_html=True,
-    )
-    search_results = deepwebs.queries_to_search_results(search_config)
-    formatted_results = []
-    for result in search_results[0]:  # Assuming only one query
-        formatted_results.append(f"Title: {result['title']}\nURL: {result['url']}\n")
-    return "\n".join(formatted_results)
-
-# Load your model
-repo_id = "OEvortex/HelpingAI-9B" 
-filename = "helpingai-9b.Q4_0.gguf"
-model_path = download_model(repo_id, filename, token='')
 
-# 2. Load the model 
-model = Model(model_path, n_gpu_layers=10)
-
-# Create a Thread
-system_prompt = "You are a helpful AI assistant. Respond to user queries concisely. If a user asks for information that requires a web search, use the `deepwebs_search` tool. Do not call the tool if it is not necessary."
-sampler = SamplerSettings(temp=0.7, top_p=0.9)  # Adjust these values as needed
-# 4. Create a custom chatml format with your system prompt
-custom_chatml = formats.chatml.copy()
-custom_chatml['system_content'] = system_prompt
-thread = Thread(model, custom_chatml, sampler=sampler)
-# Add the deepwebs_search tool
-thread.add_tool({
-    "type": "function",
-    "function": {
-        "name": "deepwebs_search",
-        "description": "Performs a web search using DeepWEBS and returns the title and URLs of the results.",
-        "execute": deepwebs_search,
-        "parameters": {
-            "type": "object",
-            "properties": {
-                "query": {
-                    "type": "string",
-                    "description": "The query to search on the web",
-                },
-                "max_results": {
-                    "type": "integer",
-                    "description": "Maximum number of search results (default: 5)",
-                },
-            },
-            "required": ["query"],
-        },
-    },
-})
-
-# Start interacting with the model
-while True:
-    user_input = input("You: ")
-    response = thread.send(user_input)
-    print("Bot: ", response) 
-```
 ### `Local-rawdog`
 ```python
 import webscout.Local as ws
@@ -1532,94 +1551,7 @@ if __name__ == "__main__":
         else:
             print("No response")
 ```
-### LLM with deepwebs
-```python
-from __future__ import annotations
-from typing import List, Optional
-from webscout.LLM import LLM
-from webscout import DeepWEBS
-import warnings
-
-system_message: str = (
-    "As an AI assistant, I have been designed with advanced capabilities, including real-time access to online resources. This enables me to enrich our conversations and provide you with informed and accurate responses, drawing from a vast array of information. With each interaction, my goal is to create a seamless and meaningful connection, offering insights and sharing relevant content."
-    "My directives emphasize the importance of respect, impartiality, and intellectual integrity. I am here to provide unbiased responses, ensuring an ethical and respectful exchange. I will respect your privacy and refrain from sharing any personal information that may be obtained during our conversations or through web searches, only utilizing web search functionality when necessary to provide the most accurate and up-to-date information."
-    "Together, let's explore a diverse range of topics, creating an enjoyable and informative experience, all while maintaining the highest standards of privacy and respect"
-)
-
-# Ignore the specific UserWarning
-warnings.filterwarnings("ignore", category=UserWarning, module="curl_cffio", lineno=205)
-
-LLM = LLM(model="mistralai/Mixtral-8x22B-Instruct-v0.1", system_message=system_message)
-
-def perform_web_search(query):
-    # Initialize the DeepWEBS class
-    D = DeepWEBS()
-
-    # Set up the search parameters
-    search_params = D.DeepSearch(
-        queries=[query],  # Query to search
-        result_num=10,  # Number of search results
-        safe=True,  # Enable SafeSearch
-        types=["web"],  # Search type: web
-        extract_webpage=True,  # True for extracting webpages
-        overwrite_query_html=True,
-        overwrite_webpage_html=True,
-    )
 
-    # Execute the search and retrieve results
-    results = D.queries_to_search_results(search_params)
-    return results
-
-def chat(user_input: str, result_num: int = 10) -> Optional[str]:
-    """
-    Chat function to perform a web search based on the user input and generate a response using the LLM model.
-
-    Parameters
-    ----------
-    user_input : str
-        The user input to be used for the web search
-    max_results : int, optional
-        The maximum number of search results to include in the response, by default 10
-
-    Returns
-    -------
-    Optional[str]
-        The response generated by the LLM model, or None if there is no response
-    """
-    # Perform a web search based on the user input
-    search_results = perform_web_search(user_input)
-
-    # Extract URLs from search results
-    url_results = []
-    for result in search_results[0]['query_results']:
-        url_results.append(f"{result['title']} ({result['site']}): {result['url']}")
-
-    # Format search results
-    formatted_results = "\n".join(url_results)
-
-    # Define the messages to be sent, including the user input, search results, and system message
-    messages = [
-        {"role": "user", "content": f"User question is:\n{user_input}\nwebsearch results are:\n{formatted_results}"},
-    ]
-
-    # Use the chat method to get the response
-    response = LLM.chat(messages)
-    return response
-
-if __name__ == "__main__":
-    while True:
-        # Get the user input
-        user_input = input("User: ")
-
-        # Perform a web search based on the user input
-        response = chat(user_input)
-
-        # Print the response
-        if response:
-            print("AI:", response)
-        else:
-            print("No response")
-```
 ## `Webai` - terminal gpt and a open interpeter
 
 ```python

{webscout-3.3.dist-info → webscout-3.5.dist-info}/RECORD

@@ -1,37 +1,26 @@
-DeepWEBS/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-DeepWEBS/documents/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-DeepWEBS/documents/query_results_extractor.py,sha256=whd0NKLpcxW_6q3SkBOhMukr1K_c1PPYN92rf5EHRPM,4049
-DeepWEBS/documents/webpage_content_extractor.py,sha256=P4yHCkPTiBvMbORd8SKVt64rQFPJuj3iixcQoRU34Lw,5272
-DeepWEBS/networks/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-DeepWEBS/networks/filepath_converter.py,sha256=JKMBew1TYe4TVoGTqgTWerq2Pam49_9u9TVUFCTDQyk,3183
-DeepWEBS/networks/google_searcher.py,sha256=-AdIpVkRgemsARnOt8WPkF2Id1baVlqDHyqX2qz8Aew,1966
-DeepWEBS/networks/network_configs.py,sha256=-Hb78_7SBx32h219FnU14qcHTvBdDUf_QAU6-RTL_e0,726
-DeepWEBS/networks/webpage_fetcher.py,sha256=vRB9T3o-nMgrMkG2NPHTDctNeXaPSKCmBXqu189h2ZI,3590
-DeepWEBS/utilsdw/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-DeepWEBS/utilsdw/enver.py,sha256=vpI7s4_o_VL9govSryOv-z1zYK3pTEW3-H9QNN8JYtc,2472
-DeepWEBS/utilsdw/logger.py,sha256=Z0nFUcEGyU8r28yKiIyvEtO26xxpmJgbvNToTfwZecc,8174
 webscout/AIauto.py,sha256=xPGr_Z0h27XXNh4Wiufjn9TksDOqxqlaGcLUYKNP55w,18246
 webscout/AIbase.py,sha256=GoHbN8r0gq2saYRZv6LA-Fr9Jlcjv80STKFXUq2ZeGU,4710
-webscout/AIutel.py,sha256=Ghe9w1gqnCZTwjc3NzXymSamY3nP0zEep4NiATW32Qk,33454
-webscout/DWEBS.py,sha256=QT-7-dUgWhQ_H7EVZD53AVyXxyskoPMKCkFIpzkN56Q,7332
+webscout/AIutel.py,sha256=0SquAHyAa9zqEfcBdfnplM8hCuQ_7a8CyxK6b83_n5k,33471
+webscout/DWEBS.py,sha256=QLuT1IKu0lnwdl7W6c-ctBAO7Jj0Zk3PYm6-13BC7rU,25740
 webscout/LLM.py,sha256=LbGCZdJf8A5dwfoGS4tyy39tAh5BDdhMZP0ScKaaQfU,4184
-webscout/__init__.py,sha256=pOqM5UGmljQN4jTrL3kyrjMv66VuTPyzfNlXZra9KLQ,1872
+webscout/__init__.py,sha256=QX-26QQFOsL_JhUT4bGR7H7JGI58ClFcg4dF6XnlJUQ,1847
 webscout/__main__.py,sha256=ZtTRgsRjUi2JOvYFLF1ZCh55Sdoz94I-BS-TlJC7WDU,126
 webscout/async_providers.py,sha256=holBv5SxanxVXc_92CBBaXHlB2IakB_fHnhyZaFjYF8,684
-webscout/cli.py,sha256=174iWc0NxwfYMq9vyIk_NNnd3Q8bkzEiCa_BE6a0WZY,18743
-webscout/exceptions.py,sha256=Wx8bEN3bz1nNZ9PAZHX8jwvFPddF9Y2pHAEwCMu_VJc,498
+webscout/cli.py,sha256=enw_dPTCG3sNC1TXt96XccnpRmF4Etr99nh-RbGYags,18784
+webscout/exceptions.py,sha256=YtIs-vXBwcjbt9TZ_wB7yI0dO7ANYIZAmEEeLmoQ2fI,487
 webscout/g4f.py,sha256=NNcnlOtIWV9R93UsBN4jBGBEJ9sJ-Np1WbgjkGVDcYc,24487
 webscout/models.py,sha256=5iQIdtedT18YuTZ3npoG7kLMwcrKwhQ7928dl_7qZW0,692
 webscout/tempid.py,sha256=5oc3UbXhPGKxrMRTfRABT-V-dNzH_hOKWtLYM6iCWd4,5896
 webscout/transcriber.py,sha256=EddvTSq7dPJ42V3pQVnGuEiYQ7WjJ9uyeR9kMSxN7uY,20622
 webscout/utils.py,sha256=CxeXvp0rWIulUrEaPZMaNfg_tSuQLRSV8uuHA2chyKE,2603
-webscout/version.py,sha256=ZXcHuaEBaIH89v_gIvqcKVULdIIv9r489XLDrqSqDUo,23
+webscout/version.py,sha256=pTj22SSXb7rieyMXdGyEFmljJmZMa6FL_DaETjfeLwA,23
 webscout/voice.py,sha256=0QjXTHAQmCK07IDZXRc7JXem47cnPJH7u3X0sVP1-UQ,967
-webscout/webai.py,sha256=hnRfUI9AT3MgltP68bAmW5Tq4_aWcYytYeTFEsgS7u0,85991
-webscout/webscout_search.py,sha256=8tDmlskNtIUAM41dqIc387ufC1YunovTm6w5NqeM_yQ,42650
-webscout/webscout_search_async.py,sha256=ecn9b0J6YtAxMER80iUF1cgn_eh3Ysj7jFpievJzDbE,14471
+webscout/webai.py,sha256=qkvhYdyF5wNdmW4rNdH3RbfQxabEWlGvCyAk2SbH04k,86602
+webscout/webscout_search.py,sha256=lFAot1-Qil_YfXieeLakDVDEX8Ckcima4ueXdOYwiMc,42804
+webscout/webscout_search_async.py,sha256=dooKGwLm0cwTml55Vy6NHPPY-nymEqX2h8laX94Zg5A,14537
+webscout/websx_search.py,sha256=Nz4FfdSiXHaWoUGIUH1x1FzYlm8Ns736MMXM3EKm9_Y,11724
 webscout/Local/__init__.py,sha256=RN6klpbabPGNX2YzPm_hdeUcQvieUwvJt22uAO2RKSM,238
-webscout/Local/_version.py,sha256=_K1dEsde19FCHSmC1pCe9Ut_PzboJoV9w3ZMY835kqQ,83
+webscout/Local/_version.py,sha256=hC_EHWR519ZOsyRw9i6gXEfU5IAIR_B9d3THLVmkWXw,83
 webscout/Local/formats.py,sha256=BiZZSoN3e8S6-S-ykBL9ogSUs0vK11GaZ3ghc9U8GRk,18994
 webscout/Local/model.py,sha256=T_bzNNrxEyOyLyhp6fKwiuVBBkXC2a37LzJVCxFIxOU,30710
 webscout/Local/rawdog.py,sha256=LtA7bck2HyvWmovuaG86Iiquiz7XiMcxBlebo9IuGBY,35744
@@ -43,6 +32,7 @@ webscout/Provider/Berlin4h.py,sha256=zMpmWmdFCbcE3UWB-F9xbbTWZTfx4GnjnRf6sDoaiC0
 webscout/Provider/Blackboxai.py,sha256=HUk0moEGsgGvidD1LF9tbfaKdx7bPnGU_SrYPdcfHU8,17182
 webscout/Provider/ChatGPTUK.py,sha256=qmuCb_a71GNE5LelOb5AKJUBndvj7soebiNey4VdDvE,8570
 webscout/Provider/Cohere.py,sha256=IXnRosYOaMAA65nvsKmN6ZkJGSdZFYQYBidzuNaCqX8,8711
+webscout/Provider/Deepinfra.py,sha256=13H1h4FCon3GzjV9vCrLH_v9NcWHV6nl6i6NgnpXPGQ,18272
 webscout/Provider/Deepseek.py,sha256=HKsC-ePLSPqcrQbafy-IzR0BNqId3LfiIEhk9j9oTs4,10285
 webscout/Provider/Gemini.py,sha256=_4DHWvlWuNAmVHPwHB1RjmryjTZZCthLa6lvPEHLvkQ,8451
 webscout/Provider/Groq.py,sha256=QfgP3hKUcqq5vUA4Pzuu3HAgpJkKwLWNjjsnxtkCYd8,21094
@@ -59,10 +49,10 @@ webscout/Provider/ThinkAnyAI.py,sha256=_qFjj0djxxrranyEY33w14oizyRjzlVwMv_hzvVtw
 webscout/Provider/Xjai.py,sha256=BIlk2ouz9Kh_0Gg9hPvTqhI7XtcmWdg5vHSX_4uGrIs,9039
 webscout/Provider/Yepchat.py,sha256=2Eit-A7w1ph1GQKNQuur_yaDzI64r0yBGxCIjDefJxQ,19875
 webscout/Provider/Youchat.py,sha256=UVGBuGSjv4uRibn1xflmCjYcfrRTKnDvX3adhag6T98,7976
-webscout/Provider/__init__.py,sha256=lxvdW9dUk7rSjZTUaI4PrfTMGEwktJ_c82iyg-n9LIk,1506
-webscout-3.3.dist-info/LICENSE.md,sha256=9P0imsudI7MEvZe2pOcg8rKBn6E5FGHQ-riYozZI-Bk,2942
-webscout-3.3.dist-info/METADATA,sha256=X8VHG0hDnfdEOxYBPJ5qOLiq7ufjdJPCrLagDBFF8qQ,67184
-webscout-3.3.dist-info/WHEEL,sha256=oiQVh_5PnQM0E3gPdiz09WCNmwiHDMaGer_elqB3coM,92
-webscout-3.3.dist-info/entry_points.txt,sha256=Hh4YIIjvkqB9SVxZ2ri4DZUkgEu_WF_5_r_nZDIvfG8,73
-webscout-3.3.dist-info/top_level.txt,sha256=OD5YKy6Y3hldL7SmuxsiEDxAG4LgdSSWwzYk22MF9fk,18
-webscout-3.3.dist-info/RECORD,,
+webscout/Provider/__init__.py,sha256=j132gtLgDqUJl9GNNgO4k9GUmKdlYUG0dn4AjpS8fZA,1616
+webscout-3.5.dist-info/LICENSE.md,sha256=9P0imsudI7MEvZe2pOcg8rKBn6E5FGHQ-riYozZI-Bk,2942
+webscout-3.5.dist-info/METADATA,sha256=I_dj9LHVC_6WMkg0V4K2ilywFffLPElOdLTwgfI1dIQ,63504
+webscout-3.5.dist-info/WHEEL,sha256=oiQVh_5PnQM0E3gPdiz09WCNmwiHDMaGer_elqB3coM,92
+webscout-3.5.dist-info/entry_points.txt,sha256=Hh4YIIjvkqB9SVxZ2ri4DZUkgEu_WF_5_r_nZDIvfG8,73
+webscout-3.5.dist-info/top_level.txt,sha256=nYIw7OKBQDr_Z33IzZUKidRD3zQEo8jOJYkMVMeN334,9
+webscout-3.5.dist-info/RECORD,,

{webscout-3.3.dist-info → webscout-3.5.dist-info}/top_level.txt

@@ -1,2 +1 @@
-DeepWEBS
 webscout

DeepWEBS/documents/query_results_extractor.py

@@ -1,99 +0,0 @@
-from bs4 import BeautifulSoup
-from pathlib import Path
-from DeepWEBS.utilsdw.logger import logger
-
-class QueryResultsExtractor:
-    def __init__(self) -> None:
-        self.query_results = []
-        self.related_questions = []
-
-    def load_html(self, html_path):
-        try:
-            with open(html_path, "r", encoding="utf-8") as f:
-                html = f.read()
-            self.soup = BeautifulSoup(html, "html.parser")
-        except FileNotFoundError:
-            logger.error(f"File not found: {html_path}")
-        except Exception as e:
-            logger.error(f"Error loading HTML: {e}")
-
-    def extract_query_results(self):
-        try:
-            self.query = self.soup.find("textarea").text.strip()
-            query_result_elements = self.soup.find_all("div", class_="g")
-            for idx, result in enumerate(query_result_elements):
-                try:
-                    site = result.find("cite").find_previous("span").text.strip()
-                    url = result.find("a")["href"]
-                    title = result.find("h3").text.strip()
-                    abstract_element_conditions = [
-                        {"data-sncf": "1"},
-                        {"class_": "ITZIwc"},
-                    ]
-                    for condition in abstract_element_conditions:
-                        abstract_element = result.find("div", condition)
-                        if abstract_element is not None:
-                            abstract = abstract_element.text.strip()
-                            break
-                    else:
-                        abstract = ""
-                    logger.mesg(
-                        f"{title}\n"
-                        f" - {site}\n"
-                        f" - {url}\n"
-                        f" - {abstract}\n"
-                        f"\n"
-                    )
-                    self.query_results.append(
-                        {
-                            "title": title,
-                            "site": site,
-                            "url": url,
-                            "abstract": abstract,
-                            "index": idx,
-                            "type": "web",
-                        }
-                    )
-                except Exception as e:
-                    logger.error(f"Error extracting query result: {e}")
-            logger.success(f"- {len(query_result_elements)} query results")
-        except Exception as e:
-            logger.error(f"Error extracting query results: {e}")
-
-    def extract_related_questions(self):
-        try:
-            related_question_elements = self.soup.find_all(
-                "div", class_="related-question-pair"
-            )
-            for question_element in related_question_elements:
-                try:
-                    question = question_element.find("span").text.strip()
-                    print(question)
-                    self.related_questions.append(question)
-                except Exception as e:
-                    logger.error(f"Error extracting related question: {e}")
-            logger.success(f"- {len(self.related_questions)} related questions")
-        except Exception as e:
-            logger.error(f"Error extracting related questions: {e}")
-
-    def extract(self, html_path):
-        self.load_html(html_path)
-        self.extract_query_results()
-        self.extract_related_questions()
-        self.search_results = {
-            "query": self.query,
-            "query_results": self.query_results,
-            "related_questions": self.related_questions,
-        }
-        return self.search_results
-
-
-if __name__ == "__main__":
-    html_path_root = Path(__file__).parents[1] / "files"
-    html_filename = "python_tutorials"
-    html_path = html_path_root / f"{html_filename}.html"
-    extractor = QueryResultsExtractor()
-    try:
-        extractor.extract(html_path)
-    except Exception as e:
-        logger.error(f"Error in main function: {e}")