SmartWebSearch 1.2.1__tar.gz

smartwebsearch-1.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 LIN WAI CHON
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

smartwebsearch-1.2.1/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.4
+Name: SmartWebSearch
+Version: 1.2.1
+Summary: SmartWebSearch is a Python package that combines the Tavily search API with Retrieval-Augmented Generation (RAG), LLM-powered query expansion, and web content extraction to perform intelligent, deep web searches with automated summarization.
+Author: LIN WAI CHON
+Author-email: jacksonlam.temp@gmail.com
+License-File: LICENSE
+Requires-Dist: requests
+Requires-Dist: bs4
+Requires-Dist: selenium
+Requires-Dist: markdownify
+Requires-Dist: tavily
+Requires-Dist: numpy
+Requires-Dist: sentence_transformers
+Requires-Dist: langchain_text_splitters
+Dynamic: author
+Dynamic: author-email
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: summary

smartwebsearch-1.2.1/README.md

@@ -0,0 +1,90 @@
+# Smart Web Search Package
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+
+SmartWebSearch is a Python package that combines the Tavily search API with Retrieval-Augmented Generation (RAG), LLM-powered query expansion, and web content extraction to perform intelligent, deep web searches with automated summarization.
+
+## Package Version
+- 1.2.1
+
+## Features
+- 🌐 **Web Search** – Uses Tavily API to fetch relevant search results.
+- 🧠 **Query Expansion** – Leverages LLMs (e.g., DeepSeek) to decompose complex queries and generate auxiliary searches.
+- 📄 **Content Extraction** – Fetches full page content using headless Chrome and filters noise.
+- 🔍 **RAG Pipeline** – Embeds documents with multilingual models (e.g., multilingual-e5-base) and retrieves context-aware chunks.
+- 📝 **Summarization** – Summarizes retrieved content using LLMs.
+
+## Environment
+- **Python 3.12 or above**
+- **Windows 11 Pro 64-bit** (macOS haven't tested)
+- **Python Packages** (requests, bs4, selenium, markdownify, tavily, numpy, sentence_transformers, langchain_text_splitters)
+
+## Installation
+- **The SmartWebSearch Package**: Install the SmartWebSearch package [here](https://github.com/LittleWai07/smart-web-search-package/archive/refs/heads/main.zip) or with git command `git clone https://github.com/LittleWai07/smart-web-search-package.git` (Git is required to run this command)
+- **Required Python Packages**: Install the required Python packages by command `pip install -r requirements.txt`
+
+## API Keys
+You need two API keys
+- **Tavily API key**: Sign up and get the API key [here](https://www.tavily.com) (1,000 free quotas per month)
+- **OpenAI Compatible API key**: eg., from [OpenAI](https://platform.openai.com/), [DeepSeek](https://platform.deepseek.com/), etc.
+
+## 🔒 Security Note
+
+For security reasons, **never hard-code your API keys directly in your source code**. 
+Instead, store them in environment variables, a `.env` file or a `*.json` file and load them into your program.
+
+## Quick Start
+Fill in the API keys and following required parameters manually.
+- **Tavily API Key**: The Tavily search API key (The key starts with `tvly-dev-`).
+- **OpenAI Compatible API Key**: The API key for the OpenAI Compatible API platform (The key usually starts with `sk-`).
+- **AI Model**: The id of the AI model used for summarization. (Default: `deepseek-chat`)
+- **OpenAI Compatible API Base URL**: The base url of the OpenAI Compatible API platform (The URL usually end with `/chat/completions`) (Default: `https://api.deepseek.com/chat/completions`)
+
+```python
+"""
+SmartWebSearch
+~~~~~~~~~~~~
+An example of how to use the SmartWebSearch package.
+"""
+
+# Import the SmartWebSearch package
+import SmartWebSearch as sws
+
+# --------------------------------------------------------------------
+# You can configure for different API providers by changing the 
+# model and base_url. Below are some examples:
+# --------------------------------------------------------------------
+
+# Example 1: Using DeepSeek (default)
+search = sws.SmartWebSearch(
+    "<Tavily API Key>",
+    "<OpenAI Compatible API Key>",
+    model="deepseek-chat",
+    openai_comp_api_base_url="https://api.deepseek.com/chat/completions"
+)
+
+# Example 2: Using OpenAI
+# search = sws.SmartWebSearch(
+#     "<Tavily API Key>",
+#     "<OpenAI Compatible API Key>",
+#     model="gpt-4-turbo-preview",
+#     openai_comp_api_base_url="https://api.openai.com/v1/chat/completions"
+# )
+
+# --------------------------------------------------------------------
+# Run a search
+# --------------------------------------------------------------------
+prompt = input("Enter a prompt: ")
+
+print("=== Normal Search (Tavily summaries) ===")
+print(search.search(prompt))
+
+print("\n=== Deep Search (full page content + RAG) ===")
+print(search.deepsearch(prompt))
+```
+
+**Note**: The documentation of this package will be completed in the future.
+
+## License
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details

smartwebsearch-1.2.1/SmartWebSearch/ChromeDriver.py

@@ -0,0 +1,43 @@
+"""
+SmartWebSearch.ChromeDriver
+~~~~~~~~~~~~
+
+This module implements the ChromeDriver.
+"""
+
+# Import the required modules
+from selenium.webdriver import Chrome
+from selenium.webdriver.common.by import By
+from selenium.webdriver.chrome.options import Options
+
+# The ChromeDriver class
+class ChromeDriver:
+    """
+    A class for interacting with a ChromeDriver.
+    """
+
+    def __init__(self) -> None:
+        """
+        Initialize the ChromeDriver object.
+
+        Returns:
+            None
+        """
+
+        # Create a headless Chrome browser
+        self.chrome_options: Options = Options()
+        self.chrome_options.add_argument("--headless")
+        self.chrome_options.add_argument("--no-sandbox")
+
+        self.driver: Chrome = Chrome(options = self.chrome_options)
+        self.driver.set_page_load_timeout(20)
+
+    def quit(self) -> None:
+        """
+        Quit the ChromeDriver object.
+
+        Returns:
+            None
+        """
+
+        self.driver.quit()

smartwebsearch-1.2.1/SmartWebSearch/Debugger.py

@@ -0,0 +1,111 @@
+"""
+SmartWebSearch.RAGTool
+~~~~~~~~~~~~
+
+This module implements the Debugger Tool for the package.
+"""
+
+# Import the required modules
+import os
+import datetime
+from typing import Any, TypeAlias, Literal
+
+# Type Alias
+_DebugType: TypeAlias = Literal['INFO', 'WARNING', 'ERROR', 'FILE']
+_DebugImportance: TypeAlias = Literal['LOW', 'MEDIUM', 'HIGH']
+
+# Configuration Class
+class DebuggerConfiguration:
+    """
+    DebuggerConfiguration class for the Debugger Tool.
+    """
+
+    # Whether to enable debugging
+    DEBUGGING: bool = False
+
+    # Whether to enable creating debug files
+    CREATE_DEBUG_FILES: bool = True
+
+    # Whether to skip low importance debug messages
+    SKIP_LOW_IMPORTANCE: bool = False
+
+    # Functions
+    def clear_debug_files() -> None:
+        """
+        Clear all debug files in the current directory.
+
+        Returns:
+            None
+        """
+
+        # Get all files in the current directory
+        files: list[str] = os.listdir()
+
+        # Loop through the files
+        for file in files:
+            # Check if the file is a debug file
+            if file.startswith("debug-"):
+                # Delete the file
+                os.remove(file)
+
+    # Run the clear_debug_files function
+    clear_debug_files()
+
+# Functions
+def show_debug(*values: tuple[Any], type: _DebugType = 'INFO', importance: _DebugImportance = 'MEDIUM') -> None:
+    """
+    Print the values to the console if DEBUGGING is True.
+    
+    Args:
+        *values (tuple[Any]): The values to print.
+        type (_DebugType) = 'INFO': The type of debug message.
+
+    Returns:
+        None
+    """
+
+    # If type is error, set importance to high
+    if type == 'ERROR': importance = 'HIGH'
+
+    # If importance is low and SKIP_LOW_IMPORTANCE is True, return
+    if importance == 'LOW' and DebuggerConfiguration.SKIP_LOW_IMPORTANCE: return
+
+    # Print the values if DEBUGGING is True
+    if DebuggerConfiguration.DEBUGGING:
+        print(f'[DEBUGGER] <{type} - {importance[0]}>', *values)
+
+def create_debug_file(filename: str, ext: str, content: str) -> None:
+    """
+    Create a debug file with the given filename and content.
+
+    Args:
+        filename (str): The name of the file to create.
+        ext (str): The extension of the file to create.
+        content (str): The content to write to the file.
+
+    Returns:
+        None
+    """
+
+    # If not debugging, return
+    if not DebuggerConfiguration.DEBUGGING: return
+
+    # If not creating debug files, return
+    if not DebuggerConfiguration.CREATE_DEBUG_FILES: return
+
+    # Replace all spaces in the filename to dash
+    filename: str = filename.replace(" ", "-")
+
+    # Replace all underscores in the filename to dash
+    filename: str = filename.replace("_", "-")
+
+    # Create the directory if it doesn't exist
+    if os.path.dirname(filename):
+        os.makedirs(os.path.dirname(filename), exist_ok = True)
+
+    # Write the content to the file
+    with open(f"debug-{filename}-{datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}.{ext}", "w", encoding = "utf-8") as f:
+        f.write(content)
+
+    # Show debug message
+    show_debug(f"Created debug file: 'debug-{filename}-{datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}.{ext}', content length: {len(content)}", type = 'FILE')

smartwebsearch-1.2.1/SmartWebSearch/KeyCheck.py

@@ -0,0 +1,97 @@
+"""
+SmartWebSearch.KeyCheck
+~~~~~~~~~~~~
+
+This module implements the KeyCheck Tool for the package.
+"""
+
+# Import the required modules
+import requests
+
+# Exception Class
+class InvalidKeyError(Exception):
+    """
+    An exception for invalid API keys.
+    """
+
+    def __init__(self, message: str) -> None:
+        """
+        Initialize the InvalidKeyError object.
+
+        Args:
+            message (str): The error message.
+        """
+
+        self.message: str = message
+        super().__init__(self.message)
+
+# KeyCheck Class
+class KeyCheck:
+    """
+    A class for checking the validity of API keys.
+    """
+
+    RAISE_ERROR: bool = True
+
+    # Check if the OpenAI Compatible API key is valid
+    @staticmethod
+    def check_openai_comp_api_key(openai_comp_api_key: str, model: str = "deepseek-chat", openai_comp_api_base_url: str = "https://api.deepseek.com/chat/completions") -> bool:
+        """
+        Check if the OpenAI Compatible API key is valid.
+
+        Args:
+            openai_comp_api_key (str): The OpenAI Compatible API key.
+            model (str) = "deepseek-chat": The model to use.
+            openai_comp_api_base_url (str) = "https://api.deepseek.com/chat/completions": The OpenAI Compatible API base URL.
+
+        Returns:
+            bool: True if the key is valid, False otherwise.
+        """
+
+        # Send a request to the OpenAI Compatible API to check if the key is valid
+        res: requests.Response = requests.post(
+            openai_comp_api_base_url,
+            headers = {
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {openai_comp_api_key}"
+            },
+            json = {
+                "model": model,
+                "messages": [{"role": "user", "content": "Hello!"}]
+            }
+        )
+
+        # If the key is invalid, raise an exception
+        if res.status_code != 200 and KeyCheck.RAISE_ERROR:
+            raise InvalidKeyError(f"Invalid OpenAI Compatible API key: {openai_comp_api_key}")
+
+        # Return True if the key is valid, False otherwise
+        return res.status_code == 200
+    
+    # Check if the Tavily API key is valid
+    @staticmethod
+    def check_tavily_api_key(tavily_api_key: str) -> bool:
+        """
+        Check if the Tavily API key is valid.
+
+        Args:
+            tavily_api_key (str): The Tavily API key.
+
+        Returns:
+            bool: True if the key is valid, False otherwise.
+        """
+
+        # Send a request to the Tavily API to check if the key is valid
+        res: requests.Response = requests.get(
+            "https://api.tavily.com/usage",
+            headers = {
+                "Authorization": f"Bearer {tavily_api_key}"
+            }
+        )
+
+        # If the key is invalid, raise an exception
+        if res.status_code != 200 and KeyCheck.RAISE_ERROR:
+            raise InvalidKeyError(f"Invalid Tavily API key: {tavily_api_key}")
+
+        # Return True if the key is valid, False otherwise
+        return res.status_code == 200

smartwebsearch-1.2.1/SmartWebSearch/Progress.py

@@ -0,0 +1,190 @@
+"""
+SmartWebSearch.Progress
+~~~~~~~~~~~~
+
+This module implements the progress for the web searching module.
+"""
+
+# Import the required modules
+from typing import Any, TypeAlias, Literal, Callable
+from datetime import datetime
+
+# Type Alias
+_ProgressStatus: TypeAlias = Literal['IDLE', 'STORMING', 'STORMED', 'SEARCHING', 'SEARCHED', 'PARSING', 'PARSED', 'KL_BASE_CREATING', 'KL_BASE_CREATED', 'KL_BASE_MATCHING', 'KL_BASE_MATCHED', 'CONCLUDING', 'CONCLUDED', 'PART_COMPLETED', 'COMPLETED', 'REQUEST_TIMEOUT']
+
+# Progress Classes
+class ProgressStatusSelector:
+    """
+    A class representing the status of a web searching operation.
+    """
+
+    # Constants
+    IDLE: _ProgressStatus = 'IDLE'
+    STORMING: _ProgressStatus = 'STORMING'
+    STORMED: _ProgressStatus = 'STORMED'
+    SEARCHING: _ProgressStatus = 'SEARCHING'
+    SEARCHED: _ProgressStatus = 'SEARCHED'
+    PARSING: _ProgressStatus = 'PARSING'
+    PARSED: _ProgressStatus = 'PARSED'
+    KL_BASE_CREATING: _ProgressStatus = 'KL_BASE_CREATING'
+    KL_BASE_CREATED: _ProgressStatus = 'KL_BASE_CREATED'
+    KL_BASE_MATCHING: _ProgressStatus = 'KL_BASE_MATCHING'
+    KL_BASE_MATCHED: _ProgressStatus = 'KL_BASE_MATCHED'
+    CONCLUDING: _ProgressStatus = 'CONCLUDING'
+    CONCLUDED: _ProgressStatus = 'CONCLUDED'
+    PART_COMPLETED: _ProgressStatus = 'PART_COMPLETED'
+    COMPLETED: _ProgressStatus = 'COMPLETED'
+    REQUEST_TIMEOUT: _ProgressStatus = 'REQUEST_TIMEOUT'
+
+class _ProgressData:
+    """
+    A class representing the data of a web searching operation.
+    """
+
+    def __init__(self, status: _ProgressStatus = 'IDLE', message: str = None, data: Any = None, progress: float = None, timestamp: datetime = None) -> None:
+        """
+        Initializes a new instance of the _ProgressData class.
+
+        Args:
+            status (_ProgressStatus) = 'IDLE': The status of the progress.
+            message (str) = None: The message to display. Defaults to None.
+            data (Any) = None: The data to display. Defaults to None.
+            progress (float) = None: The progress of the operation. Defaults to None. Range: [0.0, 1.0].
+            timestamp (datetime) = datetime.now(): The timestamp of the progress. Defaults to current datetime.
+        
+        Returns:
+            None
+        """
+
+        self.__status: _ProgressStatus = status
+        self.__message: str = message
+        self.__data: Any = data
+        self.__progress: float = progress
+        self.__timestamp: datetime = timestamp if timestamp else datetime.now()
+
+    def __str__(self) -> str:
+        """
+        Returns the string representation of the _ProgressData class.
+
+        Returns:
+            str: The string representation of the _ProgressData class.
+        """
+
+        return f"_ProgressData(status='{self.__status}', message='{self.__message}', data='{self.__data}', progress='{self.__progress}', timestamp='{self.__timestamp}')"
+
+    @property
+    def status(self) -> _ProgressStatus:
+        """
+        Returns the status of the progress.
+
+        Returns:
+            _ProgressStatus: The status of the progress.
+        """
+        return self.__status
+    
+    @property
+    def message(self) -> str:
+        """
+        Returns the message of the progress.
+
+        Returns:
+            str: The message of the progress.
+        """
+        return self.__message
+    
+    @property
+    def data(self) -> Any:
+        """
+        Returns the data of the progress.
+
+        Returns:
+            Any: The data of the progress.
+        """
+        return self.__data
+    
+    @property
+    def progress(self) -> float:
+        """
+        Returns the progress of the progress.
+
+        Returns:
+            float: The progress of the progress.
+        """
+        return self.__progress
+    
+    @property
+    def timestamp(self) -> datetime:
+        """
+        Returns the timestamp of the progress.
+
+        Returns:
+            datetime: The timestamp of the progress.
+        """
+        return self.__timestamp
+
+class Progress:
+    """
+    A class representing the progress of a web searching operation.
+    """
+
+    def __init__(self) -> None:
+        """
+        Initializes a new instance of the Progress class.
+        
+        Returns:
+            None
+        """
+
+        # Initialize the class attributes
+        self.__current_progress: _ProgressData = _ProgressData()
+        self.__progress_listeners = []
+    
+    def add_progress_listener(self, listener: Callable[[_ProgressStatus], None]) -> None:
+        """
+        Adds a listener to the progress of a web searching operation.
+
+        Args:
+            listener (Callable[[_ProgressStatus], None]): The callback function to add.
+        
+        Returns:
+            None
+        """
+
+        # Add the listener to the list of listeners
+        self.__progress_listeners.append(listener)
+
+    def remove_progress_listener(self, listener: Callable) -> None:
+        """
+        Removes a listener from the progress of a web searching operation.
+
+        Args:
+            listener (Callable[]): The callback function to remove.
+        
+        Returns:
+            None
+        """
+
+        # Remove the listener from the list of listeners
+        self.__progress_listeners.remove(listener)
+
+    def _update_progress(self, status: _ProgressStatus, message: str = None, data: Any = None, progress: float = None, timestamp: datetime = None) -> None:
+        """
+        Updates the progress of a web searching operation.
+
+        Args:
+            status (_ProgressStatus): The status of the progress.
+            message (str, optional): The message to display. Defaults to None.
+            data (Any, optional): The data to display. Defaults to None.
+            progress (float, optional): The progress of the operation. Defaults to None. Range: [0.0, 1.0].
+            timestamp (datetime, optional): The timestamp of the progress. Defaults to current datetime.
+        
+        Returns:
+            None
+        """
+
+        # Update the progress
+        self.__current_progress: _ProgressData = _ProgressData(status, message, data, progress, timestamp)
+
+        # Call the listeners
+        for listener in self.__progress_listeners:
+            listener(self.__current_progress)