indoxJudge 0.0.0__tar.gz

indoxjudge-0.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 osllm.ai
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

indoxjudge-0.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+recursive-include indoxJudge *

indoxjudge-0.0.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.1
+Name: indoxJudge
+Version: 0.0.0
+Summary: Indox Judge
+Home-page: https://github.com/osllmai/inDoxJudge
+Author: osllm
+Author-email: ashkan@nematifamilyfundation.onmicrosoft.com
+License: MIT
+Keywords: RAG,LLM
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: dash
+Requires-Dist: dash_bootstrap_components
+Requires-Dist: dash_daq
+Requires-Dist: ipython
+Requires-Dist: ipython
+Requires-Dist: loguru
+Requires-Dist: mistralai
+Requires-Dist: nltk
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: plotly
+Requires-Dist: plotly
+Requires-Dist: protobuf
+Requires-Dist: pydantic
+Requires-Dist: Requests
+Requires-Dist: tenacity
+Requires-Dist: torch
+Requires-Dist: tqdm
+Requires-Dist: transformers
+
+# IndoxJudge
+
+Welcome to IndoxJudge! This repository provides a comprehensive suite of evaluation metrics for assessing the performance and quality of large language models (LLMs). Whether you're a researcher, developer, or enthusiast, this toolkit offers essential tools to measure various aspects of LLMs, including knowledge retention, bias, toxicity, and more.
+
+## Overview
+
+IndoxJudge is designed to provide a standardized and extensible framework for evaluating LLMs. With a focus on accuracy, fairness, and relevancy, this toolkit supports a wide range of evaluation metrics and is continuously updated to include the latest advancements in the field.
+
+## Features
+
+- **Comprehensive Metrics**: Evaluate LLMs across multiple dimensions, including accuracy, bias, toxicity, and contextual relevancy.
+- **Extensible Framework**: Easily integrate new metrics or customize existing ones to suit specific needs.
+- **User-Friendly Interface**: Intuitive and easy-to-use interface for seamless evaluation.
+- **Continuous Updates**: Regular updates to incorporate new metrics and improvements.
+
+## Supported Models
+
+IndoxJudge currently supports the following LLM models:
+
+- **OpenAi**
+- **GoogleAi**
+- **IndoxApi**
+- **HuggingFaceModel**
+- **Mistral**
+- **Ollama**
+
+## Metrics
+
+IndoxJudge includes the following metrics, with more being added:
+
+- **GEval**: General evaluation metric for LLMs.
+- **KnowledgeRetention**: Assesses the ability of LLMs to retain factual information.
+- **BertScore**: Measures the similarity between generated and reference sentences.
+- **Toxicity**: Evaluates the presence of toxic content in model outputs.
+- **Bias**: Analyzes the potential biases in LLM outputs.
+- **Hallucination**: Identifies instances where the model generates false or misleading information.
+- **Faithfulness**: Checks the alignment of generated content with source material.
+- **ContextualRelevancy**: Assesses the relevance of responses in context.
+- **Rouge**: Measures the overlap of n-grams between generated and reference texts.
+- **BLEU**: Evaluates the quality of text generation based on precision.
+- **AnswerRelevancy**: Assesses the relevance of answers to questions.
+- **METEOR**: Evaluates machine translation quality.
+- **Gruen**: Measures the quality of generated text by assessing grammaticality, redundancy, and focus.
+
+## Installation
+
+To install IndoxJudge, follow these steps:
+
+```bash
+git clone https://github.com/yourusername/indoxjudge.git
+cd indoxjudge
+```
+## Setting Up the Python Environment
+
+If you are running this project in your local IDE, please create a Python environment to ensure all dependencies are correctly managed. You can follow the steps below to set up a virtual environment named `indox_judge`:
+
+### Windows
+
+1. **Create the virtual environment:**
+```bash
+python -m venv indox_judge
+```
+2. **Activate the virtual environment:**
+```bash
+indox_judge\Scripts\activate
+```
+
+### macOS/Linux
+
+1. **Create the virtual environment:**
+   ```bash
+   python3 -m venv indox_judge
+```
+
+2. **Activate the virtual environment:**
+    ```bash
+   source indox_judge/bin/activate
+```
+### Install Dependencies
+
+Once the virtual environment is activated, install the required dependencies by running:
+
+```bash
+pip install -r requirements.txt
+```
+
+
+
+
+## Usage
+
+To use IndoxJudge, load your API key, select the model, and choose the evaluation metrics. Here's an example demonstrating how to evaluate a model's response for faithfulness:
+
+```python
+import os
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+INDOX_API_KEY = os.getenv("INDOX_API_KEY")
+
+# Import IndoxJudge and supported models
+from indoxJudge.piplines import CustomEvaluator
+from indoxJudge.models import IndoxApi
+from indoxJudge.metrics import Faithfulness
+
+# Initialize the model with your API key
+model = IndoxApi(api_key=INDOX_API_KEY)
+
+# Define your query and retrieval context
+query = "What are the benefits of a Mediterranean diet?"
+retrieval_context = [
+    "The Mediterranean diet emphasizes eating primarily plant-based foods, such as fruits and vegetables, whole grains, legumes, and nuts. It also includes moderate amounts of fish and poultry, and low consumption of red meat. Olive oil is the main source of fat, providing monounsaturated fats which are beneficial for heart health.",
+    "Research has shown that the Mediterranean diet can reduce the risk of heart disease, stroke, and type 2 diabetes. It is also associated with improved cognitive function and a lower risk of Alzheimer's disease. The diet's high content of fiber, antioxidants, and healthy fats contributes to its numerous health benefits.",
+    "A Mediterranean diet has been linked to a longer lifespan and a reduced risk of chronic diseases. It promotes healthy aging and weight management due to its emphasis on whole, unprocessed foods and balanced nutrition."
+]
+
+# Obtain the model's response
+response = "The Mediterranean diet is known for its health benefits, including reducing the risk of heart disease, stroke, and diabetes. It encourages the consumption of fruits, vegetables, whole grains, nuts, and olive oil, while limiting red meat. Additionally, this diet has been associated with better cognitive function and a reduced risk of Alzheimer's disease, promoting longevity and overall well-being."
+
+# Initialize the Faithfulness metric
+faithfulness_metrics = Faithfulness(llm_response=response, retrieval_context=retrieval_context)
+
+# Create an evaluator with the selected metrics
+evaluator = CustomEvaluator(metrics=[faithfulness_metrics], model=model)
+
+# Evaluate the response
+faithfulness_result = evaluator.evaluate()
+
+# Output the evaluation result
+print(faithfulness_result)
+```
+## Example Output
+
+```json
+{
+  "faithfulness": {
+    "claims": [
+      "The Mediterranean diet is known for its health benefits.",
+      "The Mediterranean diet reduces the risk of heart disease.",
+      "The Mediterranean diet reduces the risk of stroke.",
+      "The Mediterranean diet reduces the risk of diabetes.",
+      "The Mediterranean diet encourages the consumption of fruits.",
+      "The Mediterranean diet encourages the consumption of vegetables.",
+      "The Mediterranean diet encourages the consumption of whole grains.",
+      "The Mediterranean diet encourages the consumption of nuts.",
+      "The Mediterranean diet encourages the consumption of olive oil.",
+      "The Mediterranean diet limits red meat consumption.",
+      "The Mediterranean diet is associated with better cognitive function.",
+      "The Mediterranean diet is associated with a reduced risk of Alzheimer's disease.",
+      "The Mediterranean diet promotes longevity.",
+      "The Mediterranean diet promotes overall well-being."
+    ],
+    "truths": [
+      "The Mediterranean diet is known for its health benefits.",
+      "The Mediterranean diet reduces the risk of heart disease, stroke, and diabetes.",
+      "The Mediterranean diet encourages the consumption of fruits, vegetables, whole grains, nuts, and olive oil.",
+      "The Mediterranean diet limits red meat consumption.",
+      "The Mediterranean diet has been associated with better cognitive function.",
+      "The Mediterranean diet has been associated with a reduced risk of Alzheimer's disease.",
+      "The Mediterranean diet promotes longevity and overall well-being."
+    ],
+    "reason": "The score is 1.0 because the 'actual output' aligns perfectly with the information presented in the 'retrieval context', showcasing the health benefits, disease risk reduction, cognitive function improvement, and overall well-being promotion of the Mediterranean diet."
+  }
+}
+```
+## Roadmap
+
+We have an exciting roadmap planned for IndoxJudge:
+
+  | Plan                                                                 |
+| -------------------------------------------------------------------- |
+ | Integration of additional metrics such as Diversity and Coherence.   |
+| Introduction of a graphical user interface (GUI) for easier evaluation. |
+  | Expansion of the toolkit to support evaluation in multiple languages. |
+  | Release of a benchmarking suite for standardizing LLM evaluations.   |
+
+## Contributing
+
+We welcome contributions from the community! If you'd like to contribute, please fork the repository and create a pull request. For major changes, please open an issue first to discuss what you would like to change.
+
+1. Fork the repository
+2. Create a new branch (`git checkout -b feature-branch`)
+3. Commit your changes (`git commit -am 'Add new feature'`)
+4. Push to the branch (`git push origin feature-branch`)
+5. Create a pull request
+
+

indoxjudge-0.0.0/README.md

@@ -0,0 +1,188 @@
+# IndoxJudge
+
+Welcome to IndoxJudge! This repository provides a comprehensive suite of evaluation metrics for assessing the performance and quality of large language models (LLMs). Whether you're a researcher, developer, or enthusiast, this toolkit offers essential tools to measure various aspects of LLMs, including knowledge retention, bias, toxicity, and more.
+
+## Overview
+
+IndoxJudge is designed to provide a standardized and extensible framework for evaluating LLMs. With a focus on accuracy, fairness, and relevancy, this toolkit supports a wide range of evaluation metrics and is continuously updated to include the latest advancements in the field.
+
+## Features
+
+- **Comprehensive Metrics**: Evaluate LLMs across multiple dimensions, including accuracy, bias, toxicity, and contextual relevancy.
+- **Extensible Framework**: Easily integrate new metrics or customize existing ones to suit specific needs.
+- **User-Friendly Interface**: Intuitive and easy-to-use interface for seamless evaluation.
+- **Continuous Updates**: Regular updates to incorporate new metrics and improvements.
+
+## Supported Models
+
+IndoxJudge currently supports the following LLM models:
+
+- **OpenAi**
+- **GoogleAi**
+- **IndoxApi**
+- **HuggingFaceModel**
+- **Mistral**
+- **Ollama**
+
+## Metrics
+
+IndoxJudge includes the following metrics, with more being added:
+
+- **GEval**: General evaluation metric for LLMs.
+- **KnowledgeRetention**: Assesses the ability of LLMs to retain factual information.
+- **BertScore**: Measures the similarity between generated and reference sentences.
+- **Toxicity**: Evaluates the presence of toxic content in model outputs.
+- **Bias**: Analyzes the potential biases in LLM outputs.
+- **Hallucination**: Identifies instances where the model generates false or misleading information.
+- **Faithfulness**: Checks the alignment of generated content with source material.
+- **ContextualRelevancy**: Assesses the relevance of responses in context.
+- **Rouge**: Measures the overlap of n-grams between generated and reference texts.
+- **BLEU**: Evaluates the quality of text generation based on precision.
+- **AnswerRelevancy**: Assesses the relevance of answers to questions.
+- **METEOR**: Evaluates machine translation quality.
+- **Gruen**: Measures the quality of generated text by assessing grammaticality, redundancy, and focus.
+
+## Installation
+
+To install IndoxJudge, follow these steps:
+
+```bash
+git clone https://github.com/yourusername/indoxjudge.git
+cd indoxjudge
+```
+## Setting Up the Python Environment
+
+If you are running this project in your local IDE, please create a Python environment to ensure all dependencies are correctly managed. You can follow the steps below to set up a virtual environment named `indox_judge`:
+
+### Windows
+
+1. **Create the virtual environment:**
+```bash
+python -m venv indox_judge
+```
+2. **Activate the virtual environment:**
+```bash
+indox_judge\Scripts\activate
+```
+
+### macOS/Linux
+
+1. **Create the virtual environment:**
+   ```bash
+   python3 -m venv indox_judge
+```
+
+2. **Activate the virtual environment:**
+    ```bash
+   source indox_judge/bin/activate
+```
+### Install Dependencies
+
+Once the virtual environment is activated, install the required dependencies by running:
+
+```bash
+pip install -r requirements.txt
+```
+
+
+
+
+## Usage
+
+To use IndoxJudge, load your API key, select the model, and choose the evaluation metrics. Here's an example demonstrating how to evaluate a model's response for faithfulness:
+
+```python
+import os
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+INDOX_API_KEY = os.getenv("INDOX_API_KEY")
+
+# Import IndoxJudge and supported models
+from indoxJudge.piplines import CustomEvaluator
+from indoxJudge.models import IndoxApi
+from indoxJudge.metrics import Faithfulness
+
+# Initialize the model with your API key
+model = IndoxApi(api_key=INDOX_API_KEY)
+
+# Define your query and retrieval context
+query = "What are the benefits of a Mediterranean diet?"
+retrieval_context = [
+    "The Mediterranean diet emphasizes eating primarily plant-based foods, such as fruits and vegetables, whole grains, legumes, and nuts. It also includes moderate amounts of fish and poultry, and low consumption of red meat. Olive oil is the main source of fat, providing monounsaturated fats which are beneficial for heart health.",
+    "Research has shown that the Mediterranean diet can reduce the risk of heart disease, stroke, and type 2 diabetes. It is also associated with improved cognitive function and a lower risk of Alzheimer's disease. The diet's high content of fiber, antioxidants, and healthy fats contributes to its numerous health benefits.",
+    "A Mediterranean diet has been linked to a longer lifespan and a reduced risk of chronic diseases. It promotes healthy aging and weight management due to its emphasis on whole, unprocessed foods and balanced nutrition."
+]
+
+# Obtain the model's response
+response = "The Mediterranean diet is known for its health benefits, including reducing the risk of heart disease, stroke, and diabetes. It encourages the consumption of fruits, vegetables, whole grains, nuts, and olive oil, while limiting red meat. Additionally, this diet has been associated with better cognitive function and a reduced risk of Alzheimer's disease, promoting longevity and overall well-being."
+
+# Initialize the Faithfulness metric
+faithfulness_metrics = Faithfulness(llm_response=response, retrieval_context=retrieval_context)
+
+# Create an evaluator with the selected metrics
+evaluator = CustomEvaluator(metrics=[faithfulness_metrics], model=model)
+
+# Evaluate the response
+faithfulness_result = evaluator.evaluate()
+
+# Output the evaluation result
+print(faithfulness_result)
+```
+## Example Output
+
+```json
+{
+  "faithfulness": {
+    "claims": [
+      "The Mediterranean diet is known for its health benefits.",
+      "The Mediterranean diet reduces the risk of heart disease.",
+      "The Mediterranean diet reduces the risk of stroke.",
+      "The Mediterranean diet reduces the risk of diabetes.",
+      "The Mediterranean diet encourages the consumption of fruits.",
+      "The Mediterranean diet encourages the consumption of vegetables.",
+      "The Mediterranean diet encourages the consumption of whole grains.",
+      "The Mediterranean diet encourages the consumption of nuts.",
+      "The Mediterranean diet encourages the consumption of olive oil.",
+      "The Mediterranean diet limits red meat consumption.",
+      "The Mediterranean diet is associated with better cognitive function.",
+      "The Mediterranean diet is associated with a reduced risk of Alzheimer's disease.",
+      "The Mediterranean diet promotes longevity.",
+      "The Mediterranean diet promotes overall well-being."
+    ],
+    "truths": [
+      "The Mediterranean diet is known for its health benefits.",
+      "The Mediterranean diet reduces the risk of heart disease, stroke, and diabetes.",
+      "The Mediterranean diet encourages the consumption of fruits, vegetables, whole grains, nuts, and olive oil.",
+      "The Mediterranean diet limits red meat consumption.",
+      "The Mediterranean diet has been associated with better cognitive function.",
+      "The Mediterranean diet has been associated with a reduced risk of Alzheimer's disease.",
+      "The Mediterranean diet promotes longevity and overall well-being."
+    ],
+    "reason": "The score is 1.0 because the 'actual output' aligns perfectly with the information presented in the 'retrieval context', showcasing the health benefits, disease risk reduction, cognitive function improvement, and overall well-being promotion of the Mediterranean diet."
+  }
+}
+```
+## Roadmap
+
+We have an exciting roadmap planned for IndoxJudge:
+
+  | Plan                                                                 |
+| -------------------------------------------------------------------- |
+ | Integration of additional metrics such as Diversity and Coherence.   |
+| Introduction of a graphical user interface (GUI) for easier evaluation. |
+  | Expansion of the toolkit to support evaluation in multiple languages. |
+  | Release of a benchmarking suite for standardizing LLM evaluations.   |
+
+## Contributing
+
+We welcome contributions from the community! If you'd like to contribute, please fork the repository and create a pull request. For major changes, please open an issue first to discuss what you would like to change.
+
+1. Fork the repository
+2. Create a new branch (`git checkout -b feature-branch`)
+3. Commit your changes (`git commit -am 'Add new feature'`)
+4. Push to the branch (`git push origin feature-branch`)
+5. Create a pull request
+
+

indoxjudge-0.0.0/indoxJudge/graph/__init__.py

@@ -0,0 +1,2 @@
+from .visualization import Visualization
+

indoxjudge-0.0.0/indoxJudge/graph/assets/style.css

@@ -0,0 +1,188 @@
+/* Light Theme */
+.bg-light-custom {
+    background: rgb(246, 246, 246) !important;
+    color: black !important;
+    transition: all 0.2s;
+}
+
+.header {
+    position: relative;
+    top: 10px;
+    padding: 15px 63px;
+}
+
+.navbar-custom {
+    justify-content: center;
+    margin: 20px 0 50px 0;
+    background-color: rgb(255, 255, 255);
+    color: black;
+    padding: 20px;
+    border-radius: 8px;
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+}
+
+.nav-link {
+    color: black !important;
+}
+
+.nav-link:hover {
+    color: rgb(255, 255, 255) !important;
+    background-color: rgb(53, 92, 125);
+}
+
+.col-12 {
+    padding: 0;
+}
+
+.card-header {
+    background-color: transparent;
+    border: none;
+    padding: 0 20px;
+    margin-bottom: 10px;
+}
+
+.cart-title {
+    font-size: 26px;
+    font-family: 'Roboto', sans-serif;
+    color: rgb(53, 92, 125);
+    text-align: center;
+    margin-bottom: 10px;
+    padding-bottom: 10px;
+    border-bottom: 2px solid rgba(53, 92, 125, 0.3);
+}
+
+.card-text {
+    height: 100%;
+    color: black;
+    backdrop-filter: blur(80px);
+}
+
+.card-body {
+    border: none;
+    box-shadow: 0 1px 3px 2px rgba(0, 0, 0, 0.1);
+    margin-bottom: 20px;
+    border-radius: 8px;
+    padding: 40px;
+    background-color: aliceblue;
+}
+
+.btn-gtt {
+    background-color: rgba(215, 215, 215, 0.891);
+    border: none;
+    outline: none;
+}
+
+.btn-gtt:hover {
+    background-color: rgb(181, 181, 181);
+}
+
+.change-bg {
+    display: flex;
+    align-items: center;
+}
+
+.modebtn {
+    margin: 0 14px;
+}
+
+/* sun icon */
+.bi-brightness-high-fill {
+    font-size: 24px;
+    color: rgb(255, 165, 0); /* Orange color */
+}
+
+/* moon icon */
+.bi-moon-fill {
+    font-size: 24px;
+    color: rgb(72, 61, 139); /* Dark Slate Blue color */
+}
+
+.form-check-input {
+    background-color: rgb(233, 233, 233);
+}
+
+/* toggle body */
+.sc-bxivhb {
+    background-color: rgb(233, 233, 233);
+    margin: 0 5px;
+    border: 1px solid rgb(218, 218, 218);
+    width: 52px;
+}
+
+/* toggle circle */
+.sc-ifAKCX {
+    background-color: rgb(133, 133, 133);
+}
+
+/* toggle body */
+.daq-toggleswitch--light__indicator--off {
+    background-color: rgb(217, 217, 217);
+}
+
+.daq-toggleswitch--light__indicator--on {
+    background-color: rgb(33, 33, 33);
+}
+
+/* Dark Theme */
+.bg-dark-custom {
+    background-color: rgb(48, 48, 48);
+    color: white;
+}
+
+.navbar-custom-dark {
+    justify-content: center;
+    margin: 20px 0 50px 0;
+    background-color: rgb(64, 64, 64);
+    color: rgb(217, 217, 217);
+    padding: 20px;
+    border-radius: 8px;
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.3);
+    transition: all 0.2s;
+}
+
+.nav-link-dark {
+    color: white !important;
+}
+
+.card-header-dark {
+    border: none;
+    margin-bottom: 20px;
+    border-radius: 8px;
+    padding: 0 20px;
+}
+
+.cart-title-dark {
+    font-size: 26px;
+    font-family: 'Roboto', sans-serif;
+    color: rgb(255, 215, 0); /* Gold color */
+    text-align: center;
+    margin-bottom: 10px;
+    padding-bottom: 10px;
+    border-bottom: 2px solid rgba(255, 215, 0, 0.3);
+}
+
+.card-text-dark {
+    height: 100%;
+    color: white;
+    backdrop-filter: blur(80px);
+}
+
+.card-body-dark {
+    border: none;
+    box-shadow: 0 1px 3px 2px rgba(0, 0, 0, 0.2);
+    margin-bottom: 20px;
+    border-radius: 8px;
+    padding: 40px;
+    background-color: rgb(64, 64, 64);
+}
+
+/* button */
+.btn-gtt-dark {
+    background-color: rgba(105, 105, 105, 0.891);
+    border: none;
+    outline: none;
+}
+
+.btn-gtt-dark:hover {
+    background-color: rgb(85, 85, 85);
+}