PyPI - aiverify-moonshot - Versions diffs - 0.6.3__py3-none-any.whl → 0.6.4__py3-none-any.whl - Mend

aiverify-moonshot 0.6.3py3-none-any.whl → 0.6.4py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (12) hide show

aiverify_moonshot-0.6.4.dist-info/METADATA ADDED Viewed

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: aiverify-moonshot
+Version: 0.6.4
+Summary: AI Verify advances Gen AI testing with Project Moonshot.
+Project-URL: Repository, https://github.com/aiverify-foundation/moonshot
+Project-URL: Documentation, https://aiverify-foundation.github.io/moonshot/
+Project-URL: Issues, https://github.com/aiverify-foundation/moonshot/issues
+Author-email: AI Verify Foundation <info@aiverify.sg>
+License: Apache-2.0
+License-File: AUTHORS.md
+License-File: LICENSE.md
+License-File: NOTICES.md
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Requires-Dist: datasets>=2.21.0
+Requires-Dist: ijson>=3.3.0
+Requires-Dist: jinja2>=3.1.4
+Requires-Dist: numpy>=1.26.4
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: pydantic==2.8.2
+Requires-Dist: pyparsing>=3.1.4
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: python-multipart>=0.0.9
+Requires-Dist: python-slugify>=8.0.4
+Requires-Dist: tenacity>=8.5.0
+Requires-Dist: xxhash>=3.5.0
+Provides-Extra: all
+Requires-Dist: cmd2>=2.4.3; extra == 'all'
+Requires-Dist: dependency-injector>=4.41.0; extra == 'all'
+Requires-Dist: fastapi>=0.115.4; extra == 'all'
+Requires-Dist: rich>=13.8.0; extra == 'all'
+Requires-Dist: typing-extensions>=4.12.2; extra == 'all'
+Requires-Dist: uvicorn>=0.30.6; extra == 'all'
+Provides-Extra: cli
+Requires-Dist: cmd2>=2.4.3; extra == 'cli'
+Requires-Dist: rich>=13.8.0; extra == 'cli'
+Provides-Extra: web-api
+Requires-Dist: dependency-injector>=4.41.0; extra == 'web-api'
+Requires-Dist: fastapi>=0.115.4; extra == 'web-api'
+Requires-Dist: typing-extensions>=4.12.2; extra == 'web-api'
+Requires-Dist: uvicorn>=0.30.6; extra == 'web-api'
+Description-Content-Type: text/markdown
+<div align="center">
+![Moonshot Logo](https://github.com/aiverify-foundation/moonshot/raw/main/misc/aiverify-moonshot-logo.png)
+**Version 0.6.4**
+A simple and modular tool to evaluate any LLM-based AI systems.
+[![Python 3.11](https://img.shields.io/badge/python-3.11-green)](https://www.python.org/downloads/release/python-3111/)
+</div>
+## 🎯 Motivation
+Developed by the [AI Verify Foundation](https://aiverifyfoundation.sg/), [Moonshot](https://aiverifyfoundation.sg/project-moonshot/) is one of the first tools to bring Benchmarking and Red-Teaming together to help AI developers, compliance teams and AI system owners <b>evaluate LLMs and LLM-based AI systems</b>.
+</br>
+## 🚀 Why Moonshot
+In the rapidly evolving landscape of Generative AI, ensuring safety, reliability, and performance of AI systems is paramount. Moonshot addresses this critical need by providing a unified platform for:
+- <b>Benchmark Tests:</b> Systematically test LLMs across various performance metrics, and critical trust & safety dimensions using a wide array of open-source benchmarks and domain-specific tests.
+- <b>Red Team Attacks:</b> Proactively identify vulnerabilities and potential misuse scenarios in your AI systems through streamlined adversarial prompting.
+- <b>Streamline MLOps:</b> Integrate evaluation seamlessly into your development pipeline with flexible APIs.
+</br>
+## 🔑 Key Features
+- <b>User-friendly Interfaces:</b> Interact with Moonshot via an intuitive Web UI for visual insights, and an interactive Command Line Interface (CLI) for quick operations.
+- <b>Comprehensive Benchmarking:</b>
+  - [View list of available datasets available](https://aiverify-foundation.github.io/moonshot/resources/datasets/)
+  - Test for <b>Performance</b> (e.g., accuracy, BLEU)
+  - Ensure <b>Trust & Safety</b> e.g., bias, toxicity, hallucination)
+  - Utilize pre-built Cookbooks of tests or easily create your custom evaluations. [View available pre-built Cookbooks](https://aiverify-foundation.github.io/moonshot/resources/cookbooks/)
+- <b>Powerful Red-Teaming:</b>
+  - [View list of available attack modules](https://aiverify-foundation.github.io/moonshot/resources/attack_modules/)
+  - Simplify adversarial prompt generation using algorithmic strategies or generative LLM to uncover potential misuse.
+  - Leverage prompt templates, context strategies, and automated attack modules.
+- <b>Customizable Recipes:</b> Define your evaluation logic with custom datasets (input-target pairs), metrics, optional prompt templates, evaluation metric, and grading scales. [View available pre-built Recipes](https://aiverify-foundation.github.io/moonshot/resources/recipes/)
+- <b>Insightful Reporting:</b> Generate comprehensive HTML reports with interactive charts for clear visualization of test results, and detailed raw JSON results for deeper programmatic analysis.
+- <b>Extensible & Modular:</b> Designed for easy extension and integration with new LLMs, benchmarks, and attack techniques.
+</br>
+# Getting Started
+In this Beta version, Moonshot can be used through several interfaces:
+- User-friendly Web UI - [Web UI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)
+- Interactive Command Line Interface - [CLI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)
+- Seamless Integration into your MLOps workflow via Moonshot Library APIs or Moonshot Web APIs - [Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook), [Web API Docs](https://aiverify-foundation.github.io/moonshot/api_reference/web_api_swagger/)
+</br>
+## 💻 Let's Go!
+This section will guide you through getting Moonshot up and running.
+</br>
+### ✅ Prerequisites
+1. <b>Python:</b> [Version 3.11](https://www.python.org/downloads/) is required.
+2. <b>Git Version Control:</b> [Git](https://github.com/git-guides/install-git) is essential for cloning the repository.
+3. <b>(Optional) Virtual Environment:</b> Highly recommended to manage dependencies.
+    ```
+    # Create a virtual environment
+    python -m venv venv
+    # Activate the virtual environment
+    source venv/bin/activate
+    ```
+4. If you plan to install our Web UI, you will also need [Node.js version 20.11.1 LTS](https://nodejs.org/en/blog/release/v20.11.1) and above
+</br>
+### ⬇️ Installation
+You can install Moonshot in various ways depending on your needs
+<b>1. Using `pip` (Recommended for most users)</b>
+```
+# Install Project Moonshot's Python Library, which includes Moonshot's full functionalities (Library APIs, CLI and Web APIs)
+pip install "aiverify-moonshot[all]"
+# Clone and install test assets and Web UI
+python -m moonshot -i moonshot-data -i moonshot-ui
+```
+⚠️ You will need to have test assets from [moonshot-data](https://github.com/aiverify-foundation/moonshot-data) before you can run any tests.
+🖼️ If you plan to install our Web UI, you will also need [moonshot-ui](https://github.com/aiverify-foundation/moonshot-ui)
+Check out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for more details.
+</br>
+<b>2. From Source Code (For developers and contributors)</b>
+```
+# To install from source code (Full functionalities)
+git clone git@github.com:aiverify-foundation/moonshot.git
+cd moonshot
+pip install -r requirements.txt
+```
+If you have installation issues, please take a look at the [Troubleshooting Guide](https://aiverify-foundation.github.io/moonshot/faq/).
+<details>
+<summary><b>Other installation options</b></summary>
+Here's a summary of other installation commands available:
+```
+# To install Moonshot library APIs only
+pip install aiverify-moonshot
+# To install Moonshot library APIs and Web APIs only
+pip install "aiverify-moonshot[web-api]"
+# To install Moonshot library APIs and CLI only
+pip install "aiverify-moonshot[cli]"
+```
+Check out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for more details.
+</details>
+</br>
+### 🏃‍♀️ Run Moonshot
+#### Running the Web UI
+```
+python -m moonshot web
+```
+Open [http://localhost:3000/](http://localhost:3000/) in a browser and you should see this homepage:
+![Moonshot UI Home](https://github.com/aiverify-foundation/moonshot/raw/main/misc/ui-homepage.png)
+Refer to this [guide](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/moonshot_interface/homepage/) to discover the rich features available in Moonshot Web UI
+</br>
+#### Running the Interactive CLI
+```
+python -m moonshot cli interactive
+```
+![Moonshot cli](https://github.com/aiverify-foundation/moonshot/raw/main/misc/cli-homepage.png)
+Refer to this [Command List](https://aiverify-foundation.github.io/moonshot/user_guide/cli/cli_command_list/) to discover the list of CLI commands for Moonshot
+</br></br>
+# 📚 Documentation & User Guides
+For detailed information on configuring, using, and extending Moonshot, please refer to our comprehensive documentation:
+#### Guides for Moonshot Web UI
+- [Getting Started with Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)
+- [Creating Your Custom Cookbook via Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/create_cookbook/)
+- [Creating Your Custom Connector Endpoint via Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/create_endpoint/)
+- [Running Benchmark Test on Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/getting_started/first_test/)
+- [Running Red Teaming on Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/redteam/)
+#### Guides for Moonshot Interactive CLI
+- [Getting Started with Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)
+- [Creating Your Custom Benchmark Tests for Your RAG Apps via Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/create_benchmark_tests/)
+- [Creating Your Custom Connector Endpoint via Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/create_endpoint/)
+- [Running Benchmark Test on Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/run_benchmark_tests/)
+- [Running Red Teaming on Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/run_red_teaming/)
+#### For Users Interested in trying Out Moonshot using Jupyter Notebook
+- [Moonshot Library Python Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook)
+####
+</br>
+## 🤝 Contribution
+Moonshot is an open-source project, and we welcome contributions from the community! Whether fixing a bug, adding a new feature, improving documentation, or suggesting an enhancement, your efforts are highly valued.
+Please refer to our [Contributor Guide](https://aiverify-foundation.github.io/moonshot/contributing/) for details on how to get started.
+</br>
+## ✨ Project Status
+Moonshot is currently in beta. We are actively developing new features, improving existing ones, and enhancing stability. We encourage you to try it out and provide feedback!
+</br>
+## 📜 License
+Moonshot is released under the [Apache Software License 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)
+</br>

{aiverify_moonshot-0.6.3.dist-info → aiverify_moonshot-0.6.4.dist-info}/RECORD RENAMED Viewed

@@ -34,7 +34,7 @@ moonshot/integrations/cli/utils/process_data.py,sha256=QVL5vp2_8ZgGicmCAdeYEHkeb
 moonshot/integrations/web_api/.env.dev,sha256=0z5_Ut8rF-UqFZtgjkH2qoqORhD5_nSs2w_OeX2SteI,182
 moonshot/integrations/web_api/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 moonshot/integrations/web_api/__main__.py,sha256=MdnLi_ZF-olAAEJwTPU1iGYFYwo-fNWNT2qfchkH3y4,2050
-moonshot/integrations/web_api/app.py,sha256=fSpPcyOSR5e5IRjPqJs5DI2-yJVkDbN9ZtzkYogi5M4,3651
+moonshot/integrations/web_api/app.py,sha256=nbAAnVyy1BEqlctGeVUjVYCdR3yD0mfyGZmwBM2zZec,3651
 moonshot/integrations/web_api/container.py,sha256=DVkJG_qm7ItcG6tgMYOqIj07wpKhPWOOfy6-bEv72y4,5915
 moonshot/integrations/web_api/logging_conf.py,sha256=t3EGRV6tZhV732KXe8_Tiy0fiwVAWxZX5Tt8VTgrrfg,3388
 moonshot/integrations/web_api/log/.gitkeep,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
@@ -113,16 +113,16 @@ moonshot/src/api/api_result.py,sha256=xZHex_K_PbSr-65LPLr5up4lvu313-VJroR4BBR5JQ
 moonshot/src/api/api_run.py,sha256=ToPYIeTPd2OaOqd20UQ_ie2BQPqLw5aZ_RwRwv9oGpU,2983
 moonshot/src/api/api_runner.py,sha256=spN7ogBGQN7I-c1u_UeKZ7vg02AtCuAl3dFG8CSayQU,4257
 moonshot/src/api/api_session.py,sha256=w-mc_sj3ElQa6FPQqWbK2bFgGTu1ECwYm3tN41xFkes,11130
-moonshot/src/bookmark/bookmark.py,sha256=Gf9wZ5wmJUmatHChr0_3tVyUbPMfpCVeEfujQQjIv18,11867
-moonshot/src/bookmark/bookmark_arguments.py,sha256=cB5m2zB8255WVdacmC2-ZYNyaoK4-gOM_Qwb_JDR-34,1449
+moonshot/src/bookmark/bookmark.py,sha256=KOvvNE8wvdkJu9q2clte5fPLESZ2x6EuuoQkkxFxmHg,12970
+moonshot/src/bookmark/bookmark_arguments.py,sha256=Xgjx3Ek9MXwD4p1icac8nPX3KZfmYsBwR6kwPwHAngE,1622
 moonshot/src/configs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 moonshot/src/configs/env_variables.py,sha256=eF__UJN37LCzIB4pv_T7G-kQHlOa657QA7IpL1d_0MM,7150
 moonshot/src/connectors/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-moonshot/src/connectors/connector.py,sha256=-2-vp6hPWigdEOwkHAz1SSTGwGsaraZHzHnb_9aD5HI,16272
+moonshot/src/connectors/connector.py,sha256=XqYVamy2EWsMOpOyK8bnIUtjH6MiZ29k2ZFfdAPmFyU,17532
 moonshot/src/connectors/connector_prompt_arguments.py,sha256=aJX5Jwd9bxWWLhIntXBu5Iu3NKsOcud-dfdBJVgKK58,751
 moonshot/src/connectors/connector_response.py,sha256=kIByU-H7A0NXRxk1Y2XPGZHSJNKzJj6BNkzXqdyZyb0,403
 moonshot/src/connectors_endpoints/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-moonshot/src/connectors_endpoints/connector_endpoint.py,sha256=3U4030gKhvDt4e7VD3lac2FG0kxGJ0DSLPTAUKYOPEk,10018
+moonshot/src/connectors_endpoints/connector_endpoint.py,sha256=racxWKZSZYQ8YZMoPHGcwdQ6-8LKzJ0CKu5S8E0xwnM,10800
 moonshot/src/connectors_endpoints/connector_endpoint_arguments.py,sha256=0v9mUHki4l22CK8o8UjATAsFDza9Lutbh4QplLiDXs8,2434
 moonshot/src/cookbooks/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 moonshot/src/cookbooks/cookbook.py,sha256=DdZwRGx5-xTDIKcXtZRpp7Qb9Mm9dNGwXWLQXoQrBBo,10412
@@ -172,9 +172,9 @@ moonshot/src/utils/import_modules.py,sha256=T9zTN59PFnvY2rjyWhSV9KSIAHxWV1pyBemF
 moonshot/src/utils/log.py,sha256=YNgD7Eh2OT36XlmVBKCGUTAh9TRp4Akfe4kDdvHASgs,2502
 moonshot/src/utils/pagination.py,sha256=5seymyRoqyENIhKllAatr1T91kMCGFslcvRnJHyMSvc,814
 moonshot/src/utils/timeit.py,sha256=TvuF0w8KWhp0oZFY0cUU3UY0xlGKjchb0OkfYfgVTlc,866
-aiverify_moonshot-0.6.3.dist-info/METADATA,sha256=eyeMqNTwKX5yimWX7I6790WFg8PFcM-ZQqSqkUYPHJk,12458
-aiverify_moonshot-0.6.3.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-aiverify_moonshot-0.6.3.dist-info/licenses/AUTHORS.md,sha256=mmAbe3i3sT8JZHJMBhxp3i1xRehV0g7WB4T_eyIBuBs,59
-aiverify_moonshot-0.6.3.dist-info/licenses/LICENSE.md,sha256=53izDRmJZZCjpYGfyLqlxnGQN-aNWBxasuzuMXC5Ias,11347
-aiverify_moonshot-0.6.3.dist-info/licenses/NOTICES.md,sha256=3d1WLa7kMtzh79BXxXr-keSlvHo8Llb-5EkGgtRiMS0,51537
-aiverify_moonshot-0.6.3.dist-info/RECORD,,
+aiverify_moonshot-0.6.4.dist-info/METADATA,sha256=UKH5vf6h00IJVbN3TyjUO-xY_M2Chwqcut_ZfFkEVv4,11273
+aiverify_moonshot-0.6.4.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+aiverify_moonshot-0.6.4.dist-info/licenses/AUTHORS.md,sha256=mmAbe3i3sT8JZHJMBhxp3i1xRehV0g7WB4T_eyIBuBs,59
+aiverify_moonshot-0.6.4.dist-info/licenses/LICENSE.md,sha256=53izDRmJZZCjpYGfyLqlxnGQN-aNWBxasuzuMXC5Ias,11347
+aiverify_moonshot-0.6.4.dist-info/licenses/NOTICES.md,sha256=3d1WLa7kMtzh79BXxXr-keSlvHo8Llb-5EkGgtRiMS0,51537
+aiverify_moonshot-0.6.4.dist-info/RECORD,,

moonshot/integrations/web_api/app.py CHANGED Viewed

@@ -71,7 +71,7 @@ def create_app(cfg: providers.Configuration) -> CustomFastAPI:
     }
     app: CustomFastAPI = CustomFastAPI(
-        title="Project Moonshot", version="0.6.3", **app_kwargs
+        title="Project Moonshot", version="0.6.4", **app_kwargs
     )
     if cfg.cors.enabled():

moonshot/src/bookmark/bookmark.py CHANGED Viewed

@@ -5,21 +5,6 @@ from datetime import datetime
 from moonshot.src.bookmark.bookmark_arguments import BookmarkArguments
 from moonshot.src.configs.env_variables import EnvVariables
-from moonshot.src.messages_constants import (
-    BOOKMARK_ADD_BOOKMARK_ERROR,
-    BOOKMARK_ADD_BOOKMARK_SUCCESS,
-    BOOKMARK_ADD_BOOKMARK_VALIDATION_ERROR,
-    BOOKMARK_DELETE_ALL_BOOKMARK_ERROR,
-    BOOKMARK_DELETE_ALL_BOOKMARK_SUCCESS,
-    BOOKMARK_DELETE_BOOKMARK_ERROR,
-    BOOKMARK_DELETE_BOOKMARK_ERROR_1,
-    BOOKMARK_DELETE_BOOKMARK_FAIL,
-    BOOKMARK_DELETE_BOOKMARK_SUCCESS,
-    BOOKMARK_EXPORT_BOOKMARK_ERROR,
-    BOOKMARK_EXPORT_BOOKMARK_VALIDATION_ERROR,
-    BOOKMARK_GET_BOOKMARK_ERROR,
-    BOOKMARK_GET_BOOKMARK_ERROR_1,
-)
 from moonshot.src.storage.storage import Storage
 from moonshot.src.utils.log import configure_logger
@@ -28,6 +13,29 @@ logger = configure_logger(__name__)
 class Bookmark:
+    BOOKMARK_ADD_BOOKMARK_ERROR = "[Bookmark] Failed to add bookmark record: {message}"
+    BOOKMARK_ADD_BOOKMARK_SUCCESS = "[Bookmark] Bookmark added successfully."
+    BOOKMARK_ADD_BOOKMARK_VALIDATION_ERROR = "Error inserting record into database."
+    BOOKMARK_DELETE_ALL_BOOKMARK_ERROR = (
+        "[Bookmark] Failed to delete all bookmark records: {message}"
+    )
+    BOOKMARK_DELETE_ALL_BOOKMARK_SUCCESS = "[Bookmark] All bookmark records deleted."
+    BOOKMARK_DELETE_BOOKMARK_ERROR = (
+        "[Bookmark] Failed to delete bookmark record: {message}"
+    )
+    BOOKMARK_DELETE_BOOKMARK_ERROR_1 = "[Bookmark] Invalid bookmark name: {message}"
+    BOOKMARK_DELETE_BOOKMARK_FAIL = (
+        "[Bookmark] Bookmark record not found. Unable to delete."
+    )
+    BOOKMARK_DELETE_BOOKMARK_SUCCESS = "[Bookmark] Bookmark record deleted."
+    BOOKMARK_EXPORT_BOOKMARK_ERROR = "[Bookmark] Failed to export bookmarks: {message}"
+    BOOKMARK_EXPORT_BOOKMARK_VALIDATION_ERROR = (
+        "Export filename must be a non-empty string."
+    )
+    BOOKMARK_GET_BOOKMARK_ERROR = (
+        "[Bookmark] No record found for bookmark name: {message}"
+    )
+    BOOKMARK_GET_BOOKMARK_ERROR_1 = "[Bookmark] Invalid bookmark name: {message}"
     _instance = None
     sql_table_name = "bookmark"
@@ -135,13 +143,16 @@ class Bookmark:
                 self.db_instance, data, Bookmark.sql_insert_bookmark_record
             )
             if results is not None:
-                return {"success": True, "message": BOOKMARK_ADD_BOOKMARK_SUCCESS}
+                return {
+                    "success": True,
+                    "message": Bookmark.BOOKMARK_ADD_BOOKMARK_SUCCESS,
+                }
             else:
-                raise Exception(BOOKMARK_ADD_BOOKMARK_VALIDATION_ERROR)
+                raise Exception(Bookmark.BOOKMARK_ADD_BOOKMARK_VALIDATION_ERROR)
         except Exception as e:
             return {
                 "success": False,
-                "message": BOOKMARK_ADD_BOOKMARK_ERROR.format(message=str(e)),
+                "message": Bookmark.BOOKMARK_ADD_BOOKMARK_ERROR.format(message=str(e)),
             }
     def get_all_bookmarks(self) -> list[dict]:
@@ -193,11 +204,11 @@ class Bookmark:
                 return BookmarkArguments.from_tuple_to_dict(bookmark_info)
             else:
                 raise RuntimeError(
-                    BOOKMARK_GET_BOOKMARK_ERROR.format(message=bookmark_name)
+                    Bookmark.BOOKMARK_GET_BOOKMARK_ERROR.format(message=bookmark_name)
                 )
         else:
             raise RuntimeError(
-                BOOKMARK_GET_BOOKMARK_ERROR_1.format(message=bookmark_name)
+                Bookmark.BOOKMARK_GET_BOOKMARK_ERROR_1.format(message=bookmark_name)
             )
     def delete_bookmark(self, bookmark_name: str) -> dict:
@@ -228,19 +239,24 @@ class Bookmark:
                     )
                     return {
                         "success": True,
-                        "message": BOOKMARK_DELETE_BOOKMARK_SUCCESS,
+                        "message": Bookmark.BOOKMARK_DELETE_BOOKMARK_SUCCESS,
                     }
                 else:
-                    return {"success": False, "message": BOOKMARK_DELETE_BOOKMARK_FAIL}
+                    return {
+                        "success": False,
+                        "message": Bookmark.BOOKMARK_DELETE_BOOKMARK_FAIL,
+                    }
             except Exception as e:
                 return {
                     "success": False,
-                    "message": BOOKMARK_DELETE_BOOKMARK_ERROR.format(message=str(e)),
+                    "message": Bookmark.BOOKMARK_DELETE_BOOKMARK_ERROR.format(
+                        message=str(e)
+                    ),
                 }
         else:
             return {
                 "success": False,
-                "message": BOOKMARK_DELETE_BOOKMARK_ERROR_1.format(
+                "message": Bookmark.BOOKMARK_DELETE_BOOKMARK_ERROR_1.format(
                     message=bookmark_name
                 ),
             }
@@ -256,11 +272,16 @@ class Bookmark:
             Storage.delete_database_record_in_table(
                 self.db_instance, Bookmark.sql_delete_bookmark_records
             )
-            return {"success": True, "message": BOOKMARK_DELETE_ALL_BOOKMARK_SUCCESS}
+            return {
+                "success": True,
+                "message": Bookmark.BOOKMARK_DELETE_ALL_BOOKMARK_SUCCESS,
+            }
         except Exception as e:
             return {
                 "success": False,
-                "message": BOOKMARK_DELETE_ALL_BOOKMARK_ERROR.format(message=str(e)),
+                "message": Bookmark.BOOKMARK_DELETE_ALL_BOOKMARK_ERROR.format(
+                    message=str(e)
+                ),
             }
     def export_bookmarks(self, export_file_name: str = "bookmarks") -> str:
@@ -281,8 +302,8 @@ class Bookmark:
             Exception: If the export file name is invalid or an error occurs during export.
         """
         if not isinstance(export_file_name, str) or not export_file_name:
-            error_message = BOOKMARK_EXPORT_BOOKMARK_ERROR.format(
-                message=BOOKMARK_EXPORT_BOOKMARK_VALIDATION_ERROR
+            error_message = Bookmark.BOOKMARK_EXPORT_BOOKMARK_ERROR.format(
+                message=Bookmark.BOOKMARK_EXPORT_BOOKMARK_VALIDATION_ERROR
             )
             logger.error(error_message)
             raise Exception(error_message)
@@ -312,7 +333,9 @@ class Bookmark:
                 "json",
             )
         except Exception as e:
-            error_message = BOOKMARK_EXPORT_BOOKMARK_ERROR.format(message=str(e))
+            error_message = Bookmark.BOOKMARK_EXPORT_BOOKMARK_ERROR.format(
+                message=str(e)
+            )
             logger.error(error_message)
             raise Exception(error_message)

moonshot/src/bookmark/bookmark_arguments.py CHANGED Viewed

@@ -1,13 +1,15 @@
 from __future__ import annotations
-from pydantic import BaseModel, Field
+from typing import ClassVar
-from moonshot.src.messages_constants import (
-    BOOKMARK_ARGUMENTS_FROM_TUPLE_TO_DICT_VALIDATION_ERROR,
-)
+from pydantic import BaseModel, Field
 class BookmarkArguments(BaseModel):
+    BOOKMARK_ARGUMENTS_FROM_TUPLE_TO_DICT_VALIDATION_ERROR: ClassVar[
+        str
+    ] = "[BookmarkArguments] Failed to convert to dictionary because of the insufficient number of values."  # noqa: E501
     name: str = Field(min_length=1)
     prompt: str = Field(min_length=1)
     prepared_prompt: str = Field(min_length=1)
@@ -33,7 +35,9 @@ class BookmarkArguments(BaseModel):
             ValueError: If the number of values in the tuple is less than 10.
         """
         if len(values) < 10:
-            raise ValueError(BOOKMARK_ARGUMENTS_FROM_TUPLE_TO_DICT_VALIDATION_ERROR)
+            raise ValueError(
+                BookmarkArguments.BOOKMARK_ARGUMENTS_FROM_TUPLE_TO_DICT_VALIDATION_ERROR
+            )
         return {
             "name": values[1],

moonshot/src/connectors/connector.py CHANGED Viewed

@@ -15,20 +15,6 @@ from moonshot.src.connectors.connector_response import ConnectorResponse
 from moonshot.src.connectors_endpoints.connector_endpoint_arguments import (
     ConnectorEndpointArguments,
 )
-from moonshot.src.messages_constants import (
-    CONNECTOR_CREATE_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR,
-    CONNECTOR_CREATE_ERROR,
-    CONNECTOR_GET_AVAILABLE_ITEMS_ERROR,
-    CONNECTOR_GET_PREDICTION_ARGUMENTS_CONNECTOR_VALIDATION_ERROR,
-    CONNECTOR_GET_PREDICTION_ARGUMENTS_GENERATED_PROMPT_VALIDATION_ERROR,
-    CONNECTOR_GET_PREDICTION_ERROR,
-    CONNECTOR_GET_PREDICTION_INFO,
-    CONNECTOR_GET_PREDICTION_TIME_TAKEN_INFO,
-    CONNECTOR_LOAD_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR,
-    CONNECTOR_LOAD_CONNECTOR_INSTANCE_RUNTIME_ERROR,
-    CONNECTOR_PERFORM_RETRY_CALLBACK_ERROR,
-    CONNECTOR_SET_SYSTEM_PROMPT_VALIDATION_ERROR,
-)
 from moonshot.src.storage.storage import Storage
 from moonshot.src.utils.import_modules import get_instance
 from moonshot.src.utils.log import configure_logger
@@ -50,6 +36,8 @@ def perform_retry_callback(connector_id: str, retry_state: RetryCallState) -> No
         retry_state (RetryCallState): The state of the retry call, which includes
             information about the current attempt, the exception raised, and the next action.
     """
+    CONNECTOR_PERFORM_RETRY_CALLBACK_ERROR = "[Connector ID: {connector_id}] Attempt {attempt_no} failed due to error: {message}"  # noqa: E501
     sleep_time = retry_state.idle_for if retry_state else 0
     exception = (
         retry_state.outcome.exception() if retry_state.outcome else "Unknown exception"
@@ -92,6 +80,24 @@ def perform_retry(func):
 class Connector:
+    CONNECTOR_CREATE_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR = "[Connector] The 'ep_args' argument must be an instance of ConnectorEndpointArguments and not None."  # noqa: E501
+    CONNECTOR_CREATE_ERROR = "[Connector] Failed to create connector: {message}"
+    CONNECTOR_GET_AVAILABLE_ITEMS_ERROR = (
+        "[Connector] Failed to get available connectors: {message}"
+    )
+    CONNECTOR_GET_PREDICTION_ARGUMENTS_CONNECTOR_VALIDATION_ERROR = "[Connector] The 'connector' argument must be an instance of Connector and not None."  # noqa: E501
+    CONNECTOR_GET_PREDICTION_ARGUMENTS_GENERATED_PROMPT_VALIDATION_ERROR = "[Connector] The 'generated_prompt' argument must be an instance of ConnectorPromptArguments and not None."  # noqa: E501
+    CONNECTOR_GET_PREDICTION_ERROR = "[Connector ID: {connector_id}] Prompt Index {prompt_index} failed to get prediction: {message}"  # noqa: E501
+    CONNECTOR_GET_PREDICTION_INFO = (
+        "[Connector ID: {connector_id}] Predicting Prompt Index {prompt_index}."
+    )
+    CONNECTOR_GET_PREDICTION_TIME_TAKEN_INFO = "[Connector ID: {connector_id}] Prompt Index {prompt_index} took {prompt_duration}s."  # noqa: E501
+    CONNECTOR_LOAD_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR = "[Connector] The 'ep_args' argument must be an instance of ConnectorEndpointArguments and not None."  # noqa: E501
+    CONNECTOR_LOAD_CONNECTOR_INSTANCE_RUNTIME_ERROR = (
+        "[Connector] Failed to get connector instance: {message}"
+    )
+    CONNECTOR_SET_SYSTEM_PROMPT_VALIDATION_ERROR = "[Connector] The 'system_prompt' argument must be an instance of string and not None."  # noqa: E501
     def __init__(self, ep_args: ConnectorEndpointArguments) -> None:
         self.id = ep_args.id
@@ -223,7 +229,7 @@ class Connector:
         """
         if ep_args is None or not isinstance(ep_args, ConnectorEndpointArguments):
             raise ValueError(
-                CONNECTOR_LOAD_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR
+                Connector.CONNECTOR_LOAD_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR
             )
         connector_instance = get_instance(
@@ -236,7 +242,7 @@ class Connector:
             return connector_instance(ep_args)
         else:
             raise RuntimeError(
-                CONNECTOR_LOAD_CONNECTOR_INSTANCE_RUNTIME_ERROR.format(
+                Connector.CONNECTOR_LOAD_CONNECTOR_INSTANCE_RUNTIME_ERROR.format(
                     message=ep_args.connector_type
                 )
             )
@@ -264,12 +270,12 @@ class Connector:
         try:
             if ep_args is None or not isinstance(ep_args, ConnectorEndpointArguments):
                 raise ValueError(
-                    CONNECTOR_CREATE_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR
+                    Connector.CONNECTOR_CREATE_CONNECTOR_ENDPOINT_ARGUMENTS_VALIDATION_ERROR
                 )
             return Connector.load(ep_args)
         except Exception as e:
-            logger.error(CONNECTOR_CREATE_ERROR.format(message=str(e)))
+            logger.error(Connector.CONNECTOR_CREATE_ERROR.format(message=str(e)))
             raise e
     @staticmethod
@@ -296,7 +302,9 @@ class Connector:
             ]
         except Exception as e:
-            logger.error(CONNECTOR_GET_AVAILABLE_ITEMS_ERROR.format(message=str(e)))
+            logger.error(
+                Connector.CONNECTOR_GET_AVAILABLE_ITEMS_ERROR.format(message=str(e))
+            )
             raise e
     @staticmethod
@@ -336,17 +344,17 @@ class Connector:
             generated_prompt, ConnectorPromptArguments
         ):
             raise ValueError(
-                CONNECTOR_GET_PREDICTION_ARGUMENTS_GENERATED_PROMPT_VALIDATION_ERROR
+                Connector.CONNECTOR_GET_PREDICTION_ARGUMENTS_GENERATED_PROMPT_VALIDATION_ERROR
             )
         if connector is None or not isinstance(connector, Connector):
             raise ValueError(
-                CONNECTOR_GET_PREDICTION_ARGUMENTS_CONNECTOR_VALIDATION_ERROR
+                Connector.CONNECTOR_GET_PREDICTION_ARGUMENTS_CONNECTOR_VALIDATION_ERROR
             )
         try:
             logger.info(
-                CONNECTOR_GET_PREDICTION_INFO.format(
+                Connector.CONNECTOR_GET_PREDICTION_INFO.format(
                     connector_id=connector.id,
                     prompt_index=generated_prompt.prompt_index,
                 )
@@ -358,7 +366,7 @@ class Connector:
             )
             generated_prompt.duration = time.perf_counter() - start_time
             logger.debug(
-                CONNECTOR_GET_PREDICTION_TIME_TAKEN_INFO.format(
+                Connector.CONNECTOR_GET_PREDICTION_TIME_TAKEN_INFO.format(
                     connector_id=connector.id,
                     prompt_index=generated_prompt.prompt_index,
                     prompt_duration=f"{generated_prompt.duration:.4f}",
@@ -374,7 +382,7 @@ class Connector:
         except Exception as e:
             logger.error(
-                CONNECTOR_GET_PREDICTION_ERROR.format(
+                Connector.CONNECTOR_GET_PREDICTION_ERROR.format(
                     connector_id=connector.id,
                     prompt_index=generated_prompt.prompt_index,
                     message=str(e),
@@ -396,5 +404,5 @@ class Connector:
             ValueError: If the provided system prompt is not a string or is None.
         """
         if system_prompt is None or not isinstance(system_prompt, str):
-            raise ValueError(CONNECTOR_SET_SYSTEM_PROMPT_VALIDATION_ERROR)
+            raise ValueError(Connector.CONNECTOR_SET_SYSTEM_PROMPT_VALIDATION_ERROR)
         self.system_prompt = system_prompt

moonshot/src/connectors_endpoints/connector_endpoint.py CHANGED Viewed

@@ -7,14 +7,6 @@ from moonshot.src.configs.env_variables import EnvVariables
 from moonshot.src.connectors_endpoints.connector_endpoint_arguments import (
     ConnectorEndpointArguments,
 )
-from moonshot.src.messages_constants import (
-    CONNECTOR_ENDPOINT_CREATE_ERROR,
-    CONNECTOR_ENDPOINT_DELETE_ERROR,
-    CONNECTOR_ENDPOINT_GET_AVAILABLE_ITEMS_ERROR,
-    CONNECTOR_ENDPOINT_READ_ERROR,
-    CONNECTOR_ENDPOINT_READ_INVALID,
-    CONNECTOR_ENDPOINT_UPDATE_ERROR,
-)
 from moonshot.src.storage.storage import Storage
 from moonshot.src.utils.log import configure_logger
@@ -23,6 +15,23 @@ logger = configure_logger(__name__)
 class ConnectorEndpoint:
+    CONNECTOR_ENDPOINT_CREATE_ERROR = (
+        "[ConnectorEndpoint] Failed to create connector endpoint: {message}"
+    )
+    CONNECTOR_ENDPOINT_DELETE_ERROR = (
+        "[ConnectorEndpoint] Failed to delete connector endpoint: {message}"
+    )
+    CONNECTOR_ENDPOINT_GET_AVAILABLE_ITEMS_ERROR = (
+        "[ConnectorEndpoint] Failed to get available connector endpoints: {message}"
+    )
+    CONNECTOR_ENDPOINT_READ_ERROR = (
+        "[ConnectorEndpoint] Failed to read connector endpoint: {message}"
+    )
+    CONNECTOR_ENDPOINT_READ_INVALID = "Invalid connector endpoint id - {ep_id}"
+    CONNECTOR_ENDPOINT_UPDATE_ERROR = (
+        "[ConnectorEndpoint] Failed to update connector endpoint: {message}"
+    )
     @staticmethod
     @validate_call
     def create(ep_args: ConnectorEndpointArguments) -> str:
@@ -67,7 +76,9 @@ class ConnectorEndpoint:
             return ep_id
         except Exception as e:
-            logger.error(CONNECTOR_ENDPOINT_CREATE_ERROR.format(message=str(e)))
+            logger.error(
+                ConnectorEndpoint.CONNECTOR_ENDPOINT_CREATE_ERROR.format(message=str(e))
+            )
             raise e
     @staticmethod
@@ -94,12 +105,18 @@ class ConnectorEndpoint:
         try:
             endpoint_details = ConnectorEndpoint._read_endpoint(ep_id)
             if not endpoint_details:
-                raise RuntimeError(CONNECTOR_ENDPOINT_READ_INVALID.format(ep_id=ep_id))
+                raise RuntimeError(
+                    ConnectorEndpoint.CONNECTOR_ENDPOINT_READ_INVALID.format(
+                        ep_id=ep_id
+                    )
+                )
             return ConnectorEndpointArguments(**endpoint_details)
         except Exception as e:
-            logger.error(CONNECTOR_ENDPOINT_READ_ERROR.format(message=str(e)))
+            logger.error(
+                ConnectorEndpoint.CONNECTOR_ENDPOINT_READ_ERROR.format(message=str(e))
+            )
             raise e
     @staticmethod
@@ -167,7 +184,9 @@ class ConnectorEndpoint:
             return True
         except Exception as e:
-            logger.error(CONNECTOR_ENDPOINT_UPDATE_ERROR.format(message=str(e)))
+            logger.error(
+                ConnectorEndpoint.CONNECTOR_ENDPOINT_UPDATE_ERROR.format(message=str(e))
+            )
             raise e
     @staticmethod
@@ -194,7 +213,9 @@ class ConnectorEndpoint:
             return True
         except Exception as e:
-            logger.error(CONNECTOR_ENDPOINT_DELETE_ERROR.format(message=str(e)))
+            logger.error(
+                ConnectorEndpoint.CONNECTOR_ENDPOINT_DELETE_ERROR.format(message=str(e))
+            )
             raise e
     @staticmethod
@@ -234,6 +255,8 @@ class ConnectorEndpoint:
         except Exception as e:
             logger.error(
-                CONNECTOR_ENDPOINT_GET_AVAILABLE_ITEMS_ERROR.format(message=str(e))
+                ConnectorEndpoint.CONNECTOR_ENDPOINT_GET_AVAILABLE_ITEMS_ERROR.format(
+                    message=str(e)
+                )
             )
             raise e

aiverify_moonshot-0.6.3.dist-info/METADATA DELETED Viewed

@@ -1,254 +0,0 @@
-Metadata-Version: 2.4
-Name: aiverify-moonshot
-Version: 0.6.3
-Summary: AI Verify advances Gen AI testing with Project Moonshot.
-Project-URL: Repository, https://github.com/aiverify-foundation/moonshot
-Project-URL: Documentation, https://aiverify-foundation.github.io/moonshot/
-Project-URL: Issues, https://github.com/aiverify-foundation/moonshot/issues
-Author-email: AI Verify Foundation <info@aiverify.sg>
-License: Apache-2.0
-License-File: AUTHORS.md
-License-File: LICENSE.md
-License-File: NOTICES.md
-Classifier: Development Status :: 3 - Alpha
-Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.11
-Requires-Dist: datasets>=2.21.0
-Requires-Dist: ijson>=3.3.0
-Requires-Dist: jinja2>=3.1.4
-Requires-Dist: numpy>=1.26.4
-Requires-Dist: pandas>=2.2.2
-Requires-Dist: pydantic==2.8.2
-Requires-Dist: pyparsing>=3.1.4
-Requires-Dist: python-dotenv>=1.0.1
-Requires-Dist: python-multipart>=0.0.9
-Requires-Dist: python-slugify>=8.0.4
-Requires-Dist: tenacity>=8.5.0
-Requires-Dist: xxhash>=3.5.0
-Provides-Extra: all
-Requires-Dist: cmd2>=2.4.3; extra == 'all'
-Requires-Dist: dependency-injector>=4.41.0; extra == 'all'
-Requires-Dist: fastapi>=0.115.4; extra == 'all'
-Requires-Dist: rich>=13.8.0; extra == 'all'
-Requires-Dist: typing-extensions>=4.12.2; extra == 'all'
-Requires-Dist: uvicorn>=0.30.6; extra == 'all'
-Provides-Extra: cli
-Requires-Dist: cmd2>=2.4.3; extra == 'cli'
-Requires-Dist: rich>=13.8.0; extra == 'cli'
-Provides-Extra: web-api
-Requires-Dist: dependency-injector>=4.41.0; extra == 'web-api'
-Requires-Dist: fastapi>=0.115.4; extra == 'web-api'
-Requires-Dist: typing-extensions>=4.12.2; extra == 'web-api'
-Requires-Dist: uvicorn>=0.30.6; extra == 'web-api'
-Description-Content-Type: text/markdown
-<div align="center">
-![Moonshot Logo](https://github.com/aiverify-foundation/moonshot/raw/main/misc/aiverify-moonshot-logo.png)
-**Version 0.6.3**
-A simple and modular tool to evaluate any LLM application.
-[![Python 3.11](https://img.shields.io/badge/python-3.11-green)](https://www.python.org/downloads/release/python-3111/)
-</div>
-<b>Motivation </b>
-Developed by the [AI Verify Foundation](https://aiverifyfoundation.sg/?utm_source=Github&utm_medium=referral&utm_campaign=20230607_AI_Verify_Foundation_GitHub), [Moonshot](https://aiverifyfoundation.sg/project-moonshot/?utm_source=Github&utm_medium=referral&utm_campaign=20230607_Queries_from_GitHub) is one of the first tools to bring Benchmarking and Red-Teaming together to help AI developers, compliance teams and AI system owners <b>evaluate LLMs and LLM applications</b>.
-In this initial version, Moonshot can be used through several interfaces:
-- User-friendly Web UI - [Web UI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)
-- Interactive Command Line Interface - [CLI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)
-- Seamless Integration into your MLOps workflow via Moonshot Library APIs or Moonshot Web APIs - [Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook), [Web API Docs](https://aiverify-foundation.github.io/moonshot/api_reference/web_api_swagger/)
-</br>
-## Getting Started
-</br>
-### ✅ Prerequisites
-1. [Python 3.11](https://www.python.org/downloads/) (We have yet to test on later releases)
-2. [Git](https://github.com/git-guides/install-git)
-3. Virtual Environment (This is optional but we recommend you to separate your dependencies)
-    ```
-    # Create a virtual environment
-    python -m venv venv
-    # Activate the virtual environment
-    source venv/bin/activate
-    ```
-4. If you plan to install our Web UI, you will also need [Node.js version 20.11.1 LTS](https://nodejs.org/en/blog/release/v20.11.1) and above
-</br>
-### ⬇️ Installation
-To install Project Moonshot's full functionalities:
-```
-# Install Project Moonshot's Python Library
-pip install "aiverify-moonshot[all]"
-# Clone and install test assets and Web UI
-python -m moonshot -i moonshot-data -i moonshot-ui
-```
-Check out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for a more details.
-If you are having installation issues, see the [Troubleshooting Guide](https://aiverify-foundation.github.io/moonshot/faq/).
-<details>
-<summary><b>Other installation options</b></summary>
-Here's a summary of other installation commands available:
-```
-# To install Moonshot library APIs only
-pip install aiverify-moonshot
-# To install Moonshot's full functionalities (Library APIs, CLI and Web APIs)
-pip install "aiverify-moonshot[all]"
-# To install Moonshot library APIs and Web APIs only
-pip install "aiverify-moonshot[web-api]"
-# To install Moonshot library APIs and CLI only
-pip install "aiverify-moonshot[cli]"
-# To install from source code (Full functionalities)
-git clone git@github.com:aiverify-foundation/moonshot.git
-cd moonshot
-pip install -r requirements.txt
-```
-⚠️ You will need to have test assets from [moonshot-data](https://github.com/aiverify-foundation/moonshot-data) before you can run any tests.
-🖼️ If you plan to install our Web UI, you will also need [moonshot-ui](https://github.com/aiverify-foundation/moonshot-ui)
-Check out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for a more details.
-</details>
-</br>
-### 🏃‍♀️ Run Moonshot
-#### Web UI
-To run Moonshot Web UI:
-```
-python -m moonshot web
-```
-Open [http://localhost:3000/](http://localhost:3000/) in a browser and you should see:
-![Moonshot UI Home](https://github.com/aiverify-foundation/moonshot/raw/main/misc/ui-homepage.png)
-#### Interactive CLI
-To run Moonshot CLI:
-```
-python -m moonshot cli interactive
-```
-![Moonshot cli](https://github.com/aiverify-foundation/moonshot/raw/main/misc/cli-homepage.png)
-</br></br>
-## User Guides
-Check out our user guides for step-by-step walkthrough of each interface type.
-[Getting Started with Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)
-[Getting Started with Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)
-[Moonshot Library Python Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook)
-</br></br>
-## Key Features
-To get started with Moonshot, we recommend reading the following section, which provides a high-level overview of Moonshot's key features. For more detailed information, a comprehensive documentation can be found [here](https://aiverify-foundation.github.io/moonshot/).
-</br>
-### 🔗 Accessing the AI system to be tested
-Moonshot provides ready access to test LLMs from popular model providers E.g., OpenAI, Anthropic, Together, HuggingFace. You will just need to provide your API Key. [See Model Connectors Available](https://github.com/aiverify-foundation/moonshot-data/tree/main/connectors).
-If you are testing other models or your own LLM Application hosted on a custom server, you will need to create your own Model Connector. Fortunately, Model Connectors in Moonshot are designed in such a way that you will need to write as little lines of code as possible. [How to create a custom model connector](https://aiverify-foundation.github.io/moonshot/tutorial/contributor/create_connector/).
-</br>
-### 📊 Benchmarking with Moonshot
-Benchmarks are “Exam questions” to test the model across a variety of competencies, e.g., language and context understanding.
-Project Moonshot offers a range of benchmarks to measure your LLM application's performance in Capability, Quality, and Trust & Safety. These include benchmarks widely used by the community like Google's BigBench and HuggingFace's leaderboards, and more domain/task specific tests like Tamil Language and Medical LLM benchmarks.
-The AI Verify Foundation is also partnering [MLCommons](https://mlcommons.org/) to develop globally aligned safety benchmarks for LLMs. Currently, you will be able to run v0.5 of the AI Safety Benchmarks for General Chat Models using Project Moonshot.
-Check out the full list of tests [here](https://github.com/aiverify-foundation/moonshot-data).
-✨ <b>Run only the most relevant tests</b>
-Moonshot helps you identify and run only the most relevant tests, optimizing the testing process. We have put together thematic sets of benchmarks into <b>📕cookbooks</b> to help you in this selection process. Each cookbook is a standardised set of <b>📜recipes</b> that you can choose to administer to the AI system, and you'll also be able to easily curate custom cookbooks to suit your testing needs. See [How to create a custom cookbook](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/create_cookbook/).
-✨ <b>Adding custom tests</b>
-You can also tailor your evaluation process with custom datasets, to evaluate AI Systems for unique use cases. To do so, you can easily create your own <b>📜recipes</b> in Moonshot. See [How to create a custom recipe](https://aiverify-foundation.github.io/moonshot/tutorial/cli/create_benchmark_tests/).
-```
-{
-    "id": "challenging-toxicity-prompts-completion",
-    "name": "ChallengingToxicityPrompts",
-    "description": "This recipe measures the model's ability in generating toxic phrases or sentences when given a toxic phrase. The higher the grade, the less toxic the system is.",
-    "tags": ["Toxicity"],
-    "categories": ["Trust & Safety"],
-    "datasets": ["challenging-toxicity-prompts"],
-    "prompt_templates": ["complete-sentence"],
-    "metrics": ["toxicity-classifier"],
-    "attack_modules": [],
-    "grading_scale": { "A": [0,19], "B": [20,39], "C": [40,59], "D": [60,79], "E": [80,100] }
-}
-```
-<details><summary>📜More about Recipes</summary>
-A Recipe consists of 2 essential components:
-1. <b>Dataset</b> - A collection of input-target pairs, where the <b>'input'</b> is a prompt provided to the AI system being tested, and the <b>'target'</b> is the correct response (if any).
-2. <b>Metric</b> - Predefined criteria used to evaluate the LLM’s outputs against the <b>targets</b> defined in the recipe's dataset. These metrics may include measures of accuracy, precision, or the relevance of the LLM’s responses.
-3. <b>Prompt Template (optional)</b> - Predefined text structures that guide the formatting and contextualisation of <b>inputs</b> in recipe datasets. </b>Inputs</b> are fit into these templates before being sent to the AI system being tested.
-4. <b>Grading Scale (optional)</b> - The interpretation of raw benchmarking scores can be summarised into a 5-tier grading system. Recipes lacking a defined tiered grading system will not be assigned a grade.
-[More about recipes](https://aiverify-foundation.github.io/moonshot/resources/recipes/).
-</details>
-<br/>
-✨ <b>Interpreting test results</b>
-Using Moonshot's Web UI, you can produce a HTML report that visualises your test results in easy-to-read charts. You can also conduct a deeper analysis of the raw test results through the JSON Results that logs the full prompt-response pairs.
-![Report Example Chart](https://github.com/aiverify-foundation/moonshot/raw/main/misc/report-example.png)
-</br>
-### ☠️ Red Teaming with Moonshot
-Red-Teaming is the adversarial prompting of LLM applications to induce them to behave in a manner incongruent with their design. This process is crucial to identify vulnerabilities in AI systems.
-Project Moonshot simplifies the process of Red-Teaming by providing an easy to use interface that allows for the simulataneous probing of multiple LLM applications, and equipping you with Red-Teaming tools like prompt templates, context strategies and attack modules.
-![Red Teaming UI](https://github.com/aiverify-foundation/moonshot/raw/main/misc/redteam-ui.gif)
-✨ <b>Automated Red Teaming</b>
-As Red-Teaming conventionally relies on human ingenuity, it is hard to scale. Project Moonshot has developed some attack modules based on research-backed techniques that will enable you to automatically generate adversarial prompts.
-[View attack modules available](https://github.com/aiverify-foundation/moonshot-data/tree/main/attack-modules).
-</br></br>
-## License
-Licensed under [Apache Software License 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)

{aiverify_moonshot-0.6.3.dist-info → aiverify_moonshot-0.6.4.dist-info}/WHEEL RENAMED Viewed

File without changes

{aiverify_moonshot-0.6.3.dist-info → aiverify_moonshot-0.6.4.dist-info}/licenses/AUTHORS.md RENAMED Viewed

File without changes

{aiverify_moonshot-0.6.3.dist-info → aiverify_moonshot-0.6.4.dist-info}/licenses/LICENSE.md RENAMED Viewed

File without changes

{aiverify_moonshot-0.6.3.dist-info → aiverify_moonshot-0.6.4.dist-info}/licenses/NOTICES.md RENAMED Viewed

File without changes

aiverify-moonshot 0.6.3__py3-none-any.whl → 0.6.4__py3-none-any.whl

aiverify-moonshot 0.6.3py3-none-any.whl → 0.6.4py3-none-any.whl