trismik 0.9.0__py3-none-any.whl → 0.9.4__py3-none-any.whl

trismik-0.9.4.dist-info/METADATA

@@ -0,0 +1,172 @@
+Metadata-Version: 2.3
+Name: trismik
+Version: 0.9.4
+Summary: 
+Author: Bartosz Kielczewski
+Author-email: bk352@cam.ac.uk
+Requires-Python: >=3.9
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Provides-Extra: examples
+Requires-Dist: accelerate (>=1.7.0,<2.0.0) ; extra == "examples"
+Requires-Dist: notebook (>=7.4.4,<8.0.0) ; extra == "examples"
+Requires-Dist: openai (>=1.81.0,<2.0.0) ; extra == "examples"
+Requires-Dist: torch (>=2.7.0,<3.0.0) ; extra == "examples"
+Requires-Dist: torchaudio (>=2.7.0,<3.0.0) ; extra == "examples"
+Requires-Dist: torchvision (>=0.22.0,<1.0.0) ; extra == "examples"
+Requires-Dist: transformers (>=4.51.3,<5.0.0) ; extra == "examples"
+Description-Content-Type: text/markdown
+
+<h1 align="center"> Trismik SDK</h1>
+
+<p align="center">
+  <img alt="PyPI - Version" src="https://img.shields.io/pypi/v/trismik">
+  <img alt="Python Version" src="https://img.shields.io/badge/python-3.9%2B-blue">
+  <img alt="License" src="https://img.shields.io/badge/license-MIT-green">
+</p>
+
+## Table of Contents
+- [Overview](#overview)
+- [Quick Start](#quick-start)
+  - [Installation](#installation)
+  - [API Key Setup](#api-key-setup)
+  - [Basic Usage](#basic-usage)
+- [Interpreting Results](#interpreting-results)
+  - [Theta (θ)](#theta-θ)
+  - [Other Metrics](#other-metrics)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Overview
+
+[**Trismik**](https://trismik.com) is a Cambridge, UK based startup offering adversarial testing for LLMs. The APIs we provide through this library allow you to call our adaptive test engine and evaluate LLMs up to 95% faster (and cheaper!) than traditional evaluation techniques.
+
+Our **adaptive testing** algorithm allows to estimate the precision of the model by looking only at a small portion of a dataset. Through this library, we provide access to a number of open source datasets over several dimensions (reasoning, toxicity, tool use...) to speed up model testing in several scenarios, like foundation model training, supervised fine tuning, prompt engineering, and so on.
+
+## Quick Start
+
+### Installation
+
+To use our API, you need to get an API key first. Please register on [dashboard.trismik.com](https://dashboard.trismik.com) and obtain an API key.
+
+Trismik is available via [pypi](https://pypi.org/project/trismik/). To install Trismik, run the following in your terminal (in a virtualenv, if you use one):
+
+```bash
+pip install trismik
+```
+
+### API Key Setup
+
+You can provide your API key in one of the following ways:
+
+1. **Environment Variable**:
+   ```bash
+   export TRISMIK_API_KEY="your-api-key"
+   ```
+
+2. **`.env` File**:
+   ```bash
+   # .env
+   TRISMIK_API_KEY=your-api-key
+   ```
+   Then load it with `python-dotenv`:
+   ```python
+   from dotenv import load_dotenv
+   load_dotenv()
+   ```
+
+3. **Direct Initialization**:
+   ```python
+   client = TrismikAsyncClient(api_key="YOUR_API_KEY")
+   ```
+
+### Basic Usage
+
+Running a test is straightforward:
+
+1. Implement a method that wraps model inference over a dataset item
+2. Create an `AdaptiveTest` instance
+3. Run the test!
+
+Here's a basic example:
+
+```python
+def model_inference(item: TrismikItem) -> Any:
+    model_output = ...  # call your model here
+    return model_output
+
+
+# Initialize the test runner
+runner = AdaptiveTest(model_inference)
+
+# Run the test
+results = await runner.run_async(
+    "MMLUPro2025",  # or any dataset we support
+    with_responses=True,
+    run_metadata=sample_metadata,
+)
+
+# Print the test output
+for result in results:
+    print(f"{result.trait} ({result.name}): {result.value}")
+```
+
+### Examples
+
+You can find more examples in the `examples` folder:
+- [`example_transformers.py`](examples/example_transformers.py) - Example using Hugging Face Transformers models
+- [`example_openai.py`](examples/example_openai.py) - Example using OpenAI models
+- [`example_adaptive_test.py`](examples/example_adaptive_test.py) - Example of adaptive testing configuration
+
+To run the examples, you will need to clone this repo, navigate to the
+source folder, and then run:
+
+```bash
+poetry install --with examples
+poetry run python examples/example_adaptive_test.py # or any other example
+```
+
+## Interpreting Results
+
+### Theta (θ)
+
+Our adversarial test returns several values; however, you will be interested mainly in `theta`. Theta ($\theta$) is our metric; it measures the ability of the model on a certain dataset, and it can be used as a proxy to approximate the original metric used on that dataset. For example, on an accuracy-based dataset, a high theta correlates with a high accuracy, and low theta correlates with low accuracy.
+
+To interpret a theta score, consider that $\theta=0$ corresponds to a 50% chance for a model to get an answer right - in other words, to an accuracy of 50%.
+A negative theta means that the model will give more bad answers then good ones, while a positive theta means that the model will give more good answers then bad answers.
+While theta is unbounded in our implementation (i.e. $-\infty < \theta < \infty$), in practice we have that for most cases $\theta$ will take values between -3 and 3.
+
+Compared to classical benchmark testing, the estimated accuracy from adaptive testing uses fewer but more informative items while avoiding noise from overly easy or difficult questions. This makes it a more efficient and stable measure, especially on very large datasets.
+
+### Other Metrics
+
+- **Standard Deviation (`std`)**:
+  - A measure of the uncertainty or error in the theta estimate
+  - A smaller `std` indicates a more precise estimate
+  - You should see a `std` around or below 0.25
+
+- **Correct Responses (`responsesCorrect`)**:
+  - The number of correct answers delivered by the model
+
+  - **Important note**: A higher number of correct answers does not necessarily
+  correlate with a high theta. Our algorithm navigates the dataset to find a
+   balance of “hard” and “easy” items for your model, so by the end of the test,
+  it encounters a representative mix of inputs it can and cannot handle. In
+   practice, expect responsesCorrect to be roughly half of responsesTotal.
+
+- **Total Responses (`responsesTotal`)**:
+  - The number of items processed before reaching a stable theta.
+  - Expected range: 60 ≤ responses_total ≤ 80
+
+## Contributing
+
+See `CONTRIBUTING.md`.
+
+## License
+
+This library is licensed under the MIT license. See `LICENSE` file.
+

trismik-0.9.4.dist-info/RECORD

@@ -0,0 +1,12 @@
+trismik/__init__.py,sha256=20SwXrda9YsgykaoPohwz6foj2FkraniPA-GTQS9m00,197
+trismik/_mapper.py,sha256=A1mik1jlF7hm9oyq5Of-BZtA16MVhVJv1AI_20PKv2k,10315
+trismik/_utils.py,sha256=WZ7x0EaG7PdXdFZMIs1wzgwqiPQm59QDDmQzBKYGCEg,3753
+trismik/adaptive_test.py,sha256=hLGdf0jgLmJJ55I9c5QjaEd1bCA8QNfhJJoOBkF38ik,20002
+trismik/client_async.py,sha256=ZOvAH-nTnh_S3mrpLuprxtByZBasTnNCysBwUPwXMb0,12523
+trismik/exceptions.py,sha256=2wb4_K7GdDf00s3xUaiSfw6718ZV3Eaa4M2lYbiEZl4,1945
+trismik/settings.py,sha256=FCP-d8ZEiYWUTWoa9nOVzSwTOLvg8y0pU08dAseiOwY,412
+trismik/types.py,sha256=OS0npQV6FgN5DRE1Ebd4tY1Z8_R_K6SSzRdHvKdosug,6240
+trismik-0.9.4.dist-info/LICENSE,sha256=tgetRhapGLh7ZxfknW6Mm-WobfziPd64nAK52X5XKaw,1077
+trismik-0.9.4.dist-info/METADATA,sha256=2N19pz_qSvCRolCF_2Cw7TaScnXRopj7gy87KlXiyoQ,6619
+trismik-0.9.4.dist-info/WHEEL,sha256=b4K_helf-jlQoXBBETfwnf4B04YC67LOev0jo4fX5m8,88
+trismik-0.9.4.dist-info/RECORD,,

{trismik-0.9.0.dist-info → trismik-0.9.4.dist-info}/WHEEL

@@ -1,4 +1,4 @@
 Wheel-Version: 1.0
-Generator: poetry-core 1.9.0
+Generator: poetry-core 2.1.3
 Root-Is-Purelib: true
 Tag: py3-none-any

trismik/client.py

@@ -1,274 +0,0 @@
-from typing import List, Any, Optional
-
-import httpx
-
-from ._mapper import TrismikResponseMapper
-from ._utils import TrismikUtils
-from .exceptions import TrismikApiError
-from .types import (
-    TrismikTest,
-    TrismikAuth,
-    TrismikSession,
-    TrismikItem,
-    TrismikResult,
-    TrismikResponse,
-)
-
-
-class TrismikClient:
-    _serviceUrl: str = "https://trismik.e-psychometrics.com/api"
-
-    def __init__(
-            self,
-            service_url: Optional[str] = None,
-            api_key: Optional[str] = None,
-            http_client: Optional[httpx.Client] | None = None,
-    ) -> None:
-        """
-        Initializes a new Trismik client.
-
-        Args:
-            service_url (Optional[str]): URL of the Trismik service.
-            api_key (Optional[str]): API key for the Trismik service.
-            http_client (Optional[httpx.Client]): HTTP client to use for requests.
-
-        Raises:
-            TrismikError: If service_url or api_key are not provided and not found in environment.
-            TrismikApiError: If API request fails.
-        """
-        self._service_url = TrismikUtils.option(
-                service_url, self._serviceUrl, "TRISMIK_SERVICE_URL"
-        )
-        self._api_key = TrismikUtils.required_option(
-                api_key, "api_key", "TRISMIK_API_KEY"
-        )
-        self._http_client = http_client or httpx.Client(
-                base_url=self._service_url)
-
-    def authenticate(self) -> TrismikAuth:
-        """
-        Authenticates with the Trismik service.
-
-        Returns:
-            TrismikAuth: Authentication token.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = "/client/auth"
-            body = {"apiKey": self._api_key}
-            response = self._http_client.post(url, json=body)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_auth(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def refresh_token(self, token: str) -> TrismikAuth:
-        """
-        Refreshes the authentication token.
-
-        Args:
-            token (str): Current authentication token.
-
-        Returns:
-            TrismikAuth: New authentication token.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = "/client/token"
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.get(url, headers=headers)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_auth(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def available_tests(self, token: str) -> List[TrismikTest]:
-        """
-        Retrieves a list of available tests.
-
-        Args:
-            token (str): Authentication token.
-
-        Returns:
-            List[TrismikTest]: List of available tests.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = "/client/tests"
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.get(url, headers=headers)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_tests(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def create_session(self, test_id: str, token: str) -> TrismikSession:
-        """
-        Creates a new session for a test.
-
-        Args:
-            test_id (str): ID of the test.
-            token (str): Authentication token.
-
-        Returns:
-            TrismikSession: New session
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = "/client/sessions"
-            headers = {"Authorization": f"Bearer {token}"}
-            body = {"testId": test_id, }
-            response = self._http_client.post(url, headers=headers, json=body)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_session(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def current_item(
-            self,
-            session_url: str,
-            token: str
-    ) -> TrismikItem:
-        """
-        Retrieves the current test item.
-
-        Args:
-            session_url (str): URL of the session.
-            token (str): Authentication token.
-
-        Returns:
-            TrismikItem: Current test item.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = f"{session_url}/item"
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.get(url, headers=headers)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_item(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def respond_to_current_item(
-            self,
-            session_url: str,
-            value: Any,
-            token: str
-    ) -> TrismikItem | None:
-        """
-        Responds to the current test item.
-
-        Args:
-            session_url (str): URL of the session.
-            value (Any): Response value.
-            token (str): Authentication token.
-
-        Returns:
-            TrismikItem | None: Next test item or None if session is finished.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = f"{session_url}/item"
-            body = {"value": value}
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.post(url, headers=headers, json=body)
-            response.raise_for_status()
-            if response.status_code == 204:
-                return None
-            else:
-                json = response.json()
-                return TrismikResponseMapper.to_item(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def results(self, session_url: str, token: str) -> List[TrismikResult]:
-        """
-        Retrieves the results of a session.
-
-        Args:
-            session_url (str): URL of the session.
-            token (str): Authentication token.
-
-        Returns:
-            List[TrismikResult]: Results of the session.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = f"{session_url}/results"
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.get(url, headers=headers)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_results(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e
-
-    def responses(self,
-            session_url: str,
-            token: str
-    ) -> List[TrismikResponse]:
-        """
-        Retrieves responses to session items.
-
-        Args:
-            session_url (str): URL of the session.
-            token (str): Authentication token.
-
-        Returns:
-            List[TrismikResponse]: Responses of the session.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        try:
-            url = f"{session_url}/responses"
-            headers = {"Authorization": f"Bearer {token}"}
-            response = self._http_client.get(url, headers=headers)
-            response.raise_for_status()
-            json = response.json()
-            return TrismikResponseMapper.to_responses(json)
-        except httpx.HTTPStatusError as e:
-            raise TrismikApiError(
-                    TrismikUtils.get_error_message(e.response)) from e
-        except httpx.HTTPError as e:
-            raise TrismikApiError(str(e)) from e

trismik/runner.py

@@ -1,85 +0,0 @@
-from datetime import datetime, timedelta
-from typing import List, Callable, Any, Optional
-
-from .client import TrismikClient
-from .types import (
-    TrismikAuth,
-    TrismikItem,
-    TrismikResult,
-    TrismikResultsAndResponses,
-)
-
-
-class TrismikRunner:
-    def __init__(
-            self,
-            item_processor: Callable[[TrismikItem], Any],
-            client: Optional[TrismikClient] = None,
-            auth: Optional[TrismikAuth] = None,
-    ) -> None:
-        """
-        Initializes a new Trismik runner.
-
-        Args:
-            item_processor (Callable[[TrismikItem], Any]): Function to process test items.
-            client (Optional[TrismikClient]): Trismik client to use for requests.
-            auth (Optional[TrismikAuth]): Authentication token to use for requests
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        self._item_processor = item_processor
-        self._client = client
-        self._auth = auth
-
-    def run(self,
-            test_id: str,
-            with_responses: bool = False,
-    ) -> List[TrismikResult] | TrismikResultsAndResponses:
-        """
-        Runs a test.
-
-        Args:
-            test_id (str): ID of the test to run.
-            with_responses (bool): If True, responses will be included with the results.
-
-        Returns:
-            List[TrismikResult] | TrismikResultsAndResponses: Either just test results, or with responses.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        self._init()
-        self._refresh_token_if_needed()
-        session = self._client.create_session(test_id, self._auth.token)
-        self._run_session(session.url)
-        results = self._client.results(session.url, self._auth.token)
-
-        if with_responses:
-            responses = self._client.responses(session.url, self._auth.token)
-            return TrismikResultsAndResponses(results, responses)
-        else:
-            return results
-
-    def _run_session(self, session_url: str) -> None:
-        item = self._client.current_item(session_url, self._auth.token)
-        while item is not None:
-            self._refresh_token_if_needed()
-            response = self._item_processor(item)
-            item = self._client.respond_to_current_item(
-                    session_url, response, self._auth.token
-            )
-
-    def _init(self) -> None:
-        if self._client is None:
-            self._client = TrismikClient()
-
-        if self._auth is None:
-            self._auth = self._client.authenticate()
-
-    def _refresh_token_if_needed(self) -> None:
-        if self._token_needs_refresh():
-            self._auth = self._client.refresh_token(self._auth.token)
-
-    def _token_needs_refresh(self) -> bool:
-        return self._auth.expires < (datetime.now() + timedelta(minutes=5))

trismik/runner_async.py

@@ -1,87 +0,0 @@
-from datetime import datetime, timedelta
-from typing import List, Callable, Any, Awaitable, Optional
-
-from .client_async import TrismikAsyncClient
-from .types import (
-    TrismikAuth,
-    TrismikItem,
-    TrismikResult,
-    TrismikResultsAndResponses,
-)
-
-
-class TrismikAsyncRunner:
-    def __init__(
-            self,
-            item_processor: Callable[[TrismikItem], Awaitable[Any]],
-            client: Optional[TrismikAsyncClient] = None,
-            auth: Optional[TrismikAuth] = None,
-    ) -> None:
-        """
-        Initializes a new Trismik runner (async version).
-
-        Args:
-            item_processor (Callable[[TrismikItem], Any]): Function to process test items.
-            client (Optional[TrismikClient]): Trismik client to use for requests.
-            auth (Optional[TrismikAuth]): Authentication token to use for requests
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        self._item_processor = item_processor
-        self._client = client
-        self._auth = auth
-
-    async def run(self,
-            test_id: str,
-            with_responses: bool = False,
-    ) -> List[TrismikResult] | TrismikResultsAndResponses:
-        """
-        Runs a test.
-
-        Args:
-            test_id (str): ID of the test to run.
-            with_responses (bool): If True, responses will be included with the results.
-
-        Returns:
-            List[TrismikResult] | TrismikResultsAndResponses: Either just test results, or with responses.
-
-        Raises:
-            TrismikApiError: If API request fails.
-        """
-        await self._init()
-        await self._refresh_token_if_needed()
-        session = await self._client.create_session(test_id, self._auth.token)
-        await self._run_session(session.url)
-        results = await self._client.results(session.url, self._auth.token)
-
-        if with_responses:
-            responses = await self._client.responses(session.url,
-                                                     self._auth.token)
-            return TrismikResultsAndResponses(results, responses)
-        else:
-            return results
-
-    async def _run_session(self, session_url: str) -> None:
-        await self._init()
-        await self._refresh_token_if_needed()
-        item = await self._client.current_item(session_url, self._auth.token)
-        while item is not None:
-            await self._refresh_token_if_needed()
-            response = await self._item_processor(item)
-            item = await self._client.respond_to_current_item(
-                    session_url, response, self._auth.token)
-
-    async def _init(self) -> None:
-        if self._client is None:
-            self._client = TrismikAsyncClient()
-
-        if self._auth is None:
-            self._auth = await self._client.authenticate()
-
-    async def _refresh_token_if_needed(self) -> None:
-        if self._token_needs_refresh():
-            self._auth = await self._client.refresh_token(self._auth.token)
-
-    def _token_needs_refresh(self) -> bool:
-        return self._auth.expires < (datetime.now() + timedelta(minutes=5))

trismik-0.9.0.dist-info/METADATA

@@ -1,60 +0,0 @@
-Metadata-Version: 2.1
-Name: trismik
-Version: 0.9.0
-Summary: 
-Author: Bartosz Kielczewski
-Author-email: bk352@cam.ac.uk
-Requires-Python: >=3.8
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: httpx (>=0.27.2,<0.28.0)
-Requires-Dist: python-dateutil (>=2.9.0.post0,<3.0.0)
-Description-Content-Type: text/markdown
-
-Trismik Python SDK
-==================
-
-This is the official Python SDK for Trismik. It provides a simple way to interact with the Trismik
-API.
-
-Usage
------
-
-1. ```pip install trismik```
-2. Set the following environment variable. Alternatively, put it into `.env` file
-   in the root of your project, and load them using `python-dotenv` package:
-
-   ```
-   TRISMIK_API_KEY=<api_key>
-   ```
-
-3. Refer to examples:
-   * [example_runner.py](./examples/example_runner.py) - run test using high-level `TrismikRunner`
-     class
-   * [example_runner_async.py](./examples/example_runner_async.py) - like above, but with async
-     support
-   * [example_client.py](./examples/example_client.py) - run test using `TrismikClient` directly
-   * [example_client_async.py](./examples/example_client_async.py) - like above, but with async
-     support
-
-Contributing
-------------
-
-1. Install [Python Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer)
-2. ```git clone https://github.com/trismik/trismik-python.git```, or if cloned previously:
-   ```git pull``` to update
-3. ```cd ./trismik-python```
-4. ```poetry install```
-5. ```poetry run pytest```
-
-Publishing to TestPyPi
-----------------------
-
-1. ```poetry config repositories.testpypi https://test.pypi.org/legacy/```
-2. ```poetry config pypi-token.testpypi <token>```
-3. ```poetry publish --build --repository testpypi```
-