mankinds-sdk 1.0.0__tar.gz

mankinds_sdk-1.0.0/PKG-INFO

@@ -0,0 +1,413 @@
+Metadata-Version: 2.4
+Name: mankinds-sdk
+Version: 1.0.0
+Summary: Python SDK for Mankinds API
+Author-email: Mankinds <info@mankinds.com>
+License: MIT
+Keywords: mankinds,api,sdk
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: isort>=5.11; extra == "dev"
+Requires-Dist: flake8>=5.0; extra == "dev"
+Requires-Dist: mypy>=0.990; extra == "dev"
+
+<div align="center">
+
+
+<h1>
+    <img src="https://gitlab.com/mankinds1/mankinds-sdk/-/raw/main/assets/logo.png" alt="Mankinds" width="32" height="27" />
+    Mankinds SDK
+</h1>
+
+[![PyPI version](https://img.shields.io/pypi/v/mankinds-sdk?logo=pypi&logoColor=white)](https://pypi.org/project/mankinds-sdk/)
+[![Build Status](https://img.shields.io/gitlab/pipeline/mankinds1/mankinds-sdk?branch=main&logo=gitlab&logoColor=white)](https://gitlab.com/mankinds1/mankinds-sdk/-/pipelines)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?logo=opensourceinitiative&logoColor=white)](../../LICENSE)
+[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg?logo=readthedocs&logoColor=white)](https://docs.mankinds.io)
+
+*Evaluate AI system with automated tests.*
+
+Register an AI system, optionally attach connectors (logs, databases), import or generate your golden dataset, and run automated evaluations covering privacy, security, performance, fairness, explainability, transparency and accountability.
+
+</div>
+
+---
+
+## Features
+
+- **System Management** — Create, update, and configure AI systems with custom API endpoints
+- **Endpoint Configuration** — Support for REST, SSE streaming, and multi-turn conversations
+- **Dataset Generation** — Auto-generate or provide custom test scenarios
+- **Evaluation** — Run evaluations with real-time polling and configurable profiles
+- **Connectors** — Attach data sources (log files, Datadog, SQLite, PostgreSQL)
+- **Error Handling** — Typed exceptions for all error cases
+
+## Documentation
+
+- [Mankinds Documentation](https://docs.mankinds.io)
+
+## Requirements
+
+- Python ≥ 3.8
+
+## Installation
+
+```bash
+pip install mankinds-sdk
+```
+
+## Usage
+
+The SDK follows a simple 3-step workflow: **create a system, generate test data, run an evaluation**.
+
+### Initialize the Client
+
+```python
+from mankinds_sdk import MankindsClient
+
+client = MankindsClient(api_key="mk_...")
+```
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `api_key` | `str` | Yes | — | Your API key |
+| `base_url` | `str` | No | `https://app.mankinds.io` | Custom API base URL |
+| `timeout` | `int` | No | `120` | Request timeout in seconds |
+
+### Create an AI System
+
+Register your AI system by providing its name, description, and API endpoint. The endpoint defines how your AI is called during evaluation.
+
+```python
+system = client.create_system(
+    name="Customer Support Bot",
+    description="A chatbot that handles order inquiries and returns for an e-commerce platform.",
+    endpoint={
+        "url": "https://api.example.com/chat",
+        "method": "POST",
+        "headers": {"Authorization": "Bearer your-token"},
+        "body": {"message": "{{input}}"},
+        "response": {"answer": "{{output}}"}
+    }
+)
+
+system_id = system["id"]
+```
+
+Use `{{input}}` in the request body and `{{output}}` in the response mapping so test inputs and expected outputs are bound during evaluation.
+
+### Endpoint Configuration
+
+The **endpoint** defines how the API calls your AI system during evaluation. It's a JSON object that describes your API's request/response format.
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `url` | string | Yes | API endpoint URL |
+| `method` | string | Yes | HTTP method (`POST`, `GET`, etc.) |
+| `body` | object | Yes | Request body with `{{input}}` placeholder |
+| `response` | object | Yes | Response mapping with `{{output}}` placeholder |
+| `headers` | object | No | HTTP headers |
+| `streaming` | object | No | SSE streaming configuration |
+| `multiturn` | object | No | Multi-turn conversation configuration |
+
+**Placeholders:**
+
+- `{{input}}` in `body`: replaced with test inputs during evaluation
+- `{{output}}` in `response`: indicates which field contains the AI response
+
+```python
+"body": {"message": "{{input}}"},
+"response": {"answer": "{{output}}"}
+```
+
+**Streaming (SSE):**
+
+```python
+endpoint = {
+    "url": "https://api.example.com/chat",
+    "method": "POST",
+    "body": {"message": "{{input}}"},
+    "response": {"answer": "{{output}}"},
+    "streaming": {
+        "enabled": True,
+        "format": "openai",  # "openai" | "anthropic" | "custom"
+        "content_path": "choices[0].delta.content"
+    }
+}
+```
+
+**Multi-turn conversations:**
+
+```python
+endpoint = {
+    "url": "https://api.example.com/chat",
+    "method": "POST",
+    "body": {"message": "{{input}}", "session_id": "{{session}}"},
+    "response": {"answer": "{{output}}"},
+    "multiturn": {
+        "type": "session_id",  # "none" | "session_id" | "history"
+        "field": "conversation_id",
+        "location": "body"
+    }
+}
+```
+
+### Generate Evaluation Dataset
+
+Test scenarios can be auto-generated based on your system description, or you can provide custom scenarios.
+
+**Auto-generate scenarios:**
+
+```python
+dataset = client.generate_dataset(system_id, num_scenarios=20)
+```
+
+**Provide custom scenarios:**
+
+```python
+dataset = client.generate_dataset(
+    system_id,
+    scenarios=[
+        {"input": "Where is my order?", "outputs": ["I can help you track your order."]},
+        {"input": "I want a refund", "outputs": ["I'll process your refund request."]}
+    ]
+)
+```
+
+**Refine an existing dataset:**
+
+```python
+dataset = client.update_dataset(
+    system_id,
+    orientation="Add more edge cases about payment failures"
+)
+```
+
+> **Note:** `generate_dataset` requires a validated system description. If validation fails, a `DescriptionNotValidatedError` is raised with recommendations.
+
+### Run Evaluation
+
+Start an evaluation to test your AI system. By default, the call blocks until the evaluation completes.
+
+**Block until complete (default):**
+
+```python
+result = client.evaluate(system_id)
+print(f"Score: {result['summary']}")
+```
+
+**Start without waiting:**
+
+```python
+run_info = client.evaluate(system_id, wait=False)
+run_id = run_info["run_id"]
+
+# Check status later
+result = client.get_evaluation(run_id)
+print(f"Status: {result['status']}")
+```
+
+**With specific thematics:**
+
+```python
+result = client.evaluate(
+    system_id,
+    thematics_config={
+        "explainability": {"justification": {"nb_tests": 5}},
+        "robustness": {"prompt_injection": {"nb_tests": 10}}
+    }
+)
+```
+
+**With evaluation profile:**
+
+```python
+result = client.evaluate(system_id, profile="extended")
+```
+
+**With progress callback:**
+
+```python
+result = client.evaluate(
+    system_id,
+    poll_interval=10,
+    on_poll=lambda status, elapsed: print(f"  {status} ({elapsed}s)")
+)
+```
+
+### Connectors
+
+**Connectors** attach external data sources (logs, databases) to your system for richer evaluation context.
+
+**File logs:**
+
+```python
+from mankinds_sdk.connectors import FileConnector
+
+connector = FileConnector(file_path="/path/to/logs.json")
+client.add_connector(system_id, connector)
+```
+
+**Datadog logs:**
+
+```python
+from mankinds_sdk.connectors import DatadogConnector
+
+connector = DatadogConnector(
+    api_key="dd-api-key",
+    app_key="dd-app-key",
+    site="datadoghq.eu",  # default
+)
+client.add_connector(system_id, connector)
+```
+
+**SQLite database:**
+
+```python
+from mankinds_sdk.connectors import SqliteConnector
+
+connector = SqliteConnector(file_path="/path/to/database.db")
+client.add_connector(system_id, connector)
+```
+
+**PostgreSQL database:**
+
+```python
+from mankinds_sdk.connectors import PostgresqlConnector
+
+connector = PostgresqlConnector(
+    host="localhost",
+    database="mydb",
+    user="admin",
+    password="secret",
+    port=5432,
+)
+client.add_connector(system_id, connector)
+```
+
+**Manage connectors:**
+
+```python
+# List all connectors
+connectors = client.get_connectors(system_id)
+
+# Update a connector
+connector = FileConnector(file_path="/path/to/new-logs.json")
+client.update_connector(system_id, connector)
+
+# Remove a connector
+client.delete_connector(system_id, connector)
+```
+
+> Only one connector per category (logs, database) is allowed per system. Adding a duplicate raises `ConnectorAlreadyExistsError`.
+
+## Complete Example
+
+```python
+from mankinds_sdk import MankindsClient
+from mankinds_sdk.connectors import FileConnector
+
+client = MankindsClient(api_key="mk_...")
+
+# Create system
+system = client.create_system(
+    name="Support Bot",
+    description="A customer support chatbot for order tracking and returns.",
+    endpoint={
+        "url": "https://api.example.com/chat",
+        "method": "POST",
+        "body": {"message": "{{input}}"},
+        "response": {"answer": "{{output}}"}
+    }
+)
+system_id = system["id"]
+
+# Attach production logs
+connector = FileConnector(file_path="./logs/production.json")
+client.add_connector(system_id, connector)
+
+# Generate dataset and evaluate
+dataset = client.generate_dataset(system_id, num_scenarios=15)
+result = client.evaluate(system_id, profile="extended")
+
+print(f"Status: {result['status']}")
+print(f"Score: {result['summary']}")
+```
+
+## API Reference
+
+### MankindsClient
+
+| Method | Description |
+|--------|-------------|
+| `get_system(system_id)` | Get system details and configuration |
+| `create_system(name, description, endpoint)` | Create a new AI system |
+| `update_system(system_id, name?, description?, endpoint?)` | Update an existing system |
+| `generate_dataset(system_id, num_scenarios?, scenarios?)` | Generate and validate evaluation scenarios |
+| `update_dataset(system_id, orientation?, scenarios?)` | Refine or replace dataset scenarios |
+| `evaluate(system_id, ...)` | Run an evaluation |
+| `get_evaluation(run_id)` | Get evaluation status and results |
+| `add_connector(system_id, connector)` | Add a data source connector |
+| `get_connectors(system_id)` | List all connectors for a system |
+| `update_connector(system_id, connector)` | Update a connector |
+| `delete_connector(system_id, connector)` | Remove a connector |
+
+### Exceptions
+
+| Exception | When Raised |
+|-----------|-------------|
+| `CredentialsError` | Missing API key |
+| `AuthenticationError` | Invalid or expired API key (401) |
+| `NotFoundError` | Resource not found (404) |
+| `ValidationError` | Request validation failed (422) |
+| `RateLimitError` | Too many requests (429) |
+| `ServerError` | Server error (5xx) |
+| `InvalidEndpointError` | Endpoint missing required fields |
+| `EndpointNotConfiguredError` | Evaluation without endpoint |
+| `DescriptionNotValidatedError` | Dataset generation before validation |
+| `ConnectorAlreadyExistsError` | Duplicate connector category |
+
+```python
+from mankinds_sdk.exceptions import (
+    CredentialsError,
+    AuthenticationError,
+    InvalidEndpointError,
+    DescriptionNotValidatedError,
+    ConnectorAlreadyExistsError,
+)
+
+try:
+    result = client.evaluate(system_id)
+except AuthenticationError:
+    print("Invalid API key")
+except InvalidEndpointError as e:
+    print(f"Missing fields: {e.missing_fields}")
+except DescriptionNotValidatedError as e:
+    print(f"Fix description: {e.recommendations}")
+```
+
+---
+
+<div align="center">
+
+## License
+
+[MIT](../../LICENSE)
+
+© 2026 [Mankinds](https://mankinds.io). All rights reserved.
+
+</div>