prodloop-observability-sdk 0.1.2__tar.gz → 0.1.5__tar.gz

prodloop_observability_sdk-0.1.5/PKG-INFO

@@ -0,0 +1,234 @@
+Metadata-Version: 2.4
+Name: prodloop-observability-sdk
+Version: 0.1.5
+Summary: Python SDK for evaluating AI voice bot calls via Prodloop APIs.
+Project-URL: Homepage, https://prodloop.com
+Project-URL: Documentation, https://observability-sdk-docs.pages.dev/
+Project-URL: Repository, https://github.com/prodloop/prodloop-observability-sdk
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: litellm
+Requires-Dist: requests>=2.31.0
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.6.0; extra == "docs"
+Requires-Dist: mkdocs-material>=9.5.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.25.0; extra == "docs"
+
+# Prodloop Observability SDK
+
+Python SDK to evaluate AI voice bot calls through the Prodloop evaluation service.
+
+## Install
+
+```bash
+pip install prodloop-observability-sdk
+```
+
+## Quickstart
+
+```python
+from prodloop import ProdloopClient, EvaluationParameter
+
+client = ProdloopClient(api_key="sk_live_...")
+
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[
+        EvaluationParameter.E2E_RESPONSE_TIME,
+        EvaluationParameter.HALLUCINATION,
+    ],
+    thresholds={"e2e_response_time_max_ms": 800},
+    input_prompt="Bot instructions used during this call...",
+)
+
+print(result)
+```
+
+## Extraction Validation
+
+To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
+    extraction_schema={"customer_name": "string"},
+    bot_captured_variables={"customer_name": "ram"},
+)
+```
+
+Response includes:
+
+- `extraction_variables`
+- `extraction_validation`
+
+## Hallucination Input Requirement
+
+When requesting `hallucination`, pass the bot's original call prompt as `input_prompt`:
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    input_prompt="You are a polite admissions bot. Never invent course details.",
+)
+```
+
+## Supported Parameters
+
+- `e2e_response_time`
+- `turn_by_turn_latency`
+- `pause_profile`
+- `audio_artifacts`
+- `hallucination`
+- `extraction_variables`
+- `interruption_behavior`
+
+## Parameter Purpose
+
+- `e2e_response_time`: average response latency in milliseconds.
+- `turn_by_turn_latency`: per-turn latency series with `turn_index` and `latency_ms`.
+- `pause_profile`: deterministic pause summary (`pause_count`, `total_pause_time_ms`, `longest_pause_ms`).
+- `audio_artifacts`: deterministic audio-signal checks (`clipping_ratio`, `dc_offset`, clipping/DC flags).
+- `hallucination`: whether the bot produced fabricated or incorrect claims.
+- `extraction_variables`: structured variable extraction from call audio.
+- `interruption_behavior`: whether the bot handled interruptions gracefully.
+
+Deterministic parameters are computed directly from the audio signal:
+`e2e_response_time`, `turn_by_turn_latency`, `pause_profile`, `audio_artifacts`.
+
+## Authentication
+
+Pass your Prodloop API key in the SDK constructor.  
+The SDK sends it as a `Bearer` token in the `Authorization` header.
+
+## Errors
+
+- `ValidationError`: invalid local inputs (file path, parameters, etc.)
+- `APIError`: backend/API-level failures (`status_code`, `message`)
+
+## Documentation Site
+
+Live documentation: https://observability-sdk-docs.pages.dev/
+
+## Prompt Simulation
+
+The SDK can test a bot prompt without an audio file by simulating a text conversation between a tester LLM and your bot.
+
+There are two modes:
+
+- `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
+- `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.
+
+Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.
+
+Discover currently enabled simulation parameters at runtime:
+
+```python
+params = client.get_simulation_parameters()
+print(params["parameters"])  # [{"key": "hallucination"}, ...]
+```
+
+The response intentionally exposes only public parameter keys. Whether a parameter is LLM-judged or deterministic is backend implementation detail.
+
+For prompt simulation, `turn_by_turn_latency` is the only timing parameter exposed. It is also emitted in every simulation result automatically at no extra LLM cost, even when you select another parameter such as `hallucination`. Older audio-evaluation parameters like `e2e_response_time` and `pause_profile` remain relevant for call-audio evaluation, but they are not separate prompt-simulation parameters.
+
+### Self Simulation
+
+Use the generic LiteLLM connector. The currently supported backend model routes are `vertex_ai/gemini-2.5-pro` for Gemini on Vertex AI and `azure/<deployment-name>` for Azure OpenAI deployments.
+
+```python
+import os
+import time
+from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins
+
+client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])
+
+start = client.simulate_prompt(
+    simulation_mode=SimulationMode.SELF_SIMULATION,
+    prompt="You are a concise support bot. Do not invent policy details.",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    bot_llm=plugins.LiteLLM(
+        model="vertex_ai/gemini-2.5-pro",
+        temperature=0.2,
+        max_tokens=512,
+    ),
+    max_turns=5,
+    scenario="A customer reports a delayed order and asks about compensation.",
+)
+
+chat_id = start["chat_id"]
+print("chat_id:", chat_id)
+
+while True:
+    result = client.get_simulation(chat_id)
+    print(result["status"])
+    if result["status"] in {"completed", "failed"}:
+        print(result)
+        break
+    time.sleep(2)
+```
+
+Backend provider configuration is required for `self_simulation`. Examples:
+
+- Vertex AI: `VERTEX_PROJECT`, `VERTEX_LOCATION`, and the deployed service account permissions.
+- Azure OpenAI: `AZURE_API_BASE`, `AZURE_API_VERSION`, and `AZURE_API_KEY` or `AZURE_AD_TOKEN`. You can also derive the base URL from a resource name in your own deployment setup.
+
+If the backend is not configured for the selected route, the API returns a safe user-facing error asking you to configure provider credentials or use `user_orchestrated` mode.
+
+### User Orchestrated Simulation
+
+Use `user_orchestrated` when the bot must run in your own process with your own credentials. Prodloop never receives your bot provider credentials.
+
+```python
+import os
+from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins
+
+bot = plugins.LiteLLMBot(
+    model="azure/<deployment-name>",
+    system_prompt="You are a concise support bot. Do not invent policy details.",
+    options={"temperature": 0.2, "max_tokens": 256},
+)
+
+client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])
+
+result = client.simulate_prompt(
+    simulation_mode=SimulationMode.USER_ORCHESTRATED,
+    prompt="You are a concise support bot. Do not invent policy details.",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    max_turns=5,
+    scenario="A customer pressures the bot to confirm a fake policy.",
+    bot_turn_handler=bot,
+)
+
+print(result)
+```
+
+For `user_orchestrated`, configure bot credentials locally for either Vertex AI or Azure OpenAI. For Vertex AI, use ADC/service-account configuration and project/location. For Azure OpenAI, use your Azure endpoint, API version, and API key/token locally. The SDK sends only bot response text and bot LLM processing latency to Prodloop.
+
+### Result Shape
+
+Simulation responses include:
+
+- `chat_id`: stable simulation identifier.
+- `status`: `running`, `completed`, or `failed`.
+- `turns`: tester message, bot response, `turn_id`, grading result, and per-turn timing.
+- `final_result`: overall pass/fail, per-parameter result, summary, and prompt patch suggestions when a test fails.
+
+If a parameter fails, `final_result.parameter_results[*].prompt_patch_lines` and `prompt_patch_location` provide copy-paste prompt guidance.
+
+
+
+## Credit Balance
+
+Check the credits left on the API key without running an evaluation:
+
+```python
+balance = client.get_credit_balance()
+print(balance["credits_remaining"])
+
+# Alias with the same response
+balance = client.get_credits_remaining()
+```
+
+This endpoint authenticates the API key but does not debit credits.

prodloop_observability_sdk-0.1.5/README.md

@@ -0,0 +1,218 @@
+# Prodloop Observability SDK
+
+Python SDK to evaluate AI voice bot calls through the Prodloop evaluation service.
+
+## Install
+
+```bash
+pip install prodloop-observability-sdk
+```
+
+## Quickstart
+
+```python
+from prodloop import ProdloopClient, EvaluationParameter
+
+client = ProdloopClient(api_key="sk_live_...")
+
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[
+        EvaluationParameter.E2E_RESPONSE_TIME,
+        EvaluationParameter.HALLUCINATION,
+    ],
+    thresholds={"e2e_response_time_max_ms": 800},
+    input_prompt="Bot instructions used during this call...",
+)
+
+print(result)
+```
+
+## Extraction Validation
+
+To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.EXTRACTION_VARIABLES],
+    extraction_schema={"customer_name": "string"},
+    bot_captured_variables={"customer_name": "ram"},
+)
+```
+
+Response includes:
+
+- `extraction_variables`
+- `extraction_validation`
+
+## Hallucination Input Requirement
+
+When requesting `hallucination`, pass the bot's original call prompt as `input_prompt`:
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    input_prompt="You are a polite admissions bot. Never invent course details.",
+)
+```
+
+## Supported Parameters
+
+- `e2e_response_time`
+- `turn_by_turn_latency`
+- `pause_profile`
+- `audio_artifacts`
+- `hallucination`
+- `extraction_variables`
+- `interruption_behavior`
+
+## Parameter Purpose
+
+- `e2e_response_time`: average response latency in milliseconds.
+- `turn_by_turn_latency`: per-turn latency series with `turn_index` and `latency_ms`.
+- `pause_profile`: deterministic pause summary (`pause_count`, `total_pause_time_ms`, `longest_pause_ms`).
+- `audio_artifacts`: deterministic audio-signal checks (`clipping_ratio`, `dc_offset`, clipping/DC flags).
+- `hallucination`: whether the bot produced fabricated or incorrect claims.
+- `extraction_variables`: structured variable extraction from call audio.
+- `interruption_behavior`: whether the bot handled interruptions gracefully.
+
+Deterministic parameters are computed directly from the audio signal:
+`e2e_response_time`, `turn_by_turn_latency`, `pause_profile`, `audio_artifacts`.
+
+## Authentication
+
+Pass your Prodloop API key in the SDK constructor.  
+The SDK sends it as a `Bearer` token in the `Authorization` header.
+
+## Errors
+
+- `ValidationError`: invalid local inputs (file path, parameters, etc.)
+- `APIError`: backend/API-level failures (`status_code`, `message`)
+
+## Documentation Site
+
+Live documentation: https://observability-sdk-docs.pages.dev/
+
+## Prompt Simulation
+
+The SDK can test a bot prompt without an audio file by simulating a text conversation between a tester LLM and your bot.
+
+There are two modes:
+
+- `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
+- `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.
+
+Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.
+
+Discover currently enabled simulation parameters at runtime:
+
+```python
+params = client.get_simulation_parameters()
+print(params["parameters"])  # [{"key": "hallucination"}, ...]
+```
+
+The response intentionally exposes only public parameter keys. Whether a parameter is LLM-judged or deterministic is backend implementation detail.
+
+For prompt simulation, `turn_by_turn_latency` is the only timing parameter exposed. It is also emitted in every simulation result automatically at no extra LLM cost, even when you select another parameter such as `hallucination`. Older audio-evaluation parameters like `e2e_response_time` and `pause_profile` remain relevant for call-audio evaluation, but they are not separate prompt-simulation parameters.
+
+### Self Simulation
+
+Use the generic LiteLLM connector. The currently supported backend model routes are `vertex_ai/gemini-2.5-pro` for Gemini on Vertex AI and `azure/<deployment-name>` for Azure OpenAI deployments.
+
+```python
+import os
+import time
+from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins
+
+client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])
+
+start = client.simulate_prompt(
+    simulation_mode=SimulationMode.SELF_SIMULATION,
+    prompt="You are a concise support bot. Do not invent policy details.",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    bot_llm=plugins.LiteLLM(
+        model="vertex_ai/gemini-2.5-pro",
+        temperature=0.2,
+        max_tokens=512,
+    ),
+    max_turns=5,
+    scenario="A customer reports a delayed order and asks about compensation.",
+)
+
+chat_id = start["chat_id"]
+print("chat_id:", chat_id)
+
+while True:
+    result = client.get_simulation(chat_id)
+    print(result["status"])
+    if result["status"] in {"completed", "failed"}:
+        print(result)
+        break
+    time.sleep(2)
+```
+
+Backend provider configuration is required for `self_simulation`. Examples:
+
+- Vertex AI: `VERTEX_PROJECT`, `VERTEX_LOCATION`, and the deployed service account permissions.
+- Azure OpenAI: `AZURE_API_BASE`, `AZURE_API_VERSION`, and `AZURE_API_KEY` or `AZURE_AD_TOKEN`. You can also derive the base URL from a resource name in your own deployment setup.
+
+If the backend is not configured for the selected route, the API returns a safe user-facing error asking you to configure provider credentials or use `user_orchestrated` mode.
+
+### User Orchestrated Simulation
+
+Use `user_orchestrated` when the bot must run in your own process with your own credentials. Prodloop never receives your bot provider credentials.
+
+```python
+import os
+from prodloop import EvaluationParameter, ProdloopClient, SimulationMode, plugins
+
+bot = plugins.LiteLLMBot(
+    model="azure/<deployment-name>",
+    system_prompt="You are a concise support bot. Do not invent policy details.",
+    options={"temperature": 0.2, "max_tokens": 256},
+)
+
+client = ProdloopClient(api_key=os.environ["PRODLOOP_API_KEY"])
+
+result = client.simulate_prompt(
+    simulation_mode=SimulationMode.USER_ORCHESTRATED,
+    prompt="You are a concise support bot. Do not invent policy details.",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    max_turns=5,
+    scenario="A customer pressures the bot to confirm a fake policy.",
+    bot_turn_handler=bot,
+)
+
+print(result)
+```
+
+For `user_orchestrated`, configure bot credentials locally for either Vertex AI or Azure OpenAI. For Vertex AI, use ADC/service-account configuration and project/location. For Azure OpenAI, use your Azure endpoint, API version, and API key/token locally. The SDK sends only bot response text and bot LLM processing latency to Prodloop.
+
+### Result Shape
+
+Simulation responses include:
+
+- `chat_id`: stable simulation identifier.
+- `status`: `running`, `completed`, or `failed`.
+- `turns`: tester message, bot response, `turn_id`, grading result, and per-turn timing.
+- `final_result`: overall pass/fail, per-parameter result, summary, and prompt patch suggestions when a test fails.
+
+If a parameter fails, `final_result.parameter_results[*].prompt_patch_lines` and `prompt_patch_location` provide copy-paste prompt guidance.
+
+
+
+## Credit Balance
+
+Check the credits left on the API key without running an evaluation:
+
+```python
+balance = client.get_credit_balance()
+print(balance["credits_remaining"])
+
+# Alias with the same response
+balance = client.get_credits_remaining()
+```
+
+This endpoint authenticates the API key but does not debit credits.

prodloop_observability_sdk-0.1.5/prodloop/__init__.py

@@ -0,0 +1,22 @@
+import prodloop.plugins as plugins
+from prodloop.client import ProdloopClient
+from prodloop.exceptions import APIError, ProdloopError, ValidationError
+from prodloop.models import (
+    BotLLMConnection,
+    EvaluationParameter,
+    SimulationMode,
+)
+from prodloop.plugins import LiteLLM, LiteLLMBot
+
+__all__ = [
+    "ProdloopClient",
+    "BotLLMConnection",
+    "LiteLLM",
+    "LiteLLMBot",
+    "EvaluationParameter",
+    "SimulationMode",
+    "plugins",
+    "ProdloopError",
+    "ValidationError",
+    "APIError",
+]