prodloop-observability-sdk 0.1.8__tar.gz → 0.1.9__tar.gz

{prodloop_observability_sdk-0.1.8/prodloop_observability_sdk.egg-info → prodloop_observability_sdk-0.1.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: prodloop-observability-sdk
-Version: 0.1.8
+Version: 0.1.9
 Summary: Python SDK for evaluating AI voice bot calls via Prodloop APIs.
 Project-URL: Homepage, https://prodloop.com
 Project-URL: Documentation, https://observability-sdk-docs.pages.dev/
@@ -27,7 +27,7 @@ pip install prodloop-observability-sdk
 ## Quickstart
 
 ```python
-from prodloop import ProdloopClient, EvaluationParameter
+from prodloop import CustomEvaluationParameter, EvaluationParameter, ProdloopClient
 
 client = ProdloopClient(api_key="sk_live_...")
 
@@ -38,12 +38,40 @@ result = client.evaluate_call(
         EvaluationParameter.HALLUCINATION,
     ],
     thresholds={"e2e_response_time_max_ms": 800},
+    custom_parameters=[
+        CustomEvaluationParameter(
+            key="resolution_quality",
+            label="Resolution quality",
+            description="Check whether the bot correctly understood the issue and reached a useful final outcome.",
+        ),
+    ],
     input_prompt="Bot instructions used during this call...",
 )
 
 print(result)
 ```
 
+## Custom Parameters
+
+Use `custom_parameters` for audit dimensions that are not part of the fixed `EvaluationParameter` enum. Each custom parameter needs a stable `key` and clear `description`; `label` is optional.
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    custom_parameters=[
+        {
+            "key": "driver_resolution_quality",
+            "label": "Driver resolution quality",
+            "description": "Evaluate whether the bot handled driver-not-found or cancellation cases correctly and empathetically.",
+        }
+    ],
+    input_prompt="Use the Namma Yatri cancellation support policy as context.",
+)
+```
+
+Custom checks are sent as `custom_parameters` metadata and evaluated from their descriptions plus optional `input_prompt` context.
+
 ## Extraction Validation
 
 To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:
@@ -89,6 +117,27 @@ Prompt-aware parameter results use a compact shape:
 
 `passed` can be `"true"`, `"false"`, or `"N/A"`. `"N/A"` means the parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.
 
+Example prompt-aware response:
+
+```json
+{
+  "section_sequencing": {
+    "passed": "false",
+    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
+  },
+  "mandatory_field_gating": {
+    "passed": "N/A",
+    "explanation": "The prompt-defined gated action was not triggered in this call."
+  },
+  "prompt_injection": {
+    "passed": "N/A",
+    "explanation": "The caller did not attempt to override instructions or inject commands."
+  }
+}
+```
+
+Runnable example: `examples/post_call_prompt_aware_demo.py`. The same flow was production-tested with a partial parameter set and then with all prompt-aware parameters.
+
 ## Supported Parameters
 
 - `e2e_response_time`
@@ -169,6 +218,8 @@ There are two modes:
 - `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
 - `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.
 
+The production backend also supports `audit_discovery` for deeper prompt-risk discovery. Runnable examples are available in `examples/audit_discovery_demo.py` and `simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py`.
+
 Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.
 
 Discover currently enabled simulation parameters at runtime:
@@ -259,6 +310,10 @@ For `user_orchestrated`, configure bot credentials locally for either Vertex AI
 
 For adaptive simulations, `max_turns` controls turns per conversation and `adaptive_max_conversations` controls the maximum number of conversations to explore.
 
+### Audit Discovery
+
+Audit discovery plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios plus patch guidance for failures. A production smoke test for `section_sequencing` completed successfully with `status="completed"`, `final_result.overall_pass=true`, and `final_result.stop_reason="audit_discovery_completed"`.
+
 ### Result Shape
 
 Simulation responses include:

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/README.md

@@ -11,7 +11,7 @@ pip install prodloop-observability-sdk
 ## Quickstart
 
 ```python
-from prodloop import ProdloopClient, EvaluationParameter
+from prodloop import CustomEvaluationParameter, EvaluationParameter, ProdloopClient
 
 client = ProdloopClient(api_key="sk_live_...")
 
@@ -22,12 +22,40 @@ result = client.evaluate_call(
         EvaluationParameter.HALLUCINATION,
     ],
     thresholds={"e2e_response_time_max_ms": 800},
+    custom_parameters=[
+        CustomEvaluationParameter(
+            key="resolution_quality",
+            label="Resolution quality",
+            description="Check whether the bot correctly understood the issue and reached a useful final outcome.",
+        ),
+    ],
     input_prompt="Bot instructions used during this call...",
 )
 
 print(result)
 ```
 
+## Custom Parameters
+
+Use `custom_parameters` for audit dimensions that are not part of the fixed `EvaluationParameter` enum. Each custom parameter needs a stable `key` and clear `description`; `label` is optional.
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    custom_parameters=[
+        {
+            "key": "driver_resolution_quality",
+            "label": "Driver resolution quality",
+            "description": "Evaluate whether the bot handled driver-not-found or cancellation cases correctly and empathetically.",
+        }
+    ],
+    input_prompt="Use the Namma Yatri cancellation support policy as context.",
+)
+```
+
+Custom checks are sent as `custom_parameters` metadata and evaluated from their descriptions plus optional `input_prompt` context.
+
 ## Extraction Validation
 
 To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:
@@ -73,6 +101,27 @@ Prompt-aware parameter results use a compact shape:
 
 `passed` can be `"true"`, `"false"`, or `"N/A"`. `"N/A"` means the parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.
 
+Example prompt-aware response:
+
+```json
+{
+  "section_sequencing": {
+    "passed": "false",
+    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
+  },
+  "mandatory_field_gating": {
+    "passed": "N/A",
+    "explanation": "The prompt-defined gated action was not triggered in this call."
+  },
+  "prompt_injection": {
+    "passed": "N/A",
+    "explanation": "The caller did not attempt to override instructions or inject commands."
+  }
+}
+```
+
+Runnable example: `examples/post_call_prompt_aware_demo.py`. The same flow was production-tested with a partial parameter set and then with all prompt-aware parameters.
+
 ## Supported Parameters
 
 - `e2e_response_time`
@@ -153,6 +202,8 @@ There are two modes:
 - `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
 - `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.
 
+The production backend also supports `audit_discovery` for deeper prompt-risk discovery. Runnable examples are available in `examples/audit_discovery_demo.py` and `simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py`.
+
 Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.
 
 Discover currently enabled simulation parameters at runtime:
@@ -243,6 +294,10 @@ For `user_orchestrated`, configure bot credentials locally for either Vertex AI
 
 For adaptive simulations, `max_turns` controls turns per conversation and `adaptive_max_conversations` controls the maximum number of conversations to explore.
 
+### Audit Discovery
+
+Audit discovery plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios plus patch guidance for failures. A production smoke test for `section_sequencing` completed successfully with `status="completed"`, `final_result.overall_pass=true`, and `final_result.stop_reason="audit_discovery_completed"`.
+
 ### Result Shape
 
 Simulation responses include:

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/docs/examples.md

@@ -69,6 +69,25 @@ print(result["section_sequencing"])
 
 For prompt-aware parameters, `passed` is `"true"`, `"false"`, or `"N/A"`. The model returns `"N/A"` when the parameter is not relevant to the supplied prompt or the call does not exercise enough behavior to judge it.
 
+This flow was tested against production with both a small subset and all prompt-aware parameters. Example response for a call that did not match the supplied bot prompt:
+
+```json
+{
+  "section_sequencing": {
+    "passed": "false",
+    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
+  },
+  "mandatory_field_gating": {
+    "passed": "N/A",
+    "explanation": "The prompt-defined gated action was not triggered in this call."
+  },
+  "prompt_injection": {
+    "passed": "N/A",
+    "explanation": "The caller did not attempt to override instructions or inject commands."
+  }
+}
+```
+
 
 ## Self Simulation
 
@@ -101,6 +120,34 @@ while True:
     time.sleep(2)
 ```
 
+## Audit Discovery
+
+Audit discovery is a production backend mode for deeper prompt-risk discovery. It plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios with patch guidance for failures.
+
+The after-PyPI production demo lives at `simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py`. A production smoke test for `section_sequencing` completed with:
+
+```json
+{
+  "status": "completed",
+  "final_result": {
+    "overall_pass": true,
+    "stop_reason": "audit_discovery_completed",
+    "stop_message": "Audit discovery completed across planned risk scenarios.",
+    "audit_discovery": {
+      "enabled": true,
+      "passed_scenarios": [
+        {
+          "risk_id": "fatal_emergency_interruption",
+          "planned_risk_passed": true
+        }
+      ],
+      "failed_scenarios": [],
+      "error_scenarios": []
+    }
+  }
+}
+```
+
 ## User Orchestrated Simulation
 
 ```python
@@ -166,6 +213,18 @@ The repository includes copy-pasteable examples in `examples/`. These are embedd
 --8<-- "examples/demo_gpt.py"
 ```
 
+### Prompt-Aware Post-Call Evaluation
+
+```python
+--8<-- "examples/post_call_prompt_aware_demo.py"
+```
+
+### Audit Discovery
+
+```python
+--8<-- "examples/audit_discovery_demo.py"
+```
+
 ### Vertex AI User Orchestrated Simulation
 
 ```python

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/examples/.env.example

@@ -6,6 +6,18 @@
 # Bot provider credentials for self_simulation are configured on the Prodloop backend.
 PRODLOOP_API_KEY=
 
+# Post-call prompt-aware demo:
+# Used by post_call_prompt_aware_demo.py.
+POST_CALL_AUDIO_FILE=sample_call.mp3
+POST_CALL_PROMPT_FILE=sample_prompt.txt
+
+# Audit discovery demo:
+# Used by audit_discovery_demo.py.
+AUDIT_DISCOVERY_PARAMETER=section_sequencing
+AUDIT_DISCOVERY_BOT_MODEL=azure/<deployment-name>
+AUDIT_DISCOVERY_MAX_SCENARIOS=1
+AUDIT_DISCOVERY_MAX_TURNS=6
+
 # Azure OpenAI: fill this section when running GPT/Azure demos:
 # - demo_gpt.py
 # - user_orchestrated_demo_gpt.py

prodloop_observability_sdk-0.1.9/examples/audit_discovery_demo.py

@@ -0,0 +1,96 @@
+"""Audit discovery using the installed Prodloop SDK.
+
+Set POST_CALL_PROMPT_FILE, AUDIT_DISCOVERY_PARAMETER, and bot model settings
+in examples/.env first. Audit discovery runs one selected prompt-risk parameter
+through planned backend scenarios and returns passed/failed scenario evidence.
+"""
+import json
+import os
+import time
+from pathlib import Path
+from typing import Any
+
+from prodloop import ProdloopClient, plugins
+
+
+ENV_PATH = Path(__file__).with_name(".env")
+
+
+def _load_env() -> None:
+    try:
+        from dotenv import load_dotenv
+    except Exception:
+        return
+    load_dotenv(dotenv_path=ENV_PATH, override=False)
+
+
+def _required_path(env_name: str) -> Path:
+    raw_value = os.getenv(env_name, "").strip()
+    if not raw_value:
+        raise RuntimeError(f"Set {env_name} in {ENV_PATH}.")
+    path = Path(raw_value)
+    if not path.exists():
+        raise RuntimeError(f"{env_name} path not found: {path}")
+    return path
+
+
+def _int_env(name: str, default: int) -> int:
+    raw_value = os.getenv(name, "").strip()
+    return int(raw_value) if raw_value else default
+
+
+def _safe_bot_payload(client: ProdloopClient, model: str, max_tokens: int) -> dict[str, Any]:
+    bot_llm = plugins.LiteLLM(
+        model=model,
+        temperature=0.2,
+        max_tokens=max_tokens,
+    )
+    return client._self_simulation_bot_payload(bot_llm.to_payload())  # noqa: SLF001
+
+
+def main() -> None:
+    _load_env()
+    api_key = os.getenv("PRODLOOP_API_KEY", "").strip()
+    if not api_key:
+        raise RuntimeError(f"Set PRODLOOP_API_KEY in {ENV_PATH}.")
+
+    client = ProdloopClient(api_key=api_key, timeout_seconds=1800)
+    prompt_path = _required_path("POST_CALL_PROMPT_FILE")
+    parameter = os.getenv("AUDIT_DISCOVERY_PARAMETER", "section_sequencing").strip()
+    bot_model = os.getenv("AUDIT_DISCOVERY_BOT_MODEL", "azure/<deployment-name>").strip()
+    max_scenarios = _int_env("AUDIT_DISCOVERY_MAX_SCENARIOS", 1)
+    max_turns = _int_env("AUDIT_DISCOVERY_MAX_TURNS", 6)
+
+    start = client._post_json(  # noqa: SLF001
+        "/simulate/start",
+        {
+            "simulation_mode": "audit_discovery",
+            "prompt": prompt_path.read_text(encoding="utf-8"),
+            "parameters": [parameter],
+            "bot_llm": _safe_bot_payload(client, bot_model, max_tokens=700),
+            "max_turns": max_turns,
+            "adaptive_max_conversations": max_scenarios,
+            "scenario": "Audit discovery smoke test for the selected parameter.",
+        },
+        None,
+    )
+    chat_id = start["chat_id"]
+    print("chat_id:", chat_id, flush=True)
+
+    while True:
+        result = client.get_simulation(chat_id)
+        print(
+            "status=",
+            result.get("status"),
+            "turns=",
+            len(result.get("turns") or []),
+            flush=True,
+        )
+        if result.get("status") in {"completed", "failed"}:
+            print(json.dumps(result, indent=2, ensure_ascii=False), flush=True)
+            return
+        time.sleep(5)
+
+
+if __name__ == "__main__":
+    main()

prodloop_observability_sdk-0.1.9/examples/post_call_prompt_aware_demo.py

@@ -0,0 +1,97 @@
+"""Post-call prompt-aware evaluation using the installed Prodloop SDK.
+
+Run:
+    python post_call_prompt_aware_demo.py partial
+    python post_call_prompt_aware_demo.py all
+
+Set POST_CALL_AUDIO_FILE and POST_CALL_PROMPT_FILE in examples/.env first.
+Prompt-aware parameters return passed="true", "false", or "N/A".
+"""
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Sequence
+
+from prodloop import EvaluationParameter, ProdloopClient
+
+
+ENV_PATH = Path(__file__).with_name(".env")
+
+PROMPT_AWARE_PARAMETERS: tuple[EvaluationParameter, ...] = (
+    EvaluationParameter.SECTION_SEQUENCING,
+    EvaluationParameter.MANDATORY_FIELD_GATING,
+    EvaluationParameter.INTERRUPT_RESUME_PRECISION,
+    EvaluationParameter.CLOSING_VERBATIM_DELIVERY,
+    EvaluationParameter.SINGLE_ATTEMPT_CONSTRAINTS,
+    EvaluationParameter.INFO_DUMP_HANDLING,
+    EvaluationParameter.MID_FLOW_INTENT_SWITCH,
+    EvaluationParameter.SIDE_TALK_LEAKAGE,
+    EvaluationParameter.AMBIGUOUS_PARTIAL_RESPONSES,
+    EvaluationParameter.INTERNAL_JARGON_LEAKAGE,
+    EvaluationParameter.IDENTITY_EXTRACTION,
+    EvaluationParameter.PROMPT_INJECTION,
+    EvaluationParameter.COMMITMENT_EXTRACTION,
+    EvaluationParameter.SCOPE_BOUNDARY_TESTING,
+    EvaluationParameter.ROLEPLAY_JAILBREAK,
+    EvaluationParameter.CONTEXT_MEMORY_ACROSS_TURNS,
+    EvaluationParameter.HALLUCINATION_FABRICATION,
+)
+
+PARTIAL_PARAMETERS: tuple[EvaluationParameter, ...] = (
+    EvaluationParameter.SECTION_SEQUENCING,
+    EvaluationParameter.MANDATORY_FIELD_GATING,
+    EvaluationParameter.PROMPT_INJECTION,
+)
+
+
+def _load_env() -> None:
+    try:
+        from dotenv import load_dotenv
+    except Exception:
+        return
+    load_dotenv(dotenv_path=ENV_PATH, override=False)
+
+
+def _parameters_for_mode(mode: str) -> Sequence[EvaluationParameter]:
+    if mode == "partial":
+        return PARTIAL_PARAMETERS
+    if mode == "all":
+        return PROMPT_AWARE_PARAMETERS
+    raise RuntimeError("Usage: python post_call_prompt_aware_demo.py [partial|all]")
+
+
+def _required_path(env_name: str) -> Path:
+    raw_value = os.getenv(env_name, "").strip()
+    if not raw_value:
+        raise RuntimeError(f"Set {env_name} in {ENV_PATH}.")
+    path = Path(raw_value)
+    if not path.exists():
+        raise RuntimeError(f"{env_name} path not found: {path}")
+    return path
+
+
+def main() -> None:
+    _load_env()
+    api_key = os.getenv("PRODLOOP_API_KEY", "").strip()
+    if not api_key:
+        raise RuntimeError(f"Set PRODLOOP_API_KEY in {ENV_PATH}.")
+
+    mode = sys.argv[1].strip().lower() if len(sys.argv) > 1 else "partial"
+    parameters = _parameters_for_mode(mode)
+    audio_path = _required_path("POST_CALL_AUDIO_FILE")
+    prompt_path = _required_path("POST_CALL_PROMPT_FILE")
+
+    client = ProdloopClient(api_key=api_key, timeout_seconds=1800)
+    result = client.evaluate_call(
+        audio_file_path=str(audio_path),
+        input_prompt=prompt_path.read_text(encoding="utf-8"),
+        parameters=parameters,
+        timeout_seconds=1800,
+    )
+
+    print(json.dumps(result, indent=2, ensure_ascii=False), flush=True)
+
+
+if __name__ == "__main__":
+    main()

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/prodloop/__init__.py

@@ -3,6 +3,7 @@ from prodloop.client import ProdloopClient
 from prodloop.exceptions import APIError, ProdloopError, ValidationError
 from prodloop.models import (
     BotLLMConnection,
+    CustomEvaluationParameter,
     EvaluationParameter,
     SimulationMode,
 )
@@ -11,6 +12,7 @@ from prodloop.plugins import LiteLLM, LiteLLMBot
 __all__ = [
     "ProdloopClient",
     "BotLLMConnection",
+    "CustomEvaluationParameter",
     "LiteLLM",
     "LiteLLMBot",
     "EvaluationParameter",

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/prodloop/client.py

@@ -2,23 +2,37 @@ import json
 import mimetypes
 import time
 from pathlib import Path
-from typing import Any, Callable, Dict, Iterable, Mapping, Optional, Sequence, Tuple, Union
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    Iterable,
+    Mapping,
+    Optional,
+    Sequence,
+    Tuple,
+    Union,
+)
 
 import requests
 
 from prodloop.exceptions import APIError, ValidationError
 from prodloop.models import (
     BotLLMInput,
+    CustomParameterInput,
     EvaluationParameter,
     ParameterInput,
     SimulationMode,
     coerce_bot_llm,
+    coerce_custom_parameter,
     coerce_parameter,
 )
 
 
 class ProdloopClient:
-    _SERVICE_URL = "https://asia-south1-prodloop.cloudfunctions.net/prodloop-evaluator-fn"
+    _SERVICE_URL = (
+        "https://asia-south1-prodloop.cloudfunctions.net/prodloop-evaluator-fn"
+    )
 
     def __init__(
         self,
@@ -38,16 +52,25 @@ class ProdloopClient:
         audio_file_path: str,
         parameters: Sequence[ParameterInput],
         thresholds: Optional[Mapping[str, Any]] = None,
+        custom_parameters: Optional[Sequence[CustomParameterInput]] = None,
         extraction_schema: Optional[Mapping[str, str]] = None,
         bot_captured_variables: Optional[Mapping[str, Any]] = None,
         input_prompt: Optional[str] = None,
         timeout_seconds: Optional[int] = None,
     ) -> Dict[str, Any]:
         normalized_parameters = self._normalize_parameters(parameters)
+        normalized_custom_parameters = self._normalize_custom_parameters(
+            custom_parameters or []
+        )
+        if not normalized_parameters and not normalized_custom_parameters:
+            raise ValidationError("At least one evaluation parameter is required.")
+
         normalized_input_prompt = (input_prompt or "").strip()
         payload: Dict[str, Any] = {
             "parameters": [param.value for param in normalized_parameters],
         }
+        if normalized_custom_parameters:
+            payload["custom_parameters"] = normalized_custom_parameters
 
         if thresholds:
             payload["thresholds"] = dict(thresholds)
@@ -68,7 +91,9 @@ class ProdloopClient:
                     "bot_captured_variables is required when using 'extraction_variables'."
                 )
             missing_keys = [
-                key for key in extraction_schema.keys() if key not in bot_captured_variables
+                key
+                for key in extraction_schema.keys()
+                if key not in bot_captured_variables
             ]
             if missing_keys:
                 raise ValidationError(
@@ -189,7 +214,9 @@ class ProdloopClient:
 
             handler_start = time.perf_counter()
             handler_output = bot_turn_handler(tester_message, transcript)
-            measured_handler_latency_ms = round((time.perf_counter() - handler_start) * 1000.0, 2)
+            measured_handler_latency_ms = round(
+                (time.perf_counter() - handler_start) * 1000.0, 2
+            )
             bot_response, bot_response_latency_ms = self._normalize_bot_turn_output(
                 handler_output=handler_output,
                 measured_handler_latency_ms=measured_handler_latency_ms,
@@ -284,7 +311,12 @@ class ProdloopClient:
             "token",
         ):
             livekit_kwargs.pop(secret_key, None)
-        for backend_owned_key in ("project", "location", "vertex_project", "vertex_location"):
+        for backend_owned_key in (
+            "project",
+            "location",
+            "vertex_project",
+            "vertex_location",
+        ):
             endpoint.pop(backend_owned_key, None)
             livekit_kwargs.pop(backend_owned_key, None)
 
@@ -298,14 +330,29 @@ class ProdloopClient:
         }
 
     @staticmethod
-    def _normalize_parameters(parameters: Iterable[ParameterInput]) -> Sequence[EvaluationParameter]:
+    def _normalize_parameters(
+        parameters: Iterable[ParameterInput],
+    ) -> Sequence[EvaluationParameter]:
         normalized = [coerce_parameter(param) for param in parameters]
-        if not normalized:
-            raise ValidationError("At least one evaluation parameter is required.")
         return normalized
 
     @staticmethod
-    def _normalize_simulation_parameters(parameters: Iterable[ParameterInput]) -> Sequence[str]:
+    def _normalize_custom_parameters(
+        parameters: Iterable[CustomParameterInput],
+    ) -> Sequence[Dict[str, str]]:
+        normalized = [coerce_custom_parameter(param) for param in parameters]
+        seen: set[str] = set()
+        for parameter in normalized:
+            key = parameter["key"]
+            if key in seen:
+                raise ValidationError(f"Duplicate custom parameter key: {key}")
+            seen.add(key)
+        return normalized
+
+    @staticmethod
+    def _normalize_simulation_parameters(
+        parameters: Iterable[ParameterInput],
+    ) -> Sequence[str]:
         normalized: list[str] = []
         for parameter in parameters:
             if isinstance(parameter, EvaluationParameter):
@@ -342,9 +389,13 @@ class ProdloopClient:
         try:
             payload = response.json()
         except ValueError as exc:
-            raise APIError(response.status_code, "Backend returned non-JSON response.") from exc
+            raise APIError(
+                response.status_code, "Backend returned non-JSON response."
+            ) from exc
         if not isinstance(payload, dict):
-            raise APIError(response.status_code, "Backend response must be a JSON object.")
+            raise APIError(
+                response.status_code, "Backend response must be a JSON object."
+            )
         return payload
 
     @staticmethod
@@ -368,14 +419,18 @@ class ProdloopClient:
         if isinstance(handler_output, str):
             response_text = handler_output.strip()
             if not response_text:
-                raise ValidationError("bot_turn_handler must return a non-empty string.")
+                raise ValidationError(
+                    "bot_turn_handler must return a non-empty string."
+                )
             return response_text, measured_handler_latency_ms
 
         if isinstance(handler_output, tuple) and len(handler_output) == 2:
             response_raw, latency_raw = handler_output
             response_text = str(response_raw or "").strip()
             if not response_text:
-                raise ValidationError("bot_turn_handler tuple must include a non-empty response.")
+                raise ValidationError(
+                    "bot_turn_handler tuple must include a non-empty response."
+                )
             try:
                 latency_ms = float(latency_raw)
             except (TypeError, ValueError) as exc:
@@ -383,7 +438,9 @@ class ProdloopClient:
                     "bot_turn_handler tuple latency must be numeric."
                 ) from exc
             if latency_ms < 0:
-                raise ValidationError("bot_turn_handler tuple latency cannot be negative.")
+                raise ValidationError(
+                    "bot_turn_handler tuple latency cannot be negative."
+                )
             return response_text, round(latency_ms, 2)
 
         if isinstance(handler_output, Mapping):
@@ -406,11 +463,12 @@ class ProdloopClient:
                     "bot_turn_handler mapping latency must be numeric."
                 ) from exc
             if latency_ms < 0:
-                raise ValidationError("bot_turn_handler mapping latency cannot be negative.")
+                raise ValidationError(
+                    "bot_turn_handler mapping latency cannot be negative."
+                )
             return response_text, round(latency_ms, 2)
 
         raise ValidationError(
             "bot_turn_handler must return either a string, "
             "(response, llm_latency_ms), or a mapping with response and latency."
         )
-

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/prodloop/models.py

@@ -30,7 +30,27 @@ class EvaluationParameter(str, Enum):
     HALLUCINATION_FABRICATION = "hallucination_fabrication"
 
 
+@dataclass(frozen=True)
+class CustomEvaluationParameter:
+    key: str
+    description: str
+    label: Optional[str] = None
+
+    def to_payload(self) -> Dict[str, str]:
+        key = self.key.strip().lower().replace(" ", "_")
+        description = self.description.strip()
+        if not key:
+            raise ValueError("custom parameter key is required.")
+        if not description:
+            raise ValueError("custom parameter description is required.")
+        payload = {"key": key, "description": description}
+        if self.label and self.label.strip():
+            payload["label"] = self.label.strip()
+        return payload
+
+
 ParameterInput = Union[EvaluationParameter, str]
+CustomParameterInput = Union[CustomEvaluationParameter, Mapping[str, Any]]
 
 SUPPORTED_PARAMETERS = tuple(param.value for param in EvaluationParameter)
 
@@ -86,6 +106,18 @@ def coerce_parameter(value: ParameterInput) -> EvaluationParameter:
     )
 
 
+def coerce_custom_parameter(value: CustomParameterInput) -> Dict[str, str]:
+    if isinstance(value, CustomEvaluationParameter):
+        return value.to_payload()
+    if not isinstance(value, Mapping):
+        raise ValueError("custom parameter must be a CustomEvaluationParameter or mapping.")
+    return CustomEvaluationParameter(
+        key=str(value.get("key", "")),
+        label=str(value.get("label", "")) if value.get("label") is not None else None,
+        description=str(value.get("description", "")),
+    ).to_payload()
+
+
 def coerce_bot_llm(value: Optional[BotLLMInput]) -> Optional[Dict[str, Any]]:
     if value is None:
         return None

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9/prodloop_observability_sdk.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: prodloop-observability-sdk
-Version: 0.1.8
+Version: 0.1.9
 Summary: Python SDK for evaluating AI voice bot calls via Prodloop APIs.
 Project-URL: Homepage, https://prodloop.com
 Project-URL: Documentation, https://observability-sdk-docs.pages.dev/
@@ -27,7 +27,7 @@ pip install prodloop-observability-sdk
 ## Quickstart
 
 ```python
-from prodloop import ProdloopClient, EvaluationParameter
+from prodloop import CustomEvaluationParameter, EvaluationParameter, ProdloopClient
 
 client = ProdloopClient(api_key="sk_live_...")
 
@@ -38,12 +38,40 @@ result = client.evaluate_call(
         EvaluationParameter.HALLUCINATION,
     ],
     thresholds={"e2e_response_time_max_ms": 800},
+    custom_parameters=[
+        CustomEvaluationParameter(
+            key="resolution_quality",
+            label="Resolution quality",
+            description="Check whether the bot correctly understood the issue and reached a useful final outcome.",
+        ),
+    ],
     input_prompt="Bot instructions used during this call...",
 )
 
 print(result)
 ```
 
+## Custom Parameters
+
+Use `custom_parameters` for audit dimensions that are not part of the fixed `EvaluationParameter` enum. Each custom parameter needs a stable `key` and clear `description`; `label` is optional.
+
+```python
+result = client.evaluate_call(
+    audio_file_path="call.mp3",
+    parameters=[EvaluationParameter.HALLUCINATION],
+    custom_parameters=[
+        {
+            "key": "driver_resolution_quality",
+            "label": "Driver resolution quality",
+            "description": "Evaluate whether the bot handled driver-not-found or cancellation cases correctly and empathetically.",
+        }
+    ],
+    input_prompt="Use the Namma Yatri cancellation support policy as context.",
+)
+```
+
+Custom checks are sent as `custom_parameters` metadata and evaluated from their descriptions plus optional `input_prompt` context.
+
 ## Extraction Validation
 
 To validate extraction quality, pass both `extraction_schema` and `bot_captured_variables`:
@@ -89,6 +117,27 @@ Prompt-aware parameter results use a compact shape:
 
 `passed` can be `"true"`, `"false"`, or `"N/A"`. `"N/A"` means the parameter was not relevant to the supplied prompt or the call did not exercise enough behavior to judge it.
 
+Example prompt-aware response:
+
+```json
+{
+  "section_sequencing": {
+    "passed": "false",
+    "explanation": "The bot did not follow the section flow defined in the supplied prompt."
+  },
+  "mandatory_field_gating": {
+    "passed": "N/A",
+    "explanation": "The prompt-defined gated action was not triggered in this call."
+  },
+  "prompt_injection": {
+    "passed": "N/A",
+    "explanation": "The caller did not attempt to override instructions or inject commands."
+  }
+}
+```
+
+Runnable example: `examples/post_call_prompt_aware_demo.py`. The same flow was production-tested with a partial parameter set and then with all prompt-aware parameters.
+
 ## Supported Parameters
 
 - `e2e_response_time`
@@ -169,6 +218,8 @@ There are two modes:
 - `self_simulation`: Prodloop backend runs the tester and bot conversation. You select the bot model route, but Prodloop-owned backend credentials are used. No bot credentials are sent from your code.
 - `user_orchestrated`: Prodloop backend runs the tester and grader. Your SDK process runs the bot locally with your own credentials and sends only bot replies/latency back to Prodloop.
 
+The production backend also supports `audit_discovery` for deeper prompt-risk discovery. Runnable examples are available in `examples/audit_discovery_demo.py` and `simulation_demo/prod_testing/after_pypi/audit_discovery_demo.py`.
+
 Simulation currently accepts exactly one parameter per request. To test multiple parameters, start one simulation per parameter. `max_turns` is configurable from `1` to `10`.
 
 Discover currently enabled simulation parameters at runtime:
@@ -259,6 +310,10 @@ For `user_orchestrated`, configure bot credentials locally for either Vertex AI
 
 For adaptive simulations, `max_turns` controls turns per conversation and `adaptive_max_conversations` controls the maximum number of conversations to explore.
 
+### Audit Discovery
+
+Audit discovery plans targeted risk scenarios for one selected parameter, runs them against the bot, and returns passed/failed scenarios plus patch guidance for failures. A production smoke test for `section_sequencing` completed successfully with `status="completed"`, `final_result.overall_pass=true`, and `final_result.stop_reason="audit_discovery_completed"`.
+
 ### Result Shape
 
 Simulation responses include:

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/prodloop_observability_sdk.egg-info/SOURCES.txt

@@ -12,8 +12,10 @@ docs/index.md
 docs/parameters.md
 docs/visual-overview.md
 examples/.env.example
+examples/audit_discovery_demo.py
 examples/demo.py
 examples/demo_gpt.py
+examples/post_call_prompt_aware_demo.py
 examples/user_orchestrated_demo.py
 examples/user_orchestrated_demo_gpt.py
 prodloop/__init__.py

{prodloop_observability_sdk-0.1.8 → prodloop_observability_sdk-0.1.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "prodloop-observability-sdk"
-version = "0.1.8"
+version = "0.1.9"
 description = "Python SDK for evaluating AI voice bot calls via Prodloop APIs."
 readme = "README.md"
 requires-python = ">=3.9"