applied-cli 0.6.5__tar.gz → 0.6.7__tar.gz

{applied_cli-0.6.5 → applied_cli-0.6.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: applied-cli
-Version: 0.6.5
+Version: 0.6.7
 Summary: CLI and shared client library for Applied Labs AI support agents
 Author: Applied Labs
 License-Expression: MIT
@@ -80,6 +80,57 @@ applied metrics --metric-name conversation.resolve --start 2026-04-01 --end 2026
 object. `analytics` returns grouped rows and currently supports `--metrics count`.
 Raw analytics SQL is not available through the public CLI surface.
 
+## Benchmarks & Scenarios
+
+A **benchmark** is a named regression suite; a **scenario** is one test conversation
+(built from a real `input_conversation_id`) that can belong to one or more benchmarks.
+The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
+
+```bash
+# Inspect benchmarks and their scenarios
+applied benchmarks --agent-id <agent_id> --format json
+applied benchmark <benchmark_id> --format json
+applied scenarios --benchmark-id <benchmark_id> --format json
+
+# Build a suite
+applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
+applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
+  --benchmark-id <benchmark_id>
+
+# Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
+# scenarios under the destination agent; same-agent just tags them in.
+# Dry-run by default; add --apply to write.
+applied benchmark-clone <source_benchmark_id> --dest-benchmark-name "Chat Regression" \
+  --target-agent-id <chat_agent_id> --apply
+
+# Run a benchmark and wait for results in one command.
+# --contact-email runs as a contact that has an email, fixing
+# "Email is not present in the conversation" on test conversations.
+applied scenario-bulk-run --benchmark-id <benchmark_id> \
+  --contact-email test@example.com --wait
+applied scenario-bulk-status <job_id> --include-runs --format json
+
+# Kill a stuck bulk run (deletes its queued/running runs; finished runs preserved)
+applied scenario-bulk-cancel <job_id> --apply
+
+# Review pass/fail health (pass_status reflects the latest run per scenario)
+applied benchmark-results <benchmark_id> --format json
+
+# Rate scenarios as you evaluate
+applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
+
+# Safe delete — refuses to wipe scenarios unless you opt in
+applied benchmark-delete <benchmark_id> --detach-scenarios   # preserve scenarios
+applied benchmark-delete <benchmark_id> --force              # cascade delete
+
+# Recover deleted benchmark/scenario rows from a local PITR export
+applied scenario-recover-catalog --recovery-dir <dir> --apply
+```
+
+Deleting a benchmark cascades and permanently deletes its scenarios and runs, so
+`benchmark-delete` refuses a non-empty benchmark unless you pass `--detach-scenarios`
+(unlink the scenarios first so they survive under their agent) or `--force`.
+
 ## Library Usage
 
 ```python
@@ -113,6 +164,11 @@ conversations = await tools.conversation_query(
 | `analytics_report` | Read standard dashboard/report analytics views |
 | `analytics_query` | Aggregate supported conversation dimensions with count |
 | `metrics_query` | Roll up named metric events |
+| `benchmark_clone` | Copy all scenarios from one benchmark into another |
+| `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
+| `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
+| `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
 
 ## Examples
 

{applied_cli-0.6.5 → applied_cli-0.6.7}/README.md

@@ -54,6 +54,57 @@ applied metrics --metric-name conversation.resolve --start 2026-04-01 --end 2026
 object. `analytics` returns grouped rows and currently supports `--metrics count`.
 Raw analytics SQL is not available through the public CLI surface.
 
+## Benchmarks & Scenarios
+
+A **benchmark** is a named regression suite; a **scenario** is one test conversation
+(built from a real `input_conversation_id`) that can belong to one or more benchmarks.
+The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
+
+```bash
+# Inspect benchmarks and their scenarios
+applied benchmarks --agent-id <agent_id> --format json
+applied benchmark <benchmark_id> --format json
+applied scenarios --benchmark-id <benchmark_id> --format json
+
+# Build a suite
+applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
+applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
+  --benchmark-id <benchmark_id>
+
+# Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
+# scenarios under the destination agent; same-agent just tags them in.
+# Dry-run by default; add --apply to write.
+applied benchmark-clone <source_benchmark_id> --dest-benchmark-name "Chat Regression" \
+  --target-agent-id <chat_agent_id> --apply
+
+# Run a benchmark and wait for results in one command.
+# --contact-email runs as a contact that has an email, fixing
+# "Email is not present in the conversation" on test conversations.
+applied scenario-bulk-run --benchmark-id <benchmark_id> \
+  --contact-email test@example.com --wait
+applied scenario-bulk-status <job_id> --include-runs --format json
+
+# Kill a stuck bulk run (deletes its queued/running runs; finished runs preserved)
+applied scenario-bulk-cancel <job_id> --apply
+
+# Review pass/fail health (pass_status reflects the latest run per scenario)
+applied benchmark-results <benchmark_id> --format json
+
+# Rate scenarios as you evaluate
+applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
+
+# Safe delete — refuses to wipe scenarios unless you opt in
+applied benchmark-delete <benchmark_id> --detach-scenarios   # preserve scenarios
+applied benchmark-delete <benchmark_id> --force              # cascade delete
+
+# Recover deleted benchmark/scenario rows from a local PITR export
+applied scenario-recover-catalog --recovery-dir <dir> --apply
+```
+
+Deleting a benchmark cascades and permanently deletes its scenarios and runs, so
+`benchmark-delete` refuses a non-empty benchmark unless you pass `--detach-scenarios`
+(unlink the scenarios first so they survive under their agent) or `--force`.
+
 ## Library Usage
 
 ```python
@@ -87,6 +138,11 @@ conversations = await tools.conversation_query(
 | `analytics_report` | Read standard dashboard/report analytics views |
 | `analytics_query` | Aggregate supported conversation dimensions with count |
 | `metrics_query` | Roll up named metric events |
+| `benchmark_clone` | Copy all scenarios from one benchmark into another |
+| `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
+| `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
+| `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
 
 ## Examples
 

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli/cli.py

@@ -1672,6 +1672,22 @@ def benchmark_delete(
     typer.echo(result)
 
 
+@app.command("benchmark-results")
+def benchmark_results(
+    id: str = typer.Argument(..., help="Benchmark ID"),
+    shop_id: str = typer.Option(None, "--shop-id", help="Override shop ID"),
+    format: str = typer.Option(
+        "text", "--format", "-f", help="Output format: text or json"
+    ),
+) -> None:
+    """Summarize a benchmark's pass/fail/unrated health and pass rate."""
+    client = get_client(shop_id=shop_id)
+    result = asyncio.run(
+        tools.benchmark_results(client, benchmark_id=id, output_format=format)
+    )
+    typer.echo(result)
+
+
 @app.command()
 def scenarios(
     benchmark_id: str = typer.Option(

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli/tools.py

@@ -5710,6 +5710,93 @@ async def benchmark_clone(
     return "\n".join(lines)
 
 
+async def benchmark_results(
+    client: AppliedClient,
+    benchmark_id: str,
+    *,
+    output_format: str = "text",
+) -> str:
+    """
+    Summarize a benchmark's pass/fail health.
+
+    Tallies the pass_status across the benchmark's scenarios (pass / fail /
+    unrated), computes the pass rate among rated scenarios, and lists the failing
+    and still-unrated scenarios so you know what to fix or evaluate next.
+
+    Args:
+        client: Authenticated AppliedClient
+        benchmark_id: The benchmark UUID
+        output_format: 'text' (default) or 'json'
+
+    Returns:
+        Pass-rate summary with failing and unrated scenario lists.
+    """
+    try:
+        benchmark = await client.get_benchmark(benchmark_id)
+        scenarios = await client.list_scenarios(
+            benchmark_id=benchmark_id, fetch_all=True
+        )
+    except AppliedAPIError as e:
+        return _format_error(e)
+
+    tally = {"pass": 0, "fail": 0, "unrated": 0}
+    failing: list[dict[str, Any]] = []
+    unrated: list[dict[str, Any]] = []
+    for scenario in scenarios:
+        status = str(scenario.get("pass_status") or "unrated").lower()
+        if status not in tally:
+            status = "unrated"
+        tally[status] += 1
+        entry = {"id": scenario.get("id"), "name": scenario.get("name")}
+        if status == "fail":
+            failing.append(entry)
+        elif status == "unrated":
+            unrated.append(entry)
+
+    rated = tally["pass"] + tally["fail"]
+    pass_rate = round(tally["pass"] / rated, 4) if rated else None
+    summary = {
+        "benchmark_id": benchmark_id,
+        "benchmark_name": benchmark.get("name"),
+        "total_scenarios": len(scenarios),
+        "passed": tally["pass"],
+        "failed": tally["fail"],
+        "unrated": tally["unrated"],
+        "rated": rated,
+        "pass_rate": pass_rate,
+        "failing_scenarios": failing,
+        "unrated_scenarios": unrated,
+    }
+
+    if output_format == "json":
+        return to_json(summary)
+
+    pass_rate_str = (
+        f"{pass_rate * 100:.1f}% ({tally['pass']}/{rated} rated)"
+        if pass_rate is not None
+        else "n/a (no rated scenarios yet)"
+    )
+    lines = [
+        f"# Benchmark Results: {benchmark.get('name')} ({benchmark_id})",
+        f"total_scenarios: {summary['total_scenarios']}",
+        f"passed: {tally['pass']}",
+        f"failed: {tally['fail']}",
+        f"unrated: {tally['unrated']}",
+        f"pass_rate: {pass_rate_str}",
+    ]
+    if failing:
+        lines.append(f"\n# Failing ({len(failing)})")
+        lines.extend(f"  - {s['name']} ({s['id']})" for s in failing[:50])
+        if len(failing) > 50:
+            lines.append(f"  ... and {len(failing) - 50} more")
+    if unrated:
+        lines.append(f"\n# Unrated ({len(unrated)}) — evaluate these next")
+        lines.extend(f"  - {s['name']} ({s['id']})" for s in unrated[:50])
+        if len(unrated) > 50:
+            lines.append(f"  ... and {len(unrated) - 50} more")
+    return "\n".join(lines)
+
+
 # -----------------------------------------------------------------------------
 # Scenarios
 # -----------------------------------------------------------------------------

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli/v2/domains.py

@@ -43,6 +43,7 @@ DOMAIN_TOOL_RENAMES: dict[str, dict[str, str]] = {
         "benchmark_create": "benchmarks_create",
         "benchmark_delete": "benchmarks_delete",
         "benchmark_clone": "benchmarks_clone",
+        "benchmark_results": "benchmarks_results",
     },
     "connectors": {
         "connector_types": "connectors_types_list",

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli/v2/scenarios.py

@@ -71,6 +71,10 @@ class BenchmarksCloneInput(StrictInput):
     apply: bool = False
 
 
+class BenchmarksResultsInput(StrictInput):
+    benchmark_id: str
+
+
 class ScenariosListInput(StrictInput):
     benchmark_id: str | None = None
     agent_id: str | None = None
@@ -503,6 +507,44 @@ async def benchmarks_clone_handler(
     )
 
 
+async def benchmarks_results_handler(
+    client: AppliedClient,
+    params: BenchmarksResultsInput,
+) -> ToolResult[Any]:
+    from applied_cli import tools as legacy_tools
+
+    raw = await legacy_tools.benchmark_results(
+        client, benchmark_id=params.benchmark_id, output_format="json"
+    )
+    try:
+        data = json.loads(raw)
+    except (json.JSONDecodeError, TypeError):
+        return ToolResult(data={"message": raw}, summary=str(raw))
+
+    pass_rate = data.get("pass_rate")
+    rate_str = (
+        f"{pass_rate * 100:.1f}%" if pass_rate is not None else "n/a (no rated yet)"
+    )
+    next_actions = []
+    if data.get("unrated"):
+        next_actions.append(
+            "Rate the unrated scenarios with scenarios_update (pass_status)."
+        )
+    if data.get("failed"):
+        next_actions.append(
+            "Inspect failing scenarios with scenarios_get / conversations_debug_bundle."
+        )
+    return ToolResult(
+        data=data,
+        summary=(
+            f"{data.get('benchmark_name') or params.benchmark_id}: pass rate "
+            f"{rate_str} — {data.get('passed', 0)} passed, "
+            f"{data.get('failed', 0)} failed, {data.get('unrated', 0)} unrated."
+        ),
+        next_actions=next_actions,
+    )
+
+
 async def benchmarks_delete_handler(
     client: AppliedClient,
     params: BenchmarksDeleteInput,
@@ -1006,6 +1048,19 @@ def scenario_specs() -> list[ToolSpec]:
             read_write_mode="write",
             tags=["benchmark_clone", "native"],
         ),
+        ToolSpec(
+            name="benchmarks_results",
+            namespace="benchmarks",
+            description=(
+                "Summarize a benchmark's pass/fail/unrated health and pass rate, "
+                "with the failing and unrated scenario lists."
+            ),
+            input_model=BenchmarksResultsInput,
+            output_model=None,
+            handler=benchmarks_results_handler,
+            read_write_mode="read",
+            tags=["benchmark_results", "native"],
+        ),
         ToolSpec(
             name="scenarios_list",
             namespace="scenarios",

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: applied-cli
-Version: 0.6.5
+Version: 0.6.7
 Summary: CLI and shared client library for Applied Labs AI support agents
 Author: Applied Labs
 License-Expression: MIT
@@ -80,6 +80,57 @@ applied metrics --metric-name conversation.resolve --start 2026-04-01 --end 2026
 object. `analytics` returns grouped rows and currently supports `--metrics count`.
 Raw analytics SQL is not available through the public CLI surface.
 
+## Benchmarks & Scenarios
+
+A **benchmark** is a named regression suite; a **scenario** is one test conversation
+(built from a real `input_conversation_id`) that can belong to one or more benchmarks.
+The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
+
+```bash
+# Inspect benchmarks and their scenarios
+applied benchmarks --agent-id <agent_id> --format json
+applied benchmark <benchmark_id> --format json
+applied scenarios --benchmark-id <benchmark_id> --format json
+
+# Build a suite
+applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
+applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
+  --benchmark-id <benchmark_id>
+
+# Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
+# scenarios under the destination agent; same-agent just tags them in.
+# Dry-run by default; add --apply to write.
+applied benchmark-clone <source_benchmark_id> --dest-benchmark-name "Chat Regression" \
+  --target-agent-id <chat_agent_id> --apply
+
+# Run a benchmark and wait for results in one command.
+# --contact-email runs as a contact that has an email, fixing
+# "Email is not present in the conversation" on test conversations.
+applied scenario-bulk-run --benchmark-id <benchmark_id> \
+  --contact-email test@example.com --wait
+applied scenario-bulk-status <job_id> --include-runs --format json
+
+# Kill a stuck bulk run (deletes its queued/running runs; finished runs preserved)
+applied scenario-bulk-cancel <job_id> --apply
+
+# Review pass/fail health (pass_status reflects the latest run per scenario)
+applied benchmark-results <benchmark_id> --format json
+
+# Rate scenarios as you evaluate
+applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
+
+# Safe delete — refuses to wipe scenarios unless you opt in
+applied benchmark-delete <benchmark_id> --detach-scenarios   # preserve scenarios
+applied benchmark-delete <benchmark_id> --force              # cascade delete
+
+# Recover deleted benchmark/scenario rows from a local PITR export
+applied scenario-recover-catalog --recovery-dir <dir> --apply
+```
+
+Deleting a benchmark cascades and permanently deletes its scenarios and runs, so
+`benchmark-delete` refuses a non-empty benchmark unless you pass `--detach-scenarios`
+(unlink the scenarios first so they survive under their agent) or `--force`.
+
 ## Library Usage
 
 ```python
@@ -113,6 +164,11 @@ conversations = await tools.conversation_query(
 | `analytics_report` | Read standard dashboard/report analytics views |
 | `analytics_query` | Aggregate supported conversation dimensions with count |
 | `metrics_query` | Roll up named metric events |
+| `benchmark_clone` | Copy all scenarios from one benchmark into another |
+| `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
+| `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
+| `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
 
 ## Examples
 

{applied_cli-0.6.5 → applied_cli-0.6.7}/applied_cli.egg-info/SOURCES.txt

@@ -40,6 +40,7 @@ tests/test_audit_tools.py
 tests/test_auth_context.py
 tests/test_benchmark_clone.py
 tests/test_benchmark_delete_guardrail.py
+tests/test_benchmark_results.py
 tests/test_benchmark_scenario_tools.py
 tests/test_cli.py
 tests/test_cli_v2.py

{applied_cli-0.6.5 → applied_cli-0.6.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "applied-cli"
-version = "0.6.5"
+version = "0.6.7"
 description = "CLI and shared client library for Applied Labs AI support agents"
 readme = "README.md"
 requires-python = ">=3.11"

applied_cli-0.6.7/tests/test_benchmark_results.py

@@ -0,0 +1,78 @@
+import json
+
+import pytest
+
+from applied_cli import tools
+
+BENCHMARK = {"id": "bench-1", "name": "Cancel Regression"}
+
+SCENARIOS = [
+    {"id": "s1", "name": "Cancel order", "pass_status": "pass"},
+    {"id": "s2", "name": "Refund flow", "pass_status": "pass"},
+    {"id": "s3", "name": "Pause subscription", "pass_status": "fail"},
+    {"id": "s4", "name": "Address change", "pass_status": "unrated"},
+    {"id": "s5", "name": "No status field"},  # missing -> unrated
+]
+
+
+class FakeResultsClient:
+    def __init__(self, scenarios=SCENARIOS):
+        self._scenarios = scenarios
+
+    async def get_benchmark(self, benchmark_id):
+        return BENCHMARK
+
+    async def list_scenarios(self, benchmark_id=None, fetch_all=True, **kwargs):
+        return list(self._scenarios)
+
+
+@pytest.mark.asyncio
+async def test_results_tally_and_pass_rate():
+    client = FakeResultsClient()
+    data = json.loads(
+        await tools.benchmark_results(client, "bench-1", output_format="json")
+    )
+    assert data["total_scenarios"] == 5
+    assert data["passed"] == 2
+    assert data["failed"] == 1
+    assert data["unrated"] == 2
+    assert data["rated"] == 3
+    # 2 passed / 3 rated
+    assert data["pass_rate"] == round(2 / 3, 4)
+    assert [s["id"] for s in data["failing_scenarios"]] == ["s3"]
+    assert {s["id"] for s in data["unrated_scenarios"]} == {"s4", "s5"}
+
+
+@pytest.mark.asyncio
+async def test_results_no_rated_scenarios_pass_rate_none():
+    client = FakeResultsClient(
+        scenarios=[{"id": "s1", "name": "A", "pass_status": "unrated"}]
+    )
+    text = await tools.benchmark_results(client, "bench-1", output_format="text")
+    assert "n/a (no rated scenarios yet)" in text
+
+
+@pytest.mark.asyncio
+async def test_results_text_lists_failing_and_unrated():
+    client = FakeResultsClient()
+    text = await tools.benchmark_results(client, "bench-1")
+    assert "# Failing (1)" in text
+    assert "Pause subscription" in text
+    assert "# Unrated (2)" in text
+
+
+@pytest.mark.asyncio
+async def test_v2_benchmarks_results_handler_summary():
+    from applied_cli.v2.scenarios import (
+        BenchmarksResultsInput,
+        benchmarks_results_handler,
+    )
+
+    client = FakeResultsClient()
+    result = await benchmarks_results_handler(
+        client, BenchmarksResultsInput(benchmark_id="bench-1")
+    )
+    assert result.data["passed"] == 2
+    assert "pass rate" in result.summary
+    # Has unrated + failing → both follow-up actions surfaced.
+    assert len(result.next_actions) == 2