applied-cli 0.6.9__tar.gz → 0.6.11__tar.gz

{applied_cli-0.6.9 → applied_cli-0.6.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: applied-cli
-Version: 0.6.9
+Version: 0.6.11
 Summary: CLI and shared client library for Applied Labs AI support agents
 Author: Applied Labs
 License-Expression: MIT
@@ -82,6 +82,8 @@ Raw analytics SQL is not available through the public CLI surface.
 
 ## Benchmarks & Scenarios
 
+> **LEGACY surface.** Benchmarks and scenarios are the older regression-suite system. The current evaluation system is the **/evals** surface (feedback + simulations + test data), driven by the `simulation_test_*`, `test_customer_*`, and `simulation_run_*` tools (MCP / `/v1/simulation-tests/`, `/v1/test-customers/`, `/v1/simulation-runs/`). The `feedback` field on `scenario_update` / `scenario_run_update` is a legacy free-text note, NOT the new Judgment+Comment feedback surface. Use the /evals tools for new evaluation work; the tools below are for maintaining existing benchmark suites.
+
 A **benchmark** is a named regression suite; a **scenario** is one test conversation
 (built from a real `input_conversation_id`) that can belong to one or more benchmarks.
 The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
@@ -97,6 +99,11 @@ applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
 applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
   --benchmark-id <benchmark_id>
 
+# Build a suite fast from several real conversations at once. Each scenario is
+# named from its conversation's title (or "<prefix> N" with --name-prefix).
+applied scenario-create-bulk --conversation-ids <id1>,<id2>,<id3> \
+  --benchmark-id <benchmark_id>
+
 # Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
 # scenarios under the destination agent; same-agent just tags them in.
 # Dry-run by default; add --apply to write.
@@ -116,6 +123,9 @@ applied scenario-bulk-cancel <job_id> --apply
 # Review pass/fail health (pass_status reflects the latest run per scenario)
 applied benchmark-results <benchmark_id> --format json
 
+# Portfolio go/no-go: pass rates across all of an agent's benchmarks at a glance
+applied benchmarks --agent-id <agent_id> --with-results --format json
+
 # Rate scenarios as you evaluate
 applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
 
@@ -167,8 +177,22 @@ conversations = await tools.conversation_query(
 | `benchmark_clone` | Copy all scenarios from one benchmark into another |
 | `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
 | `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `benchmark_list` | List benchmarks (with per-benchmark pass rates via `with_results`) |
+| `scenario_create_bulk` | Build scenarios from several conversations at once |
 | `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
 | `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
+| `review_run` | Grade real conversations with the AI Reviewer (pass/fail) |
+| `review_status` | Verdict pairs + the Reviewer's reason for a set of conversations |
+| `review_set_verdict` | Record the human verdict with a teaching reason |
+| `review_teach` | Teach the Reviewer a rule directly |
+| `review_make_test` | Turn a flagged conversation into a proposed regression test |
+| `review_proposals_list` | The suite change-inbox (proposed adds/edits/retires/merges) |
+| `review_proposals_decide` | Accept or dismiss a proposed test |
+
+The full `review_*` namespace also includes `review_get`, `review_clear_verdict`,
+`review_add_note`, `review_trust`, and `review_lessons` — the outer review loop:
+grade real traffic, confirm/overrule with a reason that teaches the Reviewer, and
+promote failures into the simulation suite as tests.
 
 ## Examples
 

{applied_cli-0.6.9 → applied_cli-0.6.11}/README.md

@@ -56,6 +56,8 @@ Raw analytics SQL is not available through the public CLI surface.
 
 ## Benchmarks & Scenarios
 
+> **LEGACY surface.** Benchmarks and scenarios are the older regression-suite system. The current evaluation system is the **/evals** surface (feedback + simulations + test data), driven by the `simulation_test_*`, `test_customer_*`, and `simulation_run_*` tools (MCP / `/v1/simulation-tests/`, `/v1/test-customers/`, `/v1/simulation-runs/`). The `feedback` field on `scenario_update` / `scenario_run_update` is a legacy free-text note, NOT the new Judgment+Comment feedback surface. Use the /evals tools for new evaluation work; the tools below are for maintaining existing benchmark suites.
+
 A **benchmark** is a named regression suite; a **scenario** is one test conversation
 (built from a real `input_conversation_id`) that can belong to one or more benchmarks.
 The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
@@ -71,6 +73,11 @@ applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
 applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
   --benchmark-id <benchmark_id>
 
+# Build a suite fast from several real conversations at once. Each scenario is
+# named from its conversation's title (or "<prefix> N" with --name-prefix).
+applied scenario-create-bulk --conversation-ids <id1>,<id2>,<id3> \
+  --benchmark-id <benchmark_id>
+
 # Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
 # scenarios under the destination agent; same-agent just tags them in.
 # Dry-run by default; add --apply to write.
@@ -90,6 +97,9 @@ applied scenario-bulk-cancel <job_id> --apply
 # Review pass/fail health (pass_status reflects the latest run per scenario)
 applied benchmark-results <benchmark_id> --format json
 
+# Portfolio go/no-go: pass rates across all of an agent's benchmarks at a glance
+applied benchmarks --agent-id <agent_id> --with-results --format json
+
 # Rate scenarios as you evaluate
 applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
 
@@ -141,8 +151,22 @@ conversations = await tools.conversation_query(
 | `benchmark_clone` | Copy all scenarios from one benchmark into another |
 | `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
 | `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `benchmark_list` | List benchmarks (with per-benchmark pass rates via `with_results`) |
+| `scenario_create_bulk` | Build scenarios from several conversations at once |
 | `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
 | `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
+| `review_run` | Grade real conversations with the AI Reviewer (pass/fail) |
+| `review_status` | Verdict pairs + the Reviewer's reason for a set of conversations |
+| `review_set_verdict` | Record the human verdict with a teaching reason |
+| `review_teach` | Teach the Reviewer a rule directly |
+| `review_make_test` | Turn a flagged conversation into a proposed regression test |
+| `review_proposals_list` | The suite change-inbox (proposed adds/edits/retires/merges) |
+| `review_proposals_decide` | Accept or dismiss a proposed test |
+
+The full `review_*` namespace also includes `review_get`, `review_clear_verdict`,
+`review_add_note`, `review_trust`, and `review_lessons` — the outer review loop:
+grade real traffic, confirm/overrule with a reason that teaches the Reviewer, and
+promote failures into the simulation suite as tests.
 
 ## Examples
 

{applied_cli-0.6.9 → applied_cli-0.6.11}/applied_cli/__init__.py

@@ -6,7 +6,7 @@ from applied_cli.client import AppliedClient
 from applied_cli.formatters import to_csv, to_json
 from applied_cli.toolkit import ToolResult, ToolSpec
 
-__version__ = "0.6.0"
+__version__ = "0.6.11"
 
 __all__ = [
     "AppliedClient",

{applied_cli-0.6.9 → applied_cli-0.6.11}/applied_cli/v2/catalog.py

@@ -18,6 +18,7 @@ from applied_cli.v2.connectors import connector_specs
 from applied_cli.v2.content import content_specs
 from applied_cli.v2.conversations import conversation_specs
 from applied_cli.v2.domains import LEGACY_TOOL_RENAMES
+from applied_cli.v2.evals import evals_specs
 from applied_cli.v2.flows import flow_specs
 from applied_cli.v2.knowledge import knowledge_specs
 from applied_cli.v2.products import product_specs
@@ -99,7 +100,9 @@ def _input_model_for(func: Any, v2_name: str) -> type[BaseModel]:
         }:
             continue
         annotation = _safe_annotation(parameter.annotation)
-        default = ... if parameter.default is inspect.Parameter.empty else parameter.default
+        default = (
+            ... if parameter.default is inspect.Parameter.empty else parameter.default
+        )
         fields[name] = (annotation, default)
 
     if not fields:
@@ -225,6 +228,7 @@ def get_tool_catalog(include_debug: bool = False) -> ToolCatalog:
         *connector_specs(),
         *content_specs(),
         *conversation_specs(),
+        *evals_specs(),
         *flow_specs(),
         *knowledge_specs(),
         *product_specs(),

applied_cli-0.6.11/applied_cli/v2/evals.py

@@ -0,0 +1,567 @@
+"""Native structured v2 tools for the evals / review loop.
+
+The review loop over real traffic: the AI Reviewer grades a conversation, a human
+confirms or overrules with a teaching reason, the reason folds back into the
+Reviewer's criteria, and a flagged conversation becomes a regression test in one
+step. These tools expose that loop to an agent — grade conversations, record
+verdicts, teach the Reviewer, and promote a bad conversation into the simulation
+suite as a proposed test (with an auto-generated test customer).
+
+All endpoints are shop-scoped and merchant-authenticated. Discover conversations
+to review with `conversations_search` / `conversations_find`, then feed their ids
+here.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from applied_cli.client import AppliedAPIError, AppliedClient
+from applied_cli.toolkit import ToolError, ToolResult, ToolSpec
+
+
+class StrictInput(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+
+# ── inputs ──────────────────────────────────────────────────────────────────
+
+
+class ReviewRunInput(StrictInput):
+    conversation_ids: list[str] = Field(
+        description="Conversation ids to grade (max 50 per call).",
+    )
+    force: bool = Field(
+        default=False,
+        description=(
+            "Re-grade even if an AI verdict already exists. Off by default so "
+            "already-reviewed conversations are skipped."
+        ),
+    )
+
+
+class ReviewStatusInput(StrictInput):
+    conversation_ids: list[str] = Field(
+        description="Conversation ids to fetch verdict pairs + narration for.",
+    )
+
+
+class ReviewGetInput(StrictInput):
+    conversation_id: str
+
+
+class ReviewSetVerdictInput(StrictInput):
+    conversation_id: str
+    verdict: Literal["pass", "fail"]
+    reason: str | None = Field(
+        default=None,
+        description=(
+            "Why — the teaching signal. The Reviewer folds recent reasons into "
+            "its criteria automatically, so a clear reason improves future grading."
+        ),
+    )
+    pinned: bool | None = Field(
+        default=None,
+        description="Pin this reason as a durable lesson that never rotates out.",
+    )
+
+
+class ReviewClearVerdictInput(StrictInput):
+    conversation_id: str
+
+
+class ReviewAddNoteInput(StrictInput):
+    conversation_id: str
+    kind: Literal["correction", "good"] = "correction"
+    observed: str | None = Field(
+        default=None, description="What the bot did wrong (corrections only)."
+    )
+    desired: str | None = Field(
+        default=None,
+        description="What it should have done, or why a good turn is worth reusing.",
+    )
+    message_id: str | None = Field(
+        default=None, description="The turn this note anchors to (optional)."
+    )
+
+
+class ReviewTeachInput(StrictInput):
+    text: str = Field(
+        description=(
+            "A rule to teach the Reviewer directly. Lands pinned and applies to "
+            "every review from the next one on."
+        ),
+    )
+
+
+class EmptyInput(StrictInput):
+    pass
+
+
+class ReviewMakeTestInput(StrictInput):
+    conversation_id: str = Field(
+        description="The conversation to turn into a proposed regression test.",
+    )
+
+
+class ProposalsListInput(StrictInput):
+    status: Literal["proposed", "accepted", "dismissed"] = "proposed"
+
+
+class ProposalsDecideInput(StrictInput):
+    proposal_id: str
+    decision: Literal["accepted", "dismissed"]
+    customer_state_ref: str | None = Field(
+        default=None,
+        description=(
+            "When accepting an add/merge that has no fixture yet, the test "
+            "customer (CustomerState) id to bind the test to."
+        ),
+    )
+    reason: str | None = Field(
+        default=None,
+        description="When dismissing, why — a negative training signal for miners.",
+    )
+
+
+# ── helpers ─────────────────────────────────────────────────────────────────
+
+
+def _api_error_result(exc: AppliedAPIError) -> ToolResult[Any]:
+    return ToolResult(
+        ok=False,
+        error=ToolError(
+            code=f"api.{exc.status_code}",
+            message=str(exc),
+            status_code=exc.status_code,
+            retryable=exc.status_code in {408, 429, 500, 502, 503, 504},
+            recovery_hint=exc.suggestion,
+        ),
+    )
+
+
+# ── handlers ────────────────────────────────────────────────────────────────
+
+
+async def review_run_handler(
+    client: AppliedClient, params: ReviewRunInput
+) -> ToolResult[Any]:
+    ids = params.conversation_ids[:50]
+    if not ids:
+        return ToolResult(
+            ok=False,
+            error=ToolError(
+                code="input.invalid",
+                message="Provide at least one conversation id.",
+                retryable=False,
+            ),
+        )
+    try:
+        data = await client.request(
+            "POST",
+            "/v1/evals/review/",
+            body={"ids": ids, "force": params.force},
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary=f"Ran the AI Reviewer over {len(ids)} conversation(s).",
+        next_actions=[
+            "Use review_status to read the verdicts and the Reviewer's reasons.",
+            "Use review_set_verdict to confirm or overrule a verdict.",
+        ],
+    )
+
+
+async def review_status_handler(
+    client: AppliedClient, params: ReviewStatusInput
+) -> ToolResult[Any]:
+    ids = params.conversation_ids[:200]
+    try:
+        data = await client.request(
+            "GET", "/v1/evals/status/", params={"ids": ",".join(ids)}
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    rows = data.get("rows", []) if isinstance(data, dict) else []
+    return ToolResult(
+        data=data,
+        summary=f"Verdict pairs for {len(rows)} of {len(ids)} conversation(s).",
+        next_actions=[
+            "Use review_get for the full review pair + notes on one conversation.",
+            "Use review_run with the unreviewed ids to grade them.",
+        ],
+    )
+
+
+async def review_get_handler(
+    client: AppliedClient, params: ReviewGetInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request("GET", f"/v1/evals/{params.conversation_id}/")
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary=f"Review pair + notes for conversation {params.conversation_id}.",
+        next_actions=[
+            "Use review_set_verdict to record the human verdict.",
+            "Use review_make_test to turn a flagged conversation into a test.",
+        ],
+    )
+
+
+async def review_set_verdict_handler(
+    client: AppliedClient, params: ReviewSetVerdictInput
+) -> ToolResult[Any]:
+    body: dict[str, Any] = {"verdict": params.verdict}
+    if params.reason is not None:
+        body["reason"] = params.reason
+    if params.pinned is not None:
+        body["pinned"] = params.pinned
+    try:
+        data = await client.request(
+            "POST", f"/v1/evals/{params.conversation_id}/review/", body=body
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary=f"Recorded a '{params.verdict}' verdict.",
+        next_actions=(
+            [
+                "Use review_make_test to add this as a regression test.",
+                "Use review_add_note to flag the specific failing turn.",
+            ]
+            if params.verdict == "fail"
+            else []
+        ),
+    )
+
+
+async def review_clear_verdict_handler(
+    client: AppliedClient, params: ReviewClearVerdictInput
+) -> ToolResult[Any]:
+    try:
+        await client.request("DELETE", f"/v1/evals/{params.conversation_id}/review/")
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data={"conversation_id": params.conversation_id, "cleared": True},
+        summary="Withdrew the human verdict.",
+    )
+
+
+async def review_add_note_handler(
+    client: AppliedClient, params: ReviewAddNoteInput
+) -> ToolResult[Any]:
+    if not (params.observed or params.desired):
+        return ToolResult(
+            ok=False,
+            error=ToolError(
+                code="input.invalid",
+                message="A note needs observed and/or desired text.",
+                retryable=False,
+            ),
+        )
+    body: dict[str, Any] = {"kind": params.kind}
+    if params.observed is not None:
+        body["observed"] = params.observed
+    if params.desired is not None:
+        body["desired"] = params.desired
+    if params.message_id is not None:
+        body["message_id"] = params.message_id
+    try:
+        data = await client.request(
+            "POST", f"/v1/evals/{params.conversation_id}/notes/", body=body
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(data=data, summary=f"Added a '{params.kind}' note.")
+
+
+async def review_trust_handler(
+    client: AppliedClient, params: EmptyInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request("GET", "/v1/evals/trust/")
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    pairs = data.get("total_pairs") if isinstance(data, dict) else None
+    return ToolResult(
+        data=data,
+        summary=(
+            f"Reviewer↔human calibration over {pairs} judged pair(s)."
+            if pairs is not None
+            else "Reviewer↔human calibration by topic."
+        ),
+    )
+
+
+async def review_lessons_handler(
+    client: AppliedClient, params: EmptyInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request("GET", "/v1/evals/lessons/")
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    lessons = data.get("lessons", []) if isinstance(data, dict) else []
+    return ToolResult(
+        data=data,
+        summary=f"The Reviewer has learned {len(lessons)} lesson(s).",
+        next_actions=["Use review_teach to add a rule directly."],
+    )
+
+
+async def review_teach_handler(
+    client: AppliedClient, params: ReviewTeachInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request(
+            "POST", "/v1/evals/lessons/", body={"text": params.text}
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary="Taught the Reviewer a new rule (pinned).",
+    )
+
+
+async def review_make_test_handler(
+    client: AppliedClient, params: ReviewMakeTestInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request(
+            "POST",
+            "/v1/simulation-test-proposals/from-conversation/",
+            body={"conversation_id": params.conversation_id},
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary="Proposed a regression test (with an auto-built test customer).",
+        next_actions=[
+            "Use review_proposals_list to see the change inbox.",
+            "Use review_proposals_decide to accept it into the suite.",
+        ],
+    )
+
+
+async def review_proposals_list_handler(
+    client: AppliedClient, params: ProposalsListInput
+) -> ToolResult[Any]:
+    try:
+        data = await client.request(
+            "GET",
+            "/v1/simulation-test-proposals/",
+            params={"status": params.status},
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    items = data.get("items", []) if isinstance(data, dict) else []
+    return ToolResult(
+        data=data,
+        summary=f"{len(items)} {params.status} test proposal(s).",
+        next_actions=[
+            "Use review_proposals_decide to accept or dismiss a proposal.",
+        ],
+    )
+
+
+async def review_proposals_decide_handler(
+    client: AppliedClient, params: ProposalsDecideInput
+) -> ToolResult[Any]:
+    body: dict[str, Any] = {"status": params.decision}
+    if params.customer_state_ref is not None:
+        body["customer_state_ref"] = params.customer_state_ref
+    if params.reason is not None:
+        body["reason"] = params.reason
+    try:
+        data = await client.request(
+            "PATCH",
+            f"/v1/simulation-test-proposals/{params.proposal_id}/",
+            body=body,
+        )
+    except AppliedAPIError as exc:
+        return _api_error_result(exc)
+    return ToolResult(
+        data=data,
+        summary=f"Marked proposal {params.proposal_id} {params.decision}.",
+        next_actions=(
+            ["Use scenarios_bulk_run to run the newly added test."]
+            if params.decision == "accepted"
+            else []
+        ),
+    )
+
+
+# ── specs ───────────────────────────────────────────────────────────────────
+
+
+def evals_specs() -> list[ToolSpec]:
+    return [
+        ToolSpec(
+            name="review_run",
+            namespace="review",
+            description=(
+                "Run the AI Reviewer over one or more real conversations. It grades "
+                "each pass/fail against learned criteria without ever reading the "
+                "human verdict. Skips already-reviewed conversations unless force=true."
+            ),
+            input_model=ReviewRunInput,
+            output_model=None,
+            handler=review_run_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_status",
+            namespace="review",
+            description=(
+                "Fetch the verdict pair (AI + human) and the Reviewer's one-line "
+                "reason for a set of conversations, plus shop-wide review counts."
+            ),
+            input_model=ReviewStatusInput,
+            output_model=None,
+            handler=review_status_handler,
+            read_write_mode="read",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_get",
+            namespace="review",
+            description=(
+                "Get the full review of one conversation: the AI verdict + reason, "
+                "the human verdict, and every turn-level note."
+            ),
+            input_model=ReviewGetInput,
+            output_model=None,
+            handler=review_get_handler,
+            read_write_mode="read",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_set_verdict",
+            namespace="review",
+            description=(
+                "Record the human verdict on a conversation (pass/fail) with an "
+                "optional teaching reason. The reason folds into the Reviewer's "
+                "criteria automatically. Upserts — call again to change it."
+            ),
+            input_model=ReviewSetVerdictInput,
+            output_model=None,
+            handler=review_set_verdict_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_clear_verdict",
+            namespace="review",
+            description="Withdraw the human verdict on a conversation (undo).",
+            input_model=ReviewClearVerdictInput,
+            output_model=None,
+            handler=review_clear_verdict_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_add_note",
+            namespace="review",
+            description=(
+                "Attach a turn-level note to a conversation: a correction (what the "
+                "bot did wrong / what it should have done) or a good example."
+            ),
+            input_model=ReviewAddNoteInput,
+            output_model=None,
+            handler=review_add_note_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_trust",
+            namespace="review",
+            description=(
+                "Per-topic Reviewer↔human calibration: agreement, and false passes "
+                "(the Reviewer passed what a human flagged) — the direction to watch."
+            ),
+            input_model=EmptyInput,
+            output_model=None,
+            handler=review_trust_handler,
+            read_write_mode="read",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_lessons",
+            namespace="review",
+            description=(
+                "What the Reviewer has learned — the capped set of human signals "
+                "(verdict reasons, turn notes, taught rules, merged lessons) its "
+                "criteria compile from, with provenance."
+            ),
+            input_model=EmptyInput,
+            output_model=None,
+            handler=review_lessons_handler,
+            read_write_mode="read",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_teach",
+            namespace="review",
+            description=(
+                "Teach the Reviewer a rule directly. Lands pinned and applies to "
+                "every review from the next one on."
+            ),
+            input_model=ReviewTeachInput,
+            output_model=None,
+            handler=review_teach_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_make_test",
+            namespace="review",
+            description=(
+                "Turn a reviewed conversation into a proposed regression test — "
+                "drafts both the test and a fictionalized test customer, and lands "
+                "it in the suite's change inbox. The one path from a bad "
+                "conversation to a suite candidate."
+            ),
+            input_model=ReviewMakeTestInput,
+            output_model=None,
+            handler=review_make_test_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_proposals_list",
+            namespace="review",
+            description=(
+                "List test proposals in the change inbox (proposed / accepted / "
+                "dismissed) — adds, edits, retires, and merges awaiting a decision."
+            ),
+            input_model=ProposalsListInput,
+            output_model=None,
+            handler=review_proposals_list_handler,
+            read_write_mode="read",
+            tags=["evals", "native"],
+        ),
+        ToolSpec(
+            name="review_proposals_decide",
+            namespace="review",
+            description=(
+                "Accept or dismiss a test proposal. Accepting an add/merge without "
+                "a fixture needs a customer_state_ref; dismissing takes an optional "
+                "reason used as a negative training signal."
+            ),
+            input_model=ProposalsDecideInput,
+            output_model=None,
+            handler=review_proposals_decide_handler,
+            read_write_mode="write",
+            tags=["evals", "native"],
+        ),
+    ]

{applied_cli-0.6.9 → applied_cli-0.6.11}/applied_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: applied-cli
-Version: 0.6.9
+Version: 0.6.11
 Summary: CLI and shared client library for Applied Labs AI support agents
 Author: Applied Labs
 License-Expression: MIT
@@ -82,6 +82,8 @@ Raw analytics SQL is not available through the public CLI surface.
 
 ## Benchmarks & Scenarios
 
+> **LEGACY surface.** Benchmarks and scenarios are the older regression-suite system. The current evaluation system is the **/evals** surface (feedback + simulations + test data), driven by the `simulation_test_*`, `test_customer_*`, and `simulation_run_*` tools (MCP / `/v1/simulation-tests/`, `/v1/test-customers/`, `/v1/simulation-runs/`). The `feedback` field on `scenario_update` / `scenario_run_update` is a legacy free-text note, NOT the new Judgment+Comment feedback surface. Use the /evals tools for new evaluation work; the tools below are for maintaining existing benchmark suites.
+
 A **benchmark** is a named regression suite; a **scenario** is one test conversation
 (built from a real `input_conversation_id`) that can belong to one or more benchmarks.
 The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
@@ -97,6 +99,11 @@ applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
 applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
   --benchmark-id <benchmark_id>
 
+# Build a suite fast from several real conversations at once. Each scenario is
+# named from its conversation's title (or "<prefix> N" with --name-prefix).
+applied scenario-create-bulk --conversation-ids <id1>,<id2>,<id3> \
+  --benchmark-id <benchmark_id>
+
 # Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
 # scenarios under the destination agent; same-agent just tags them in.
 # Dry-run by default; add --apply to write.
@@ -116,6 +123,9 @@ applied scenario-bulk-cancel <job_id> --apply
 # Review pass/fail health (pass_status reflects the latest run per scenario)
 applied benchmark-results <benchmark_id> --format json
 
+# Portfolio go/no-go: pass rates across all of an agent's benchmarks at a glance
+applied benchmarks --agent-id <agent_id> --with-results --format json
+
 # Rate scenarios as you evaluate
 applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
 
@@ -167,8 +177,22 @@ conversations = await tools.conversation_query(
 | `benchmark_clone` | Copy all scenarios from one benchmark into another |
 | `benchmark_delete` | Delete a benchmark (guards against wiping scenarios) |
 | `benchmark_results` | Pass/fail/unrated tally and pass rate for a benchmark |
+| `benchmark_list` | List benchmarks (with per-benchmark pass rates via `with_results`) |
+| `scenario_create_bulk` | Build scenarios from several conversations at once |
 | `scenario_bulk_run` | Run scenarios (contact override + wait-to-completion) |
 | `scenario_bulk_cancel` | Cancel a stuck bulk run's queued/running scenario runs |
+| `review_run` | Grade real conversations with the AI Reviewer (pass/fail) |
+| `review_status` | Verdict pairs + the Reviewer's reason for a set of conversations |
+| `review_set_verdict` | Record the human verdict with a teaching reason |
+| `review_teach` | Teach the Reviewer a rule directly |
+| `review_make_test` | Turn a flagged conversation into a proposed regression test |
+| `review_proposals_list` | The suite change-inbox (proposed adds/edits/retires/merges) |
+| `review_proposals_decide` | Accept or dismiss a proposed test |
+
+The full `review_*` namespace also includes `review_get`, `review_clear_verdict`,
+`review_add_note`, `review_trust`, and `review_lessons` — the outer review loop:
+grade real traffic, confirm/overrule with a reason that teaches the Reviewer, and
+promote failures into the simulation suite as tests.
 
 ## Examples
 

{applied_cli-0.6.9 → applied_cli-0.6.11}/applied_cli.egg-info/SOURCES.txt

@@ -28,6 +28,7 @@ applied_cli/v2/connectors.py
 applied_cli/v2/content.py
 applied_cli/v2/conversations.py
 applied_cli/v2/domains.py
+applied_cli/v2/evals.py
 applied_cli/v2/flows.py
 applied_cli/v2/knowledge.py
 applied_cli/v2/manifest.py
@@ -62,6 +63,7 @@ tests/test_v2_catalog_and_mcp.py
 tests/test_v2_connectors.py
 tests/test_v2_content.py
 tests/test_v2_conversations.py
+tests/test_v2_evals.py
 tests/test_v2_flows.py
 tests/test_v2_knowledge.py
 tests/test_v2_products.py

{applied_cli-0.6.9 → applied_cli-0.6.11}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "applied-cli"
-version = "0.6.9"
+version = "0.6.11"
 description = "CLI and shared client library for Applied Labs AI support agents"
 readme = "README.md"
 requires-python = ">=3.11"

applied_cli-0.6.11/tests/test_v2_evals.py

@@ -0,0 +1,163 @@
+import pytest
+
+from applied_cli.v2.catalog import get_tool_catalog
+
+
+class FakeEvalsClient:
+    """Records the raw request(method, path, params, body) each tool makes and
+    returns a canned response, so we assert the exact wire shape."""
+
+    def __init__(self, response=None):
+        self.access_mode = "full_access"
+        self.shop_id = "shop-1"
+        self.calls: list[dict] = []
+        self._response = response if response is not None else {}
+
+    async def request(self, method, path, params=None, body=None, shop_id=None):
+        self.calls.append(
+            {"method": method, "path": path, "params": params, "body": body}
+        )
+        return self._response
+
+
+async def _run(tool_name, client, **kwargs):
+    spec = get_tool_catalog()[tool_name]
+    return await spec.run(client, kwargs)
+
+
+def test_all_review_tools_registered():
+    catalog = get_tool_catalog()
+    review = {s.name for s in catalog if s.namespace == "review"}
+    assert review == {
+        "review_run",
+        "review_status",
+        "review_get",
+        "review_set_verdict",
+        "review_clear_verdict",
+        "review_add_note",
+        "review_trust",
+        "review_lessons",
+        "review_teach",
+        "review_make_test",
+        "review_proposals_list",
+        "review_proposals_decide",
+    }
+
+
+@pytest.mark.asyncio
+async def test_review_run_posts_ids_and_force():
+    client = FakeEvalsClient({"graded": 2, "flagged": 1})
+    result = await _run("review_run", client, conversation_ids=["a", "b"], force=True)
+    assert result.ok
+    assert client.calls[0] == {
+        "method": "POST",
+        "path": "/v1/evals/review/",
+        "params": None,
+        "body": {"ids": ["a", "b"], "force": True},
+    }
+    assert result.data == {"graded": 2, "flagged": 1}
+
+
+@pytest.mark.asyncio
+async def test_review_run_rejects_empty_ids_without_calling_api():
+    client = FakeEvalsClient()
+    result = await _run("review_run", client, conversation_ids=[])
+    assert not result.ok
+    assert result.error.code == "input.invalid"
+    assert client.calls == []
+
+
+@pytest.mark.asyncio
+async def test_review_status_joins_ids_into_query():
+    client = FakeEvalsClient({"rows": [{"conversation_id": "a"}], "counts": {}})
+    await _run("review_status", client, conversation_ids=["a", "b", "c"])
+    assert client.calls[0]["method"] == "GET"
+    assert client.calls[0]["path"] == "/v1/evals/status/"
+    assert client.calls[0]["params"] == {"ids": "a,b,c"}
+
+
+@pytest.mark.asyncio
+async def test_review_set_verdict_omits_unset_optionals():
+    client = FakeEvalsClient({"verdict": "fail"})
+    await _run(
+        "review_set_verdict",
+        client,
+        conversation_id="conv-1",
+        verdict="fail",
+        reason="Promised a refund it never issued.",
+    )
+    call = client.calls[0]
+    assert call["path"] == "/v1/evals/conv-1/review/"
+    assert call["body"] == {
+        "verdict": "fail",
+        "reason": "Promised a refund it never issued.",
+    }
+    assert "pinned" not in call["body"]
+
+
+@pytest.mark.asyncio
+async def test_review_clear_verdict_deletes():
+    client = FakeEvalsClient(None)
+    result = await _run("review_clear_verdict", client, conversation_id="conv-1")
+    assert result.ok
+    assert client.calls[0]["method"] == "DELETE"
+    assert client.calls[0]["path"] == "/v1/evals/conv-1/review/"
+
+
+@pytest.mark.asyncio
+async def test_review_add_note_requires_text():
+    client = FakeEvalsClient()
+    result = await _run(
+        "review_add_note", client, conversation_id="conv-1", kind="correction"
+    )
+    assert not result.ok
+    assert result.error.code == "input.invalid"
+    assert client.calls == []
+
+
+@pytest.mark.asyncio
+async def test_review_teach_posts_rule_text():
+    client = FakeEvalsClient({"id": "note-1"})
+    await _run("review_teach", client, text="Always verify the subscription first.")
+    assert client.calls[0] == {
+        "method": "POST",
+        "path": "/v1/evals/lessons/",
+        "params": None,
+        "body": {"text": "Always verify the subscription first."},
+    }
+
+
+@pytest.mark.asyncio
+async def test_review_make_test_hits_from_conversation():
+    client = FakeEvalsClient({"id": "prop-1", "kind": "add"})
+    await _run("review_make_test", client, conversation_id="conv-9")
+    assert client.calls[0]["path"] == (
+        "/v1/simulation-test-proposals/from-conversation/"
+    )
+    assert client.calls[0]["body"] == {"conversation_id": "conv-9"}
+
+
+@pytest.mark.asyncio
+async def test_review_proposals_decide_accept_with_customer():
+    client = FakeEvalsClient({"status": "accepted"})
+    await _run(
+        "review_proposals_decide",
+        client,
+        proposal_id="prop-1",
+        decision="accepted",
+        customer_state_ref="cust-1",
+    )
+    call = client.calls[0]
+    assert call["method"] == "PATCH"
+    assert call["path"] == "/v1/simulation-test-proposals/prop-1/"
+    assert call["body"] == {"status": "accepted", "customer_state_ref": "cust-1"}
+
+
+@pytest.mark.asyncio
+async def test_unknown_field_rejected_before_handler():
+    client = FakeEvalsClient()
+    spec = get_tool_catalog()["review_trust"]
+    result = await spec.run(client, {"bogus": 1})
+    assert not result.ok
+    assert result.error.code == "input.invalid"
+    assert client.calls == []