cassis-cli 3.3.0__tar.gz → 3.4.0__tar.gz

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cassis-cli
-Version: 3.3.0
+Version: 3.4.0
 Summary: Validate, test and evaluate your Cassis context from your terminal, then publish it
 License: Apache-2.0
 License-File: LICENSE
@@ -33,7 +33,8 @@ Validate, test and evaluate your context from your terminal, then publish it. Th
 - The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version. When you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local context files (scored in-memory, nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
 - `cassis context test` runs individual questions through the text-to-SQL agent using your local context files, so you can check that a change actually works (e.g. a new column gets picked), where `eval run` only checks for regressions on existing eval cases.
-- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite: after fixing a context issue, add the question users were failing on so `eval run` guards it from regressing.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite: after fixing a context issue, add the question users were failing on so `eval run` guards it from regressing. Cassis then runs the new case three times in the background, and `eval list-cases` flags it if the attempts fail or disagree.
+- `cassis eval run --repeat N` runs every case N times (2 to 5) on the same context and model and compares the attempts per case: which cases flipped, and whether a flip came from the agent writing different SQL or the judge ruling differently on the same SQL. `cassis eval compare <run-id>` prints that comparison again. A repeat run measures noise: it opens no issue, is not the run shown before publishing, and exits 1 only when a case failed every attempt.
 - `cassis eval list-cases`, `cassis eval history` and `cassis eval delete-case` maintain the suite: list the current cases with their ids (flagging those whose gold SQL reads tables the context lacks), see how one case has behaved across runs (pass rate, whether repeated runs under one configuration disagree, divergences and timings), and prune one that is stale or wrong (e.g. its gold SQL encodes a definition the context has since changed).
 - `cassis schema plan <ddl>` (or `--warehouse` on a project connected to a warehouse) previews what a schema update would change before anything is applied: the source schema diff, the context changes Cassis will make (every change on a table placed in the context, with everything a drop takes with it) and warnings, terraform-style. `cassis schema apply <ddl>` (or `--plan <id>`) writes the resulting context files into the local checkout, app untouched, for review with `git diff`. `cassis schema push <ddl> [--publish]` pushes the new schema and the local context to the app (`--yes` in CI). The file speaks only for the schemas it contains: pass `--complete` when it is the project's complete source schema so schemas absent from it are treated as dropped. With `--warehouse` the server introspects the connected warehouse instead of parsing a file; the plan is always whole-source. `cassis schema plan <ddl> --dry-run` is the prepare-ahead variant: the plan is computed synchronously and nothing is kept in Cassis (no plan to apply or resume, the current plan untouched), so the schema snapshot for a change still in a PR can be planned against safely; `--write-checkout` writes the context files it would produce into the checkout, to commit alongside the schema change.
 - `cassis projects list` lists the projects your API key can reach: id (what `--project` and `CASSIS_PROJECT_ID` take), name, published context version, and data-source dialect. A pipeline or agent can discover the project id from the terminal instead of fishing it out of a webapp URL.
@@ -141,6 +142,10 @@ cassis eval list-cases --project ...
 cassis eval history 019f0000-0000-7000-8000-0000000000ca --project ...
 cassis eval delete-case 019f0000-0000-7000-8000-0000000000ca --project ...
 
+# Run the suite 3 times to see which cases flip, and why; print it again later:
+cassis eval run --repeat 3
+cassis eval compare 019f0000-0000-7000-8000-00000000aaaa --project ...
+
 # Pull the source schema into <base-path>/.schema.json (gitignored local snapshot):
 cassis schema pull
 
@@ -203,7 +208,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | none (required)             |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis`, must match the project's git-sync "Path" setting |
-| `--project` (check, pull, upload, schema pull, eval run, eval add-case, eval list-cases, eval history, eval delete-case, test, issues, dbt push, dbt check) | `CASSIS_PROJECT_ID` | the id in `<base-path>/project.yml` (required before the first pull; `check` alone falls back to the project-less validation when unbound) |
+| `--project` (check, pull, upload, schema pull, eval run, eval add-case, eval list-cases, eval history, eval compare, eval delete-case, test, issues, dbt push, dbt check) | `CASSIS_PROJECT_ID` | the id in `<base-path>/project.yml` (required before the first pull; `check` alone falls back to the project-less validation when unbound) |
 
 `cassis eval run` also accepts `--case <id>` (repeatable; run only the named
 cases, ids from `eval list-cases` or `add-case`), `--label` (run label in the Evals page; defaults

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/README.md

@@ -11,7 +11,8 @@ Validate, test and evaluate your context from your terminal, then publish it. Th
 - The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version. When you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local context files (scored in-memory, nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
 - `cassis context test` runs individual questions through the text-to-SQL agent using your local context files, so you can check that a change actually works (e.g. a new column gets picked), where `eval run` only checks for regressions on existing eval cases.
-- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite: after fixing a context issue, add the question users were failing on so `eval run` guards it from regressing.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite: after fixing a context issue, add the question users were failing on so `eval run` guards it from regressing. Cassis then runs the new case three times in the background, and `eval list-cases` flags it if the attempts fail or disagree.
+- `cassis eval run --repeat N` runs every case N times (2 to 5) on the same context and model and compares the attempts per case: which cases flipped, and whether a flip came from the agent writing different SQL or the judge ruling differently on the same SQL. `cassis eval compare <run-id>` prints that comparison again. A repeat run measures noise: it opens no issue, is not the run shown before publishing, and exits 1 only when a case failed every attempt.
 - `cassis eval list-cases`, `cassis eval history` and `cassis eval delete-case` maintain the suite: list the current cases with their ids (flagging those whose gold SQL reads tables the context lacks), see how one case has behaved across runs (pass rate, whether repeated runs under one configuration disagree, divergences and timings), and prune one that is stale or wrong (e.g. its gold SQL encodes a definition the context has since changed).
 - `cassis schema plan <ddl>` (or `--warehouse` on a project connected to a warehouse) previews what a schema update would change before anything is applied: the source schema diff, the context changes Cassis will make (every change on a table placed in the context, with everything a drop takes with it) and warnings, terraform-style. `cassis schema apply <ddl>` (or `--plan <id>`) writes the resulting context files into the local checkout, app untouched, for review with `git diff`. `cassis schema push <ddl> [--publish]` pushes the new schema and the local context to the app (`--yes` in CI). The file speaks only for the schemas it contains: pass `--complete` when it is the project's complete source schema so schemas absent from it are treated as dropped. With `--warehouse` the server introspects the connected warehouse instead of parsing a file; the plan is always whole-source. `cassis schema plan <ddl> --dry-run` is the prepare-ahead variant: the plan is computed synchronously and nothing is kept in Cassis (no plan to apply or resume, the current plan untouched), so the schema snapshot for a change still in a PR can be planned against safely; `--write-checkout` writes the context files it would produce into the checkout, to commit alongside the schema change.
 - `cassis projects list` lists the projects your API key can reach: id (what `--project` and `CASSIS_PROJECT_ID` take), name, published context version, and data-source dialect. A pipeline or agent can discover the project id from the terminal instead of fishing it out of a webapp URL.
@@ -119,6 +120,10 @@ cassis eval list-cases --project ...
 cassis eval history 019f0000-0000-7000-8000-0000000000ca --project ...
 cassis eval delete-case 019f0000-0000-7000-8000-0000000000ca --project ...
 
+# Run the suite 3 times to see which cases flip, and why; print it again later:
+cassis eval run --repeat 3
+cassis eval compare 019f0000-0000-7000-8000-00000000aaaa --project ...
+
 # Pull the source schema into <base-path>/.schema.json (gitignored local snapshot):
 cassis schema pull
 
@@ -181,7 +186,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | none (required)             |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis`, must match the project's git-sync "Path" setting |
-| `--project` (check, pull, upload, schema pull, eval run, eval add-case, eval list-cases, eval history, eval delete-case, test, issues, dbt push, dbt check) | `CASSIS_PROJECT_ID` | the id in `<base-path>/project.yml` (required before the first pull; `check` alone falls back to the project-less validation when unbound) |
+| `--project` (check, pull, upload, schema pull, eval run, eval add-case, eval list-cases, eval history, eval compare, eval delete-case, test, issues, dbt push, dbt check) | `CASSIS_PROJECT_ID` | the id in `<base-path>/project.yml` (required before the first pull; `check` alone falls back to the project-less validation when unbound) |
 
 `cassis eval run` also accepts `--case <id>` (repeatable; run only the named
 cases, ids from `eval list-cases` or `add-case`), `--label` (run label in the Evals page; defaults

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/api.py

@@ -594,6 +594,7 @@ def post_eval_run_start(
     branch: Optional[str] = None,
     label: Optional[str] = None,
     case_ids: Optional[list[str]] = None,
+    repeat_count: Optional[int] = None,
     transport: Optional[httpx.BaseTransport] = None,
 ) -> dict[str, Any]:
     """POST to /api/ci/projects/{project_id}/eval/runs and return the run record."""
@@ -607,6 +608,8 @@ def post_eval_run_start(
         body["label"] = label
     if case_ids is not None:
         body["test_case_ids"] = case_ids
+    if repeat_count is not None:
+        body["repeat_count"] = repeat_count
     try:
         # May carry the whole ontology tree in `files` — same budget as the
         # other whole-tree endpoints.
@@ -833,6 +836,41 @@ def get_eval_run_results(
     return result
 
 
+class EvalCompareUnsupportedError(ApiError):
+    """The server predates run comparisons."""
+
+
+def get_eval_run_comparison(
+    *,
+    api_url: str,
+    api_key: str,
+    project_id: str,
+    run_id: str,
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """GET /api/ci/projects/{project_id}/eval/runs/{run_id}/comparison: the run's attempts compared per case."""
+    url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/eval/runs/{run_id}/comparison"
+    try:
+        with _client(transport=transport) as client:
+            response = client.get(url, headers={"Authorization": f"Bearer {api_key}"})
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    # A server without the route answers the framework's own "Not Found",
+    # distinct from the "Eval run not found" a real lookup returns.
+    if response.status_code == 404 and _detail_or_text(response) == "Not Found":
+        raise EvalCompareUnsupportedError("This Cassis server does not compare runs yet; it needs a newer server.")
+    if response.status_code in (403, 404):
+        raise _project_scope_error(response)
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if not isinstance(result, dict) or not all(key in result for key in ("run_id", "summary", "cases")):
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result
+
+
 def post_eval_run_cancel(
     *,
     api_url: str,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/dbt.py

@@ -111,7 +111,9 @@ _BASE_PATH_OPTION = typer.Option(
     DEFAULT_BASE_PATH, "--base-path", envvar="CASSIS_BASE_PATH", help="Context directory holding project.yml."
 )
 _POLL_INTERVAL_OPTION = typer.Option(2.0, "--poll-interval", help="Seconds between polls.")
-_TIMEOUT_OPTION = typer.Option(600.0, "--timeout", help="Give up waiting after this many seconds.")
+_TIMEOUT_OPTION = typer.Option(
+    600.0, "--timeout", help="Give up waiting after this many seconds (exit 3; the upload keeps processing)."
+)
 _JSON_OPTION = typer.Option(False, "--json", help="Print the upload record as raw JSON (the summary goes to stderr).")
 
 
@@ -139,7 +141,9 @@ def push(
     baseline, and files each part of the context the change puts in doubt as
     an issue (impact "at risk"). Exits 0 when accepted or superseded by a
     newer upload, 1 when rejected (invalid files, or no relation matches the
-    data source), 2 on usage errors, 3 on transport errors.
+    data source), 2 on usage errors, 3 on transport errors or a timeout (the
+    upload keeps processing server-side), or when Cassis fails on its side
+    while processing it.
     """
     record = _upload_and_wait(
         "push",
@@ -189,7 +193,8 @@ def check(
     nothing is filed and the baseline doesn't move. Exits 0 with or without
     findings, 1 when the upload is rejected (invalid files, or no baseline
     yet: run `cassis dbt push` on the production branch first), 2 on usage
-    errors, 3 on transport errors.
+    errors, 3 on transport errors or a timeout (the upload keeps processing
+    server-side), or when Cassis fails on its side while processing it.
     """
     record = _upload_and_wait(
         "check",
@@ -287,7 +292,7 @@ def _upload_and_wait(
 
     def on_timeout(last: "dict[str, Any]") -> None:
         typer.secho(
-            f"Timed out after {timeout:.0f}s: upload {observation_id} is still {last.get('status', '?')}. "
+            f"Timed out after {timeout:g}s: upload {observation_id} is still {last.get('status', '?')}. "
             "It keeps processing server-side.",
             fg=typer.colors.YELLOW,
             err=True,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/eval.py

@@ -25,6 +25,7 @@ from cassis_cli.api import (
     get_eval_case_history,
     get_eval_cases,
     get_eval_run,
+    get_eval_run_comparison,
     get_eval_run_results,
     post_eval_case_create,
     post_eval_run_cancel,
@@ -272,6 +273,22 @@ def _footprint_note(case: dict[str, Any]) -> Optional[str]:
     return "; ".join(reasons)
 
 
+def _entry_check_note(case: dict[str, Any]) -> Optional[str]:
+    """One line on a case whose creation check flags it; None when stable or never checked."""
+    check = case.get("entry_check")
+    if not isinstance(check, dict):
+        return None
+    state = check.get("state")
+    score = f"{check.get('passed')}/{check.get('total')}" if check.get("total") else ""
+    if state == "flaky":
+        return f"flaky at creation ({score} attempts passed): its verdict is noise until the question or gold SQL is tightened"
+    if state == "failing":
+        return f"failing at creation ({score} attempts passed): check the gold SQL, or the context gap it exposes"
+    if state == "running":
+        return "entry check running (3 attempts on the current context)"
+    return None
+
+
 @app.command(name="add-case")
 def add_case(
     path: Path = typer.Argument(
@@ -328,6 +345,8 @@ def add_case(
     case is stored, so a case that cannot execute never enters the suite.
     The SQL comes from --gold-sql (inline) or --gold-sql-file (a file path;
     prefer it for multi-line SQL, which shell quoting mangles inline).
+    Cassis then runs the new case three times in the background, and
+    `cassis eval list-cases` flags it if the attempts fail or disagree.
     Exits 0 on creation, 1 on a duplicate question or gold SQL that does not
     run, 2 on usage errors, 3 on transport/API errors.
     """
@@ -379,6 +398,12 @@ def add_case(
                 fg=typer.colors.YELLOW,
                 err=True,
             )
+        check = case.get("entry_check")
+        if isinstance(check, dict) and check.get("state") == "running":
+            typer.echo(
+                "  Checking it: 3 attempts run on the current context in the background; "
+                "`cassis eval list-cases` flags it if they fail or disagree."
+            )
     raise typer.Exit(EXIT_OK)
 
 
@@ -427,7 +452,9 @@ def list_cases(
     what `cassis eval delete-case` and `cassis eval history` take. A case
     whose gold SQL does not fully resolve against the context (a table the
     context lacks, an ambiguous name, SQL that does not parse) is flagged
-    under its line, since such a case cannot be trusted to score a change.
+    under its line, since such a case cannot be trusted to score a change,
+    and so is a case whose three-attempt check after creation failed or
+    disagreed.
     Exits 0 on success, 2 on usage errors (including a --branch the project
     does not have), 3 on transport/API errors.
     """
@@ -459,6 +486,13 @@ def list_cases(
             if note:
                 flagged += 1
                 typer.secho(f"    ⚠ {note}", fg=typer.colors.YELLOW)
+            check_note = _entry_check_note(case)
+            if check_note:
+                running = (case.get("entry_check") or {}).get("state") == "running"
+                typer.secho(
+                    f"    {'…' if running else '⚠'} {check_note}",
+                    fg=None if running else typer.colors.YELLOW,
+                )
         if flagged:
             typer.secho(
                 f"{flagged} case(s) have a gold SQL that does not fully resolve against the context; "
@@ -469,6 +503,55 @@ def list_cases(
     raise typer.Exit(EXIT_OK)
 
 
+# How a flip reads to the person fixing it: which side of the evaluation moved.
+_CAUSE_HINTS = {
+    "judge": "the judge ruled differently on the same SQL: check the gold SQL and the question's wording",
+    "generation": "the agent wrote different SQL from the same context: the context leaves room for both",
+    "both": "the agent wrote different SQL and the judge also wavered",
+    "unattributed": "the evidence decides neither side (a re-judge failed, or the same SQL returned different rows)",
+}
+
+
+def _print_comparison(comparison: dict[str, Any]) -> None:
+    """A repeat run's attempts per case: the same comparison the webapp's run page shows."""
+    summary = comparison.get("summary") or {}
+    repeat = comparison.get("repeat_count") or 1
+    cases = comparison.get("cases") or []
+    typer.secho(f"Repeat comparison: {repeat} attempts per case, {summary.get('cases', len(cases))} cases", bold=True)
+    causes = summary.get("causes") or {}
+    flipped = f"{summary.get('flipped', 0)} flipped"
+    if causes:
+        flipped += " (" + ", ".join(f"{k} {n}" for k, n in causes.items()) + ")"
+    typer.echo(
+        f"  {summary.get('all_passed', 0)} passed every attempt · "
+        f"{summary.get('all_failed', 0)} failed every attempt · {flipped}"
+    )
+    if repeat < 2:
+        typer.echo("  Not a repeat run: start one with `cassis eval run --repeat N` to measure noise.")
+        return
+    shown = [c for c in cases if c.get("agreement") != "all_passed"]
+    if not shown:
+        return
+    typer.echo("")
+    for c in shown:
+        agreement = c.get("agreement")
+        icon = "~" if agreement == "flipped" else "✗"
+        color = typer.colors.YELLOW if agreement == "flipped" else typer.colors.RED
+        distinct = int(c.get("distinct_sql") or 0)
+        sqls = f"{distinct} SQL" if distinct == 1 else f"{distinct} SQLs"
+        cause = str(c.get("cause") or "")
+        question = str(c.get("question") or "").replace("\n", " ")
+        if len(question) > 60:
+            question = question[:57] + "..."
+        typer.secho(f"  {icon} {c.get('passed', 0)}/{c.get('total', 0)}  {cause:<13} {sqls:<8} {question}", fg=color)
+        typer.echo(f"      {c.get('test_case_id')}")
+    seen = [k for k in _CAUSE_HINTS if k in causes]
+    if seen:
+        typer.echo("")
+        for k in seen:
+            typer.echo(f"  {k}: {_CAUSE_HINTS[k]}")
+
+
 def _format_when(iso: str) -> str:
     """`2026-10-23 18:14` from an ISO timestamp; the raw text when it does not parse."""
     try:
@@ -633,6 +716,76 @@ def history(
     raise typer.Exit(EXIT_OK)
 
 
+@app.command()
+def compare(
+    run_id: str = typer.Argument(
+        ...,
+        help="Id of the eval run (printed by `cassis eval run`).",
+    ),
+    path: Path = typer.Option(
+        Path("."),
+        "--path",
+        help="Repository checkout root (holds <base-path>/project.yml for the --project default).",
+    ),
+    project_id: Optional[str] = typer.Option(
+        None,
+        "--project",
+        envvar="CASSIS_PROJECT_ID",
+        help="Target Cassis project ID (UUID). Defaults to the id in <base-path>/project.yml.",
+    ),
+    api_key: Optional[str] = typer.Option(
+        None,
+        "--api-key",
+        envvar="CASSIS_API_KEY",
+        help="Cassis API key (sk-k6-...). Create one in Organization settings -> API keys.",
+    ),
+    api_url: str = typer.Option(
+        DEFAULT_API_URL,
+        "--api-url",
+        envvar="CASSIS_API_URL",
+        help="Cassis API base URL.",
+    ),
+    base_path: str = typer.Option(
+        DEFAULT_BASE_PATH,
+        "--base-path",
+        envvar="CASSIS_BASE_PATH",
+        help="Repository directory the context is exported under (holds project.yml for the --project default).",
+    ),
+    json_output: bool = typer.Option(False, "--json", help="Print the comparison as raw JSON."),
+) -> None:
+    """Compare the attempts of a repeat run, case by case.
+
+    For a run started with `cassis eval run --repeat N`: how many cases passed
+    every attempt, failed every attempt or flipped, and for each flipping case
+    how many distinct SQLs the agent wrote and what moved between attempts:
+    `generation` (the agent wrote different SQL), `judge` (the judge ruled
+    differently on the same SQL), `both`, or `unattributed`. The same
+    comparison the webapp's run page shows. Exits 0 on success, 2 on usage
+    errors, 3 on transport/API errors (including a server too old to compare).
+    """
+    api_key = require_api_key(api_key)
+    project_id = resolve_project_id(project_id, path / Path(base_path))
+    try:
+        UUID(run_id)
+    except ValueError:
+        typer.secho(f"RUN_ID must be a UUID, got {run_id!r}.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+    try:
+        comparison = get_eval_run_comparison(api_url=api_url, api_key=api_key, project_id=project_id, run_id=run_id)
+    except ApiError as exc:  # covers AuthError and EvalCompareUnsupportedError
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_TRANSPORT) from exc
+
+    if json_output:
+        typer.echo(json.dumps(comparison, indent=2))
+        raise typer.Exit(EXIT_OK)
+    if comparison.get("status") == "running":
+        typer.echo("The run is still in progress; its attempts are compared once it completes.")
+        raise typer.Exit(EXIT_OK)
+    _print_comparison(comparison)
+    raise typer.Exit(EXIT_OK)
+
+
 @app.command(name="delete-case")
 def delete_case(
     case_id: str = typer.Argument(
@@ -745,6 +898,13 @@ def run(
         "--label",
         help="Run label shown in the webapp's Evals page (default: the local git branch name).",
     ),
+    repeat: Optional[int] = typer.Option(
+        None,
+        "--repeat",
+        min=2,
+        max=5,
+        help="Run every case N times (2 to 5) as one diagnostic run, then compare the attempts per case.",
+    ),
     wait: bool = typer.Option(
         True,
         "--wait/--no-wait",
@@ -767,9 +927,14 @@ def run(
     an existing Cassis branch instead (no files are sent). With --case, only the
     named case(s) run — e.g. proving one fresh `add-case` in seconds instead of
     rerunning the whole suite; passed cases then print their SQL and evidence
-    too, so you can check the case passes for the right reason. Exits 0 when
-    the run completes with every case passed, 1 on any failed case / failed
-    run / invalid tree, 2 on usage errors (including a --branch the project
+    too, so you can check the case passes for the right reason. With --repeat
+    N, every case runs N times under the same context and model, and the
+    command prints how the attempts compare per case (`cassis eval compare`);
+    a repeat run is diagnostic, so it files no issue and is not the run the
+    webapp shows before publishing. Exits 0 when
+    the run completes with every case passed (with --repeat: no case failed
+    every attempt, since a flip is the noise being measured), 1 on any failed
+    case / failed run / invalid tree, 2 on usage errors (including a --branch the project
     does not have), 3 on transport errors or --timeout.
 
     Failed cases (and, with --case, passed ones) print the generated SQL as
@@ -808,6 +973,7 @@ def run(
             branch=branch,
             label=label,
             case_ids=case,
+            repeat_count=repeat,
         )
     except EvalStartValidationError as exc:
         _print_validation_failure(exc.detail, base_path)
@@ -831,6 +997,15 @@ def run(
     total = run_record.get("total_cases", 0)
     ontology_label = run_record.get("ontology_label") or "unpublished context"
     run_url = _run_page_url(app_url or api_url, project_id, run_id)
+    repeat_count = run_record.get("repeat_count")
+    if repeat is not None and repeat > 1 and not repeat_count:
+        # An older server drops the field: the run is an ordinary one, so say so
+        # rather than print a comparison it never measured.
+        typer.secho(
+            "This Cassis server ignored --repeat: the run executes each case once, as an ordinary run.",
+            fg=typer.colors.YELLOW,
+            err=True,
+        )
 
     if not wait:
         if json_output:
@@ -841,7 +1016,10 @@ def run(
         raise typer.Exit(EXIT_OK)
 
     # To stderr under --json so `cassis eval run --json | jq` gets only the record.
-    typer.echo(f"Eval run {run_id} started: {total} cases against {ontology_label!r}.", err=json_output)
+    started = f"Eval run {run_id} started: {total} cases against {ontology_label!r}"
+    if repeat_count:
+        started += f", {repeat_count} attempts each"
+    typer.echo(f"{started}.", err=json_output)
 
     try:
         final_run, results = _wait_for_run(
@@ -849,7 +1027,7 @@ def run(
             api_key=api_key,
             project_id=project_id,
             run_id=run_id,
-            total=total,
+            total=total * (repeat_count or 1),
             poll_interval=poll_interval,
             timeout=timeout,
             json_output=json_output,
@@ -865,21 +1043,44 @@ def run(
             typer.secho(f"Could not cancel the run: {exc}", fg=typer.colors.RED, err=True)
         raise typer.Exit(EXIT_INTERRUPTED)
 
+    comparison: Optional[dict[str, Any]] = None
+    if repeat_count and final_run.get("status") == "completed":
+        try:
+            comparison = get_eval_run_comparison(api_url=api_url, api_key=api_key, project_id=project_id, run_id=run_id)
+        except ApiError as exc:
+            typer.secho(f"Could not compare the attempts: {exc}", fg=typer.colors.YELLOW, err=True)
+
     if json_output:
-        typer.echo(json.dumps({"run": final_run, "results": results, "run_url": run_url}, indent=2))
+        record: dict[str, Any] = {"run": final_run, "results": results, "run_url": run_url}
+        if comparison is not None:
+            record["comparison"] = comparison
+        typer.echo(json.dumps(record, indent=2))
     else:
         typer.echo("")
-        _print_results_table(results, passed_evidence=bool(case))
+        if comparison is not None:
+            _print_comparison(comparison)
+        else:
+            _print_results_table(results, passed_evidence=bool(case))
         typer.echo("")
         _print_summary(final_run)
         typer.echo(f"View details: {run_url}")
 
-    status = final_run.get("status")
-    if status == "completed" and all(r.get("status") == _PASSED for r in results):
+    if final_run.get("status") == "completed" and _suite_passed(results, repeated=bool(repeat_count)):
         raise typer.Exit(EXIT_OK)
     raise typer.Exit(EXIT_VALIDATION_FAILED)
 
 
+def _suite_passed(results: "list[dict[str, Any]]", *, repeated: bool) -> bool:
+    """Whether a completed run passes: every case passed, or on a repeat run, no case failed every attempt."""
+    if not repeated:
+        return all(r.get("status") == _PASSED for r in results)
+    passed_once: dict[str, bool] = {}
+    for r in results:
+        key = str(r.get("test_case_id") or r.get("question"))
+        passed_once[key] = passed_once.get(key, False) or r.get("status") == _PASSED
+    return all(passed_once.values())
+
+
 def _wait_for_run(
     *,
     api_url: str,
@@ -928,7 +1129,7 @@ def _wait_for_run(
 
     def on_timeout(_snapshot: "tuple[dict[str, Any], list[dict[str, Any]]]") -> None:
         typer.secho(
-            f"Timed out after {timeout:.0f}s waiting for run {run_id}. "
+            f"Timed out after {timeout:g}s waiting for run {run_id}. "
             "The run keeps going server-side — see the webapp's Evals page.",
             fg=typer.colors.YELLOW,
             err=True,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/issues.py

@@ -133,8 +133,9 @@ def list_issues(
 
     The order: editorial priority (P1, P2, P3, then unranked), then impact,
     then reach, then how the issue was found (dbt findings first), then
-    recurrence. Prints each issue's id, impact, occurrence count, reach (the
-    chat answers that used its context tables in the last 30 days; `-` when
+    recurrence. Prints each issue's priority (`-` when unranked), id, impact,
+    origin (conversation, eval or dbt), occurrence count, reach (the chat
+    answers that used any of its context tables in the last 30 days; `-` when
     not measured, because its sources name no context table), status, primary
     context domain (`-` when Cassis could not attach one) and title; the id is what
     `cassis issues show`, `resolve`, `dismiss` and `reopen` take. Triage one
@@ -176,6 +177,8 @@ def list_issues(
         reach = issue.get("reach_count")
         rows.append(
             [
+                # The editorial priority the list is sorted by first; `-` when unranked.
+                str(issue.get("priority") or "-"),
                 str(issue.get("id")),
                 str(issue.get("impact")),
                 # Always printed (conversation, eval or dbt), so every line has the same columns.
@@ -621,7 +624,7 @@ def _wait_for_analysis(
 
     def on_timeout(_run: "dict[str, Any]") -> None:
         typer.secho(
-            f"Timed out after {timeout:.0f}s waiting for analysis run {run_id}. "
+            f"Timed out after {timeout:g}s waiting for analysis run {run_id}. "
             "The analysis keeps going server-side — see the webapp's Issues page.",
             fg=typer.colors.YELLOW,
             err=True,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/schema.py

@@ -889,7 +889,7 @@ def _wait_for_plan(
             else "`cassis status` shows the outcome once it finishes"
         )
         typer.secho(
-            f"Timed out after {timeout:.0f}s: the {what} is still {record.get('status', '?')}. "
+            f"Timed out after {timeout:g}s: the {what} is still {record.get('status', '?')}. "
             f"It keeps running server-side; {resume}",
             fg=typer.colors.YELLOW,
             err=True,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/status.py

@@ -197,7 +197,7 @@ def status(
 
     def on_timeout(_snapshot: "Tuple[dict[str, Any], str, bool]") -> None:
         typer.secho(
-            f"Timed out after {timeout:.0f}s: the published version still does not match the local HEAD.",
+            f"Timed out after {timeout:g}s: the published version still does not match the local HEAD.",
             fg=typer.colors.YELLOW,
             err=True,
         )

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/cassis_cli/verify.py

@@ -109,6 +109,7 @@ def verify(
                 branch=None,
                 case=None,
                 label=None,
+                repeat=None,
                 wait=True,
                 poll_interval=5.0,
                 timeout=1800.0,

{cassis_cli-3.3.0 → cassis_cli-3.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cassis-cli"
-version = "3.3.0"
+version = "3.4.0"
 description = "Validate, test and evaluate your Cassis context from your terminal, then publish it"
 readme = "README.md"
 license = { text = "Apache-2.0" }