cassis-cli 2.0.0__tar.gz → 2.1.0__tar.gz

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cassis-cli
-Version: 2.0.0
+Version: 2.1.0
 Summary: Validate, test and evaluate your Cassis ontology from your terminal, then publish it
 License: Apache-2.0
 License-File: LICENSE
@@ -104,6 +104,17 @@ cassis eval run --project ... --case 019f0000-0000-7000-8000-0000000000ca
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+```
+
+Under each failed case `eval run` prints what is needed to diagnose it: the SQL the agent
+generated, the gold SQL it was compared against, how many rows each side returned, any concepts
+the agent found missing, and the judge's reasoning when a judge graded the case. The expected and
+actual row *values* are deliberately not printed — they are in `--json` and in the Cassis app.
+Note that the **generated SQL is printed as the agent wrote it**, and an agent that read your data
+while planning can carry a value it saw into a literal in that SQL. Treat `eval run` output as
+carrying the same sensitivity as the queries themselves when you decide who can read your CI logs.
+
+```bash
 
 # Probe questions through the text-to-SQL agent using the local ontology files
 # (one full agent run per question, expect ~30-90s each; repeat -q for several):
@@ -197,7 +208,7 @@ cassis ontology fmt --check
 | ---- | ------------------------------------------------------------------------------ |
 | 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
 | 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed; add-case: duplicate question or gold SQL that does not run; delete-case: no such case in the project; issues: no such issue or occurrence in the project; issues analyze: failed or cancelled analysis run; schema plan/apply: the plan failed (unparseable or truncated DDL), is stale or expired, the apply failed, or the project won't accept it (a plan is being applied, a DDL was given for a warehouse-connected project, or --warehouse for a DDL-only one)) |
-| 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
+| 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits, `eval run --branch` naming an ontology branch the project does not have) |
 | 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another eval run or issue analysis already active, out of credits, or `--timeout` reached |
 
 Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/README.md

@@ -82,6 +82,17 @@ cassis eval run --project ... --case 019f0000-0000-7000-8000-0000000000ca
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+```
+
+Under each failed case `eval run` prints what is needed to diagnose it: the SQL the agent
+generated, the gold SQL it was compared against, how many rows each side returned, any concepts
+the agent found missing, and the judge's reasoning when a judge graded the case. The expected and
+actual row *values* are deliberately not printed — they are in `--json` and in the Cassis app.
+Note that the **generated SQL is printed as the agent wrote it**, and an agent that read your data
+while planning can carry a value it saw into a literal in that SQL. Treat `eval run` output as
+carrying the same sensitivity as the queries themselves when you decide who can read your CI logs.
+
+```bash
 
 # Probe questions through the text-to-SQL agent using the local ontology files
 # (one full agent run per question, expect ~30-90s each; repeat -q for several):
@@ -175,7 +186,7 @@ cassis ontology fmt --check
 | ---- | ------------------------------------------------------------------------------ |
 | 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
 | 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed; add-case: duplicate question or gold SQL that does not run; delete-case: no such case in the project; issues: no such issue or occurrence in the project; issues analyze: failed or cancelled analysis run; schema plan/apply: the plan failed (unparseable or truncated DDL), is stale or expired, the apply failed, or the project won't accept it (a plan is being applied, a DDL was given for a warehouse-connected project, or --warehouse for a DDL-only one)) |
-| 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
+| 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits, `eval run --branch` naming an ontology branch the project does not have) |
 | 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another eval run or issue analysis already active, out of credits, or `--timeout` reached |
 
 Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/cassis_cli/api.py

@@ -106,6 +106,21 @@ class EvalRunActiveError(ApiError):
     """Another eval run is already active for the project (409)."""
 
 
+class EvalBranchNotFoundError(ApiError):
+    """`--branch` named an ontology branch the project does not have (404)."""
+
+
+def _is_branch_not_found(detail: object) -> bool:
+    """Whether a 404 detail is the start endpoint's missing-branch message.
+
+    Wire contract with `start_eval_run` in backend `endpoints/ci/eval.py`,
+    matched the way `delete_eval_case` matches its own 404 text. A server that
+    words it differently falls through to the generic project-scope error, so an
+    older server degrades rather than misreporting.
+    """
+    return isinstance(detail, str) and detail.startswith("Branch ") and detail.endswith(" not found")
+
+
 def _project_scope_error(response: httpx.Response) -> ApiError:
     # Surface the server's own message when it names the missing resource
     # (e.g. "Branch 'x' not found") — the generic hint covers the rest.
@@ -544,6 +559,11 @@ def post_eval_run_start(
             "An eval run is already active for this project — wait for it to finish or cancel it "
             "(in the webapp's Evals page, or with the run id printed when it was started)."
         )
+    if response.status_code == 404 and _is_branch_not_found(_detail_or_text(response)):
+        # Distinct from a project-scope 404: the key and project are fine, the
+        # branch name is not, and the generic copy sends the user to check
+        # permissions instead.
+        raise EvalBranchNotFoundError(_detail_or_text(response))
     if response.status_code in (403, 404):
         raise _project_scope_error(response)
     if response.status_code >= 400:

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/cassis_cli/eval.py

@@ -14,6 +14,7 @@ from cassis_cli.api import (
     DEFAULT_API_URL,
     ApiError,
     AuthError,
+    EvalBranchNotFoundError,
     EvalCaseExistsError,
     EvalCaseGoldSqlError,
     EvalCaseNotFoundError,
@@ -56,6 +57,15 @@ _STATUS_COLORS = {
     "plan_unexecutable": typer.colors.YELLOW,
 }
 
+# Caps on the evidence printed under a failed case. Bounded so a suite of
+# failures stays readable in a CI log, and always announced when they bite.
+_MAX_ERROR_CHARS = 200
+_MAX_REASONING_CHARS = 500
+_MAX_SQL_LINES = 20
+# A line cap alone bounds nothing: a gold SQL written as one long line in a YAML
+# case is a single line of any size, printed twice per failing case.
+_MAX_SQL_CHARS = 2000
+
 
 def _run_page_url(app_url: str, project_id: str, run_id: str) -> str:
     """Webapp URL of the run's detail view (Evals page, runs tab)."""
@@ -118,6 +128,84 @@ def _print_validation_failure(detail: object, base_path: str) -> None:
         typer.secho(str(detail), fg=typer.colors.RED, err=True)
 
 
+def _shorten(text: str, limit: int) -> str:
+    """Shorten to `limit` characters and say so. A silent cut can drop the cause."""
+    text = " ".join(text.split())
+    if len(text) <= limit:
+        return text
+    return f"{text[:limit]}... (truncated, {len(text)} chars)"
+
+
+def _print_sql(label: str, sql: str) -> None:
+    """Print SQL under a failed case, bounded on both lines and characters."""
+    lines = sql.strip().splitlines()
+    typer.echo(f"      {label}:")
+    budget = _MAX_SQL_CHARS
+    shown = 0
+    for line in lines[:_MAX_SQL_LINES]:
+        if budget <= 0:
+            break
+        if len(line) > budget:
+            typer.echo(f"        {line[:budget]}... (truncated, {len(line)} chars)")
+            budget = 0
+        else:
+            typer.echo(f"        {line}")
+            budget -= len(line)
+        shown += 1
+    if shown < len(lines):
+        typer.echo(f"        ... ({len(lines) - shown} more lines)")
+
+
+def _print_failure_evidence(r: dict[str, Any]) -> None:
+    """Print what a failed case needs to be diagnosed here, without row values.
+
+    Which field carries the reason depends on the status: a missing-concept
+    failure explains itself through `missing_concepts`, a judge rejection
+    through `judge_reasoning`, a data mismatch through `error` and the row
+    counts. Printing only `error` — as this did — showed nothing at all for the
+    first two.
+
+    Expected and actual row values are deliberately not printed. They ride in
+    `--json` and the webapp instead: `cassis verify` output is a CI job log, and
+    warehouse values in a job log travel further than whoever reads it intends.
+
+    Every field is read with `.get`, so an older server that returns none of
+    them degrades to the error line alone.
+    """
+    error = r.get("error")
+    if error:
+        typer.echo(f"      {_shorten(str(error), _MAX_ERROR_CHARS)}")
+
+    names = [str(c["name"]) for c in (r.get("missing_concepts") or []) if isinstance(c, dict) and c.get("name")]
+    if names:
+        typer.echo(f"      Missing concepts: {', '.join(names)}")
+
+    if r.get("judge_verdict"):
+        typer.echo(f"      Judge verdict: {r['judge_verdict']}")
+    if r.get("judge_reasoning"):
+        typer.echo(f"      Judge reasoning: {_shorten(str(r['judge_reasoning']), _MAX_REASONING_CHARS)}")
+
+    expected, actual = r.get("expected_row_count"), r.get("actual_row_count")
+    if expected is not None or actual is not None:
+        shown_actual = "-" if actual is None else str(actual)
+        shown_expected = "-" if expected is None else str(expected)
+        typer.echo(f"      Rows: {shown_actual} actual vs {shown_expected} expected")
+
+    # The agent's own words, for the statuses that carry no other explanation
+    # (a conversational reply instead of a plan, for one).
+    content = r.get("content")
+    if content and not error and not names and not r.get("judge_reasoning"):
+        typer.echo(f"      {_shorten(str(content), _MAX_ERROR_CHARS)}")
+
+    if r.get("generated_sql"):
+        _print_sql("Generated SQL", str(r["generated_sql"]))
+    if r.get("gold_sql"):
+        label = "Gold SQL"
+        if r.get("gold_sql_from_live_case"):
+            label = "Gold SQL (the case as it stands now; this result predates snapshotting)"
+        _print_sql(label, str(r["gold_sql"]))
+
+
 def _print_results_table(results: list[dict[str, Any]]) -> None:
     for r in sorted(results, key=lambda r: (r.get("status") == _PASSED, r.get("question") or "")):
         status = r.get("status", "?")
@@ -129,8 +217,9 @@ def _print_results_table(results: list[dict[str, Any]]) -> None:
             question = question[:67] + "..."
         line = f"  {icon} {status:<17} {duration:>5}  {question}"
         typer.secho(line, fg=color)
-        if r.get("error"):
-            typer.echo(f"      {str(r['error'])[:200]}")
+        if status == _PASSED:
+            continue
+        _print_failure_evidence(r)
 
 
 def _print_summary(run: dict[str, Any]) -> None:
@@ -448,7 +537,13 @@ def run(
     named case(s) run — e.g. proving one fresh `add-case` in seconds instead of
     rerunning the whole suite. Exits 0 when the run completes with every case
     passed, 1 on any failed case / failed run / invalid tree, 2 on usage
-    errors, 3 on transport errors or --timeout.
+    errors (including a --branch the project does not have), 3 on transport
+    errors or --timeout.
+
+    Failed cases print the generated SQL as the agent wrote it. Row values are
+    never printed (they are in --json), but an agent that read your data while
+    planning can carry a value it saw into a SQL literal — so treat this output
+    as sensitive as the queries themselves.
     """
     api_key = require_api_key(api_key)
     for case_id in case or []:
@@ -485,6 +580,17 @@ def run(
     except EvalStartValidationError as exc:
         _print_validation_failure(exc.detail, base_path)
         raise typer.Exit(EXIT_VALIDATION_FAILED) from exc
+    except EvalBranchNotFoundError as exc:
+        # Bad input, not a transport or permissions problem — so exit 2, and say
+        # what --branch actually takes instead of pointing at the API key.
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        typer.echo(
+            "--branch runs against an ontology branch that already exists in Cassis "
+            "(create it in the webapp, or push it with `cassis ontology push`). "
+            "To label a run after your local checkout's branch, use --label instead.",
+            err=True,
+        )
+        raise typer.Exit(EXIT_USAGE) from exc
     except ApiError as exc:  # covers AuthError and EvalRunActiveError too
         typer.secho(str(exc), fg=typer.colors.RED, err=True)
         raise typer.Exit(EXIT_TRANSPORT) from exc

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/cassis_cli/guide.py

@@ -31,7 +31,7 @@ GUIDE_FILENAME = "AGENTS.md"
 # Monotonic version of the doctrine text below. Bump it whenever
 # ontology_design_guide.md changes (a backend test enforces the pairing) — it
 # is what lets an older writer recognize a newer guide and leave it alone.
-DOCTRINE_VERSION = 6
+DOCTRINE_VERSION = 7
 
 # Must stay byte-identical to backend/app/services/ontology_guide.py::_BANNER —
 # the server-side git export writes the same file, and differing banners would

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/cassis_cli/ontology_design_guide.md

@@ -107,7 +107,7 @@ Put each fact at the lowest layer that fully owns it:
 
 ```
 column synonym / column description      single-column facts: meaning, encoding, unit, date format, nulls
-  → table description                    single-table facts: grain, scope, mandatory filters, "use this not that"
+  → table description                    single-table facts: scope, mandatory filters, "use this not that"
     → domain context_md                  cross-table routing, disambiguation, vocabulary spanning tables
       → root context                     truly global rules only: corporate identity, entity hierarchy, global conventions
 ```
@@ -134,13 +134,18 @@ otherwise push it down.
   Per-table facts go on that table's `description`; keep only genuinely
   cross-table routing in `context_md`.
 - Metrics are surfaced with their full definition → don't re-paste a formula in
-  prose. Point to a metric by name **only** when it lives in a *different* domain
+  prose, and that includes the metric's **own** `description` and `notes` (§8).
+  Point to a metric by name **only** when it lives in a *different* domain
   than the prose discussing it; a domain's own metrics are surfaced alongside its
   `context_md` already.
 - Joins are surfaced with their condition and cardinality → don't enumerate join
   keys in prose. The only join content that belongs in prose is a join+filter
   *recipe* no single structured join can express.
 
+**Short or empty beats redundant.** When the structured fields already say
+everything worth saying, leave the prose field short or empty. An empty
+description costs nothing; a redundant one goes stale.
+
 ---
 
 ## 4. Domains and `context_md`
@@ -188,9 +193,9 @@ domain. Everything domain-specific moves down.
 
 A table's `description` answers *"what is this, and what must the SQL agent know
 to use it correctly?"* — business meaning **plus** the correctness facts (scope,
-mandatory filters, gotchas). Always **state the grain explicitly** ("one row per
-order"; "one row per (user_id, day)") so the agent aggregates correctly, and set
-the `grain` field to the identifying column(s). Don't enumerate the columns — the
+mandatory filters, gotchas). The grain goes in the `grain` field — the identifying
+column(s) — not in the description: the agent reads it structurally, and "one row
+per order" in prose is a second copy. Don't enumerate the columns either — the
 column list is surfaced automatically.
 
 Use `synonyms` for alternate names a user might say ("purchases", "transactions"),
@@ -219,10 +224,10 @@ in the `sql` per §7.
 ## 6. Columns
 
 Column enrichment is `description`, `unit`, and `synonyms` — you cannot add or
-rename columns (they come from introspection). The `description` is the SQL
-agent's only per-column guidance, so it carries both the semantic meaning and the
-SQL-correctness facts: possible values, encoding, units, date format, null
-behavior, any mandatory filter.
+rename columns (they come from introspection). The agent sees a column's name,
+type, `unit` and `synonyms` structurally; the `description` carries what those
+don't: meaning, possible values, encoding, date format, null behavior, any
+mandatory filter.
 
 - **Prioritize the trap columns.** Disambiguate similarly-named columns
   (`created_at` vs `updated_at`, `amount` vs `net_amount`), numeric-encoded
@@ -232,6 +237,14 @@ behavior, any mandatory filter.
 - **Set `unit`** whenever a number is a currency, percentage, or duration —
   aggregation correctness depends on it.
 - **Use `synonyms`** for 1–3 short, plain alternate terms a user might use.
+- **Facts, not SQL.** Say what the column holds and how it is encoded ("TRUE =
+  the account can sign in"), never a fragment to paste ("filter with
+  `is_active = TRUE`", "count people with `COUNT(DISTINCT id)`"). Writing the SQL
+  is the agent's job.
+- **Relationships live in joins.** A foreign-key column says what it identifies
+  ("the owning organization"), not which table it references or on what
+  condition; the join carries that, and a description repeating it drifts when
+  the join changes.
 - **Keep the join key queryable.** A foreign-key column stays a real, described
   column even when a join covers the relationship — hiding it leaves the agent
   unable to filter or group on it without traversing the join. Test: imagine a
@@ -312,12 +325,33 @@ unit: euro cents
 synonyms:
 - net sales
 - revenue net of refunds
-description: Gross order amount minus refunds, excluding cancelled and refunded orders.
+description: Revenue kept after refunds; the top-line sales figure.
 ```
 
 A metric definition beats a prose "revenue = net revenue excluding tax" note —
 the metric is surfaced structurally and can be reused; the prose drifts.
 
+**Definition in the fields, meaning in the prose.** `expression`, `filters`,
+`table_schema`/`table_name` and `unit` *are* the definition. Every prose field —
+including the metric's own `description` and `notes` — says what the number means
+and when to use it, never how it is computed: no columns, thresholds, time windows,
+or numerator/denominator. A rule copied into prose is a second definition, and the
+two drift apart (one metric shipped with a 90-day window in `filters` and "30 days"
+in its own description). Business intent is fine ("draft bookings are not
+commitments"); its encoding (`status <> 'DRAFT'`) is not. This holds even when the
+metric's name *is* the term being defined ("active partners"): the reader sees
+`filters` right next to the description. Disambiguate from sibling metrics by the
+question each answers, never by restating their windows or filters. If the fields
+say it all, keep the description to a phrase or leave it empty.
+
+```yaml
+# Bad — restates the definition
+description: Active partners = COUNT(DISTINCT partner_id) with a non-cancelled
+  booking in the last 90 days.
+# Good — what the number is for
+description: Partners still trading; the activity count used for outreach targets.
+```
+
 ---
 
 ## 9. Identifiers and case — where SQL silently breaks
@@ -351,7 +385,9 @@ description, so a wrong example is worse than none.
 
 - **Use real stored enum values verbatim.** Never hand-wave a list with "etc." —
   the agent reads `etc.` as license to invent values. If you don't have the exact
-  values, omit the list rather than approximate it. When two columns supposedly
+  values, omit the list rather than approximate it. A `DISTINCT` over the column
+  *is* the list of stored values: write it plainly ("Stored values: 'EDITOR'"),
+  with no "observed so far" hedge and no counts. When two columns supposedly
   share an enum, verify they're identical before encoding both.
 - **Be wary of analogies to external products** — they mislead the moment the
   real mechanic diverges.
@@ -390,6 +426,14 @@ description, so a wrong example is worse than none.
 - **Ground writes in the real schema.** Before referencing a physical table or
   column, confirm it exists. Don't invent tables or columns; the physical schema
   is fixed.
+- **Never persist a measurement.** Prose holds stable encodings and business
+  definitions, not what the data shows today. Row counts, distinct counts, shares,
+  distributions, funnel rates and "as of" / "currently" / "observed" statements go
+  stale silently, and the answering agent repeats them as fact. Query the data to
+  *verify* a stable fact (stored enum values, a date format, null behaviour) and
+  write only the fact: "Stored values: 'EDITOR', 'ADMIN'", not "only 'EDITOR'
+  observed (6 of 6, 100%)". Complete value lists and constants the business fixes
+  (a 14-day trial) are facts, not measurements.
 - **Ask for the business meaning; don't guess.** A term like "active user" or
   "last month" has a project-specific definition (time window, criteria, filters).
   Encode the business's actual definition, not a textbook default.

{cassis_cli-2.0.0 → cassis_cli-2.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "cassis-cli"
-version = "2.0.0"
+version = "2.1.0"
 description = "Validate, test and evaluate your Cassis ontology from your terminal, then publish it"
 readme = "README.md"
 license = { text = "Apache-2.0" }