cassis-cli 0.3.0__tar.gz → 1.0.0__tar.gz

{cassis_cli-0.3.0 → cassis_cli-1.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cassis-cli
-Version: 0.3.0
+Version: 1.0.0
 Summary: Cassis CLI — run Cassis actions (ontology validation, upload and publish) from your CI pipelines
 License: Proprietary
 Keywords: cassis,ontology,ci,text-to-sql
@@ -23,9 +23,14 @@ Description-Content-Type: text/markdown
 Run Cassis actions from your CI pipelines:
 
 - `cassis ontology check` validates the ontology files in your repository with the exact same checks as the Cassis GitHub PR check (YAML parsing, round-trip, import validation) — so you can gate merges in any CI system, not just GitHub.
+- `cassis ontology fmt` rewrites the ontology files in canonical form (think `black`/`gofmt` for the ontology), so hand or agent edits pass the round-trip check.
 - `cassis ontology upload` uploads the ontology files to a Cassis project (full replace) and, by default, publishes them immediately as a new version — so a merge to your main branch can go live in one CI step.
 - `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local YAML files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` and `cassis ontology fmt` also write `<base-path>/AGENTS.md`, the Cassis ontology modeling guide, into the checkout (default `cassis/AGENTS.md`) — a managed file (generated banner; the CLI overwrites local edits) so a repo-aware coding agent loads current Cassis modeling doctrine by convention. It sits inside the ontology directory but is not part of the ontology tree (the CLI reads only `*.yml`/`*.yaml`), so it is never uploaded, validated, or pruned. Commit it alongside your ontology changes. The guide text ships inside the CLI package, so its version tracks the **installed cassis-cli version** — upgrade the CLI (`pip install -U cassis-cli`) and re-run `fmt` to pick up doctrine updates; an unpinned `pip install cassis-cli` in CI gets them automatically. The banner stamps a doctrine version, and the CLI never *downgrades* the file: if the checkout's `AGENTS.md` was written by a newer doctrine (a newer CLI, or Cassis itself on a publish), `fmt`/`pull` leave it in place, print an upgrade notice, and `fmt --check` still passes.
+- The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version — when you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local ontology files (scored in-memory — nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
+- `cassis ontology test` runs individual questions through the text-to-SQL agent using your local ontology files, so you can check that a change actually works (e.g. a new column gets picked) — where `eval run` only checks for regressions on existing eval cases.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite — after fixing an ontology issue, add the question users were failing on so `eval run` guards it from regressing.
 
 ## Install
 
@@ -76,6 +81,14 @@ cassis eval run --project ... --branch feature-x
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+
+# Probe questions through the text-to-SQL agent using the local ontology files
+# (one full agent run per question, expect ~30-90s each; repeat -q for several):
+cassis ontology test --project ... -q "How much was refunded last month?" -q "Net revenue in Q1?"
+
+# Add a gold case to the eval suite (rejected if the exact question already exists):
+cassis eval add-case --project ... -q "How much was refunded last month?" \
+  --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..."
 ```
 
 Configuration (flags take precedence over env vars):
@@ -85,7 +98,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | — (required)                |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis` — must match the project's git-sync "Path" setting |
-| `--project` (pull, upload, eval run) | `CASSIS_PROJECT_ID` | — (required)      |
+| `--project` (pull, upload, eval run, eval add-case, test) | `CASSIS_PROJECT_ID` | — (required)      |
 
 `cassis eval run` also accepts `--label` (run label in the Evals page; defaults
 to the branch name from the CI environment or the local git checkout; rejected
@@ -95,16 +108,30 @@ and Ctrl-C cancels the run (exit 130). It prints a deep link to the run's page
 in the Evals UI; `--app-url` / `CASSIS_APP_URL` overrides the link's base URL
 when the webapp is not served from the API host (defaults to `--api-url`).
 
+### Formatting
+
+```bash
+# Rewrite the ontology files in canonical form (in place)
+cassis ontology fmt
+
+# CI mode: fail (exit 1) if any file is not canonical, write nothing
+cassis ontology fmt --check
+```
+
+`fmt` uses the exact serializer the validation round-trip compares against, so a formatted tree cannot fail that stage. Formatting does not run import validation — `check` remains the pass/fail gate for semantic problems (dangling references, incomplete metrics).
+
+**Review the diff before committing**: canonical form keeps exactly the fields Cassis understands. Unknown fields (typos) are dropped — the rewrite makes them visible in `git diff` instead of losing them silently at sync time. Files with duplicate YAML keys are rejected (fix them by hand: the formatter can't know which value you meant).
+
 ### Exit codes
 
 | Code | Meaning                                                                        |
 | ---- | ------------------------------------------------------------------------------ |
-| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) |
-| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run) |
+| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
+| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed) |
 | 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
-| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, or `--timeout` reached |
+| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, out of credits, or `--timeout` reached |
 
-Commands that send the local tree (`check`, `upload`, `eval run`) accept up to
+Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to
 2000 YAML files / 5 MB total — far above real ontologies (a few hundred small
 files). Beyond that the CLI fails fast with exit 2 before uploading anything;
 double-check `--base-path` if you hit it.

{cassis_cli-0.3.0 → cassis_cli-1.0.0}/README.md

@@ -3,9 +3,14 @@
 Run Cassis actions from your CI pipelines:
 
 - `cassis ontology check` validates the ontology files in your repository with the exact same checks as the Cassis GitHub PR check (YAML parsing, round-trip, import validation) — so you can gate merges in any CI system, not just GitHub.
+- `cassis ontology fmt` rewrites the ontology files in canonical form (think `black`/`gofmt` for the ontology), so hand or agent edits pass the round-trip check.
 - `cassis ontology upload` uploads the ontology files to a Cassis project (full replace) and, by default, publishes them immediately as a new version — so a merge to your main branch can go live in one CI step.
 - `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local YAML files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` and `cassis ontology fmt` also write `<base-path>/AGENTS.md`, the Cassis ontology modeling guide, into the checkout (default `cassis/AGENTS.md`) — a managed file (generated banner; the CLI overwrites local edits) so a repo-aware coding agent loads current Cassis modeling doctrine by convention. It sits inside the ontology directory but is not part of the ontology tree (the CLI reads only `*.yml`/`*.yaml`), so it is never uploaded, validated, or pruned. Commit it alongside your ontology changes. The guide text ships inside the CLI package, so its version tracks the **installed cassis-cli version** — upgrade the CLI (`pip install -U cassis-cli`) and re-run `fmt` to pick up doctrine updates; an unpinned `pip install cassis-cli` in CI gets them automatically. The banner stamps a doctrine version, and the CLI never *downgrades* the file: if the checkout's `AGENTS.md` was written by a newer doctrine (a newer CLI, or Cassis itself on a publish), `fmt`/`pull` leave it in place, print an upgrade notice, and `fmt --check` still passes.
+- The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version — when you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local ontology files (scored in-memory — nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
+- `cassis ontology test` runs individual questions through the text-to-SQL agent using your local ontology files, so you can check that a change actually works (e.g. a new column gets picked) — where `eval run` only checks for regressions on existing eval cases.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite — after fixing an ontology issue, add the question users were failing on so `eval run` guards it from regressing.
 
 ## Install
 
@@ -56,6 +61,14 @@ cassis eval run --project ... --branch feature-x
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+
+# Probe questions through the text-to-SQL agent using the local ontology files
+# (one full agent run per question, expect ~30-90s each; repeat -q for several):
+cassis ontology test --project ... -q "How much was refunded last month?" -q "Net revenue in Q1?"
+
+# Add a gold case to the eval suite (rejected if the exact question already exists):
+cassis eval add-case --project ... -q "How much was refunded last month?" \
+  --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..."
 ```
 
 Configuration (flags take precedence over env vars):
@@ -65,7 +78,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | — (required)                |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis` — must match the project's git-sync "Path" setting |
-| `--project` (pull, upload, eval run) | `CASSIS_PROJECT_ID` | — (required)      |
+| `--project` (pull, upload, eval run, eval add-case, test) | `CASSIS_PROJECT_ID` | — (required)      |
 
 `cassis eval run` also accepts `--label` (run label in the Evals page; defaults
 to the branch name from the CI environment or the local git checkout; rejected
@@ -75,16 +88,30 @@ and Ctrl-C cancels the run (exit 130). It prints a deep link to the run's page
 in the Evals UI; `--app-url` / `CASSIS_APP_URL` overrides the link's base URL
 when the webapp is not served from the API host (defaults to `--api-url`).
 
+### Formatting
+
+```bash
+# Rewrite the ontology files in canonical form (in place)
+cassis ontology fmt
+
+# CI mode: fail (exit 1) if any file is not canonical, write nothing
+cassis ontology fmt --check
+```
+
+`fmt` uses the exact serializer the validation round-trip compares against, so a formatted tree cannot fail that stage. Formatting does not run import validation — `check` remains the pass/fail gate for semantic problems (dangling references, incomplete metrics).
+
+**Review the diff before committing**: canonical form keeps exactly the fields Cassis understands. Unknown fields (typos) are dropped — the rewrite makes them visible in `git diff` instead of losing them silently at sync time. Files with duplicate YAML keys are rejected (fix them by hand: the formatter can't know which value you meant).
+
 ### Exit codes
 
 | Code | Meaning                                                                        |
 | ---- | ------------------------------------------------------------------------------ |
-| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) |
-| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run) |
+| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
+| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed) |
 | 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
-| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, or `--timeout` reached |
+| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, out of credits, or `--timeout` reached |
 
-Commands that send the local tree (`check`, `upload`, `eval run`) accept up to
+Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to
 2000 YAML files / 5 MB total — far above real ontologies (a few hundred small
 files). Beyond that the CLI fails fast with exit 2 before uploading anything;
 double-check `--base-path` if you hit it.

{cassis_cli-0.3.0 → cassis_cli-1.0.0}/cassis_cli/__init__.py

@@ -1,3 +1,3 @@
 """Cassis CLI — run Cassis actions from your CI pipelines."""
 
-__version__ = "0.3.0"
+__version__ = "1.0.0"

{cassis_cli-0.3.0 → cassis_cli-1.0.0}/cassis_cli/api.py

@@ -2,13 +2,73 @@
 
 from __future__ import annotations
 
+import sys
 from typing import Any, Optional
 
 import httpx
+from cassis_cli import __version__
 
 DEFAULT_API_URL = "https://app.getcassis.com"
 TIMEOUT_SECONDS = 60.0
 
+USER_AGENT = f"cassis-cli/{__version__}"
+
+# Response header the CI endpoints set to the newest cassis-cli on PyPI.
+LATEST_VERSION_HEADER = "x-cassis-cli-latest"
+
+_upgrade_notice_shown = False
+
+
+def _version_tuple(version: str) -> Optional[tuple[int, ...]]:
+    try:
+        return tuple(int(part) for part in version.strip().split("."))
+    except ValueError:
+        return None
+
+
+def _maybe_print_upgrade_notice(response: httpx.Response) -> None:
+    """Print a one-time stderr notice when the server advertises a newer CLI.
+
+    Purely informational — never changes behavior or exit codes. Staying
+    current matters beyond bugfixes: the ontology modeling guide written to
+    ``cassis/AGENTS.md`` ships inside this package, so an old CLI keeps old
+    doctrine in the repo.
+    """
+    global _upgrade_notice_shown
+    if _upgrade_notice_shown:
+        return
+    latest = response.headers.get(LATEST_VERSION_HEADER)
+    if not latest:
+        return
+    mine, theirs = _version_tuple(__version__), _version_tuple(latest)
+    if mine is None or theirs is None:
+        return
+    # Zero-pad to equal length so "0.6" == "0.6.0" (same rule as the webapp's
+    # Agent setup page — the comparison logic exists on both surfaces).
+    width = max(len(mine), len(theirs))
+    if theirs + (0,) * (width - len(theirs)) <= mine + (0,) * (width - len(mine)):
+        return
+    _upgrade_notice_shown = True
+    print(
+        f"notice: cassis-cli {latest} is available (you have {__version__}) — "
+        "run `pip install -U cassis-cli`, then `cassis ontology fmt` to refresh cassis/AGENTS.md.",
+        file=sys.stderr,
+    )
+
+
+def _client(*, timeout: float = TIMEOUT_SECONDS, transport: Optional[httpx.BaseTransport] = None) -> httpx.Client:
+    """Build the HTTP client every API call goes through.
+
+    Identifies the CLI to the server (User-Agent) and watches responses for
+    the newer-version advertisement.
+    """
+    return httpx.Client(
+        timeout=timeout,
+        transport=transport,
+        headers={"User-Agent": USER_AGENT},
+        event_hooks={"response": [_maybe_print_upgrade_notice]},
+    )
+
 
 class ApiError(Exception):
     """Transport or HTTP-level failure talking to the Cassis API."""
@@ -52,6 +112,14 @@ def _project_scope_error(response: httpx.Response) -> ApiError:
     )
 
 
+def _detail_or_text(response: httpx.Response) -> Any:
+    """Return the error response's ``detail`` (str or structured), falling back to its body."""
+    try:
+        return response.json().get("detail") or response.text[:500]
+    except ValueError:
+        return response.text[:500]
+
+
 def _parse_json_response(response: httpx.Response, url: str) -> Any:
     try:
         return response.json()
@@ -71,7 +139,7 @@ def post_ontology_check(
     """POST the ontology tree to /api/ci/ontology-check and return the response body."""
     url = api_url.rstrip("/") + "/api/ci/ontology-check"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(
                 url,
                 json={"files": files},
@@ -110,7 +178,7 @@ def post_ontology_import(
     if label is not None:
         body["label"] = label
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(
                 url,
                 json=body,
@@ -122,11 +190,7 @@ def post_ontology_import(
     if response.status_code == 401:
         raise AuthError("The Cassis API rejected the API key (invalid or expired).")
     if response.status_code == 400:
-        try:
-            detail = response.json().get("detail") or response.text[:500]
-        except ValueError:
-            detail = response.text[:500]
-        raise UploadValidationError(str(detail))
+        raise UploadValidationError(str(_detail_or_text(response)))
     if response.status_code in (403, 404):
         raise _project_scope_error(response)
     if response.status_code >= 400:
@@ -149,7 +213,7 @@ def get_ontology_export(
     """GET /api/ci/projects/{project_id}/ontology/export and return the files tree."""
     url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/ontology/export"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.get(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -186,7 +250,7 @@ def post_eval_run_start(
     if label is not None:
         body["label"] = label
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(url, json=body, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -194,11 +258,7 @@ def post_eval_run_start(
     if response.status_code == 401:
         raise AuthError("The Cassis API rejected the API key (invalid or expired).")
     if response.status_code == 400:
-        try:
-            detail = response.json().get("detail") or response.text[:500]
-        except ValueError:
-            detail = response.text[:500]
-        raise EvalStartValidationError(detail)
+        raise EvalStartValidationError(_detail_or_text(response))
     if response.status_code == 409:
         raise EvalRunActiveError(
             "An eval run is already active for this project — wait for it to finish or cancel it "
@@ -214,10 +274,55 @@ def post_eval_run_start(
     return result
 
 
+class EvalCaseExistsError(ApiError):
+    """The project already has an eval case with this exact question."""
+
+
+class EvalCaseGoldSqlError(ApiError):
+    """The API rejected the gold SQL (400): it does not run against the project's data source."""
+
+
+def post_eval_case_create(
+    *,
+    api_url: str,
+    api_key: str,
+    project_id: str,
+    question: str,
+    gold_sql: str,
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """POST to /api/ci/projects/{project_id}/eval/cases and return the created case."""
+    url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/eval/cases"
+    try:
+        with _client(transport=transport) as client:
+            response = client.post(
+                url,
+                json={"question": question, "gold_sql": gold_sql},
+                headers={"Authorization": f"Bearer {api_key}"},
+            )
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    if response.status_code == 409:
+        raise EvalCaseExistsError(str(_detail_or_text(response)))
+    if response.status_code == 400:
+        raise EvalCaseGoldSqlError(str(_detail_or_text(response)))
+    if response.status_code in (403, 404):
+        raise _project_scope_error(response)
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if not isinstance(result, dict) or not all(key in result for key in ("id", "question", "gold_sql")):
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result
+
+
 def _get_eval_json(url: str, api_key: str, transport: Optional[httpx.BaseTransport]) -> Any:
     """GET an eval-run URL with the shared error mapping."""
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.get(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -274,7 +379,7 @@ def post_eval_run_cancel(
     """POST /api/ci/projects/{project_id}/eval/runs/{run_id}/cancel."""
     url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/eval/runs/{run_id}/cancel"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -284,3 +389,95 @@ def post_eval_run_cancel(
         raise _project_scope_error(response)
     if response.status_code >= 400:
         raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+
+
+def post_ontology_fmt(
+    *,
+    api_url: str,
+    api_key: str,
+    files: dict[str, str],
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """POST the ontology tree to /api/ci/ontology-fmt and return the response body."""
+    url = api_url.rstrip("/") + "/api/ci/ontology-fmt"
+    try:
+        with _client(transport=transport) as client:
+            response = client.post(
+                url,
+                json={"files": files},
+                headers={"Authorization": f"Bearer {api_key}"},
+            )
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if (
+        not isinstance(result, dict)
+        or "ok" not in result
+        or not isinstance(result.get("findings"), list)
+        or not isinstance(result.get("changed_paths"), list)
+        or not isinstance(result.get("removed_paths"), list)
+        or (result["ok"] and not isinstance(result.get("files"), dict))
+    ):
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result
+
+
+# The server-side probe budget is 300s; leave headroom for transport.
+ONTOLOGY_TEST_TIMEOUT_SECONDS = 330.0
+
+
+class OntologyTestValidationError(ApiError):
+    """The API rejected the ontology tree as invalid (400).
+
+    ``detail`` keeps the structured payload (``{"message", "findings"}``) for
+    display.
+    """
+
+    def __init__(self, detail: object) -> None:
+        super().__init__(str(detail))
+        self.detail = detail
+
+
+def post_ontology_test(
+    *,
+    api_url: str,
+    api_key: str,
+    project_id: str,
+    files: dict[str, str],
+    question: str,
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """POST to /api/ci/projects/{project_id}/ontology/test and return the probe outcome.
+
+    Blocks for the duration of the agent run (up to ~5 minutes server-side).
+    """
+    url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/ontology/test"
+    try:
+        with _client(timeout=ONTOLOGY_TEST_TIMEOUT_SECONDS, transport=transport) as client:
+            response = client.post(
+                url,
+                json={"files": files, "question": question},
+                headers={"Authorization": f"Bearer {api_key}"},
+            )
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    if response.status_code == 400:
+        raise OntologyTestValidationError(_detail_or_text(response))
+    if response.status_code == 402:
+        raise ApiError("Your organization has run out of credits. Contact your administrator to top up.")
+    if response.status_code in (403, 404):
+        raise _project_scope_error(response)
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if not isinstance(result, dict) or "status" not in result:
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result

{cassis_cli-0.3.0 → cassis_cli-1.0.0}/cassis_cli/eval.py

@@ -15,10 +15,13 @@ from cassis_cli.api import (
     DEFAULT_API_URL,
     ApiError,
     AuthError,
+    EvalCaseExistsError,
+    EvalCaseGoldSqlError,
     EvalRunActiveError,
     EvalStartValidationError,
     get_eval_run,
     get_eval_run_results,
+    post_eval_case_create,
     post_eval_run_cancel,
     post_eval_run_start,
 )
@@ -139,6 +142,79 @@ def _print_summary(run: dict[str, Any]) -> None:
     typer.echo(", ".join(parts))
 
 
+@app.command(name="add-case")
+def add_case(
+    project_id: str = typer.Option(
+        ...,
+        "--project",
+        envvar="CASSIS_PROJECT_ID",
+        help="Target Cassis project ID (UUID, shown in the project's URL).",
+    ),
+    question: str = typer.Option(
+        ...,
+        "-q",
+        "--question",
+        help="The natural-language question the case guards.",
+    ),
+    gold_sql: str = typer.Option(
+        ...,
+        "--gold-sql",
+        help="The correct SQL for the question; executed at run time to produce the expected output.",
+    ),
+    api_key: Optional[str] = typer.Option(
+        None,
+        "--api-key",
+        envvar="CASSIS_API_KEY",
+        help="Cassis API key (sk-k6-...). Create one in Organization settings -> API keys.",
+    ),
+    api_url: str = typer.Option(
+        DEFAULT_API_URL,
+        "--api-url",
+        envvar="CASSIS_API_URL",
+        help="Cassis API base URL.",
+    ),
+    json_output: bool = typer.Option(False, "--json", help="Print the created case as raw JSON."),
+) -> None:
+    """Add a gold test case to the project's eval suite.
+
+    Closes the loop after fixing an ontology issue: the question users were
+    failing on becomes a gold case, so `cassis eval run` guards it from
+    regressing. On an executable data source the gold SQL is run before the
+    case is stored, so a case that cannot execute never enters the suite.
+    Exits 0 on creation, 1 on a duplicate question or gold SQL that does not
+    run, 2 on usage errors, 3 on transport/API errors.
+    """
+    api_key = require_api_key(api_key)
+    try:
+        UUID(project_id)
+    except ValueError:
+        typer.secho(f"--project must be a project ID (UUID), got {project_id!r}.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+    if not question.strip() or not gold_sql.strip():
+        typer.secho("--question and --gold-sql must not be empty.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+
+    try:
+        case = post_eval_case_create(
+            api_url=api_url, api_key=api_key, project_id=project_id, question=question, gold_sql=gold_sql
+        )
+    except (EvalCaseExistsError, EvalCaseGoldSqlError) as exc:
+        typer.secho(str(exc), fg=typer.colors.YELLOW, err=True)
+        raise typer.Exit(EXIT_VALIDATION_FAILED) from exc
+    except AuthError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_TRANSPORT) from exc
+    except ApiError as exc:
+        typer.secho(str(exc), fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_TRANSPORT) from exc
+
+    if json_output:
+        typer.echo(json.dumps(case, indent=2))
+    else:
+        typer.secho(f"✓ Added eval case {case['id']}: {case['question']}", fg=typer.colors.GREEN)
+    raise typer.Exit(EXIT_OK)
+
+
 @app.command()
 def run(
     path: Path = typer.Argument(

cassis_cli-1.0.0/cassis_cli/guide.py

@@ -0,0 +1,97 @@
+"""The ontology modeling guide the CLI writes into a git-synced checkout.
+
+``ontology_design_guide.md`` in this package is a byte-for-byte copy of the
+canonical ``docs/ontology-design-guide.md`` (a repo-root test enforces they stay
+identical — the backend image doesn't ship ``docs/``, so the CLI carries its own
+copy). ``pull`` writes it into the checkout as ``<base_path>/AGENTS.md`` and
+``fmt`` keeps it canonical, so a repo-aware agent loads current Cassis modeling
+doctrine by convention. The file is managed: a banner marks it generated and the
+CLI overwrites local edits, exactly as ``fmt`` rewrites drifted ontology YAML.
+
+Two writers manage the file — this CLI and the Cassis server's git export — and
+they may run different doctrine versions (the guide ships inside each). The
+banner stamps its ``DOCTRINE_VERSION`` so an older CLI never *downgrades* a
+guide a newer writer produced: ``refresh_guide`` leaves a newer-stamped file in
+place and the CLI tells the user to upgrade instead.
+
+``AGENTS.md`` sits inside the ontology base path but is never part of the
+ontology tree — ``collect_files`` globs only ``*.yml`` / ``*.yaml``, so check,
+fmt-of-YAML, upload, and pull-prune all ignore it.
+"""
+
+from __future__ import annotations
+
+import re
+from importlib import resources
+from pathlib import Path
+from typing import Literal
+
+GUIDE_FILENAME = "AGENTS.md"
+
+# Monotonic version of the doctrine text below. Bump it whenever
+# ontology_design_guide.md changes (a backend test enforces the pairing) — it
+# is what lets an older writer recognize a newer guide and leave it alone.
+DOCTRINE_VERSION = 1
+
+# Must stay byte-identical to backend/app/services/ontology_guide.py::_BANNER —
+# the server-side git export writes the same file, and differing banners would
+# make the two writers churn the file against each other.
+_BANNER = (
+    "<!--\n"
+    f"Generated by Cassis — the ontology modeling guide (doctrine v{DOCTRINE_VERSION}).\n"
+    "Do NOT edit: Cassis and cassis-cli (`ontology pull` / `ontology fmt`) overwrite this file.\n"
+    "-->\n\n"
+)
+
+_DOCTRINE_VERSION_RE = re.compile(r"doctrine v(\d+)\b")
+
+GuideStatus = Literal["current", "stale", "newer", "missing"]
+
+
+def canonical_guide() -> str:
+    """Return the managed ``AGENTS.md`` content: banner + the packaged guide."""
+    body = resources.files("cassis_cli").joinpath("ontology_design_guide.md").read_text(encoding="utf-8")
+    return _BANNER + body
+
+
+def guide_status(ontology_dir: Path) -> GuideStatus:
+    """Classify ``<ontology_dir>/AGENTS.md`` against the guide this CLI carries.
+
+    - ``current``: byte-identical to what this CLI would write.
+    - ``newer``: stamped with a higher doctrine version (written by a newer
+      CLI or by the Cassis server) — must not be overwritten by this CLI.
+    - ``stale``: anything else that exists (older doctrine, local edits, or an
+      unversioned banner).
+    - ``missing``: absent or unreadable.
+    """
+    try:
+        existing = (ontology_dir / GUIDE_FILENAME).read_text(encoding="utf-8")
+    except OSError:
+        return "missing"
+    if existing == canonical_guide():
+        return "current"
+    # Only a leading banner comment carries the stamp — the guide body may
+    # legitimately mention "doctrine vN" as prose, and a hand-written file has
+    # no banner at all. Both must read as "stale", never as "newer" (which is
+    # never repaired and nags the user to upgrade a CLI that is already current).
+    if existing.lstrip().startswith("<!--") and "-->" in existing:
+        match = _DOCTRINE_VERSION_RE.search(existing.split("-->", 1)[0])
+        if match is not None and int(match.group(1)) > DOCTRINE_VERSION:
+            return "newer"
+    return "stale"
+
+
+def refresh_guide(ontology_dir: Path) -> GuideStatus:
+    """Bring ``<ontology_dir>/AGENTS.md`` up to this CLI's doctrine, never down.
+
+    Writes the managed guide when the file is ``stale`` or ``missing``; leaves
+    ``current`` and ``newer`` files untouched. Returns the status found before
+    writing, so the caller can report what happened (and nudge an upgrade on
+    ``newer``). Raises OSError on a local write failure (the caller maps it to
+    a usage-level exit).
+    """
+    status = guide_status(ontology_dir)
+    if status in ("stale", "missing"):
+        ontology_dir.mkdir(parents=True, exist_ok=True)
+        (ontology_dir / GUIDE_FILENAME).write_text(canonical_guide(), encoding="utf-8")
+    return status