cassis-cli 0.4.0__tar.gz → 1.1.0__tar.gz

{cassis_cli-0.4.0 → cassis_cli-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cassis-cli
-Version: 0.4.0
+Version: 1.1.0
 Summary: Cassis CLI — run Cassis actions (ontology validation, upload and publish) from your CI pipelines
 License: Proprietary
 Keywords: cassis,ontology,ci,text-to-sql
@@ -25,8 +25,12 @@ Run Cassis actions from your CI pipelines:
 - `cassis ontology check` validates the ontology files in your repository with the exact same checks as the Cassis GitHub PR check (YAML parsing, round-trip, import validation) — so you can gate merges in any CI system, not just GitHub.
 - `cassis ontology fmt` rewrites the ontology files in canonical form (think `black`/`gofmt` for the ontology), so hand or agent edits pass the round-trip check.
 - `cassis ontology upload` uploads the ontology files to a Cassis project (full replace) and, by default, publishes them immediately as a new version — so a merge to your main branch can go live in one CI step.
-- `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local YAML files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local ontology files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` and `cassis ontology fmt` also write `<base-path>/AGENTS.md`, the Cassis ontology modeling guide, into the checkout (default `cassis/AGENTS.md`) — a managed file (generated banner; the CLI overwrites local edits) so a repo-aware coding agent loads current Cassis modeling doctrine by convention. It sits inside the ontology directory but is not part of the ontology tree (which is the YAML files plus the domain Markdown files `domains/**/README.md`), so it is never uploaded, validated, or pruned. Commit it alongside your ontology changes. The guide text ships inside the CLI package, so its version tracks the **installed cassis-cli version** — upgrade the CLI (`pip install -U cassis-cli`) and re-run `fmt` to pick up doctrine updates; an unpinned `pip install cassis-cli` in CI gets them automatically. The banner stamps a doctrine version, and the CLI never *downgrades* the file: if the checkout's `AGENTS.md` was written by a newer doctrine (a newer CLI, or Cassis itself on a publish), `fmt`/`pull` leave it in place, print an upgrade notice, and `fmt --check` still passes.
+- The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version — when you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local ontology files (scored in-memory — nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
+- `cassis ontology test` runs individual questions through the text-to-SQL agent using your local ontology files, so you can check that a change actually works (e.g. a new column gets picked) — where `eval run` only checks for regressions on existing eval cases.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite — after fixing an ontology issue, add the question users were failing on so `eval run` guards it from regressing.
 
 ## Install
 
@@ -34,11 +38,21 @@ Run Cassis actions from your CI pipelines:
 pip install cassis-cli
 ```
 
+## Ontology file format
+
+The ontology tree under `<base-path>` (default `cassis/`) is:
+
+- **Project identity** — `project.yml`: the Cassis project id and format version. Written by `pull` and by server-side publish (the contexts that know the id); a local `fmt` won't create it.
+- **Domains** — Markdown files: every domain is the `README.md` of its folder — `domains/README.md` for the root, `domains/<path>/README.md` for each sub-domain. Each has a small YAML frontmatter block (`type`, `title`, `description`) and a Markdown body carrying the domain's `context_md`; a generated section at the bottom links the domain's tables and metrics (kept current by `fmt`/`pull` — edit your prose above it, and the PR check fails if the links are stale, so re-run `fmt`). The layout is a Cassis profile inspired by [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf): the files render on GitHub and read in any Markdown editor, but Cassis validates them strictly (unknown keys are flagged, not preserved).
+- **Tables, joins, metrics** — YAML, unchanged: `tables/<schema>/<table>.yml`, `joins.yml`, `metrics/<name>.yml`.
+
+**Migrating an existing repo** (domains were YAML `_project.yml` / `_domain.yml` before cassis-cli 1.1.0): upgrade and run `cassis ontology fmt` (or `cassis ontology pull` if you have no local edits) — it rewrites the domain files to Markdown and removes the old ones. Review the diff and commit. Cassis reads the old YAML domain files too, so an un-migrated repo keeps working until you convert it. **Uploading requires cassis-cli ≥ 1.1.0** — the server rejects an older CLI (which would drop the Markdown domain files) with a clear upgrade error.
+
 ## Setup
 
 1. Create an API key in Cassis under **Organization settings → API keys** (keys start with `sk-k6-`).
 2. Store it as a CI secret and expose it as `CASSIS_API_KEY`.
-3. For `pull`, `upload` and `eval run`: find the project ID (UUID) in the project's URL and expose it as `CASSIS_PROJECT_ID` (or pass `--project`).
+3. For `pull`, `upload`, `eval run`, `ontology test`, and `eval add-case`: the project ID (UUID) is taken from `<base-path>/project.yml` in the checkout (written by `pull` and by publishing) — so once a repo is pulled you don't need to pass it. To override, or before the first pull, set `CASSIS_PROJECT_ID` or pass `--project` (find the UUID in the project's URL).
 
 ## Usage
 
@@ -77,6 +91,14 @@ cassis eval run --project ... --branch feature-x
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+
+# Probe questions through the text-to-SQL agent using the local ontology files
+# (one full agent run per question, expect ~30-90s each; repeat -q for several):
+cassis ontology test --project ... -q "How much was refunded last month?" -q "Net revenue in Q1?"
+
+# Add a gold case to the eval suite (rejected if the exact question already exists):
+cassis eval add-case --project ... -q "How much was refunded last month?" \
+  --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..."
 ```
 
 Configuration (flags take precedence over env vars):
@@ -86,7 +108,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | — (required)                |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis` — must match the project's git-sync "Path" setting |
-| `--project` (pull, upload, eval run) | `CASSIS_PROJECT_ID` | — (required)      |
+| `--project` (pull, upload, eval run, eval add-case, test) | `CASSIS_PROJECT_ID` | — (required)      |
 
 `cassis eval run` also accepts `--label` (run label in the Evals page; defaults
 to the branch name from the CI environment or the local git checkout; rejected
@@ -114,13 +136,13 @@ cassis ontology fmt --check
 
 | Code | Meaning                                                                        |
 | ---- | ------------------------------------------------------------------------------ |
-| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) |
-| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run) |
+| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
+| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed) |
 | 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
-| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, or `--timeout` reached |
+| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, out of credits, or `--timeout` reached |
 
-Commands that send the local tree (`check`, `upload`, `eval run`) accept up to
-2000 YAML files / 5 MB total — far above real ontologies (a few hundred small
+Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to
+2000 ontology files / 5 MB total — far above real ontologies (a few hundred small
 files). Beyond that the CLI fails fast with exit 2 before uploading anything;
 double-check `--base-path` if you hit it.
 

{cassis_cli-0.4.0 → cassis_cli-1.1.0}/README.md

@@ -5,8 +5,12 @@ Run Cassis actions from your CI pipelines:
 - `cassis ontology check` validates the ontology files in your repository with the exact same checks as the Cassis GitHub PR check (YAML parsing, round-trip, import validation) — so you can gate merges in any CI system, not just GitHub.
 - `cassis ontology fmt` rewrites the ontology files in canonical form (think `black`/`gofmt` for the ontology), so hand or agent edits pass the round-trip check.
 - `cassis ontology upload` uploads the ontology files to a Cassis project (full replace) and, by default, publishes them immediately as a new version — so a merge to your main branch can go live in one CI step.
-- `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local YAML files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` downloads the project's unpublished ontology into your repository checkout (full sync — stale local ontology files are pruned), so you can start editing from the current state, or bootstrap a repo that isn't git-synced (e.g. Bitbucket).
+- `cassis ontology pull` and `cassis ontology fmt` also write `<base-path>/AGENTS.md`, the Cassis ontology modeling guide, into the checkout (default `cassis/AGENTS.md`) — a managed file (generated banner; the CLI overwrites local edits) so a repo-aware coding agent loads current Cassis modeling doctrine by convention. It sits inside the ontology directory but is not part of the ontology tree (which is the YAML files plus the domain Markdown files `domains/**/README.md`), so it is never uploaded, validated, or pruned. Commit it alongside your ontology changes. The guide text ships inside the CLI package, so its version tracks the **installed cassis-cli version** — upgrade the CLI (`pip install -U cassis-cli`) and re-run `fmt` to pick up doctrine updates; an unpinned `pip install cassis-cli` in CI gets them automatically. The banner stamps a doctrine version, and the CLI never *downgrades* the file: if the checkout's `AGENTS.md` was written by a newer doctrine (a newer CLI, or Cassis itself on a publish), `fmt`/`pull` leave it in place, print an upgrade notice, and `fmt --check` still passes.
+- The CLI identifies itself to the API (`User-Agent: cassis-cli/<version>`), and successful API responses advertise the newest published version — when you are behind, commands print a one-line upgrade notice on stderr (purely informational; output and exit codes are unchanged).
 - `cassis eval run` runs the project's eval suite against your local ontology files (scored in-memory — nothing is pushed to Cassis) and prints per-question results, so you can test the changes on your git branch before merging.
+- `cassis ontology test` runs individual questions through the text-to-SQL agent using your local ontology files, so you can check that a change actually works (e.g. a new column gets picked) — where `eval run` only checks for regressions on existing eval cases.
+- `cassis eval add-case` adds a gold question/SQL case to the project's eval suite — after fixing an ontology issue, add the question users were failing on so `eval run` guards it from regressing.
 
 ## Install
 
@@ -14,11 +18,21 @@ Run Cassis actions from your CI pipelines:
 pip install cassis-cli
 ```
 
+## Ontology file format
+
+The ontology tree under `<base-path>` (default `cassis/`) is:
+
+- **Project identity** — `project.yml`: the Cassis project id and format version. Written by `pull` and by server-side publish (the contexts that know the id); a local `fmt` won't create it.
+- **Domains** — Markdown files: every domain is the `README.md` of its folder — `domains/README.md` for the root, `domains/<path>/README.md` for each sub-domain. Each has a small YAML frontmatter block (`type`, `title`, `description`) and a Markdown body carrying the domain's `context_md`; a generated section at the bottom links the domain's tables and metrics (kept current by `fmt`/`pull` — edit your prose above it, and the PR check fails if the links are stale, so re-run `fmt`). The layout is a Cassis profile inspired by [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf): the files render on GitHub and read in any Markdown editor, but Cassis validates them strictly (unknown keys are flagged, not preserved).
+- **Tables, joins, metrics** — YAML, unchanged: `tables/<schema>/<table>.yml`, `joins.yml`, `metrics/<name>.yml`.
+
+**Migrating an existing repo** (domains were YAML `_project.yml` / `_domain.yml` before cassis-cli 1.1.0): upgrade and run `cassis ontology fmt` (or `cassis ontology pull` if you have no local edits) — it rewrites the domain files to Markdown and removes the old ones. Review the diff and commit. Cassis reads the old YAML domain files too, so an un-migrated repo keeps working until you convert it. **Uploading requires cassis-cli ≥ 1.1.0** — the server rejects an older CLI (which would drop the Markdown domain files) with a clear upgrade error.
+
 ## Setup
 
 1. Create an API key in Cassis under **Organization settings → API keys** (keys start with `sk-k6-`).
 2. Store it as a CI secret and expose it as `CASSIS_API_KEY`.
-3. For `pull`, `upload` and `eval run`: find the project ID (UUID) in the project's URL and expose it as `CASSIS_PROJECT_ID` (or pass `--project`).
+3. For `pull`, `upload`, `eval run`, `ontology test`, and `eval add-case`: the project ID (UUID) is taken from `<base-path>/project.yml` in the checkout (written by `pull` and by publishing) — so once a repo is pulled you don't need to pass it. To override, or before the first pull, set `CASSIS_PROJECT_ID` or pass `--project` (find the UUID in the project's URL).
 
 ## Usage
 
@@ -57,6 +71,14 @@ cassis eval run --project ... --branch feature-x
 
 # Start the run and return immediately (poll in the webapp):
 cassis eval run --project ... --no-wait
+
+# Probe questions through the text-to-SQL agent using the local ontology files
+# (one full agent run per question, expect ~30-90s each; repeat -q for several):
+cassis ontology test --project ... -q "How much was refunded last month?" -q "Net revenue in Q1?"
+
+# Add a gold case to the eval suite (rejected if the exact question already exists):
+cassis eval add-case --project ... -q "How much was refunded last month?" \
+  --gold-sql "SELECT SUM(refunded_cents) / 100.0 FROM public.orders WHERE ..."
 ```
 
 Configuration (flags take precedence over env vars):
@@ -66,7 +88,7 @@ Configuration (flags take precedence over env vars):
 | `--api-key` | `CASSIS_API_KEY` | — (required)                |
 | `--api-url` | `CASSIS_API_URL` | `https://app.getcassis.com` |
 | `--base-path` | `CASSIS_BASE_PATH` | `cassis` — must match the project's git-sync "Path" setting |
-| `--project` (pull, upload, eval run) | `CASSIS_PROJECT_ID` | — (required)      |
+| `--project` (pull, upload, eval run, eval add-case, test) | `CASSIS_PROJECT_ID` | — (required)      |
 
 `cassis eval run` also accepts `--label` (run label in the Evals page; defaults
 to the branch name from the CI environment or the local git checkout; rejected
@@ -94,13 +116,13 @@ cassis ontology fmt --check
 
 | Code | Meaning                                                                        |
 | ---- | ------------------------------------------------------------------------------ |
-| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) |
-| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run) |
+| 0    | Ontology is valid (check) / pulled (pull) / uploaded (upload) / eval run completed all-passed (eval run) / every probe completed (test — whatever its outcome; probes are informational, don't gate CI on them) |
+| 1    | Validation failed (check: findings printed; upload: nothing imported; eval run: invalid tree, failed cases, or failed/cancelled run; test: invalid tree or a probe failed) |
 | 2    | Usage error (missing API key or project, no ontology directory, unreadable file, tree over the size limits) |
-| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, or `--timeout` reached |
+| 3    | Transport/API error (unreachable API, invalid key, inaccessible project, unexpected response), another run already active, out of credits, or `--timeout` reached |
 
-Commands that send the local tree (`check`, `upload`, `eval run`) accept up to
-2000 YAML files / 5 MB total — far above real ontologies (a few hundred small
+Commands that send the local tree (`check`, `fmt`, `upload`, `eval run`, `test`) accept up to
+2000 ontology files / 5 MB total — far above real ontologies (a few hundred small
 files). Beyond that the CLI fails fast with exit 2 before uploading anything;
 double-check `--base-path` if you hit it.
 

{cassis_cli-0.4.0 → cassis_cli-1.1.0}/cassis_cli/__init__.py

@@ -1,3 +1,3 @@
 """Cassis CLI — run Cassis actions from your CI pipelines."""
 
-__version__ = "0.4.0"
+__version__ = "1.1.0"

{cassis_cli-0.4.0 → cassis_cli-1.1.0}/cassis_cli/api.py

@@ -2,13 +2,73 @@
 
 from __future__ import annotations
 
+import sys
 from typing import Any, Optional
 
 import httpx
+from cassis_cli import __version__
 
 DEFAULT_API_URL = "https://app.getcassis.com"
 TIMEOUT_SECONDS = 60.0
 
+USER_AGENT = f"cassis-cli/{__version__}"
+
+# Response header the CI endpoints set to the newest cassis-cli on PyPI.
+LATEST_VERSION_HEADER = "x-cassis-cli-latest"
+
+_upgrade_notice_shown = False
+
+
+def _version_tuple(version: str) -> Optional[tuple[int, ...]]:
+    try:
+        return tuple(int(part) for part in version.strip().split("."))
+    except ValueError:
+        return None
+
+
+def _maybe_print_upgrade_notice(response: httpx.Response) -> None:
+    """Print a one-time stderr notice when the server advertises a newer CLI.
+
+    Purely informational — never changes behavior or exit codes. Staying
+    current matters beyond bugfixes: the ontology modeling guide written to
+    ``cassis/AGENTS.md`` ships inside this package, so an old CLI keeps old
+    doctrine in the repo.
+    """
+    global _upgrade_notice_shown
+    if _upgrade_notice_shown:
+        return
+    latest = response.headers.get(LATEST_VERSION_HEADER)
+    if not latest:
+        return
+    mine, theirs = _version_tuple(__version__), _version_tuple(latest)
+    if mine is None or theirs is None:
+        return
+    # Zero-pad to equal length so "0.6" == "0.6.0" (same rule as the webapp's
+    # Agent setup page — the comparison logic exists on both surfaces).
+    width = max(len(mine), len(theirs))
+    if theirs + (0,) * (width - len(theirs)) <= mine + (0,) * (width - len(mine)):
+        return
+    _upgrade_notice_shown = True
+    print(
+        f"notice: cassis-cli {latest} is available (you have {__version__}) — "
+        "run `pip install -U cassis-cli`, then `cassis ontology fmt` to refresh cassis/AGENTS.md.",
+        file=sys.stderr,
+    )
+
+
+def _client(*, timeout: float = TIMEOUT_SECONDS, transport: Optional[httpx.BaseTransport] = None) -> httpx.Client:
+    """Build the HTTP client every API call goes through.
+
+    Identifies the CLI to the server (User-Agent) and watches responses for
+    the newer-version advertisement.
+    """
+    return httpx.Client(
+        timeout=timeout,
+        transport=transport,
+        headers={"User-Agent": USER_AGENT},
+        event_hooks={"response": [_maybe_print_upgrade_notice]},
+    )
+
 
 class ApiError(Exception):
     """Transport or HTTP-level failure talking to the Cassis API."""
@@ -52,6 +112,14 @@ def _project_scope_error(response: httpx.Response) -> ApiError:
     )
 
 
+def _detail_or_text(response: httpx.Response) -> Any:
+    """Return the error response's ``detail`` (str or structured), falling back to its body."""
+    try:
+        return response.json().get("detail") or response.text[:500]
+    except ValueError:
+        return response.text[:500]
+
+
 def _parse_json_response(response: httpx.Response, url: str) -> Any:
     try:
         return response.json()
@@ -71,7 +139,7 @@ def post_ontology_check(
     """POST the ontology tree to /api/ci/ontology-check and return the response body."""
     url = api_url.rstrip("/") + "/api/ci/ontology-check"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(
                 url,
                 json={"files": files},
@@ -110,7 +178,7 @@ def post_ontology_import(
     if label is not None:
         body["label"] = label
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(
                 url,
                 json=body,
@@ -122,11 +190,9 @@ def post_ontology_import(
     if response.status_code == 401:
         raise AuthError("The Cassis API rejected the API key (invalid or expired).")
     if response.status_code == 400:
-        try:
-            detail = response.json().get("detail") or response.text[:500]
-        except ValueError:
-            detail = response.text[:500]
-        raise UploadValidationError(str(detail))
+        raise UploadValidationError(str(_detail_or_text(response)))
+    if response.status_code == 426:  # this CLI is too old for the server's ontology format
+        raise UploadValidationError(str(_detail_or_text(response)))
     if response.status_code in (403, 404):
         raise _project_scope_error(response)
     if response.status_code >= 400:
@@ -149,7 +215,7 @@ def get_ontology_export(
     """GET /api/ci/projects/{project_id}/ontology/export and return the files tree."""
     url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/ontology/export"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.get(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -186,7 +252,7 @@ def post_eval_run_start(
     if label is not None:
         body["label"] = label
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(url, json=body, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -194,11 +260,7 @@ def post_eval_run_start(
     if response.status_code == 401:
         raise AuthError("The Cassis API rejected the API key (invalid or expired).")
     if response.status_code == 400:
-        try:
-            detail = response.json().get("detail") or response.text[:500]
-        except ValueError:
-            detail = response.text[:500]
-        raise EvalStartValidationError(detail)
+        raise EvalStartValidationError(_detail_or_text(response))
     if response.status_code == 409:
         raise EvalRunActiveError(
             "An eval run is already active for this project — wait for it to finish or cancel it "
@@ -214,10 +276,55 @@ def post_eval_run_start(
     return result
 
 
+class EvalCaseExistsError(ApiError):
+    """The project already has an eval case with this exact question."""
+
+
+class EvalCaseGoldSqlError(ApiError):
+    """The API rejected the gold SQL (400): it does not run against the project's data source."""
+
+
+def post_eval_case_create(
+    *,
+    api_url: str,
+    api_key: str,
+    project_id: str,
+    question: str,
+    gold_sql: str,
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """POST to /api/ci/projects/{project_id}/eval/cases and return the created case."""
+    url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/eval/cases"
+    try:
+        with _client(transport=transport) as client:
+            response = client.post(
+                url,
+                json={"question": question, "gold_sql": gold_sql},
+                headers={"Authorization": f"Bearer {api_key}"},
+            )
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    if response.status_code == 409:
+        raise EvalCaseExistsError(str(_detail_or_text(response)))
+    if response.status_code == 400:
+        raise EvalCaseGoldSqlError(str(_detail_or_text(response)))
+    if response.status_code in (403, 404):
+        raise _project_scope_error(response)
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if not isinstance(result, dict) or not all(key in result for key in ("id", "question", "gold_sql")):
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result
+
+
 def _get_eval_json(url: str, api_key: str, transport: Optional[httpx.BaseTransport]) -> Any:
     """GET an eval-run URL with the shared error mapping."""
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.get(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -274,7 +381,7 @@ def post_eval_run_cancel(
     """POST /api/ci/projects/{project_id}/eval/runs/{run_id}/cancel."""
     url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/eval/runs/{run_id}/cancel"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(url, headers={"Authorization": f"Bearer {api_key}"})
     except httpx.HTTPError as exc:
         raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
@@ -296,7 +403,7 @@ def post_ontology_fmt(
     """POST the ontology tree to /api/ci/ontology-fmt and return the response body."""
     url = api_url.rstrip("/") + "/api/ci/ontology-fmt"
     try:
-        with httpx.Client(timeout=TIMEOUT_SECONDS, transport=transport) as client:
+        with _client(transport=transport) as client:
             response = client.post(
                 url,
                 json={"files": files},
@@ -320,3 +427,59 @@ def post_ontology_fmt(
     ):
         raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
     return result
+
+
+# The server-side probe budget is 300s; leave headroom for transport.
+ONTOLOGY_TEST_TIMEOUT_SECONDS = 330.0
+
+
+class OntologyTestValidationError(ApiError):
+    """The API rejected the ontology tree as invalid (400).
+
+    ``detail`` keeps the structured payload (``{"message", "findings"}``) for
+    display.
+    """
+
+    def __init__(self, detail: object) -> None:
+        super().__init__(str(detail))
+        self.detail = detail
+
+
+def post_ontology_test(
+    *,
+    api_url: str,
+    api_key: str,
+    project_id: str,
+    files: dict[str, str],
+    question: str,
+    transport: Optional[httpx.BaseTransport] = None,
+) -> dict[str, Any]:
+    """POST to /api/ci/projects/{project_id}/ontology/test and return the probe outcome.
+
+    Blocks for the duration of the agent run (up to ~5 minutes server-side).
+    """
+    url = api_url.rstrip("/") + f"/api/ci/projects/{project_id}/ontology/test"
+    try:
+        with _client(timeout=ONTOLOGY_TEST_TIMEOUT_SECONDS, transport=transport) as client:
+            response = client.post(
+                url,
+                json={"files": files, "question": question},
+                headers={"Authorization": f"Bearer {api_key}"},
+            )
+    except httpx.HTTPError as exc:
+        raise ApiError(f"Could not reach the Cassis API at {url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise AuthError("The Cassis API rejected the API key (invalid or expired).")
+    if response.status_code == 400:
+        raise OntologyTestValidationError(_detail_or_text(response))
+    if response.status_code == 402:
+        raise ApiError("Your organization has run out of credits. Contact your administrator to top up.")
+    if response.status_code in (403, 404):
+        raise _project_scope_error(response)
+    if response.status_code >= 400:
+        raise ApiError(f"Cassis API returned HTTP {response.status_code}: {response.text[:500]}")
+    result = _parse_json_response(response, url)
+    if not isinstance(result, dict) or "status" not in result:
+        raise ApiError(f"Unexpected response shape from the Cassis API at {url}.")
+    return result

cassis_cli-1.1.0/cassis_cli/common.py

@@ -0,0 +1,177 @@
+"""Helpers shared by the `cassis` subcommands (tree collection, auth, exit codes)."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+from typing import Optional
+from uuid import UUID
+
+import typer
+
+# Repository directory the ontology tree is exported under. Must match the
+# project's git-sync "Path" setting in Cassis (default "cassis").
+DEFAULT_BASE_PATH = "cassis"
+
+# Exit codes (documented in the README; stable contract for CI scripts).
+EXIT_OK = 0
+EXIT_VALIDATION_FAILED = 1
+EXIT_USAGE = 2
+EXIT_TRANSPORT = 3
+
+# Request ceilings of the /api/ci file-tree endpoints, mirrored so oversized
+# trees fail fast with a clear message before any upload. Source of truth:
+# backend/app/schemas/ci.py (the server's 422 remains the backstop).
+MAX_FILES = 2000
+MAX_TOTAL_BYTES = 5 * 1024 * 1024
+
+
+def is_ontology_file(rel_path: str) -> bool:
+    """Whether a base-relative path is an ontology file the server reads.
+
+    YAML (``project.yml``, tables, joins, metrics, legacy domains) plus domain
+    Markdown — every domain is the ``README.md`` of its folder, root included
+    (``domains/README.md``), so ``domains/**/README.md`` covers them all. Mirrors
+    the server's ``ontology_fs.is_ontology_tree_file`` (kept in sync by hand —
+    the CLI can't import the backend). Excludes the managed ``AGENTS.md`` and any
+    stray Markdown note, so neither is uploaded nor deleted by ``pull --prune``.
+    """
+    if rel_path.endswith((".yml", ".yaml")):
+        return True
+    return rel_path.startswith("domains/") and rel_path.endswith("/README.md")
+
+
+def is_legacy_domain_file(rel_path: str) -> bool:
+    """Whether a base-relative path is a legacy (pre-Markdown) domain file.
+
+    Used to report the one-time migration to the Markdown domain format, when
+    ``pull``/``fmt`` remove a ``_project.yml``/``_domain.yml`` and write the
+    ``domains/README.md`` (and sub-domain ``README.md``) that replaces it.
+    """
+    return rel_path == "_project.yml" or rel_path.endswith("/_domain.yml")
+
+
+def collect_files(ontology_dir: Path) -> dict[str, str]:
+    """Read every ontology file under the ontology dir, keyed by posix relpath.
+
+    Ontology files are YAML and domain Markdown (see ``is_ontology_file``); the
+    managed ``AGENTS.md`` and stray notes are skipped. Exits 2 (usage) on an
+    unreadable or non-UTF-8 file — a local checkout problem, reported before
+    anything is sent to the API.
+    """
+    files: dict[str, str] = {}
+    for pattern in ("**/*.yml", "**/*.yaml", "**/*.md"):
+        for file in sorted(ontology_dir.glob(pattern)):
+            if not file.is_file():
+                continue
+            rel = file.relative_to(ontology_dir).as_posix()
+            if not is_ontology_file(rel):
+                continue
+            try:
+                files[rel] = file.read_text(encoding="utf-8")
+            except (UnicodeDecodeError, OSError) as exc:
+                typer.secho(f"Cannot read {file}: {exc}", fg=typer.colors.RED, err=True)
+                raise typer.Exit(EXIT_USAGE) from exc
+    return files
+
+
+# project.yml is a two-line machine-written file (`cassis_format_version`, `project_id`);
+# match the id line directly rather than pull in a YAML parser just for this.
+_PROJECT_ID_LINE = re.compile(r"^project_id:\s*['\"]?([^'\"\s]+)['\"]?\s*$")
+
+
+def read_project_id_from_dir(ontology_dir: Path) -> Optional[str]:
+    """Return the ``project_id`` recorded in ``<ontology_dir>/project.yml``, or None."""
+    try:
+        text = (ontology_dir / "project.yml").read_text(encoding="utf-8")
+    except (OSError, UnicodeDecodeError) as _exc:  # `as` keeps black from stripping the parens (3.14-only syntax)
+        return None
+    for line in text.splitlines():
+        match = _PROJECT_ID_LINE.match(line.strip())
+        if match:
+            return match.group(1)
+    return None
+
+
+def resolve_project_id(project_id: Optional[str], ontology_dir: Path) -> str:
+    """Resolve the target project id, defaulting to the checkout's ``project.yml``.
+
+    Precedence: an explicit ``--project`` / ``CASSIS_PROJECT_ID`` wins; otherwise
+    the ``project_id`` recorded in ``<base-path>/project.yml`` (written by
+    ``pull`` / publish) is used, and where it came from is noted on stderr so a
+    stale value in a copied repo is visible. Exits 2 (usage) when neither is
+    available or the value isn't a UUID.
+    """
+    from_file = False
+    if not project_id:
+        project_id = read_project_id_from_dir(ontology_dir)
+        from_file = project_id is not None
+    if not project_id:
+        typer.secho(
+            f"No project. Pass --project (or set CASSIS_PROJECT_ID), or run in a checkout whose "
+            f"{ontology_dir.name}/project.yml records it (written by `cassis ontology pull` or a publish).",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(EXIT_USAGE)
+    try:
+        UUID(project_id)
+    except ValueError:
+        typer.secho(f"--project must be a project ID (UUID), got {project_id!r}.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+    if from_file:
+        typer.secho(f"Using project {project_id} from {ontology_dir.name}/project.yml.", fg=typer.colors.CYAN, err=True)
+    return project_id
+
+
+def require_api_key(api_key: Optional[str]) -> str:
+    """Exit 2 (usage) when no API key was provided."""
+    if not api_key:
+        typer.secho(
+            "No API key. Set CASSIS_API_KEY or pass --api-key "
+            "(create one in Cassis under Organization settings -> API keys).",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(EXIT_USAGE)
+    return api_key
+
+
+def collect_tree(path: Path, base_path: str) -> "tuple[dict[str, str], str]":
+    """Resolve the ontology dir under the checkout and read its file tree.
+
+    Returns ``(files, normalized_base_path)``. Exits 2 (usage) on an empty or
+    missing dir, or a tree beyond the API request ceilings — all local checkout
+    problems, reported before anything is sent to the API.
+    """
+    base_path = base_path.strip().strip("/")
+    if not base_path:
+        typer.secho("--base-path must not be empty.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+
+    ontology_dir = path / Path(base_path)
+    if not ontology_dir.is_dir():
+        typer.secho(
+            f"No {base_path}/ directory found under {path}. "
+            "If the project exports to a custom path, pass it with --base-path (or CASSIS_BASE_PATH).",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(EXIT_USAGE)
+
+    files = collect_files(ontology_dir)
+    if not files:
+        typer.secho(f"No ontology files found under {ontology_dir}.", fg=typer.colors.RED, err=True)
+        raise typer.Exit(EXIT_USAGE)
+
+    total_bytes = sum(len(content.encode()) for content in files.values())
+    if len(files) > MAX_FILES or total_bytes > MAX_TOTAL_BYTES:
+        typer.secho(
+            f"Ontology tree too large: {len(files)} files / {total_bytes / (1024 * 1024):.1f} MB "
+            f"(limits: {MAX_FILES} files / {MAX_TOTAL_BYTES // (1024 * 1024)} MB). "
+            "Check that --base-path points at the ontology directory, not a larger tree.",
+            fg=typer.colors.RED,
+            err=True,
+        )
+        raise typer.Exit(EXIT_USAGE)
+    return files, base_path