@oxy-hq/cli 0.7.0 → 0.9.0

package/skills/oxy-agents/SKILL.md

@@ -0,0 +1,318 @@
+---
+name: oxy-agents
+description: >-
+  Use when creating, editing, reviewing or debugging an Oxy agent, a `.agentic.yml` file that
+  answers questions from the semantic model by clarifying, specifying, solving (SQL), executing
+  and interpreting. Triggers include "create an analytics agent", "my agent picks the wrong
+  metric", "my solving instructions are ignored", "the agent never uses my verified query", "the
+  agent can't see a database", "which tools can a state call", "set thinking, effort,
+  budget_tokens or extended thinking", "per-state model or max_retries", "validation rules",
+  "Cube or Looker semantic_engine", "analytics vs app-builder agent", "`.agent.yml` or
+  `.agentic.yml`", and validating, running (`oxy agentic run`) or testing (`oxy test`) an agent.
+  Not for writing the semantic model itself (views, topics) or automations.
+---
+
+# Oxy agents (`.agentic.yml`)
+
+An Oxy agent is one YAML file, `<name>.agentic.yml`. It configures a fixed
+pipeline that turns a natural-language question into a grounded query against
+the semantic model, runs it, checks the result and explains it. You configure
+the pipeline. You do not define its states or its tools.
+
+## There is one kind of agent
+
+- **`.agentic.yml` is the agent.** The classic `.agent.yml` format (a single
+  LLM call with custom tools) has been removed. Do not create one, and do not
+  copy one from an old workspace.
+- **There is no `kind:` field.** Every `.agentic.yml` runs the same analytics
+  pipeline. A file named `app_builder.agentic.yml` is not special. Generating
+  `.app.yml` data apps, views or agent files is the in-product **Builder Agent**
+  (chat Build mode), and no `.agentic.yml` configures it.
+- **Automations are a different file.** Deterministic multi-step work is an
+  automation, `.automation.yml` (`.procedure.yml` still works). `.workflow.yml`
+  is no longer recognised at all. An agent can *run* an automation it finds in
+  its context. It cannot *be* one.
+- **Instruction-only agents skip the pipeline.** A file with `instructions:`
+  and no `databases:`, `context:` or `states:` is answered with one LLM call
+  and queries nothing. Add any of those three and you get the full pipeline.
+
+**Use an agent** for conversational questions over data that has a semantic
+model. **Use a verified query** (a `.sql` file with an `oxy:` header, below)
+when one question must always get exactly the same SQL. **Use an automation**
+when the steps are fixed and need no LLM to choose between them.
+
+## Minimal file
+
+```yaml
+llm:
+  ref: claude            # a `name:` under config.yml `models:`
+context:
+  - ./semantics/**/*.view.yml
+  - ./semantics/**/*.topic.yml
+```
+
+That is enough when the views declare a `datasource`. The agent can query
+every database its context references. The full annotated file is
+[templates/agentic-template.yml](templates/agentic-template.yml).
+
+Top-level keys: `instructions`, `timezone`, `databases`, `context`, `llm`,
+`thinking`, `states`, `validation`, `semantic_engine`. All are optional.
+**Unknown keys are ignored silently at every level**, by the runtime and by
+both validators. A typo such as `stats:` or `solve:` costs you the setting
+with no error.
+
+## How a question flows
+
+```
+clarifying → specifying → solving → executing → interpreting → done
+     ↑            ↑           ↑                        ↑
+     └──────── diagnosing (when a check fails) ────────┘
+```
+
+1. **clarifying** classifies the question as a trend, comparison, breakdown,
+   single value, distribution, general inquiry, root cause or opportunity. It
+   searches the catalog and automations, and may ask the user to clarify.
+   Three outcomes leave the normal path early:
+   - **General inquiry** ("what metrics do you have?"). Answered directly,
+     with no query.
+   - **Root cause / opportunity** ("why did revenue drop?", "where can we
+     grow?"). Handled by metric-tree and anomaly tools, skipping
+     specifying/solving.
+   - **Confident semantic query.** If the model proposes the exact measures
+     and dimensions with high confidence and the semantic model compiles them,
+     specifying and solving are both skipped. Likewise when it selects a
+     verified query or automation, that is run instead.
+2. **specifying** turns the question into a semantic query. If the semantic
+   model compiles it, **solving is skipped** and the compiled SQL runs.
+3. **solving** writes SQL itself. This happens only when the semantic model
+   could not answer.
+4. **executing** runs the SQL. It has no LLM. Then the result checks run.
+5. **interpreting** writes the answer and picks a chart.
+
+**Diagnosing** routes a failure back upstream:
+
+- An unresolved metric or join, an empty result, a compile error or bad SQL
+  goes back to specifying.
+- An ambiguous column goes back to clarifying.
+- Outliers, a stale period or a bad chart go to interpreting, which still
+  answers and states the caveat.
+
+The practical consequence: **guidance every answer needs goes in the global
+`instructions:`, `clarifying` or `interpreting`, never only in `solving`.** On a
+well-modelled workspace, solving rarely runs.
+
+## Tools each state can call
+
+The tool set is fixed by the runtime and is not configurable. There is no
+`tools:` key.
+
+| State | Tools |
+| --- | --- |
+| clarifying | `search_catalog`, `search_automations`, `propose_semantic_query`, `ask_user` |
+| clarifying, root cause / opportunity | `explain_metric`, `find_opportunities`, `metric_sensitivity`, `predict_impact`, plus `list_anomalies`, `detect_anomalies` and `explain_anomaly` when anomaly monitoring is set up |
+| specifying | `search_catalog`, `sample_columns`, `ask_user`. With a semantic model: `check_data_freshness`. Without one: `get_join_path`, `list_tables`, `describe_table` |
+| solving | `execute_preview` (runs a candidate query with `LIMIT 5`) |
+| executing | none |
+| interpreting | `render_chart` |
+
+`search_automations` searches only the automations and verified queries this
+agent's `context:` reaches. With none, it finds nothing. The metric-tree and
+anomaly tools are offered in interactive runs. They are not offered in
+workspace previews or scheduled runs.
+
+## The fields that matter
+
+**`instructions`** go into every LLM state's system prompt. Use them for the
+persona, business conventions ("fiscal year starts in February", "customer
+means the `customers` view") and which views or topics to prefer. Per-state
+`instructions` are **appended** to these, not substituted, so don't repeat the
+global text in them.
+
+**`timezone`** is an IANA name used for "today", "yesterday" and "last week".
+It falls back to config.yml's `timezone:`, then UTC. An invalid name fails the
+run.
+
+**`databases`** lists names from config.yml `databases:`. The agent can query
+exactly these plus every database its context files reference: a view's
+`datasource` (or `data_source`), a verified query's `database:`, or an
+automation task's `database:`. There is **no fallback to every database in
+config.yml**. With none at all, the run fails with `no databases configured`.
+
+**`context`** takes globs relative to the agent file's directory. Each match is
+bucketed by its file name:
+
+| Matches | Becomes |
+| --- | --- |
+| `*.view.yml`, `*.topic.yml` | the semantic catalog |
+| `*.sql` with an `oxy:` header | a **verified query**: found by `search_automations` and run verbatim |
+| `*.sql` without the header | SQL style examples in the solving prompt only |
+| `*.md` | domain notes in the clarifying and interpreting prompts |
+| `*.automation.yml`, `*.procedure.yml` | automations the agent may pick and run |
+| anything else | ignored |
+
+A verified query opts in with a leading comment block:
+
+```sql
+/*
+  oxy:
+    description: Monthly revenue by region
+    database: warehouse
+*/
+SELECT ...
+```
+
+Without the header, the same file is only a style hint. Context files under a
+dot-directory, `build/`, `dist/`, `target/` or `node_modules/`, or with
+`.test.` in the name, are not deployed, so a glob that works on your machine
+can come back empty in the cloud.
+
+**`llm`** picks the model.
+
+- `ref` names a model in config.yml `models:`, and the vendor, API key and base
+  URL come from that entry. In the cloud the key is a workspace secret, so
+  prefer `ref`.
+- With neither `ref` nor `model`, the first model in config.yml is used.
+- `model` overrides only the model ID. On its own (no `ref`) it uses
+  `api_key` or the server's `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`.
+- `vendor` (`anthropic`, `open_ai`, `open_ai_compat`) overrides the ref's wire
+  protocol. Use `open_ai_compat` for gateways that serve only
+  `/chat/completions`.
+- Google/Gemini models are not supported in agents.
+
+**`states`** honours four keys: `clarifying`, `specifying`, `solving` and
+`interpreting`.
+
+- `executing` has no LLM, so it is accepted and does nothing. Every other key
+  is ignored.
+- Each state takes `instructions`, `thinking`, `model` and `max_retries`.
+- `model` is a provider model ID. It inherits the vendor, key and base URL
+  from `llm:`, so it must be the same provider.
+- `max_retries` is the number of **tool-call rounds** per visit. When they run
+  out, the run pauses and asks the user whether to continue. It is not a
+  failure-retry count. Defaults: specifying 5, solving 3, interpreting 5.
+  Clarifying has a fixed limit and ignores the setting.
+
+**`validation`** sets the result checks. Leave it out and 15 default checks
+run. If you include it, it **replaces** the defaults: only the rules you list
+run, and a stage you leave empty is not checked. To switch one check off or
+retune it, list every rule that should still run. Rule names and parameters
+are in [references/quick-reference.md](references/quick-reference.md).
+
+**`semantic_engine`** routes queries through Cube or Looker ahead of Oxy's own
+semantic model. `${VAR}` placeholders are read from the server's environment,
+not from workspace secrets. The engine is pinged when a run starts, and an
+unreachable engine fails the run.
+
+## Thinking and models
+
+Thinking is resolved per state, in this order:
+
+1. the state's own `thinking`
+2. `llm.thinking`
+3. top-level `thinking`
+4. the state's built-in default: **adaptive for solving, disabled for
+   everything else**
+
+A global value replaces every built-in default. So top-level
+`thinking: adaptive` makes clarifying, specifying and interpreting think too,
+and `llm.thinking: disabled` turns thinking off in solving. Prefer per-state
+settings.
+
+| Form | Meaning |
+| --- | --- |
+| `adaptive` | the model decides when and how much to think |
+| `disabled` | no thinking (bare or quoted; **never** `false`, which is a parse error) |
+| `effort:high` or `{effort: high}` | depth hint: `low`, `medium`, `high`, `xhigh`, `max` |
+| `{budget_tokens: 10000}` | hard token ceiling (Claude 4.6 and earlier) |
+
+How each provider treats these forms:
+
+- **Anthropic.** `adaptive` and `effort` are native from Claude 4.6, and
+  `xhigh` from 4.7. Claude 4.7 and later **reject** `budget_tokens`, so Oxy
+  sends it as the nearest effort level and logs a warning. On older models it
+  goes the other way: `adaptive` and `effort` become a token budget, and
+  `xhigh` becomes `high`. Write `effort` on current models.
+- **OpenAI** (`open_ai`, Responses API). `effort` maps to reasoning effort,
+  with `xhigh` and `max` sent as `high`. `adaptive` leaves the model's default
+  effort. `budget_tokens` is ignored.
+- **`open_ai_compat`.** Any setting other than `disabled` adds a
+  chain-of-thought instruction to the prompt, because there is no native
+  control.
+
+A string that isn't one of the forms above (`thinking: enabled`,
+`thinking: high`) silently means **disabled**. An unknown effort level
+silently means `medium`.
+
+`llm.extended_thinking: {model, thinking}` is used while the user has the
+**extended thinking** toggle on (`--thinking-mode extended_thinking` from the
+CLI). It replaces the model and the thinking setting in every state. Without
+the block, the toggle changes nothing.
+
+Leave `llm.max_tokens` unset unless you need a hard cap. When it is unset, Oxy
+uses 4,096 without thinking and 16,384 (or the budget plus 4,096) with
+thinking. A value you set is used as-is, and thinking tokens count against it.
+
+## Validate, run, test
+
+```bash
+oxyc validate --file agents/sales.agentic.yml   # offline schema check, no server
+oxy validate --file agents/sales.agentic.yml    # the runtime's own parser
+oxy agentic run agents/sales.agentic.yml -q "Revenue by region last quarter?"
+oxy test agents/sales.agentic.test.yml          # LLM-judged evals
+```
+
+- **Both validators are structural only.** They catch a bad shape (a
+  malformed `thinking`, a wrong type, a duplicate key). `oxyc validate` also
+  rejects the vendor spellings `openai` and `openai_compat`, which the runtime
+  accepts, so write `open_ai` and `open_ai_compat` to satisfy both. Neither
+  validator resolves `databases:` or `llm.ref`, expands globs, reads `${VAR}`
+  placeholders, checks rule names or timezones, or notices unknown keys.
+- **The first full check is a run.** `oxy agentic run <path> -q "<question>"`
+  builds the agent exactly as the server does and streams each state. It runs
+  in a workspace directory and needs Oxy's Postgres database (`OXY_DATABASE_URL`),
+  like `oxy serve`. Add `--json` for JSONL events, or `--thinking-mode
+  extended_thinking`. A CLI run that pauses to ask the user something can't be
+  usefully resumed from the CLI, because `oxy agentic answer` only marks the
+  run as running and does not deliver the text. Ask a more specific question,
+  or continue in the app.
+- **`oxy build` does not check agents.** It builds search embeddings and syncs
+  integrations.
+- **`oxy test`** runs `.test.yml` evals against an agent and scores each case
+  with an LLM judge. The format and flags are in
+  [references/quick-reference.md](references/quick-reference.md).
+
+To point an editor at the schema, start the file with:
+
+```yaml
+# yaml-language-server: $schema=https://raw.githubusercontent.com/oxy-hq/oxygen/refs/heads/main/json-schemas/agentic.json
+```
+
+## Pitfalls
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `no databases configured` | nothing in `databases:` and no context file references a database | add a name from config.yml, or views with `datasource` |
+| agent can't see a table it "should" | the agent only reaches databases it declares or references | add it to `databases:` |
+| wrong model answers, or an auth error on the first question | `llm.ref` is not a `name:` in config.yml, so the agent fell back to a built-in Claude model and the server's `ANTHROPIC_API_KEY`; or the model's key secret is unset | fix the name; set the key |
+| "does not have a metric tree configured" | the question was classified as root cause or opportunity, and this workspace or run has no metric tree | define a metric tree, or ask for the numbers directly |
+| `solving` instructions never take effect | the semantic model compiled the question, so solving was skipped | move them to the global `instructions:` or `clarifying` |
+| verified query never chosen | `.sql` has no `oxy:` header, or isn't matched by `context:` | add the header and a `description`; check the glob |
+| state override does nothing | misspelled key, or `executing` / `clarifying.max_retries` | use the four honoured keys |
+| thinking silently off | `thinking: enabled` or another unknown string | use one of the forms above |
+| `thinking: false` rejected | boolean, not a mode | `thinking: disabled` |
+| a default check stopped running | a `validation:` section replaced the defaults | list every rule you still want |
+| `validation config error: unknown … rule` | rule name typo, or a rule under the wrong stage | names and stages are in the quick reference |
+| `invalid timezone '…'` | not an IANA name | e.g. `America/New_York`, `UTC` |
+| `environment variable '${X}' … is not set` | `semantic_engine` placeholder unset on the server | set it in the server environment |
+| `semantic engine connection error` | Cube/Looker unreachable or rejected the credentials | check `base_url` and credentials |
+| agent missing in the cloud, fine locally | `.agentic.yaml` extension, a dot-directory/`build/` path, or `.test.` in the name | rename or move it |
+
+## Read next
+
+- [references/quick-reference.md](references/quick-reference.md): every field,
+  default, rule and parameter in tables, plus the `.test.yml` format.
+- [templates/agentic-template.yml](templates/agentic-template.yml): an annotated
+  starter file that validates against the schema.
+- Related skills: **oxy-semantic-modeling** for the views and topics an agent
+  queries, **oxy-evals** for writing and repairing `.test.yml` evals, and
+  **oxy-automations** for automations.

package/skills/oxy-agents/references/quick-reference.md

@@ -0,0 +1,222 @@
+# `.agentic.yml` quick reference
+
+Dense tables. The reasoning behind them is in [../SKILL.md](../SKILL.md); a
+starter file is [../templates/agentic-template.yml](../templates/agentic-template.yml).
+
+## File
+
+| Rule | Detail |
+| --- | --- |
+| Name | `<name>.agentic.yml`. `.agentic.yaml` is not deployed to the cloud |
+| Location | anywhere in the workspace, except under a dot-directory, `build/`, `dist/`, `target/` or `node_modules/` |
+| Not deployed | any file with `.test.` in its name (that is the eval-file convention) |
+| Referenced as | its workspace-relative path, e.g. `agents/sales.agentic.yml`. The UI shows the stem (`sales`) |
+| Unknown keys | ignored silently at every level, by the runtime and by both validators |
+| Duplicate keys | a YAML parse error |
+| Editor schema | `# yaml-language-server: $schema=https://raw.githubusercontent.com/oxy-hq/oxygen/refs/heads/main/json-schemas/agentic.json` |
+
+## Top-level fields
+
+| Field | Type | Default | Notes |
+| --- | --- | --- | --- |
+| `instructions` | string | none | Added to every LLM state's system prompt |
+| `timezone` | IANA name | config.yml `timezone:`, then UTC | Used for "today" and relative dates. An invalid name fails the run |
+| `databases` | list of config.yml database names | `[]` | Combined with the databases the context references. There is no all-databases fallback |
+| `context` | list of globs | `[]` | Relative to the agent file's directory |
+| `llm` | map | see below | Model, key, thinking |
+| `thinking` | thinking form | none | Global thinking. Overrides each state's built-in default; `llm.thinking` overrides it |
+| `states` | map | `{}` | Per-state overrides |
+| `validation` | map | 15 default rules | **Replaces** the defaults when present |
+| `semantic_engine` | map | none | Cube or Looker ahead of Oxy's semantic model |
+
+An instruction-only agent (`instructions`, and no `databases`, `context` or
+`states`) is one LLM call. It runs no pipeline and no queries.
+
+## `llm:`
+
+| Field | Default | Notes |
+| --- | --- | --- |
+| `ref` | none | A `name:` under config.yml `models:`. Supplies the vendor, key (a workspace secret in the cloud), base URL, Azure settings and headers |
+| `model` | from `ref`, else the first config.yml model | A provider model ID. Setting `model` without `ref` skips config.yml entirely, so the key must come from `api_key` or the server environment |
+| `vendor` | from `ref`, else `anthropic` | `anthropic`, `open_ai` (Responses API) or `open_ai_compat` (`/chat/completions`). The runtime also accepts `openai` and `openai_compat`; `oxyc validate` does not |
+| `api_key` | from `ref`, else `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` on the server | Prefer `ref` |
+| `base_url` | from `ref`, else the provider default | `open_ai_compat` defaults to `http://localhost:11434/v1` |
+| `max_tokens` | 4,096 without thinking; 16,384 (or budget + 4,096) with thinking | A value you set is used as-is, and thinking tokens count against it |
+| `thinking` | none | Beats top-level `thinking` |
+| `extended_thinking` | none | `{model, thinking}`. Replaces both in every state while the user's extended-thinking toggle is on |
+
+A `ref` that is not in config.yml is not an error at load time. The agent falls
+back to a built-in Claude model and the server's `ANTHROPIC_API_KEY`.
+Google/Gemini models are not supported in agents.
+
+## `states:`
+
+| Key | LLM | Thinking default | `max_retries` default | Prompt extras |
+| --- | --- | --- | --- | --- |
+| `clarifying` | yes | disabled | fixed; the setting is ignored | `.md` context docs |
+| `specifying` | yes | disabled | 5 | SQL dialect |
+| `solving` | yes | adaptive | 3 | SQL dialect, `.sql` examples |
+| `executing` | **no** | n/a | n/a | key accepted, does nothing |
+| `interpreting` | yes | disabled | 5 | `.md` context docs |
+
+Any other key (`solve`, `diagnosing`, …) is ignored silently.
+
+| Per-state field | Notes |
+| --- | --- |
+| `instructions` | **Appended** after the global `instructions` for this state |
+| `thinking` | Any thinking form. Highest precedence (except the extended-thinking toggle) |
+| `model` | Provider model ID. Inherits the vendor, key and base URL from `llm:`, so it must be the same provider |
+| `max_retries` | Tool-call rounds per visit. When they run out, the run pauses and asks the user whether to continue |
+
+## Thinking
+
+| Form | Meaning |
+| --- | --- |
+| `adaptive` | the model decides |
+| `disabled` / `"disabled"` | off |
+| `effort:<level>` / `{effort: <level>}` | `low`, `medium`, `high`, `xhigh`, `max`. An unknown level means `medium` |
+| `{budget_tokens: N}` | hard ceiling. Native on Claude 3.7 to 4.6 |
+| any other string | silently **disabled** |
+| `false` / `true` | parse error |
+
+Precedence, highest first: the extended-thinking toggle, then `states.<s>.thinking`,
+then `llm.thinking`, then top-level `thinking`, then the state default.
+
+| Provider | `adaptive` | `effort` | `budget_tokens` |
+| --- | --- | --- | --- |
+| Anthropic, Claude 4.7+ | native | native, all 5 levels | API rejects it; Oxy sends the nearest effort level and warns |
+| Anthropic, Claude 4.6 | native | native; `xhigh` → `high` | native |
+| Anthropic, 3.7 to 4.5 | sent as a mid-range budget | sent as a budget; `xhigh` → `high` | native |
+| OpenAI (`open_ai`) | the model's default reasoning | reasoning effort; `xhigh` and `max` → `high` | ignored |
+| `open_ai_compat` | chain-of-thought prompt | chain-of-thought prompt | chain-of-thought prompt |
+
+## `context:` buckets
+
+| Matches | Becomes | Also |
+| --- | --- | --- |
+| `*.view.yml`, `*.view.yaml` | semantic catalog | its `datasource` / `data_source` adds a database |
+| `*.topic.yml`, `*.topic.yaml` | semantic catalog | |
+| `*.sql` with an `oxy:` header | verified query, run verbatim when chosen | header `database:` adds a database |
+| `*.sql` without the header | SQL examples in the solving prompt | |
+| `*.md` | domain notes for clarifying and interpreting | |
+| `*.automation.yml/.yaml`, `*.procedure.yml/.yaml` | automations `search_automations` can return | task `database:` values add databases |
+| anything else | ignored | |
+
+Verified-query header fields: `description` (used to match the question) and
+`database` (the connection to run on).
+
+```sql
+/*
+  oxy:
+    description: Weekly sales for all stores
+    database: warehouse
+*/
+```
+
+## Tools by state (fixed; no `tools:` key)
+
+| State | Tools |
+| --- | --- |
+| clarifying | `search_catalog`, `search_automations`, `propose_semantic_query`, `ask_user` |
+| clarifying → root cause / opportunity | `explain_metric`, `find_opportunities`, `metric_sensitivity`, `predict_impact`, `search_catalog`, `search_automations`, `propose_semantic_query`; with anomaly monitoring also `list_anomalies`, `detect_anomalies`, `explain_anomaly` |
+| specifying | `search_catalog`, `sample_columns`, `ask_user`; with a semantic model `check_data_freshness`; without one `get_join_path`, `list_tables`, `describe_table` |
+| solving | `execute_preview` |
+| executing | none |
+| interpreting | `render_chart` |
+
+Root-cause and opportunity questions need a metric tree. Without one the agent
+replies that the workspace "does not have a metric tree configured". Metric-tree
+and anomaly tools are not offered in workspace previews or scheduled runs.
+
+## `validation:`
+
+Absent: the 15 rules marked default run. Present: only the listed rules run (an
+entry with `enabled: false` is skipped), and a stage with no list is not
+checked. Entry shape: `- name: <rule>`, optional `enabled`, plus the rule's
+parameters at the same level. Misspelled parameters are ignored silently.
+
+| Stage (checked after) | Rule | Default | Params (default) | Fails when |
+| --- | --- | --- | --- | --- |
+| `specified` (specifying) | `metric_resolves` | yes | | a resolved metric's table or column is not in the catalog |
+| | `join_key_exists` | yes | | a join's tables or key are not in the catalog |
+| | `filter_unambiguous` | yes | | a filter column is unknown or matches several tables |
+| `solvable` (solving) | `sql_syntax` | yes | `dialect` (`generic`): accepted, but parsing is generic whatever its value | the SQL does not parse |
+| | `tables_exist_in_catalog` | yes | | `FROM`/`JOIN` names a table the catalog lacks |
+| | `spec_tables_present` | yes | | a table the spec resolved is missing from the SQL |
+| | `column_refs_valid` | yes | | a `table.column` reference does not exist |
+| | `timeseries_order_by_check` | yes | | a time series has no outer `ORDER BY` |
+| `solved` (executing) | `non_empty` | yes | | zero rows |
+| | `truncation_warning` | yes | | a scalar or series result hit the row cap |
+| | `no_nan_inf` | yes | | NaN or ±∞ in a numeric cell |
+| | `outlier_detection` | yes | `threshold_sigma` (5.0), `min_rows` (4) | a value is beyond the z-score threshold |
+| | `null_ratio_check` | yes | `threshold` (0.5) | a numeric column's NULL share is above the threshold |
+| | `duplicate_row_check` | yes | `max_duplicate_ratio` (0.1) | the duplicate-row share is above the ratio |
+| | `freshness_check` | yes | `threshold_days` (1) | the requested range runs past the latest date in the result |
+| | `shape_match` | no | | the result doesn't have the expected shape |
+| | `timeseries_date_check` | no | | a time series' first column is not date-like |
+
+Where a failure goes (diagnosing):
+
+| Failure | Back to |
+| --- | --- |
+| unresolved metric or join, empty result, semantic compile error | specifying |
+| SQL syntax, null ratio or duplicates found after solving | specifying (fresh attempt) |
+| ambiguous column | clarifying |
+| outlier, freshness gap, bad chart config | interpreting (answers and states the caveat) |
+
+## `semantic_engine:`
+
+| Field | Cube | Looker |
+| --- | --- | --- |
+| `vendor` | `cube` | `looker` |
+| `base_url` | required | required |
+| `api_token` | used | n/a |
+| `client_id`, `client_secret` | n/a | used |
+
+`${VAR}` is read from the **server's environment**. If the variable is unset,
+or the engine is unreachable at run start, the run fails.
+
+## Commands
+
+| Command | Checks |
+| --- | --- |
+| `oxyc validate --file <path>` | JSON-schema shape, offline. Catches wrong types, a malformed `thinking` and vendor spelling. Misses unknown keys, `ref`, databases, globs, rule names, env vars |
+| `oxy validate --file <path>` | parses with the runtime's own parser. Structural only, with the same blind spots |
+| `oxy agentic run <path> -q "<question>"` | the real build and run, streamed by state. `--json`, `--thinking-mode extended_thinking`. Needs Oxy's Postgres (`OXY_DATABASE_URL`) |
+| `oxy test [<file>.test.yml]` | LLM-judged evals; with no file, every `*.test.yml` in the workspace |
+| `oxy build` | not an agent check: it builds embeddings and syncs integrations |
+
+## `.test.yml` (evals)
+
+```yaml
+target: agents/sales.agentic.yml   # workspace-relative; omit to infer it
+settings:
+  runs: 3                          # runs per case (default 3), used for consistency
+  concurrency: 5                   # default 5
+  judge_model: gpt-5-mini          # config.yml model name; use a different family
+cases:
+  - name: weekly-total             # optional
+    prompt: "What were total sales last week?"
+    expected: "Total sales were $1.2M"
+    tags: [sales, critical]
+```
+
+| Detail | Value |
+| --- | --- |
+| Target inference | `sales.agentic.test.yml` tests `sales.agentic.yml` in the same directory |
+| Target kind | must be an `.agentic.yml` |
+| `tool:` on a case | parsed, not yet checked |
+| Useful flags | `--tag <t>`, `--case <index-or-name>`, `--min-accuracy 0.8` (exit 1 below), `--threshold-mode average\|all`, `--format json`, `--output-json`, `-v` |
+
+## Errors → fix
+
+| Message | Fix |
+| --- | --- |
+| `no databases configured` | name a database in `databases:`, or give context views a `datasource` |
+| `YAML parse error: …` | malformed YAML, a duplicate key, `thinking: false`, or an unknown `semantic_engine.vendor` |
+| `validation config error: unknown <stage> rule: '…'` | fix the rule name or move it to its stage |
+| `validation config error: invalid params for rule '…'` | fix the parameter's type |
+| `invalid timezone '…'` | use an IANA name |
+| `environment variable '${X}' referenced in config is not set` | set it in the server environment |
+| `semantic engine connection error: …` | check `base_url` and credentials |
+| `glob pattern error: …` | fix the `context:` pattern |