@oxy-hq/cli 0.7.0 → 0.9.0

package/skills/oxy-evals/SKILL.md

@@ -0,0 +1,362 @@
+---
+name: oxy-evals
+description: >-
+  Use when writing, running, or acting on evals for an Oxy agent — `.test.yml` files that run an
+  `.agentic.yml` agent and score its answers with an LLM judge — and when the agent's answers are
+  wrong. Triggers: "add a test", "write a test case", "write evals", "test this agent", "evaluate
+  my agent", "bootstrap an eval suite", "fill in expected answers", "run the tests"; and the
+  repair side: "tests are failing", "this test is flaky", "the agent's answer is wrong", "the
+  answer should be X but the agent says Y", "debug this eval", "make this test pass", or pasted
+  `oxy test --output-json` output or a prompt/response pair to diagnose. Drafts expected answers
+  from observed runs, then diagnoses failures and repairs the agent's semantic model, context and
+  instructions — never the test.
+---
+
+# Oxy evals: draft, run, diagnose, repair
+
+An Oxy eval is a `.test.yml` file: a list of prompts, each with the answer a correct agent
+gives. `oxy test` runs the target agent on every prompt several times and asks an LLM judge
+whether each answer carries the expected facts.
+
+Drafting the tests and repairing the agent are one loop:
+
+1. **Draft** cases and their `expected` answers from evidence.
+2. **Run** them and read the results.
+3. **Diagnose** each failure and **repair the agent** until it passes reliably.
+
+**The hard rule: the expected answer is the specification. Never weaken a test to make it
+pass.** Don't loosen `expected` to match what the agent said, delete or skip a failing case, or
+rewrite a prompt to dodge the failure. If a test itself is wrong (a bad expected value, an
+ambiguous prompt), say so, show the evidence, and let the user decide. That's a different
+change from a repair, and you don't make it silently.
+
+Reference docs (field tables, CI usage):
+<https://www.oxygen-hq.com/docs/guide/build/test-evaluate/testing>
+
+## What can be tested
+
+- **Only `.agentic.yml` agents.** `oxy test` rejects any other target, including classic
+  `.agent.yml` agents (that runtime was removed). Inline `tests:` blocks inside agent or
+  automation files are gone too. A `.test.yml` file is the only entry point.
+- **Only the final answer text is judged.** The judge sees the question, `expected`, and the
+  agent's answer. It doesn't see the SQL, the tool calls, or the rows.
+- **`tool:` on a case is accepted but not enforced.** It has no effect on pass/fail. Don't
+  rely on it to assert that the agent used a particular path.
+
+## The `.test.yml` format
+
+```yaml
+name: Sales performance          # optional display name
+target: agents/sales.agentic.yml # path from the project root; see "target" below
+
+settings:
+  runs: 3                        # times each case runs (default 3)
+  concurrency: 5                 # agent runs in flight at once (default 5)
+  judge_model: judge             # a model `name:` from config.yml (default: first model there)
+
+cases:
+  - name: total-revenue-2025     # optional, but always add it: it makes --case usable
+    prompt: What was total revenue across all stores in 2025?
+    expected: Total revenue across all stores in 2025 was approximately $6.7 billion.
+    tags: [revenue, critical]
+```
+
+Rules the parser enforces:
+
+- **Unknown keys are rejected at every level** (file, `settings`, each case). A typo such as
+  `judge-model:` or `expect:` makes the file fail to parse. Run with an explicit file and you
+  get `Failed to deserialize test config …`. **Run `oxy test` with no file and a broken file
+  is skipped silently.** It just isn't in the run, with no error. After editing a test file,
+  run it by path once.
+- **`prompt` and `expected` are required strings.** Everything else is optional.
+- **`target`** is resolved from the project root (the directory holding `config.yml`). If you
+  omit it, it's inferred from the filename, in the same directory: `sales.agentic.test.yml`
+  targets `sales.agentic.yml` beside it. Tests kept in a separate `tests/` folder must set
+  `target` explicitly.
+- **`judge_model` names a model by its `name:` under `models:` in `config.yml`.** It isn't a
+  vendor model ID. If you leave it out, the judge is the **first** model listed in
+  `config.yml`. An unknown name fails the run before any judging (`LLM model '<name>' could not
+  be resolved from project config`). Google/Gemini models can't be used. Prefer a judge from a
+  different model family than the agent's, to avoid self-preference bias.
+
+Naming and placement: `<agent-stem>.<topic>.test.yml` (for example
+`sales.performance.test.yml`) in the folder the project already uses for tests. Discovery
+(`oxy test` with no file) finds every `*.test.yml` in the project, except under hidden
+(dot-prefixed) directories and `target/`, `node_modules/`, `dist/` or `build/`.
+
+## When to write evals
+
+- Before changing an agent, a semantic model, or instructions, so there's a baseline to
+  compare against.
+- When a user reports a wrong answer. Turn the prompt into a case first, then repair.
+- For the questions the business actually asks, and for the edges: a question the agent must
+  decline, a ranking, a trend, a filter on a rare value.
+
+## Drafting cases
+
+### Prompts
+
+- **Use absolute periods.** The agent resolves "last month" or "this week" against *today*
+  in its `timezone:` (UTC when unset). The expected answer for a relative prompt goes stale
+  the next month. Write "November 2025", not "last month".
+- **Make prompts specific enough to answer without a follow-up question.** Tests run
+  headless. If the agent stops to ask the user something, the run ends as an **error**
+  (`agentic pipeline asked a clarifying question: …`), which counts as a FAIL. An ambiguous
+  prompt shows up as a run error, not as a wrong answer.
+- **Test one thing per case.** "Revenue by region and the top product in each" is two cases.
+- **Name every case** (`name:`), with a short stable slug.
+
+### Scaffolding with placeholders
+
+When the right answer isn't known yet, scaffold the cases with a placeholder and fill it in
+from evidence:
+
+```yaml
+expected: "DRAFT: to be synthesized from repeated runs."
+```
+
+A placeholder case's verdict means nothing, because the judge has no facts to check. Never
+report a suite as passing while any `DRAFT:` remains in it.
+
+### Writing `expected`
+
+`expected` is what the judge checks the answer against. The judge's instruction is to PASS an
+answer that contains the **core factual content** of `expected`. It ignores formatting,
+phrasing, and level of detail, and it allows extra correct information. So write the facts
+that must be present, and nothing that may legitimately vary.
+
+- State the facts in plain business language, 1–4 sentences. Don't start with "The response
+  should…". `expected` is an answer, not a grading instruction.
+- **Approximate numbers** (`~`, "about", "roughly") unless an exact value is reliably stable.
+  The judge compares facts, and a cent-level total invites false failures.
+- **Rankings:** name the criterion once, then the top 2–3 entities with anchor values. Don't
+  list every row.
+- **Trends:** the direction, the current value, and one or two notable points.
+- **Questions the agent can't answer:** state why, and what a correct answer offers instead.
+- No internal field, view or schema names unless the user would say them.
+
+```text
+Good: Total revenue across all locations in 2025 is approximately $6.7 billion.
+
+Good: The top 5 items by first-week engagements in February 2026: #1 had ~25K engagements
+      and #5 had ~10K. Each entry gives the title, channel and engagement count.
+
+Good: Regional plan attainment can't be calculated because targets are set at the company
+      level only. A correct answer says so and offers company-level attainment instead.
+
+Bad:  The response should explain total revenue across all stores.      (grading instruction)
+Bad:  Total revenue is $6,737,218,004.32.                               (false precision)
+Bad:  Sum of store__weekly_sales across all store__store_id values.     (schema names)
+Bad:  Store A $73.3M, Store B $275M, Store C $402K, …                   (exhaustive list)
+```
+
+More worked examples, and the evidence rules for each kind of question:
+[references/writing-expected.md](references/writing-expected.md).
+
+## Running
+
+Run from the project root (the directory holding `config.yml`). The file argument is resolved
+from your current directory, and `target` from the project root.
+
+```bash
+oxy test tests/sales.performance.test.yml                                  # one file
+oxy test tests/sales.performance.test.yml --case total-revenue-2025 --output-json
+oxy test tests/sales.performance.test.yml --case 0 --output-json           # by 0-based index
+oxy test tests/sales.performance.test.yml --tag critical --output-json     # by tag
+oxy test                                                                   # every *.test.yml
+oxy test --min-accuracy 0.8 --quiet                                        # CI gate, exit 1 below
+```
+
+- `--case` takes a 0-based index, a case `name`, or the exact prompt. It requires a file. With
+  `--tag` as well, a case must match both. If two cases share a name, the first one runs.
+- `--tag` takes one tag.
+- `--output-json` writes the full results next to the test file, replacing `.yml` with
+  `.results.json` (`tests/sales.performance.test.results.json`). With no file argument it
+  writes `test-results.json` in the current directory. `--format json` prints the same JSON
+  to stdout instead of the pretty report.
+- `-v` adds each failing run's actual output to the pretty report. `-q` shows only the
+  summary.
+- `--min-accuracy` compares **per-file** scores. `--threshold-mode average` (the default)
+  checks the mean across files, and `all` checks every file. It doesn't check individual
+  cases.
+- **Cost:** every case runs `runs` times through the full agent, plus one judge call per run,
+  with `concurrency` runs against the warehouse at once. When you iterate, run one case.
+
+## Reading results
+
+The pretty report gives one line per case:
+
+| Status | Meaning |
+| --- | --- |
+| `PASS (3/3)` | every run passed |
+| `FLKY (1/3)` | some runs passed. The answer is unstable, which is a real defect |
+| `FAIL (0/3)` | no run passed |
+
+The file's **Correctness** score is the mean over every case × run. Each run scores 1 (PASS)
+or 0 (FAIL). There's no partial credit.
+
+In the JSON, each run is a record with `prompt`, `expected`, `actual_output`, `cot` (the
+judge's full response), `choice` (`PASS`/`FAIL`) and `score`. The JSON doesn't include the
+SQL or the tool calls. Three things to check first:
+
+- **`actual_output` starting with `[ERROR]`** is a run that never produced an answer: a
+  clarifying question, `max iterations exceeded`, a connector failure, a timeout. Its error
+  text is also in the top-level `errors` array. Errored runs are appended at the **end** of
+  `records`, not next to their case, so group records by `prompt`.
+- **An answer with no numbers, or rows whose metric values are all empty,** is a failed
+  attempt, even if the judge passed it.
+- **A `cot` that argues PASS while `choice` is `FAIL`** is usually a judge parse problem, not
+  an agent bug. See the judge category in the diagnosis reference.
+
+The full schema, `jq` recipes, and how to see what the agent actually queried:
+[references/results-json.md](references/results-json.md).
+
+## Filling expected answers from evidence
+
+1. Run the case, or the whole file if several placeholders need filling, with
+   `--output-json`. Re-run if a run failed on a transient error. Up to three runs in total is
+   reasonable.
+2. For each case, compare `actual_output` across every run and classify it:
+
+| Class | Evidence | What to write |
+| --- | --- | --- |
+| `stable` | 2+ runs agree on the business facts | a full expected, numbers approximated |
+| `stable_but_partial` | ranking or direction agrees, values vary | the ranking plus 2–3 anchors |
+| `flaky` | runs disagree materially | only the facts that agree. Flag the rest for repair |
+| `ambiguous` | runs answer different readings of the prompt | propose a sharper prompt to the user |
+| `cannot_answer_as_asked` | runs consistently decline, with a valid reason | encode the reason and the alternative offered |
+| `likely_unsupported` | no run finds the data | leave a narrowed `DRAFT:` note. This is a gap to report |
+
+3. **Check every number you're about to write down.** An observed answer isn't automatically a
+   correct one. If you can, verify it independently (for example, run the agent's SQL or a
+   simple query yourself). **Never encode a clearly wrong answer as `expected` just because
+   it was stable.** A consistently wrong agent is a repair, not a spec.
+4. Replace the placeholders, turn relative periods into absolute ones, and report:
+
+```text
+## Eval drafting summary
+File: tests/sales.performance.test.yml   Cases: 9   Drafted: 6   Partial: 2   Flaky: 1
+| Case                 | Class              | Notes                        |
+| total-revenue-2025   | stable             | 3/3 runs agree, ~$6.7B       |
+| top-items-feb-2026   | stable_but_partial | ranking stable; values vary  |
+Follow-up: <cases to repair, prompts to sharpen, data gaps>
+```
+
+## Diagnose and repair
+
+When a case fails or is flaky, the agent is the thing under repair. The test stays fixed,
+apart from mechanical edits such as adding a missing `name:`.
+
+### 1. Reproduce
+
+```bash
+oxy test <file> --case <name> --output-json
+```
+
+Read every run of that case: `expected`, `actual_output`, `cot`, `choice`.
+
+### 2. See what the agent did
+
+The results JSON has only the final answer. To see the path, run the same prompt on its own
+and read the events:
+
+```bash
+oxy agentic run <path/to/agent.agentic.yml> -q "<the case's prompt>" --json
+```
+
+This streams the run's events as JSON lines and needs `OXY_DATABASE_URL`. Look at
+`query_executed`: its `query`, its `rows`, and its `source`. `source` is `semantic` (compiled
+by the semantic model), `llm` (SQL the model wrote), `verified_sql` (a Verified Query), or
+`vendor`. Also look at `analytics_validation_failed`, the checks that bounced the agent. If
+the project exports traces to Oxy's observability, the run's trace shows the same steps.
+
+### 3. Classify the root cause before editing anything
+
+| Category | Typical evidence | Fix lives in |
+| --- | --- | --- |
+| A. Semantic model gap: a grouping or dimension is missing | many queries, raw rows, the agent aggregates in prose | a dimension on the view |
+| B. Missing or unfindable measure | `source: llm` where a measure should have answered, or arithmetic done in the answer | a measure, or its name and description |
+| C. Instructions vague or misdirecting | wrong interpretation, varying definitions across runs | `instructions:` or a `.md` context doc |
+| D. Canonical answer exists as SQL | the same question keeps producing different SQL | a Verified Query |
+| E. Validation rejects a correct result | `analytics_validation_failed` loops, `max iterations exceeded` | `validation:` |
+| F. Judge false negative | `cot` agrees with the expected answer, `choice` is FAIL | report it. Don't touch the agent |
+| G. Prompt underspecified | clarifying-question errors, different readings across runs | report it, and propose a sharper prompt |
+| H. Data unavailable | no view or table holds it | report it. Don't fake support |
+
+Symptoms, evidence, and a worked fix for each:
+[references/diagnosis.md](references/diagnosis.md).
+
+### 4. Repair: levers in an `.agentic.yml` agent, in this order
+
+1. **Semantic model** (`.view.yml` / `.topic.yml` matched by the agent's `context:` globs):
+   add the missing measure or dimension, or fix an aggregation or an entity. The agent's
+   catalog search matches a measure's **name**, its **description** and its **view name**. It
+   doesn't read `synonyms:`, so the words users ask with belong in the description. A view
+   that no `context:` glob matches doesn't exist as far as the agent is concerned.
+2. **Context** (`context:` globs, relative to the agent file):
+   - **`.md` docs** go into the prompts for clarifying the question and writing the answer.
+     Use them for business definitions: the fiscal calendar, what "active customer" means,
+     which of two similar measures to use.
+   - **Verified Queries**: a `.sql` file with a `/* oxy: … */` comment block at the top that
+     carries a `description:`. When the agent's search matches its file name or description,
+     the SQL runs verbatim, with no SQL generation. A `.sql` file **without** that block is
+     only a style example for SQL generation, and is never run as an answer.
+3. **Instructions:** top-level `instructions:` reaches every LLM call. `states.<state>.
+   instructions:` reaches one stage: `clarifying`, `specifying`, `solving` or `interpreting`.
+   Write general rules ("revenue means net of refunds", "rank by units unless asked
+   otherwise"). Never put the expected answer, or a number from it, into instructions.
+4. **Validation:** the `validation:` block **replaces** the 15 default rules rather than
+   adjusting them. To switch off or retune one rule, list every rule that should still run.
+   An empty `validation: {}` disables them all.
+
+Make the smallest fix that generalizes. If the same logical gap exists in several views, fix
+all of them. Preserve the file's existing style. Explain the root cause alongside the edit.
+
+### 5. Validate
+
+There's no build step. `oxy test` reloads the agent file, its context and its semantic files
+from disk on every run, and `oxy build` (embeddings and integration sync) plays no part. Check
+the files you edited, then re-run:
+
+```bash
+oxy validate agents/sales.agentic.yml
+oxy validate semantics/views/orders.view.yml
+oxy test <file> --case <name> --output-json
+```
+
+**One pass isn't proof.** A case needs `runs`/`runs` on at least two rounds. Then run the
+whole file to catch regressions in other cases. If the repair doesn't hold after three
+diagnose-edit-validate cycles, stop and report what you know.
+
+### 6. Report
+
+```text
+## Repair summary
+Case: tests/sales.performance.test.yml › top-items-feb-2026
+Root cause: B — the measure existed, but its description never said "engagements"
+Changed: semantics/views/posts.view.yml (description of total_engagements)
+Validation: round 1 3/3 · round 2 3/3 · full file 9/9 cases
+Notes: <judge observations, remaining risks, anything left for the user>
+```
+
+## Never
+
+- Edit `expected`, delete or skip a case, or reword a prompt to make a failure go away.
+- Encode an observed answer you haven't checked, or a wrong one because it was consistent.
+- Fabricate an expected value. A narrowed `DRAFT:` note is honest, an invented number isn't.
+- Hard-code an answer, or a threshold derived from the test data, into instructions or a view.
+- Treat a judge false negative as an agent bug, or a clarifying-question error as a wrong
+  answer.
+- Stop at one lucky pass, or make "while I'm here" edits to unrelated files.
+- Explain a root cause you didn't find. "Unclear, needs investigation" is a valid finding.
+
+## Read next
+
+- [references/writing-expected.md](references/writing-expected.md): worked `expected` examples
+  by question type, and the evidence rules.
+- [references/results-json.md](references/results-json.md): the results schema, `jq`
+  recipes, and how the judge's verdict is parsed.
+- [references/diagnosis.md](references/diagnosis.md): the failure taxonomy, with a repair
+  pattern for each.
+- The `oxy-semantic-modeling` skill, for view, measure and entity craft when the repair is in
+  the semantic model.

package/skills/oxy-evals/references/diagnosis.md

@@ -0,0 +1,243 @@
+# Diagnosing and repairing a failing eval
+
+Diagnose before editing. For every failing or flaky case, collect:
+
+1. Every run's `actual_output`, `cot` and `choice` from the results JSON
+   ([results-json.md](results-json.md)).
+2. The route the agent took, from `oxy agentic run <agent> -q "<prompt>" --json`. Run it two
+   or three times if the case is flaky. Read `query_executed.source` and `query`, and any
+   `analytics_validation_failed` events.
+3. The files the agent actually loads: the `context:` globs in its `.agentic.yml`, resolved
+   relative to that file's directory. A view, doc or SQL file outside those globs doesn't
+   exist for this agent.
+
+Then name one category below. If two apply, fix the earlier one first. Most failures are
+**A** or **B**.
+
+## A. The semantic model lacks a grouping or dimension
+
+**Evidence:** several queries per run, `source: llm` with long hand-written SQL, raw
+row-level results, the answer doing arithmetic or bucketing in prose, and runs that disagree
+because each one improvises the grouping differently.
+
+**Fix:** add a dimension that names the grouping, so one semantic query returns a small,
+finished result.
+
+```yaml
+dimensions:
+  - name: order_size_band
+    type: string
+    description: >-
+      Order size band: Small (under $50), Medium ($50–$200), Large (over $200). Use to compare
+      small, medium and large orders or baskets.
+    expr: |
+      CASE
+        WHEN order_total < 50 THEN 'Small'
+        WHEN order_total <= 200 THEN 'Medium'
+        ELSE 'Large'
+      END
+    samples: ["Small", "Medium", "Large"]
+```
+
+The thresholds must be **business definitions**, not numbers read off the current data. A
+cut-off derived from today's average drifts as the data moves, and a cut-off derived from the
+test's expected answer is hard-coding the answer. If the same grouping makes sense in several
+views, add it to each of them.
+
+## B. A measure is missing, wrong, or can't be found
+
+**Missing.** The answer computes a ratio, rate or average itself, and the arithmetic varies
+across runs. Add the measure:
+
+```yaml
+measures:
+  - name: avg_order_value
+    type: number            # `number`: the expression already aggregates
+    description: >-
+      Average order value (AOV): net revenue per order. Also called average basket or
+      average ticket.
+    expr: "SUM(net_revenue) / NULLIF(COUNT(DISTINCT order_id), 0)"
+```
+
+**Wrong.** The runs agree on a number that is consistently wrong. Check the measure's
+aggregation (`sum` where `count_distinct` was meant), a join that fans out and double-counts,
+the wrong date column (record-created rather than order date), and the measure's `filters:`.
+Fix the definition, not the answer.
+
+**Can't be found.** The measure exists but the agent never uses it (`source: llm`, or "I
+don't have that data"). The analytics agent's catalog search matches a measure's `name`, its
+`description` and its view's name, and **does not read `synonyms:`**. Put the user's words
+(the ones in the failing prompt) into the measure's `description`.
+
+**A dimension that exists but never gets used.** Check three things: its view is matched by
+`context:`; the views are linked by entities (a `primary` entity on one, a `foreign` entity
+with the same key on the other); and its `description` says what it is in the user's terms.
+For view, measure and entity craft, see the `oxy-semantic-modeling` skill.
+
+**Two similar members, chosen differently on each run** (for example `created_date` against
+`order_date`). Make the descriptions decisive:
+
+```yaml
+  - name: order_date
+    type: date
+    description: Date the order was placed. Use for sales, revenue and order trends by date.
+  - name: created_date
+    type: date
+    description: When the database row was written. System metadata, not for business questions.
+```
+
+## C. Instructions are vague, missing, or misdirect
+
+**Evidence:** the agent picks a different reasonable definition on each run (gross or net,
+calendar or fiscal), formats or scopes the answer inconsistently, or ignores a business
+convention.
+
+**Fix, by where the knowledge belongs:**
+
+- **A definition or a business fact** goes in a `.md` doc matched by `context:`. Docs are
+  injected into the prompts that clarify the question and write the answer.
+- **A rule for every step** goes in top-level `instructions:`.
+- **A rule for one stage** goes in `states.<state>.instructions`, for one of `clarifying`
+  (reading the question), `specifying` (choosing measures and filters), `solving` (writing
+  SQL) or `interpreting` (writing the answer).
+
+```yaml
+instructions: |
+  "Revenue" means net revenue (after refunds and discounts) unless the user says gross.
+states:
+  clarifying:
+    instructions: |
+      Read "last quarter" as the most recent completed fiscal quarter; the fiscal year starts
+      on 1 February.
+  interpreting:
+    instructions: State the period and every filter applied in the answer's first sentence.
+
+context:
+  - ../semantics/**/*.view.yml
+  - ../semantics/**/*.topic.yml
+  - ./docs/*.md
+```
+
+Keep instructions general and reusable. A sentence that only makes sense for one test case
+("the answer to the Q4 question is $1.2M") is hard-coding the test. It's forbidden, and it
+fails the next question anyway.
+
+## D. One canonical query should answer the question every time
+
+**Evidence:** an important, frequently asked question whose correct SQL is known and fixed,
+but the agent writes different SQL each run.
+
+**Fix:** a Verified Query, a `.sql` file matched by `context:` that carries an `oxy:` comment
+block:
+
+```sql
+/*
+  oxy:
+    description: "Current active headcount by department"
+    database: warehouse        # optional: a database name from config.yml
+*/
+SELECT department, COUNT(*) AS active_headcount
+FROM hr.employees
+WHERE status = 'active'
+GROUP BY department
+```
+
+When the agent's search matches the file's name or `description`, the SQL runs **verbatim**.
+Nothing is substituted into it, so it only fits questions whose answer is exactly that query.
+It's no use for "…in a given month". A `.sql` file without the `oxy:` block is only a style
+example for the SQL the model writes.
+
+## E. Validation rejects a correct result
+
+**Evidence:** `analytics_validation_failed` events repeating on a result that was right, runs
+that end in `max iterations exceeded`, or answers that drift away from the right one after a
+retry. Common causes: `non_empty`, when "none" is a correct answer; `outlier_detection` on
+naturally spiky data; and `freshness_check` on a source that loads late.
+
+**Fix:** retune the rule. The `validation:` block **replaces** the default set, so list every
+rule that should still run. A stage you leave out isn't checked at all, and a misspelled rule
+name stops the agent from loading. The 15 defaults:
+
+```yaml
+validation:
+  rules:
+    specified:
+      - name: metric_resolves
+      - name: join_key_exists
+      - name: filter_unambiguous
+    solvable:
+      - name: sql_syntax
+      - name: tables_exist_in_catalog
+      - name: spec_tables_present
+      - name: column_refs_valid
+      - name: timeseries_order_by_check
+    solved:
+      - name: non_empty
+        enabled: false               # "none" is a valid answer for this agent
+      - name: truncation_warning
+      - name: no_nan_inf
+      - name: outlier_detection
+        threshold_sigma: 4.0         # looser than the default for spiky daily data
+        min_rows: 12
+      - name: null_ratio_check
+      - name: duplicate_row_check
+      - name: freshness_check
+```
+
+Loosen only the rule the evidence names. Validation is what stops wrong answers. Don't switch
+it off to make a test pass.
+
+## F. Judge false negative (not an agent problem)
+
+**Evidence:** the answer is correct when you read it, the `cot` reasoning says so, and
+`choice` is still `FAIL`. Usually the verdict line is missing from the last five lines of the
+judge's response, or it reads `PASS or FAIL`, which parses as FAIL.
+
+**Fix:** don't touch the agent. Report it with the `cot` excerpt. Recommend a stronger
+`judge_model`: a model that already exists in `config.yml`, preferably from a different family
+than the agent's. Changing the test's settings is the user's call.
+
+A false **positive** (an answer that is wrong but PASSed) is the dangerous one. It usually
+means `expected` is too vague to contradict anything. Report it, and propose a sharper
+`expected` to the user. Don't change it yourself.
+
+## G. The prompt is underspecified
+
+**Evidence:** runs that end in `agentic pipeline asked a clarifying question: …`, or runs
+that answer different reasonable readings of the prompt.
+
+**Fix:** report the readings you saw. The test owner chooses between two remedies:
+
+- **Sharpen the prompt.** This is a change to the test, so the user decides.
+- **Teach the agent a default reading**, through instructions or a doc, when the business
+  wants one (for example, "when no region is named, report all regions"). That's a legitimate
+  agent repair.
+
+## H. The data isn't there
+
+**Evidence:** no view or table holds the needed facts, and every run declines or answers
+around it.
+
+**Fix:** say so plainly, and name the missing source. Don't add semantic or instruction hacks
+that make the agent sound as if it can answer.
+
+## Also check
+
+- **Dates off by a day, or the wrong month.** The agent resolves "today" and relative dates
+  in its `timezone:` (UTC when unset). Set it to the business's zone.
+- **The right number for the wrong period.** The prompt is relative ("last month") and
+  `expected` was written for an earlier month. Report it: the test needs an absolute period.
+- **Flaky runs with different `source` values.** Some runs take the semantic model and others
+  write SQL. Make the semantic path win: make the measure findable (B), make the dimensions
+  reachable (B), or add a Verified Query (D).
+
+## Order of repair
+
+1. Semantic model: measures, dimensions, entities, descriptions.
+2. Context: `.md` definitions, Verified Queries, and the `context:` globs themselves.
+3. Instructions: global first, then per-state.
+4. Validation tuning.
+
+After each change, `oxy validate <edited file>`, then re-run the case at least twice
+(`oxy test <file> --case <name> --output-json`), then the whole file. Nothing needs to be
+rebuilt between edits. Stop after three cycles that don't hold, and report.