@malloy-publisher/skills 0.1.13 → 0.1.17

package/README.md

@@ -53,11 +53,11 @@ skill points to for detail it does not inline. Start with `malloy-getting-starte
 `malloy-review` to check Malloy for correctness.
 
 A few skills (`malloy-modeling`, `malloy-publish`, `malloy-document`, `malloy-getting-started`, and
-the `malloy` index) are written for a Publisher host and name Publisher's `malloy_*` MCP tools
+the `malloy` index) are written for a Publisher host and name Publisher's MCP tools
 directly. The rest describe Malloy itself and refer to tools by bare name (`get_context`,
 `execute_query`, `search_malloy_docs`), because the prefix depends on the
-host. `search_database_schema` maps the same way if a shared skill starts using it. On Publisher those are `malloy_getContext`, `malloy_executeQuery`, `malloy_searchDocs`, and
-`malloy_searchDatabaseSchema`.
+host. `search_database_schema` maps the same way if a shared skill starts using it. On Publisher those are `get_context`, `execute_query`, `search_malloy_docs`, and
+`search_database_schema`.
 
 ## Versioning
 

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.13",
+   "version": "0.1.17",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.13
+version: 0.1.17
 ---
 
 # Evaluate One Answer
@@ -36,7 +36,10 @@ The checklist. An attempt is contaminated when its log shows any of:
    `modelPath` argument on an MCP `execute_query` is NOT contamination; the
    server resolves it, the answerer never reads the file);
 3. `reported_calls` greater than `host_tool_uses` (the detectable
-   under-report floor is reported at most total tool uses).
+   under-report floor is reported at most total tool uses). `host_tool_uses`
+   is EVERY tool use the host logged, MCP calls included; while it counted only
+   the non-MCP ones this comparison was true of almost every clean attempt, so
+   a run whose attempts predate the split cannot be checked this way.
 
 `skills/eval-answer/scripts/check_contamination.py` is a reference aid that
 mechanizes the same checklist over a JSON log; your reading of the transcript
@@ -54,10 +57,23 @@ Never score the agent's reported rows. Take its final query, execute it with
 `execute_query`, and write a prediction CSV under the run's
 `artifacts/` directory.
 
-`submitted: false` when there is no final query. That is not a wrong answer.
-No verdict can be issued (`verdict: null`, with the reason) when the attempt
-is not submitted, when the golden is missing, provisional, invalid, or
-ambiguous, or when a verified golden has no local artifact to compare.
+A named view is a submitted query. `execute_query` takes either ad-hoc Malloy
+or a `queryName` plus `sourceName`, and its own tool description steers an
+answerer to the named form; record it as the Malloy it stands for,
+`run: <source> -> <view>`, which re-executes and reads the same as any other.
+Capturing only the ad-hoc form recorded an attempt that did query as
+`submitted: false` with no query to re-execute, and the judge then graded prose.
+
+`submitted: false` when there is no final query. That is not a wrong answer, and
+it is not by itself a reason to withhold a verdict. No verdict can be issued
+(`verdict: null`, with the reason) when the attempt produced neither a query nor
+any answer text, when the golden is missing, provisional, invalid, or ambiguous,
+or when a verified golden that holds a value has no local artifact to compare.
+A `criteria` golden holds no value and needs no artifact: its clauses are the
+whole comparison, and withholding a verdict for a missing artifact there drops
+a scorable case out of the pass rate. An attempt that
+wrote prose and ran nothing IS judged: against a golden holding a value, an
+answer containing none of it is `no_match` however well it reasons.
 
 ## Step 3: Judge the answer
 
@@ -83,6 +99,18 @@ relevant source and field definitions from the model. It returns
 - Large row sets are still the judge's job. There is no scripted row oracle:
   a script that can pass a wrong answer is worse than none, and the rubric's
   containment and column-pairing rules are what the comparison needs.
+- `golden.mustNotUse` is the exception, and it is not the judge's. It names the
+  similar-but-wrong field, and using one is a failure however good the number
+  looks, which is a question about query TEXT. Run
+  `scripts/check_must_not_use.py` over the final query: a named field found
+  there forces `no_match` and records `must_not_use_hits`, keeping the judge's
+  own verdict beside it as `judge_verdict`. Only a BARE name vetoes. An entry
+  with a connective (`weekly_active_users as a cumulative series`, `product.cost
+  through the order_items join`) objects to a use of the field rather than to
+  the field, so it goes to the judge as prose, as does an entry that names no
+  field at all and a path's bare leaf. A veto that fires on a correct answer is
+  worse than one that misses, and reading the head of `X as ...` as "ban X"
+  failed an answer whose only sin was showing X as an extra column.
 
 ## Step 4: Score what retrieval delivered
 
@@ -103,6 +131,64 @@ this number worth having. It uses the search terms the answerer chose, so it
 attributes a failure *within* an arm and does not compare retrieval across arms
 -- that is the engine-side `eval-retrieval` skill, which does not ship here.
 
+`scripts/check_coverage.py` measures the other half, and it is worth knowing
+which question each answers. Recall asks whether search surfaced the entities a
+case names. Coverage asks whether the model holds the concepts at all, decided
+by READING the model: no answerer, no judge, no goldens, no warehouse. A version
+can score full recall on a case the model could never have answered, and then
+recall is scoring search against an expectation that was never satisfiable.
+
+Because it runs from the model text alone it is cheap enough to point at every
+published version and read as a trend, which is what it is for. Its verdicts are
+`eval-diagnose`'s codes verbatim, validated against that table at startup, so
+`COVERAGE`, `AMBIGUOUS`, `NO-DISAMBIG` and `CONVENTION` mean there exactly what
+they mean here. It writes nothing: not a golden, not the case-level `coverage`
+field, not a run directory. Where its verdict and that field disagree is where a
+version regressed, and the field is the standing judgement about the question
+while this is a measurement against one build.
+
+How to invoke it. Either `--model <file-or-dir>` for a local package or
+`--publisher <url> --package <pkg>` for a served one, plus `--set <dir>`, and
+`--version <label>` to stamp the report so a trend has an x-axis:
+
+```
+python3 check_coverage.py --set evals/ecommerce --model model.malloy --version 0.0.58
+```
+
+Two flags change what the number means, so choose them rather than inheriting
+them. `--repeat N` samples each case N times and takes the majority; it defaults
+to **1 for a set**, because the score is a trend over many cases rather than a
+verdict on one, and to 3 for `--self-check`, where a single fixture is the whole
+measurement. A case that flips between samples is arguable rather than covered,
+so a tie goes to the gap, and two DIFFERENT gaps tying leaves the case undecided
+and out of the denominator. `--self-check` runs the shipped fixture instead of a
+set, which is how to confirm the checker still detects a gap it is known to
+detect; run it after editing the prompt, the verdict vocabulary **or the fixture
+itself**. The tests pin the fixture's shape and cannot pin its verdict, because
+the judge needs a live model, so a fixture swap that goes unmeasured leaves the
+whole metric unguarded.
+
+Undecided cases are excluded from the percentage and reported on their own line.
+Read that line: a coverage number over a handful of decided cases is not a
+measurement, it is a sample size.
+
+**Read `reference/coverage-limits.md` before quoting a coverage number.** Run
+against all 49 hand-labelled cases of the `evals/ecommerce` set it disagreed
+with the authored label on 17, while the two headline percentages landed two
+points apart because the disagreements cancel. Checking each one against the
+model found the LABEL was the stale side more often than the verdict was: four
+notes call a measure missing that the model declares, and one prescribes a
+filter on a field the model documents as "not evidence of a sale". So read a
+coverage number as a rough signal over many cases, never as a verdict on one and
+never as a small movement between versions, and use `--compare-labels` to print
+the disagreements and triage them by reading the model. Agreement with the
+labels is not a score for the checker; the two answer different questions on
+purpose. Separately, the whole model goes into one prompt per case, so past a
+few thousand lines it stops running on Linux and above about 100 KB the verdict
+stops being stable. Do not take the over-size message's advice to narrow
+`--model` to one file, which drops every imported source and manufactures
+`COVERAGE` verdicts.
+
 ## Step 5: Distrust the golden
 
 A reference answer can be wrong (parent-column fanout, a join on a shared
@@ -121,7 +207,7 @@ event unchanged, and where it says nothing, fall back to the case's standing
 Do not encode a rewrite of a bad golden into the model. A `suspect` or
 `verified_wrong`, or a no_match whose why indicts the golden rather than the
 prediction, routes to the golden side door in `skill:eval-loop` as a **dataset**
-issue. It is never a model failure, and it must be settled before improve runs --
+issue, which is where the audit procedure for settling it lives. It is never a model failure, and it must be settled before improve runs --
 otherwise a modelling agent is dispatched to fix a model that is already right.
 
 ## Step 6: Append events, then stop

package/skills/eval-answer/reference/coverage-limits.md

@@ -0,0 +1,215 @@
+<!-- What a coverage number does and does not measure. Read before quoting one. Measured against a labelled set and across model sizes; none of it is visible in the output the script prints. -->
+
+# Coverage limits
+
+`check_coverage.py` was measured against a real labelled set and at a range of
+model sizes. Every number below is measured, and the method is named beside it
+so it can be re-measured when the harness changes. None of it is visible in the
+output the script prints today.
+
+The short version. **Read it as a rough signal over many cases, never as a
+verdict on one, and never as a small movement between versions.** Where it
+disagrees with a set's authored `coverage` label, either side can be the stale
+one, and on the one set measured the LABEL was wrong more often than the verdict
+was. Separately, the whole model goes into one prompt per case, which caps the
+package size it can run on at all and destabilises the verdict well before that
+cap.
+
+## Measured against a labelled set: the label drifts too
+
+The `evals/ecommerce` set in `credibledata/malloy-samples` labels every case
+`covered`, `derivable` or `absent` by hand, and names the entity that would
+close each one. Running all 49 cases against that package's model, `--repeat 1`,
+19,258 chars of model text over three files:
+
+| | |
+|---|---|
+| script | coverage 45%, 22 of 49 |
+| set, counting `covered` | 43%, 21 of 49 |
+| cases where the two disagree | 17 |
+
+**Read the disagreements, not the two-point gap.** The headlines land close
+because the two directions of disagreement nearly cancel, 8 cases the script
+calls a gap that the label calls covered against 9 the other way. A number that
+agrees for that reason can move without the model changing and hold still when
+it does.
+
+**And do not assume the checker is the wrong side.** Every disagreement was
+checked against the model by hand. On the majority the LABEL was stale or
+wrong, because the model gained measures and the standing judgement was not
+revisited:
+
+| case | label says | the model actually declares |
+|---|---|---|
+| `ecom_time_to_ship` | "no measure; shipped_at - created_at" | `avg_days_to_ship is avg(days_to_ship)`, documented |
+| `ecom_gross_margin_pct` | "average_gross_margin is per item, not margin / sales" | `margin_rate is total_gross_margin / nullif(total_sales, 0)` |
+| `ecom_top_margin_pct_categories` | "no margin-pct measure to group by category" | the same `margin_rate`, groupable |
+| `ecom_return_rate` | "no return_rate measure, the model has no rate measure at all" | `percent_purchases_returned`, a returned-over-all ratio |
+
+The reverse direction is the same story with the sides swapped.
+`ecom_unsold_inventory` is labelled `covered` with the note
+"`inventory_item_count`, filtered to unsold", and the checker returns a gap. The
+checker is right: the model's doc on `sold_at` says "Despite the name it is not
+evidence of a sale: it is a copy of order_items.created_at and is set even when
+that order was cancelled", and `inventory_items` declares no join back to
+`order_items`, so the anti-join the question needs cannot be expressed at all.
+The note prescribes exactly the filter the model warns against.
+
+So the honest reading of this run is that it audited the SET as much as the
+model, and that a coverage judgement is only as good as the docs it reads.
+`--compare-labels` exists for this: it joins the verdicts against the authored
+field and prints the disagreements with their notes, taking no position on which
+side is wrong. Check the label first, against the model, before treating a
+disagreement as a coverage regression.
+
+One number does hold up on its own. `absent` came back `COVERAGE` 4 times out
+of 4, so where the model holds nothing at all the verdict is dependable. That is
+also the easiest call in the set.
+
+Do NOT read agreement with these labels as a score for the checker. The two
+answer different questions on purpose, which the script's own docstring says,
+and this run is the evidence that either side can be the one that has gone
+stale.
+
+## It stops RUNNING at about 3,500 lines, on Linux only
+
+Measured separately, on model size rather than on accuracy. Density first,
+measured by concatenating example packages and reading `usage` off a real judge
+call: Malloy text runs **2.70 chars per token** and **37 chars per line**, so a
+10,000-line model is about 370 KB and 136,000 tokens.
+
+The prompt goes in `argv`. Linux caps a single argument string at
+`MAX_ARG_STRLEN`, 131,072 bytes, independently of `ARG_MAX`; measured in
+`python:3.12-slim`, 131,000 bytes execs and 131,072 raises `OSError 7 Argument
+list too long`. macOS has no per-argument cap below its roughly 1 MB total, so
+the same model that works on a developer's laptop fails in CI and in a Linux
+checkout.
+
+`MAX_PROMPT` is 400,000, so it does not catch this: three times the real Linux
+ceiling, and it counts CHARACTERS while `argv` spends UTF-8 BYTES, so a model
+with non-ASCII doc comments passes the guard by a wider margin still. Anything
+over roughly **3,500 lines** clears the guard and then fails the exec.
+
+The failure is shaped like a different bug. `run_cli` retries on `no_text`, so a
+deterministic `E2BIG` is retried with the full backoff and lands as `no reply
+from the model ([Errno 7] Argument list too long)`, once per case, which is what
+a rate limit also looks like. Every case comes back undecided, `coverage` prints
+`n/a`, and the process still **exits 0**.
+
+## It stops being STABLE at about 100 KB, everywhere
+
+The shipped fixture exists to prove the measurement judges expressibility rather
+than matching the question's words against field names. Growing only the
+haystack around it, needle and prompt untouched:
+
+| model text | verdict | |
+|---|---|---|
+| 2.8 KB, the fixture of the day | `CONVENTION` | correct |
+| 103 KB | `ok` | WRONG, reproduced at `--repeat 5` |
+| 373 KB | `CONVENTION` | correct |
+
+**These three rows predate the current fixture and have not been re-measured.**
+They were taken against a 2.8 KB excerpt that was replaced, when a customer
+model was scrubbed out of `check_coverage.py`, by the 1.4 KB `support_desk`
+fixture that ships today. What survives the swap is the SHAPE of the curve --
+the needle was untouched in all three rows and only the haystack grew -- so read
+the 100 KB cliff as the finding and the specific verdicts as history. The
+shipped fixture on its own reads `CONVENTION` 3 of 3 (2026-09-08, sonnet).
+
+At 103 KB the fixture reads `ok`, precisely the field-name matching it was
+written to detect, and the majority of five samples did not rescue it. The
+pattern is instability rather than decay: above roughly 100 KB the verdict stops
+being a function of the model's content, and it drifts toward `ok`, the
+optimistic direction `majority()` is written to avoid. 103 KB is a quarter of
+`MAX_PROMPT` and under a third of a 10,000-line model, so a package can sit
+inside every limit the script enforces and still produce a number that means
+nothing.
+
+One caveat on method: the haystack was example packages repeated with renamed
+sources, so this measures dilution and recall rather than a genuinely distinct
+200-file package. Treat the direction as established and the threshold as
+approximate.
+
+## What it costs
+
+The model text and the question share ONE user message block, so the question,
+which changes per case, sits inside the cached prefix and the model is re-cached
+every call. Measured on two calls with the same 100 KB model and different
+questions: `cache_creation_input_tokens` 37,033 and `cache_read` 0 for the
+model. Moving the model into `--append-system-prompt` and leaving only the
+question in the message gave `cache_read` 50,754 and `cache_creation` 1,159 on
+the second call.
+
+| per case | as written | model cached |
+|---|---|---|
+| 19 KB model (the ecommerce set) | $0.08 | $0.05 |
+| 100 KB model | $0.181 | $0.016 |
+| 370 KB model, 10k lines | $0.55 | $0.08 |
+
+The 49-case run above cost about $5.72 and took about 11 minutes at
+`--parallel 8`. The script reports neither, which for a measurement sold on
+being cheap is its own gap: nothing in the output says what the number spent.
+
+## Do NOT narrow to one file to get under the cap
+
+The over-size message suggests pointing `--model` at a single `.malloy` file. Do
+not, and do not read a coverage number produced that way. `model_text()`
+concatenates every `*.malloy` under the tree and does **not** resolve `import`;
+real packages split across files exactly that way (in
+`credibledata/malloy-samples`, `ecommerce/brand_synergy.malloy` opens with
+`import "ecommerce.malloy"`). Narrowing to one file drops every imported source,
+so the judge correctly answers `COVERAGE`, no entity represents the concept
+anywhere, for cases the model answers perfectly well. That is a false gap
+indistinguishable from a real one, and it lands in the numerator of a published
+trend.
+
+If the model does not fit, the honest outcome is a failed measurement, not a
+measurement of a fragment.
+
+## Where to take it
+
+Ordered by what each fixes.
+
+**Run `--compare-labels` whenever a set carries authored `coverage` labels, and
+triage what it prints.** It is the cheapest audit available on both the set and
+the model, and it needs no extra model calls beyond the run itself. Resolve each
+disagreement by reading the model, and expect to fix labels as often as
+verdicts. What it must NOT become is a score for the checker: agreement is not
+the goal, since the two answer different questions on purpose.
+
+**Do not add a prompt rule on this evidence.** The first reading of that run
+was that the checker had two defects, a named measure plus a filter reading as
+a gap and the prompt's ratio rule going unapplied. Checking each case against
+the model dissolved both: the model documents `sold_at` as NOT evidence of a
+sale and declares no join for the anti-join, and the measures the labels called
+missing exist. A prompt rule written from that first reading would have taught
+the judge to ignore a caveat the model states plainly.
+
+**Pass the prompt on stdin.** `claude -p` reads it there (verified), which
+removes the `argv` ceiling outright. Then `MAX_PROMPT` should bound bytes
+against the agent model's context window rather than characters against a Linux
+limit it does not match.
+
+**Exit 1 when `decided` is 0.** Done: a run that decided nothing now exits 1
+and says so, instead of reporting `n/a` and a success. This is the whole-set
+shape of the `argv` failure above, where every case fails identically.
+
+**Note that the two cheap fixes pull against each other.**
+`--append-system-prompt` is what buys the caching above, and it is also `argv`,
+so on Linux it is stuck under the same 131,072 bytes. There is no
+`--append-system-prompt-file`. Both cannot be had from this CLI surface, which
+is the argument for not making this judge a `claude -p` call at all: it holds no
+tools, is granted a second turn only to absorb a stray tool call, and scans
+braces out of prose because the reply is not structured. A cached system block
+for the model, a volatile user block for the question, and a structured output
+format address the cost, the ceiling and `json_objects()` together. Stdlib
+`urllib` keeps the stdlib-only promise; the real cost is auth, since `claude -p`
+rides the CLI's credentials.
+
+**And for a large package, stop sending the whole model.** Rule out `COVERAGE`
+against a complete inventory of names and one-line docs, a small fraction of the
+bytes, which keeps "no candidate anywhere" decidable; then send full text for
+the candidate sources only, to settle `AMBIGUOUS`, `NO-DISAMBIG` and
+`CONVENTION`. Deliberately NOT retrieval ranking: folding retrieval into this
+number would collapse the distinction from recall that the metric exists to
+draw.

package/skills/eval-answer/reference/ledger-schema.md

@@ -65,8 +65,9 @@ Rules that make the ledger trustworthy:
 | `description` | |
 | `datasetVersion` | Integer. Bump on any golden repair or case change. Runs record the version they scored against. |
 | `targetModelPath` | Model path within the package. |
-| `truthPackage` | Name of the package holding the semantics-free sources every golden is re-derived from -- a package NAME, not an object. `verify_goldens.py` skips every check without it, reporting "nothing to re-derive against", so a set that omits it silently has no golden verification at all. `init_truth_package.py` scaffolds the package. |
+| `truthPackage` | Name of the package holding the semantics-free sources every golden is re-derived from -- a package NAME, not an object. Without it `verify_goldens.py` re-derives nothing: it reports "nothing to re-derive against" and **exits 3**, which a caller must read as unverified rather than as a pass. The checks that need no truth server -- rubric figures, verification axis, stale rubric claims, set names -- still run and still report. `init_truth_package.py` scaffolds the package. |
 | `truthModel` | Model file inside that package. Default `truth.malloy`. |
+| `targetPackage` | Name of the package UNDER TEST. `verify_goldens.py` refuses a "truth" server that also serves it, because an answerer on that server can retrieve the raw truth sources beside the model and quietly change what is under test. `run_baseline.py` and `improve.py` pass the package they were given, so this field is the fallback for a direct call; without either the guard cannot fire. |
 | `truthTableRewrite` | Boolean, default false. Rewrites `duckdb.table('data/x.parquet')` refs to bare `x` in canonical queries, for a truth package whose tables are registered rather than read from files. |
 
 ## `cases.jsonl`
@@ -81,7 +82,8 @@ One JSON object per line:
 | `tags` | list |
 | `state` | `candidate` / `selected` / `excluded`. |
 | `source` | Where the case came from. |
-| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous`), `kind`, `value` or `path` (artifact under the set directory), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`judge.md` rules 9 and 10). |
+| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous` / `verified_wrong`, the last being a key a person has established is wrong, which `run_baseline.py` drops from every aggregate), `kind`, `value` or `path` (artifact under the set directory), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`skill:eval-judge` rules 10 and 11). |
+| `golden.kind` | What the key IS. A value (`scalar`, rows), `unanswerable`, or `criteria`: prose with no value, where the author's criteria are the whole key. `criteria` is the one kind that is legitimately `verified` on arrival, because there is no derivation to perform and nothing that can be numerically wrong; an imported key holding a VALUE is `provisional` until re-derived (`skill:eval-import`). **`verify_goldens.py --promote` is the only thing that writes `golden.status`**: it marks a `provisional` golden `verified` when its value re-derived cleanly from the truth package AND a second derivation exists. `--refresh` rewrites a drifted VALUE and never changes a status; `invalid` and `ambiguous` are settled by a person through the golden side door. |
 | `golden.rubric` | **On the golden, not on the case.** The prose the judge is shown as `CASE RUBRIC`, naming what counts as correct and what does not. `run_baseline.py` reads `golden.rubric`; a rubric written at the case's top level is silently not passed, the judge is told `CASE RUBRIC: none`, and it then scores from the golden value and the answer alone -- which reads as a judge that ignores its instructions. |
 | `golden.mustState` | **On the golden, not on the case.** What the answer has to say out loud. Read from `golden.mustState` by the run-package builder. |
 | `goldenRevision` | Integer. Bump on every golden change; `score` events stamp the revision they compared against. |
@@ -119,20 +121,27 @@ on that axis.
 | `skillsVersion` *(pin)* | HEAD of the checkout the agents' skills were loaded from, dirty-marked (`ledger.skills_git_sha(root)`). The skills are the doctrine the agents load; a run that cannot name their version cannot anchor a skills A/B. |
 | `skillsRoot` / `harnessVersion` | Which checkout supplied the doctrine (`--skills-root`, e.g. a Publisher checkout for the open-source skills; default this one), and this checkout's own HEAD. The eval-* skills always come from the harness checkout, whatever `skillsRoot` says. Two runs whose `skillsRoot` differ are a skills A/B only if their manifests name the same skills. |
 | `diagnoserModel` / `improverModel` | Written by `diagnose.py` / `improve.py` when those stages run, so the run names every LLM that touched it. Absent on a run that was only answered and judged. |
-| `modelGitSha` | Commit of the model repo, when the target served working files; `-dirty` suffix when the tree had uncommitted changes -- fine for a band measurement, not for an A/B pin. Omit for a platform target, where a local commit pins nothing. |
+| `modelGitSha` | Commit of the model repo, from `--model-repo`; `-dirty` suffix when that path had uncommitted changes -- fine for a band measurement, not for an A/B pin. **Null unless `--model-repo` names the repo.** It cannot be inferred: Publisher serves a COPY under `publisher_data/`, so the tree around the served file is the server's storage, not the model's history, and the field named that until 2026-09-03. `modelSha` above is the pin that always works; this is the pointer to where those bytes are versioned, and absent beats wrong. |
+| `modelRepo` | The path `--model-repo` named, so a reader can tell an absent pin from an unrecorded one. |
 | `environment` / `package` / `modelPath` | What was served, and from where. On a platform target `environment` is the organization and `package` is the workspace the MCP URL is scoped to. |
-| `scope` | Platform target only: the `environment/package` the answerer was told to pass as an explicit `scopes` entry on every `get_context` / `execute_query` call. A workspace can serve many packages (a personal workspace serves every package the user can read), so without this the run also measures whether retrieval picks the right package -- a different measurement. Absent means unscoped. |
+| `scope` | Platform target only: the `environment/package[@version]` the answerer was told to pass as an explicit `scopes` entry on every `get_context` / `execute_query` call. A workspace can serve many packages (a personal workspace serves every package the user can read), so without this the run also measures whether retrieval picks the right package -- a different measurement. Absent means unscoped. |
+| `targetVersion` *(pin)* | Platform target only: the published version the answers came from. Both hosted tools default an omitted version to the PINNED one, meaning whatever the workspace serves at the moment of the call, so this only describes the run when `--scope` carries it to every call. `--target-version` supplies it when the scope omits it; a scope naming a different version is refused rather than treated as an override, because a run cannot say which build answered if two pins disagree. |
 | `mcpUrl` / `publisher` | The endpoints the answerer and the re-execution used. |
-| `predictionsReExecuted` | Whether the judge saw re-executed rows (false when the served bytes no longer match the pinned snapshot). |
+| `predictionsReExecuted` | Whether re-execution was POSSIBLE: the server was serving the bytes this run is pinned to. It is one bool for the whole run and says nothing about any given query, so it reads stronger than it is; `reExecution` below is what actually happened. |
+| `reExecution` | `{attempted, ok, failed, noQuery, notReExecuted, notJudged, missing}` over the run's predictions. `failed` means the query ran and the server returned an error, which is a different thing from an answer with no query at all. `notJudged` is a case the judge was never handed: a golden refused before judging (`provisional`, `invalid`, `ambiguous`, `verified_wrong`, missing), an attempt that submitted nothing, or every case under `--no-judge`. Those used to land in `missing` and read as corrupt artifacts on a freshly imported set. `ok`, `failed`, `noQuery`, `notReExecuted`, `notJudged` and `missing` partition the cases, so they sum to the case count; `attempted` is `ok + failed` and sits outside that sum. |
+| `retrievalMode` | Which retriever answered: `semantic`, `lexical`, `mixed` (it changed mid-run, so nothing can be averaged across it) or `unreported` (nothing ranked, which on a configured server means no embedding provider). Only a RANKING call names a retriever, so a call that enumerates a type or looks a name up is `unreported` on a fully semantic server and must not make the run read as `mixed`. Local retrieval degrades to lexical SILENTLY without an embedding key, and a pair compared across that reads as a model change, so `flip_table.py` refuses the pair unless both sides match. |
+| `retrievalCalls` | `{semantic, lexical, unreported}` call counts behind `retrievalMode`. `unreported` counts the calls that did not rank, which is a different thing from calls that fell back. |
+| `retrievalGate` | What the pre-arm retrieval gate found, or why it did not run. A run is held until `get_context` answers semantic twice in a row, because the first calls after a restart are lexical by design and an arm started immediately measures two retrievers. `not run (no answering phase)` under `--rebuild`/`--rejudge`, and `skipped by --no-retrieval-gate` when the operator opted out -- which is a different fact from a gate that passed, and the field is what tells them apart. |
 | `label` | `<set>-<phase>-<nn>`, assigned by `run_baseline.py` from the set name, the run's phase and the next free number beside it (`ecommerce-baseline-01`, `ecommerce-baseline-02`, `ecommerce-blind_gate-01`). The A/A pair is two runs of the same phase; the post-edit arms are two runs of `blind_gate`. Hand-typed names do not survive one afternoon of runs -- `base`, `rejudged2`, `r3`, `post1` sort wrongly, group not at all, and cannot be matched to an arm. `--label` overrides for a run that genuinely needs a human name. |
 | `effort` | |
 | `answererCostUsd` / `judgeCostUsd` | What the arm cost, split by role. The judge's half was discarded until 2026-09-02, so every "cost per arm" quoted before then was the answerer alone. |
-| `goldenCheck` | What `verify_goldens.py` said before the run started: `N ok, M drifted, K other finding(s)`, or why it did not run (no truth server on a platform target; `--skip-golden-check`; rebuild). A run that started on a drifted set says so here rather than pretending its verdicts mean something. |
+| `goldenCheck` | What `verify_goldens.py` said before the run started: `N ok, M drifted, K other finding(s)`, or why it did not run (no truth server on a platform target; `--skip-golden-check`; rebuild). A run that started on a drifted set says so here rather than pretending its verdicts mean something. `run_baseline` calls `verify()` without a model, so the set-name audit (check 5) does not run in-arm; the trailing `N stale entity name(s)` is the in-arm lint's own count, and `entity-name lint not run` says there was no model text to lint against. Without it the field read `0 other finding(s)` for a check that never happened. |
+| `staleEntityNames` | The `expectedEntities` names that appear nowhere in the served model, from the in-arm lint. Each one depresses the run's own recall on every case listing it, so a stale set reads as a retrieval failure until someone checks by hand. `null` when there was no model text to lint against (a platform target), which is a different fact from `[]`. |
 | `status` | `complete`, or `aborted` when four consecutive attempts errored or found the server dead and the harness stopped rather than spend the rest of the budget on attempts nobody will trust. |
 | `packageSha` / `servedRevision` *(pins)* | Taken from the server, not recomputed: `sourceContentSha` is a content hash over EVERY model path in the package, so an edit to an imported file moves it where a sha of the one `--model-path` does not; `servedRevision` is minted per load, so it identifies a load rather than content and is a poor pin alone. Measured: `publisher.json`'s `version` moves neither, and nothing in Publisher reads it -- it is not in the Package API schema and never returned, so it pins nothing. |
 | `datasetSha` *(pin)* | Content hash of `set.json` + `cases.jsonl`. Deliberately SEPARATE from the model's sha: a golden repair is not a model change, and one pin covering both would make every answer-key fix read as an edit to the model, which is the distinction an A/B rests on. Automatic, so nobody has to remember it; `datasetVersion` stays beside it as the human-readable sequence. **Local targets only.** A hosted target that publishes IMMUTABLE versions needs none of this: the set rides inside the version, and immutability -- not hashing -- is what makes a pin trustworthy. There, `targetVersion` alone identifies model and set together. |
-| `doubtedGoldens` | The cases whose golden the judge did not believe: `qid`, `gold_status` (`suspect` or `verified_wrong`), `gold_note`. **Read this before diagnose.** These are dataset issues, not model failures, and they go through the golden side door in `skill:eval-loop`. Empty list when the judge believed every key. Written from the same scan that prints the end-of-run warning, because a warning that lives only in console text is one scrollback away from sending a modelling agent at a model that is already right. |
-| `mode` / `setName` / `targetVersion` / `serverVersion` / `traceMode` / `callBudget` / `status` | Defined and accepted, **not yet written by any harness** -- kept in the schema for the platform target and the conductor, which need them. |
+| `doubtedGoldens` | The cases whose golden is not believed: `qid`, `gold_status` (`suspect` or `verified_wrong`), `gold_note`, and `declaredBy` (`judge` or `set`). A status the SET declared is a key somebody already settled, not an opinion the judge formed this run, and attributing both to the judge claimed a judgement that never happened. **Read this before diagnose.** These are dataset issues, not model failures, and they go through the golden side door in `skill:eval-loop`. Empty list when no key is in doubt from either source. Written from the same scan that prints the end-of-run warning, because a warning that lives only in console text is one scrollback away from sending a modelling agent at a model that is already right. |
+| `mode` / `setName` / `serverVersion` / `traceMode` / `callBudget` | Defined and accepted, **not yet written by any harness** -- kept in the schema for the platform target and the conductor, which need them. (`targetVersion` and `status` were in this list and are both written now.) |
 
 ## Events
 
@@ -152,13 +161,15 @@ exist in the flat shape, so the flat shape is the contract.) `kind` is one of:
 | `qid` | string | |
 | `sample` | int or null | Which repeat. Required even when null. |
 | `phase` | string | `baseline` / `loop` / `blind_gate` / `canary` / `final`. `phase` lives here, on the attempt, not in run config. |
-| `question_sha` | string | Hash of the exact text the answerer saw. |
+| `question_sha` | string | Hash of the exact text the answerer saw, so two runs can be compared on whether they asked the same thing. A case may carry `questionSha`, the seal stamped at conversion, and that wins when present; the ecommerce set writes a 16-char prefix of it, and a set without one left this null. Neither form proves the question was not edited, since both follow the case as it now reads: that comparison is check 6 of `verify_goldens.py`. |
 | `submitted` | bool | False when there was no final query. Not a wrong answer. |
-| `final_query` | string or null | Required to replay. |
+| `final_query` | string or null | Required to replay. A named view is recorded as the Malloy it stands for, `run: <source> -> <view>`, so every consumer sees one shape and the query re-executes. |
+| `final_query_source` | string or null | How `final_query` was chosen: `declared` (the answer printed it), `last_ok` (the last call the server answered) or `last`. `last` is a warning: a trailing sanity probe may be standing in for the answer's own query. |
 | `servedRevision` | string or null | From the package actually queried. |
 | `n_get_context` / `n_execute` / `n_execute_errors` | int | |
-| `host_tool_uses` | int | Host-side count, including Read and Shell. |
-| `reported_calls` | int | MCP calls the answerer claimed. |
+| `host_tool_uses` | int | EVERY tool use the host logged, MCP calls included. It counted only the non-MCP ones until 2026-09-03, which made the under-report check below true of almost every clean attempt. |
+| `mcp_tool_uses` | int or null | The MCP subset: `get_context` plus `execute_query`. Null on a run written before the split. |
+| `reported_calls` | int | MCP calls the answerer claimed. `reported_calls > host_tool_uses` is the under-report floor from `skill:eval-answer`: the answerer cannot have made more calls than the host logged. `validate_run` warns on it, and only for runs that carry `mcp_tool_uses`, because the field meant something narrower before. |
 | `contaminated` | bool or `"unknown"` | `"unknown"` when no host log exists. Read it with `ledger.is_contaminated`, never for truthiness: runs written before 2026-09-03 carry the strings `"true"`/`"false"`, and `bool("false")` is True. `ledger.event` now rejects those strings on write; `validate_run` grandfathers them on read as a warning. |
 | `contamination_reasons` | list | Empty when clean. |
 | `input_tokens` / `output_tokens` / `cache_read_tokens` | int or null | Answerer token usage. Null when the host does not report it. |
@@ -182,6 +193,9 @@ One event per MCP `get_context` or `execute_query` the attempt made.
 | `traceId` | string or null | `get_context` only; look up in your host's trace store. |
 | `targets` | string, list or null | **What the answerer asked for**: the search terms it sent to `get_context`. Null for `execute_query`. |
 | `rankedSummary` | object | Copied at capture from the trace so evidence survives trace eviction: `entityIds`, `ranks`, `resultCount`, and per-target `targets` with within-target ranks. |
+| `retrieval_mode` | string or null | `get_context` only: the `retrieval` field of the response that answered, `semantic` or `lexical`. Null when the server named none, which means no embedding provider. Recorded per call because the semantic path can fall over partway through a run. |
+| `query` | string or null | `execute_query` only: the Malloy the call sent, so the final query can be chosen by which call the server actually answered. |
+| `modelPath` | string or null | `execute_query` only: the model file the call named. A source does not resolve outside the file that declares it, so re-execution needs this. |
 | `error` | string or null | |
 
 Never persist `execute_query` result rows, givens, or credentials.
@@ -208,19 +222,38 @@ The answer judge's verdict for one attempt (protocol in
 | Field | Type | Notes |
 |---|---|---|
 | `verdict` | string or null | `match` / `near_match` / `no_match` / `needs_human`; null when the attempt is not scorable. Only `match` and `no_match` are decisions; see below. |
-| `reason` | string | Why, from the judge; for a null verdict, why not scorable (`not_submitted`, `golden_missing`, `golden_ambiguous`, `contaminated`). |
+| `reason` | string | Why, from the judge; for a null verdict, why not scorable: `not_submitted`, `contaminated`, or `golden_<status>` -- `golden_missing`, `golden_provisional`, `golden_invalid`, `golden_ambiguous`, `golden_verified_wrong`. The `golden_*` reasons are decided before the judge is called, so an unestablished key costs nothing to refuse. |
 | `confidence` | int or null | 1 to 10. Confidence of 5 or lower forces `needs_human`. |
 | `column_pairing` | object or null | The judge's named gold-to-prediction column correspondence. |
 | `judge_version` / `rubric_sha` | string | Pins which rubric produced this verdict. |
 | `golden_revision` | int | From the case at score time. |
 | `contaminated` | bool or `"unknown"` | Copied from the attempt; true or unknown means `verdict: null`. |
+| `must_not_use_hits` | list or null | Entries of `golden.mustNotUse` a script found in the final query. A hit forces `verdict: no_match`. Decided by `check_must_not_use.py`, never by the judge: it is a question about query text. Only a BARE name vetoes; an entry with a connective (`X as ...`, `X through ...`) objects to a use of X rather than to X, and goes to the judge with the prose entries and a path's bare leaf. |
+| `judge_verdict` | string or null | What the judge said when a `must_not_use_hits` veto overrode it. Null otherwise, so the judge's own agreement rate stays measurable across vetoes. |
 | `artifactPath` | string | The full judge output under `artifacts/`. |
-| `gold_status` | string | `verified` / `verified_benign` / `suspect` / `verified_wrong`. **From the judge**, which scored against the golden as written and reports separately whether it believes it; falls back to the case's standing `golden.status` when the judge does not say. `verified_wrong` excludes the case from run aggregates. `suspect` and `verified_wrong` route to the golden side door as `dataset` issues, never to improve. |
+| `gold_status` | string | `verified` / `verified_benign` / `suspect` / `verified_wrong`, and never anything else: the case file's `golden.status` is a DIFFERENT vocabulary that shares only `verified` and `verified_wrong`, and only those two fall back into this field. A `provisional` / `invalid` / `ambiguous` key is reported through the verdict's `reason` (`golden_<status>`) instead, so nothing downstream has to match on two vocabularies in one field. **From the judge**, which scored against the golden as written and reports separately whether it believes it; falls back to the case's standing `golden.status` when the judge does not say, written onto the verdict itself so the run summary and the aggregates see it too. `verified_wrong` excludes the case from run aggregates -- from the judge OR from the case file; assigning the fallback into a copy of the verdict meant a key the set had marked wrong stayed in the aggregates unless the judge independently agreed. `suspect` and `verified_wrong` route to the golden side door as `dataset` issues, never to improve. |
 | `gold_note` | string or null | The judge's evidence for a non-`verified` status: the two values, or the model line against the rubric sentence. Null when `verified`. |
 
-A `submitted: false` attempt gets `verdict: null, reason: "not_submitted"`,
-except for an `unanswerable` golden, where a refusal that names the gap is the
-pass and a confident numeric answer is the fail.
+A `golden_*` reason means the ANSWER KEY is not something a verdict can be
+issued against. A golden the set stamps `provisional`, `invalid` or `ambiguous`
+is refused, as is a case with no golden at all and one the set marks
+`verified_wrong`; a golden with NO `status` is unguarded rather than
+unscorable, because sets that predate the field would otherwise score zero of
+zero. A golden holding no value is not refused either: that is how a
+`kind: unanswerable` case and a `kind: criteria` golden are written. This
+matters most for an imported set, because `skill:eval-import` stamps
+`provisional` on every golden it writes and enforces it -- so a set is
+unscorable by construction until its keys are re-derived, and `run_baseline`
+refuses to spend at all when every case in a set is in that state.
+
+`not_submitted` means the attempt produced NOTHING to judge: no answer text and
+no query. An attempt that wrote prose without querying **is judged**, and
+against a golden that holds a value an answer containing none of it is
+`no_match` (`skill:eval-judge`, `reference/refusal.md` rule 10). Excusing those
+as unscorable dropped a confident refusal on an answerable case out of the pass
+rate, which is the one thing the answerable-sounds-unanswerable cases exist to
+measure. A refusal is exempt only where `golden.kind` is `unanswerable`, and
+there naming the gap is the pass and a confident number is the fail.
 
 Aggregates count decided verdicts only. `match` and `no_match` are the pass and
 the fail; **`near_match`, `needs_human` and null are none of the above** and stay

package/skills/eval-answer/scripts/check_contamination.py

@@ -21,7 +21,7 @@ The log is a JSON object:
     "toolUses": [
       {"name": "Read", "input": {"path": "..."}},
       {"name": "Shell", "input": {"command": "..."}},
-      {"name": "malloy_getContext", "input": {}}
+      {"name": "get_context", "input": {}}
     ],
     "reportedCalls": 4
   }
@@ -148,7 +148,7 @@ def check(
         name = _tool_name(entry) or f"use[{i}]"
         hay = _haystack(entry)
         hits = _matches(hay, needles)
-        # modelPath on malloy_executeQuery is required, not a file read --
+        # modelPath on execute_query is required, not a file read --
         # including when the MCP call rides over Shell as a curl JSON-RPC.
         if model_needles and _is_host_file_tool(name) and not _is_mcp_transport(hay):
             hits.extend(_matches(hay, model_needles))