@malloy-publisher/skills 0.1.20 → 0.1.22

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.20",
+   "version": "0.1.22",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.20
+version: 0.1.22
 ---
 
 # Evaluate One Answer
@@ -123,11 +123,16 @@ a returned source's documentation -- text the answerer reads and acts on. Only
 `missing` is a retrieval miss; the route per entity is recorded so the strict
 count is still there.
 
-Recall 1.0 with a wrong answer exonerates retrieval: the failure is in the query.
+Recall 1.0 with a wrong answer exonerates retrieval: everything arrived. Whether
+the agent misused it or the docs never said how to use it is `eval-diagnose`'s
+call, sufficiency first, so the row reads `delivered, wrong` and names no owner.
 Recall below 1.0 and `coverage: covered` means the entity existed and search did
-not surface it; `derivable` or `absent` means there was nothing to surface. Those
-look identical in an answer score and have opposite owners, which is what makes
-this number worth having. It uses the search terms the answerer chose, so it
+not surface it -- a documentation finding, because the retrieval algorithm is
+fixed (semantic search over doc strings) and an entity that exists and does not
+come back is one whose docs do not say what people ask; `eval-diagnose` calls it
+`NOT-RETURNED`, owner model. `derivable` or `absent` means there was nothing to
+surface. Those look identical in an answer score and have different owners,
+which is what makes this number worth having. It uses the search terms the answerer chose, so it
 attributes a failure *within* an arm and does not compare retrieval across arms
 -- that is the engine-side `eval-retrieval` skill, which does not ship here.
 
@@ -155,6 +160,13 @@ How to invoke it. Either `--model <file-or-dir>` for a local package or
 python3 check_coverage.py --set evals/ecommerce --model model.malloy --version 0.0.58
 ```
 
+Then hand the report back to the run: `run_baseline.py --coverage <report>`.
+Its per-case verdict beats the case's authored `coverage` label in retrieval
+attribution, `run.json` records which report was read, and each retrieval row
+says whether `measured`, `authored` or `none` charged the failure. Without a
+report, a case with no label is attributed to nobody rather than to the model,
+which is what used to happen.
+
 Two flags change what the number means, so choose them rather than inheriting
 them. `--repeat N` samples each case N times and takes the majority; it defaults
 to **1 for a set**, because the score is a trend over many cases rather than a
@@ -189,6 +201,34 @@ stops being stable. Do not take the over-size message's advice to narrow
 `--model` to one file, which drops every imported source and manufactures
 `COVERAGE` verdicts.
 
+## Validate the definitions, not every answer
+
+A golden must not be derived through the definitions the question TESTS. It may
+reuse everything the question does not turn on -- joins, base sources, date
+handling -- because circularity only bites where the key and the answer share the
+step under measurement. That is what makes this affordable on a model too large
+to reimplement.
+
+`scripts/verify_definitions.py` checks each definition once against the layer
+directly beneath it, and every case depending on it inherits the result. A golden
+is trustworthy if it was derived independently, OR if every definition it tests
+has itself been validated.
+
+Two things it will not do, and both matter more than what it does. It never
+reports a definition reaching through a join as validated, because fanout
+inflates the measure and the control expression equally and the comparison stays
+green on a broken join. And building the ledger without a server exits 3, not 0:
+nothing was checked, and a caller must not read that as a pass.
+
+A raw check -- a definition that reaches through a join -- is an authored
+control: a person writes the population down as a query on the ledger record and
+the tool re-runs it every time. It is not found unaided, and the docs say so.
+`verify_goldens.py --definitions <ledger>` then applies the composition rule at
+the gate: a set with no truth package exits 0 when every value-bearing case's
+tested definitions are validated, and names the cases that are not.
+
+**Read `reference/definition-ledger.md` before building or quoting one.**
+
 ## Step 5: Distrust the golden
 
 A reference answer can be wrong (parent-column fanout, a join on a shared

package/skills/eval-answer/reference/definition-ledger.md

@@ -0,0 +1,305 @@
+<!-- Validating a model's definitions once, so answer keys need not be re-derived per case. Read before building or reading a ledger. -->
+
+# The definition ledger
+
+## Why it exists
+
+The loop's standing rule is that a golden must never be derived through the model
+under test, because a model bug would otherwise certify its own golden. That rule
+is sound, and it does not scale: on a model with real transformation, deriving
+every key independently means reimplementing the model. You end up with two
+implementations and no reason to trust either, and the eval task becomes *rebuild
+the model to verify the model*.
+
+The ledger is the other half. **A definition is checked once, against the layer
+directly beneath it, and every case depending on it inherits the result.** Work
+scales with the number of definitions rather than the number of cases, and stops
+growing as cases are added.
+
+The rule this replaces:
+
+> A key must be independent of **the definitions the question tests**, not of the
+> whole model.
+
+And the rule that makes that sound:
+
+> A golden is trustworthy if it was derived independently, **or** if every
+> definition it tests has itself been validated.
+
+## Deep models are cheaper per case, not dearer
+
+The tractability argument is about definitions being **shared**, not about models
+being shallow. An earlier draft claimed every measure sits one or two hops from a
+raw column; measured across four models, that is false:
+
+| Model | Definitions | within 2 hops | deepest chain |
+|---|---|---|---|
+| ecommerce (sample) | 32 | 90.6% | 3 |
+| storefront (sample) | 19 | 94.7% | 3 |
+| a 492-definition benchmark | 492 | 100% | 2 |
+| a real 548-definition customer model | 548 | **42.5%** | **12** |
+
+On that customer model, the twenty deepest measures close over **812**
+definition-checks with no reuse and **116** with the ledger: an 86% saving.
+Reuse pays off *more* as chains deepen, because deep chains share their lower
+layers. Depth is what makes the ledger worth having.
+
+## What a record holds
+
+```json
+{
+  "entityId": "measure:order_items:total_sales",
+  "kind": "measure",
+  "source": "order_items",
+  "name": "total_sales",
+  "expr": "sale_price.sum()",
+  "exprSha": "3f9a…",
+  "depends": ["measure:order_items:order_item_count"],
+  "check": {"kind": "within_model", "slice": "first 200 rows"},
+  "verdict": "agrees",
+  "needs": null
+}
+```
+
+`exprSha` is a **Merkle chain**, not a hash of the one line: it covers the
+definition and everything beneath it. A measure whose own text never moved is
+still stale once something it builds on changes, and a flat hash would report it
+as current. `packageSha` exists already but is whole-package, so it invalidates
+everything on any edit and distinguishes nothing.
+
+## The two check kinds
+
+### `within_model` -- no raw access, no truth package
+
+Ask the model for the entity and for the expression it **claims** to be, in one
+query, and compare:
+
+```malloy
+run: order_items -> {
+  aggregate:
+    stated is total_sales
+    control is sale_price.sum()
+}
+```
+
+Both sides come from **one** query against one source, so they see identical
+rows: a difference is the definition, not a moving population.
+
+A dimension is scalar, and `aggregate:` rejects it outright ("Cannot use a scalar
+field in an aggregate"), so a dimension is compared row by row over a slice. The
+slice is recorded, because a slice is weaker evidence than a whole table and the
+report must not imply otherwise.
+
+This catches a definition that is not what it says: a filter nobody mentioned, a
+renamed column, the wrong aggregate, or a served model that has moved away from
+the text the ledger recorded.
+
+### `raw` -- needs a lens onto the base tables
+
+**A within-model check is blind to fanout.** A join that duplicates rows inflates
+the stated measure and the control expression equally, and the comparison stays
+green on a genuinely broken join. So an expression reaching through a join records
+`unchecked` with a reason, **never `agrees`** -- reporting it as validated would be
+the tool committing the exact error it exists to find.
+
+The exception is aggregates that survive uniform duplication: `AVG`, `STDDEV`,
+`MIN`, `MAX`. Those stay checkable across a join.
+
+Detection is by join **name**, not by spotting a dot. `sale_price.sum()` is a
+method call on a column in the same source and is perfectly checkable;
+`inventory_items.cost.sum()` traverses a join and is not. A first implementation
+matched any `word.word` and held back `total_sales` -- the measure twelve of the
+ecommerce set's cases depend on -- for a join it does not cross.
+
+## A raw check is a control a person wrote
+
+An earlier version of this page, and the plan behind it, said raw checks would
+"catch a definition wrong about the business, like the cogs case." **That is
+retracted.** Re-deriving `total_gross_margin`'s own expression over the base
+tables -- reconstructing the model's join from its declared key -- computes the
+same quantity, cancelled lines included, and agrees. Mechanically that catches
+only fanout from a mis-declared join, which Malloy's symmetric aggregates already
+mostly prevent. It cannot see that the population is wrong.
+
+What caught the cogs case was a person who read the `status` doc ("Cancelled
+lines never ship and are not sales") and wrote `where: status != 'Cancelled'`
+into the golden's query. That is an **authored assertion**. So a raw check is
+exactly that, made durable: the conductor writes a control query onto the ledger
+record, and `verify_definitions.py` re-runs and compares it on every rebuild.
+
+```json
+{
+  "entityId": "measure:order_items:total_gross_margin",
+  "check": {
+    "kind": "raw",
+    "against": "model",
+    "query": "run: order_items -> { aggregate: control is sale_price.sum() { where: status != 'Cancelled' } - inventory_items.cost.sum() { where: status != 'Cancelled' } }",
+    "note": "excludes Cancelled, per status's own doc and units_sold"
+  },
+  "cause": "CONVENTION"
+}
+```
+
+A control may be authored on **any** measure record, not only a `raw` one, and
+it supersedes the within-model check. That check can only say a measure is what
+its expression says; a person can say what its population should be. `total_sales
+is sale_price.sum()` agrees with itself while including the cancelled lines its
+own docs exclude -- the within-model check reads `agrees`, and it is the measure
+twelve of the sample set's cases depend on. A one-line control on it is the
+whole difference.
+
+The control returns one row with a `control` column. `against: model` runs it on
+the same server as the measure and is the default -- it reuses the model's joins
+and columns and asserts only the population. `against: truth` runs it on a truth
+package of raw tables (`--truth-publisher`, `--truth-package`) for a fully
+independent derivation, join included. The measure's own value is always fetched
+through the model, separately, because the control is *supposed* to differ in
+population and may run elsewhere.
+
+Authoring a control is judgement, and the same rules as authoring a golden apply:
+where two populations are both defensible from the docs, do not pick one --
+record the ambiguity in `note` and leave the record `unchecked`. `cause` is the
+`eval-diagnose` code the author expects a disagreement to mean (the cogs case is
+`CONVENTION`); it is carried onto the finding, not inferred.
+
+This is **validate what someone asserted**, not **find business-wrong definitions
+unaided**. It turns "I believe `total_gross_margin` should exclude cancelled
+lines" into a checked, re-runnable assertion with a sha, so the belief is tested
+on every rebuild instead of once in someone's head. The difference is the whole
+point, and the run report must not blur it.
+
+### Build once, then maintain
+
+`--out` writes a fresh ledger. `--ledger <file>` reads an existing one, keeps
+every field a person wrote (`check.query`, `check.against`, `check.note`,
+`cause`), re-parses the model, re-runs every check, and writes it back. Nothing
+else is carried: a verdict is never inherited from a definition that may have
+moved, and a record whose definition the model no longer declares is dropped,
+control and all -- a control for a measure that does not exist is not a
+validated measure.
+
+## What it will not claim
+
+Every one of these records `unchecked`, and a reader must treat `unchecked` as
+"nobody has established this", never as "fine":
+
+| Situation | Why not checked |
+|---|---|
+| reaches through a join, and no control has been authored | fanout moves both sides of a within-model check equally, and nobody has written down what the population should be |
+| spans more than one line | only the first line was read, so the recorded expression is a fragment |
+| the query failed | a check that could not run is not a check that passed |
+| no rows came back | nothing to compare |
+| every sampled row null on both sides | proves nothing about the definition |
+| no enclosing source | nothing to query it against |
+
+## Exit codes
+
+Matching `verify_goldens.py`, deliberately:
+
+| | |
+|---|---|
+| **0** | every check that could run ran, and agreed |
+| **1** | a definition disagrees with its own stated expression |
+| **3** | could not run: no model, no server, or a ledger built and never checked |
+
+**3 is load-bearing.** Building the ledger without `--publisher` returns 3, not 0:
+it validated nothing, and a caller must never read that as a pass.
+
+## Running it
+
+```bash
+# build the ledger only (exits 3: nothing was checked)
+python3 verify_definitions.py --model ecommerce.malloy --out evals/definitions/ecommerce.jsonl
+
+# build and check
+python3 verify_definitions.py --model ecommerce.malloy \
+  --publisher http://localhost:4811 --environment samples \
+  --package ecommerce --model-path ecommerce.malloy \
+  --out evals/definitions/ecommerce.jsonl
+```
+
+Measured on the ecommerce sample: **28 definitions, 15 agree, 13 unchecked**, of
+which 4 need a raw check. The remainder are multi-line definitions and fields
+declared inside a nested block, which the line scanner attributes to the
+enclosing source and cannot address.
+
+## What a run reports
+
+`run_baseline.py --definitions <ledger>` adds an EVIDENCE block saying what the
+pass rate rests on, per case:
+
+```
+EVIDENCE
+  basis         5 independent, 8 definitions, 36 unchecked
+                ! those cases rest on a definition nobody has validated. Not a
+                  failure, and not a pass either.
+```
+
+| basis | meaning |
+|---|---|
+| `independent` | the key was established outside the model: two differently shaped derivations agree, or it holds no value at all (`criteria`, `unanswerable`) |
+| `definitions` | the key went through the model, and every definition it tests is validated |
+| `unchecked` | at least one tested definition is unvalidated, stale, or absent from the ledger |
+| `disagrees` | a definition the case depends on contradicts its own expression, so the model is wrong before the answer is |
+
+Independence is read from `golden.verification` (or `gold/<qid>.json`) through
+`verify_goldens.has_second_derivation` -- the structured record of "two
+differently shaped derivations agree". **Not from `verifiedBy`,** which is free
+text: a first implementation matched it by prefix and classified 34 goldens of
+the ecommerce set as `unchecked` when every one of them reads "authored and
+re-derived against ecommerce-truth". A well-founded set reading as unvalidated
+is the same over-claim as an unfounded one reading as validated, pointed the
+other way.
+
+Which definitions a case tests comes from `expectedEntities`, which already
+names entities in the `kind:source:name` form the ledger keys on. No new
+hand-maintained field: a wrong entity id is the failure mode that cost a real
+set two days, so this reuses a link the set already maintains and that
+`verify_goldens` check 5 audits against the model.
+
+**Without `--definitions` there is no EVIDENCE block at all.** Absent is a
+different fact from checked, and a reassuring empty block would be the
+over-claim this whole direction exists to remove.
+
+Staleness is a hash comparison against the run's own `model.malloy` snapshot --
+the bytes that actually answered -- so it costs nothing and runs every arm.
+
+## The composition rule at the gate
+
+`verify_goldens.py` used to exit 3 -- "could not run" -- on any set that named no
+truth package, because no golden value could be re-derived. That was right when
+re-derivation was the only evidence there was. With a ledger it is one of two:
+
+> A golden is trustworthy if it was derived independently, **or** if every
+> definition it tests has itself been validated.
+
+`verify_goldens.py --definitions <ledger>` applies the second half. For each
+value-bearing case it takes the entities the case names in `expectedEntities` --
+the same ids the ledger is keyed on -- and asks whether every one is `agrees` in
+the ledger and not stale against `--model`. If every case passes, the absent truth
+server is not a check that failed to happen, and the audit exits 0 with a line
+saying what validated it; `--promote` may then promote a `provisional` golden that
+also carries a second derivation, and `verifiedBy` names the ledger. If any case
+does not pass, the exit stays 3 and the message names the cases and the
+definitions that let them down. Nothing is inferred about a case that names no
+entities: it is unvalidated, and it says so.
+
+`improve.py` passes `--definitions` through when it has one, so the acceptance
+check can pass on a set that never had a truth package.
+
+## Where it goes
+
+`evals/definitions/<package>.jsonl` -- keyed by **package, not by set**, because
+definitions belong to the model and outlive any one set, and several sets against
+the same package share them. Never under the served package tree; the golden side
+door forbids eval artifacts there.
+
+## What this is not
+
+It is not a unit test. Publisher's roadmap plans fixture-based unit tests (a CSV
+per source plus an expected result) alongside data tests in `malloy-pub lint`.
+A fixture unit test proves an expression computes what you wrote -- so it would
+**not** catch a measure whose expression is correct and whose *population* is
+wrong, which is the defect class this exists for. The ledger is a data test, and
+it should hand off to `lint` when that governance work lands rather than becoming
+a permanent parallel mechanism.

package/skills/eval-answer/reference/ledger-schema.md

@@ -87,7 +87,7 @@ One JSON object per line:
 | `golden.rubric` | **On the golden, not on the case.** The prose the judge is shown as `CASE RUBRIC`, naming what counts as correct and what does not. `run_baseline.py` reads `golden.rubric`; a rubric written at the case's top level is silently not passed, the judge is told `CASE RUBRIC: none`, and it then scores from the golden value and the answer alone -- which reads as a judge that ignores its instructions. |
 | `golden.mustState` | **On the golden, not on the case.** What the answer has to say out loud. Read from `golden.mustState` by the run-package builder. |
 | `goldenRevision` | Integer. Bump on every golden change; `score` events stamp the revision they compared against. |
-| `expectedEntities` | `required`: entity ids (`kind:source:name`) the answer cannot be produced without. `requiredAnyOf`: a list of **groups**, each a list of ids of which any one suffices -- for a case the model can answer through more than one route. Naming only one route scores the other as a retrieval miss and steers diagnosis to "retrieval ranking" for a failure that was never retrieval's. `acceptable`: ids that are not noise if returned. An entity counts as delivered when returned under its exact id, under the same type and name on a sibling source, or by name in a returned source's documentation; `score_retrieval.py` records the route per entity as `delivery`. |
+| `expectedEntities` | `required`: entity ids (`kind:source:name`) the answer cannot be produced without. `requiredAnyOf`: a list of **groups**, each a list of ids of which any one suffices -- for a case the model can answer through more than one route. Naming only one route scores the other as a retrieval miss and steers the fix to the docs for an entity whose docs were never the problem. `acceptable`: ids that are not noise if returned. An entity counts as delivered when returned under its exact id, under the same type and name on a sibling source, or by name in a returned source's documentation; `score_retrieval.py` records the route per entity as `delivery`. |
 
 ## Term files (`eval-retrieval` only)
 
@@ -116,6 +116,7 @@ on that axis.
 | `answererModel` *(required)* | |
 | `phase` / `started` *(required)* | |
 | `judgeModel` / `judgeVersion` / `rubricSha` *(pins)* | The judge's model, and the version + content sha of `skill:eval-judge`. |
+| `judgePromptSha` *(pin)* | Content sha of the prompt the judge is SHOWN, which is the harness's own template and not the skill file `rubricSha` covers. The template carries the golden rendering, the evidence blocks and the tie-breaking rules, so two runs whose prompts differed compared as the same judge -- the blind spot that let a golden-rendering defect run through three arms with every test passing, because the tests covered the code paths and not the text the model sees. Absent on runs written before it existed. |
 | `datasetVersion` *(pin)* | From `set.json` at run time. |
 | `modelSha` *(pin)* | Content sha of the `model.malloy` snapshot in the run directory -- the bytes the answerer actually queried. A git sha is not enough: a snapshot host serves a copy, often of a dirty tree no commit names. |
 | `skillsVersion` *(pin)* | HEAD of the checkout the agents' skills were loaded from, dirty-marked (`ledger.skills_git_sha(root)`). The skills are the doctrine the agents load; a run that cannot name their version cannot anchor a skills A/B. |
@@ -132,12 +133,16 @@ on that axis.
 | `retrievalMode` | Which retriever answered: `semantic`, `lexical`, `mixed` (it changed mid-run, so nothing can be averaged across it) or `unreported` (nothing ranked, which on a configured server means no embedding provider). Only a RANKING call names a retriever, so a call that enumerates a type or looks a name up is `unreported` on a fully semantic server and must not make the run read as `mixed`. Local retrieval degrades to lexical SILENTLY without an embedding key, and a pair compared across that reads as a model change, so `flip_table.py` refuses the pair unless both sides match. |
 | `retrievalCalls` | `{semantic, lexical, unreported}` call counts behind `retrievalMode`. `unreported` counts the calls that did not rank, which is a different thing from calls that fell back. |
 | `retrievalGate` | What the pre-arm retrieval gate found, or why it did not run. A run is held until `get_context` answers semantic twice in a row, because the first calls after a restart are lexical by design and an arm started immediately measures two retrievers. `not run (no answering phase)` under `--rebuild`/`--rejudge`, and `skipped by --no-retrieval-gate` when the operator opted out -- which is a different fact from a gate that passed, and the field is what tells them apart. |
+| `coverageReport` | The `check_coverage.py --out` report this run read, when `--coverage` named one: its path, the model version it was stamped with, its `agentModel`, and `decided` of `cases`. Coverage is a read of the MODEL, so it is measured separately and costs a judge call per case; the run does not run it, it consumes it. Its per-case verdict then beats the authored `coverage` label in retrieval attribution, and each `score_retrieval` row records `coverage_source` (`measured`, `authored`, `none`) so nobody has to guess which one charged the failure. Absent when no report was given, which is a different fact from a report that found everything covered. |
 | `label` | `<set>-<phase>-<nn>`, assigned by `run_baseline.py` from the set name, the run's phase and the next free number beside it (`ecommerce-baseline-01`, `ecommerce-baseline-02`, `ecommerce-blind_gate-01`). The A/A pair is two runs of the same phase; the post-edit arms are two runs of `blind_gate`. Hand-typed names do not survive one afternoon of runs -- `base`, `rejudged2`, `r3`, `post1` sort wrongly, group not at all, and cannot be matched to an arm. `--label` overrides for a run that genuinely needs a human name. |
 | `effort` | |
-| `answererCostUsd` / `judgeCostUsd` | What the arm cost, split by role. The judge's half was discarded until 2026-09-02, so every "cost per arm" quoted before then was the answerer alone. |
+| `maxTurns` / `answererTimeout` *(pins)* | The answerer's turn cap and per-attempt timeout, as the arm ran them. `skill:eval-loop` step 6 lists the call budget among the pins to freeze for a whole run, and nothing wrote one, so no two runs could be compared on the setting that decides whether a case got to finish at all. Distinct from `callBudget`, which no harness has ever written. Size the cap from a pilot rather than taking the default: on a 29-case arm at the default 30, completed attempts had a median of 17 turns and a 90th percentile of 26, and the four cases that ran past it were recorded as wrong answers. |
+| `truncated` / `contaminated` | The qids excluded for a HARNESS-owned reason: the cap cut the answer off, or the attempt breached isolation. Both leave the denominator, both are named here rather than only in the console, and either one makes `status` `incomplete`. |
+| `answererCostUsd` / `judgeCostUsd` | What the arm cost, split by role. The judge's half was discarded until 2026-09-02, so every "cost per arm" quoted before then was the answerer alone. Per-case judge cost is on each `score` event. |
+| `answererCostCopiedFrom` | The run whose attempts `answererCostUsd` was COPIED from, when this run spawned no answerer -- a `--rejudge`, a `--rebuild`, a `--from`. Those copy the source's attempt events verbatim, `cost_usd` included, so the figure is right about the attempts and wrong as this run's spend: one re-judge reported $17.56 of answering that never happened. Null when this run paid, so a total over several run files can skip the copies. |
 | `goldenCheck` | What `verify_goldens.py` said before the run started: `N ok, M drifted, K other finding(s)`, or why it did not run (no truth server on a platform target; `--skip-golden-check`; rebuild). A run that started on a drifted set says so here rather than pretending its verdicts mean something. `run_baseline` calls `verify()` without a model, so the set-name audit (check 5) does not run in-arm; the trailing `N stale entity name(s)` is the in-arm lint's own count, and `entity-name lint not run` says there was no model text to lint against. Without it the field read `0 other finding(s)` for a check that never happened. |
 | `staleEntityNames` | The `expectedEntities` names that appear nowhere in the served model, from the in-arm lint. Each one depresses the run's own recall on every case listing it, so a stale set reads as a retrieval failure until someone checks by hand. `null` when there was no model text to lint against (a platform target), which is a different fact from `[]`. |
-| `status` | `complete`, or `aborted` when four consecutive attempts errored or found the server dead and the harness stopped rather than spend the rest of the budget on attempts nobody will trust. |
+| `status` | `complete`; `aborted` when four consecutive attempts errored or found the server dead and the harness stopped rather than spend the rest of the budget on attempts nobody will trust; or `incomplete` when the arm ran to the end and still cannot report a rate, because `truncated` or `contaminated` is non-empty. `incomplete` is a third state, not a soft `complete`: the arm did everything asked of it and the number is still unearned. It does NOT stop the run being compared: `flip_table.py` scores the cases both arms did, and names the ones this arm left unscored -- an excluded case carries no verdict, so it can never be counted as a flip. |
 | `packageSha` / `servedRevision` *(pins)* | Taken from the server, not recomputed: `sourceContentSha` is a content hash over EVERY model path in the package, so an edit to an imported file moves it where a sha of the one `--model-path` does not; `servedRevision` is minted per load, so it identifies a load rather than content and is a poor pin alone. Measured: `publisher.json`'s `version` moves neither, and nothing in Publisher reads it -- it is not in the Package API schema and never returned, so it pins nothing. |
 | `datasetSha` *(pin)* | Content hash of `set.json` + `cases.jsonl`. Deliberately SEPARATE from the model's sha: a golden repair is not a model change, and one pin covering both would make every answer-key fix read as an edit to the model, which is the distinction an A/B rests on. Automatic, so nobody has to remember it; `datasetVersion` stays beside it as the human-readable sequence. **Local targets only.** A hosted target that publishes IMMUTABLE versions needs none of this: the set rides inside the version, and immutability -- not hashing -- is what makes a pin trustworthy. There, `targetVersion` alone identifies model and set together. |
 | `doubtedGoldens` | The cases whose golden is not believed: `qid`, `gold_status` (`suspect` or `verified_wrong`), `gold_note`, and `declaredBy` (`judge` or `set`). A status the SET declared is a key somebody already settled, not an opinion the judge formed this run, and attributing both to the judge claimed a judgement that never happened. **Read this before diagnose.** These are dataset issues, not model failures, and they go through the golden side door in `skill:eval-loop`. Empty list when no key is in doubt from either source. Written from the same scan that prints the end-of-run warning, because a warning that lives only in console text is one scrollback away from sending a modelling agent at a model that is already right. |
@@ -172,10 +177,11 @@ exist in the flat shape, so the flat shape is the contract.) `kind` is one of:
 | `reported_calls` | int | MCP calls the answerer claimed. `reported_calls > host_tool_uses` is the under-report floor from `skill:eval-answer`: the answerer cannot have made more calls than the host logged. `validate_run` warns on it, and only for runs that carry `mcp_tool_uses`, because the field meant something narrower before. |
 | `contaminated` | bool or `"unknown"` | `"unknown"` when no host log exists. Read it with `ledger.is_contaminated`, never for truthiness: runs written before 2026-09-03 carry the strings `"true"`/`"false"`, and `bool("false")` is True. `ledger.event` now rejects those strings on write; `validate_run` grandfathers them on read as a warning. |
 | `contamination_reasons` | list | Empty when clean. |
-| `input_tokens` / `output_tokens` / `cache_read_tokens` | int or null | Answerer token usage. Null when the host does not report it. |
+| `input_tokens` / `output_tokens` / `cache_read_tokens` / `cache_write_tokens` | int or null | Answerer token usage, the four counts that reprice a run from its own ledger. Null when the host does not report it. `cost_usd` beside them comes from the CLI's `total_cost_usd`, which prices cache reads and writes at their rates, so it was always cache-inclusive; what was missing was the breakdown that reproduces it. `cache_write_tokens` (`cache_creation_input_tokens` upstream) was never captured before 2026-09-15, and on one analysed run cache writes were 44% of the agent's cost, the single largest line. **Read `input_tokens` with care:** it counts only the tokens after the last cache breakpoint. One run recorded 224 for 24 questions beside 2.4M cache reads; the 224 is not the context volume, and a token total that omits the cache columns is not a size. |
 | `cost_usd` | float or null | Answerer cost for this attempt. |
 | `num_turns` / `wall_seconds` | int, float or null | |
 | `answer_text` | string or null | The answer the judge scored. Kept so a verdict can be re-read without the transcript. |
+| `skills_invoked` | list of string | The skills the answerer invoked through the `Skill` tool, deduplicated and sorted, counted from its own transcript. Always a list on a run that recorded it, so an EMPTY list is the finding: the arm granted a manifest and the agent reached for none of it. Absent on runs written before 2026-09-15, which is why every reader takes `or []`. Only explicit `Skill` calls are counted, and only skills the arm granted -- one outside `--answerer-skills` is a contamination breach and lands in `contamination_reasons` instead. What it is for: a manifest names what an arm GRANTED, and comparing two arms' doctrine is meaningless if the doctrine never loaded. |
 | `transcriptPath` | string | The answerer's transcript under `artifacts/`. |
 
 Token counts sit here rather than being derived later because the claim a
@@ -192,7 +198,8 @@ One event per MCP `get_context` or `execute_query` the attempt made.
 | `tool` | string | `get_context` or `execute_query`. |
 | `traceId` | string or null | `get_context` only; look up in your host's trace store. |
 | `targets` | string, list or null | **What the answerer asked for**: the search terms it sent to `get_context`. Null for `execute_query`. |
-| `rankedSummary` | object | Copied at capture from the trace so evidence survives trace eviction: `entityIds`, `ranks`, `resultCount`, and per-target `targets` with within-target ranks. |
+| `rankedSummary` | object | Copied at capture so evidence survives trace eviction: `entityIds` (identity, in response order), `hits` (one row per entity: `entity_id`, `source`, `relevance`, `matched_targets`, `position`, `position_in_source`), `resultCount`, `docTokens`. **There is no rank.** This field previously documented `ranks` and "per-target `targets` with within-target ranks"; no writer ever wrote either, and what `ranks` did hold was `range(1, n+1)` over a list already flattened source-card by source-card -- a position, not a rank, and not even a relevance order, since the response sorts globally and then buckets into cards. Ask relevance questions of `hits[].relevance`. |
+| `rankedSummary.hits[].matched_targets` | list or null | Which of the caller's search targets matched this entity, each with the server's own relevance. **`null` means no attribution was published**, which is not the same as `[]`: the server attaches it to entities only (a source card's attribution is discarded before an entity object exists) and only on the semantic path, because a lexical score is relative to its own query and comparing two of them means nothing. A lexical run therefore cannot say which target found what, and a reader must report that as unavailable rather than as "never asked". |
 | `retrieval_mode` | string or null | `get_context` only: the `retrieval` field of the response that answered, `semantic` or `lexical`. Null when the server named none, which means no embedding provider. Recorded per call because the semantic path can fall over partway through a run. |
 | `query` | string or null | `execute_query` only: the Malloy the call sent, so the final query can be chosen by which call the server actually answered. |
 | `modelPath` | string or null | `execute_query` only: the model file the call named. A source does not resolve outside the file that declares it, so re-execution needs this. |
@@ -222,14 +229,14 @@ The answer judge's verdict for one attempt (protocol in
 | Field | Type | Notes |
 |---|---|---|
 | `verdict` | string or null | `match` / `near_match` / `no_match` / `needs_human`; null when the attempt is not scorable. Only `match` and `no_match` are decisions; see below. |
-| `reason` | string | Why, from the judge; for a null verdict, why not scorable: `not_submitted`, `contaminated`, or `golden_<status>` -- `golden_missing`, `golden_provisional`, `golden_invalid`, `golden_ambiguous`, `golden_verified_wrong`. The `golden_*` reasons are decided before the judge is called, so an unestablished key costs nothing to refuse. |
+| `reason` | string | Why, from the judge; for a null verdict, why not scorable, and it is ALWAYS one of these: `not_submitted`, `answerer_truncated`, `contaminated`, `judge_unparseable`, `no_saved_verdict`, `judge_error: <text>`, or `golden_<status>` -- `golden_missing`, `golden_provisional`, `golden_invalid`, `golden_ambiguous`, `golden_verified_wrong`. A null verdict never arrives without one: a nulling that does not say which of these it was is indistinguishable from the field being dropped on the write path, and was read as exactly that from a real run. The `golden_*` reasons and `answerer_truncated` are decided before the judge is called, so neither an unestablished key nor a cut-off attempt costs a judge call. |
 | `confidence` | int or null | 1 to 10. Confidence of 5 or lower forces `needs_human`. |
 | `column_pairing` | object or null | The judge's named gold-to-prediction column correspondence. |
 | `judge_version` / `rubric_sha` | string | Pins which rubric produced this verdict. |
 | `golden_revision` | int | From the case at score time. |
 | `contaminated` | bool or `"unknown"` | Copied from the attempt; true or unknown means `verdict: null`. |
 | `must_not_use_hits` | list or null | Entries of `golden.mustNotUse` a script found in the final query. A hit forces `verdict: no_match`. Decided by `check_must_not_use.py`, never by the judge: it is a question about query text. Only a BARE name vetoes; an entry with a connective (`X as ...`, `X through ...`) objects to a use of X rather than to X, and goes to the judge with the prose entries and a path's bare leaf. |
-| `judge_verdict` | string or null | What the judge said when a `must_not_use_hits` veto overrode it. Null otherwise, so the judge's own agreement rate stays measurable across vetoes. |
+| `judge_verdict` | string or null | What the judge said when a MECHANICAL override replaced it: a `must_not_use_hits` veto, or contamination voiding the verdict. Null otherwise, so the judge's own agreement rate stays measurable across both. When a case is vetoed AND contaminated this still holds the judge's read, not the veto's `no_match`. |
 | `artifactPath` | string | The full judge output under `artifacts/`. |
 | `gold_status` | string | `verified` / `verified_benign` / `suspect` / `verified_wrong`, and never anything else: the case file's `golden.status` is a DIFFERENT vocabulary that shares only `verified` and `verified_wrong`, and only those two fall back into this field. A `provisional` / `invalid` / `ambiguous` key is reported through the verdict's `reason` (`golden_<status>`) instead, so nothing downstream has to match on two vocabularies in one field. **From the judge**, which scored against the golden as written and reports separately whether it believes it; falls back to the case's standing `golden.status` when the judge does not say, written onto the verdict itself so the run summary and the aggregates see it too. `verified_wrong` excludes the case from run aggregates -- from the judge OR from the case file; assigning the fallback into a copy of the verdict meant a key the set had marked wrong stayed in the aggregates unless the judge independently agreed. `suspect` and `verified_wrong` route to the golden side door as `dataset` issues, never to improve. |
 | `gold_note` | string or null | The judge's evidence for a non-`verified` status: the two values, or the model line against the rubric sentence. Null when `verified`. |
@@ -255,6 +262,28 @@ rate, which is the one thing the answerable-sounds-unanswerable cases exist to
 measure. A refusal is exempt only where `golden.kind` is `unanswerable`, and
 there naming the gap is the pass and a confident number is the fail.
 
+`answerer_truncated` means the answerer hit the run's `maxTurns` cap and the
+CLI returned whatever it had, which is usually a sentence stopping mid-thought.
+It is refused BEFORE the golden gate and before a saved verdict is reused, so
+`--rebuild` over a run made without this re-derives the right ledger from its
+transcripts. A cap is a harness setting, so a case cut off by one says nothing
+about whether the model could answer it: measured on one 29-case arm, four
+attempts ended at exactly 31 turns and each was judged as a wrong answer, one
+of them on the text "Let me build the correct monthly trend query."
+
+`contaminated` means the attempt breached isolation (`contamination_reasons`
+names how) and the verdict is withheld rather than lost. The judge's own read
+moves to `judge_verdict`. Before this reason was written, a voided verdict kept
+the judge's prose in `reason` beside `verdict: null`, and a run report
+reasonably concluded the ledger was dropping the field -- it recovered nine
+"lost" verdicts by re-parsing the stored judge replies, every one of which had
+been voided on purpose.
+
+Both are HARNESS-owned, not model-owned, and a run containing either reports
+`status: incomplete` in `run.json` and **prints no pass rate**. `unscorable`
+(no established golden) and `judge_unparseable` do not suppress the rate: the
+first is a dataset state no arm can fix, and the second already has a retry.
+
 Aggregates count decided verdicts only. `match` and `no_match` are the pass and
 the fail; **`near_match`, `needs_human` and null are none of the above** and stay
 out of acceptance arithmetic. `near_match` is excluded because it means "defensibly

package/skills/eval-answer/scripts/check_coverage.py

@@ -43,6 +43,7 @@ from __future__ import annotations
 
 import argparse
 import concurrent.futures as futures
+import hashlib
 import json
 import pathlib
 import re
@@ -56,6 +57,7 @@ HERE = pathlib.Path(__file__).resolve().parent
 SKILLS_ROOT = HERE.parent.parent
 sys.path.insert(0, str(HERE))
 
+from json_scan import json_objects  # noqa: E402
 from verify_goldens import model_text as local_model_text  # noqa: E402
 
 # The `claude -p` invocation lives once, in the harness, rather than being
@@ -234,27 +236,6 @@ one candidate and its `ruled_out` entry is null, the verdict is not `ok`.
 """
 
 
-def json_objects(text: str) -> list[dict[str, Any]]:
-    """Every JSON object in `text`, in the order they appear.
-
-    Decoded from each `{` rather than matched with `\\{.*\\}`, which is greedy
-    and spans the FIRST brace to the LAST. The prompt hands the agent the model
-    text, so one `extend { ... }` quoted back in the narration would swallow the
-    verdict and a good reply would read as unparseable.
-    """
-    dec, out, i = json.JSONDecoder(), [], 0
-    while (i := text.find("{", i)) >= 0:
-        try:
-            v, end = dec.raw_decode(text, i)
-        except json.JSONDecodeError:
-            i += 1
-            continue
-        if isinstance(v, dict):
-            out.append(v)
-        i = max(end, i + 1)
-    return out
-
-
 def parse_reply(text: str, allowed: tuple[str, ...]) -> dict[str, Any]:
     """The agent's JSON, or a null verdict saying why not.
 
@@ -688,6 +669,15 @@ def main(argv: list[str] | None = None) -> int:
     else:
         raise SystemExit("pass --model <path>, or --publisher with --package")
 
+    # What was measured, pinned by content. Coverage is sold as a per-version
+    # trend, and a trend needs each point tied to the bytes behind it: a
+    # `--model <dir>` run stamped `version: null` and named no path, so two
+    # runs reading 38% and 62% could not afterwards be told apart. The sha is
+    # over the same text the agent is shown, which is already in memory.
+    model_source = (str(a.model_path.resolve()) if a.model_path
+                    else f"{a.publisher} {a.environment}/{a.package}")
+    model_sha = hashlib.sha256(model.encode()).hexdigest()
+
     cases = [json.loads(l) for l in (a.set_dir / "cases.jsonl").read_text()
              .splitlines() if l.strip()]
     if a.only:
@@ -715,6 +705,7 @@ def main(argv: list[str] | None = None) -> int:
     if a.out:
         a.out.write_text(json.dumps(
             {"version": a.version, "set": str(a.set_dir),
+             "modelSource": model_source, "modelSha256": model_sha,
              "agentModel": a.agent_model, **s, "cases_detail": rows,
              **({"labelComparison": serialisable(cmp)} if cmp else {})},
             indent=2))

package/skills/eval-answer/scripts/check_coverage_test.py

@@ -404,5 +404,27 @@ class TheComparisonSurvivesTheArtifactWrite(unittest.TestCase):
         # The console report still reads the tuple-keyed original.
         self.assertIn("covered", cc.label_report(cmp))
 
+class ModelIdentity(unittest.TestCase):
+    """Coverage is sold as a per-version trend, and a trend needs each point
+    tied to the bytes behind it. A `--model <dir>` run stamped `version: null`
+    and named no path, so two runs reading 38% and 62% could not afterwards be
+    told apart -- the numbers became unciteable."""
+
+    SRC = (pathlib.Path(__file__).resolve().parent / "check_coverage.py").read_text()
+
+    def test_the_report_carries_the_model_source_and_sha(self):
+        self.assertIn('"modelSource": model_source', self.SRC)
+        self.assertIn('"modelSha256": model_sha', self.SRC)
+
+    def test_the_sha_is_over_the_text_the_agent_was_shown(self):
+        self.assertIn("hashlib.sha256(model.encode()).hexdigest()", self.SRC)
+
+    def test_a_local_run_names_the_resolved_path(self):
+        self.assertIn("str(a.model_path.resolve()) if a.model_path", self.SRC)
+
+    def test_a_served_run_names_the_publisher_and_package(self):
+        self.assertIn('f"{a.publisher} {a.environment}/{a.package}"', self.SRC)
+
+
 if __name__ == "__main__":
     unittest.main()