@malloy-publisher/skills 0.1.31 → 0.1.33

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.31",
+   "version": "0.1.33",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.31
+version: 0.1.33
 ---
 
 # Evaluate One Answer
@@ -257,7 +257,8 @@ otherwise a modelling agent is dispatched to fix a model that is already right.
 
 ## Step 6: Append events, then stop
 
-Append to `evals/<set>/runs/<runId>/events.jsonl` with `caseId` set. Shapes
+Append to `<workdir>/runs/<runId>/events.jsonl` with `caseId` set (the run
+directory `eval.py run` printed; the workdir is the set's, from `eval.toml`). Shapes
 live in `reference/ledger-schema.md`.
 
 1. `attempt`: qid, sample, phase, question_sha, submitted, final_query,

package/skills/eval-answer/reference/ledger-schema.md

@@ -29,20 +29,28 @@ grandfathered unknown fields on old runs.
 ## Layout
 
 ```
-evals/<set>/
+evals/<set>/                # the set, in the model's repository
   set.json                  # set metadata (below)
+  eval.toml                 # servers and paths for eval.py (config.py)
   cases.jsonl               # one case per line
   judge-regressions.jsonl   # judge verdicts a human overruled
+
+<workdir>/                  # [paths] workdir, default ~/.malloy-eval/<set>-<hash>/
   runs/<runId>/
     run.json                # run config, the attribution pins
     events.jsonl            # append-only event lines
     artifacts/              # prediction CSVs, judge outputs, transcripts
+  packages/                 # built report packages
+  servers/<role>/           # server roots for eval.py serve
 ```
 
 The set directory lives in the SAME git repository as the model it evaluates,
-so a checkpoint (a git commit) pins the model and the ledger together. Never
-place `evals/` inside the directory tree the answerer's package serves: gold
-in the served tree is a contamination path.
+so a checkpoint (a git commit) pins the model and the answer key together.
+The workdir is never inside the model package: a run holds a `model.malloy`
+snapshot and a report is a Malloy package, and either one nested there puts
+the package into `loadErrors`. For a checkpoint to pin the ledger as well,
+point the workdir at a directory in the repository outside the package, and
+gitignore its `servers/` and `packages/`.
 
 Rules that make the ledger trustworthy:
 
@@ -82,7 +90,7 @@ One JSON object per line:
 | `tags` | list |
 | `state` | `candidate` / `selected` / `excluded`. |
 | `source` | Where the case came from. |
-| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous` / `verified_wrong`, the last being a key a person has established is wrong, which `run_baseline.py` drops from every aggregate), `kind`, `value` or `path` (artifact under the set directory), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`skill:eval-judge` rules 10 and 11). |
+| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous` / `verified_wrong`, the last being a key a person has established is wrong, which `run_baseline.py` drops from every aggregate), `kind`, `value` or `path` (artifact under the set directory; a `scalar` value is an object naming the canonical query's column, `{"net_sales": 834216}`, never a bare number), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`skill:eval-judge` rules 10 and 11). |
 | `golden.kind` | What the key IS. A value (`scalar`, rows), `unanswerable`, or `criteria`: prose with no value, where the author's criteria are the whole key. `criteria` is the one kind that is legitimately `verified` on arrival, because there is no derivation to perform and nothing that can be numerically wrong; an imported key holding a VALUE is `provisional` until re-derived (`skill:eval-import`). **`verify_goldens.py --promote` is the only thing that writes `golden.status`**: it marks a `provisional` golden `verified` when its value re-derived cleanly from the truth package AND a second derivation exists. `--refresh` rewrites a drifted VALUE and never changes a status; `invalid` and `ambiguous` are settled by a person through the golden side door. |
 | `golden.rubric` | **On the golden, not on the case.** The prose the judge is shown as `CASE RUBRIC`, naming what counts as correct and what does not. `run_baseline.py` reads `golden.rubric`; a rubric written at the case's top level is silently not passed, the judge is told `CASE RUBRIC: none`, and it then scores from the golden value and the answer alone -- which reads as a judge that ignores its instructions. |
 | `golden.mustState` | **On the golden, not on the case.** What the answer has to say out loud. Read from `golden.mustState` by the run-package builder. |
@@ -129,6 +137,7 @@ on that axis.
 | `scope` | Platform target only: the `environment/package[@version]` the answerer was told to pass as an explicit `scopes` entry on every `get_context` / `execute_query` call. A workspace can serve many packages (a personal workspace serves every package the user can read), so without this the run also measures whether retrieval picks the right package -- a different measurement. Absent means unscoped. |
 | `targetVersion` *(pin)* | Platform target only: the published version the answers came from. Both hosted tools default an omitted version to the PINNED one, meaning whatever the workspace serves at the moment of the call, so this only describes the run when `--scope` carries it to every call. `--target-version` supplies it when the scope omits it; a scope naming a different version is refused rather than treated as an override, because a run cannot say which build answered if two pins disagree. |
 | `mcpUrl` / `publisher` | The endpoints the answerer and the re-execution used. |
+| `evalConfig` | The set's `eval.toml` (or `eval.json`) the run read, as `{path, values}`; `path` is null when the set has none. |
 | `predictionsReExecuted` | Whether re-execution was POSSIBLE: the server was serving the bytes this run is pinned to. It is one bool for the whole run and says nothing about any given query, so it reads stronger than it is; `reExecution` below is what actually happened. |
 | `reExecution` | `{attempted, ok, failed, noQuery, notReExecuted, notJudged, missing}` over the run's predictions. `failed` means the query ran and the server returned an error, which is a different thing from an answer with no query at all. `notJudged` is a case the judge was never handed: a golden refused before judging (`provisional`, `invalid`, `ambiguous`, `verified_wrong`, missing), an attempt that submitted nothing, or every case under `--no-judge`. Those used to land in `missing` and read as corrupt artifacts on a freshly imported set. `ok`, `failed`, `noQuery`, `notReExecuted`, `notJudged` and `missing` partition the cases, so they sum to the case count; `attempted` is `ok + failed` and sits outside that sum. |
 | `retrievalMode` | Which retriever answered: `semantic`, `lexical`, `mixed` (it changed mid-run, so nothing can be averaged across it) or `unreported` (nothing ranked, which on a configured server means no embedding provider). Only a RANKING call names a retriever, so a call that enumerates a type or looks a name up is `unreported` on a fully semantic server and must not make the run read as `mixed`. Local retrieval degrades to lexical SILENTLY without an embedding key, and a pair compared across that reads as a model change, so `flip_table.py` refuses the pair unless both sides match. |
@@ -142,7 +151,7 @@ on that axis.
 | `answererCostUsd` / `judgeCostUsd` | What the arm cost, split by role. The judge's half was discarded until 2026-09-02, so every "cost per arm" quoted before then was the answerer alone. Per-case judge cost is on each `score` event. |
 | `answererCostCopiedFrom` | The run whose attempts `answererCostUsd` was COPIED from, when this run spawned no answerer -- a `--rejudge`, a `--rebuild`, a `--from`. Those copy the source's attempt events verbatim, `cost_usd` included, so the figure is right about the attempts and wrong as this run's spend: one re-judge reported $17.56 of answering that never happened. Null when this run paid, so a total over several run files can skip the copies. |
 | `goldenCheck` | What `verify_goldens.py` said before the run started: `N ok, M drifted, K other finding(s)`, or why it did not run (no truth server on a platform target; `--skip-golden-check`; rebuild). A run that started on a drifted set says so here rather than pretending its verdicts mean something. `run_baseline` calls `verify()` without a model, so the set-name audit (check 5) does not run in-arm; the trailing `N stale entity name(s)` is the in-arm lint's own count, and `entity-name lint not run` says there was no model text to lint against. Without it the field read `0 other finding(s)` for a check that never happened. |
-| `staleEntityNames` | The `expectedEntities` names that appear nowhere in the served model, from the in-arm lint. Each one depresses the run's own recall on every case listing it, so a stale set reads as a retrieval failure until someone checks by hand. `null` when there was no model text to lint against (a platform target), which is a different fact from `[]`. |
+| `staleEntityNames` | The `expectedEntities` ids the compiled served model does not declare, from the in-arm lint (bare names from a search of the model text when the compiled model could not be read). Each one depresses the run's own recall on every case listing it, so a stale set reads as a retrieval failure until someone checks by hand. `null` when there was no model text to lint against (a platform target), which is a different fact from `[]`. |
 | `status` | `complete`; `aborted` when four consecutive attempts errored or found the server dead and the harness stopped rather than spend the rest of the budget on attempts nobody will trust; or `incomplete` when the arm ran to the end and still cannot report a rate, because `truncated` or `contaminated` is non-empty. `incomplete` is a third state, not a soft `complete`: the arm did everything asked of it and the number is still unearned. It does NOT stop the run being compared: `flip_table.py` scores the cases both arms did, and names the ones this arm left unscored -- an excluded case carries no verdict, so it can never be counted as a flip. |
 | `packageSha` / `servedRevision` *(pins)* | Taken from the server, not recomputed: `sourceContentSha` is a content hash over EVERY model path in the package, so an edit to an imported file moves it where a sha of the one `--model-path` does not; `servedRevision` is minted per load, so it identifies a load rather than content and is a poor pin alone. Measured: `publisher.json`'s `version` moves neither, and nothing in Publisher reads it -- it is not in the Package API schema and never returned, so it pins nothing. |
 | `datasetSha` *(pin)* | Content hash of `set.json` + `cases.jsonl`. Deliberately SEPARATE from the model's sha: a golden repair is not a model change, and one pin covering both would make every answer-key fix read as an edit to the model, which is the distinction an A/B rests on. Automatic, so nobody has to remember it; `datasetVersion` stays beside it as the human-readable sequence. **Local targets only.** A hosted target that publishes IMMUTABLE versions needs none of this: the set rides inside the version, and immutability -- not hashing -- is what makes a pin trustworthy. There, `targetVersion` alone identifies model and set together. |

package/skills/eval-answer/scripts/check_coverage.py

@@ -9,7 +9,7 @@ version and read as a trend, which is the point of it.
 
   python check_coverage.py --set <set-dir> --model <package-dir-or-file>
   python check_coverage.py --set <set-dir> --publisher http://localhost:4811 \
-      --environment samples --package ecommerce --version 0.0.58
+      --version 0.0.58   # environment and package from the set's eval.toml
 
 It answers a different question from retrieval recall, and the difference is the
 reason it exists. Recall asks whether `get_context` surfaced the entities a case
@@ -57,9 +57,10 @@ HERE = pathlib.Path(__file__).resolve().parent
 SKILLS_ROOT = HERE.parent.parent
 sys.path.insert(0, str(HERE))
 
+import config  # noqa: E402
 from json_scan import json_objects  # noqa: E402
 from verify_goldens import model_text as local_model_text  # noqa: E402
-from check_findable import compiled_entities  # noqa: E402
+from check_findable import current_entities  # noqa: E402
 
 # The `claude -p` invocation lives once, in the harness, rather than being
 # rewritten here. It costs this script a reach into a sibling skill, which the
@@ -158,16 +159,20 @@ def compiled_surface(base: str, environment: str, package: str) -> str:
     reads those as absent and returns MISSING on a question the model answers
     -- measured, two of this skill's three false gaps on one set.
 
-    Empty string when the compiled model cannot be read, which is honest: the
-    prompt then says the list is unavailable rather than implying the text is
-    the whole surface.
+    Empty string when the compiled model cannot be read, or is stale, which is
+    honest: the prompt then says the list is unavailable rather than implying
+    the text is the whole surface.
     """
-    declared = compiled_entities(base, environment, package)
+    declared, warning = current_entities(base, environment, package)
+    if warning:
+        print(f"! {warning}", file=sys.stderr)
     if not declared:
         return ""
     lines = []
     for src in sorted(declared):
-        fields = sorted(f for f in declared[src] if not f.startswith("source:"))
+        # Own fields only: each joined source is listed under its own name.
+        fields = sorted(f for f in declared[src]
+                        if not f.startswith("source:") and "." not in f)
         if fields:
             lines.append(f"{src}: " + ", ".join(fields))
     return "\n".join(lines)
@@ -694,8 +699,13 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--publisher", default=None,
                     help="read the model over REST instead, from a running "
                          "Publisher serving the version to measure")
-    ap.add_argument("--environment", default="samples")
-    ap.add_argument("--package", default=None)
+    ap.add_argument("--environment", default=None,
+                    help="with --publisher: the environment serving --package. "
+                         "Default: [model] environment in the set's eval.toml")
+    ap.add_argument("--package", default=None,
+                    help="with --publisher: the package to measure. Default: "
+                         "[model] package in the set's eval.toml, then "
+                         "set.json's targetPackage")
     ap.add_argument("--model-path", dest="one_model", default=None,
                     help="with --publisher: measure one model file rather than "
                          "every model in the package")
@@ -754,7 +764,11 @@ def main(argv: list[str] | None = None) -> int:
         model = local_model_text(a.model_path)
         if not model:
             raise SystemExit(f"no .malloy text under {a.model_path}")
-    elif a.publisher and a.package:
+    elif a.publisher:
+        cfg = config.load(a.set_dir)
+        a.environment = cfg.need(a.environment, "model", "environment",
+                                 "--environment")
+        a.package = cfg.need(a.package, "model", "package", "--package")
         try:
             model = rest_model_text(a.publisher, a.environment, a.package,
                                     a.one_model)
@@ -764,7 +778,7 @@ def main(argv: list[str] | None = None) -> int:
             raise SystemExit(f"{a.publisher} served no model text for "
                              f"{a.environment}/{a.package}")
     else:
-        raise SystemExit("pass --model <path>, or --publisher with --package")
+        raise SystemExit("pass --model <path>, or --publisher <url>")
 
     # The compiled surface names the columns a source exposes without declaring
     # them. Only REST can supply it; a --model run reads text off disk and has

package/skills/eval-answer/scripts/check_coverage_test.py

@@ -440,10 +440,10 @@ class CompiledSurface(unittest.TestCase):
     """
 
     def test_it_lists_fields_per_source(self):
-        with mock.patch.object(cc, "compiled_entities", return_value={
+        with mock.patch.object(cc, "current_entities", return_value=({
                 "products": {"source:products", "dimension:retail_price",
                              "dimension:cost"},
-                "customers": {"source:customers", "dimension:signup_date"}}):
+                "customers": {"source:customers", "dimension:signup_date"}}, None)):
             out = cc.compiled_surface("http://p", "env", "pkg")
         self.assertIn("products: dimension:cost, dimension:retail_price", out)
         self.assertIn("customers: dimension:signup_date", out)
@@ -451,7 +451,9 @@ class CompiledSurface(unittest.TestCase):
         self.assertNotIn("source:products", out)
 
     def test_an_unreadable_model_is_empty_not_a_guess(self):
-        with mock.patch.object(cc, "compiled_entities", return_value=None):
+        with mock.patch.object(cc, "current_entities",
+                               return_value=(None, "could not be read")), \
+             mock.patch("builtins.print"):
             self.assertEqual(cc.compiled_surface("http://p", "env", "pkg"), "")
 
 
@@ -537,17 +539,18 @@ class MainWiring(unittest.TestCase):
             prompts[qid] = prompt
             return [{"type": "assistant"}], json.dumps(replies[qid]), "", 1, 0.1
 
-        entities = mock.Mock(side_effect=surface_raises, return_value=surface)
+        entities = mock.Mock(side_effect=surface_raises, return_value=(surface, None))
         err = []
         with mock.patch.object(cc, "rest_model_text",
                                return_value="source: products is t"), \
-             mock.patch.object(cc, "compiled_entities", entities), \
+             mock.patch.object(cc, "current_entities", entities), \
              mock.patch.object(cc, "run_cli", side_effect=fake_cli), \
              mock.patch("builtins.print",
                         lambda *a, **k: err.append(" ".join(map(str, a)))
                         if k.get("file") is sys.stderr else None):
             cc.main(["--set", str(d), "--publisher", "http://p",
-                     "--package", "pkg", "--parallel", "1",
+                     "--environment", "env", "--package", "pkg",
+                     "--parallel", "1",
                      "--out", str(d / "out.json"), *extra])
         return prompts, json.loads((d / "out.json").read_text()), "\n".join(err)
 

package/skills/eval-answer/scripts/check_findable.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 """Can a right-typed search actually retrieve each `required` entity? Stdlib only.
 
+  python3 check_findable.py --set <set-dir>   # server and names from its eval.toml
   python3 check_findable.py --set evals/faa-v1 --mcp-url http://localhost:4045/mcp \
       --environment samples --package faa
 
@@ -58,6 +59,7 @@ import urllib.request
 from typing import Any
 
 sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+import config  # noqa: E402
 from mcp_payload import entity_hits  # noqa: E402
 
 # An entity kind, and the target type that can return it. The inverse of the
@@ -104,6 +106,24 @@ def get_context(mcp_url: str, targets: list[dict[str, str]],
     return json.loads(text)
 
 
+def add_fields(bucket: set[str], fields: list[Any], prefix: str = "") -> None:
+    """Every field as `kind:name`, and every field reached through a join as
+    `kind:join.name`, the dotted path get_context names it by.
+
+    A join field carries the joined source's schema, so each hop of a path is
+    checked against the join the source actually declares. Looking the last
+    hop up as a source name instead passed a bogus first hop, and failed a
+    join whose name is not its source's (`join_one: buyer is customers`).
+    """
+    for f in fields:
+        if not (isinstance(f, dict) and f.get("name") and f.get("kind")):
+            continue
+        bucket.add(f"{f['kind']}:{prefix}{f['name']}")
+        if f["kind"] == "join":
+            add_fields(bucket, (f.get("schema") or {}).get("fields") or [],
+                       f"{prefix}{f['name']}.")
+
+
 def compiled_entities(rest_url: str, environment: str,
                       package: str) -> dict[str, set[str]] | None:
     """`{source: {"kind:name", ...}}` from the COMPILED model, or None.
@@ -161,9 +181,7 @@ def compiled_entities(rest_url: str, environment: str,
                 continue
             bucket = out.setdefault(src, set())
             bucket.add(f"source:{src}")
-            for f in fields:
-                if isinstance(f, dict) and f.get("name") and f.get("kind"):
-                    bucket.add(f"{f['kind']}:{f['name']}")
+            add_fields(bucket, fields)
         for q in doc.get("queries") or []:
             if not isinstance(q, dict):
                 continue
@@ -224,14 +242,46 @@ def stale_packages(rest_url: str, timeout: int = 30) -> set[tuple[str, str]] | N
             if isinstance(e, dict) and e.get("stale")}
 
 
+def current_entities(rest_url: str, environment: str,
+                     package: str) -> tuple[dict[str, set[str]] | None, str | None]:
+    """`compiled_entities`, unless the package is stale, and a warning if any.
+
+    Staleness first. A stale package answers the models endpoint normally
+    while describing the compile BEFORE the last save, so an entity deleted
+    since then reads as declared. Refusing to check says so; checking anyway
+    would be one more number nobody had earned. Every caller that treats the
+    compiled model as the authority goes through here.
+    """
+    stale = stale_packages(rest_url)
+    if stale and (environment, package) in stale:
+        return None, (f"{environment}/{package} is serving a STALE model (its "
+                      f"last reload failed to compile), so the compiled model "
+                      f"is not the authority on what exists; existence and "
+                      f"kind were NOT checked. Fix the model and reload, then "
+                      f"re-run.")
+    declared = compiled_entities(rest_url, environment, package)
+    if declared is None:
+        return None, ("the compiled model could not be read; existence and "
+                      "kind were NOT checked")
+    if stale is None:
+        return declared, ("the server's status could not be read, so "
+                          "staleness is unknown; treating the compiled model "
+                          "as current")
+    return declared, None
+
+
 def declared_findings(cases: list[dict[str, Any]],
                       declared: dict[str, set[str]]) -> list[str]:
     """Ids the compiled model does not declare, or declares as another kind."""
     out = []
     for eid, qids in sorted(required_ids(cases).items()):
         parts = eid.split(":", 2)
-        if len(parts) < 3:
-            continue          # malformed; `check` reports it on its own
+        if len(parts) < 3 or not all(parts):
+            # Reported here as well as by `check`: the in-run lint calls only
+            # this, and an id it skips is one no retrieval can ever match.
+            out.append(f"{eid}: not a kind:source:name id "
+                       f"(required by {', '.join(qids)})")
+            continue
         kind, src, name = parts
         if src not in declared:
             out.append(f"{eid}: the compiled model has no source {src!r} "
@@ -239,25 +289,23 @@ def declared_findings(cases: list[dict[str, Any]],
             continue
         if f"{kind}:{name}" in declared[src]:
             continue
-        # A DOTTED name is a join path, not a field name. `declared[src]` holds
-        # the source's OWN fields, so `dimension:order_items:products.brand`
-        # never matched and every dotted id was reported as a missing field --
-        # five of them on one set, against a get_context that returns each as
-        # its top result. Resolve it where it lives: the last segment is the
-        # field, the segment before it names the joined source.
+        # A DOTTED name is a join path. `compiled_entities` records every
+        # field reached through a join under its path, so a miss above means a
+        # hop or the leaf is wrong. Name the first hop that does not exist.
         if "." in name:
             *path, leaf = name.split(".")
-            hop = path[-1]
-            if hop in declared and f"{kind}:{leaf}" in declared[hop]:
-                continue
-            if hop not in declared:
-                out.append(f"{eid}: the join path names {hop!r}, which the "
-                           f"compiled model has no source for "
-                           f"(required by {', '.join(qids)})")
+            reached = ""
+            for hop in path:
+                if f"join:{reached}{hop}" not in declared[src]:
+                    out.append(f"{eid}: the compiled {src} source has no join "
+                               f"{reached + hop!r} (required by "
+                               f"{', '.join(qids)})")
+                    break
+                reached += hop + "."
             else:
-                out.append(f"{eid}: the compiled {hop} source declares no "
-                           f"{kind} {leaf!r} to reach through that join "
-                           f"(required by {', '.join(qids)})")
+                out.append(f"{eid}: the join {'.'.join(path)!r} from {src} "
+                           f"reaches no {kind} {leaf!r} (required by "
+                           f"{', '.join(qids)})")
             continue
         other = sorted(k.split(":", 1)[0] for k in declared[src]
                        if k.split(":", 1)[1] == name)
@@ -341,11 +389,15 @@ def check(cases: list[dict[str, Any]], mcp_url: str, environment: str,
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("--set", dest="set_dir", required=True, type=pathlib.Path)
-    ap.add_argument("--mcp-url", required=True,
+    ap.add_argument("--mcp-url", default=None,
                     help="the MCP endpoint of the model UNDER TEST, not the "
-                         "truth server")
-    ap.add_argument("--environment", required=True)
-    ap.add_argument("--package", required=True)
+                         "truth server. Default: from [model] in the set's "
+                         "eval.toml")
+    ap.add_argument("--environment", default=None,
+                    help="Default: [model] environment in the set's eval.toml")
+    ap.add_argument("--package", default=None,
+                    help="Default: [model] package in the set's eval.toml, then "
+                         "set.json's targetPackage")
     ap.add_argument("--publisher", default=None,
                     help="REST URL of the same server, e.g. "
                          "http://localhost:4811. With it, each id is also "
@@ -361,6 +413,10 @@ def main(argv: list[str] | None = None) -> int:
                          "wait, which risks reading the lexical matcher as a "
                          "missing entity")
     a = ap.parse_args(argv)
+    cfg = config.load(a.set_dir)
+    a.environment = cfg.need(a.environment, "model", "environment", "--environment")
+    a.package = cfg.need(a.package, "model", "package", "--package")
+    a.mcp_url = a.mcp_url or cfg.model_mcp_url()
 
     f = a.set_dir / "cases.jsonl"
     if not f.exists():
@@ -419,29 +475,10 @@ def main(argv: list[str] | None = None) -> int:
 
     declared_out: list[str] = []
     if a.publisher:
-        # Staleness first. A stale package answers the models endpoint normally
-        # while describing the compile BEFORE the last save, so reading it as
-        # the authority prints findings nobody can act on: the id may be fine
-        # and the served model simply old. Refusing to check says so; checking
-        # anyway would be one more number the harness had not earned.
-        stale = stale_packages(a.publisher)
-        if stale is None:
-            print("! the server's status could not be read, so staleness is "
-                  "unknown; treating the compiled model as current",
-                  file=sys.stderr)
-        is_stale = bool(stale) and (a.environment, a.package) in stale
-        declared = (None if is_stale
-                    else compiled_entities(a.publisher, a.environment, a.package))
-        if is_stale:
-            print(f"! {a.environment}/{a.package} is serving a STALE model "
-                  f"(its last reload failed to compile), so the compiled model "
-                  f"is not the authority on what exists; existence and kind "
-                  f"were NOT checked. Fix the model and reload, then re-run.",
-                  file=sys.stderr)
-        elif declared is None:
-            print("! the compiled model could not be read; existence and kind "
-                  "were NOT checked", file=sys.stderr)
-        else:
+        declared, warning = current_entities(a.publisher, a.environment, a.package)
+        if warning:
+            print(f"! {warning}", file=sys.stderr)
+        if declared is not None:
             declared_out = declared_findings(cases, declared)
             print(f"compiled model: {sum(len(v) for v in declared.values())} "
                   f"entities across {len(declared)} sources")

package/skills/eval-answer/scripts/check_findable_test.py

@@ -11,6 +11,7 @@ from __future__ import annotations
 import os
 import sys
 import unittest
+from unittest import mock
 
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 import check_findable  # noqa: E402
@@ -181,10 +182,12 @@ class CompiledModel(unittest.TestCase):
         self.assertEqual(self.find("measure:flights:flight_count",
                                    "dimension:airports:state"), [])
 
-    def test_a_malformed_id_is_left_to_the_other_check(self):
-        # `check` reports it before any search; reporting it twice reads as
-        # two defects.
-        self.assertEqual(self.find("flight_count"), [])
+    def test_a_malformed_id_is_reported(self):
+        # run_baseline's lint calls only this, so skipping it here dropped the
+        # id there. main() dedups on the id, so the CLI still says it once.
+        out = self.find("flight_count")
+        self.assertEqual(len(out), 1)
+        self.assertEqual(check_findable.finding_id(out[0]), "flight_count")
 
     def test_a_finding_is_identified_by_its_whole_entity_id(self):
         # Splitting on the bare colon returns the KIND, and deduplicating the
@@ -321,44 +324,82 @@ class Staleness(unittest.TestCase):
         # silently treating the model as current.
         self.assertIsNone(check_findable.stale_packages("http://127.0.0.1:9"))
 
+    def test_current_entities_refuses_a_stale_package(self):
+        # run_baseline read the compiled model with no staleness check, so an
+        # entity deleted since the last good compile read as declared.
+        with mock.patch.object(check_findable, "stale_packages",
+                               return_value={("e", "p")}), \
+             mock.patch.object(check_findable, "compiled_entities") as compiled:
+            declared, warning = check_findable.current_entities("http://x", "e", "p")
+        self.assertIsNone(declared)
+        self.assertIn("STALE", warning)
+        compiled.assert_not_called()
+
+    def test_current_entities_passes_a_current_package_through(self):
+        with mock.patch.object(check_findable, "stale_packages", return_value=set()), \
+             mock.patch.object(check_findable, "compiled_entities",
+                               return_value={"s": {"source:s"}}):
+            self.assertEqual(check_findable.current_entities("http://x", "e", "p"),
+                             ({"s": {"source:s"}}, None))
+
 
 class DottedJoinPaths(unittest.TestCase):
-    """A dotted name is a join path, resolved on the joined source.
+    """A dotted name is a join path, checked hop by hop against the joins the
+    source declares.
 
-    `declared[src]` holds a source's OWN fields, so every dotted id read as a
-    field the source does not declare. On the storefront tour set that was five
-    of eight reported misses, each one an id get_context returns as its top
-    result -- and the check ends by telling you to fix the key, so a false miss
-    here sends you to break an answer key that is correct.
+    Reading only a source's OWN fields made every dotted id a miss: on the
+    storefront tour set, five of eight reported misses, each one an id
+    get_context returns as its top result. Resolving only the LAST hop, as a
+    source name, then passed a bogus first hop and failed an aliased join.
     """
 
-    DECLARED = {
-        "order_items": {"source:order_items", "measure:total_sales"},
-        "products": {"source:products", "dimension:brand", "measure:product_count"},
-    }
+    PRODUCTS = [{"kind": "dimension", "name": "brand"},
+                {"kind": "measure", "name": "product_count"}]
+    # The shape the server returns: a join field carries the joined schema.
+    FIELDS = [{"kind": "measure", "name": "total_sales"},
+              {"kind": "join", "name": "products", "relationship": "one",
+               "schema": {"fields": PRODUCTS}},
+              {"kind": "join", "name": "buyer", "relationship": "one",
+               "schema": {"fields": [{"kind": "dimension", "name": "state"}]}}]
 
     def findings(self, eid):
+        declared = {"order_items": {"source:order_items"}}
+        check_findable.add_fields(declared["order_items"], self.FIELDS)
         cases = [{"qid": "q1", "expectedEntities": {"required": [eid]}}]
-        return check_findable.declared_findings(cases, self.DECLARED)
+        return check_findable.declared_findings(cases, declared)
 
     def test_a_reachable_join_path_is_not_a_finding(self):
         self.assertEqual(self.findings("dimension:order_items:products.brand"), [])
 
+    def test_a_join_named_other_than_its_source_is_reachable(self):
+        # `join_one: buyer is customers`: no source is called `buyer`.
+        self.assertEqual(self.findings("dimension:order_items:buyer.state"), [])
+
     def test_the_kind_still_has_to_match_on_the_joined_source(self):
         out = self.findings("measure:order_items:products.brand")
         self.assertEqual(len(out), 1)
-        self.assertIn("declares no measure 'brand'", out[0])
+        self.assertIn("reaches no measure 'brand'", out[0])
 
     def test_an_unknown_hop_is_named_as_the_hop(self):
         out = self.findings("dimension:order_items:suppliers.name")
         self.assertEqual(len(out), 1)
-        self.assertIn("'suppliers'", out[0])
+        self.assertIn("has no join 'suppliers'", out[0])
+
+    def test_a_bogus_first_hop_is_caught_even_when_the_last_hop_exists(self):
+        out = self.findings("dimension:order_items:nowhere.products.brand")
+        self.assertEqual(len(out), 1)
+        self.assertIn("has no join 'nowhere'", out[0])
 
     def test_a_field_missing_on_the_joined_source_is_a_finding(self):
         out = self.findings("dimension:order_items:products.colour")
         self.assertEqual(len(out), 1)
         self.assertIn("'colour'", out[0])
 
+    def test_a_malformed_id_is_reported_not_dropped(self):
+        out = self.findings("dimension:brand")
+        self.assertEqual(len(out), 1)
+        self.assertIn("not a kind:source:name id", out[0])
+
 
 if __name__ == "__main__":
     unittest.main()