@malloy-publisher/skills 0.1.27 → 0.1.28

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.27",
+   "version": "0.1.28",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # Evaluate One Answer

package/skills/eval-answer/reference/ledger-schema.md

@@ -122,7 +122,7 @@ on that axis.
 | `skillsVersion` *(pin)* | HEAD of the checkout the agents' skills were loaded from, dirty-marked (`ledger.skills_git_sha(root)`). The skills are the doctrine the agents load; a run that cannot name their version cannot anchor a skills A/B. |
 | `skillsRoot` / `harnessVersion` | Which checkout supplied the doctrine (`--skills-root`, e.g. a Publisher checkout for the open-source skills; default this one), and this checkout's own HEAD. The eval-* skills always come from the harness checkout, whatever `skillsRoot` says. Two runs whose `skillsRoot` differ are a skills A/B only if their manifests name the same skills. |
 | `diagnoserModel` / `improverModel` | Written by `diagnose.py` / `improve.py` when those stages run, so the run names every LLM that touched it. Absent on a run that was only answered and judged. |
-| `modelGitSha` | Commit of the model repo, from `--model-repo`; `-dirty` suffix when that path had uncommitted changes -- fine for a band measurement, not for an A/B pin. **Null unless `--model-repo` names the repo.** It cannot be inferred: Publisher serves a COPY under `publisher_data/`, so the tree around the served file is the server's storage, not the model's history, and the field named that until 2026-09-03. `modelSha` above is the pin that always works; this is the pointer to where those bytes are versioned, and absent beats wrong. |
+| `modelGitSha` | Commit of the model repo, from `--model-repo`; `-dirty` suffix when `--model-dir` (or the whole repo, without it) had uncommitted changes -- fine for a band measurement, not for an A/B pin. Null when git cannot answer for the scope (a path outside the repo): absent beats wrong. **Null unless `--model-repo` names the repo.** It cannot be inferred: Publisher serves a COPY under `publisher_data/`, so the tree around the served file is the server's storage, not the model's history, and the field named that until 2026-09-03. `modelSha` above is the pin that always works; this is the pointer to where those bytes are versioned, and absent beats wrong. |
 | `modelRepo` | The path `--model-repo` named, so a reader can tell an absent pin from an unrecorded one. |
 | `modelDir` | The package directory `--model-dir` named, when it was: the path the `-dirty` marker on `modelGitSha` was decided over. Absent means the whole repo, where any unrelated untracked file marks the model dirty. |
 | `environment` / `package` / `modelPath` | What was served, and from where. On a platform target `environment` is the organization and `package` is the workspace the MCP URL is scoped to. |

package/skills/eval-answer/scripts/verify_goldens_test.py

@@ -1192,8 +1192,6 @@ class VerifyQuotedFigures(unittest.TestCase):
         self.assertEqual(called, ["opus"])
 
 
-if __name__ == "__main__":
-    unittest.main()
 
 
 class HeldGoldensAreNotReDerived(unittest.TestCase):
@@ -1258,3 +1256,23 @@ class ANotQueryableKeyNamesItsRootSource(unittest.TestCase):
                                         return_value=(None, "HTTP 500: boom")):
             got, detail, _ = check_value(case, a)
         self.assertEqual((got, detail), ("error", "HTTP 500: boom"))
+
+
+class NothingIsDefinedBelowTheMainGuard(unittest.TestCase):
+    """CI runs this file as a script, `python3 <file>`, and `unittest.main()`
+    runs what is defined so far and exits. A test class written below the
+    guard is collected by `-m unittest` and never by CI: four classes sat
+    there after one PR, and one review earlier had moved two more up for the
+    same reason. Reading the file is the check that does not depend on how the
+    tests were invoked."""
+
+    def test_the_guard_is_the_last_statement(self):
+        src = pathlib.Path(__file__).read_text().splitlines()
+        guard = [i for i, l in enumerate(src) if l.startswith('if __name__ == "__main__":')]
+        self.assertEqual(len(guard), 1)
+        below = [l for l in src[guard[0]:] if l.startswith(("class ", "def "))]
+        self.assertEqual(below, [], f"defined below the main guard: {below}")
+
+
+if __name__ == "__main__":
+    unittest.main()

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # Diagnose One Answer

package/skills/eval-diagnose/scripts/diagnose.py

@@ -706,7 +706,10 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--run", required=True, type=pathlib.Path)
     ap.add_argument("--set", dest="set_dir", required=True, type=pathlib.Path)
     ap.add_argument("--model-dir", type=pathlib.Path, default=None,
-                    help="the Malloy package under test; the agent's cwd")
+                    help="the Malloy package under test; the agent's cwd. A "
+                         "relative path is taken from the working directory "
+                         "(run_baseline's --model-dir is relative to its "
+                         "--model-repo instead)")
     # The split is cheap-on-per-case, expensive-on-clustering, and one measured
     # run says it is the wrong way round: every failure in it came from per-case
     # work (wrong output shape, a probe at the wrong grain) while the clustering

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-improve
 description: 'Make the smallest safe Malloy model edit that closes a diagnosed model-owned gap, with a probe receipt for every factual claim. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # Improve the Model

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files under the model package''s evals/ directory; checkpoints are git commits of the model repo. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.27
+version: 0.1.28
 ---
 
 # The Evaluation Loop

package/skills/eval-loop/reference/running-a-run.md

@@ -62,10 +62,14 @@ python3 skills/eval-loop/scripts/run_baseline.py \
 python3 skills/eval-loop/scripts/run_baseline.py \
   --set <repo>/evals/ecommerce --out results/<arm> \
   --parallel 4 --truth-publisher http://localhost:4881 \
-  --model-repo <repo>   # the checkout the MODEL is versioned in, recorded as
-                        # modelGitSha. It cannot be inferred: Publisher serves
-                        # a copy under publisher_data/, whose surrounding tree
-                        # is the server's storage, not the model's history.
+  --model-repo <repo> --model-dir <package>   # the checkout the MODEL is
+                        # versioned in, recorded as modelGitSha, and the package
+                        # directory inside it the -dirty marker is decided over
+                        # (relative to the repo). Without --model-dir any
+                        # untracked file anywhere in the repo marks the model
+                        # dirty. The repo cannot be inferred: Publisher serves a
+                        # copy under publisher_data/, whose surrounding tree is
+                        # the server's storage, not the model's history.
 #    the run names itself <set>-<phase>-<nn> (ecommerce-baseline-01, then -02
 #    for the second arm of the A/A). Pass --label only for a run that needs a
 #    human name; hand-typed arm names stop being readable within an afternoon.

package/skills/eval-loop/scripts/build_run_package.py

@@ -122,13 +122,12 @@ def golden_display(g: dict[str, Any]) -> str:
 _IDENT = re.compile(r"(?<![A-Za-z0-9_])[a-z][a-z0-9]*(?:_[a-z0-9]+)+(?![A-Za-z0-9_])")
 
 
-def _result_text(block: dict[str, Any]) -> str:
-    c = block.get("content")
-    if isinstance(c, str):
-        return c
-    if isinstance(c, list):
-        return "\n".join(x.get("text", "") for x in c if isinstance(x, dict))
-    return ""
+# The run viewer used to keep its own copy of the raw reader, so a get_context
+# result the CLI had spilled to a file showed the "<persisted-output>" note as
+# its detail and no result count. run_baseline.result_text reads the file back
+# when it is still there; when it is gone the note stays, and the summary the
+# ledger recorded (none, for an unmeasured call) is what the viewer shows.
+from run_baseline import result_text as _result_text  # noqa: E402  (same directory)
 
 
 def _payload(text: str) -> Any:
@@ -373,14 +372,21 @@ def build(run_dirs: list[pathlib.Path], set_dir: pathlib.Path,
 
             mine = tool_calls.get(kk, [])
             for i, t in enumerate(mine, 1):
-                rs = t.get("rankedSummary") or {}
-                eids = rs.get("entityIds") or []
+                # A get_context call with no rankedSummary was never measured:
+                # the CLI's saved result file was gone by the time of a rebuild,
+                # or the call errored. It used to show as n_returned 0, the same
+                # row a search that found nothing writes, and empty_calls
+                # counted it. n_returned stays empty and `unmeasured` says why.
+                rs = t.get("rankedSummary")
+                unmeasured = t.get("tool") == "get_context" and rs is None
+                eids = (rs or {}).get("entityIds") or []
                 calls.append({
                     "attempt_key": ak,
                     "run_id": run_id, "qid": qid, "sample": e.get("sample"),
                     "call_index": i, "tool": t.get("tool"),
                     "targets": t.get("targets"),
-                    "n_returned": len(eids),
+                    "n_returned": None if unmeasured else len(eids),
+                    "unmeasured": unmeasured,
                     "entity_ids": eids[:40],
                     "error": t.get("error"),
                     # Which retriever answered this call. Absent on a server
@@ -512,7 +518,7 @@ def build(run_dirs: list[pathlib.Path], set_dir: pathlib.Path,
         "missing", "noise", "component", "owner", "where_to_fix", "why"])
     write_csv(data / "calls.csv", calls, [
         "attempt_key", "run_id", "qid", "sample", "call_index", "tool", "targets",
-        "n_returned", "entity_ids", "error", "retrieval_mode"])
+        "n_returned", "unmeasured", "entity_ids", "error", "retrieval_mode"])
     write_csv(data / "entities.csv", ents, [
         "run_id", "qid", "sample", "entity_id", "entity_kind", "entity_source",
         "entity_name", "role"])

package/skills/eval-loop/scripts/run_baseline.py

@@ -288,6 +288,11 @@ def git_sha(path: pathlib.Path, scope: pathlib.Path | None = None) -> str | None
     # so a dirty model repo read clean and the pin silently stopped pinning.
     path = path.resolve()
     scope = scope.resolve() if scope is not None else None
+    if scope is not None and not scope.exists():
+        # `git status -- <missing path>` exits 0 with empty output, so a typo
+        # (`pgk`) or a doubled path (`--model-repo repo/pkg --model-dir pkg`)
+        # read as a clean model. No pin is the honest answer here too.
+        return None
     try:
         d = path if path.is_dir() else path.parent
         head = subprocess.run(["git", "-C", str(d), "rev-parse", "HEAD"],
@@ -298,11 +303,29 @@ def git_sha(path: pathlib.Path, scope: pathlib.Path | None = None) -> str | None
         if scope is not None:
             cmd += ["--", str(scope)]
         dirty = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
+        if dirty.returncode != 0:
+            # A scope outside the repo makes git exit 128 with empty stdout,
+            # which read as "clean" and dropped the marker on a dirty model.
+            # No pin beats a wrong one: the schema says absent is honest.
+            return None
         return head.stdout.strip() + ("-dirty" if dirty.stdout.strip() else "")
     except Exception:  # noqa: BLE001
         return None
 
 
+def model_scope(repo: pathlib.Path | None, model_dir: pathlib.Path | None
+                ) -> pathlib.Path | None:
+    """The path the -dirty marker is decided over. A relative --model-dir is
+    resolved against --model-repo, not the working directory: given from
+    anywhere but the repo root it pointed outside the repo, git status exited
+    128 with empty stdout, and a dirty model pinned as clean."""
+    if repo is None:
+        return None
+    if model_dir is None:
+        return repo
+    return model_dir if model_dir.is_absolute() else repo / model_dir
+
+
 # --- Run naming --------------------------------------------------------------
 #
 # A run's label is what every comparison, flip table and report prints, so a set
@@ -354,6 +377,18 @@ def next_run_label(out: pathlib.Path, set_name: str, phase: str) -> str:
     return f"{stem}-{n:02d}"
 
 
+def imply_flags(a: argparse.Namespace) -> argparse.Namespace:
+    """`--rejudge` is `--rebuild`'s answer half plus a fresh judge, so it sets
+    `rebuild`. Without this, `--rejudge` alone reached every `if a.rebuild`
+    gate as a fresh run: it spawned a new answerer per case ($0.33 each) against
+    the flag's own help text, and with `--only` it took the ledger's full-write
+    path and left a 12-case run holding one case. `--from` already set both;
+    this makes the bare flag mean what it says."""
+    if getattr(a, "rejudge", False):
+        a.rebuild = True
+    return a
+
+
 def existing_run_refusal(out: pathlib.Path) -> str | None:
     """Why a fresh arm may not write into `out`, or None when it may.
 
@@ -474,22 +509,43 @@ def format_rows(rows: list[dict], limit: int = 60) -> str:
 # did not measure. Seen twice in one arm: 86.4% printed against 95% actual, and
 # diagnose went on to explain the phantom miss with an index-readiness story
 # that was not true.
-OFFLOADED = re.compile(r"(?:saved to|written to)\s+(/[^\s'\"]+)")
-
-
-def offloaded_json(text: str) -> dict[str, Any] | None:
-    """The response body a host spilled to a file, read back, or None.
-
-    Only ever reads a path the host itself named in the result it returned.
+# One note, one regex, one reader. The CLI names the file it spilled a result
+# to in two spellings ("<persisted-output> ... Full output saved to: <path>" and
+# "Output has been saved to <path>.txt."), sometimes with a colon, sometimes
+# with a trailing period. Two regexes and two readers for this one concept
+# handled those differently, and a rebuild after the CLI's temporary file was
+# gone matched one, missed the other, and scored the call as zero entities.
+# Only the CLI's own note counts, in either of its spellings, and only a
+# .json or .txt path: a looser "saved to|written to" also matched a source doc
+# that said "nightly extract is written to /etc/hosts" inside an ordinary JSON
+# result, replaced the result with that file, and scored the call as zero.
+SAVED_TO = re.compile(
+    r"(?:<persisted-output>.*?Full output saved to|Output has been saved to):?"
+    r"\s+(/[^\s'\"]+?\.(?:json|txt))\.?(?=\s|$)", re.S)
+
+
+def saved_result(text: str) -> tuple[pathlib.Path | None, str | None]:
+    """(path, body) for a result the host spilled to a file.
+
+    (None, None) when the text names no file; (path, None) when it names one
+    that cannot be read, which is the ordinary state of a rebuild since the
+    CLI's tool-results/ files are temporary; (path, body) when it can. Only
+    ever reads a path the host itself named in the result it returned.
     """
-    m = OFFLOADED.search(text or "")
+    m = SAVED_TO.search(text or "")
     if not m:
-        return None
+        return None, None
+    path = pathlib.Path(m.group(1))
     try:
-        body = pathlib.Path(m.group(1)).read_text()
+        return path, path.read_text()
     except OSError:
-        return None
-    return resource_json(body)
+        return path, None
+
+
+def offloaded_json(text: str) -> dict[str, Any] | None:
+    """The response body a host spilled to a file, read back, or None."""
+    _, body = saved_result(text)
+    return resource_json(body) if body is not None else None
 
 
 def resource_json(text: str) -> dict[str, Any] | None:
@@ -594,40 +650,39 @@ def path_breaches(events: list[dict[str, Any]],
     return list(r["reasons"])
 
 
-# Two spellings, from two CLI paths: the persisted-output stub, and the MCP
-# token-cap error ("result ... exceeds maximum allowed tokens. Output has been
-# saved to <path>"). Both leave the whole payload on disk.
-PERSISTED_STUB = re.compile(
-    r"(?:<persisted-output>.*?Full output saved to|Output has been saved to):? "
-    r"(/\S+?\.(?:json|txt))\.?(?=\s|$)", re.S)
-
-
 def result_text(block: dict[str, Any]) -> str:
-    """The text of one tool_result block, with a persisted stub resolved.
-
-    Above a size the CLI decides, a tool result reaches the answerer as a
-    `<persisted-output>` stub: a file path and a 2 KB preview. Measured on the
-    first arm against a real model, 14 of 74 get_context results (53 to 70 KB each) arrived that
-    way, and the answerer followed the path with Read every time. Reading the
-    stub as the payload scored those calls as zero entities delivered, which
-    is the opposite of what happened: the whole ranking was on disk. So when
-    the stub names a file that still exists, its content is the result.
+    """The text of one tool_result block, with a spilled result read back.
+
+    Above a size the CLI decides, a tool result reaches the answerer as a note
+    naming a file plus a 2 KB preview. Measured on the first arm against a real
+    model, 14 of 74 get_context results (53 to 70 KB each) arrived that way and
+    the answerer followed the path with Read every time. Reading the note as
+    the payload scored those calls as zero entities delivered. When the file is
+    still there its content is the result; when it is gone (the CLI's
+    tool-results/ files are temporary, so a later rebuild always lands here)
+    the note comes back unchanged and the caller records the call as
+    unmeasured through `saved_result`, never as zero. Text that already parses
+    as a result is returned as it is, whatever paths it mentions.
     """
     text = _raw_result_text(block)
-    m = PERSISTED_STUB.search(text)
-    if m:
-        path = pathlib.Path(m.group(1))
-        if path.exists():
-            try:
-                blocks = json.loads(path.read_text())
-            except (OSError, json.JSONDecodeError):
-                return path.read_text()
-            if isinstance(blocks, list):
-                return "\n".join(b.get("text", "") for b in blocks
-                                 if isinstance(b, dict) and b.get("type") == "text")
-            if isinstance(blocks, dict) and blocks.get("type") == "text":
-                return blocks.get("text", "")
-    return text
+    if resource_json(text) is not None:
+        # Already a result. A note naming a file is looked for only when the
+        # text is not one, so a doc string that mentions a real path inside a
+        # JSON result is never read in its place.
+        return text
+    _, body = saved_result(text)
+    if body is None:
+        return text
+    try:
+        blocks = json.loads(body)
+    except json.JSONDecodeError:
+        return body
+    if isinstance(blocks, list):
+        return "\n".join(b.get("text", "") for b in blocks
+                         if isinstance(b, dict) and b.get("type") == "text")
+    if isinstance(blocks, dict) and blocks.get("type") == "text":
+        return blocks.get("text", "")
+    return body
 
 
 def _raw_result_text(block: dict[str, Any]) -> str:
@@ -1225,8 +1280,44 @@ def _norm(q: str) -> str:
     never compared equal and `declared` never fired -- the harness fell through
     to `last_ok` and re-executed a probe. On one acceptance arm that read a
     model edit as a regression: the answer led with the filtered figure and
-    the harness graded the unfiltered probe it ran afterwards."""
-    return " ".join((q or "").replace(";", " ").split())
+    the harness graded the unfiltered probe it ran afterwards.
+
+    Only a `;` outside a string literal is a clause separator, and the lexer's
+    other two rules are honoured too: inside a string a backslash escapes the
+    next character (`'O\\'Brien'` does not end at the escaped quote), and
+    outside one `--` or `//` starts a comment that runs to the end of the line
+    (an apostrophe in it opens no string). Without those, an escaped quote or a
+    comment apostrophe made the printed form and the executed form normalise
+    differently, and the harness graded the probe query instead."""
+    out: list[str] = []
+    quote: str | None = None
+    q = q or ""
+    i, n = 0, len(q)
+    while i < n:
+        ch = q[i]
+        if quote:
+            out.append(ch)
+            if ch == "\\" and i + 1 < n:
+                out.append(q[i + 1])
+                i += 2
+                continue
+            if ch == quote:
+                quote = None
+        elif q.startswith("--", i) or q.startswith("//", i):
+            j = q.find("\n", i)
+            j = n if j < 0 else j
+            out.append(q[i:j])
+            i = j
+            continue
+        elif ch in ("'", '"', "`"):
+            quote = ch
+            out.append(ch)
+        elif ch == ";":
+            out.append(" ")
+        else:
+            out.append(ch)
+        i += 1
+    return " ".join("".join(out).split())
 
 
 def _model_path_of(query: str, runs: list[dict[str, Any]]) -> str | None:
@@ -2019,12 +2110,23 @@ def run_answerer(case: dict[str, Any], a: argparse.Namespace,
                     calls.append({**info, "error": text[:300] if failed else None,
                                   "rankedSummary": None})
                 else:
+                    if failed:
+                        # The call errored (an answerer that left out `scopes`
+                        # gets an MCP validation error). That is no ranking at
+                        # all, not a ranking of zero: recorded like a spilled
+                        # file that cannot be read, with no summary, so the
+                        # retrieval score never counts it as a search that
+                        # found nothing.
+                        calls.append({**info, "error": text[:300],
+                                      "rankedSummary": None})
+                        continue
                     # The host may have spilled the body to a file. Read it
                     # back rather than scoring the notice as an empty response.
                     if payload is None:
                         payload = offloaded_json(text)
-                    if payload is None and OFFLOADED.search(text or ""):
-                        # Named a file we could not read. Record NO summary
+                    if payload is None and saved_result(text)[0] is not None:
+                        # Named a file we could not read (a rebuild after the
+                        # CLI's temporary file is gone). Record NO summary
                         # rather than an empty one: unmeasured is the truth,
                         # and a zero here is a miss the run did not observe.
                         calls.append({**info,
@@ -2745,7 +2847,8 @@ def main(argv: list[str] | None = None) -> int:
                     help="model within the package; defaults to set.json targetModelPath")
     ap.add_argument("--model-repo", default=None, type=pathlib.Path,
                     help="the git checkout the MODEL is versioned in, recorded "
-                         "as modelGitSha (dirty-marked over this path). The "
+                         "as modelGitSha (dirty-marked over --model-dir, or over "
+                         "this whole path without it). The "
                          "Publisher serves a copy under publisher_data/, so "
                          "there is no way to infer it; without this the run "
                          "carries modelSha, the content pin, and no git pin")
@@ -2754,8 +2857,11 @@ def main(argv: list[str] | None = None) -> int:
                          "was served from. The -dirty marker on modelGitSha is "
                          "decided over this path, so an unrelated untracked file "
                          "elsewhere in the repo (a scratch notebook, a run "
-                         "directory) does not stamp a clean model dirty. "
-                         "Recorded as modelDir; defaults to the whole repo")
+                         "directory) does not stamp a clean model dirty. A "
+                         "relative path is taken from --model-repo (not from the "
+                         "working directory, unlike diagnose.py's --model-dir); "
+                         "a path that does not exist records no pin. Recorded as "
+                         "modelDir; defaults to the whole repo")
     ap.add_argument("--skills-root", default=None,
                     help="a checkout holding skills/ and manifests/ to load the "
                          "answerer's and judge's doctrine from -- a Publisher "
@@ -2831,9 +2937,10 @@ def main(argv: list[str] | None = None) -> int:
                          "delete events.jsonl, find -name judge.md -delete) was "
                          "done three times on one set")
     ap.add_argument("--rejudge", action="store_true",
-                    help="with --rebuild: reuse the answers but score them "
-                         "again, for a judge or rubric change")
+                    help="reuse the saved answers but score them again, for a "
+                         "judge or rubric change. Implies --rebuild")
     a = ap.parse_args(argv)
+    imply_flags(a)
     # Resolved once here rather than per attempt: the answerer's granted tool
     # list is part of what a run measured, so it must not vary within a run.
     a.hosted_tools = hosted_tools(
@@ -3269,7 +3376,7 @@ def main(argv: list[str] | None = None) -> int:
         # nobody said where that is.
         modelRepo=str(a.model_repo) if a.model_repo else None,
         modelDir=str(a.model_dir) if a.model_dir else None,
-        modelGitSha=(git_sha(a.model_repo, scope=a.model_dir or a.model_repo)
+        modelGitSha=(git_sha(a.model_repo, scope=model_scope(a.model_repo, a.model_dir))
                      if a.model_repo else None),
         skillsVersion=ledger.skills_git_sha(a.roots[0]),
         skillsRoot=str(a.roots[0]),