@malloy-publisher/skills 0.1.35 → 0.1.37

package/README.md

@@ -77,8 +77,7 @@ skill points to for detail it does not inline. Start with `malloy-getting-starte
 `malloy-modeling` to build or change a model, `malloy-analysis` to explore and answer questions, and
 `malloy-review` to check Malloy for correctness.
 
-A few skills (`malloy-modeling`, `malloy-publish`, `malloy-document`, `malloy-getting-started`, and
-the `malloy` index) are written for a Publisher host and name Publisher's MCP tools
+A few skills (`malloy-modeling`, `malloy-publish`, `malloy-document`, and `malloy-getting-started`) are written for a Publisher host and name Publisher's MCP tools
 directly. The rest describe Malloy itself and refer to tools by bare name (`get_context`,
 `execute_query`, `search_malloy_docs`), because the prefix depends on the
 host. `search_database_schema` maps the same way if a shared skill starts using it. On Publisher those are `get_context`, `execute_query`, `search_malloy_docs`, and

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.35",
+   "version": "0.1.37",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.35
+version: 0.1.37
 ---
 
 # Evaluate One Answer
@@ -290,5 +290,4 @@ original answerer to "try again" with the new key in context.
 
 - `skill:eval-diagnose`: why it failed, after this record exists.
 - `skill:eval-improve`: smallest model edit, model-owned issues only.
-- The `malloy-analysis-pitfalls` skill: checks before you trust a result you ran.
-  The judge loads it by name; it is not part of the `eval` group.
+- Step 5 of the `malloy-analysis` skill: checks before you trust a result you ran.

package/skills/eval-answer/reference/ledger-schema.md

@@ -90,7 +90,7 @@ One JSON object per line:
 | `tags` | list |
 | `state` | `candidate` / `selected` / `excluded`. |
 | `source` | Where the case came from. |
-| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous` / `verified_wrong`, the last being a key a person has established is wrong, which `run_baseline.py` drops from every aggregate), `kind`, `value` or `path` (artifact under the set directory; a `scalar` value is an object naming the canonical query's column, `{"net_sales": 834216}`, never a bare number), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`skill:eval-judge` rules 10 and 11). |
+| `golden` | `status` (`verified` / `provisional` / `invalid` / `ambiguous` / `verified_wrong`, the last being a key a person has established is wrong, which `run_baseline.py` drops from every aggregate), `kind`, `value` or `path` (a `scalar` value is an object naming the canonical query's column, `{"net_sales": 834216}`, never a bare number; a `rows` golden's `path` is relative to the set directory and names either a CSV with a header row or a JSON array of row objects, and `value` wins when both are present -- `gold/<qid>.json`, the derivation record below, is neither and is refused; a file with no header or no data rows is an error, so a key that really is no rows is written `value: []`), `canonicalQuery` (runs against the **truth** package, never the model under test), `verifiedBy`, and `verification` -- `{primaryAxis, variesAxis, note}` naming what the second derivation varied. **A golden is written only after two differently shaped derivations agree** (the truth query and a second one that varies a different axis -- group by something else, sum a different way, take a different route through the raw tables); `gold/<qid>.json` holds both (`truthRows`, `verifyRows`, `agreement`). Two derivations that vary the same axis share every other blind spot: that is how a golden that summed three overlapping time slices, 3x too high, passed its check. `verify_goldens.py` re-derives the value, reads the accepting clause of the rubric against the rows, and flags a second derivation that names no axis. **The question asks for data, never for an interpretation of it**, and either fixes the grain or the rubric accepts a correct figure at any stated grain (`skill:eval-judge` rules 10 and 11). |
 | `golden.kind` | What the key IS. A value (`scalar`, rows), `unanswerable`, or `criteria`: prose with no value, where the author's criteria are the whole key. `criteria` is the one kind that is legitimately `verified` on arrival, because there is no derivation to perform and nothing that can be numerically wrong; an imported key holding a VALUE is `provisional` until re-derived (`skill:eval-import`). **`verify_goldens.py --promote` is the only thing that writes `golden.status`**: it marks a `provisional` golden `verified` when its value re-derived cleanly from the truth package AND a second derivation exists. `--refresh` rewrites a drifted VALUE and never changes a status; `invalid` and `ambiguous` are settled by a person through the golden side door. |
 | `golden.rubric` | **On the golden, not on the case.** The prose the judge is shown as `CASE RUBRIC`, naming what counts as correct and what does not. `run_baseline.py` reads `golden.rubric`; a rubric written at the case's top level is silently not passed, the judge is told `CASE RUBRIC: none`, and it then scores from the golden value and the answer alone -- which reads as a judge that ignores its instructions. |
 | `golden.mustState` | **On the golden, not on the case.** What the answer has to say out loud. Read from `golden.mustState` by the run-package builder. |
@@ -180,6 +180,7 @@ exist in the flat shape, so the flat shape is the contract.) `kind` is one of:
 | `submitted` | bool | False when there was no final query. Not a wrong answer. |
 | `final_query` | string or null | Required to replay. A named view is recorded as the Malloy it stands for, `run: <source> -> <view>`, so every consumer sees one shape and the query re-executes. |
 | `final_query_source` | string or null | How `final_query` was chosen: `declared` (the answer printed it), `last_ok` (the last call the server answered) or `last`. `last` is a warning: a trailing sanity probe may be standing in for the answer's own query. |
+| `final_givens` | object or null | The `givens` (Publisher runtime parameters, `{name: value}`) the call behind `final_query` was sent with. A query written against a given returns unfiltered rows when re-run without it. Null when the call passed none. |
 | `servedRevision` | string or null | From the package actually queried. |
 | `n_get_context` / `n_execute` / `n_execute_errors` | int | |
 | `host_tool_uses` | int | EVERY tool use the host logged, MCP calls included. It counted only the non-MCP ones until 2026-09-03, which made the under-report check below true of almost every clean attempt. |
@@ -213,9 +214,11 @@ One event per MCP `get_context` or `execute_query` the attempt made.
 | `retrieval_mode` | string or null | `get_context` only: the `retrieval` field of the response that answered, `semantic` or `lexical`. Null when the server named none, which means no embedding provider. Recorded per call because the semantic path can fall over partway through a run. |
 | `query` | string or null | `execute_query` only: the Malloy the call sent, so the final query can be chosen by which call the server actually answered. |
 | `modelPath` | string or null | `execute_query` only: the model file the call named. A source does not resolve outside the file that declares it, so re-execution needs this. |
+| `givens` | object or null | `execute_query` only: the `{name: value}` runtime parameters the call was sent with. Part of the same fact as `query` and `modelPath`: the text alone, re-run without them, returns a different population. |
+| `filterParams` | object or null | `execute_query` only: values for the model's legacy `#(filter)` source filters, a separate mechanism from `givens`. The harness captures `givens`; this stays allowed for records that carry it. |
 | `error` | string or null | |
 
-Never persist `execute_query` result rows, givens, or credentials.
+Never persist `execute_query` result rows or credentials. `givens` are recorded because they decide what the query means; they are filter values, not secrets, so do not put a credential in one.
 
 `targets` records the request; `rankedSummary` records the response. Without
 both, a low per-attempt recall has two readings that cannot be told apart: the

package/skills/eval-answer/scripts/golden_rows.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python3
+"""Read a `rows` golden, wherever it keeps its rows.
+
+A rows golden holds its key in `golden.value` (a list of row objects) or, for a
+long one, in a CSV named by `golden.path`, relative to the set folder. Code
+that read `value` alone treated a path-held golden as having no key: the judge
+was told the question was unanswerable, and the golden audit reported an error.
+
+`path` names a CSV with a header row, or a JSON array of row objects. `value`
+wins when both exist. A `path` that leaves the set folder, names a file that is
+not there, or holds no header or no data rows raises `GoldenRowsError` with the
+case and the path.
+"""
+from __future__ import annotations
+
+import csv
+import json
+import pathlib
+import re
+from typing import Any
+
+# The most rows a judge prompt carries. A golden past this is cut, and the
+# prompt says so, rather than letting one case fill the context.
+ROWS_CAP = 500
+
+_INT = re.compile(r"-?(0|[1-9]\d*)")
+_FLOAT = re.compile(r"-?(0|[1-9]\d*)?\.\d+([eE][-+]?\d+)?")
+
+
+class GoldenRowsError(ValueError):
+    """A rows golden's `path` cannot be read; the message names case and path."""
+
+
+def _cell(text: str | None) -> Any:
+    """A CSV cell as a value: numbers as numbers, empty as null.
+
+    Only canonical numerals convert, so an identifier such as `007` stays text.
+    `true` and `false` (lower case, as JSON and Publisher write them) become
+    booleans, so they compare equal to what a query returns.
+    """
+    if text is None or text == "":
+        return None
+    if text == "true":
+        return True
+    if text == "false":
+        return False
+    if _INT.fullmatch(text):
+        return int(text)
+    if _FLOAT.fullmatch(text):
+        return float(text)
+    return text
+
+
+def load_rows(golden: dict[str, Any] | None, set_dir: pathlib.Path | None,
+              qid: str) -> list[dict[str, Any]] | None:
+    """The golden's rows, all of them, or None when it holds none.
+
+    `value` is used as it stands when it is a list. Otherwise `path` is read
+    (CSV, or a JSON list of row objects).
+    """
+    g = golden or {}
+    if isinstance(g.get("value"), list):
+        return g["value"]
+    rel = g.get("path")
+    if g.get("value") is not None or not rel:
+        return None
+    if set_dir is None:
+        raise GoldenRowsError(
+            f"{qid}: golden.path {rel!r} cannot be read without the set folder")
+    root = pathlib.Path(set_dir).resolve()
+    target = (root / rel).resolve()
+    if not target.is_relative_to(root):
+        raise GoldenRowsError(
+            f"{qid}: golden.path {rel!r} is outside the set folder {root}; "
+            f"keep the file under the set and give its path relative to it")
+    if not target.is_file():
+        raise GoldenRowsError(
+            f"{qid}: golden.path {rel!r} does not exist under {root}")
+    try:
+        if target.suffix.lower() == ".json":
+            rows = json.loads(target.read_text())
+            if not (isinstance(rows, list)
+                    and all(isinstance(r, dict) for r in rows)):
+                raise GoldenRowsError(
+                    f"{qid}: golden.path {rel!r} is JSON but not a list of "
+                    f"row objects")
+        else:
+            rows = _read_csv(target, rel, qid)
+    except (OSError, ValueError, csv.Error) as exc:
+        if isinstance(exc, GoldenRowsError):
+            raise
+        raise GoldenRowsError(
+            f"{qid}: golden.path {rel!r} could not be read: {exc}") from exc
+    # A truncated export reads as zero rows, and a judge shown an empty key
+    # fails every correct answer. A key that really is "no rows" belongs in
+    # `value: []`, where nobody can mistake it for a lost file.
+    if not rows:
+        raise GoldenRowsError(
+            f"{qid}: golden.path {rel!r} has no data rows; re-export it, or "
+            f"write `value: []` on the golden if the answer is no rows")
+    return rows
+
+
+def _read_csv(target: pathlib.Path, rel: str, qid: str
+              ) -> list[dict[str, Any]]:
+    """The CSV's rows, keyed by its header.
+
+    `csv.reader`, not `DictReader`: DictReader skips a blank line, and DuckDB
+    writes a NULL in a one-column file as exactly that, so the row dropped. In
+    a one-column file a blank line is one empty value. In a wider file it
+    carries no value at all and is skipped, as before.
+    """
+    with target.open(newline="", encoding="utf-8-sig") as fh:
+        reader = csv.reader(fh)
+        header = next(reader, None)
+        if not header:
+            raise GoldenRowsError(
+                f"{qid}: golden.path {rel!r} has no header row; the first "
+                f"line must name the columns")
+        out = []
+        for row in reader:
+            if not row:
+                if len(header) > 1:
+                    continue
+                row = [""]
+            if len(row) != len(header):
+                raise GoldenRowsError(
+                    f"{qid}: golden.path {rel!r} line {reader.line_num} "
+                    f"has a different number of fields than the header "
+                    f"({len(header)}); fix the row")
+            out.append({k: _cell(v) for k, v in zip(header, row)})
+        return out
+
+
+def key_value(golden: dict[str, Any] | None, set_dir: pathlib.Path | None,
+              qid: str) -> Any:
+    """The golden's key as a reader should see it: its rows when they live in
+    `path`, otherwise `value` as it stands. Raises `GoldenRowsError`."""
+    rows = load_rows(golden, set_dir, qid)
+    return rows if rows is not None else (golden or {}).get("value")
+
+
+def key_value_or_note(golden: dict[str, Any] | None,
+                      set_dir: pathlib.Path | None, qid: str) -> Any:
+    """`key_value`, with an unreadable `path` given as a note in its place.
+
+    For a prompt that shows the key to an agent: a note that says the file is
+    broken is evidence, while a missing key reads as a keyless case.
+    """
+    try:
+        return key_value(golden, set_dir, qid)
+    except GoldenRowsError as exc:
+        return f"(the golden's rows could not be read: {exc})"
+
+
+def render(rows: list[dict[str, Any]], total: int | None = None) -> str:
+    """The rows as the judge sees them, cut at ROWS_CAP with the cut stated."""
+    total = len(rows) if total is None else total
+    shown = rows[:ROWS_CAP]
+    text = json.dumps(shown)
+    if total > len(shown):
+        text += (f" (first {len(shown)} of {total} rows; the rest are not "
+                 f"shown)")
+    return text

package/skills/eval-answer/scripts/golden_rows_test.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Tests for golden_rows.py: reading a `rows` golden that lives in a file."""
+import pathlib
+import shutil
+import tempfile
+import unittest
+
+import sys
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+import golden_rows  # noqa: E402
+
+
+class LoadRows(unittest.TestCase):
+    def setUp(self):
+        self.tmp = pathlib.Path(tempfile.mkdtemp())
+        self.set_dir = self.tmp / "myset"
+        (self.set_dir / "gold").mkdir(parents=True)
+
+    def tearDown(self):
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def csv(self, name, text):
+        (self.set_dir / "gold" / name).write_text(text)
+        return {"kind": "rows", "path": f"gold/{name}"}
+
+    def test_rows_come_from_the_csv_with_numbers_as_numbers(self):
+        g = self.csv("q1.csv", "region,total,share\nWest,12,0.5\nEast,7,0.25\n")
+        self.assertEqual(
+            golden_rows.load_rows(g, self.set_dir, "q1"),
+            [{"region": "West", "total": 12, "share": 0.5},
+             {"region": "East", "total": 7, "share": 0.25}])
+
+    def test_an_identifier_with_leading_zeros_stays_text(self):
+        g = self.csv("q1.csv", "code,n\n007,1\n")
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"code": "007", "n": 1}])
+
+    def test_an_empty_cell_is_null(self):
+        g = self.csv("q1.csv", "a,b\n1,\n")
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"a": 1, "b": None}])
+
+    def test_true_and_false_become_booleans(self):
+        g = self.csv("q1.csv", "id,active,note\n1,true,False\n2,false,truex\n")
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"id": 1, "active": True, "note": "False"},
+                          {"id": 2, "active": False, "note": "truex"}])
+
+    def test_a_row_longer_than_the_header_is_a_golden_rows_error(self):
+        g = self.csv("q1.csv", "region,total\nWest,12,EXTRA\n")
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows(g, self.set_dir, "q1")
+        self.assertEqual(
+            str(cm.exception),
+            "q1: golden.path 'gold/q1.csv' line 2 has a different number of "
+            "fields than the header (2); fix the row")
+
+    def test_a_row_shorter_than_the_header_is_a_golden_rows_error(self):
+        g = self.csv("q1.csv", "region,total\nWest,12\nEast\n")
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows(g, self.set_dir, "q1")
+        self.assertIn("line 3", str(cm.exception))
+
+    def test_an_empty_file_is_a_golden_rows_error(self):
+        g = self.csv("q1.csv", "")
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows(g, self.set_dir, "q1")
+        self.assertEqual(
+            str(cm.exception),
+            "q1: golden.path 'gold/q1.csv' has no header row; the first line "
+            "must name the columns")
+
+    def test_a_header_with_no_rows_is_a_golden_rows_error(self):
+        g = self.csv("q1.csv", "region,total\n")
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows(g, self.set_dir, "q1")
+        self.assertEqual(
+            str(cm.exception),
+            "q1: golden.path 'gold/q1.csv' has no data rows; re-export it, or "
+            "write `value: []` on the golden if the answer is no rows")
+
+    def test_an_empty_json_array_is_a_golden_rows_error(self):
+        g = self.csv("q1.json", "[]")
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows(g, self.set_dir, "q1")
+        self.assertIn("has no data rows", str(cm.exception))
+
+    def test_a_blank_line_in_a_one_column_file_is_a_null(self):
+        # DuckDB writes a one-column NULL as a blank line.
+        g = self.csv("q1.csv", "region\nWest\n\nEast\n")
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"region": "West"}, {"region": None},
+                          {"region": "East"}])
+
+    def test_a_blank_line_in_a_wider_file_is_skipped(self):
+        g = self.csv("q1.csv", "a,b\n1,2\n\n3,4\n")
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+
+    def test_value_wins_when_both_exist(self):
+        g = self.csv("q1.csv", "a\n1\n")
+        g["value"] = [{"a": 99}]
+        self.assertEqual(golden_rows.load_rows(g, self.set_dir, "q1"),
+                         [{"a": 99}])
+
+    def test_neither_value_nor_path_is_none(self):
+        self.assertIsNone(golden_rows.load_rows({"kind": "rows"},
+                                                self.set_dir, "q1"))
+
+    def test_a_missing_file_names_the_case_and_the_path(self):
+        with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+            golden_rows.load_rows({"kind": "rows", "path": "gold/nope.csv"},
+                                  self.set_dir, "q7")
+        self.assertIn("q7", str(cm.exception))
+        self.assertIn("gold/nope.csv", str(cm.exception))
+
+    def test_a_path_outside_the_set_is_refused(self):
+        (self.tmp / "secret.csv").write_text("a\n1\n")
+        for bad in ("../secret.csv", str(self.tmp / "secret.csv")):
+            with self.subTest(bad=bad):
+                with self.assertRaises(golden_rows.GoldenRowsError) as cm:
+                    golden_rows.load_rows({"kind": "rows", "path": bad},
+                                          self.set_dir, "q1")
+                self.assertIn("outside", str(cm.exception))
+                self.assertIn("q1", str(cm.exception))
+
+    def test_a_symlink_out_of_the_set_is_refused(self):
+        (self.tmp / "secret.csv").write_text("a\n1\n")
+        (self.set_dir / "gold" / "link.csv").symlink_to(self.tmp / "secret.csv")
+        with self.assertRaises(golden_rows.GoldenRowsError):
+            golden_rows.load_rows({"kind": "rows", "path": "gold/link.csv"},
+                                  self.set_dir, "q1")
+
+
+class KeyValue(unittest.TestCase):
+    """What diagnose, improve and the rubric audit show as the key."""
+
+    def setUp(self):
+        self.set_dir = pathlib.Path(tempfile.mkdtemp())
+        (self.set_dir / "gold").mkdir()
+        (self.set_dir / "gold" / "q1.csv").write_text("a\n1\n")
+
+    def tearDown(self):
+        shutil.rmtree(self.set_dir, ignore_errors=True)
+
+    def test_a_path_held_golden_gives_its_rows(self):
+        self.assertEqual(golden_rows.key_value(
+            {"kind": "rows", "path": "gold/q1.csv"}, self.set_dir, "q1"),
+            [{"a": 1}])
+
+    def test_a_value_golden_gives_its_value(self):
+        self.assertEqual(golden_rows.key_value(
+            {"kind": "scalar", "value": {"n": 3}}, self.set_dir, "q1"),
+            {"n": 3})
+
+    def test_an_unreadable_path_becomes_a_note(self):
+        self.assertEqual(
+            golden_rows.key_value_or_note(
+                {"kind": "rows", "path": "gold/nope.csv"}, self.set_dir, "q1"),
+            "(the golden's rows could not be read: q1: golden.path "
+            f"'gold/nope.csv' does not exist under {self.set_dir.resolve()})")
+
+
+class Render(unittest.TestCase):
+    def test_a_short_list_renders_whole(self):
+        text = golden_rows.render([{"a": 1}], total=1)
+        self.assertEqual(text, '[{"a": 1}]')
+
+    def test_a_long_list_is_cut_and_says_so(self):
+        rows = [{"a": i} for i in range(golden_rows.ROWS_CAP + 5)]
+        text = golden_rows.render(rows, total=len(rows))
+        self.assertIn(f"first {golden_rows.ROWS_CAP} of {len(rows)} rows", text)
+        self.assertNotIn(f'"a": {golden_rows.ROWS_CAP + 1}}}', text)
+
+
+if __name__ == "__main__":
+    unittest.main()

package/skills/eval-answer/scripts/init_truth_package.py

@@ -96,7 +96,14 @@ def place_ref(package: pathlib.Path, src: pathlib.Path,
         return ref, None
     if "/" not in ref and not DATA_EXT.search(ref):
         return ref, None  # a warehouse table name, not a file
+    # Beside the model file first (what this has always done). If nothing is
+    # there, the ref was written from the package root, as a model in a
+    # subfolder such as `_shared/` does: resolving it against that folder again
+    # doubles the folder name and finds nothing.
     target = (src.parent / ref).resolve()
+    from_root = (package / ref).resolve()
+    if not target.exists() and from_root.exists():
+        target = from_root
     try:
         rel = target.relative_to(package.resolve())
     except ValueError:

package/skills/eval-answer/scripts/init_truth_package_test.py

@@ -74,6 +74,26 @@ class RefsWorkOnceServed(unittest.TestCase):
         self.assertEqual(itp.place_ref(self.pkg, src, "u.parquet"),
                          (str(self.pkg / "models" / "u.parquet"), None))
 
+    def test_a_ref_written_from_the_package_root_is_not_resolved_against_its_own_folder(self):
+        # A model in `_shared/` names its data relative to the package root:
+        # `_shared/data/u.parquet`. Resolved against `_shared/` itself that is
+        # `_shared/_shared/data/u.parquet`, which does not exist.
+        (self.pkg / "_shared" / "data").mkdir(parents=True)
+        (self.pkg / "_shared" / "data" / "u.parquet").write_text("")
+        (self.pkg / "_shared" / "x.malloy").write_text("")
+        src = self.pkg / "_shared" / "x.malloy"
+        served, _ = itp.place_ref(self.pkg, src, "_shared/data/u.parquet")
+        self.assertEqual(pathlib.Path(served),
+                         self.pkg / "_shared" / "data" / "u.parquet")
+
+    def test_a_ref_beside_its_model_still_resolves_against_that_folder(self):
+        (self.pkg / "models").mkdir()
+        (self.pkg / "models" / "x.malloy").write_text("")
+        (self.pkg / "models" / "u.parquet").write_text("")
+        served, _ = itp.place_ref(self.pkg, self.pkg / "models" / "x.malloy",
+                                  "u.parquet")
+        self.assertEqual(pathlib.Path(served), self.pkg / "models" / "u.parquet")
+
     def test_a_warehouse_table_is_left_alone(self):
         self.assertEqual(itp.place_ref(self.pkg, self.pkg / "m.malloy",
                                        "analytics.public.orders"),

package/skills/eval-answer/scripts/ledger.py

@@ -65,6 +65,7 @@ EVENTS: dict[str, dict[str, set[str]]] = {
         "optional": {"question_sha", "servedRevision", "n_get_context",
                      "n_execute", "n_execute_errors", "host_tool_uses",
                      "mcp_tool_uses", "final_query_source",
+                     "final_givens",
                      "reported_calls", "contaminated", "contamination_reasons",
                      "input_tokens", "output_tokens", "cache_read_tokens",
                      "cache_write_tokens", "skills_invoked",
@@ -80,9 +81,11 @@ EVENTS: dict[str, dict[str, set[str]]] = {
         # either. Which retriever answered decides whether two runs are
         # comparable at all, and it is only knowable from the response that
         # answered.
-        # `query`, `modelPath` and `filterParams` are one fact about one
-        # call: the text that ran, the file it was written against, and the
-        # filter values it ran under. Recorded apart they drift, and two of
+        # `query`, `modelPath`, `givens` and `filterParams` are one fact about
+        # one call: the text that ran, the file it was written against, and
+        # the runtime parameter values it ran under (`givens` is what the
+        # harness captures from the answerer's call; `filterParams` is the
+        # legacy `#(filter)` mechanism). Recorded apart they drift, and two of
         # the three drift SILENTLY -- a replay against the wrong file at least
         # errors, while a replay under another report's filter values returns
         # real rows for the wrong population. Keeping all three on the event
@@ -94,7 +97,8 @@ EVENTS: dict[str, dict[str, set[str]]] = {
         # transcripts, and transcripts get pruned.
         "optional": {"targets", "target_shapes", "scopes", "rankedSummary",
                      "error", "traceId",
-                     "query", "modelPath", "filterParams", "retrieval_mode",
+                     "query", "modelPath", "givens", "filterParams",
+                     "retrieval_mode",
                      "at"},
     },
     "score": {

package/skills/eval-answer/scripts/ledger_test.py

@@ -178,6 +178,17 @@ class WriteContract(unittest.TestCase):
                          filterParams={"report_id": "123"})
         self.assertEqual(e["filterParams"], {"report_id": "123"})
 
+    def test_a_tool_call_and_an_attempt_may_carry_givens(self):
+        g = {"region": "West"}
+        e = ledger.event("tool_call", qid="q1", sample=0, phase="baseline",
+                         tool="execute_query", query="run: x -> y",
+                         modelPath="a.malloy", givens=g)
+        self.assertEqual(e["givens"], g)
+        e = ledger.event("attempt", qid="q1", sample=0, phase="baseline",
+                         submitted=True, final_query="run: x -> y",
+                         answer_text="a", transcriptPath="t", final_givens=g)
+        self.assertEqual(e["final_givens"], g)
+
     def test_an_attempt_may_carry_cache_write_tokens(self):
         # The fourth token column. Without it the ledger held a cache-inclusive
         # cost_usd and a breakdown that could not reproduce it.

package/skills/eval-answer/scripts/publisher_rest.py

@@ -105,18 +105,28 @@ def rows_from_result(body: Any) -> list[dict[str, Any]]:
 
 
 def query(base: str, environment: str, package: str, model: str, malloy: str,
-          timeout: int = 180) -> list[dict[str, Any]]:
-    """Run a Malloy query. Raises on transport, HTTP or parse failure."""
+          timeout: int = 180, givens: dict[str, Any] | None = None
+          ) -> list[dict[str, Any]]:
+    """Run a Malloy query. Raises on transport, HTTP or parse failure.
+
+    `givens` are the model's runtime parameters, sent as the request's own
+    `givens` field. A query written against a given and run without it returns
+    unfiltered rows, so a caller re-running someone else's query passes theirs.
+    """
+    body: dict[str, Any] = {"query": malloy, "compactJson": True}
+    if givens:
+        body["givens"] = givens
     req = urllib.request.Request(
         _query_url(base, environment, package, model),
-        data=json.dumps({"query": malloy, "compactJson": True}).encode(),
+        data=json.dumps(body).encode(),
         headers={"Content-Type": "application/json"})
     with urllib.request.urlopen(req, timeout=timeout) as r:
         return rows_from_result(json.loads(r.read().decode()))
 
 
 def try_query(base: str, environment: str, package: str, model: str,
-              malloy: str, timeout: int = 120
+              malloy: str, timeout: int = 120,
+              givens: dict[str, Any] | None = None
               ) -> tuple[list[dict[str, Any]], str | None]:
     """Run a Malloy query. Returns (rows, error); never raises.
 
@@ -125,7 +135,7 @@ def try_query(base: str, environment: str, package: str, model: str,
     """
     try:
         return query(base, environment, package, model, malloy,
-                     timeout=timeout), None
+                     timeout=timeout, givens=givens), None
     except urllib.error.HTTPError as exc:
         return [], f"HTTP {exc.code}: {exc.read().decode()[:300]}"
     except Exception as exc:                        # noqa: BLE001

package/skills/eval-answer/scripts/publisher_rest_test.py

@@ -111,6 +111,26 @@ class CompactJson(unittest.TestCase):
         self.assertIs(payload["compactJson"], True)
         self.assertEqual(payload["query"], "run: x -> { ... }")
 
+    def test_givens_travel_in_the_request_body(self):
+        seen = {}
+
+        class Resp:
+            def read(self_): return b'{"result": "[]"}'
+            def __enter__(self_): return self_
+            def __exit__(self_, *a): return False
+
+        def fake(req, timeout=None):
+            seen["payload"] = json.loads(req.data.decode())
+            return Resp()
+
+        with unittest.mock.patch.object(publisher_rest.urllib.request,
+                                        "urlopen", fake):
+            publisher_rest.query("http://x", "e", "p", "m.malloy", "run: x",
+                                 givens={"region": "West"})
+            self.assertEqual(seen["payload"]["givens"], {"region": "West"})
+            publisher_rest.query("http://x", "e", "p", "m.malloy", "run: x")
+            self.assertNotIn("givens", seen["payload"])
+
     def test_a_nest_keeps_its_field_names(self):
         # What the server returns under compactJson: `result` is a JSON
         # STRING of plain rows, nested names intact.