@malloy-publisher/skills 0.1.28 → 0.1.30

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.28",
+   "version": "0.1.30",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # Evaluate One Answer

package/skills/eval-answer/scripts/score_retrieval_test.py

@@ -601,6 +601,16 @@ class LabelsMatchTheRunPackage(unittest.TestCase):
                                "eval-run-package", *parts)) as fh:
             return fh.read()
 
+    def notebook_text(self):
+        """The notebook, at its `.malloy` path or the legacy `.malloynb` one that
+        run packages built before the switch still carry."""
+        for parts in (("notebooks", "eval_run.malloy"), ("eval_run.malloynb",)):
+            try:
+                return self.package_file(*parts)
+            except FileNotFoundError:
+                continue
+        raise FileNotFoundError("eval-run-package has no notebook")
+
     def emitted(self):
         from score_retrieval import (DELIVERED, MODEL, NEVER_ASKED,
                                      NOT_RETURNED, REFUSAL, UNMEASURED)
@@ -611,7 +621,7 @@ class LabelsMatchTheRunPackage(unittest.TestCase):
         """The legend is what a reader consults to interpret the column, so a
         label missing from it is worse than no legend. It named `documentation`
         -- a value that never appears -- and omitted `never asked` entirely."""
-        text = self.package_file("eval_run.malloynb")
+        text = self.notebook_text()
         for label in self.emitted():
             self.assertIn(f"**{label}**", text, label)
 
@@ -643,11 +653,11 @@ class LabelsMatchTheRunPackage(unittest.TestCase):
 
     def test_no_retired_label_survives_in_the_package(self):
         # Each of these was a real value once; each now matches nothing.
-        for f in (("eval_run.malloy",), ("eval_run.malloynb",),
-                  ("public", "app.js"), ("README.md",)):
-            text = self.package_file(*f)
+        for text in (self.package_file("eval_run.malloy"), self.notebook_text(),
+                     self.package_file("public", "app.js"),
+                     self.package_file("README.md")):
             for retired in ("query construction", "retrieval ranking"):
-                self.assertNotIn(retired, text, f"{f}: {retired}")
+                self.assertNotIn(retired, text, retired)
 
 
 class Summary(unittest.TestCase):

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # Diagnose One Answer

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-improve
 description: 'Make the smallest safe Malloy model edit that closes a diagnosed model-owned gap, with a probe receipt for every factual claim. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # Improve the Model

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files under the model package''s evals/ directory; checkpoints are git commits of the model repo. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.28
+version: 0.1.30
 ---
 
 # The Evaluation Loop

package/skills/eval-loop/reference/running-a-run.md

@@ -121,7 +121,7 @@ Three layers, in this order, and the order is what makes it readable:
 4. **DEEP DIVE.** A run directory is JSONL, which is a record and not a
    report, so this layer is the two commands that turn it into something you
    read: `build_run_package.py` (a Malloy model over the run's CSVs,
-   `eval_run.malloynb` for the aggregate tables, and an in-package HTML app for
+   `notebooks/eval_run.malloy` for the aggregate tables, and an in-package HTML app for
    the case matrix and its per-case drawer), then a `POST .../packages` that
    registers it on the Publisher already running, with no restart. It prints
    the resulting URL with the run's own paths filled in.

package/skills/eval-loop/scripts/build_run_package.py

@@ -10,7 +10,7 @@ an in-package HTML app for the case matrix.
 WHY BOTH A NOTEBOOK AND AN APP
 
 They are not redundant, and the split is not stylistic. Publisher renders a
-`.malloynb` natively, so the aggregate tables -- pass rate, effort, most-missed
+`.malloy` notebook natively, so the aggregate tables -- pass rate, effort, most-missed
 entities, the backlog -- are best expressed as Malloy and left alone: no
 JavaScript to drift out of step with the model. What a notebook cannot do is
 open a drawer, and reading an eval is mostly drilling into one case. So the
@@ -530,9 +530,11 @@ def build(run_dirs: list[pathlib.Path], set_dir: pathlib.Path,
         "n_cases", "proposed_edit", "evidence", "confidence"])
     write_csv(data / "cluster_members.csv", member_rows, ["cluster_id", "qid"])
 
-    for name in ("publisher.json", "eval_run.malloy", "eval_run.malloynb"):
+    for name in ("publisher.json", "eval_run.malloy"):
         if (TEMPLATE / name).exists():
             shutil.copy(TEMPLATE / name, out / name)
+    if (TEMPLATE / "notebooks").exists():
+        shutil.copytree(TEMPLATE / "notebooks", out / "notebooks", dirs_exist_ok=True)
     if (TEMPLATE / "public").exists():
         shutil.copytree(TEMPLATE / "public", out / "public", dirs_exist_ok=True)
 

package/skills/eval-loop/scripts/build_run_package_test.py

@@ -75,6 +75,7 @@ class SourcesMatchTheCsvs(unittest.TestCase):
             capture_output=True, text=True, timeout=300)
         assert p.returncode == 0, p.stderr
         cls.data = out / "data"
+        cls.out = out
 
     @classmethod
     def tearDownClass(cls):
@@ -92,6 +93,10 @@ class SourcesMatchTheCsvs(unittest.TestCase):
         self.assertTrue(found, "no duckdb.table sources parsed from the model")
         return found
 
+    def test_the_package_carries_the_notebook_at_its_malloy_path(self):
+        self.assertTrue((self.out / "notebooks" / "eval_run.malloy").exists())
+        self.assertFalse((self.out / "eval_run.malloynb").exists())
+
     def test_every_declared_column_is_actually_written(self):
         for name, stem, block in self.sources():
             if not (self.data / f"{stem}.csv").exists():

package/skills/eval-loop/scripts/run_baseline.py

@@ -1748,7 +1748,7 @@ def summary_lines(*, out: pathlib.Path, set_dir: pathlib.Path, events_n: int,
     lines += ["", "DEEP DIVE",
               "  A run directory is JSONL, which is a record, not a report.",
               "  build_run_package.py turns it into a servable Malloy package:",
-              "  a model over the run's CSVs, eval_run.malloynb for the",
+              "  a model over the run's CSVs, notebooks/eval_run.malloy for the",
               "  aggregate tables, and an in-package HTML app for the case",
               "  matrix and its per-case drawer. Register it and open it:",
               "",

package/skills/eval-loop/templates/eval-run-package/README.md

@@ -1,7 +1,7 @@
 # Eval-run package
 
 A finished run as a Malloy package you can open: the semantic model in
-`eval_run.malloy`, a notebook in `eval_run.malloynb`, an HTML data app in
+`eval_run.malloy`, a notebook in `notebooks/eval_run.malloy`, an HTML data app in
 `public/`, and CSV under `data/` written by `../../scripts/build_run_package.py`.
 
 ```bash
@@ -32,22 +32,27 @@ product, found on data we control and understand.
 
 Both, and the split is not stylistic.
 
-**`eval_run.malloynb`** holds the analytical tables: pass rate, cost, effort,
+**`notebooks/eval_run.malloy`** holds the analytical tables: pass rate, cost, effort,
 where the failures are, retrieval, the backlog. Publisher renders it natively, so
 these are Malloy reading the model directly with no JavaScript in between and
 nothing to drift.
 
-**`public/index.html`** is the case list -- one expandable row per case, with a
-verdict pill and a row of dots per arm (the entities the golden depends on: a
-ranked entity, a sibling-source alias, in a returned source's docs, or missing).
-Opening a row shows the golden as a table, then per arm: the effort line (turns,
-calls, errors, seconds, dollars), the judge's reasoning, the re-executed rows,
-and the attempt as a **timeline** -- every `get_context` and `execute_query` with
-its input and result, and the prose between them. That is the view a verdict
-cannot give: whether a wrong number came out of a wrong query or a right query
-read wrongly. A notebook cannot do that, and drilling into one case is most of
-what reading an eval consists of. Data comes from `steps.csv`, `required.csv`
-and the `prediction` column on attempts, all built from the run's transcripts.
+**`public/index.html`** leads with the score and where every question landed,
+then the diagnosis backlog (failures grouped by shared cause), an effort strip
+(one dot per question), and the case list: one row per case, with a verdict pill
+and a row of dots per arm (the entities the golden depends on: a ranked entity, a
+sibling-source alias, in a returned source's docs, or missing). Clicking a row
+opens that case in a side panel. Its **Summary** tab says why the judge decided
+as it did, puts the golden beside the rows the agent's final query returns when
+the harness re-runs it, shows the agent's final message, which required entities
+search delivered, and what the attempt cost. Its **Step by step** tab is the
+attempt as a list -- every `get_context` and `execute_query` and the prose
+between them, one line each, opened in place to show the full query and its
+result. That is the view a verdict cannot give: whether a wrong number came out
+of a wrong query or a right query read wrongly. A notebook cannot do that, and
+drilling into one case is most of what reading an eval consists of. Data comes
+from `steps.csv`, `required.csv` and the `prediction` column on attempts, all
+built from the run's transcripts.
 
 Both read the same model, which is what stops the two from disagreeing.
 
@@ -116,7 +121,7 @@ hole -- that exact bug is why the tables are cross-checked rather than trusted.
 ```bash
 publisher --server_root <parent-of-package> --mcp_port 4049
 # app      http://localhost:4000/environments/evals/packages/eval-run/index.html
-# notebook http://localhost:4000/evals/eval-run/eval_run.malloynb
+# notebook http://localhost:4000/evals/eval-run/notebooks/eval_run
 ```
 
 The two URLs are different on purpose and neither is guessable. `public/` is
@@ -133,13 +138,14 @@ file:
 
 ```bash
 curl -s .../api/v0/environments/<env>/packages/<pkg>/models      # compiles the model
-curl -s .../api/v0/environments/<env>/packages/<pkg>/notebooks   # lists .malloynb
+curl -s .../api/v0/environments/<env>/packages/<pkg>/notebooks   # lists the notebook
 ```
 
 `models` returns a 424 with the full compiler error when the Malloy is broken,
 which is the fastest way to see a compile failure. Notebooks are listed by
-`notebooks` and are **not** in `models`; asking for one under `models` returns
-`404 "<file> is a notebook"`. A notebook fetched from `notebooks/<file>` reports
+`notebooks` and are **not** in the `models` list. A `.malloy` notebook still compiles
+as a model, so `models/notebooks/eval_run.malloy` shows its compile errors; only a
+legacy `.malloynb` returns `404 "<file> is a notebook"` there. A notebook fetched from `notebooks/<file>` reports
 its cells' compiled schemas under `modelInfo`, but only the cells it treats as
 anonymous queries -- do not read a low count there as cells failing to compile. To
 verify every cell, run each named query through `execute_query`.

package/skills/eval-loop/templates/eval-run-package/eval_run.malloy

@@ -775,7 +775,7 @@ query: case_matrix is scores -> {
     arms is count()
     arms_passed is passed
   nest: by_arm is {
-    group_by: arm is runs.label, verdict, outcome, confidence,
+    group_by: arm is runs.label, verdict, outcome, confidence, counts,
               where_to_fix is retrieval.where_to_fix,
               recall is retrieval.recall
     order_by: arm

package/skills/eval-loop/templates/eval-run-package/{eval_run.malloynb → notebooks/eval_run.malloy}

@@ -1,4 +1,8 @@
->>>markdown
+##" One or more scored runs over one evaluation set: the analytical tables.
+## artifact { kind=notebook title="Eval run" }
+import "../eval_run.malloy"
+
+##|(markdown)
 # Eval run
 
 One or more scored runs over one evaluation set. Publisher renders this natively,
@@ -12,9 +16,9 @@ open a drawer.
 
 Read it top to bottom: the headline, then whether the headline is trustworthy,
 then where the failures are, then what to do about them.
->>>malloy
-import 'eval_run.malloy'
->>>markdown
+|##
+
+##|(markdown)
 ## 1. The headline
 
 Pass rate over **decided** verdicts -- `match` against `no_match`. Two things are
@@ -28,9 +32,11 @@ quietly depresses every arm:
 Read `confident_count` alongside the rate: two arms are only comparable when they
 answered the same cases. Read `near_matches` too. It is the leading indicator of
 rubrics going vague, and it moves before the pass rate does.
->>>malloy
+|##
+
 run: run_summary
->>>markdown
+
+##|(markdown)
 ## 2. Is the headline trustworthy?
 
 A pass rate means nothing if the attempts behind it were contaminated, if the
@@ -39,7 +45,8 @@ harness fell over, or if the arms did not face the same set.
 Any contamination at all voids the run -- an attempt that saw the gold answer is
 not evidence about the model. Harness errors are excluded from the rate rather
 than counted as failures: a timeout is the eval failing, not the answerer.
->>>malloy
+|##
+
 run: attempts -> {
   group_by: arm is runs.label
   aggregate:
@@ -50,18 +57,21 @@ run: attempts -> {
     judge_version is runs.judge_version.max()
     set_version is runs.set_version.max()
 }
->>>markdown
+
+##|(markdown)
 ## 3. What it cost
 
 The claim a semantic model makes is not only that the agent gets the right
 answer, but that it gets there quickly: fewer turns, fewer tokens, less money.
 An arm that wins on accuracy while costing three times as much has not obviously
 won.
->>>malloy
+|##
+
 run: cost
->>>malloy
+
 run: effort
->>>markdown
+
+##|(markdown)
 ## 4. Where the failures are
 
 Every failure carries exactly one **where to fix**, so these counts sum to the
@@ -82,9 +92,11 @@ tables below disagree with each other.
 - **coverage not measured** -- a miss on a case with no measured coverage label.
   Retrieval's or the model's, and nothing says which, so it is charged to
   neither.
->>>malloy
+|##
+
 run: failures_by_where_to_fix
->>>markdown
+
+##|(markdown)
 ### Pass rate by coverage
 
 Coverage is the ceiling on what any agent can do. A model-shaped problem shows up
@@ -93,9 +105,11 @@ only by assembling it, and the agent could not.
 
 `absent` cases are scored on the **refusal** -- passing means the agent correctly
 declined.
->>>malloy
+|##
+
 run: pass_by_coverage
->>>markdown
+
+##|(markdown)
 ## 5. Retrieval
 
 **Per-question entity recall**: of the entities each golden answer depends on, how
@@ -103,16 +117,20 @@ many the agent's own `get_context` calls delivered -- as a ranked entity, under
 sibling source, or named in a returned source's documentation. It uses the search
 terms the answerer chose, so it moves when the answerer changes; use it to
 attribute a failure *within* one arm, not to compare retrieval across arms.
->>>malloy
+|##
+
 run: retrieval_as_used
->>>markdown
+
+##|(markdown)
 ### Entities most often needed and not returned
 
 The ranked retrieval backlog. An entity high on this list is one fix that several
 cases are waiting on.
->>>malloy
+|##
+
 run: most_missed_entities
->>>markdown
+
+##|(markdown)
 ## 6. What to do next
 
 Failures grouped by shared root cause, ordered by how many cases one edit would
@@ -122,17 +140,21 @@ clear. Eight failures are rarely eight problems.
 proposal is an open question, and that is an honest state -- a generated proposal
 that nobody stands behind is worse than a blank, because a reviewer has to
 disprove it.
->>>malloy
+|##
+
 run: backlog
->>>markdown
+
+##|(markdown)
 ### Coverage gaps in the set itself
 
 Independent of any run: cases the model cannot answer outright, with the entity
 that would close the gap. This is the model-improvement backlog even before an
 agent has attempted anything, and it is the part a customer can act on directly.
->>>malloy
+|##
+
 run: coverage_gaps
->>>markdown
+
+##|(markdown)
 ## 7. Refusal behaviour
 
 Refusal is scored in both directions, and getting one right while getting the
@@ -143,9 +165,11 @@ other wrong is a common and misleading result.
 - A case tagged **answerable-sounds-unanswerable** passes by *answering*. An agent
   that has learned to refuse whenever a question sounds hard will fail these while
   looking well-calibrated on the ones above.
->>>malloy
+|##
+
 run: refusal_behaviour
->>>markdown
+
+##|(markdown)
 ## 8. Comparing arms
 
 Only meaningful with two or more arms loaded.
@@ -156,12 +180,15 @@ one case. What matters is which cases actually moved, and in which direction.
 
 `arms_passed` between 1 and `arms` means the arms disagree. Read the nested
 verdicts to see who won each one.
->>>malloy
+|##
+
 run: flips
->>>markdown
+
+##|(markdown)
 ### Cases every arm failed
 
 Changing the answerer will not move these. They are where model and retrieval
 work pays off, and they are the most durable finding in the whole run.
->>>malloy
+|##
+
 run: failed_everywhere