@malloy-publisher/skills 0.1.28 → 0.1.29

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.28",
+   "version": "0.1.29",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # Evaluate One Answer

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # Diagnose One Answer

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-improve
 description: 'Make the smallest safe Malloy model edit that closes a diagnosed model-owned gap, with a probe receipt for every factual claim. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # Improve the Model

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files under the model package''s evals/ directory; checkpoints are git commits of the model repo. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.28
+version: 0.1.29
 ---
 
 # The Evaluation Loop

package/skills/eval-loop/templates/eval-run-package/README.md

@@ -37,17 +37,22 @@ where the failures are, retrieval, the backlog. Publisher renders it natively, s
 these are Malloy reading the model directly with no JavaScript in between and
 nothing to drift.
 
-**`public/index.html`** is the case list -- one expandable row per case, with a
-verdict pill and a row of dots per arm (the entities the golden depends on: a
-ranked entity, a sibling-source alias, in a returned source's docs, or missing).
-Opening a row shows the golden as a table, then per arm: the effort line (turns,
-calls, errors, seconds, dollars), the judge's reasoning, the re-executed rows,
-and the attempt as a **timeline** -- every `get_context` and `execute_query` with
-its input and result, and the prose between them. That is the view a verdict
-cannot give: whether a wrong number came out of a wrong query or a right query
-read wrongly. A notebook cannot do that, and drilling into one case is most of
-what reading an eval consists of. Data comes from `steps.csv`, `required.csv`
-and the `prediction` column on attempts, all built from the run's transcripts.
+**`public/index.html`** leads with the score and where every question landed,
+then the diagnosis backlog (failures grouped by shared cause), an effort strip
+(one dot per question), and the case list: one row per case, with a verdict pill
+and a row of dots per arm (the entities the golden depends on: a ranked entity, a
+sibling-source alias, in a returned source's docs, or missing). Clicking a row
+opens that case in a side panel. Its **Summary** tab says why the judge decided
+as it did, puts the golden beside the rows the agent's final query returns when
+the harness re-runs it, shows the agent's final message, which required entities
+search delivered, and what the attempt cost. Its **Step by step** tab is the
+attempt as a list -- every `get_context` and `execute_query` and the prose
+between them, one line each, opened in place to show the full query and its
+result. That is the view a verdict cannot give: whether a wrong number came out
+of a wrong query or a right query read wrongly. A notebook cannot do that, and
+drilling into one case is most of what reading an eval consists of. Data comes
+from `steps.csv`, `required.csv` and the `prediction` column on attempts, all
+built from the run's transcripts.
 
 Both read the same model, which is what stops the two from disagreeing.
 

package/skills/eval-loop/templates/eval-run-package/eval_run.malloy

@@ -775,7 +775,7 @@ query: case_matrix is scores -> {
     arms is count()
     arms_passed is passed
   nest: by_arm is {
-    group_by: arm is runs.label, verdict, outcome, confidence,
+    group_by: arm is runs.label, verdict, outcome, confidence, counts,
               where_to_fix is retrieval.where_to_fix,
               recall is retrieval.recall
     order_by: arm