@malloy-publisher/skills 0.1.38 → 0.1.40

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.38",
+   "version": "0.1.40",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.38
+version: 0.1.40
 ---
 
 # Evaluate One Answer

package/skills/eval-answer/reference/ledger-schema.md

@@ -153,7 +153,8 @@ on that axis.
 | `goldenCheck` | What `verify_goldens.py` said before the run started: `N ok, M drifted, K other finding(s)`, or why it did not run (no truth server on a platform target; `--skip-golden-check`; rebuild). A run that started on a drifted set says so here rather than pretending its verdicts mean something. `run_baseline` calls `verify()` without a model, so the set-name audit (check 5) does not run in-arm; the trailing `N stale entity name(s)` is the in-arm lint's own count, and `entity-name lint not run` says there was no model text to lint against. Without it the field read `0 other finding(s)` for a check that never happened. |
 | `staleEntityNames` | The `expectedEntities` ids the compiled served model does not declare, from the in-arm lint (bare names from a search of the model text when the compiled model could not be read). Each one depresses the run's own recall on every case listing it, so a stale set reads as a retrieval failure until someone checks by hand. `null` when there was no model text to lint against (a platform target), which is a different fact from `[]`. |
 | `status` | `complete`; `aborted` when four consecutive attempts errored or found the server dead and the harness stopped rather than spend the rest of the budget on attempts nobody will trust; or `incomplete` when the arm ran to the end and still cannot report a rate, because `truncated` or `contaminated` is non-empty. `incomplete` is a third state, not a soft `complete`: the arm did everything asked of it and the number is still unearned. It does NOT stop the run being compared: `flip_table.py` scores the cases both arms did, and names the ones this arm left unscored -- an excluded case carries no verdict, so it can never be counted as a flip. |
-| `packageSha` / `servedRevision` *(pins)* | Taken from the server, not recomputed: `sourceContentSha` is a content hash over EVERY model path in the package, so an edit to an imported file moves it where a sha of the one `--model-path` does not; `servedRevision` is minted per load, so it identifies a load rather than content and is a poor pin alone. Measured: `publisher.json`'s `version` moves neither, and nothing in Publisher reads it -- it is not in the Package API schema and never returned, so it pins nothing. |
+| `packageSha` / `servedRevision` *(pins)* | Taken from the server, not recomputed: `sourceContentSha` is a content hash over EVERY model path in the package AND its `skills/` files AND its declared agents (their files and the `publisher.json` declaration), so an edit to an imported file or to a skill moves it where a sha of the one `--model-path` does not; `servedRevision` is minted per load, so it identifies a load rather than content and is a poor pin alone. Measured: `publisher.json`'s `version` moves neither, and nothing in Publisher reads it -- it is not in the Package API schema and never returned, so it pins nothing. |
+| `packageSkillsMode` / `packageSkillsSha` *(pins)* | How the package's own skills reached the answerer (`install` copied them into its workspace, `tool` left it to fetch them with `get_skill`, `off` withheld them, which is also what a package that ships none is recorded as), and a content hash of that `skills/` tree. A run that spawns no answerer (`--rebuild`, `--rejudge`, `--from`) carries the prior run's values, and a platform run records null. Two runs can differ only in guidance; without these the ledger cannot tell them apart, and a flip table between them would look like noise. |
 | `datasetSha` *(pin)* | Content hash of `set.json` + `cases.jsonl`. Deliberately SEPARATE from the model's sha: a golden repair is not a model change, and one pin covering both would make every answer-key fix read as an edit to the model, which is the distinction an A/B rests on. Automatic, so nobody has to remember it; `datasetVersion` stays beside it as the human-readable sequence. **Local targets only.** A hosted target that publishes IMMUTABLE versions needs none of this: the set rides inside the version, and immutability -- not hashing -- is what makes a pin trustworthy. There, `targetVersion` alone identifies model and set together. |
 | `doubtedGoldens` | The cases whose golden is not believed: `qid`, `gold_status` (`suspect` or `verified_wrong`), `gold_note`, and `declaredBy` (`judge` or `set`). A status the SET declared is a key somebody already settled, not an opinion the judge formed this run, and attributing both to the judge claimed a judgement that never happened. **Read this before diagnose.** These are dataset issues, not model failures, and they go through the golden side door in `skill:eval-loop`. Empty list when no key is in doubt from either source. Written from the same scan that prints the end-of-run warning, because a warning that lives only in console text is one scrollback away from sending a modelling agent at a model that is already right. |
 | `mode` / `setName` / `serverVersion` / `traceMode` / `callBudget` | Defined and accepted, **not yet written by any harness** -- kept in the schema for the platform target and the conductor, which need them. (`targetVersion` and `status` were in this list and are both written now.) |
@@ -357,7 +358,7 @@ rejected direction keeps its record.
 | `issue_ids` | |
 | `files` | Paths the edit touched. |
 | `diffSummary` | One line per file. |
-| `probes` | Query and result for each factual claim. |
+| `probes` | Query and result for each factual claim. May be empty for a `class: skill` candidate that asserts nothing about the data: a guide making no factual claim has nothing to probe. Empty is not a missing receipt there; a guide that DOES quote a number still needs one. |
 | `meaningChanged` | Entities whose *meaning* the edit changed; `[]` for a docs-only edit. |
 | `goldenSuspect` | Each `{qid, entity, stored, rederived}`: a golden this edit may have invalidated. Reported by the improver, never repaired by it. **Non-empty halts the acceptance check** until adjudicated through the golden side door. |
 | `goldenAudit` | The set's `verify_goldens.py` run against the edited model: `{ran, clean, model, tail}`. Catches drift and rubric-vs-model contradictions only; a golden whose value silently moved is invisible to it, which is why `goldenSuspect` exists alongside. |
@@ -373,7 +374,7 @@ Written by `eval-loop`, one per acceptance check decision, BEFORE any checkpoint
 |---|---|
 | `issue_ids` | |
 | `decision` | `accepted` / `rejected`. |
-| `class` | `docs` / `definition` / `retrieval` / `justified`. |
+| `class` | `docs` / `definition` / `skill` / `retrieval` / `justified`. `skill` is an edit to a package skill under the package's `skills/`; it never takes the documentation shortcut in the acceptance check. |
 | `baselineRunId` / `finalRunIds` | Plural: acceptance needs two independent runs. |
 | `regressions` | Case ids whose verdict got worse vs baseline. Must be empty to accept. |
 | `holdoutDelta` | Confident-verdict delta on the holdout slice. |

package/skills/eval-answer/scripts/ledger.py

@@ -217,7 +217,11 @@ RUN_OPTIONAL = {"label", "effort", "environment", "package", "modelPath",
                 "staleEntityNames",
                 "retrievalMode", "retrievalCalls", "retrievalGate",
                 "reExecution", "coverageReport",
-                "modelRepo"} | RUN_RECOMMENDED
+                "modelRepo",
+                # How the package's own skills reached the answerer, and a hash
+                # of the tree they came from. Two runs can differ ONLY in
+                # guidance, and without these the ledger cannot tell them apart.
+                "packageSkillsMode", "packageSkillsSha"} | RUN_RECOMMENDED
 
 
 def dataset_sha(set_dir: pathlib.Path) -> str | None:

package/skills/eval-answer/scripts/publisher_rest.py

@@ -148,6 +148,45 @@ def get_json(base: str, path: str, timeout: int = 30) -> Any:
         return json.loads(resp.read())
 
 
+def served_package_dir(base: str, environment: str,
+                       package: str) -> pathlib.Path | None:
+    """The directory the running Publisher serves this package FROM, or None.
+
+    A host commonly serves a copy rather than your working tree, so this is the
+    only path whose bytes are the ones answering queries.
+    """
+    try:
+        for proj in get_json(base, "api/v0/projects"):
+            if proj.get("name") == environment and proj.get("location"):
+                p = pathlib.Path(proj["location"]) / package
+                return p if p.is_dir() else None
+    except (OSError, ValueError):   # unreachable or non-JSON: absence is reported
+        return None
+    return None
+
+
+def package_skill_names(base: str, environment: str, package: str) -> list[str]:
+    """Skills the server says this package ships, or [] if it ships none.
+
+    Read from the server rather than from the working tree, because those are
+    two different questions: a host serves a COPY of the package. What the
+    answerer can actually reach is what this returns.
+
+    An unreachable or older server returns [] rather than raising: the caller
+    decides whether "the server serves none" contradicts what it asked for.
+    """
+    try:
+        rows = get_json(
+            base,
+            f"api/v0/environments/{urllib.parse.quote(environment)}"
+            f"/packages/{urllib.parse.quote(package)}/skills")
+    except (OSError, ValueError):                       # unreachable or non-JSON
+        return []
+    return sorted(r["name"] for r in rows
+                  if isinstance(r, dict) and r.get("origin") == "package"
+                  and isinstance(r.get("name"), str))
+
+
 def served_model_path(base: str, environment: str, package: str,
                       model_path: str) -> pathlib.Path | None:
     """Where the running Publisher reads this model from, or None.

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.38
+version: 0.1.40
 ---
 
 # Diagnose One Answer
@@ -68,9 +68,19 @@ never "C1" / "C2" / "C3":
 | `construction` | Did sufficient context arrive, and the agent still built the wrong query? |
 | `model-definition` | Is a measure, join, filter convention, or source semantically wrong? |
 
-`owner` is separate: `model`, `retrieval`, `agent-skill`, or `dataset`. There
-is no environment owner: an environment failure stops the run before
-diagnosis (see the boundary above), so no issue can carry it.
+`owner` is separate: `model`, `package-skill`, `retrieval`, `agent-skill`, or
+`dataset`. There is no environment owner: an environment failure stops the run
+before diagnosis (see the boundary above), so no issue can carry it.
+
+`package-skill` and `agent-skill` are both "the guidance was wrong", split by
+who can fix it here. A package can ship its own guides under `skills/`, and one
+whose name matches a built-in REPLACES it for that package, so a fix scoped to
+this package is a file in the model repo and lands in the same commit as a model
+edit. `agent-skill` is the shared skill, outside this repo and outside this
+loop. Choose `package-skill` only when the fix is genuinely specific to this
+package: this customer's vocabulary, this model's conventions, a trap only these
+sources have. A gap every package would hit is `agent-skill`, and widening a
+shared skill by editing one package's copy hides it from every other package.
 
 ### Assigning owner: one question
 
@@ -202,14 +212,20 @@ Prior `score` events are not rewritten. They keep the old `golden_revision`.
 | Code | When | Owner |
 |---|---|---|
 | `NEVER-ASKED` | no utterance targeted a needed concept | agent-skill, and model if nothing would have prompted the ask |
-| `VAGUE` | compound or generic utterances, so nothing could rank | agent-skill |
-| `QUESTION-VOCAB` | utterances parroted the question where the data uses other words | agent-skill, and model if that vocabulary is undocumented |
-| `RULE_UNWRITTEN` | the data is present and the model does not encode the rule for combining or filtering it, so whoever answers invents one. Qualified `(guessable)` when the data forces the rule -- list price minus sale price -- and `(arbitrary)` when it is a business decision nobody could derive, such as a season window or whether revenue is net of tax. Only the stated conventions tell those apart: from the model alone the two look identical, and an unqualified reading defaults to the harmless one | model: write the rule down |
+| `VAGUE` | compound or generic utterances, so nothing could rank | agent-skill: decomposition is a general skill. package-skill only if this package's concepts are what defeated it |
+| `QUESTION-VOCAB` | utterances parroted the question where the data uses other words | package-skill: map this business's words to the model's. model if the vocabulary belongs on the entity itself |
+| `RULE_UNWRITTEN` | the data is present and the model does not encode the rule for combining or filtering it, so whoever answers invents one. Qualified `(guessable)` when the data forces the rule -- list price minus sale price -- and `(arbitrary)` when it is a business decision nobody could derive, such as a season window or whether revenue is net of tax. Only the stated conventions tell those apart: from the model alone the two look identical, and an unqualified reading defaults to the harmless one | model: write the rule down, or expose a named measure. package-skill when no measure or doc can carry it, e.g. which population to default to |
 | `ASSUMED` | assumed a scope or convention instead of checking | model if nothing warned; agent-skill otherwise |
 | `WRONG-TYPE-OR-SCOPE` | asked, but with the wrong target type or an empty/wrong scope | agent-skill |
 
 If the agent could not reasonably have known to ask, that is a model gap.
 
+Between `package-skill` and `model` on any of these rows: if the fact belongs to
+one entity, put it on that entity's `#(doc)`, because a doc reaches every agent
+whether or not it loaded a guide. Reach for `package-skill` when the fact spans
+entities or is a rule rather than a property: which source to prefer, what the
+house default population is, what this business calls things.
+
 ### get_context / model
 
 | Code | When | Owner |
@@ -218,8 +234,8 @@ If the agent could not reasonably have known to ask, that is a model gap.
 | `NOT-RETURNED` | it exists, the ask was on target, it never came back | model: labels, docs, synonyms, index |
 | `LOW-RANK` | returned, buried under noise the agent reasonably skipped | model |
 | `AMBIGUOUS` | several candidates and nothing says which this question means. Judged over the entities AND their docs: a `#(doc)` that resolves the choice makes it `MODELLED`, which is why a sentence of documentation is usually the whole fix | model: "use X for …, Y when …" |
-| `GUIDANCE-NOT-RETRIEVED` | entities came back, governing guidance did not | model: put guidance on the entities agents search for |
-| `GUIDANCE-DECLINED` | guidance was retrieved and judged inapplicable | model: state the business default, not a caveat |
+| `GUIDANCE-NOT-RETRIEVED` | entities came back, governing guidance did not | model: put guidance on the entities agents search for. package-skill when it governs several sources and belongs on none |
+| `GUIDANCE-DECLINED` | guidance was retrieved and judged inapplicable | model: state the business default, not a caveat. package-skill if the default is a house rule rather than a property of one field |
 
 **Before diagnosing a `NOT-RETURNED`, check the call actually returned
 nothing.** The `tool_call` event carries `retrieval_mode` and `rankedSummary`.
@@ -254,7 +270,7 @@ with both queries. Otherwise it is still `NOT-RETURNED` / `LOW-RANK`.
 
 | Code | When | Owner |
 |---|---|---|
-| `WRONG-PICK` | needed entity returned, used a different one | model if indistinguishable; agent-skill if docs distinguished them |
+| `WRONG-PICK` | needed entity returned, used a different one | model if indistinguishable; package-skill if choosing between them is a rule of this package; agent-skill if the docs already distinguished them |
 | `SCOPE` | right entities, wrong population | model if the scope rule was undocumented |
 | `GRAIN` | right entities, wrong grain | model or agent-skill |
 | `FILTER-LITERAL` | filter literal did not match stored values | model (document the stored form) and agent-skill |
@@ -352,9 +368,10 @@ convention explains both. Same `owner` and same `component` is a hint, never a
 criterion: two `MISSING` issues about different missing entities are two
 clusters, and merging them produces an edit that fixes neither cleanly.
 
-Order clusters by how many cases they would fix. Cluster the non-model owners
-too, in their own clusters, so nothing is lost on the way to the backlog --
-but keep them separate, because only `owner: model` may proceed to an edit.
+Order clusters by how many cases they would fix. Cluster every owner in its own
+clusters, so nothing is lost on the way to the backlog, and keep the owners
+separate: `model` and `package-skill` each license a different edit, and the
+rest license none here.
 
 Say what you considered merging and chose not to. A cluster is a claim that one
 change fixes N cases, and the near-misses are what a reviewer needs to falsify
@@ -381,13 +398,20 @@ can design the edit is the whole job here; the edit itself is
 `skill:eval-improve`, working from this. A cluster carrying an honest open
 question is more useful than one carrying a remedy nobody probed.
 
-**Only `owner: model` proceeds to `eval-improve`.** Skill findings go back into
-the analysis or phrase-detection skill. Retrieval findings go to the tool.
+**`owner: model` and `owner: package-skill` proceed to `eval-improve`.** Both
+are fixable in the model repo, so both are checkpointed and rolled back by the
+same git commit. They are still separate owners, because the edit they license
+is different and the acceptance check treats them differently: a model edit
+changes what the data means, a package-skill edit changes what the agent is told.
+
+`owner: agent-skill` findings go back into the shared analysis or
+phrase-detection skill, outside this loop. Retrieval findings go to the tool.
 `BAD-REFERENCE` and `AMBIGUOUS-REFERENCE` go to the golden side door in
 `skill:eval-loop` (repair or hold). Do not send them to improve. Other dataset
 findings (a bad question, a case worth excluding) go back to the case in
-`cases.jsonl` via the conductor. Routing a skill bug into the model is
-how models accumulate scar tissue.
+`cases.jsonl` via the conductor. Routing a guidance bug into the model is how
+models accumulate scar tissue: if the entity was there, correctly defined, and
+the agent still misused it, the fix is guidance, not a new field.
 
 ## Anti-patterns
 
@@ -401,5 +425,6 @@ how models accumulate scar tissue.
 ## Related skills
 
 - `skill:eval-answer`: the score this consumes.
-- `skill:eval-improve`: smallest model edit, `owner: model` only.
+- `skill:eval-improve`: the smallest edit, for `owner: model` and
+  `owner: package-skill`.
 - `skill:eval-loop`: golden hold/repair, the acceptance check, and checkpoint.

package/skills/eval-diagnose/scripts/cluster_failures.py

@@ -76,6 +76,7 @@ WHERE_BY_OWNER = {
     "retrieval": NOT_RETURNED[2],
     "agent-skill": DELIVERED[2],
     "dataset": "dataset",
+    "package-skill": "package guidance",
 }
 # Which artifact the edit lands in. An owner nobody has named yet -- eval-
 # diagnose has not run, or it ran and declined (`undecided` for a
@@ -87,6 +88,7 @@ LEVER_BY_OWNER = {
     "retrieval": "retrieval",
     "agent-skill": "skill",
     "dataset": "dataset",
+    "package-skill": "package-skill",
 }
 
 

package/skills/eval-diagnose/scripts/diagnose.py

@@ -94,7 +94,7 @@ READ_ONLY = (*NO_EDITS, *NO_SHELL)
 CODE_IN_TABLE = re.compile(r"^\|\s*`([A-Z][A-Z_-]+)`\s*\|", re.M)
 COMPONENTS = ("dataset", "agent-call", "get_context/model",
               "get_context/retrieval", "construction", "model-definition")
-OWNERS = ("model", "retrieval", "agent-skill", "dataset")
+OWNERS = ("model", "package-skill", "retrieval", "agent-skill", "dataset")
 SUFFICIENCY = ("sufficient", "insufficient", "unknown")
 SEVERITY = ("low", "medium", "high")
 
@@ -1108,11 +1108,18 @@ def main(argv: list[str] | None = None) -> int:
     if invalid:
         print(f"{len(invalid)} broke the skill's vocabulary "
               f"(see `_invalid` in diagnoses.jsonl)")
+    # Both owners are fixable in the model repo, so both are what improve may
+    # touch. Counted separately because they license different edits and the
+    # acceptance check treats them differently.
+    fixable = ("model", "package-skill")
     actionable = sum(len(c.get("qids", []))
                      for c in clusters.get("clusters", [])
-                     if c.get("owner") == "model")
-    print(f"{actionable} cases sit behind a model-owned cluster "
-          f"(the only ones eval-improve may touch)")
+                     if c.get("owner") in fixable)
+    by_owner = ", ".join(
+        f"{o}: {sum(len(c.get('qids', [])) for c in clusters.get('clusters', []) if c.get('owner') == o)}"
+        for o in fixable)
+    print(f"{actionable} cases sit behind a cluster eval-improve may touch "
+          f"({by_owner})")
     print(f"cost ${spend:.2f}")
     return 0
 

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.38
+version: 0.1.40
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,13 +1,27 @@
 ---
 name: eval-improve
-description: 'Make the smallest safe Malloy model edit that closes a diagnosed model-owned gap, with a probe receipt for every factual claim. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.38
+description: 'Make the smallest safe edit that closes a diagnosed gap the model repo owns: a Malloy model edit, or a package skill under skills/. Every factual claim carries a probe receipt. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
+version: 0.1.40
 ---
 
 # Improve the Model
 
-Takes an issue with `owner: model` and produces **one smallest edit** that closes
-the gap. Every factual claim is backed by a query you ran.
+Takes an issue with `owner: model` or `owner: package-skill` and produces **one
+smallest edit** that closes the gap. Every factual claim is backed by a query
+you ran.
+
+The two owners license different edits and are never interchangeable:
+
+- **`model`** changes what the data means or how it is described:
+  a `.malloy` file.
+- **`package-skill`** changes what the agent is told about using this package:
+  a file under the package's `skills/`. Use it for a rule that spans entities or
+  is not a property of any one of them. A fact about a single field belongs in
+  that field's `#(doc)`, which reaches every agent whether or not it loaded a
+  guide.
+
+Both live in the model repo, so one git commit checkpoints and rolls back
+either.
 
 **Two hard boundaries:**
 
@@ -30,11 +44,18 @@ the gap. Every factual claim is backed by a query you ran.
 | Doubt only, or retrieval-only (no verdict) | Docs, labels, index only, and only where the transcript shows a concrete confusion. |
 | Silence | **No edit.** |
 
+A `package-skill` edit sits at the docs tier throughout: it is permitted
+wherever the table permits docs, and never on silence. It is the easiest artifact
+in this loop to overfit, because it is prompt text sitting next to the question
+it just failed, so the expert test in Step 1 applies to it word for word.
+
 Do not edit for `BAD-REFERENCE` or `AMBIGUOUS-REFERENCE`. Those are the
 golden side door in `skill:eval-loop`: repair or hold the golden, bump
 `goldenRevision` on the case, and open a new baseline run. Being right and unmatched
-beats encoding a defect or an unsettled key. Do not edit for a skill,
-retrieval, or dataset owner.
+beats encoding a defect or an unsettled key. Do not edit for an `agent-skill`,
+retrieval, or dataset owner: `agent-skill` means the shared skill, which is not
+in this repo, and widening it by editing one package's copy hides the gap from
+every other package.
 
 ## Step 1: What a correct answer may teach
 
@@ -66,6 +87,20 @@ A false `primary_key` compiles and silently corrupts every aggregate. Of one
 pilot's 11 accepted edits, 4 of 5 wrong ones died to a single
 `COUNT(*)` vs `COUNT(DISTINCT …)` probe that was never run.
 
+**A package-skill edit still needs probes, for a different reason.** It makes no
+claim about a join key, so there is nothing structural to verify, and the
+`probes` array can legitimately be short. But a guide that states a fact about
+the data is asserting it to every agent that reads it, with no compiler and no
+query to contradict it. Probe every number and every claim about what a measure
+counts before writing it into a guide, exactly as you would before writing it
+into a `#(doc)`. A guide that quotes a stale figure is worse than one that
+quotes none: it is confidently wrong at the moment of use.
+
+Its reload receipt is the ordinary one. Skill files are part of the package's
+content hash, so a skill-only edit moves `sourceContentSha` and an unchanged sha
+means the edit did not reach what is served. After the reload, read the guide
+back from the server and confirm it is your text.
+
 Compile-check the edit before saving (scope `file` for an edit), then reload
 the package. Confirm it is not serving a stale model.
 
@@ -98,6 +133,11 @@ displace answers that already worked.
 | 4 | Declared join on a *probed* key |
 | 5 | A new source: last resort, at most one |
 
+For a `package-skill` issue the ranking is the same shape, and rung 1 is still
+the model: if the fact fits on one entity's `#(doc)`, put it there instead. A
+guide is the right home only for what a doc cannot hold, and it reaches only the
+agents that read it.
+
 ### Four ways a doc edit backfires, each measured
 
 **Do not put an emphatic claim next to an unsettled one.** A dimension doc was

package/skills/eval-improve/scripts/improve.py

@@ -4,9 +4,10 @@
   python improve.py --run <workdir>/runs/<label> --set <set-dir> --watch-mode
   # the model directory comes from [model] repo in the set's eval.toml
 
-One agent per `owner: model` cluster, each holding `skill:eval-improve`, each
-producing at most one smallest edit with probe receipts. Appends one `candidate`
-event per cluster to the run's `events.jsonl`.
+One agent per fixable cluster (`owner: model` or `owner: package-skill`), each
+holding `skill:eval-improve`, each producing at most one smallest edit with
+probe receipts. Appends one `candidate` event per cluster to the run's
+`events.jsonl`.
 
 THIS SCRIPT CONTAINS NO MODELING DOCTRINE
 
@@ -71,13 +72,13 @@ IMPROVE_TOOLS = ("mcp__publisher__get_context",
                  "mcp__publisher__execute_query",
                  "mcp__publisher__compile_model",
                  "mcp__publisher__reload_package",
+                 "mcp__publisher__get_skill",
+                 "mcp__publisher__get_status",
                  "Read", "Edit", "Write", "Grep", "Glob",
                  "Bash(bash ./sync_and_reload.sh)")
 
-RELOAD = """curl -s -m 60 -X POST "{mcp_url}" \\
-  -H "Content-Type: application/json" \\
-  -H "Accept: application/json, text/event-stream" \\
-  -d '{{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{{"name":"reload_package","arguments":{{"environmentName":"{environment}","packageName":"{package}"}}}}}}' \\
+# REST, not MCP: an MCP reply escapes its inner JSON, so grepping it for the sha matches nothing.
+RELOAD = """curl -s -m 60 "{rest_url}/api/v0/environments/{environment}/packages/{package}?reload=true" \\
   | grep -o '"sourceContentSha":"[^"]*"' | head -1"""
 
 SYNC_SCRIPT = """#!/bin/bash
@@ -87,11 +88,31 @@ echo "reloaded; if sourceContentSha did not change, the edit did not reach the s
 """
 
 
+SYNC_HELPER = "sync_and_reload.sh"
+
+
 def git(args: list[str], cwd: pathlib.Path) -> str:
     return subprocess.run(["git", *args], cwd=cwd, capture_output=True,
                           text=True).stdout
 
 
+def candidate_diff(model_dir: pathlib.Path, extra: tuple[str, ...] = ()) -> str:
+    """The agent's edit: tracked changes plus new files, minus our own helper.
+
+    `git diff` alone omits untracked files, so a new skills/ file left an empty
+    patch that `git clean` then deleted. `--no-index` diffs each against
+    /dev/null without touching the index.
+    """
+    out = git(["diff", *extra, "--", "."], model_dir)
+    new = git(["ls-files", "--others", "--exclude-standard", "--", "."],
+              model_dir).splitlines()
+    for f in new:
+        if f != SYNC_HELPER:
+            out += git(["diff", "--no-index", *extra, "--", "/dev/null", f],
+                       model_dir)
+    return out
+
+
 # Thin on purpose: the cluster, the cases, the one operational fact the skill
 # cannot know (how to reach the server), and the output shape. Everything about
 # how to choose an edit is in the loaded skill.
@@ -111,7 +132,8 @@ MAKING YOUR EDIT REACH THE SERVER
 
 After each edit run `bash ./sync_and_reload.sh`. It prints `sourceContentSha`.
 If that value does not change, your edit did NOT reach the server and every
-probe you run afterwards is testing the old model.
+probe you run afterwards is testing the old model. This covers a package skill
+under skills/ as well as a .malloy file: both are part of the hash.
 
 One edit for the cluster's shared root cause, not one per case.
 
@@ -259,7 +281,8 @@ def improve_cluster(issue: dict[str, Any], cases: dict[str, Any],
         return {**json.loads(out.read_text()), "_cached": True}
     d.mkdir(parents=True, exist_ok=True)
 
-    reload_cmd = RELOAD.format(mcp_url=a.mcp_url, environment=a.environment,
+    reload_cmd = RELOAD.format(rest_url=a.rest_url.rstrip("/"),
+                               environment=a.environment,
                                package=a.package)
     sync = ""
     if not a.watch_mode:
@@ -268,7 +291,7 @@ def improve_cluster(issue: dict[str, Any], cases: dict[str, Any],
         if served:
             sync = (f'rsync -a --delete --exclude .git '
                     f'"{a.model_dir.resolve()}/" "{served.resolve()}/"\n')
-    helper = a.model_dir / "sync_and_reload.sh"
+    helper = a.model_dir / SYNC_HELPER
     helper.write_text(SYNC_SCRIPT.format(sync=sync, reload=reload_cmd))
     helper.chmod(0o755)
 
@@ -298,9 +321,9 @@ def improve_cluster(issue: dict[str, Any], cases: dict[str, Any],
         save_transcript=d / "improver.jsonl")
     (d / "improver.md").write_text(r.text)
 
-    diff = git(["diff", "--", "."], a.model_dir)
+    diff = candidate_diff(a.model_dir)
     (d / "edit.patch").write_text(diff)
-    stat = git(["diff", "--stat", "--", "."], a.model_dir).strip()
+    stat = candidate_diff(a.model_dir, ("--stat",)).strip()
 
     # While the edit is still in the tree: the mechanical half of Step 4. Only
     # catches drift and rubric sentences that contradict the model -- a golden
@@ -347,6 +370,10 @@ def main(argv: list[str] | None = None) -> int:
     ap.add_argument("--environment", default=None)
     ap.add_argument("--package", default=None)
     ap.add_argument("--mcp-url", default=None)
+    ap.add_argument("--rest-url", default=None,
+                    help="REST base; the reload receipt is read from here, "
+                         "because the MCP reply double-encodes its JSON. "
+                         "Default: the [model] server in the set's eval.toml")
     # No built-in port: a fixed default once named the server holding the
     # model under test, which cannot verify its own goldens. The fallback is
     # the [truth] section of the set's eval.toml. With neither, the value
@@ -406,6 +433,7 @@ def main(argv: list[str] | None = None) -> int:
                              "--environment")
     a.package = cfg.need(a.package, "model", "package", "--package")
     a.mcp_url = a.mcp_url or cfg.model_mcp_url()
+    a.rest_url = a.rest_url or cfg.model_publisher()
     a.truth_publisher = a.truth_publisher or cfg.truth_publisher()
     a.truth_environment = a.truth_environment or cfg.get("truth", "environment")
     a.skills_root = a.skills_root or cfg.get("paths", "skills_root")
@@ -439,7 +467,7 @@ def main(argv: list[str] | None = None) -> int:
         if e.get("kind") == "issue_status":
             status[e["issue_id"]] = e["status"]
     issues = [e for e in events if e.get("kind") == "issue"
-              and e.get("owner") == "model"
+              and e.get("owner") in ("model", "package-skill")
               and status.get(e["issue_id"]) == "open"]
     if a.only:
         want = {x.strip() for x in a.only.split(",")}
@@ -450,13 +478,14 @@ def main(argv: list[str] | None = None) -> int:
     if a.limit is not None:
         issues = issues[:a.limit]
     if not issues:
-        print("no open model-owned issues; run diagnose.py first, or every "
-              "cluster is owned by retrieval, agent-skill, or dataset")
+        print("no open model- or package-skill-owned issues; run diagnose.py "
+              "first, or every cluster is owned by retrieval, agent-skill, "
+              "or dataset")
         return 0
 
     art = a.run / "artifacts"
     art.mkdir(parents=True, exist_ok=True)
-    print(f"{len(issues)} model-owned clusters, agent {a.model}, "
+    print(f"{len(issues)} model/package-skill clusters, agent {a.model}, "
           f"{'isolated' if a.isolate else 'cumulative'}, "
           f"{'watch mode' if a.watch_mode else 'snapshot sync'}")
 

package/skills/eval-improve/scripts/improve_test.py

@@ -247,5 +247,40 @@ class ClusterMembers(unittest.TestCase):
               "golden": [{"region": "West", "n": 3}],
               "rubric": "West has 3."}])
 
+class CandidateDiff(unittest.TestCase):
+    """A new file is part of the edit, and our own helper is not."""
+
+    def setUp(self):
+        self.repo = pathlib.Path(tempfile.mkdtemp())
+        self.addCleanup(shutil.rmtree, self.repo, True)
+        for cmd in (["init", "-q"], ["config", "user.email", "t@t"],
+                    ["config", "user.name", "t"]):
+            subprocess.run(["git", *cmd], cwd=self.repo, check=True)
+        (self.repo / "m.malloy").write_text("source: a is x\n")
+        subprocess.run(["git", "add", "."], cwd=self.repo, check=True)
+        subprocess.run(["git", "commit", "-qm", "init"], cwd=self.repo, check=True)
+
+    def test_a_new_skills_file_is_in_the_patch_and_the_helper_is_not(self):
+        (self.repo / "skills" / "s").mkdir(parents=True)
+        (self.repo / "skills" / "s" / "SKILL.md").write_text("NEW GUIDE\n")
+        (self.repo / improve.SYNC_HELPER).write_text("#!/bin/bash\n")
+        (self.repo / "m.malloy").write_text("source: a is y\n")
+        diff = improve.candidate_diff(self.repo)
+        self.assertIn("NEW GUIDE", diff)
+        self.assertIn("source: a is y", diff)
+        self.assertNotIn(improve.SYNC_HELPER, diff)
+        self.assertIn("SKILL.md", improve.candidate_diff(self.repo, ("--stat",)))
+
+    def test_only_the_helper_is_an_empty_diff(self):
+        (self.repo / improve.SYNC_HELPER).write_text("#!/bin/bash\n")
+        self.assertEqual(improve.candidate_diff(self.repo).strip(), "")
+
+
+class Grants(unittest.TestCase):
+    def test_the_tools_the_skill_names_are_granted(self):
+        for t in ("get_skill", "get_status"):
+            self.assertIn(f"mcp__publisher__{t}", improve.IMPROVE_TOOLS)
+
+
 if __name__ == "__main__":
     unittest.main()

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.38
+version: 0.1.40
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files: the set in the model package''s evals/ directory, runs in the set''s workdir; checkpoints are git commits of the model repo. `eval.py` runs each step from the set''s eval.toml. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.38
+version: 0.1.40
 ---
 
 # The Evaluation Loop
@@ -523,7 +523,8 @@ conductor. Do not:
 
 ## Prime directives
 
-- The model is the only thing improve edits. No question text, qids, or
+- Improve edits the model repo and nothing else: a `.malloy` file, or a package
+  skill under the package's `skills/`. No question text, qids, or
   expected values in any name, doc, or comment.
 - **You are measuring a model, not reviewing this harness.** Read a script when
   a number you have to report cannot be explained otherwise, and stop there. Do

package/skills/eval-loop/reference/acceptance-check.md

@@ -50,6 +50,18 @@ Acceptance rules (replacing any vague "results improve"):
 - Independent deterministic justification (a probed-wrong definition
   corrected) may accept without a measured win. Record that as the acceptance check
   `reason`.
+- **Package-skill edits (`class: skill`) get the full rules, never the
+  documentation shortcut.** A `get_context` probe proves an entity is
+  discoverable. Nothing deterministic proves an agent read a guide and behaved
+  differently, so the only evidence a skill edit has is a fresh blind
+  re-answer: both splits, twice, clearing the flip-count bar. Treat the
+  documentation shortcut as unavailable here even when the edit reads like
+  documentation, because a guide is prompt text written next to the case it
+  failed and is the easiest artifact in this loop to overfit.
+
+  Read the diff before measuring it, too. A guide that names the case, quotes
+  its expected number, or encodes the shape of one question fails the expert
+  test in `skill:eval-improve` and is rejected without a run.
 
 Write the `acceptance_check` event (decision, class, baseline and final run ids,
 regressions, holdout delta, reason) BEFORE any commit, so a rejected