@malloy-publisher/skills 0.1.50 → 0.1.52

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.50",
+   "version": "0.1.52",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # Evaluate One Answer

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # Diagnose One Answer

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-improve
 description: 'Make the smallest safe edit that closes a diagnosed gap the model repo owns: a Malloy model edit, or a package skill under skills/. Every factual claim carries a probe receipt. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # Improve the Model

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files: the set in the model package''s evals/ directory, runs in the set''s workdir; checkpoints are git commits of the model repo. `eval.py` runs each step from the set''s eval.toml. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # The Evaluation Loop

package/skills/eval-report/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-report
 description: 'Write the summary a person reads after an evaluation run: what ran, what it scored, which failures were the MODEL and which were the EVAL itself (out of turns, contaminated, unestablished golden, unreadable judge), what was learned, and what to do next. Every claim links to the artifact behind it. Use at the end of any run, before quoting a number to anyone. Never scores an answer (eval-answer) or decides who owns a failure (eval-diagnose).'
-version: 0.1.50
+version: 0.1.52
 ---
 
 # Report a Run

package/skills/malloy-analysis/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-analysis
 description: "Read before your first get_context call on any business or data question. The required workflow, not background: which source, which rows, how to verify. Skipping it causes confident wrong answers."
-version: 0.1.50
+version: 0.1.52
 ---
 
 <!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->
@@ -83,6 +83,7 @@ Your first result is a draft, not an answer. The difference between a useful ana
 - **Ask "what would make this wrong?"** then run the query that would expose that problem. A plausible-looking wrong answer is the most dangerous kind.
 - **Check the common failure modes:**
   - Fan-out / double-counting: if you joined across grain, compare `count()` with `count(base_key)`, the base source's own key (in Malloy `count(field)` is already the distinct count). A large gap means base rows are repeated and aggregates are inflated. The check reads one way only: `count(joined.field)` is a distinct count at the joined grain, so it is normally much SMALLER than `count()`, which is not missing data. Measure coverage at the base grain: `count() { where: joined.field is null }`.
+  - Grouping by an array: a field with `relationship: "many"` that is not reached through a join (`lines.sku`, `tags.each`) gives one group per distinct element value, and a row lands in every group its array holds, so its groups add up to more than the total, as across a one-to-many join. Do not print a total row across it, and count elements with `tags.count()`, not `count(tags.each)`, which is the distinct count.
   - Broken filters: a quick count confirms a filter narrowed the data as expected. Watch case, spelling and date-format mismatches; a filter that matches nothing still returns a result, just the wrong one. Confirm which dimension the user's word ("brand") maps to when several fields look alike, and that the field means what you assume (gross against net revenue): read its doc. Before a `pick` or `where` that groups values into categories, list the field's distinct values, since an unlisted one falls into `else` unnoticed. Apply the filter the question implies ("last quarter" with no time filter returns all-time data). A view can have a `where:` built in (`recent_orders`): read its code, and query the base source with explicit filters if it conflicts with the ask.
   - Null-driven loss: `count() { where: the_field is null }` shows how many rows a key field drops (`count(the_field)` counts distinct values, so subtracting it from `count()` is not a null count).
   - Parts that do not sum to the whole: if you split a total into categories, confirm they add up. If a doc says the parts are not additive (one event logged on two layers, a non-additive measure), do not print a total row across that dimension, least of all one nobody asked for; give the per-category figures and say why.

package/skills/malloy-analysis-report/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-analysis-report
 description: Combine validated Malloy queries into a report (a linear report, read top to bottom). Use when the user asks to "create a report", "combine these into a report", or wants a persistent multi-query artifact.
-version: 0.1.50
+version: 0.1.52
 ---
 
 <!--

package/skills/malloy-charts/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-charts
 description: Read before choosing a chart or adding a chart annotation. Chart types, tag syntax and scale rules, KPI cards, dashboards, sparklines, and the renderer mistakes that fail silently.
-version: 0.1.50
+version: 0.1.52
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-classify/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-classify
 description: 'Propose #(classification) annotations for a Malloy model from an access policy''s class scheme, backed by aggregate value evidence from the served data, and show them as a diff before writing anything. Use when the user asks to "classify the model", "tag PHI/PII", "label sensitive fields", or adds fields to a package that a policy governs, and before running malloy-compliance-review on a model nobody has classified.'
-version: 0.1.50
+version: 0.1.52
 ---
 
 <!--

package/skills/malloy-compliance-review/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-compliance-review
 description: 'Check a Malloy package and its controls against a written access policy: turn the policy''s prose into a control register, run compliance.py check against a served package (structure, access as each role, and the values themselves), judge what needs a person, and produce a report and a signable attestation. Use when the user asks whether a model "complies with", "satisfies" or "meets" a policy, HIPAA, SOC 2, PCI or the like, asks for a compliance review or attestation, or wants the review in CI.'
-version: 0.1.50
+version: 0.1.52
 ---
 
 <!--
@@ -22,12 +22,13 @@ The review has two halves, and they are kept apart on purpose:
 - **The register** (`policies/controls.json`) is the policy as checks. You draft it from the prose;
   the policy owner approves it. It changes when the policy changes.
 - **The check** (`scripts/compliance.py check`) runs the register against the package as it is
-  served. It changes nothing, reads only aggregates, and exits 1 when a control fails, so the same
+  served. It changes nothing, reads only aggregates, and exits 1 when a control fails (3 when nothing failed but something could not be checked), so the same
   command gates every change in CI.
 
-What Publisher already refuses at load (a domain exposing a class its `#(carries)` omits, a gate
-on a given the caller can set, a join that reaches a gated source around its gate) the check
-reports as the control it breaks. It does not re-derive them.
+What Publisher refuses at load (a domain exposing a class its `#(carries)` omits) the check reports
+as the control it breaks. A join that reaches a gated source around its gate is only a `join-bypass`
+warning on the server, in every package, so the check fails it: this review is the hard stop. It
+does not re-derive either.
 
 ## Before you start
 
@@ -35,6 +36,8 @@ reports as the control it breaks. It does not re-derive them.
   review against a server with **no gateway in front**: each persona's givens are sent with the
   request and honored, which is how the check asks "what would this role see". Never point it at a
   production endpoint, where a trusted tier sets those givens and the probe would be meaningless.
+  A pass certifies the package's controls. In production they hold only behind the gateway that
+  authenticates users, sets the givens and blocks raw SQL, so a report says that alongside the verdict.
 - The policy document.
 - A classified model. If fields are unclassified, `skill:malloy-classify` first.
 
@@ -75,8 +78,14 @@ the MCP endpoint (`--mcp`, default `http://127.0.0.1:4040/mcp`) and `GET /api/v0
 every published domain as every persona with a count, queries every source the package does not
 publish (read from the `.malloy` files in the current directory, or `--package-dir`) to confirm it
 refuses, and profiles every published dimension with aggregates (seconds to a minute for 140
-fields). It writes `review/report.json` and `review/report.md`. Exit 0 means no
-control fails.
+fields). It writes `review/report.json` and `review/report.md`. Against a server that is not on
+localhost, `--mcp` is required (a usage error otherwise).
+
+Exit codes: **0** every control holds, **1** a control fails, **2** usage (a bad flag, or a register that is missing, not JSON or has no `controls`),
+**3** nothing failed but something could not be checked, **4** an internal error (traceback on
+stderr). A probe the server neither answered nor refused (a 400, a 401, a 404 that is not the query
+boundary's refusal, or that names the `--model` itself as missing) is never a pass: it needs review, so a typo cannot make a control hold.
+`--allow-review` turns 3 into 0 for a pipeline that triages review items by hand.
 
 ### 3. Judge what needs a person
 
@@ -86,9 +95,10 @@ Read `review/report.md`. `reference/report.md` explains each section. For each c
 - **FAILS**: name the domain and field from the finding, the requirement it breaks, and the
   smallest change to the model that would fix it. A failure the server refused at load (the report
   says the package is serving a stale model) is already contained: the change was not deployed.
-  Say both.
-- **Needs review**: a value check that could not decide (free text on a de-identified domain, a
-  probe that errored). Look at the evidence and say what you concluded and why; if it needs the
+  Say both. A `join-bypass` failure is not contained: the server serves that package with a warning.
+- **Needs review**: a check that could not decide (free text on a de-identified domain, a probe
+  the server neither answered nor refused, a cap on a field that was not profiled, a field whose
+  declared class is lower than what its expression reads). Look at the evidence and say what you concluded and why; if it needs the
   policy owner, say so.
 - **Not provable here**: say who can prove it.
 
@@ -109,8 +119,21 @@ and rerun.
 ### 5. In CI
 
 Run the same `check` on every change to a `.malloy` file, against a Publisher started on the
-change's checkout, and fail the build on exit 1. `docs/compliance-review.md` has a GitHub Actions
-example. Upload `review/` as the build's artifact so the evidence travels with the change.
+change's checkout, and fail the build on any non-zero exit: 1 and 4 are never acceptable, and 3
+means a person has to look. `docs/compliance-review.md` has a GitHub Actions example. Upload
+`review/` as the build's artifact so the evidence travels with the change.
+
+```yaml
+      - name: Review
+        run: |
+          python3 <skills>/malloy-compliance-review/scripts/compliance.py check \
+            --environment ci --package <pkg> --register policies/controls.json --out-dir review
+          # add --allow-review to pass on exit 3 (needs review, none failing)
+
+      - if: always()
+        uses: actions/upload-artifact@v4
+        with: { name: compliance-review, path: review/ }
+```
 
 ## Rules
 

package/skills/malloy-compliance-review/reference/checks.md

@@ -13,14 +13,14 @@ status. The rest are aggregate queries run as the register's personas. None read
 | Check | Fails when | Reads |
 | --- | --- | --- |
 | `inventory` | The classification summary lists no source, or `get_context` could not list the published fields | classification summary, `get_context` |
-| `loads` | The package did not load, or serves a stale model because its latest files were refused. One finding per refusal, tagged with the domain and kind (`ceiling`, `secure`, `join`) | `/api/v0/status` |
+| `loads` | The package did not load, serves a stale model because its latest files were refused, or loads with a `join-bypass` warning (a join that reaches a stricter gate than the entry's). One finding per refusal or warning, tagged with the domain and kind (`ceiling`, `join`); a warning is always kind `join` | `/api/v0/status`, the package read's `warnings` |
 | `refusals` | Never fails on a refusal: a refused change is evidence that a violating change is not deployed | `/api/v0/status` |
-| `coverage` | A published field has no class | classification summary |
+| `coverage` | A published field has no class. A field that declares a lower class than the fields it reads (a `classification-downgrade` warning) needs review | classification summary |
 | `gated` | A published domain has neither `#(authorize)` nor a row filter | model read |
 | `secure_gates` | A gate reads a given that is not `#(secure)` | model read |
-| `locked_bases` | A source the package does not publish answers a count to an identity holding every role the register names. Its sources are read from the package's `.malloy` files and queried through each file's own path; with no files found it needs review | package files, a count per source |
+| `locked_bases` | A source the package does not publish answers a count to an identity holding every role the register names. Its sources are read from the package's `.malloy` files and queried through each file's own path; with no files found, or a probe the server neither answered nor refused, it needs review | package files, a count per source |
 | `ceiling` | A domain declares no `#(carries)`, declares a class its `must_not_carry` forbids, or serves a class beyond it | classification summary, register |
-| `access` | A persona's outcome on a domain differs from the register, or the register names a domain the package does not publish | a count per domain per persona |
+| `access` | A persona's outcome on a domain differs from the register, a gated domain answers a caller with no givens (a domain gated only by a row filter may answer with no rows), or the register names a domain the package does not publish. A probe the server neither answered nor refused needs review | a count per domain per persona, and per gated domain with no givens |
 | `values` | A field's values prove more than its class allows, a de-identified domain serves day dates, five-digit ZIPs or uncapped values. Free text there needs review | `get_context`, an aggregate profile per field |
 
 ## What the values prove
@@ -43,8 +43,17 @@ A floor is a lower bound. A birth date's values show dates to the day, a quasi-i
 policy's class table makes it direct. The check fails a field declared *below* its floor, never one
 declared above.
 
+A probe is **refused** when the server answers 403, or 404 with the query boundary's own refusal
+(`Query target is not queryable.` or `No queryable ...`). Any other non-2xx (a 400 from a typo, a 401,
+a 404 for a domain's model path that is not served) means the probe never reached the gate, so it needs
+review rather than counting as a refusal. `locked_bases` probes models the package keeps off its surface,
+which the server answers like a missing one, so it runs only when `--package-dir` holds a `publisher.json`,
+and is review otherwise.
+
 ## Scoping
 
 A check runs once per review, however many controls name it. A control that names `values:research`
-takes only the findings about `research`; if every finding is about another domain, the control
+takes only the findings about `research`; a scope on `coverage`, `gated`, `secure_gates`, `ceiling` or `values` that names no published domain, or on any other
+check that names no domain, declared source or refusal kind, fails as an unknown scope, and `locked_bases` scopes are checked against the sources the
+package's files declare; if every finding is about another domain, the control
 holds for that check and the report says nothing it found touches `research`.

package/skills/malloy-compliance-review/reference/register.md

@@ -30,7 +30,7 @@ is a complete one for a HIPAA minimum-necessary policy.
 | --- | --- | --- |
 | `rows` | Admitted, with every row | The count probe answers, and the domain has no row filter |
 | `scoped` | Admitted, with only the rows its givens select | The count probe answers, and the model read shows a row filter in force |
-| `refused` | Not admitted | The count probe answers 403 |
+| `refused` | Not admitted | The count probe answers 403, or 404 with the query boundary's refusal |
 
 `scoped` is read from the model's row filter, never from a smaller count: a persona may be the only
 one admitted to a domain and still be scoped.
@@ -41,14 +41,15 @@ one admitted to a domain and still be scoped.
 "research": {
   "deidentified": true,
   "must_not_carry": ["phi.direct_identifier"],
-  "caps": [{ "field": "patient.age", "max": 90 }]
+  "caps": [{ "field": "patient.age", "max": 90 }, { "field": "patient.birth_year", "min": 1930 }]
 }
 ```
 
 - `deidentified`: the domain may serve no day-level date, no five-digit ZIP, and nothing that reads
   as free text.
 - `must_not_carry`: classes the domain's `#(carries)` may not list.
-- `caps`: a numeric field's maximum, as the values show it.
+- `caps`: a numeric field's `max`, `min` or both, as the values show it. A capped field that was not
+  profiled, or whose numeric query errored, needs review.
 
 ### Controls
 
@@ -59,7 +60,7 @@ one admitted to a domain and still be scoped.
 
 - `id` and `text` are the policy's, verbatim.
 - `checks` name what proves the control. A check scoped with `:domain` takes only that domain's
-  findings; `loads:` also takes a refusal kind (`ceiling`, `secure`, `join`).
+  findings; `loads:` also takes a refusal kind (`ceiling`, `join`).
 - A control nothing in the model can prove takes `"status": "not_provable_here"` and a `why`.
 - `why` on any control is printed in the report beside it.
 

package/skills/malloy-compliance-review/reference/report.md

@@ -17,9 +17,19 @@ SPDX-License-Identifier: MIT
 | --- | --- |
 | Holds | Every check the control names passed |
 | FAILS | At least one finding failed. The command exits 1 |
-| Needs review | Nothing failed, but a finding could not decide on its own |
+| Needs review | Nothing failed, but a finding could not decide on its own. With no failure the command exits 3, or 0 under `--allow-review` |
 | Not provable here | The register says the model cannot prove it, and why |
 
+## Exit codes
+
+| Code | Means |
+| --- | --- |
+| 0 | Every control holds or is not provable here |
+| 1 | At least one control fails |
+| 2 | Usage: a bad flag, `--mcp` missing for a server that is not localhost, or a register that cannot be run |
+| 3 | Nothing failed, but at least one control needs review |
+| 4 | Internal error; the traceback is on stderr |
+
 ## A stale model
 
 When the latest files did not load, Publisher keeps serving the model it compiled before, and the
@@ -32,7 +42,8 @@ deployed. The fix is to the files on disk; the served model is still the last go
 On a fresh server, as in CI, a change Publisher refuses at load leaves no earlier model to serve, so
 the package is absent. The report says so at the top. The controls the refusal names fail; every
 other control needs review, marked "not checked", because nothing could be checked. A package the
-server does not know at all fails `loads`: check the environment and package names.
+server does not know at all, with no load error naming it, is review too (exit 3): nothing was
+checked, so check the environment and package names.
 
 ## Evidence