@malloy-publisher/skills 0.1.0 → 0.1.1

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.0",
+   "version": "0.1.1",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/malloy-getting-started/SKILL.md

@@ -13,6 +13,8 @@ At minimum you need `malloy_getContext`, `malloy_executeQuery`, and `malloy_sear
 
 If none of the tools are there, either the server is not running or your client connected before it was. Start the server (`npx @malloy-publisher/server --port 4000`, or `bun run build && bun run start` from a clone) and wait until `curl -s http://localhost:4000/api/v0/status` reports `operationalState: serving`.
 
+If there is no Publisher workspace here at all, and the user wants to work with data of their own rather than the bundled examples, `npm create @malloy-publisher/malloy-package <name>` scaffolds one: the package and a starter model, registered so the server actually serves it, plus the start script, the MCP config and these skills. Run bare, it comes with a small sample dataset, so there is something to query straight away. In a fresh directory `npm start` then runs the pinned server against the package in watch mode; if the directory already had a `package.json` the scaffolder leaves it alone and adds no script, printing the equivalent `npx` command to use instead. Where you run it matters: only the package lands in `<name>/`, and the workspace files, the agent instructions and the MCP config among them, are written to the current directory. Run it here if this directory is empty or is meant to become the workspace. If it already holds other work, scaffold into a new directory instead (`mkdir my-data && cd my-data`), because agent config is discovered by walking up, so writing those files here changes what every session beneath this directory inherits. Seed the starter model from a local file with `npm create @malloy-publisher/malloy-package <name> -- --data <path/to/their-file.csv>` (CSV, Parquet, or Excel `.xlsx`), keeping the `--`, which is how `npm create` passes options through. That path is relative to wherever you run the command, so if you scaffolded into a new directory it has to reach back out to their file; the scaffolder copies it into the package and leaves the original alone. A seeded package starts smaller than the sample one, since the scaffolder does not read their columns: expect a row count and an overview, and build the model from there. A package is just Malloy, so it can instead query a database connection the config defines. Because it writes a `.mcp.json` that did not exist when the client connected, the user has to restart or reconnect once before these tools appear, and their client will ask them to approve the new project-scoped server the first time. That only works when the workspace is at the session's own root, so if you scaffolded into a new directory below that root, the user has to open a session there instead: a `.mcp.json` further down is never discovered.
+
 If you started the server yourself in this session, the tools still will not appear: your tool list was fixed when you connected, and you cannot reconnect yourself. Tell the user the tools are missing for that reason and ask them to run `/mcp`, select `malloy`, and choose Reconnect. The panel offers `Authenticate` first and reports `Auth: not authenticated`; that is a red herring, the endpoint has no auth. Restarting Claude Code also works. Continue once the tools are there.
 
 When a user is present, do not route around it by calling the REST API with curl. It appears to work, so the user never learns their session is missing the tools, and you lose what they are for: grounded discovery instead of guessed names, `malloy_compile` instead of throwaway queries, and `malloy_reloadPackage` instead of a restart. Say the tools are missing and let the user fix it in five seconds. Running unattended, with nobody who can reconnect you, is different: there the REST API is the supported interface, not a workaround. Discovery, query, compile, and reload all have REST equivalents (`malloy_searchDocs` and `malloy_getContext`'s plain-English ranking do not; read the bundled skills for syntax and ground from model metadata instead); the running server serves the full spec at `http://localhost:4000/api-doc.yaml`, and AGENTS.md carries the endpoint map.

package/skills/malloy-gotchas-modeling/SKILL.md

@@ -238,6 +238,27 @@ source: facts is conn.table('orders') -> { group_by: user_id, aggregate: total i
 
 If a project's standards file specifies a stricter policy (e.g., a `search_malloy_docs` rationale comment requirement above every `conn.sql()` block), defer to that.
 
+## Excel Files: Read `.xlsx` In Place, Never Convert
+
+```malloy
+// RIGHT: .xlsx works like .csv/.parquet (in a Publisher package the sandbox connection is `duckdb`)
+source: budget is duckdb.table('data/budget.xlsx')
+// RIGHT: sheet selection or any read option needs read_xlsx in a SQL source
+source: q2 is duckdb.sql("""SELECT * FROM read_xlsx('data/budget.xlsx', sheet = 'Q2')""")
+// WRONG: converting the spreadsheet to Parquet or CSV first (an unnecessary extra step)
+```
+
+Do not convert spreadsheets before modeling. DuckDB's excel extension reads `.xlsx` directly and loads automatically on first use, so `duckdb.table('file.xlsx')` just works. `table()` takes a plain file path only, so anything needing `read_xlsx` options (`sheet`, `range`, `header`, `ignore_errors`, `normalize_names`, `all_varchar`, `empty_as_varchar`) goes through the SQL-source form.
+
+Quirks:
+
+- Only the FIRST sheet is read by default. Select another with `sheet = 'Name'`.
+- Every numeric column arrives as `number` (xlsx stores all numbers as doubles); there are no integer columns.
+- A column mixing numbers and text fails at query time (`Could not convert string ... to DOUBLE`). Add `ignore_errors = true` (bad cells become null) or `all_varchar = true`.
+- A sheet with no header row whose first row is all text silently loses that row to header detection. Pass `header = false`.
+- Headers with spaces are kept verbatim: backtick them in Malloy, or pass `normalize_names = true` for snake_case names.
+- `all_varchar = true` returns date cells as Excel serial strings (for example `'46027.0'`), not dates.
+
 ## Duplicate Rows: Check Before Building Measures
 
 ```malloy

package/skills/malloy-materialization/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: malloy-materialization
+description: Add and debug Malloy Persistence materializations in a package - persist an expensive source so queries read a pre-built table. Read this whenever the user wants to materialize a source, add a persist annotation, speed up a slow source, or asks why a persist source isn't building.
+---
+
+# Materialization (Malloy Persistence)
+
+Materialize an expensive source once so queries read a **pre-built warehouse table** instead of recomputing it every time. You tag a source `#@ persist`, a materialization run builds it into a physical table, and queries against it are rewritten to read that table.
+
+> **The #1 gotcha, up front:** if a persist source isn't materializing, it is almost always one of two things - a `.malloy` file in the package missing the `##! experimental.persistence` flag (which aborts the *whole* package's build plan), or no build ever ran (a standalone Publisher does not build on publish - see **Building and refreshing**). Jump to **Debugging a no-op build**.
+
+## The recipe (get this right and it just works)
+
+1. **`##! experimental.persistence` on EVERY `.malloy` file in the package** - not only the file that declares the persist source. Either form enables it:
+   - `##! experimental.persistence`, or
+   - `##! experimental { access_modifiers, sql_functions, persistence }` (add `persistence` to the existing list).
+
+   **Why every file:** the build plan is computed by asking *every* `.malloy` file in the package for its persist sources, and that call **throws on any file whose model lacks the flag** (`Model must have ##! experimental.persistence`). One unflagged helper or import file, even one with no persist source of its own, aborts the whole package's build plan, so *every* persist source in the package drops out. This is the most common cause of a no-op build.
+
+2. **`#@ persist name="..."` on a query-based source, with the name quoted:**
+   ```malloy
+   #@ persist name="my_dataset.my_table"
+   source: my_rollup is some_source -> { group_by: ...; aggregate: ... }
+   ```
+   - **Only `query_source` and `sql_select` sources are persistable** - a source whose definition has a `-> { ... }` pipeline or a `conn.sql("...")`. This **includes** one refined by a trailing `extend { ... }`. What is **not** persistable is a *plain* `extend` over a bare `conn.table(...)`; a `#@ persist` on such a source is **silently ignored** (its annotation is never read) - that one source just won't materialize, and the rest of the package still builds.
+   - **Quote the name.** `name="my_table"` (or a path `name="dataset.table"` / `name="project.dataset.table"`) is required. A **bare** `name=my_table` **always fails the build/publish** with `persist annotation name must be quoted` (a raw-source scan that hard-stops); it never silently no-ops.
+   - `name=` is the target table name. In a standalone Publisher this **is** the physical table (rebuilt in place); a hosted (control-plane) deployment builds it under a content-addressed generation name. In both, the source's identity for reuse is a content address of its connection and canonical SQL (its `sourceEntityId`), so **republishing unchanged persist logic reuses the existing table** and changing the logic builds fresh.
+
+3. **Package persistence policy in `publisher.json`** (all optional):
+   ```jsonc
+   {
+     "name": "my-package",
+     "scope": "package",  // default; "version" = each published version owns its own tables
+     "materialization": { "freshness": { "window": "24h", "fallback": "live" } }
+   }
+   ```
+   Enforced at publish (strict), on edits (strict), at load (warn, still serves), and by the scheduler (an offending package is skipped):
+   - **`scope`**: `package` (default; artifacts reused across published versions) or `version` (each artifact owned by one version). Package-level only; there is no per-source scope.
+   - **`materialization.freshness`** (`window` + `fallback` of `live`/`stale_ok`/`fail`) is the objective a **hosted control plane** enforces by refreshing the table to meet it (`fallback: "live"` serves live compute while stale/absent). A **standalone** Publisher does **not** act on `freshness` for refresh - see **Building and refreshing**.
+   - **`materialization.schedule`** is a 5-field UTC cron (`min hour dom mon dow`; `L`/`W`/`#`/`?` rejected). It **requires `scope: "version"`** and is **mutually exclusive with `freshness`**. This is how a standalone Publisher refreshes on a cadence.
+
+4. **Reads vs writes.** The persist source can *read* any dataset the connection can read; the persist *target* (`name=`'s dataset) must be a dataset the connection can **write** (typically a scratch dataset).
+
+## Building and refreshing (standalone vs. hosted)
+
+A `#@ persist` tag declares *what* to materialize; it does not by itself build anything.
+
+- **Standalone Publisher:** publishing or loading a package only computes its build plan - **no table is built until a materialization run executes.** Trigger one explicitly (`malloy-pub materialize --package <pkg> --wait`, or the materialization API), or turn on the opt-in local scheduler (off unless `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER` is set) to fire the package's `schedule` cron. Refresh is a re-run or that cron; `freshness` is not a refresh trigger here, so a freshness-only standalone package builds once and is not auto-refreshed.
+- **Hosted (control-plane) deployment:** the build runs automatically on publish, best-effort - a build failure does **not** fail the publish (which is why a broken persist can look like a silent no-op), and the control plane drives refresh to meet the `freshness` objective.
+
+Either way, a successful publish alone does not prove a table exists - confirm the build separately.
+
+## Serve-time routing is `query_source`-only (today)
+
+Both persistable types *build* a table, but only a **`query_source`** (a `-> { ... }` pipeline) is rewritten to *read* it at query time. A raw **`sql_select`** (`conn.sql("...")`, including `conn.sql("...") extend { ... }`) builds its table and then the query path re-inlines its SQL, so the table is built and never read, and queries are no faster. If you have raw SQL you want served from a table, wrap it in a thin `query_source` and persist that:
+
+```malloy
+source: x_raw is my_conn.sql("select ...")
+#@ persist name="scratch_dataset.x"
+source: x is x_raw -> { select: * }
+```
+
+## Confirming it worked
+
+After a build runs, re-run one of the source's queries - a persisted `query_source` should return quickly, reading the pre-built table instead of recomputing the upstream. Your host also reports each persisted source as **ready** with its physical table name (a materialization run detail, CLI listing, or materialization view, depending on the host); if nothing is listed, either no build ran (standalone) or the build plan was empty - see **Debugging a no-op build**.
+
+## Debugging a no-op build
+
+Symptom: no table was built and the source still recomputes on every query. Check, in order:
+
+0. **Did a build actually run?** On a standalone Publisher, publish/load does **not** build - run `malloy-pub materialize` (or enable the scheduler). "Publishes fine, no table" is the *expected* standalone state, not a model bug. On a hosted deployment the build is automatic but best-effort, so a failure is silent - look for a `FAILED` run.
+1. **A `.malloy` file missing the persistence flag** (the most common real bug). Every model file's `##!` line needs `persistence`, including pure helper/import files with no persist source - one unflagged file aborts the whole package's build plan.
+2. **An unquoted persist name** - a bare `name=foo` **always** hard-stops the build/publish with `persist annotation name must be quoted`; use `name="foo"`. (If you got *no* error at all, it isn't this.)
+3. **A `#@ persist` on a non-persistable source** - a bare `extend` over `conn.table(...)` is silently ignored, so *that* source won't materialize (the rest of the package is unaffected). Tag a `query_source` / `sql_select` instead.
+4. **A persisted raw `sql_select` that builds but is never read** - if the table exists yet queries are no faster, it's the serve-routing gap above; wrap the `sql_select` in a `query_source`.
+
+**Isolation test** - add a trivial, self-contained persist source in its own file and rebuild:
+```malloy
+##! experimental.persistence
+source: smoke_raw is my_conn.table('some_dataset.some_table')
+#@ persist name="scratch_dataset.persist_smoke_test"
+source: persist_smoke is smoke_raw -> { aggregate: n is count() }
+```
+- If **even this** doesn't build (after a real materialization run), the whole package's plan is aborting - a sibling `.malloy` file is missing the flag. Fix rule 1 across the package.
+- If the smoke source **does** build but your real one doesn't, your real source is the problem - a non-persistable type (a bare `extend`), or its own file's flag.
+
+Delete the smoke file and drop its table afterward.
+
+## Gotchas
+
+- **Every `.malloy` file needs the persistence flag** - one unflagged file aborts the whole package's build plan. (A `#@ persist` on a *non*-persistable source, by contrast, is silently ignored and does not affect other sources.)
+- **A tag doesn't build** - a standalone Publisher materializes only on an explicit run or its scheduler; only a hosted control plane builds on publish.
+- **Serve-time routing is `query_source`-only** - a raw `sql_select` builds a table the query path doesn't read; wrap it in a `query_source`.
+- **Quote the name** - a bare `name=` always hard-stops the build.
+- **Republishing unchanged persist logic reuses the table** - reuse is keyed on the content-addressed `sourceEntityId`, not the `name=`.
+- **Removing a persist source (or a smoke test) does not drop its table** - physical-table cleanup is the caller's responsibility; drop it yourself.

package/skills/malloy-model/SKILL.md

@@ -182,11 +182,36 @@ source: customer_health is customers extend {
 - When both a combined table (all types) and filtered/split tables exist, prefer the split tables
 - **DRY: define measures/dimensions in base source files, not inline in views**
 
-## Parameterizable Filters with `#(filter)`
+## Parameterizing sources with `given:` (preferred)
 
-`#(filter)` annotations declare filterable dimensions on a source. Publisher parses them, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. This gives you a clean way to expose tunable knobs (date range, region, manufacturer) without hand-rolling parameterization in every query.
+Native Malloy **`given:` parameters** are the going-forward way to expose tunable knobs (date range, region, manufacturer) on a source - prefer them over `#(filter)` when you author a new model. A `given:` is a first-class runtime parameter you reference in the model's own logic; callers supply values at query time and the model uses them however it declares. Enable them with `##! experimental.givens` at the top of the model.
 
-Filters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an analysis needs a knob the source doesn't expose, the right move is to add a `#(filter)` to the source itself, not to wedge filtering into the consumer.
+```malloy
+##! experimental.givens
+
+given:
+  manufacturer_filter :: filter<string> is f''
+  subject_filter :: filter<string> is f''
+
+source: recalls is duckdb.table('data/auto_recalls.csv') extend {
+  where: Manufacturer ~ $manufacturer_filter, Subject ~ $subject_filter
+  measure: recall_count is count()
+}
+```
+
+A given is **declared bare** but **referenced with a `$` sigil** in expressions (`$manufacturer_filter`), as above.
+
+- **Give every optional filter a neutral, match-all default** - a `filter<>` given defaulting to `f''` - so an unsupplied value returns unfiltered rows, matching how `#(filter)` behaves when a value is omitted. Because the given bakes an always-on `where:` into the source, a non-neutral default (e.g. a date floor) applies to *every* read of the source, not just the ones that opt in - so keep defaults neutral. Defaults must be Malloy literals.
+- **Givens don't auto-inject a `where:`.** Unlike `#(filter)`, you write the filter expression that references the given yourself (e.g. `where: dimension ~ $given_name`).
+- **Not every filter maps cleanly.** A filter with no neutral match-all literal default - e.g. a scalar date/number range like `> @2020-01-01` - is not a good `given:`; keep those on `#(filter)`. Two more cases keep using `#(filter)`: mandatory scoping filters (`required`) and system-injected row-level filters (`implicit`), both below.
+
+Givens are also the substrate for access control - see "Access Control: Source Gating with `#(authorize)`" below.
+
+## Legacy: Parameterizable Filters with `#(filter)`
+
+`#(filter)` is the older, Publisher-specific mechanism for the same idea. Publisher parses the annotation, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. Prefer `given:` (above) for new models; keep reading and maintaining `#(filter)` on existing models, and keep using it for the two cases `given:` can't cover yet - `required` (mandatory scoping) and `implicit` (system-injected filters), below.
+
+Filters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an existing `#(filter)`-based source needs another knob, add it to the source itself, not to the consumer.
 
 ### Syntax
 

package/skills/malloy-modeling/SKILL.md

@@ -24,6 +24,7 @@ description: Build semantic models with Malloy for the Malloy Publisher. Read th
 6. **No fixed scale on measures**: use `# currency` not `# currency=usd0m`
 7. **Cast strings for aggregates:** `avg(score::number)` not `avg(score)`
 8. **Boolean columns:** use `= true` not `= 'true'` (no quotes!)
+9. **Read `.xlsx` in place:** `duckdb.table('data/file.xlsx')` works as-is; never convert a spreadsheet to Parquet/CSV first (quirks: `skill:malloy-gotchas-modeling`)
 
 ## Planning and `modeling-notes.md`
 

package/skills/malloy-publish/SKILL.md

@@ -85,7 +85,7 @@ All `.malloy` files must be in the package root (flat layout: the publisher does
 Publishable contents:
 - `.malloy` files - Semantic model definitions (base sources + joined sources)
 - `.malloynb` files - Notebooks for exploration/documentation (see `skill:malloy-notebooks`)
-- Data files (CSV/Parquet) - Embedded data published with package
+- Data files (CSV/Parquet/XLSX) - Embedded data published with package
 
 ## Version Management
 

package/skills/malloy-review/reference/rubric-correctness.md

@@ -111,7 +111,7 @@ For every rule, the linked instruction-skill section is the canonical source for
   1. Read the SQL string.
   2. **MANDATORY pre-check, schema verification.** Use `execute_query` to run `run: <table_path> -> { select: * limit: 1 }` against the underlying table referenced in the SQL. Compare the table's column list to the SQL's `SELECT` list. **Any column in the table that's NOT in the SELECT was being intentionally hidden by the SQL.** If so, the migration must preserve gating via `extend { except: ... }`, NOT just swap `sql()` → `table()`. Especially watch for columns named `month_*`, `weekly_*`, `daily_*`, `total_*`, `cum_*`, `running_*`, or anything matching `(month|day|week|hour)_<measure>`, these are typically pre-aggregated per-period rollups that would silently double-count if exposed and summed.
   3. Identify constructs used: `SELECT`, `GROUP BY`, `JOIN`, `CASE WHEN`, window functions (`LAG`, `LEAD`, `ROW_NUMBER`, `RANK`, `OVER`), CTEs, lateral joins, UNNEST, PIVOT, latest-snapshot subqueries, DB-specific functions, column aliases (`X AS Y`).
-  4. **MANDATORY: call `search_malloy_docs` for each non-trivial construct**, window functions, CTE chains, UNNEST/array access, PIVOT/conditional aggregation, latest-snapshot lookups. "When unsure" historically read as "never", agents skipped this step and asserted `conn.sql()` was justified when in fact Malloy expressed the pattern fine. The skip is not allowed. Record the search queries you ran inline in the finding's `suggested_fix` so the user can audit them.
+  4. **MANDATORY: call `search_malloy_docs` for each non-trivial construct**, window functions, CTE chains, UNNEST/array access, PIVOT and conditional aggregation, latest-snapshot lookups. "When unsure" historically read as "never", agents skipped this step and asserted `conn.sql()` was justified when in fact Malloy expressed the pattern fine. The skip is not allowed. Record the search queries you ran inline in the finding's `suggested_fix` so the user can audit them.
   5. **If every construct has a Malloy equivalent** → flag, with a sketched rewrite that includes the preserved gating clause.
   6. **If any construct genuinely requires raw SQL** → don't flag, but the genuine-requirement list is narrow (DML/DDL, certain `MERGE` patterns). Do not accept "Malloy can't express this cleanly" without citing a specific `search_malloy_docs` query that returned no equivalent.
 - **Common mappings the agent should know:**