@malloy-publisher/skills 0.1.0 → 0.1.2

package/README.md

@@ -4,10 +4,10 @@ The agent skills that ship with [Malloy Publisher](https://github.com/malloydata
 files you can install. They are task-specific guides for writing Malloy, building and reviewing
 models, exploring data, and authoring HTML data apps.
 
-Publisher already serves these skills over MCP, as prompts. This package exists for the cases MCP
-does not cover: a host that reads skill files from disk (Claude Code and the Publisher plugin), a
-scaffolder that installs them into a new project, or any tool that wants the `reference/` files that
-the prompts point at but cannot carry.
+Publisher already serves these skills over MCP, as prompts, including each skill's `reference/`
+files. This package exists for the cases MCP does not cover: a host that reads skill files from disk
+(Claude Code and the Publisher plugin), a scaffolder that installs them into a new project, or any
+tool that wants the files themselves rather than prompt text.
 
 ## Install
 

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.0",
+   "version": "0.1.2",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/malloy-getting-started/SKILL.md

@@ -13,6 +13,8 @@ At minimum you need `malloy_getContext`, `malloy_executeQuery`, and `malloy_sear
 
 If none of the tools are there, either the server is not running or your client connected before it was. Start the server (`npx @malloy-publisher/server --port 4000`, or `bun run build && bun run start` from a clone) and wait until `curl -s http://localhost:4000/api/v0/status` reports `operationalState: serving`.
 
+If there is no Publisher workspace here at all, and the user wants to work with data of their own rather than the bundled examples, `npm create @malloy-publisher/malloy-package@latest <name>` scaffolds one: the package and a starter model, registered so the server actually serves it, plus the start script, the MCP config and these skills. Keep the `@latest` when you type it: `npm create` resolves through npm's npx cache and an unversioned name is satisfied by any copy already there, so on a machine that has scaffolded before npm never asks the registry and you get an old scaffolder pinning an old server, with nothing to say so. Run bare, it comes with a small sample dataset, so there is something to query straight away. In a fresh directory `npm start` then runs the pinned server against the package in watch mode; if the directory already had a `package.json` the scaffolder leaves it alone and adds no script, printing the equivalent `npx` command to use instead. Where you run it matters: only the package lands in `<name>/`, and the workspace files, the agent instructions and the MCP config among them, are written to the current directory. Run it here if this directory is empty or is meant to become the workspace. If it already holds other work, scaffold into a new directory instead (`mkdir my-data && cd my-data`), because agent config is discovered by walking up, so writing those files here changes what every session beneath this directory inherits. Seed the starter model from a local file with `npm create @malloy-publisher/malloy-package@latest <name> -- --data <path/to/their-file.csv>` (CSV, Parquet, or Excel `.xlsx`), keeping the `--`, which is how `npm create` passes options through. That path is relative to wherever you run the command, so if you scaffolded into a new directory it has to reach back out to their file; the scaffolder copies it into the package and leaves the original alone. A seeded package starts smaller than the sample one, since the scaffolder does not read their columns: expect a row count and an overview, and build the model from there. A package is just Malloy, so it can instead query a database connection the config defines. Because it writes a `.mcp.json` that did not exist when the client connected, the user has to restart or reconnect once before these tools appear, and their client will ask them to approve the new project-scoped server the first time. That only works when the workspace is at the session's own root, so if you scaffolded into a new directory below that root, the user has to open a session there instead: a `.mcp.json` further down is never discovered.
+
 If you started the server yourself in this session, the tools still will not appear: your tool list was fixed when you connected, and you cannot reconnect yourself. Tell the user the tools are missing for that reason and ask them to run `/mcp`, select `malloy`, and choose Reconnect. The panel offers `Authenticate` first and reports `Auth: not authenticated`; that is a red herring, the endpoint has no auth. Restarting Claude Code also works. Continue once the tools are there.
 
 When a user is present, do not route around it by calling the REST API with curl. It appears to work, so the user never learns their session is missing the tools, and you lose what they are for: grounded discovery instead of guessed names, `malloy_compile` instead of throwaway queries, and `malloy_reloadPackage` instead of a restart. Say the tools are missing and let the user fix it in five seconds. Running unattended, with nobody who can reconnect you, is different: there the REST API is the supported interface, not a workaround. Discovery, query, compile, and reload all have REST equivalents (`malloy_searchDocs` and `malloy_getContext`'s plain-English ranking do not; read the bundled skills for syntax and ground from model metadata instead); the running server serves the full spec at `http://localhost:4000/api-doc.yaml`, and AGENTS.md carries the endpoint map.

package/skills/malloy-gotchas-modeling/SKILL.md

@@ -128,6 +128,14 @@ measure: median_x is sql_number("PERCENTILE_CONT(...) ...") { is_aggregate: true
 
 **Ship `avg` instead, or defer median with a documented gap** ("median deferred: no scalar median / runtime rejects raw-SQL aggregates"). Tell the user; don't silently substitute `avg` for a metric that was specified as median.
 
+**`stddev` does work**, so reach for it when the question is about spread. It is a native Malloy aggregate rather than a raw-SQL escape, so unlike everything above it compiles both inline and as a `measure:`, and it is the sample standard deviation. `variance`, `stddev_samp`, and `stddev_pop` are not Malloy functions, and pushing them through `!` fails as a scalar exactly like `percentile_cont!`.
+
+```malloy
+// WORKS: inline, or as a measure on a source
+run: order_items -> { aggregate: sd is stddev(sale_price) }
+source: items is order_items extend { measure: price_stddev is stddev(sale_price) }
+```
+
 ## Field Management: `extend {}` vs `include {}` Don't Compose
 
 Malloy has two field-management mechanisms for base sources. **`include {}` is the curated default; `extend { except / accept / rename }` is the fallback when a `rename:` is unavoidable.** They have different capabilities and **do not combine**.
@@ -206,6 +214,17 @@ source: facts is conn.table('t') -> { group_by: user_id, aggregate: total is sum
 
 To add `#(doc)` tags to existing query columns, use `include {}` between the query and extend.
 
+## Extending a Source Cannot Reuse a Name It Already Defines
+
+```malloy
+// WRONG: "Cannot redefine 'overview'" when sales already declares view: overview
+source: wines is sales extend { view: overview is { aggregate: record_count } }
+// RIGHT: give the extension its own name
+source: wines is sales extend { view: summary is { aggregate: record_count } }
+```
+
+An extension adds to the parent's namespace, it does not override it. This bites when you extend a source to "replace" one of its views: rename the new definition, or edit the view on the parent source instead of extending it. Malloy reports the same `Cannot redefine 'X'` for dimensions and measures that collide with an inherited name, per the sections above and below.
+
 ## Never Use `conn.sql()` When Malloy Has a Native Pattern
 
 ```malloy
@@ -238,6 +257,72 @@ source: facts is conn.table('orders') -> { group_by: user_id, aggregate: total i
 
 If a project's standards file specifies a stricter policy (e.g., a `search_malloy_docs` rationale comment requirement above every `conn.sql()` block), defer to that.
 
+## JSON Files: Read Them In Place Like CSV
+
+```malloy
+// RIGHT: .json works like .csv/.parquet
+source: reviews is duckdb.table('data/reviews.json')
+// RIGHT: newline-delimited JSON is read the same way
+source: events is duckdb.table('data/events.ndjson')
+// RIGHT: read options need read_json_auto in a SQL source
+source: nested is duckdb.sql("""SELECT * FROM read_json_auto('data/reviews.json')""")
+// WRONG: shelling out to python, or converting to CSV first
+```
+
+DuckDB reads JSON directly, so never preprocess a `.json` file before modeling it and never reach for a scripting language to inspect one. Both a top-level array of objects and newline-delimited JSON work through `duckdb.table()`.
+
+Quirk: JSON carries no schema, so a value written as `"90"` arrives as a string where the same data in CSV would be inferred as a number. Cast it in the source, under a new name (reusing the column's own name is a redefinition error):
+
+```malloy
+source: reviews is duckdb.table('data/reviews.json') extend {
+  dimension: points_num is points::number
+}
+```
+
+## Excel Files: Read `.xlsx` In Place, Never Convert
+
+```malloy
+// RIGHT when the sheet is a plain table (header in row 1, data under it, no blank row inside
+// it): read it where it sits, like .csv/.parquet (in a Publisher package the sandbox
+// connection is `duckdb`)
+source: budget is duckdb.table('data/budget.xlsx')
+// RIGHT for anything messier. Profile the top rows first to find the real header row and the
+// last real column, because nothing else will tell you where they are. Put the probe in the
+// model file as its own source: Publisher refuses raw SQL in an ad-hoc query.
+//   SELECT * FROM read_xlsx('data/sales.xlsx', sheet = 'Sales Data',
+//                           range = 'A1:Z15', header = false, all_varchar = true)
+source: sales is duckdb.sql("""
+  SELECT * FROM read_xlsx('data/sales.xlsx',
+    sheet = 'Sales Data',      -- EDIT: only the first sheet is read by default
+    header = true,
+    range = 'A5:J100000'       -- EDIT: A5 is the real header row. Keep the column bound at the
+  )                            -- last real column; the row bound just has to clear the end.
+  WHERE "Order ID" LIKE 'SO-%' -- EDIT, REQUIRED: a data-row predicate. This is what ends the
+""")                           -- read; drop it and every empty row in the range comes back.
+// WRONG: converting the spreadsheet to Parquet or CSV first (an unnecessary extra step)
+```
+
+Do not convert spreadsheets before modeling. DuckDB's excel extension reads `.xlsx` directly and loads automatically on first use, so a sheet that is a plain table needs nothing more than `duckdb.table()`. Converting does not avoid any of the problems below, it just moves them into a copy that goes stale the next time someone updates the workbook.
+
+**Plenty of real exports are not plain tables, and nothing tells you.** A report title, a "generated on" banner, a merged group header, a blank line above the header, or a blank spacer row inside the data are all ordinary, and none of them is visible from Malloy. There is no error either: the package loads, the server reports serving, the query returns 200, and the number is just wrong. So make two checks before building on the read: compare `aggregate: record_count is count()` against what you know is in the file, and `select: *; limit: 1` to see what the columns really are. If either disagrees with the file, the read is wrong and so is every measure over it.
+
+`table()` takes a plain file path only, so anything needing `read_xlsx` options (`sheet`, `range`, `header`, `ignore_errors`, `normalize_names`, `all_varchar`, `empty_as_varchar`, `stop_at_empty`) goes through the SQL-source form.
+
+Quirks:
+
+- Only the FIRST sheet is read by default. Select another with `sheet = 'Name'`. There is no function that lists a workbook's sheet names, but passing one that does not exist reports a suggestion (`Sheet "x" not found ... Did you mean: "Notes"`), which is one way to find a name you were not given.
+- A title or banner row above the header collapses the read. DuckDB takes the first row it finds as the column names, so a lone title cell in A1 becomes the only column. How many rows you then get is the next quirk's business: whatever sits between the title and the first blank row, often none or one, otherwise a plausible-looking partial count. Pass a `range` that starts at the real header row.
+- With no `range`, `stop_at_empty` defaults to true and the read stops at the first blank row, which on a real sheet is usually a spacer between blocks rather than the end of the data: a 30-row sheet with one spacer after row 10 reads as 10 rows. `stop_at_empty = false` lifts that, but it only helps when the header really is in row 1; with a title above the header you need the `range` anyway, and a `range` flips the default for you. It also hands the blank rows back as all-null rows, so the count comes out one high per spacer until you filter them.
+- A `range` reads every cell inside it, so an overshot bound manufactures padding: past the last real column you get all-null fields (`A5:Z100000` on a ten-column sheet yields 26, the extras named `C10` and `_1` through `_15`), and past the last real row all-null rows (`A5:J100000` on a 1,500-row sheet reads 99,995). Spacers, subtotals, and footnotes come through as rows too. So the row filter is not tidying-up, it is the thing that ends the read: filter to what a data row looks like (`WHERE "Order ID" LIKE 'SO-%'`) rather than to `IS NOT NULL`, which keeps any footnote carrying text in the first column. A bound that falls SHORT of the data is the dangerous direction: the rows and columns past it are dropped with no error at all, so overshoot the row bound and let the filter end the read.
+- Every number in an xlsx is stored as a double, so there are no integer columns. Typing is per column and decided by the FIRST data row, and `$1,234`, `12%` and `N/A` are all text: a text cell in that first row makes the whole column a string (on one real export, all ten of them), while a text cell further down leaves the column numeric and makes the read throw instead (`Could not convert string ... to DOUBLE`). `ignore_errors = true` fixes that second case, nulling the bad cells and keeping the column a number. It does nothing for the first.
+- Sample the column's SHAPES before writing any conversion, not its values: `run: source -> { group_by: shape is replace(raw_col, r'[0-9]', '9'); aggregate: n is count(); order_by: n desc }` collapses every value to its format and counts it, so on one real price column the 16 euro-denominated rows surface beside the 1,484 in dollars. A plain `group_by raw_col; limit: 20` sorts lexicographically, which hides exactly the shapes that matter.
+- Convert in the SQL source, not in Malloy, where `::number` throws on the first bad cell. `try_cast(regexp_replace("Total Revenue", '[^0-9.-]', '', 'g') AS double)` nulls what it cannot read instead of failing and is right for a plain `$1,234.56`, but it is not a general parser. It concatenates every digit in the cell, so `1,234 (see tab 2)` becomes 12342. It understands only a leading ASCII `-`, so an accounting `(1,234)`, a Unicode minus and a `CR` suffix all come back positive, while a trailing `-` (`1,234-`) comes back null and drops the row from the sum. And it assumes `.` is the decimal point, so a European `1.234,56` comes back a thousandfold small. Handle the shapes your sample actually found, and divide a percent by 100. Failure is quiet either way: a cast that fails on every row sums to 0 rather than erroring, and a text date strips to a number rather than a null (`'01/02/2023'` becomes 1022023).
+- Check the answer against the sheet's own total row, read as raw text. Lift the data-row filter and select the footer by its label, which usually sits in a different column from the one your data-row predicate uses: on one export `WHERE "Customer Name" = 'TOTAL'` finds it and `WHERE "Order ID" = 'TOTAL'` returns nothing, and an empty result reads as a pass. Do not run the total through the same expression, because a wrong sign survives a row count, survives `select: *`, and cancels out when both sides are parsed the same broken way.
+- A sheet with no header row whose first row is all text silently loses that row to header detection. Pass `header = false`.
+- Headers with spaces are kept verbatim: backtick them in Malloy, or pass `normalize_names = true` for snake_case names.
+- `all_varchar = true` hands back each cell's stored value as text, so a date arrives as its raw Excel serial number rather than a date: `'44929'` from a sheet Excel wrote, `'44927.0'` from one DuckDB's own xlsx writer wrote, and `'44929.5'` where the cell carries a time of day. Which form you get depends on the tool that wrote the file, so do not detect serials by matching for an integer; `try_cast(... AS double)` accepts all three and returns null for a cell that was stored as text (`'01/02/2023'`), which is the test you want. Convert with `date '1899-12-30' + floor(try_cast(d AS double))::int`, not from 1900-01-01. Both wrappers earn their place: adding a double to a date does not compile, and a bare `::int` rounds, so an afternoon timestamp would land on the next day.
+- A date column that mixes both, which is what an export edited by hand gives you, needs both branches or you silently lose every row of one kind: `CASE WHEN try_cast(d AS double) IS NOT NULL THEN date '1899-12-30' + floor(try_cast(d AS double))::int ELSE try_strptime(d, '%m/%d/%Y')::date END`. Without `all_varchar`, a uniformly date-formatted column arrives as real `date` and `timestamp` values, and a stray text cell behaves exactly as the typing rule above says. Note what `ignore_errors = true` does here: it nulls that cell rather than parsing it, so the hand-typed date is lost silently.
+
 ## Duplicate Rows: Check Before Building Measures
 
 ```malloy
@@ -271,7 +356,7 @@ Malloy compiles top-to-bottom. Define lookup/dimension tables before the source
 Call `search_malloy_docs` BEFORE first use of any of these. Don't guess the syntax:
 - `pick` expressions
 - Window functions (`calculate`)
-- `percentile` or statistical functions: but see the hard limit above, raw-SQL aggregates (`sql_number` / `is_aggregate` / `percentile_cont!`) do **not** compile as measures in this build; there is no scalar median
+- `percentile` or statistical functions: but see the hard limit above, raw-SQL aggregates (`sql_number` / `is_aggregate` / `percentile_cont!`) do **not** compile as measures in this build; there is no scalar median (`stddev` is the exception and does work as a measure)
 - Time interval functions (`days()`, `seconds()`): only `seconds`/`minutes`/`hours`/`days` exist (see above)
 - Query-based sources (`from()`)
 - `!` operator / `sql_number()`

package/skills/malloy-gotchas-queries/SKILL.md

@@ -56,6 +56,27 @@ measure: cogs is inventory_items.item_cost.sum()
 
 `sum`, `avg`, `min`, and `max` over a dotted joined path all produce that compile error; the diagnostic message even tells you the exact fix. Don't worry about catching this in code review; the compiler does it for you.
 
+**Method syntax is for aggregates over a path. Scalar functions never take it.**
+
+```malloy
+// WRONG: "something is missing before 'round'"
+aggregate: avg_price_r is avg(price).round(2)
+aggregate: avg_price_r is price.avg().round(2)
+// WRONG: "Cannot call function round(number, number) with source"
+aggregate: avg_price_r is avg_price.round(2)
+dimension: rounded is price.round(2)
+// RIGHT: scalar functions are always call form
+aggregate: avg_price_r is round(avg(price), 2)
+dimension: rounded is round(price, 2)
+```
+
+Two separate rules produce those errors:
+
+- **No method call chains onto the result of a function call.** `avg(price).round(2)` and `price.avg().round(2)` are both parse errors. The message names `round` without saying it is unsupported in that position, so it reads like a typo somewhere else. `.floor()` and `.ceil()` fail identically.
+- **Scalar functions have no method form.** `round`, `floor`, and `ceil` are always `round(x, 2)`, never `x.round(2)`, whether `x` is a named measure or a plain column.
+
+`price.avg()` and `inventory_items.item_cost.sum()` are correct because `avg` and `sum` are aggregate functions over a field path, which is exactly what method syntax is for.
+
 **Exception: `count(joined.field)` is correct, not a bug.** `count(joined.field)` is the **canonical Malloy idiom** for distinct-count through a join. Keep it as-is even when nearby `sum`/`avg`/`min`/`max` calls have to use method syntax. The closest method-syntax form `joined.count()` counts *rows* in the joined source (different semantics, differs from the distinct count when the joined field has duplicates within the joined table). The Malloy docs example `joined.count(field)` does NOT compile against current Malloy (error: `Expression illegal inside path.count()`); it only works for double-nested paths like `aircraft.count(aircraft_models.code)`.
 
 ## Chart Annotation Placement

package/skills/malloy-materialization/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: malloy-materialization
+description: Add and debug Malloy Persistence materializations in a package - persist an expensive source so queries read a pre-built table. Read this whenever the user wants to materialize a source, add a persist annotation, speed up a slow source, or asks why a persist source isn't building.
+---
+
+# Materialization (Malloy Persistence)
+
+Materialize an expensive source once so queries read a **pre-built warehouse table** instead of recomputing it every time. You tag a source `#@ persist`, a materialization run builds it into a physical table, and queries against it are rewritten to read that table.
+
+> **The #1 gotcha, up front:** if a persist source isn't materializing, it is almost always one of two things - a `.malloy` file in the package missing the `##! experimental.persistence` flag (which aborts the *whole* package's build plan), or no build ever ran (a standalone Publisher does not build on publish - see **Building and refreshing**). Jump to **Debugging a no-op build**.
+
+## The recipe (get this right and it just works)
+
+1. **`##! experimental.persistence` on EVERY `.malloy` file in the package** - not only the file that declares the persist source. Either form enables it:
+   - `##! experimental.persistence`, or
+   - `##! experimental { access_modifiers, sql_functions, persistence }` (add `persistence` to the existing list).
+
+   **Why every file:** the build plan is computed by asking *every* `.malloy` file in the package for its persist sources, and that call **throws on any file whose model lacks the flag** (`Model must have ##! experimental.persistence`). One unflagged helper or import file, even one with no persist source of its own, aborts the whole package's build plan, so *every* persist source in the package drops out. This is the most common cause of a no-op build.
+
+2. **`#@ persist name="..."` on a query-based source, with the name quoted:**
+   ```malloy
+   #@ persist name="my_dataset.my_table"
+   source: my_rollup is some_source -> { group_by: ...; aggregate: ... }
+   ```
+   - **Only `query_source` and `sql_select` sources are persistable** - a source whose definition has a `-> { ... }` pipeline or a `conn.sql("...")`. This **includes** one refined by a trailing `extend { ... }`. What is **not** persistable is a *plain* `extend` over a bare `conn.table(...)`; a `#@ persist` on such a source is **silently ignored** (its annotation is never read) - that one source just won't materialize, and the rest of the package still builds.
+   - **Quote the name.** `name="my_table"` (or a path `name="dataset.table"` / `name="project.dataset.table"`) is required. A **bare** `name=my_table` **always fails the build/publish** with `persist annotation name must be quoted` (a raw-source scan that hard-stops); it never silently no-ops.
+   - `name=` is the target table name. In a standalone Publisher this **is** the physical table (rebuilt in place); a hosted (control-plane) deployment builds it under a content-addressed generation name. In both, the source's identity for reuse is a content address of its connection and canonical SQL (its `sourceEntityId`), so **republishing unchanged persist logic reuses the existing table** and changing the logic builds fresh.
+
+3. **Package persistence policy in `publisher.json`** (all optional):
+   ```jsonc
+   {
+     "name": "my-package",
+     "scope": "package",  // default; "version" = each published version owns its own tables
+     "materialization": { "freshness": { "window": "24h", "fallback": "live" } }
+   }
+   ```
+   Enforced at publish (strict), on edits (strict), at load (warn, still serves), and by the scheduler (an offending package is skipped):
+   - **`scope`**: `package` (default; artifacts reused across published versions) or `version` (each artifact owned by one version). Package-level only; there is no per-source scope.
+   - **`materialization.freshness`** (`window` + `fallback` of `live`/`stale_ok`/`fail`) is the objective a **hosted control plane** enforces by refreshing the table to meet it (`fallback: "live"` serves live compute while stale/absent). A **standalone** Publisher does **not** act on `freshness` for refresh - see **Building and refreshing**.
+   - **`materialization.schedule`** is a 5-field UTC cron (`min hour dom mon dow`; `L`/`W`/`#`/`?` rejected). It **requires `scope: "version"`** and is **mutually exclusive with `freshness`**. This is how a standalone Publisher refreshes on a cadence.
+
+4. **Reads vs writes.** The persist source can *read* any dataset the connection can read; the persist *target* (`name=`'s dataset) must be a dataset the connection can **write** (typically a scratch dataset).
+
+## Building and refreshing (standalone vs. hosted)
+
+A `#@ persist` tag declares *what* to materialize; it does not by itself build anything.
+
+- **Standalone Publisher:** publishing or loading a package only computes its build plan - **no table is built until a materialization run executes.** Trigger one explicitly (`malloy-pub materialize --package <pkg> --wait`, or the materialization API), or turn on the opt-in local scheduler (off unless `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER` is set) to fire the package's `schedule` cron. Refresh is a re-run or that cron; `freshness` is not a refresh trigger here, so a freshness-only standalone package builds once and is not auto-refreshed.
+- **Hosted (control-plane) deployment:** the build runs automatically on publish, best-effort - a build failure does **not** fail the publish (which is why a broken persist can look like a silent no-op), and the control plane drives refresh to meet the `freshness` objective.
+
+Either way, a successful publish alone does not prove a table exists - confirm the build separately.
+
+## Serve-time routing is `query_source`-only (today)
+
+Both persistable types *build* a table, but only a **`query_source`** (a `-> { ... }` pipeline) is rewritten to *read* it at query time. A raw **`sql_select`** (`conn.sql("...")`, including `conn.sql("...") extend { ... }`) builds its table and then the query path re-inlines its SQL, so the table is built and never read, and queries are no faster. If you have raw SQL you want served from a table, wrap it in a thin `query_source` and persist that:
+
+```malloy
+source: x_raw is my_conn.sql("select ...")
+#@ persist name="scratch_dataset.x"
+source: x is x_raw -> { select: * }
+```
+
+## Confirming it worked
+
+After a build runs, re-run one of the source's queries - a persisted `query_source` should return quickly, reading the pre-built table instead of recomputing the upstream. Your host also reports each persisted source as **ready** with its physical table name (a materialization run detail, CLI listing, or materialization view, depending on the host); if nothing is listed, either no build ran (standalone) or the build plan was empty - see **Debugging a no-op build**.
+
+## Debugging a no-op build
+
+Symptom: no table was built and the source still recomputes on every query. Check, in order:
+
+0. **Did a build actually run?** On a standalone Publisher, publish/load does **not** build - run `malloy-pub materialize` (or enable the scheduler). "Publishes fine, no table" is the *expected* standalone state, not a model bug. On a hosted deployment the build is automatic but best-effort, so a failure is silent - look for a `FAILED` run.
+1. **A `.malloy` file missing the persistence flag** (the most common real bug). Every model file's `##!` line needs `persistence`, including pure helper/import files with no persist source - one unflagged file aborts the whole package's build plan.
+2. **An unquoted persist name** - a bare `name=foo` **always** hard-stops the build/publish with `persist annotation name must be quoted`; use `name="foo"`. (If you got *no* error at all, it isn't this.)
+3. **A `#@ persist` on a non-persistable source** - a bare `extend` over `conn.table(...)` is silently ignored, so *that* source won't materialize (the rest of the package is unaffected). Tag a `query_source` / `sql_select` instead.
+4. **A persisted raw `sql_select` that builds but is never read** - if the table exists yet queries are no faster, it's the serve-routing gap above; wrap the `sql_select` in a `query_source`.
+
+**Isolation test** - add a trivial, self-contained persist source in its own file and rebuild:
+```malloy
+##! experimental.persistence
+source: smoke_raw is my_conn.table('some_dataset.some_table')
+#@ persist name="scratch_dataset.persist_smoke_test"
+source: persist_smoke is smoke_raw -> { aggregate: n is count() }
+```
+- If **even this** doesn't build (after a real materialization run), the whole package's plan is aborting - a sibling `.malloy` file is missing the flag. Fix rule 1 across the package.
+- If the smoke source **does** build but your real one doesn't, your real source is the problem - a non-persistable type (a bare `extend`), or its own file's flag.
+
+Delete the smoke file and drop its table afterward.
+
+## Gotchas
+
+- **Every `.malloy` file needs the persistence flag** - one unflagged file aborts the whole package's build plan. (A `#@ persist` on a *non*-persistable source, by contrast, is silently ignored and does not affect other sources.)
+- **A tag doesn't build** - a standalone Publisher materializes only on an explicit run or its scheduler; only a hosted control plane builds on publish.
+- **Serve-time routing is `query_source`-only** - a raw `sql_select` builds a table the query path doesn't read; wrap it in a `query_source`.
+- **Quote the name** - a bare `name=` always hard-stops the build.
+- **Republishing unchanged persist logic reuses the table** - reuse is keyed on the content-addressed `sourceEntityId`, not the `name=`.
+- **Removing a persist source (or a smoke test) does not drop its table** - physical-table cleanup is the caller's responsibility; drop it yourself.

package/skills/malloy-model/SKILL.md

@@ -182,11 +182,36 @@ source: customer_health is customers extend {
 - When both a combined table (all types) and filtered/split tables exist, prefer the split tables
 - **DRY: define measures/dimensions in base source files, not inline in views**
 
-## Parameterizable Filters with `#(filter)`
+## Parameterizing sources with `given:` (preferred)
 
-`#(filter)` annotations declare filterable dimensions on a source. Publisher parses them, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. This gives you a clean way to expose tunable knobs (date range, region, manufacturer) without hand-rolling parameterization in every query.
+Native Malloy **`given:` parameters** are the going-forward way to expose tunable knobs (date range, region, manufacturer) on a source - prefer them over `#(filter)` when you author a new model. A `given:` is a first-class runtime parameter you reference in the model's own logic; callers supply values at query time and the model uses them however it declares. Enable them with `##! experimental.givens` at the top of the model.
 
-Filters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an analysis needs a knob the source doesn't expose, the right move is to add a `#(filter)` to the source itself, not to wedge filtering into the consumer.
+```malloy
+##! experimental.givens
+
+given:
+  manufacturer_filter :: filter<string> is f''
+  subject_filter :: filter<string> is f''
+
+source: recalls is duckdb.table('data/auto_recalls.csv') extend {
+  where: Manufacturer ~ $manufacturer_filter, Subject ~ $subject_filter
+  measure: recall_count is count()
+}
+```
+
+A given is **declared bare** but **referenced with a `$` sigil** in expressions (`$manufacturer_filter`), as above.
+
+- **Give every optional filter a neutral, match-all default** - a `filter<>` given defaulting to `f''` - so an unsupplied value returns unfiltered rows, matching how `#(filter)` behaves when a value is omitted. Because the given bakes an always-on `where:` into the source, a non-neutral default (e.g. a date floor) applies to *every* read of the source, not just the ones that opt in - so keep defaults neutral. Defaults must be Malloy literals.
+- **Givens don't auto-inject a `where:`.** Unlike `#(filter)`, you write the filter expression that references the given yourself (e.g. `where: dimension ~ $given_name`).
+- **Not every filter maps cleanly.** A filter with no neutral match-all literal default - e.g. a scalar date/number range like `> @2020-01-01` - is not a good `given:`; keep those on `#(filter)`. Two more cases keep using `#(filter)`: mandatory scoping filters (`required`) and system-injected row-level filters (`implicit`), both below.
+
+Givens are also the substrate for access control - see "Access Control: Source Gating with `#(authorize)`" below.
+
+## Legacy: Parameterizable Filters with `#(filter)`
+
+`#(filter)` is the older, Publisher-specific mechanism for the same idea. Publisher parses the annotation, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. Prefer `given:` (above) for new models; keep reading and maintaining `#(filter)` on existing models, and keep using it for the two cases `given:` can't cover yet - `required` (mandatory scoping) and `implicit` (system-injected filters), below.
+
+Filters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an existing `#(filter)`-based source needs another knob, add it to the source itself, not to the consumer.
 
 ### Syntax
 

package/skills/malloy-modeling/SKILL.md

@@ -24,6 +24,7 @@ description: Build semantic models with Malloy for the Malloy Publisher. Read th
 6. **No fixed scale on measures**: use `# currency` not `# currency=usd0m`
 7. **Cast strings for aggregates:** `avg(score::number)` not `avg(score)`
 8. **Boolean columns:** use `= true` not `= 'true'` (no quotes!)
+9. **Read data files in place:** `.csv`, `.parquet`, `.json`, `.ndjson`, and `.xlsx` all work as-is through `duckdb.table('data/file.ext')`. Never convert a file to another format first, and never read one with python or jq to "have a look" first: query it. For `.xlsx`, check the row count before trusting it: a workbook with a title row or a blank spacer reads short and reports no error. (Per-format quirks: `skill:malloy-gotchas-modeling`)
 
 ## Planning and `modeling-notes.md`
 

package/skills/malloy-publish/SKILL.md

@@ -85,7 +85,7 @@ All `.malloy` files must be in the package root (flat layout: the publisher does
 Publishable contents:
 - `.malloy` files - Semantic model definitions (base sources + joined sources)
 - `.malloynb` files - Notebooks for exploration/documentation (see `skill:malloy-notebooks`)
-- Data files (CSV/Parquet) - Embedded data published with package
+- Data files (CSV/Parquet/XLSX) - Embedded data published with package
 
 ## Version Management
 

package/skills/malloy-review/reference/rubric-correctness.md

@@ -111,7 +111,7 @@ For every rule, the linked instruction-skill section is the canonical source for
   1. Read the SQL string.
   2. **MANDATORY pre-check, schema verification.** Use `execute_query` to run `run: <table_path> -> { select: * limit: 1 }` against the underlying table referenced in the SQL. Compare the table's column list to the SQL's `SELECT` list. **Any column in the table that's NOT in the SELECT was being intentionally hidden by the SQL.** If so, the migration must preserve gating via `extend { except: ... }`, NOT just swap `sql()` → `table()`. Especially watch for columns named `month_*`, `weekly_*`, `daily_*`, `total_*`, `cum_*`, `running_*`, or anything matching `(month|day|week|hour)_<measure>`, these are typically pre-aggregated per-period rollups that would silently double-count if exposed and summed.
   3. Identify constructs used: `SELECT`, `GROUP BY`, `JOIN`, `CASE WHEN`, window functions (`LAG`, `LEAD`, `ROW_NUMBER`, `RANK`, `OVER`), CTEs, lateral joins, UNNEST, PIVOT, latest-snapshot subqueries, DB-specific functions, column aliases (`X AS Y`).
-  4. **MANDATORY: call `search_malloy_docs` for each non-trivial construct**, window functions, CTE chains, UNNEST/array access, PIVOT/conditional aggregation, latest-snapshot lookups. "When unsure" historically read as "never", agents skipped this step and asserted `conn.sql()` was justified when in fact Malloy expressed the pattern fine. The skip is not allowed. Record the search queries you ran inline in the finding's `suggested_fix` so the user can audit them.
+  4. **MANDATORY: call `search_malloy_docs` for each non-trivial construct**, window functions, CTE chains, UNNEST/array access, PIVOT and conditional aggregation, latest-snapshot lookups. "When unsure" historically read as "never", agents skipped this step and asserted `conn.sql()` was justified when in fact Malloy expressed the pattern fine. The skip is not allowed. Record the search queries you ran inline in the finding's `suggested_fix` so the user can audit them.
   5. **If every construct has a Malloy equivalent** → flag, with a sketched rewrite that includes the preserved gating clause.
   6. **If any construct genuinely requires raw SQL** → don't flag, but the genuine-requirement list is narrow (DML/DDL, certain `MERGE` patterns). Do not accept "Malloy can't express this cleanly" without citing a specific `search_malloy_docs` query that returned no equivalent.
 - **Common mappings the agent should know:**