@malloy-publisher/skills 0.1.36 → 0.1.37

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.36",
+   "version": "0.1.37",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",

package/skills/eval-answer/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-answer
 description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # Evaluate One Answer

package/skills/eval-diagnose/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-diagnose
 description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # Diagnose One Answer

package/skills/eval-import/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-import
 description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # Import questions into an eval set

package/skills/eval-improve/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-improve
 description: 'Make the smallest safe Malloy model edit that closes a diagnosed model-owned gap, with a probe receipt for every factual claim. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # Improve the Model

package/skills/eval-judge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-judge
 description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # The judge

package/skills/eval-loop/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-loop
 description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files: the set in the model package''s evals/ directory, runs in the set''s workdir; checkpoints are git commits of the model repo. `eval.py` runs each step from the set''s eval.toml. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # The Evaluation Loop

package/skills/eval-report/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: eval-report
 description: 'Write the summary a person reads after an evaluation run: what ran, what it scored, which failures were the MODEL and which were the EVAL itself (out of turns, contaminated, unestablished golden, unreadable judge), what was learned, and what to do next. Every claim links to the artifact behind it. Use at the end of any run, before quoting a number to anyone. Never scores an answer (eval-answer) or decides who owns a failure (eval-diagnose).'
-version: 0.1.36
+version: 0.1.37
 ---
 
 # Report a Run

package/skills/malloy-analysis/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-analysis
 description: "Read before your first get_context call on any business or data question. The required workflow, not background: which source, which rows, how to verify. Skipping it causes confident wrong answers."
-version: 0.1.36
+version: 0.1.37
 ---
 
 <!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->
@@ -24,6 +24,7 @@ Find the right entities before writing any query.
 
 - If you do not already know which package to work in, confirm the environment and package with the user before continuing.
 - **Load `skill:malloy-phrase-detection`**, then call `get_context` with entity targets that describe the fields the question needs: a `measure` for the metric, a `dimension` for each breakdown or filter, and a `view` if the question sounds like a canned report. **Every target carries a `search_text`**: a target without one enumerates a capped catalog instead of searching, and the cap is blind to your question. The tool description covers what comes back and how to narrow or browse.
+- **A value the user typed (a brand, a city, a product name) is not yet a filter.** Resolve it to the string the data stores before you filter on it: a `dimensional_value` target in that `get_context` call, or, where value search isn't available, a query of the dimension's distinct values. "ray ban" may be stored as `Ray-Ban` and `Ray-Ban Junior`; a `where:` on the user's own spelling, or a regex guessed from it, can match nothing and still return a result.
 - Read the documentation on each returned entity: on an entity it is `description`, on a source it is `docs`, often with a short `one_line_summary` and sometimes a longer `summary`. Which of these a server fills varies, so read the ones your response actually carries and do not go hunting for one that is absent. Authors write it as `#(doc)` in the model, but the response never uses that label, so do not go looking for that either. It is where grain, units, null handling, and any source-level filters are described. Confirm the exact field names against the results before using them.
 - **Read the source's own documentation too, not just each field's.** The source-level `docs` often defines the grain, the universe of rows it represents, how joins behave, and source-level filters or assumptions that apply to every query rooted on it. Factor both the source and the field docstrings into how you build and later verify the query.
 - **Do not rebuild a view that already exists from its description.** A description says what a view does, not how; rebuilding from prose loses what prose does not carry: a denominator, a `partition_by`, an exact filter. Set `entity_name` in the scope to get that entity on its own instead of a ranked sweep. That also returns its `code` (for a view, its definition as written): read it first, with the source's `docs` and the entity's `description`. Then RUN the view and adapt its output rather than writing a replacement; to change a band width or grain, change that one thing and keep the rest. Where a server returns no `code`, the docs plus the view's own output are what you have, and a calculation you cannot see is a reason to reuse the view rather than guess at it.

package/skills/malloy-analysis-report/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-analysis-report
 description: Combine validated Malloy queries into a notebook report. Use when the user asks to "create a report", "combine these into a report", or wants a persistent multi-query artifact.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-charts/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-charts
 description: Read before choosing a chart or adding a chart annotation. Chart types, tag syntax and scale rules, KPI cards, dashboards, sparklines, and the renderer mistakes that fail silently.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-dashboards/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-dashboards
 description: "Build or modify a Malloy Publisher dashboard, a tagged .malloy file in a package's dashboards/ directory, with auto-rendered filter controls, a grid layout, and # drill click-through. Use when the user asks for a dashboard, a filterable operational view, or drill-through between views, and no code is wanted."
-version: 0.1.36
+version: 0.1.37
 ---
 
 <!--
@@ -68,8 +68,12 @@ endpoint named beside it.
    already-defined errors that reads as broken Malloy rather than a wrong scope. **Editing a shared
    include wants `"scope": "package"`**, which recompiles every file as saved: `file` only checks the
    one you are editing, so renaming a source in `_shared.malloy` passes it while breaking every
-   dashboard that imports it. A clean compile is not a working dashboard: some tag mistakes surface
-   at step 6, and some only when you look at the page in step 7. **A not-yet-saved dashboard wants
+   dashboard that imports it. **`"scope": "file"` does not check tiles, drills or suggest queries.**
+   To see those before you save, compile the same text at `"scope": "package"` with `modelPath` set
+   to the dashboard's path: it runs the lint step 6 reads and returns its findings with code
+   `dashboard-lint` (or `render-tag`), each at the severity a load gives it, so an error finding
+   makes the compile an error too. Even that is not a working dashboard: some mistakes only show when
+   you look at the page in step 7. **A not-yet-saved dashboard wants
    `"scope": "file"` too**, not the default: a dashboard file opens with an `import`, and the
    default `append` scope refuses one, so the new-file case fails on the import and again on the
    path that does not exist yet. `append` is for a fragment checked against a model that is
@@ -125,8 +129,9 @@ nest's pipeline starts from its own query's source and there is no way to combin
 
 Three things the form costs, so you are not surprised by them:
 
-- **A tile expression is a string in an annotation, so the compiler never checks it.** Rename a view
-  and the file still compiles; the tile fails at package load. Read the lint (step 6).
+- **A tile expression is a string in an annotation, so the Malloy compiler never checks it.** Rename
+  a view and the file still compiles; the tile fails at package load. The lint names it: at step 6,
+  or before saving with `compile_model` at `"scope": "package"` (step 5).
 - **No per-parent-row grouping.** There is no parent query to repeat a grid over.
 - **Filtering lives on the tiles**, not on the page: each view's `+ { where: ... }` names the
   controls it answers to. Below is why, and the one thing that does not work.
@@ -313,6 +318,10 @@ value, that the prefix "is not a well-formed route", because a route ends at the
 complaint is a **compile** diagnostic on a compile that still succeeds, not a package warning, so
 step 6 will not show it. Pick by which reader you care about.
 
+`dimension=` can name a field of a directly joined source (one join level). Quote it, because
+an unquoted dotted value does not parse as a tag:
+`suggest { source=sales dimension="products.department" }`.
+
 On the `query=` form of `suggest`, `dimension` names a column of that query's output, not a path into
 the model, and only the last segment of it is read. Match it to the suggest query's `group_by`. When it
 matches no column the picker falls back to the first column of the result: silently right for a

package/skills/malloy-define/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-define
 description: Propose scope, a source plan and field definitions for a Malloy model. Which tables and questions, which sources at what grain, then renames, dimensions and measures, each backed by querying data.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-discover/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-discover
 description: Silent data discovery for Malloy modeling. Used at Step 1 of the modeling workflow. Scans tables, columns, distributions, and relationships without user interaction. The agent builds an internal picture before presenting anything.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-document/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-document
 description: 'Add documentation with #(doc) tags to Malloy models so fields and sources are described in plain language. Use when user asks to "add documentation", "add doc tags", "document the model", or wants fields and sources described for natural-language search and discovery. For runtime filters, declare given: parameters (see the malloy-model skill); #(filter) is deprecated. Filters are a runtime/modeling construct (governance, latency, correctness), not a documentation tag.'
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-getting-started/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-getting-started
 description: First steps for using a Malloy Publisher deployment through its MCP tools. Use when connecting to Publisher for the first time, when you do not yet know the available environments, packages, or models, or when a user asks what data they can explore. Also use for "malloy help" or which Malloy skill to read. Covers verifying the server, discovering data with get_context, and running a first grounded query.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-gotchas-modeling/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-gotchas-modeling
 description: Malloy modeling mistakes and compile-error fixes. Read before writing sources, dimensions, measures or joins, and when a .malloy file will not compile. Reserved words, NULLs, dates, field access.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.
@@ -444,7 +444,7 @@ This is the modeling-time consequence of ignoring `skill:malloy-define`'s scope
 
 ## Thresholds Are Decisions, Not Syntax
 
-Before writing a `pick` expression or filtered measure with a numeric cutoff, see `skill:malloy-model` § Key Rules: every boundary must be user-supplied, distribution-derived (query the percentiles first), or explicitly flagged as an assumption in its `#(doc)`. Never invent one silently.
+Before writing a `pick` expression or filtered measure with a numeric cutoff, see `skill:malloy-model` § Key Rules: every boundary must be user-supplied, distribution-derived (query the percentiles first: Malloy has no percentile function, so use the **Tier boundaries** query in `skill:malloy-discover`), or explicitly flagged as an assumption in its `#(doc)`. Never invent one silently.
 
 ## `except:` Removes Fields From Namespace Entirely
 

package/skills/malloy-html-data-apps/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-html-data-apps
 description: "Build or modify an in-package HTML data app for a Malloy Publisher package (a public/ directory the package serves): design it, write its Publisher.query code, embed it. Hand-authored, no build step."
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-lookml-review/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-lookml-review
 description: Analyze LookML files as prior art for Malloy modeling. Used during Step 1 (DISCOVER) when .lkml files are present. Coordinates reference files that extract business logic, relationships, and curation decisions. Works with or without a database connection.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-lookml-review/reference/propose-fields.md

@@ -60,7 +60,7 @@ dimension: is_complete is status = 'complete'
 | `type: min` / `max` | `min(field)` / `max(field)` | Direct mapping |
 | `type: number` | Derived measure expression | Usually a ratio; use `nullif()` for division |
 | `type: list` | No direct equivalent | Flag for alternative approach |
-| `type: percentile` | No measure form: Malloy has no `percentile` function | Compute the value with the two-stage query in `skill:malloy-discover` § Example Queries, and tell the user it is a query result, not a reusable measure |
+| `type: percentile` / `type: median` | No measure form: Malloy has no `percentile` function | Compute the value with the two-stage query in `skill:malloy-discover` § Example Queries, and tell the user it is a query result, not a reusable measure |
 
 **Filtered measures:** `filters:` → Malloy `{ where: }` syntax:
 ```

package/skills/malloy-materialization/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-materialization
-description: Add and debug Malloy Persistence materializations in a package - persist an expensive source so queries read a pre-built table. Read this whenever the user wants to materialize a source, add a persist annotation, speed up a slow source, tune what to persist, or asks why a persist source isn't building.
-version: 0.1.36
+description: Add, confirm and debug Malloy Persistence materializations in a package - persist an expensive source so queries read a pre-built table, in the source's own warehouse or in a storage destination. Read this whenever the user wants to materialize a source, add a persist annotation, speed up a slow source, build one stored rollup over another, tune what to persist, or asks why a persist source isn't building or isn't being read.
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.
@@ -10,80 +10,89 @@ SPDX-License-Identifier: MIT
 
 # Materialization (Malloy Persistence)
 
-Materialize an expensive source once so queries read a **pre-built warehouse table** instead of recomputing it every time. You tag a source `#@ persist`, a materialization run builds it into a physical table, and queries against it are rewritten to read that table.
+Materialize an expensive source once so queries read a **pre-built table** instead of recomputing it every time. You tag a source `#@ persist`, a materialization run builds it into a physical table, and queries against it are rewritten to read that table.
 
-> **The #1 gotcha, up front:** if a persist source isn't materializing, it is almost always one of two things - a `.malloy` file in the package missing the `##! experimental.persistence` flag (which aborts the *whole* package's build plan), or no build ever ran (a standalone Publisher does not build on publish - see **Building and refreshing**). Jump to **Debugging a no-op build**.
+> **The #1 gotcha, up front:** if a persist source isn't materializing, it is almost always one of four things - no build ever ran (a standalone Publisher does not build on publish - see **Building and refreshing**), a build ran and failed or lost that one source (a run can finish ready with `failures`; read them), the model declaring the source has no `##! experimental.persistence` flag and nothing flagged imports it (it is skipped without a warning), or the package was reloaded since the build (a reload drops the bindings until the next run). Jump to **Debugging a no-op build**.
 
-**Deciding what to persist, what to stop persisting, and how to schedule it** (making a package cheaper or faster): read `reference/tuning.md`. It reads the materialization history with the `malloy-pub` CLI and proposes changes; it recommends and does not edit.
+This file is the core. The details live beside it, one topic per file, and each is a prompt of its own:
+
+- `reference/annotation.md` - the `#@ persist` tag: what can carry it, the name rules, identity and reuse, readers of a persisted source, and every option with where it is explained.
+- `reference/policy.md` - the `publisher.json` block: `scope`, `queryMetadata`, and where the rules are enforced.
+- `reference/storage.md` - building into a **storage destination** (`storage=`), `partition=`, where a given may sit, which sources that tier refuses, and how a stored table is served.
+- `reference/chaining.md` - one stored source built **over another stored source**: what makes it reuse the parent's table or recompute from the warehouse, and how to read which happened.
+- `reference/refresh.md` - keeping a table current: `freshness`, `schedule`, and `refresh="incremental"`.
+- `reference/access-filter.md` - persisting a `#(access_filter)`-gated source, and the bound on a stale access decision.
+- `reference/tuning.md` - reading the materialization history to decide what to persist, stop persisting, or reschedule (recommendations only).
 
 ## The recipe (get this right and it just works)
 
-1. **`##! experimental.persistence` on EVERY `.malloy` file in the package** - not only the file that declares the persist source. Either form enables it:
+1. **`##! experimental.persistence` on every `.malloy` file that declares a persist source.** Either form enables it:
    - `##! experimental.persistence`, or
    - `##! experimental { access_modifiers, sql_functions, persistence }` (add `persistence` to the existing list).
 
-   **Why every file:** the build plan is computed by asking *every* `.malloy` file in the package for its persist sources, and that call **throws on any file whose model lacks the flag** (`Model must have ##! experimental.persistence`). One unflagged helper or import file, even one with no persist source of its own, aborts the whole package's build plan, so *every* persist source in the package drops out. This is the most common cause of a no-op build.
+   **Why it matters:** the build plan walks only the models that carry the flag, and skips every other file **without a warning**. The flag is a property of the model being *walked*, not of the file that declares the annotation: a flagged model builds every persist source its walk reaches, including ones declared in a flagless file it imports, and a flagless file's persist sources build only when some flagged model imports it. So a `#@ persist` in a flagless file that nothing flagged imports is silently never built, while adding a flagged entry model that imports it starts building tables that were inert - neither state warns. Put the flag in the declaring file itself: that is the one placement whose effect does not depend on what else imports it. A file with no persist source of its own does not need the flag; adding it anyway is harmless. (`#@ preaggregate` rollups are the one exception: they build from an unflagged file, because they live in a model Publisher synthesizes.)
 
 2. **`#@ persist name="..."` on a query-based source, with the name quoted:**
    ```malloy
    #@ persist name="my_dataset.my_table"
    source: my_rollup is some_source -> { group_by: ...; aggregate: ... }
    ```
-   - **Only `query_source` and `sql_select` sources are persistable** - a source whose definition has a `-> { ... }` pipeline or a `conn.sql("...")`. This **includes** one refined by a trailing `extend { ... }`. What is **not** persistable is a *plain* `extend` over a bare `conn.table(...)`; a `#@ persist` on such a source is **silently ignored** (its annotation is never read) - that one source just won't materialize, and the rest of the package still builds.
-   - **Quote the name.** `name="my_table"` (or a path `name="dataset.table"` / `name="project.dataset.table"`) is required. A **bare** `name=my_table` **always fails the build/publish** with `persist annotation name must be quoted` (a raw-source scan that hard-stops); it never silently no-ops.
-   - `name=` is the target table name. In a standalone Publisher this **is** the physical table (rebuilt in place); a hosted (control-plane) deployment builds it under a content-addressed generation name. In both, the source's identity for reuse is a content address of its connection and canonical SQL (its `sourceEntityId`), so **republishing unchanged persist logic reuses the existing table** and changing the logic builds fresh.
+   - **Only `query_source` and `sql_select` sources are persistable** - a `-> { ... }` pipeline or a `conn.sql("...")`, including one refined by a trailing `extend { ... }`. A `#@ persist` on a *plain* `extend` over a bare `conn.table(...)` is **refused, not ignored**: the package warns, and every run that covers it fails naming the source, so a whole-package run builds nothing until you fix it. Persist a query source instead (`source: x is raw -> { where: ...; select: * }`).
+   - **Quote the name, once.** A bare `name=my_table` fails the package load with `persist annotation name must be quoted`; a name quoted inside the quotes silently serves live. The target's container must already exist and be writable by the connection.
+   - **The name is not the identity.** A persist source is identified by a content address of its connection, SQL and partition layout, so a rebuild with unchanged logic reuses the table and two sources with the same body share one. An `extend` of a persisted source reads the same table and is not a second build target.
+
+   The name rules, bound arguments, `#@ -persist`, and every option (`storage=`, `partition=`, `refresh=`, `freshness.*`, `queryMetadata.*`) are in `reference/annotation.md`.
 
 3. **Package persistence policy in `publisher.json`** (all optional):
    ```jsonc
    {
      "name": "my-package",
+     "queryMetadata": { "team": "finance" },   // root level: tags every statement a build OR a query issues
      "materialization": {
-       "scope": "package",  // default; "version" = each published version owns its own tables
-       "freshness": { "window": "24h", "fallback": "live" },
-       "queryMetadata": { "team": "finance" }  // tags the build's backend statements
+       "scope": "package",                      // default; "version" = each published version owns its own tables
+       "freshness": { "window": "24h", "fallback": "live" }
      }
    }
    ```
-   Enforced at publish (strict), on edits (strict), at load (warn, still serves), and by the scheduler (an offending package is skipped):
-   - **`scope`**: `package` (default; artifacts reused across published versions) or `version` (each artifact owned by one version). Package-level only; there is no per-source scope. A root-level `scope` is the deprecated home and still works, with a warning; declaring both homes with different values is rejected.
-   - **`materialization.freshness`** (`window` + `fallback` of `live`/`stale_ok`/`fail`) is the objective a **hosted control plane** enforces by refreshing the table to meet it (`fallback: "live"` serves live compute while stale/absent). A **standalone** Publisher does **not** act on `freshness` for refresh - see **Building and refreshing**.
-   - **`materialization.queryMetadata`** is a bag of string properties attached to every statement the build issues, for the backend's own cost attribution (Snowflake `QUERY_TAG`, BigQuery job labels, a leading SQL comment elsewhere). Overridable per source with `#@ persist queryMetadata.<name>="<value>"`. Observability only: it never changes what gets built. See `docs/query-metadata.md`.
-   - **`materialization.schedule`** is a 5-field UTC cron (`min hour dom mon dow`; `L`/`W`/`#`/`?` rejected). It **requires `scope: "version"`** and is **mutually exclusive with `freshness`**. This is how a standalone Publisher refreshes on a cadence.
+   - **`scope`** is a package-level contract about sharing tables across versions; on a standalone server it changes nothing except gating `schedule`. Its deprecated root-level home still works, but the two homes disagreeing **fails the load**.
+   - **`freshness`** is the objective a **hosted control plane** enforces by refreshing the table; a **standalone** Publisher does not act on it at all. `fail` behaves like `live` today; only `stale_ok` keeps serving a stale table.
+   - **`queryMetadata`** tags every build statement and every query for the backend's cost attribution. Root level; the home inside `materialization` is deprecated. Observability only.
+   - **`schedule`** is a 5-field UTC cron that **requires `scope: "version"`** and **excludes `freshness`**; it fires only on a standalone server whose scheduler is enabled.
 
-4. **Reads vs writes.** The persist source can *read* any dataset the connection can read; the persist *target* (`name=`'s dataset) must be a dataset the connection can **write** (typically a scratch dataset).
+   The coherence rules are rejected at publish and at a policy edit, warned at load, and skipped by the scheduler. `reference/policy.md` has `scope` and `queryMetadata` in full; `reference/refresh.md` has `freshness` and `schedule`.
 
 ## Building and refreshing (standalone vs. hosted)
 
 A `#@ persist` tag declares *what* to materialize; it does not by itself build anything.
 
-- **Standalone Publisher:** publishing or loading a package only computes its build plan - **no table is built until a materialization run executes.** Trigger one explicitly (`malloy-pub materialize --package <pkg> --wait`, or the materialization API), or turn on the opt-in local scheduler (off unless `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER` is set) to fire the package's `schedule` cron. Refresh is a re-run or that cron; `freshness` is not a refresh trigger here, so a freshness-only standalone package builds once and is not auto-refreshed.
+- **Standalone Publisher:** publishing or loading a package only computes its build plan - **no table is built until a materialization run executes.** Trigger one explicitly (`malloy-pub materialize --environment <env> --package <pkg> --wait`, or the materialization API), or turn on the opt-in local scheduler (off unless `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER` is set) to fire the package's `schedule` cron. Refresh is a re-run or that cron; `freshness` is not a refresh trigger here, so a freshness-only standalone package builds once and is not auto-refreshed. Two flags to know: `--force-refresh` builds every source even when its content address is unchanged (which is what every scheduler fire does), and `--reseed` is the only thing that rebuilds an incremental source from scratch. `--wait` gives up after **120 s by default** (`--timeout <s>`) and exits non-zero while the build keeps running; `malloy-pub get materialization <id>` then shows where it got to.
 - **Hosted (control-plane) deployment:** the build runs automatically on publish, best-effort - a build failure does **not** fail the publish (which is why a broken persist can look like a silent no-op), and the control plane drives refresh to meet the `freshness` objective.
 
-Either way, a successful publish alone does not prove a table exists - confirm the build separately.
+Two mechanics either way. **One build at a time per package**: a second run requested while one is active is rejected with a 409 conflict rather than queued, and the scheduler skips a fire while a run is active. **A build can be partial**: `sourceNames` on the request builds only the sources it names and leaves the others exactly as they were (live if never built), which is how to add one source to a package without rebuilding the rest.
 
-## Serve-time routing is `query_source`-only (today)
+Either way, a successful publish alone does not prove a table exists - confirm the build separately.
 
-Both persistable types *build* a table, but only a **`query_source`** (a `-> { ... }` pipeline) is rewritten to *read* it at query time. A raw **`sql_select`** (`conn.sql("...")`, including `conn.sql("...") extend { ... }`) builds its table and then the query path re-inlines its SQL, so the table is built and never read, and queries are no faster. If you have raw SQL you want served from a table, wrap it in a thin `query_source` and persist that:
+## Confirming it worked
 
-```malloy
-source: x_raw is my_conn.sql("select ...")
-#@ persist name="scratch_dataset.x"
-source: x is x_raw -> { select: * }
-```
+A green run is not proof, and neither is a fast query (on small data a live recompute is fast too). Check what the server says it serves:
 
-## Confirming it worked
+1. **The run lists your source, and did not lose it.** A run's detail (`malloy-pub get materialization <id> --environment <env> --package <pkg>`) carries `manifest.entries`, a map keyed by `sourceEntityId` whose values name each built or reused source (`sourceName`) and its physical table. A run can finish `MANIFEST_FILE_READY` having built nothing (every source skipped for a missing flag, or carried forward unchanged) **or having lost a source**: a source that failed is absent from `entries` and present in `manifest.failures` with its error, and a source the eligibility gate refused is in `metadata.refusedSources`, while the run still reads ready. `metadata.sourcesBuilt` / `sourcesReused` / `sourcesFailed` / `sourcesRefused` are the counts. For a source built over another stored source, the entry's `upstreamReuse` says whether it read the parent's table (`reused`) or recomputed it (`recomputed`, with `upstreamRecomputeReason`) - see `reference/chaining.md`.
+2. **The package is bound to those tables.** On a standalone Publisher the package's details report `manifestBindingStatus` and `manifestEntryCount`. The count covers tables built in the source's own warehouse only: it reads `bound` with a nonzero count when such a table is bound and `unbound` otherwise. A `storage=` table never counts there - it appears under the package's `storageServeBindings`, so a package whose only persist sources are `storage=` reads `unbound` while serving from the store. `live_fallback` is a hosted-only status: a configured `manifestLocation` could not be fetched or bound, and the previously bound manifest (if any) stays in force.
+3. **The query names the table.** A query's full result (the REST query response without `compactJson`) carries the SQL it ran. Served from the table, its `FROM` names the physical table; computed live, it carries the source's own SQL instead. This is the per-query proof.
+4. **A reload unbinds them.** Reloading the package - including the automatic reload when a watched file is saved - drops the binding, and queries compute live until the next materialization run, which reuses unchanged tables and binds them again. (A server *restart* re-establishes the bindings from its store; a reload does not.) After editing a model, run a build before you trust a timing.
 
-After a build runs, re-run one of the source's queries - a persisted `query_source` should return quickly, reading the pre-built table instead of recomputing the upstream. Your host also reports each persisted source as **ready** with its physical table name (a materialization run detail, CLI listing, or materialization view, depending on the host); if nothing is listed, either no build ran (standalone) or the build plan was empty - see **Debugging a no-op build**.
+A query response's `servedFrom` field does not answer this question for a table built in the source's own connection: it reports only the `storage=` tier's outcome (`storage` when that tier answered, `live_fallback` when it degraded to a live recompute) and is empty otherwise, served from a colocated table or not. Do not read `live_fallback` here as a signal about a colocated table.
 
 ## Debugging a no-op build
 
 Symptom: no table was built and the source still recomputes on every query. Check, in order:
 
-0. **Did a build actually run?** On a standalone Publisher, publish/load does **not** build - run `malloy-pub materialize` (or enable the scheduler). "Publishes fine, no table" is the *expected* standalone state, not a model bug. On a hosted deployment the build is automatic but best-effort, so a failure is silent - look for a `FAILED` run.
-1. **A `.malloy` file missing the persistence flag** (the most common real bug). Every model file's `##!` line needs `persistence`, including pure helper/import files with no persist source - one unflagged file aborts the whole package's build plan.
-2. **An unquoted persist name** - a bare `name=foo` **always** hard-stops the build/publish with `persist annotation name must be quoted`; use `name="foo"`. (If you got *no* error at all, it isn't this.)
-3. **A `#@ persist` on a non-persistable source** - a bare `extend` over `conn.table(...)` is silently ignored, so *that* source won't materialize (the rest of the package is unaffected). Tag a `query_source` / `sql_select` instead.
-4. **A persisted raw `sql_select` that builds but is never read** - if the table exists yet queries are no faster, it's the serve-routing gap above; wrap the `sql_select` in a `query_source`.
+0. **Did a build actually run?** On a standalone Publisher, publish/load does **not** build - run `malloy-pub materialize` (or enable the scheduler). "Publishes fine, no table" is the *expected* standalone state, not a model bug. On a hosted deployment the build is automatic but best-effort, so a failure is silent - look for a `FAILED` run, and for a ready run with `failures`.
+1. **Did the run fail, or lose this source?** A `FAILED` run's `error` names what stopped it. A ready run's `manifest.failures` names each source that did not build, and `metadata.refusedSources` each one the gate refused, with the reason to act on. The package's details flag some of these before any run: an unbuildable persist source appears in its warnings, and its `buildPlan` lists the sources a run will build and the ones it will refuse. A `#@ persist` on a non-persistable source fails every run that covers it, naming the source.
+2. **The declaring model is missing the persistence flag** and nothing flagged imports it: skipped without any warning, the run succeeds, the source is simply absent from what was built. Add the flag to the declaring file.
+3. **An unquoted persist name** - a bare `name=foo` **always** stops the load with `persist annotation name must be quoted`. (If you got *no* error at all, it isn't this.)
+4. **The package was reloaded** after the build (a saved file under watch mode counts). The tables exist and the run reads ready; the binding is gone. Run a build. And if a watched edit failed to compile, the server keeps serving the **old** model and its binding, and records the error under `staleCompileErrors` on `/status` - so "still bound" after an edit can mean the edit never took.
+5. **A `storage=` source serves live while the store is switched off.** With `PERSIST_STORAGE_MODE=off` a `storage=` source is skipped by the build and served live, with no error; it is never built into the warehouse instead. See `reference/storage.md`.
 
 **Isolation test** - add a trivial, self-contained persist source in its own file and rebuild:
 ```malloy
@@ -92,89 +101,18 @@ source: smoke_raw is my_conn.table('some_dataset.some_table')
 #@ persist name="scratch_dataset.persist_smoke_test"
 source: persist_smoke is smoke_raw -> { aggregate: n is count() }
 ```
-- If **even this** doesn't build (after a real materialization run), the whole package's plan is aborting - a sibling `.malloy` file is missing the flag. Fix rule 1 across the package.
-- If the smoke source **does** build but your real one doesn't, your real source is the problem - a non-persistable type (a bare `extend`), or its own file's flag.
-
-Delete the smoke file and drop its table afterward.
-
-## Persisting a `#(access_filter)`-gated source
-
-A gated source **can** be persisted, but only on one tier and only in one shape, and the thing to be
-careful about is not refused by anything - you have to decide it.
-
-- **`storage=` and `#@ preaggregate` always refuse a gated source**, naming it. The build skips a refused
-  source, records it on the run (`metadata.refusedSources`), and builds the rest of the package. A run fails
-  on a refusal only when every authored source it targeted was refused, or when `sourceNames` named this one;
-  a refused rollup never fails it. (`#@ persist storage=<name>` is the tier that materializes into a separate
-  registered storage destination and serves from there, rather than building in the source's own connection;
-  `#@ preaggregate` stores a rollup Publisher derives from a measure you annotated with a grain, rather than a
-  source you wrote.) A rollup also groups *across* the gated column, so it could not be row-filtered afterwards
-  even in principle.
-- **A colocated `#@ persist` (no `storage=`) is admitted** when the gate is provably the entry point's
-  **own row filter**. It is refused when the gate is reached only through a join, inherited from a base
-  the compiler cannot attribute cleanly, or does not classify as a row filter at all. The gate is found
-  through the import -> rename -> `query_source` chain, so a gate the persisted source did not declare
-  itself still counts.
-
-**What to be wary of.** Persisting does not weaken the gate: it changes only where rows are read FROM, and
-the gate still runs live on every query as that query's own `WHERE`, so filtered rows come back filtered.
-What freezes is the **column the gate filters on**. A row whose access decision changes - it changes
-owner, say - keeps being served under its OLD decision until the next rebuild. That is a stale *access
-decision*, not merely stale data, and nothing raises an error.
-
-**None of this is needed for the gate to work.** It is enforced live on every query either way; what
-needs a bound is how long a *stale* decision can survive. Of the three controls that look like that
-bound, only the first is:
-
-- **`materialization.freshness` `{ "window": "24h", "fallback": "live" }` is the bound.** The serve path
-  re-checks freshness per query, so once the artifact ages past the window it drops out of the serving set
-  and the query recomputes live, correctly filtered - whether or not a rebuild ever lands. Three details
-  decide whether you actually get that. **`fallback` must be `live`**: under `stale_ok` a stale artifact
-  keeps being served, which voids the bound, and window and fallback resolve *independently* per layer,
-  so a package-level `stale_ok` silently defeats a window you set on the source. That is a statement about
-  **layers**, which do not combine - not about siblings, below. Prefer the
-  **per-source** spelling `#@ persist name="..." freshness.window="24h" freshness.fallback="live"` over
-  the package-wide `materialization.freshness` key: the gated source is what needs the bound, and setting
-  it package-wide forces every other persisted source to recompute once stale too. And **a
-  content-identical sibling shares the artifact, so it shares the window**: reuse is keyed on the
-  content-addressed `sourceEntityId`, which folds the connection and the SQL but *not* the source name, so
-  two persist sources whose bodies compute the same SQL resolve to one table carrying one freshness
-  policy. The tightest window any of them declares governs all of them - a sibling declaring nothing
-  cannot loosen yours, and yours pulls that sibling's reads off the table once it lapses. A sibling's
-  `stale_ok` cannot void your bound either: the fold keeps whichever fallback bounds staleness, so the
-  layer rule above does not carry over here. If two sources need genuinely different windows, give them
-  genuinely different SQL.
-
-  Both of those are properties of the **host** that assembles the manifest, not of the annotation. Where
-  the host does not fold, which sibling's policy reaches the wire is unspecified; and a host that folds at
-  manifest-assembly time typically applies it when a version's manifest is next published rather than
-  retroactively to manifests already distributed - so you can declare the window correctly and not have it
-  in force yet.
-- **A cron alone is not a bound.** A failed build or a stopped scheduler leaves the source serving its old
-  decisions indefinitely. `freshness` and `schedule` are mutually exclusive; for a gated source, take the
-  window.
-- **`refresh="incremental"` does not bound revocation.** The delta only re-reads rows in
-  `[covered_through, frontier)`, so a row that changes owner *without its watermark advancing* is never
-  re-read - while the entry still reports an advancing `coveredThrough` and reads as healthy. Only a full
-  rebuild recomputes the gating column.
-
-**And the window only binds where the serving manifest carries it.** Freshness is enforced from fields a
-control plane stamps onto the manifest it distributes; a Publisher that serves what it just built binds the
-table with no `dataAsOf` and no window, and an entry carrying no window never ages out. So on a standalone
-deployment the declared window is inert and the artifact serves until the next full rebuild - which leaves a
-rebuild cadence you actually verify as the only bound, and makes leaving a revocation-sensitive source
-unpersisted the safer call.
-
-When recommending `#@ persist` on a gated source, pair it with a freshness window and say out loud what
-staleness the author is accepting. A gated source with neither a window nor a full-rebuild cadence has no
-bound on how long a revoked row keeps being served.
+If **even this** doesn't build after a real run, the problem is package-wide: read the run's error (a non-persistable source anywhere fails the whole run and names it), then check that the connection can write the target dataset. If the smoke source builds and your real one doesn't, your real source is the problem: its file's flag, or its entry in `failures` / `refusedSources`. Delete the smoke file and drop its table afterward (`malloy-pub delete materialization <id> --drop-tables`).
 
 ## Gotchas
 
-- **Every `.malloy` file needs the persistence flag** - one unflagged file aborts the whole package's build plan. (A `#@ persist` on a *non*-persistable source, by contrast, is silently ignored and does not affect other sources.)
-- **A tag doesn't build** - a standalone Publisher materializes only on an explicit run or its scheduler; only a hosted control plane builds on publish.
-- **Serve-time routing is `query_source`-only** - a raw `sql_select` builds a table the query path doesn't read; wrap it in a `query_source`.
-- **Quote the name** - a bare `name=` always hard-stops the build.
-- **Republishing unchanged persist logic reuses the table** - reuse is keyed on the content-addressed `sourceEntityId`, not the `name=`.
-- **Removing a persist source (or a smoke test) does not drop its table** - physical-table cleanup is the caller's responsibility; drop it yourself.
-- **A `#(access_filter)`-gated source freezes its gating column when persisted** - the gate still runs live, but a revoked row keeps being served under its old access decision until the next rebuild. `storage=` and `#@ preaggregate` refuse a gated source outright. See **Persisting a `#(access_filter)`-gated source**.
+- **Flag every file that declares a persist source** - a flagless model nothing flagged imports is skipped without a warning.
+- **A `#@ persist` on a plain `extend` over `conn.table(...)` fails the run** - every run that covers it, naming the source.
+- **A tag doesn't build** - only an explicit run, the standalone scheduler, or a hosted control plane does.
+- **Quote the name, once** - bare stops the load; double-quoted silently serves live.
+- **A ready run can still have lost a source** - read `manifest.failures` and `metadata.refusedSources`.
+- **A reload unbinds; a restart rebinds** - run a build before trusting a timing.
+- **Reuse is by content address, not by name** - `--force-refresh` and every scheduler fire defeat it.
+- **Removing a persist source does not drop its table** - `delete materialization --drop-tables` drops what no other ready run names, and serving reverts to live in place.
+- **A given inside the persisted query is refused; in the source's own `extend` block it is applied per caller** - `reference/storage.md`.
+- **A join to a non-persisted source, or a nested column, serves live for the queries that touch it** - the source's own fields still serve from the table.
+- **A `#(access_filter)`-gated source freezes its gating column when persisted** - `storage=` and `#@ preaggregate` refuse it outright; `reference/access-filter.md`.

package/skills/malloy-materialization/reference/access-filter.md

@@ -0,0 +1,25 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# Persisting a `#(access_filter)`-gated source
+
+A gated source **can** be persisted, but only on one tier and only in one shape, and the thing to be careful about is not refused by anything - you have to decide it.
+
+- **`storage=` and `#@ preaggregate` always refuse a gated source**, naming it. The build skips a refused source, records it on the run (`metadata.refusedSources`), and builds the rest of the package. A run fails on a refusal only when every authored source it targeted was refused, or when `sourceNames` named this one; a refused rollup never fails it. (`#@ persist storage=<name>` is the tier that materializes into a separate registered storage destination and serves from there, rather than building in the source's own connection; `#@ preaggregate` stores a rollup Publisher derives from a measure you annotated with a grain, rather than a source you wrote.) A rollup also groups *across* the gated column, so it could not be row-filtered afterwards even in principle.
+- **A colocated `#@ persist` (no `storage=`) is admitted** when the gate is provably the entry point's **own row filter**. The gate is found through the import -> rename -> `query_source` -> `extend` chain, so a gate the persisted source did not declare itself still counts, and an inherited gate is admitted like a declared one. It is refused in exactly two cases: the gate is reached only through a join, or it does not classify as a row filter at all.
+
+**What to be wary of.** Persisting does not weaken the gate: it changes only where rows are read FROM, and the gate still runs live on every query as that query's own `WHERE`, so filtered rows come back filtered. What freezes is the **column the gate filters on**. A row whose access decision changes - it changes owner, say - keeps being served under its OLD decision until the next rebuild. That is a stale *access decision*, not merely stale data, and nothing raises an error.
+
+**None of this is needed for the gate to work.** It is enforced live on every query either way; what needs a bound is how long a *stale* decision can survive. Of the three controls that look like that bound, only the first is:
+
+- **`freshness.window="24h"` with `freshness.fallback="live"` is the bound** (`reference/refresh.md`). The serve path re-checks freshness per query, so once the artifact ages past the window it drops out of the serving set and the query recomputes live, correctly filtered - whether or not a rebuild ever lands. Three details decide whether you actually get that. **`fallback` must not be `stale_ok`**: under `stale_ok` a stale artifact keeps being served, which voids the bound (`live` and `fail` both bound it today). Window and fallback resolve *independently, per field*, across the source, model-file and package layers, so a package-level `stale_ok` silently defeats a window you set on the source. Prefer the **per-source** spelling `#@ persist name="..." freshness.window="24h" freshness.fallback="live"` over the package-wide `materialization.freshness` key: the gated source is what needs the bound, and setting it package-wide forces every other persisted source to recompute once stale too. And **a content-identical sibling shares the artifact, so it shares the window**: reuse is keyed on the content-addressed `sourceEntityId`, which folds the connection and the SQL but *not* the source name, so two persist sources whose bodies compute the same SQL resolve to one table carrying one freshness policy. A host that folds the siblings' policies keeps the tightest window any of them declares - a sibling declaring nothing cannot loosen yours, and yours pulls that sibling's reads off the table once it lapses - and keeps the bounding fallback, so a sibling's `stale_ok` cannot void your bound either. If two sources need genuinely different windows, give them genuinely different SQL.
+
+  Both of those are properties of the **host** that assembles the manifest, not of the annotation: the publisher reports freshness per source and does no folding itself. Where the host does not fold, which sibling's policy reaches the wire is unspecified; a host that folds does so when it next publishes a version's manifest, not retroactively to manifests already distributed - so you can declare the window correctly and not have it in force yet.
+- **A cron alone is not a bound.** A failed build or a stopped scheduler leaves the source serving its old decisions indefinitely. `freshness` and `schedule` are mutually exclusive; for a gated source, take the window.
+- **`refresh="incremental"` does not bound revocation.** The delta only re-reads rows in `[covered_through, frontier)`, so a row that changes owner *without its watermark advancing* is never re-read - while the entry still reports an advancing `coveredThrough` and reads as healthy. Only a full rebuild recomputes the gating column.
+
+**And the window only binds where the serving manifest carries it.** Freshness is enforced from fields a control plane stamps onto the manifest it distributes; a Publisher that serves what it just built binds the table with no `dataAsOf` and no window, and an entry carrying no window never ages out. So on a standalone deployment the declared window is inert unless something binds the manifest through the API with those fields - which leaves a rebuild cadence you actually verify as the only bound, and makes leaving a revocation-sensitive source unpersisted the safer call.
+
+When recommending `#@ persist` on a gated source, pair it with a freshness window and say out loud what staleness the author is accepting. A gated source with neither a window nor a full-rebuild cadence has no bound on how long a revoked row keeps being served.

package/skills/malloy-materialization/reference/annotation.md

@@ -0,0 +1,56 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# The `#@ persist` annotation
+
+```malloy
+#@ persist name="my_dataset.my_table"
+source: my_rollup is some_source -> { group_by: ...; aggregate: ... }
+```
+
+## What can carry it
+
+**Only `query_source` and `sql_select` sources are persistable** - a source whose definition has a `-> { ... }` pipeline or a `conn.sql("...")`, including one refined by a trailing `extend { ... }`. What is **not** persistable is a *plain* `extend` over a bare `conn.table(...)` (a filtered pass-through). A `#@ persist` on such a source is **refused, not ignored**: the package carries a warning naming the source, and any materialization run that covers it fails with `annotated '#@ persist' but were not recognized as a materializable source`, so a whole-package run builds nothing until you fix it. Persist a query source instead (`source: x is raw -> { where: ...; select: * }`), or remove the annotation.
+
+**A parameterized source must have its argument bound.** `#@ persist` on `high_value(min_amount is 500) -> { ... }` builds one fixed relation; on the free template `high_value -> { ... }` it is refused (`free_parameter`), since there is no single relation to store. A `given` the persisted *query* reads is refused the same way (`given_in_persisted_query`); a given in the source's own `extend` block is applied per caller at read. `reference/storage.md` says where a given may sit.
+
+## The name
+
+- **Quote it, once.** `name="my_table"`, or a path `name="dataset.table"` / `name="project.dataset.table"`. A **bare** `name=my_table` fails the package **load** - so every publish and every restart - with `persist annotation name must be quoted`; it never silently no-ops.
+- **Plain identifier segments only**: dot-separated segments of letters, digits, underscores and hyphens. A quoted name holding a space, quote, semicolon or backtick is rejected with its own message, because the name is inlined into DDL.
+- **Never quote inside the quotes.** `name="\"Quoted Tbl\""` reaches the build with literal quote characters: the table is created with them in its name, the serve side looks for the unquoted one, and the source silently serves live after a wasted build.
+- **One name per table.** Two distinct persist sources that resolve to the same `name=` in the same destination clobber each other (the second build overwrites the first, and dropping one drops the other's). Publisher warns on the package at load and, where `PERSIST_COLLISION_ENFORCE` is set, rejects the publish. A shared *body* is fine (it shares one table by content address); a shared name over different bodies is not.
+- **The target's container must exist.** `name="analytics.daily"` puts the table in `analytics`, which the connection must be able to write and which Publisher does not create. The source can *read* anything the connection can read.
+- **The name is not the identity.** In a standalone Publisher `name=` is the physical table, rebuilt in place; a hosted deployment may assign the physical name itself. A `storage=` destination takes a single undotted name, since placement there is the destination's.
+
+## Identity and reuse
+
+A persist source's identity is its **`sourceEntityId`**: a content address of its connection, its canonical SQL and its `partition=` layout. Not the name, not the file, not the package version. So:
+
+- **A rebuild with unchanged persist logic reuses the existing table** (skip-if-unchanged); changing the SQL or the partition layout builds fresh under a new address. `--force-refresh` and every scheduler fire defeat the skip; an incremental source is exempt from it and advances by its boundary instead (`reference/refresh.md`).
+- **Two sources whose bodies compile to the same SQL share one table**, however they are named, and share one freshness policy and one incremental boundary with it.
+- **A republish by itself builds nothing on a standalone server**; reuse is a property of the next *run*.
+
+## Readers of a persisted source
+
+- **An `extend` of a persisted source reads the stored table** and is not a build target of its own: it inherits the parent's tag and the same address, so the two names are one table. To make a distinct non-persisted intermediate, pipe first: `base -> { select: * } extend { ... }`.
+- **A query over a persisted source** (`orders is _orders_fact -> { select: * }`, the private-fact / public-wrapper idiom) reads the table too, with the fact's `extend`-block filters applied per caller, and is not a build target.
+- **`#@ -persist`** on an extension opts it out: it recomputes from raw on every query and never reads the table. Reach for it when a reader must not see stale rows, and remember it forgoes exactly the work persistence was there to save.
+- Which queries over a reader route to the table is decided per query - a join to a non-persisted source or a nested column serves live for the queries that touch it (`reference/storage.md`).
+
+## Every option
+
+| On the tag | What it does | Where it is explained |
+|---|---|---|
+| `name="..."` | the target table (required) | above |
+| `storage=<destination>` | build into a storage destination and serve from it | `reference/storage.md` |
+| `partition="a,b"` | lay the stored table out by those columns (`storage=` only) | `reference/storage.md` |
+| `refresh="incremental" watermark="col" [merge_key="a,b"]` | refresh by bounded delta instead of a full rebuild | `reference/refresh.md` |
+| `freshness.window="24h" freshness.fallback="live"` | per-source freshness objective (hosted deployments) | `reference/refresh.md` |
+| `#@ queryMetadata.<key>="v"` (sibling line) | per-source cost-attribution properties | `reference/policy.md` |
+| `#@ -persist` (on a reader) | opt a reader out of the table | above |
+| `#@ preaggregate grain="..."` (on a measure) | store a derived rollup | `reference/storage.md` |
+
+An unrecognized key on the tag is a warning on the package. The retired per-source `sharing=` and `schedule=` are rejected at publish: scope and schedule are declared once for the package (`reference/policy.md`).

package/skills/malloy-materialization/reference/chaining.md

@@ -0,0 +1,48 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# Chaining: one stored source built over another
+
+A `storage=` source may read another `storage=` source. When both land in the same destination, the downstream is built by **reading the upstream's stored table** in the destination's own engine - the upstream is never re-scanned from the warehouse - and the chain is consistent by construction: the child is a pure function of the parent's stored rows. Chaining is what turns a package of independent rebuilds into one that touches the warehouse only for its roots.
+
+```malloy
+#@ persist name="daily_orders" storage=lake
+source: daily_orders is orders -> { group_by: order_date; aggregate: total_amount is amount.sum() }
+
+#@ persist name="monthly_orders" storage=lake
+source: monthly_orders is daily_orders -> { group_by: order_month is order_date.month; aggregate: monthly_total is total_amount.sum() }
+```
+
+The downstream need not name the upstream directly. Non-persisted sources between them - a `select: *` wrapper, an `extend` that adds a dimension, a named source over an inline `(parent extend { ... }) -> { ... }`, a query over the upstream, a parent reached only through an `import` - are carried into the build in dependency order, under the `##!` flags of the files that declare them, and declaring the author model's `given:`s, so the downstream still reads the upstream's table through them.
+
+## The four rules that decide reuse
+
+1. **Declare the stored roots before the sources that read them.** Malloy resolves names in file order. A rollup rewired to read a stored root that is declared further down fails to compile with `Reference to undefined object`, and so does everything after it. When you chain an existing package onto its roots, move the roots up.
+2. **An extension of a stored source is the same table, not a new one.** `rollup_base is daily_orders extend { ... }` inherits the parent's persist tag and the same content address, so the publisher treats the two names as one build target. To make a distinct non-persisted intermediate, pipe through a query first: `rollup_base is daily_orders -> { select: * } extend { ... }`. (A host that checks persist names for collisions can refuse the bare `extend` outright, since two names now point at one table.)
+3. **Every join on the path must land on a stored table.** An intermediate that joins a source nothing materializes - a `join_cross: sp is species_live` where only `species_base` is stored - reaches the warehouse, and the whole downstream is recomputed from the warehouse with the upstream inlined. Point the join at the stored sibling instead.
+4. **A chained build runs in the destination's engine, under its memory bound.** The rows are read from the lake and rolled up in DuckDB with the build session's `memory_limit`. A grouped rollup fits; a self-join or a wide hash aggregate over a large fact may not, and fails with DuckDB's `Out of Memory Error` on every attempt. Such a source belongs in the warehouse: keep its input unpersisted so it is built there and only its result is copied in. Placement is a cost decision the build does not make for you.
+
+## What the build does when it cannot reuse
+
+If the downstream cannot be built over the stored tables - it reaches the warehouse through an intermediate (rule 3), or reads a field on the parent that is not a stored column - the publisher **recomputes the upstream from the warehouse**, inlining it, and the build succeeds. Under `strictUpstreams` (what an orchestrating host sets on every build) this is the **one** recompute strict permits, because no build over the stored tables exists for that shape. Strict still refuses what it exists to refuse: a persisted upstream the build neither materialized nor was handed by reference, one whose table lives in a *different* destination, and a source over stored upstreams that the build could not carry (a refinement the ladder below drops). A refusal names the upstream and where it lives.
+
+A chained build re-declares on each parent's binding what the parent's `extend` block adds - its `where:`, dimensions, measures, joins and views - and thins them when they do not compile in the destination, in tiers: a view first, then a join, then a dimension or measure that reads a join the destination cannot bind. **The `where:` is never thinned**: a stored source declared `extend { where: total > 100 }` holds every row and is filtered when read, so a child built over it without that filter would sum the unfiltered rows.
+
+## Reading which happened
+
+Every manifest entry for a source that reads a persisted upstream carries:
+
+- `upstreamReuse: "reused"` - every persisted upstream was read from its table.
+- `upstreamReuse: "recomputed"` - at least one was recomputed from its definition; `upstreamRecomputeReason` names which and why (for example, which intermediate reached the warehouse through which join).
+
+It is reported on both tiers, since a table built in the source's own warehouse also inlines a `storage=` upstream it cannot read there. The field is not a location and never names a table. A recomputed source's rows reflect the warehouse at *that build's* time rather than the upstream's snapshot, so the two can disagree when their refresh cadences differ - this field is how to tell.
+
+The `publisher_storage_chained_build_total{outcome}` counter reports the path each chained build took: `parent_reuse`, `inline_fallback` (non-strict recompute), `strict_shape_fallback` (the strict-permitted recompute), `strict_refused`, `infra_failure` (the destination was unreachable or the build died in it - kept distinct so a store outage or an out-of-memory is not read as an un-carriable shape).
+
+## Three limits to know
+
+- A refresh of a chained `refresh="incremental"` source always rebuilds in full (`chained_storage` reason): its parent's delta can restate rows below the child's own frontier where no delta of the child's would revisit them.
+- A `#@ persist` without `storage=` (built in the source's warehouse) never substitutes a `storage=` upstream's table, because the warehouse cannot see the lake; it inlines the upstream and reports `recomputed`.
+- A chained build carries the intermediates' *text*, written for the source warehouse, into the destination's dialect. A construct the destination lacks fails the carry, and under strict that is a refusal rather than a recompute.

package/skills/malloy-materialization/reference/policy.md

@@ -0,0 +1,52 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# The package policy: `scope`, `queryMetadata`, and where the rules are enforced
+
+```jsonc
+{
+  "name": "my-package",
+  "queryMetadata": { "team": "finance" },   // root level
+  "materialization": {
+    "scope": "package",                      // default; or "version"
+    "freshness": { "window": "24h", "fallback": "live" },
+    "schedule": "0 6 * * *"                  // never together with freshness
+  }
+}
+```
+
+`freshness` and `schedule` are explained in `reference/refresh.md`. This file covers the rest of the block and the gate that checks it.
+
+## `scope`
+
+`package` (default) or `version`. It is a package-level contract about whether a persisted table may be shared across published versions: under `package` an unchanged source's table is reused by every version that computes the same SQL; under `version` each published version owns its own tables. There is no per-source scope and no per-source schedule; both are declared once for the package.
+
+Two things to know about it on a standalone server:
+
+- **It changes nothing there except gating `schedule`.** A standalone auto-run keys tables by content address regardless of scope; true per-version tables are produced when a hosted control plane assigns versioned build targets. Standalone is the place to exercise the *policy* rules, not per-version fan-out.
+- **A root-level `scope` is the deprecated home** and still works, with a warning. Whenever the server rewrites the manifest (any package PATCH, including a description-only one) it writes **both** homes with the same value, so the root key reappears after an API edit. **The two homes holding different values is not a warning: the package fails to load**, disappears from the server, and says so only in `/status` `loadErrors`. Editing by hand, change `materialization.scope` and delete the root copy; never edit the root one.
+
+## `queryMetadata`
+
+A bag of up to 20 string properties attached to **every statement a build issues and every query against the source**, for the backend's own cost attribution: Snowflake `QUERY_TAG`, BigQuery job labels, a leading SQL comment elsewhere. Publisher adds its own context on top (`class=materialize` with the package, source, trigger and run id on a build; `class=interactive` on a served query), and drops a model-side declaration that reuses one of those server-owned names.
+
+- **Declare it at the root of `publisher.json`.** The older home inside `materialization` still works but emits a deprecation warning on every load and publish.
+- **Three layers, most specific wins:** a sibling line on the source (`#@ queryMetadata.team="risk"` above the `source:`), a model-file line (`## queryMetadata.team="risk"`), then the package block.
+- **Observability only.** It never changes what gets built, is not part of the content address, and problems with it (a bad key, an over-long value, the deprecated home) are warnings, never rejections.
+
+See `docs/query-metadata.md` for the per-backend rendering.
+
+## Where the rules are enforced
+
+The scope / schedule / freshness coherence rules - a single package-level scope, `schedule` requires `scope: "version"`, `schedule` and `freshness` (including a per-source `freshness.*`) are mutually exclusive, a valid 5-field UTC cron - are checked in four places:
+
+| Where | Outcome |
+|---|---|
+| publish | rejected (400) |
+| a package edit that touches the policy | rejected (400); an edit that does not touch it is not re-checked |
+| package load | warned; the package still serves |
+| the scheduler | an offending package is skipped |
+
+Only those coherence rules are gates. `queryMetadata` findings, the deprecated homes, and an unrecognized `#@ persist` key are advisory. An invalid `refresh="incremental"` declaration is the other way round: it **fails the load** outright (`reference/refresh.md`). The `malloy-pub schedule` commands share the publish gate, so a rejection from them means the change was unsafe - and `schedule set` replaces the whole `materialization` block, dropping a freshness policy it finds there.

package/skills/malloy-materialization/reference/refresh.md

@@ -0,0 +1,53 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# Keeping a table current: freshness, schedule, incremental refresh
+
+A persisted table is a snapshot. Three declarations govern when it is rebuilt and what a reader gets in the meantime; which of them does anything depends on who drives the builds.
+
+## `freshness`: the objective tier
+
+```malloy
+#@ persist name="daily_orders" freshness.window="24h" freshness.fallback="live"
+```
+
+or package-wide in `publisher.json` under `materialization.freshness`, or per model file with `## materialization.freshness.window="24h"`. The three layers resolve **per field** - source, then model file, then package - so a source that declares only `window` takes its `fallback` from the package. That is how a package-level `stale_ok` silently defeats a window set on one source.
+
+- `window` is how old a table may be before it is stale. `fallback` says what a query over a stale table does: `live` and `fail` both skip the table and compute live (`fail` is reserved to error later; today it behaves as `live`); `stale_ok` keeps serving the stale table.
+- A **hosted control plane** enforces the objective: it refreshes the table to stay within the window and stamps each manifest entry with `dataAsOf` and the window, and the serve path re-checks freshness **per query**, so a table that ages past its window drops out of the serving set whether or not a rebuild ever lands.
+- A **standalone** Publisher does not act on it: it binds what it just built with no `dataAsOf` and no window, and an entry carrying no window never ages out. The declaration is inert there unless something binds the manifest through the API with those fields set. For a standalone server, the only refresh is a run or the scheduler below.
+- Two sources whose bodies compile to the same SQL share one table, and a host that folds their policies keeps the tightest window and the bounding fallback for both. If two sources need different windows, give them different SQL.
+
+## `schedule`: the power tier
+
+```jsonc
+{ "name": "orders", "materialization": { "scope": "version", "schedule": "0 6 * * *" } }
+```
+
+- A 5-field UTC cron; `L`, `W`, `#`, `?` are rejected at publish. It **requires `scope: "version"`** and is **mutually exclusive with `freshness`** - including a per-source `freshness.*`.
+- It fires only on a standalone Publisher whose scheduler is enabled with `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER`; without that, a schedule is inert. The scheduler sweeps loaded packages, skips one whose policy is invalid or that a control plane manages (`manifestLocation` set), fires at most a bounded number per tick, fires once to catch up after downtime, and skips a fire while a run is already active.
+- **Every scheduled fire is a forced run** (`forceRefresh: true`): every source is rebuilt whether or not its content address changed. A scheduled run therefore never reports reused sources, and a cron that is too frequent shows as build time, not as reuse.
+- `malloy-pub schedule set "<cron>"` writes the schedule and sets `scope: version` in one PATCH; it **replaces the whole `materialization` block**, dropping an existing `freshness` policy (it says so). `schedule clear` removes the cron and leaves `scope: version` in place; moving back to `package` scope is an edit to `publisher.json`.
+
+## `refresh="incremental"`: bounded deltas
+
+```malloy
+#@ persist name="daily_orders" refresh="incremental" watermark="order_date"
+source: daily_orders is orders -> { group_by: order_date; aggregate: revenue is amount.sum() }
+```
+
+- `watermark=` names one of the source's own output columns - real, orderable, non-aggregate. Publisher records how far the table is materialized (`covered_through`) and each refresh recomputes only `[covered_through, frontier)`, half-open, so the frontier value itself is left for the next run. A `date`/`timestamp` watermark takes the run's start time as the frontier; a numeric or string one reads `max(watermark)` from the source. Two refreshes within the same day of a date-watermarked source produce an empty range and skip - correct, and it looks like nothing happened.
+- Add `merge_key="col,..."` when a row can be **restated** with a new watermark value; the delta is then applied as a `MERGE` on that identity instead of a delete-and-reinsert of the range. On a caller-scoped source (a given or gate in the `extend` block) the table holds every caller's rows, so the match also carries the scope columns - declare the key as you would for an unscoped source. A scope term that reaches through a join is refused (`merge_key_scope_unresolved`) rather than scoped partially.
+- **Coverage belongs to the table.** Moving a source between tiers (adding or removing `storage=`), renaming its physical table, or changing its SQL re-seeds it in full; the boundary is measured against the table it was recorded on.
+- **An invalid declaration fails the package load**, not just the publish: `watermark=` without `refresh="incremental"`, `merge_key=` without a watermark, a watermark that names no materialized column or an aggregate, a `calculate:` field, or an unsupported dialect. Postgres, BigQuery and Snowflake sources only (the dialect that has to express the range); a DuckDB source has to say `refresh="full"`.
+- **`forceRefresh` never re-seeds**; it only defeats skip-if-unchanged, and an incremental source is exempt from that anyway. Ask for a full rebuild with `reseed` (`malloy-pub materialize --reseed`, or per source on the build instruction). Keeping them apart is what lets a schedule drive deltas at all, since every scheduled fire is forced.
+- Each manifest entry reports what the run **did**: `refresh: delta | full | none`, with `ledger.coveredThrough` as the boundary now in force. Every fallback - no recorded boundary, a boundary measured on a different table or watermark, drifted columns, `MERGE` on a Postgres older than 15 - rebuilds in full and succeeds, so a source quietly rebuilding on every run looks exactly like one advancing unless you read `refresh`. The reason codes (`forced`, `no_boundary`, `lineage_changed`, `table_renamed`, `merge_unsupported`, `table_unreadable`, `table_emptied`, `shape_mismatch`, `ledger_unreadable`, `chained_storage`, `frontier_unreadable`, `not_advanced`) say why.
+- Two sources that compile to identical SQL share one table **and one boundary**; if their incremental declarations differ, neither ever advances. Publisher warns and names both.
+- `storage=` is supported: the warehouse computes the bounded range and the DML lands in the destination. A **chained** stored source always rebuilds in full (`chained_storage`): see `reference/chaining.md`.
+- The boundary lives in the publisher's own per-process store. A host that spreads a package's refreshes over several workers holds the ledger itself: it stores each entry's `ledger` object and sends them back as `buildInstructions.ledger` on the next run. Standalone, the store is the publisher's, always.
+
+## Which refresh a gated source needs
+
+A `#(access_filter)`-gated source that is persisted freezes the column its gate filters on, so its "refresh" is also the bound on how long a revoked row keeps being served. Only a freshness window with `fallback: live` on a hosted deployment bounds that; a cron alone does not, and an incremental delta never re-reads a row whose watermark did not move. See `reference/access-filter.md`.

package/skills/malloy-materialization/reference/storage.md

@@ -0,0 +1,67 @@
+<!--
+Copyright (c) Credible Data Inc.
+SPDX-License-Identifier: MIT
+-->
+
+# The storage tier: `storage=`, `partition=`, and how a stored table is served
+
+By default a persist source is built **in its own connection** (colocated): the warehouse runs the query and the table lands beside the data. `#@ persist name="..." storage=<destination>` builds it into a **storage destination** the environment declares instead - today a DuckLake catalog over local or object storage - and serves it from there, so a query served from the table never runs in the warehouse at all.
+
+```malloy
+#@ persist name="daily_orders" storage=lake
+source: daily_orders is orders -> { group_by: order_date; aggregate: total is amount.sum() }
+```
+
+- `storage=` names a **storage destination**, never a connection. Destinations are declared on the environment (`storageDestinations: [{ "name": "lake", "type": "ducklake", "ducklakeConnection": { "catalog": {...}, "storage": {...} } }]`, by config or by environment PATCH); a build naming one the environment does not hold fails the run with a `DestinationNotFoundError`. Reads of the environment report a destination's name and type only, never its config.
+- The build is a native **query passthrough**: the source warehouse (Postgres, BigQuery or Snowflake) computes the query and only the result crosses into the store, through DuckDB. A source on a DuckDB connection is refused for `storage=` (there is nothing to push down). A `#(access_filter)`-gated source is always refused on this tier (`reference/access-filter.md`).
+- The tier is gated by `PERSIST_STORAGE_MODE` (`off` / `write-only` / `on`). Under `off` a `storage=` source is **skipped by the build and served live, with no error** - it is never built colocated instead. Under `write-only` it builds but is still served live, and the package warns. If a `storage=` source "did nothing", check the mode before the model.
+- A package may name several destinations; each source routes to its own.
+- `partition=` requires `storage=` (refused as `partition_without_storage` otherwise) and every named column must be one of the source's public output columns.
+
+## Where a given may sit
+
+A persisted table is built once, with no caller present, so a `given` has to be somewhere the build never reads it.
+
+- **Inside the persisted query - refused.** `scoped_rollup is orders -> { where: region = $REGION; group_by: ... }` has only the declaration default available at build time, so the table would hold one caller's slice and nothing at read would re-apply the filter. The build refuses it (`given_in_persisted_query`). The same applies when the given is *inherited*: a query over a source whose `extend` block carries the term reads that filter into its SQL, and is refused just the same - what decides is whether the **build** would substitute a value, not which line the author typed it on.
+- **In the persist source's own `extend` block - honoured per caller.** `#@ persist ... storage=lake` on `fact is raw -> { select: * } extend { where: org_id = $ORG_ID }` strips the term from the build, stores every caller's rows in one table, and re-applies the term per caller at read. One artifact serves every tenant, and a term a caller writes on an extension of it still scopes what is read. Partition on the scoped column (`partition="org_id"`) so each caller's reads touch one partition.
+
+A `#(access_filter)` gate is the same shape with a different source of truth; see `reference/access-filter.md`.
+
+## `partition=`: laying the table out
+
+`#@ persist ... storage=lake partition="org_id"` writes the stored table as one file set per distinct value, so an equality term on that column (`=`, `in`) reads only the files it names; `partition="org_id,day"` nests them in the order given. It is a **layout and carries no isolation**: every filter is re-applied at read whether or not its column is partitioned, so a partition list that omits the column a query filters by costs a full scan, never a wrong answer.
+
+Use it for the column the reads filter by. A rollup carrying every key at a fine grain (every species, every tenant, every product) that each query filters to one key is read **whole** per query without a partition on that key - that is what makes a pre-aggregated table answer in seconds instead of milliseconds. Partition on the filter key and read cost tracks one partition while build cost tracks the whole table. Nothing checks the column's cardinality for you: a partition per value across very many values is the many-small-files case, and that judgement is yours.
+
+`partition=` is part of the source's content address, so changing it rebuilds the table. A chained downstream of a partitioned parent reads the parent's table like any other.
+
+## Shaping a rollup for its reads
+
+A stored rollup pays at build time so that reads are cheap; the grain and the inputs decide whether they are.
+
+- **Grain to the reads, not to the data.** Group by exactly the columns the queries filter or group by. Every extra dimension multiplies the row count, and a rollup that approaches the fact table's size saves nothing.
+- **Bake the filters into the query.** A `where: is_valid = 1` inside the `-> { ... }` shrinks the table; the same clause in a trailing `extend { ... }` is re-applied at read and the table holds every row.
+- **Give a ratio both halves.** A percentage whose numerator is a rollup and whose denominator is a count over the fact table still scans the fact per query. Persist the denominator at its own grain too.
+- **List the columns a pass-through keeps.** `select: *` on a stored copy of a wide fact carries columns no reader uses; name the ones the downstream rollups read.
+
+## How a stored table is served
+
+A query against a package with bound `storage=` tables is compiled over a transient model in which each stored source is rebound to its table - a *virtual source* - under the `##!` flags of the files that declared the originals, with their `extend`-block refinements re-declared on top. A public wrapper over a stored source (`orders is _orders_fact -> { select: * }`, with or without `include { public: ... }`) is carried onto that shape verbatim and served from the table. A wrapper whose query or joins reach a source that is not materialized there - a warehouse table, or a stored source whose table is stale past its freshness window - is served **live** instead.
+
+Routing is decided per query, not per source, so one source can serve some queries from its table and others live:
+
+- **A join to a non-persisted source** serves live for the queries that traverse it; queries on the fact's own fields still route to the table. A join between two persisted sources routes both legs.
+- **A nested column** (`nest:`, a repeated record) is stored, but the serve shape carries it as opaque JSON, so every query that traverses it pays the live cost while the scalar columns serve from storage. The source looks fully materialized and is, for its flat fields.
+- **A refinement the shape cannot reproduce is thinned** (a view, then a join, then a dimension or measure reading a join); a **filter is never thinned** - a source whose filter the shape cannot reproduce loses the tier itself, and only itself.
+- A **stale** table (past its freshness window, with `fallback` `live` or `fail`) leaves the serving set, and a run-time store failure under `fallback: live` degrades that query to live.
+
+What the server reports:
+
+- `QueryResult.servedFrom`: `storage` when the tier answered, `live_fallback` when the tier was bound but a run-time store failure degraded the query to a live recompute, and empty for everything else (a colocated hit, no binding, or an ineligible query - three different things).
+- The package's `storageServeBindings` lists the stored tables the package is currently bound to, with each entry's origin (`persist` or `preaggregate`). `manifestEntryCount` does not count them.
+- `publisher_storage_serve_routing_total{outcome=storage|live_fallback|runtime_live_fallback|blocked_by_row_level_gate, origin}` counts routed queries; `publisher_storage_serve_shape_tier_drop_total{tier}` counts refinements the serve shape had to thin.
+- A query's full result carries the SQL it ran; served from the store, its `FROM` names the destination and the physical table.
+
+## Pre-aggregation uses the same tier
+
+`#@ preaggregate grain="category, order_day"` on a **measure** stores a rollup Publisher derives at that grain and routes to it when a query groups by a subset of the grain; queries do not change. It builds through the same plan, manifest and scheduler as `#@ persist`, needs no `##!` flag, inherits the base source's freshness, and refuses a gated source. A rollup on a measure the published surface hides is not built; the package loads and warns. See `docs/preaggregation.md` for grain choice.

package/skills/malloy-materialization/reference/tuning.md

@@ -5,30 +5,30 @@ SPDX-License-Identifier: MIT
 
 # Tuning materializations for cost and performance
 
-This skill turns the signals the open-source Publisher already records (the materialization history, per-run timings, and which sources were built vs reused) into concrete, **recommendations-only** advice: which sources to persist, which to stop persisting, and how to schedule and scope them. It is the local counterpart to the platform's usage-driven optimization: the Publisher has the raw signals, and you read them with the `malloy-pub` CLI.
+This skill turns the signals the open-source Publisher already records (the materialization history, per-run and per-source timings, which sources were built vs reused, and for the storage tier how queries were routed) into concrete, **recommendations-only** advice: which sources to persist, which to stop persisting, which to chain or partition, and how to schedule them. The Publisher has the raw signals, and you read them with the `malloy-pub` CLI.
 
 > **Recommendations only. Never change a model, schedule, or scope without the user's explicit go-ahead.** Present the findings and the proposed edits, then apply them only when asked. Persisting the wrong source wastes storage and rebuild time; unpersisting a hot one makes queries slow. Let the user decide.
 
-Assumes the `malloy-pub` CLI is on PATH and points at the server (`--url` or `MALLOY_PUBLISHER_URL`, default `http://localhost:4000`). Substitute the real environment and package for `<env>` / `<pkg>`.
+Assumes the `malloy-pub` CLI is on PATH and points at the server (`--url` or `MALLOY_PUBLISHER_URL`). Substitute the real environment and package for `<env>` / `<pkg>`.
 
 ## Step 1: Take inventory
 
 Establish what the package persists today and how it is governed.
 
-- **Persist sources:** the sources annotated `#@ persist name="…"` in the package's `.malloy` files. Read the models (or `get_context` the package) to list them.
+- **Persist sources:** the sources annotated `#@ persist name="..."` in the package's `.malloy` files, with their tier (`storage=` or not), `partition=`, and `refresh=`. Read the models (or `get_context` the package) to list them, and read the package's `buildPlan` for what a run will build and what it will refuse.
 - **Schedule + scope:**
 
   ```bash
   malloy-pub schedule view --environment <env> --package <pkg>
   ```
 
-  This prints the cron (or `none`, meaning publish / on-demand only), the persist **scope** (`package` = artifacts reused across versions; `version` = per published version), and whether a freshness policy is set. A control-plane-managed package (manifestLocation set) is refreshed by the control plane, not the standalone scheduler, so leave its cadence alone.
+  This prints the cron (or `none`, meaning on-demand only), the persist **scope** (`package` = artifacts reused across versions; `version` = per published version), and whether a freshness policy is set. It also says when a control plane manages the package (`manifestLocation` set): leave that package's cadence alone, the control plane refreshes it. And remember the standalone scheduler only runs with `PUBLISHER_LOCAL_MATERIALIZATION_SCHEDULER` set; a cron on a server without it is inert.
 
 ## Step 2: Read the materialization history
 
-The history is where cost lives. Each run records its trigger, timing, and how many sources were built vs reused.
+The history is where cost lives. Each run records its trigger, timing, and per source what it did.
 
-- **For one package** (newest first). A materialization is a run of one package's persist sources, so this is the listing; to review a whole environment, read each package in turn (`malloy-pub list package --environment <env>` names them):
+- **For one package** (newest first; `--limit` / `--offset` page it):
 
   ```bash
   malloy-pub list materialization --environment <env> --package <pkg>
@@ -43,41 +43,48 @@ The history is where cost lives. Each run records its trigger, timing, and how m
   ```
 
   In the JSON, read:
-  - `metadata.durationMs`: how long the build took.
-  - `metadata.sourcesBuilt` vs `metadata.sourcesReused`: how much work each run actually did. A run that is nearly all _reused_ is cheap; one that is nearly all _built_ every time is where cost accumulates.
-  - `metadata.sourcesRefused` and `metadata.refusedSources`: sources the eligibility gate skipped. Each one serves live on every query, so a refused source the user expected to be persisted is a finding in its own right; its entry's `message` says what to change.
-  - `metadata.trigger`: `SCHEDULER` (a cron fired it) or `ON_DEMAND`.
-  - `manifest.entries[*]`: the persisted sources: `sourceName`, `physicalTableName`, `realization`.
+  - `metadata.durationMs`: how long the whole run took.
+  - `metadata.sourcesBuilt` / `sourcesReused` / `sourcesFailed`: how much work the run did. A run that reused most sources was cheap. **A `SCHEDULER` run never reuses**: every scheduled fire is forced, so its `sourcesReused` is 0 by construction; judge a scheduled run by its duration and its per-source entries instead. An incremental source is counted as built even when its refresh applied nothing.
+  - `metadata.sourcesRefused` and `metadata.refusedSources` (present only when a source was refused, and only on a run the publisher planned itself): sources the eligibility gate skipped. Each one serves live on every query, so a refused source the user expected to be persisted is a finding in its own right; its entry's `message` says what to change.
+  - `manifest.entries` - a map keyed by `sourceEntityId`. Each entry carries `sourceName`, `physicalTableName`, `realization`, and the per-source cost signals: `buildDurationMs`; `queryCostBytes` where the warehouse reports it; `refresh: delta | full | none` for an incremental source (a `full` on an unchanged source every run is a source that never advances); `upstreamReuse: reused | recomputed` for a chained source (a `recomputed` with `upstreamRecomputeReason` is a chain that went back to the warehouse).
+  - `manifest.failures`: sources the run lost while still finishing ready.
 
-Look across several runs, not one: the pattern over the recent history (how often it rebuilds, how much it reuses, how long it takes) is the signal.
+Look across several runs, not one: the pattern over the recent history (how often it rebuilds, how long each source takes, what reuses) is the signal.
+
+- **For `storage=` tables, how queries were routed.** `publisher_storage_serve_routing_total{outcome, origin}` on the metrics endpoint counts queries answered from the store (`storage`) against ones that degraded (`live_fallback`, `runtime_live_fallback`), and each query response's `servedFrom` says it per query. Nothing equivalent exists for a table built in the source's own warehouse: there, "rarely queried" is a judgement from the model and the user's knowledge, not a measured metric. Say so when it matters.
 
 ## Step 3: Analyze and recommend
 
 Weigh rebuild cost against query benefit. Common findings:
 
-- **Persist candidate:** an expensive, frequently-queried source that is _not_ persisted (recomputed on every query). Recommend adding `#@ persist name="…"`. Strongest when the source is a heavy aggregate/join reused by many queries and its inputs change slowly. If the source carries a `#(access_filter)` gate, check which case it is first. A gate reached only through a `join_*`, or inherited from a base it cannot be attributed to, means the colocated persist is **refused** - the build skips it and it serves live - so do not recommend persisting it at all. Where the gate is provably the source's own row filter the persist is admitted, and then only recommend it alongside a freshness window (`freshness.fallback="live"`): the gating column freezes at build time, so without one a revoked row can be served under its old access decision indefinitely (see `SKILL.md` in this skill, which also covers where that window does and does not bind).
+- **Persist candidate:** an expensive, frequently-queried source that is _not_ persisted (recomputed on every query). Recommend adding `#@ persist name="..."`. Strongest when the source is a heavy aggregate/join reused by many queries and its inputs change slowly. If the source carries a `#(access_filter)` gate, check which case it is first. A gate reached only through a `join_*`, or not classifying as a row filter, means the colocated persist is **refused** - the build skips it and it serves live - so do not recommend persisting it at all. Where the gate is the source's own row filter the persist is admitted, and then only recommend it alongside a freshness window (`freshness.fallback="live"`): the gating column freezes at build time, so without one a revoked row can be served under its old access decision indefinitely (see `access-filter.md`).
 - **Removal candidate:** a persisted source that is cheap to compute, rarely queried, or rebuilt far more often than it is read. Recommend dropping the `#@ persist` annotation (and its table): the storage + rebuild cost is not buying anything.
-- **Cadence mismatch:** a `SCHEDULER` cadence out of step with how fast the data changes or how long a build takes. If most scheduled runs are all-reused (nothing changed), the cron is too frequent, so loosen it. If queries routinely read stale data, tighten it. If the build's `durationMs` approaches the interval, the cadence is too aggressive.
-- **Scope mismatch:** `scope: version` re-materializes per published version (right when versions must be isolated, e.g. a schedule); `scope: package` reuses one lineage across versions (cheaper when versions can share). A schedule _requires_ `version`. If a package carries a schedule it does not need, clearing the schedule frees it to use the cheaper package scope.
+- **Chain candidate:** a `storage=` rollup whose intermediates re-read the warehouse on every build while a stored root already holds the rows. Rewire the intermediates onto the stored root (declaration order, a distinct `-> { select: * }` intermediate, joins onto stored siblings - `chaining.md`), and read `upstreamReuse` on the next run to confirm the chain holds. Keep a self-join or a wide hash aggregate over a large fact **in the warehouse**: chained, it builds in the destination's engine under its memory bound.
+- **Partition candidate:** a `storage=` rollup carrying every key at a fine grain that queries filter to one key - read whole per query. Recommend `partition="<key>"` on the filter column (`storage.md`), noting the cardinality.
+- **Denominator candidate:** a ratio whose numerator is a rollup and whose denominator counts the fact table per query. Persist the denominator at its own grain.
+- **Cadence mismatch:** a `SCHEDULER` cadence out of step with how fast the data changes or how long a build takes. If the build's `durationMs` approaches the interval, the cadence is too aggressive. If queries routinely read stale data, tighten it. For an incremental source, `refresh: delta` with small `buildDurationMs` per fire is the cheap steady state; `refresh: full` every fire means the boundary is being lost (read the reason).
+- **Scope mismatch:** `scope: version` re-materializes per published version (right when versions must be isolated, e.g. a schedule); `scope: package` reuses one lineage across versions (cheaper when versions can share). A schedule _requires_ `version`. If a package carries a schedule it does not need, clearing it is step one; the scope itself is edited in `publisher.json` (the CLI does not move it back).
 
-Frame each recommendation with the evidence from Step 2 (the run IDs, timings, built/reused counts) so the user can judge it.
+Frame each recommendation with the evidence from Step 2 (the run IDs, timings, built/reused counts, per-entry fields) so the user can judge it.
 
 ## Step 4: Apply (only once approved)
 
 - **Add a persist source:** add the `#@ persist` annotation in the `.malloy` file (use the modeling workflow to validate and reload), then rebuild so it is materialized:
 
   ```bash
-  malloy-pub materialize --environment <env> --package <pkg> --wait
+  malloy-pub materialize --environment <env> --package <pkg> --wait --timeout 900
   ```
 
-- **Remove a persist source:** delete the `#@ persist` annotation, then drop the old run's tables **before** rebuilding what remains:
+  (`--wait` gives up after 120 s by default and exits non-zero while the build continues; raise `--timeout` for a real build.)
+
+- **Remove a persist source:** delete the `#@ persist` annotation, rebuild, then drop the old run's tables:
 
   ```bash
-  malloy-pub delete materialization <id> --environment <env> --package <pkg> --drop-tables
   malloy-pub materialize --environment <env> --package <pkg> --wait
+  malloy-pub delete materialization <old-id> --environment <env> --package <pkg> --drop-tables
   ```
 
-  > `--drop-tables` drops **every** physical table in that run's manifest, not just the removed source's: auto-run assigns stable table names and carries unchanged sources forward, so an old run's manifest names tables a newer manifest still serves. Do the drop **first, then rebuild**: `materialize --wait` re-creates every source that is still persisted, ending in the desired state. Dropping a run whose tables the current serving manifest depends on, without an immediate rebuild, breaks queries (`Table ... does not exist`).
+  > `--drop-tables` drops the physical tables in that run's manifest **except any a still-ready run also names** - with the stable names an auto-run assigns, a table the new run carried forward survives the delete. Deleting a run also re-derives the package's serve bindings from the latest remaining ready run (or clears them to serve live), so a delete does not break queries; it can leave a table behind. A removed source's table is therefore dropped only once no ready run names it - delete the old runs that do, or drop it in the warehouse yourself.
 
 - **Change the schedule / cadence:**
 
@@ -86,13 +93,13 @@ Frame each recommendation with the evidence from Step 2 (the run IDs, timings, b
   malloy-pub schedule clear --environment <env> --package <pkg>
   ```
 
-  `set` also sets `scope: version` (a schedule requires it); the server rejects an invalid cron or an illegal scope/freshness combination, so a rejection means the change was unsafe.
+  `set` also sets `scope: version` (a schedule requires it) and **replaces the whole `materialization` block, dropping an existing `freshness` policy**; the server rejects an invalid cron or an illegal scope/freshness combination, so a rejection means the change was unsafe. `clear` leaves `scope: version` in place.
 
 - **Change the scope:** `scope` is declared in the package's `publisher.json` under `materialization` (`"scope": "package"` or `"version"`; the root-level form is deprecated but still read). Edit it there, then reload/republish the package. Remember a schedule pins scope to `version`, so clear the schedule first if moving to `package`.
 
-After applying, re-run Step 2 on the next few builds to confirm the change did what you predicted (more reuse, shorter builds, or a table that is actually read).
+After applying, re-run Step 2 on the next few builds to confirm the change did what you predicted (more reuse, shorter builds, a chain that reports `reused`, or a table that is actually read).
 
 ## What this skill does not do
 
 - It does not decide for the user or apply changes silently; every edit needs an explicit go-ahead.
-- It cannot see per-query read counts (the open-source Publisher records build history, not query-level table usage), so "rarely queried" is a judgment from the model and the user's knowledge, not a measured metric. Say so when it matters.
+- It cannot measure reads of a table built in the source's own warehouse (only the storage tier reports routing), so "rarely queried" there is a judgment from the model and the user's knowledge, not a measured metric. Say so when it matters.

package/skills/malloy-model/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-model
 description: Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to "create a malloy model", "add dimensions", "add measures", "create a source", or any Malloy model authoring task.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-model-as-you-go/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-model-as-you-go
 description: 'After answering a data question, write down what the answer assumed so the next reader can trust the number. A field with a #(doc) in the model when you can edit it, an extend in the notebook when you can only author reports, or a stated assumption plus a Malloy snippet when you can only chat. Use after every answered question that rested on a judgment call, and whenever a question is asked against tables that have no model yet.'
-version: 0.1.36
+version: 0.1.37
 ---
 
 <!--

package/skills/malloy-modeling/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-modeling
 description: Build semantic models with Malloy for the Malloy Publisher. Read this skill whenever the user asks about modeling data or specifically mentions Malloy.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-notebooks/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-notebooks
 description: Create Malloy notebooks (a .malloy file under notebooks/, written as a one-column layout of tiles) for interactive data stories and reports. Use when user asks to "create a notebook", "make a notebook", "data story", or needs to build reports/visualizations that read top to bottom. Legacy .malloynb files are read, not newly authored.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-phrase-detection/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-phrase-detection
 description: How to phrase search_text on a get_context call so retrieval returns the fields you need instead of a truncated catalog. Covers target-type classification and decomposition patterns.
-version: 0.1.36
+version: 0.1.37
 ---
 <!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->
 

package/skills/malloy-powerbi-review/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-powerbi-review
 description: Move a Power BI semantic model (TMDL, PBIP, or .pbix) onto Malloy - read their model, transpile its DAX, and prove the numbers match. Use when Power BI artifacts are present, when a user wants to migrate or switch off Power BI, and whenever a specific DAX measure has to become Malloy - CALCULATE, ALL/ALLSELECTED/ALLEXCEPT, RANKX and top-N, time intelligence, USERELATIONSHIP, calculation groups, what-if parameters, or an RLS role. Carries a worked, executed transpile per construct and states what each port costs. Works with or without a database connection.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-powerbi-review/reference/_concepts.md

@@ -151,7 +151,7 @@ Found in real Microsoft-published TMDL and not covered by any table above. Count
 | `CALCULATETABLE` | 7 | A filtered table expression; usually collapses into the surrounding query's `where:`. With a Boolean filter argument it overwrites the column's filter like `CALCULATE` does, and routes to `FC1` |
 | `GROUPBY` | 7 | A `group_by:` stage |
 | `VALUES` | 5 | The distinct values of a column; `group_by:` or `count(c)` depending on use |
-| `PERCENTILE.INC` | 5 | No direct equivalent; use the dialect's percentile via `fn!()` |
+| `PERCENTILE.INC` | 5 | No measure equivalent. A raw-SQL percentile through `fn!()` does not compile as a measure (`skill:malloy-gotchas-modeling`). Compute it as a query with the **Tier boundaries** pattern in `skill:malloy-discover`, or list it as a gap. That pattern is nearest-rank, and `PERCENTILE.INC` interpolates, so the two can differ when the rank falls between two values |
 | `ADDCOLUMNS` / `SUMMARIZE` | 5 / 5 | A query stage, not a measure |
 | `CONCATENATEX` | 5 | `string_agg(expr, sep)`, native - no `fn!()` needed; usually report-layer |
 | `STDEV.P` | 2 | `stddev()` where the dialect has it |

package/skills/malloy-publish/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-publish
 description: Package Malloy models for serving by Malloy Publisher. Use when user asks to "publish", "package", "deploy", or wants to share models with others.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-queries/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-queries
 description: Read before writing or debugging a Malloy query. Dates, aggregates vs dimensions, joins, filters, strings, window functions, chart annotations, and the compile errors a SQL habit produces.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-review/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-review
 description: Malloy semantic-model code review. Invoke when the user asks to review, audit, or critique a `.malloy` file, a folder of Malloy models, or a GitHub PR that touches Malloy. Enforces project modeling standards and emits a navigable review file.
-version: 0.1.36
+version: 0.1.37
 ---
 <!--
 Copyright (c) Credible Data Inc.

package/skills/malloy-source-unreachable/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: malloy-source-unreachable
 description: Work out why a source is missing from get_context, or why a query against it was refused with a 404 or a 403, and what to do next. Use when a source you expect is absent from discovery, when execute_query answers "No queryable source", when a query that used to work stops working, or when deciding whether a package's curation is hiding something you need.
-version: 0.1.36
+version: 0.1.37
 ---
 <!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->