@malloy-publisher/server 0.0.230 → 0.0.231

package/dist/server.mjs

@@ -267678,7 +267678,7 @@ Each reference file is loaded by the workflow phase that needs it (via dispatch
 - Complex SQL dimensions (50+ lines): default is simplify or push upstream
 - Derived tables: classify as performance-only, transformation, or aggregation
 - Refinement structure (\`+view\`): consolidate vs. preserve layering via \`extend\`
-- Synthetic primary keys: ask about actual grain` }, { name: "malloy-model", description: 'Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to "create a malloy model", "add dimensions", "add measures", "create a source", or any Malloy model authoring task.', body: `# Building Malloy Models
+- Synthetic primary keys: ask about actual grain` }, { name: "malloy-materialization-tuning", description: "Optimize a package's Malloy Persistence materializations for cost and performance using the malloy-pub CLI and the materialization history. Recommend what to persist, what to stop persisting, and how to schedule/scope it. Use when the user asks to make a package cheaper or faster, tune persistence, decide what to materialize, or review persist/schedule choices.", body: "# Tuning materializations for cost and performance\n\nThis skill turns the signals the open-source Publisher already records (the materialization history, per-run timings, and which sources were built vs reused) into concrete, **recommendations-only** advice: which sources to persist, which to stop persisting, and how to schedule and scope them. It is the local counterpart to the platform's usage-driven optimization: the Publisher has the raw signals, and you read them with the `malloy-pub` CLI.\n\n> **Recommendations only. Never change a model, schedule, or scope without the user's explicit go-ahead.** Present the findings and the proposed edits, then apply them only when asked. Persisting the wrong source wastes storage and rebuild time; unpersisting a hot one makes queries slow. Let the user decide.\n\nAssumes the `malloy-pub` CLI is on PATH and points at the server (`--url` or `MALLOY_PUBLISHER_URL`, default `http://localhost:4000`). Substitute the real environment and package for `<env>` / `<pkg>`.\n\n## Step 1: Take inventory\n\nEstablish what the package persists today and how it is governed.\n\n- **Persist sources:** the sources annotated `#@ persist name=\"…\"` in the package's `.malloy` files. Read the models (or `malloy_getContext` the package) to list them.\n- **Schedule + scope:**\n\n  ```bash\n  malloy-pub schedule view --environment <env> --package <pkg>\n  ```\n\n  This prints the cron (or `none`, meaning publish / on-demand only), the persist **scope** (`package` = artifacts reused across versions; `version` = per published version), and whether a freshness policy is set. A control-plane-managed package (manifestLocation set) is refreshed by the control plane, not the standalone scheduler, so leave its cadence alone.\n\n## Step 2: Read the materialization history\n\nThe history is where cost lives. Each run records its trigger, timing, and how many sources were built vs reused.\n\n- **Across the whole environment** (all packages, newest first; the rows are interleaved and labeled by package, not grouped into contiguous per-package blocks):\n\n  ```bash\n  malloy-pub list materialization --environment <env>\n  ```\n\n  Columns: Package, ID, Status, **Trigger** (`SCHEDULER` vs `ON_DEMAND`), Started, Completed, Error.\n\n- **For one package:**\n\n  ```bash\n  malloy-pub list materialization --environment <env> --package <pkg>\n  ```\n\n- **A single run's detail** (the cost signals):\n\n  ```bash\n  malloy-pub get materialization <id> --environment <env> --package <pkg>\n  ```\n\n  In the JSON, read:\n  - `metadata.durationMs`: how long the build took.\n  - `metadata.sourcesBuilt` vs `metadata.sourcesReused`: how much work each run actually did. A run that is nearly all _reused_ is cheap; one that is nearly all _built_ every time is where cost accumulates.\n  - `metadata.trigger`: `SCHEDULER` (a cron fired it) or `ON_DEMAND`.\n  - `manifest.entries[*]`: the persisted sources: `sourceName`, `physicalTableName`, `realization`.\n\nLook across several runs, not one: the pattern over the recent history (how often it rebuilds, how much it reuses, how long it takes) is the signal.\n\n## Step 3: Analyze and recommend\n\nWeigh rebuild cost against query benefit. Common findings:\n\n- **Persist candidate:** an expensive, frequently-queried source that is _not_ persisted (recomputed on every query). Recommend adding `#@ persist name=\"…\"`. Strongest when the source is a heavy aggregate/join reused by many queries and its inputs change slowly.\n- **Removal candidate:** a persisted source that is cheap to compute, rarely queried, or rebuilt far more often than it is read. Recommend dropping the `#@ persist` annotation (and its table): the storage + rebuild cost is not buying anything.\n- **Cadence mismatch:** a `SCHEDULER` cadence out of step with how fast the data changes or how long a build takes. If most scheduled runs are all-reused (nothing changed), the cron is too frequent, so loosen it. If queries routinely read stale data, tighten it. If the build's `durationMs` approaches the interval, the cadence is too aggressive.\n- **Scope mismatch:** `scope: version` re-materializes per published version (right when versions must be isolated, e.g. a schedule); `scope: package` reuses one lineage across versions (cheaper when versions can share). A schedule _requires_ `version`. If a package carries a schedule it does not need, clearing the schedule frees it to use the cheaper package scope.\n\nFrame each recommendation with the evidence from Step 2 (the run IDs, timings, built/reused counts) so the user can judge it.\n\n## Step 4: Apply (only once approved)\n\n- **Add a persist source:** add the `#@ persist` annotation in the `.malloy` file (use the modeling workflow to validate and reload), then rebuild so it is materialized:\n\n  ```bash\n  malloy-pub materialize --environment <env> --package <pkg> --wait\n  ```\n\n- **Remove a persist source:** delete the `#@ persist` annotation, then drop the old run's tables **before** rebuilding what remains:\n\n  ```bash\n  malloy-pub delete materialization <id> --environment <env> --package <pkg> --drop-tables\n  malloy-pub materialize --environment <env> --package <pkg> --wait\n  ```\n\n  > `--drop-tables` drops **every** physical table in that run's manifest, not just the removed source's: auto-run assigns stable table names and carries unchanged sources forward, so an old run's manifest names tables a newer manifest still serves. Do the drop **first, then rebuild**: `materialize --wait` re-creates every source that is still persisted, ending in the desired state. Dropping a run whose tables the current serving manifest depends on, without an immediate rebuild, breaks queries (`Table ... does not exist`).\n\n- **Change the schedule / cadence:**\n\n  ```bash\n  malloy-pub schedule set \"0 6 * * *\" --environment <env> --package <pkg>   # 5-field UTC cron\n  malloy-pub schedule clear --environment <env> --package <pkg>\n  ```\n\n  `set` also sets `scope: version` (a schedule requires it); the server rejects an invalid cron or an illegal scope/freshness combination, so a rejection means the change was unsafe.\n\n- **Change the scope:** `scope` is declared at the root of the package's `publisher.json` (`\"scope\": \"package\"` or `\"version\"`). Edit it there, then reload/republish the package. Remember a schedule pins scope to `version`, so clear the schedule first if moving to `package`.\n\nAfter applying, re-run Step 2 on the next few builds to confirm the change did what you predicted (more reuse, shorter builds, or a table that is actually read).\n\n## What this skill does not do\n\n- It does not decide for the user or apply changes silently; every edit needs an explicit go-ahead.\n- It cannot see per-query read counts (the open-source Publisher records build history, not query-level table usage), so \"rarely queried\" is a judgment from the model and the user's knowledge, not a measured metric. Say so when it matters." }, { name: "malloy-model", description: 'Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to "create a malloy model", "add dimensions", "add measures", "create a source", or any Malloy model authoring task.', body: `# Building Malloy Models
 
 > **Tool names** are written bare here - \`get_context\`, \`execute_query\`, \`search_malloy_docs\`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
 

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/server",
    "description": "Malloy Publisher Server",
-   "version": "0.0.230",
+   "version": "0.0.231",
    "main": "dist/server.mjs",
    "bin": {
       "malloy-publisher": "dist/server.mjs"