@malloy-publisher/skills 0.1.2 → 0.1.4

package/README.md

@@ -50,8 +50,9 @@ skill points to for detail it does not inline. Start with `malloy-getting-starte
 A few skills (`malloy-modeling`, `malloy-publish`, `malloy-document`, `malloy-getting-started`, and
 the `malloy` index) are written for a Publisher host and name Publisher's `malloy_*` MCP tools
 directly. The rest describe Malloy itself and refer to tools by bare name (`get_context`,
-`execute_query`, `search_malloy_docs`), because the prefix depends on the host. On Publisher those
-are `malloy_getContext`, `malloy_executeQuery`, and `malloy_searchDocs`.
+`execute_query`, `search_malloy_docs`), because the prefix depends on the
+host. `search_database_schema` maps the same way if a shared skill starts using it. On Publisher those are `malloy_getContext`, `malloy_executeQuery`, `malloy_searchDocs`, and
+`malloy_searchDatabaseSchema`.
 
 ## Versioning
 

package/package.json

@@ -1,7 +1,7 @@
 {
    "name": "@malloy-publisher/skills",
    "description": "Agent skills for Malloy and the Malloy Publisher",
-   "version": "0.1.2",
+   "version": "0.1.4",
    "license": "MIT",
    "type": "module",
    "main": "dist/index.js",
@@ -26,7 +26,8 @@
    ],
    "repository": {
       "type": "git",
-      "url": "https://github.com/malloydata/publisher.git"
+      "url": "https://github.com/malloydata/publisher.git",
+      "directory": "packages/skills"
    },
    "publishConfig": {
       "access": "public"

package/skills/malloy-analysis/SKILL.md

@@ -36,6 +36,12 @@ A name is a pointer, not confirmation. A field, source, or view name you saw in
 
 Write Malloy using only the model's names. Load `skill:malloy-queries` for syntax (aggregates vs dimensions, joins and field paths, dates, `where:` vs `having:`, counting) and `skill:malloy-gotchas-queries` to avoid the common compile errors. If a model `view:` already matches, run it directly rather than rewriting it.
 
+**Check these three before your first `execute_query`** - they account for most first-attempt compile failures, and they are the ones a SQL habit gets wrong:
+
+- **Counting.** `count(field)` is already the *distinct* count of that field. Malloy has no `count(distinct field)`; it is a parse error, not a deprecation.
+- **Separators.** Within a clause, fields are separated by commas or newlines, never `;`. A semicolon fails with `no viable alternative at input '<next-field>'`.
+- **Join paths.** A dotted path like `carriers.name` resolves only if the source declares that join. Confirm the join name and the field under it in a `get_context` result instead of inferring either from a table name.
+
 If you define a calculated field that is not already in the model, treat it carefully: ad-hoc definitions are a common source of subtle errors.
 
 - Announce it: tell the user you are adding an ad-hoc field, what it computes, and why the model does not already provide it.
@@ -54,7 +60,7 @@ Your first result is a draft, not an answer. The difference between a useful ana
 - **Ground it.** Before interpreting any result, query and state the dataset scope: the time range (`min`/`max` of the primary date dimension) and the row or entity count. Every number is meaningless without it.
 - **Ask "what would make this wrong?"** then run the query that would expose that problem. A plausible-looking wrong answer is the most dangerous kind.
 - **Check the common failure modes:**
-  - Fan-out / double-counting: if you joined across grain, compare `count()` to `count(distinct key)`. A large gap means duplication is inflating the aggregates.
+  - Fan-out / double-counting: if you joined across grain, compare `count()` to `count(key)` - in Malloy `count(field)` is already the distinct count. A large gap means duplication is inflating the aggregates.
   - Broken filters: a quick count confirms a filter narrowed the data as expected. Watch case, spelling, and date-format mismatches; a filter that matches nothing still returns a result, just the wrong one.
   - Null-driven loss: `count() - count(the_field)` shows how many rows a key field drops.
   - Parts that do not sum to the whole: if you split a total into categories, confirm they add up.

package/skills/malloy-getting-started/SKILL.md

@@ -43,6 +43,8 @@ The result is JSON. Charts and dashboards defined in the model render in the Pub
 
 Use `malloy_searchDocs` for language questions (filters, aggregates, joins, nesting, renderers).
 
+If the data you want is in a connected database but not yet in any package, use `malloy_searchDatabaseSchema` instead of `malloy_getContext`: it walks a connection's schemas and tables and ranks them against a plain-English description, and hands back the `source:` line to start a model from. It returns names and types only, so to see what a column actually contains run `malloy_executeQuery` against a model in a package that uses the same connection, with an ad-hoc query like `run: my_conn.table('sales.orders') -> { group_by: order_status }`. That tool needs an existing model to run against, so a table you have not modelled yet has none of its own.
+
 ## 4. What else you can do here
 
 Answering questions is the start, not the whole surface. When the user asks what is possible, say so rather than offering queries alone. Switch skills for the deeper work:

package/skills/malloy-gotchas-queries/SKILL.md

@@ -79,6 +79,40 @@ Two separate rules produce those errors:
 
 **Exception: `count(joined.field)` is correct, not a bug.** `count(joined.field)` is the **canonical Malloy idiom** for distinct-count through a join. Keep it as-is even when nearby `sum`/`avg`/`min`/`max` calls have to use method syntax. The closest method-syntax form `joined.count()` counts *rows* in the joined source (different semantics, differs from the distinct count when the joined field has duplicates within the joined table). The Malloy docs example `joined.count(field)` does NOT compile against current Malloy (error: `Expression illegal inside path.count()`); it only works for double-nested paths like `aircraft.count(aircraft_models.code)`.
 
+## `sum`/`avg` Need a Numeric Field
+
+```malloy
+// WRONG: "Can't use type string" - status is a string column
+aggregate: avg_status is avg(status)
+// RIGHT: aggregate a numeric field; count a string one
+aggregate: avg_price is avg(price), statuses is count(status)
+```
+
+Check the field's type in the `get_context` result before aggregating it. A name that reads numeric (`order_number`, `zip`, `account_id`) is very often typed string.
+
+## Dotted Paths Must Name a Declared Join
+
+```malloy
+// WRONG: the source declares the join as `carrier`, so this fails with
+//   "'carriers.name' is not a source or join"
+run: flights -> { group_by: carriers.name }
+// RIGHT: use the join name the source actually declares
+run: flights -> { group_by: carrier.nickname }
+```
+
+A dotted path resolves only against a join declared on the source you are running. Confirm both the join name and the field under it in a `get_context` result; do not infer either from a table name or a plural/singular guess.
+
+## `order_by:` Can Only Name an Output Column
+
+```malloy
+// WRONG: "Unknown field total in output space" - total is never emitted
+run: orders -> { group_by: state, aggregate: revenue is sum(total), order_by: total }
+// RIGHT: order by a column the query actually outputs
+run: orders -> { group_by: state, aggregate: revenue is sum(total), order_by: revenue }
+```
+
+`order_by:` resolves against the query's *output* columns, not the source's fields. To order by something, `group_by` or `aggregate` it first - and if it comes through a join, alias it (see above).
+
 ## Chart Annotation Placement
 
 Place `# bar_chart` / `# line_chart` on the **nested view definition**, not on `nest:` itself. Putting it on `nest:` causes "not a repeated record" errors.

package/skills/malloy-materialization/SKILL.md

@@ -30,13 +30,17 @@ Materialize an expensive source once so queries read a **pre-built warehouse tab
    ```jsonc
    {
      "name": "my-package",
-     "scope": "package",  // default; "version" = each published version owns its own tables
-     "materialization": { "freshness": { "window": "24h", "fallback": "live" } }
+     "materialization": {
+       "scope": "package",  // default; "version" = each published version owns its own tables
+       "freshness": { "window": "24h", "fallback": "live" },
+       "queryMetadata": { "team": "finance" }  // tags the build's backend statements
+     }
    }
    ```
    Enforced at publish (strict), on edits (strict), at load (warn, still serves), and by the scheduler (an offending package is skipped):
-   - **`scope`**: `package` (default; artifacts reused across published versions) or `version` (each artifact owned by one version). Package-level only; there is no per-source scope.
+   - **`scope`**: `package` (default; artifacts reused across published versions) or `version` (each artifact owned by one version). Package-level only; there is no per-source scope. A root-level `scope` is the deprecated home and still works, with a warning; declaring both homes with different values is rejected.
    - **`materialization.freshness`** (`window` + `fallback` of `live`/`stale_ok`/`fail`) is the objective a **hosted control plane** enforces by refreshing the table to meet it (`fallback: "live"` serves live compute while stale/absent). A **standalone** Publisher does **not** act on `freshness` for refresh - see **Building and refreshing**.
+   - **`materialization.queryMetadata`** is a bag of string properties attached to every statement the build issues, for the backend's own cost attribution (Snowflake `QUERY_TAG`, BigQuery job labels, a leading SQL comment elsewhere). Overridable per source with `#@ persist queryMetadata.<name>="<value>"`. Observability only: it never changes what gets built. See `docs/query-metadata.md`.
    - **`materialization.schedule`** is a 5-field UTC cron (`min hour dom mon dow`; `L`/`W`/`#`/`?` rejected). It **requires `scope: "version"`** and is **mutually exclusive with `freshness`**. This is how a standalone Publisher refreshes on a cadence.
 
 4. **Reads vs writes.** The persist source can *read* any dataset the connection can read; the persist *target* (`name=`'s dataset) must be a dataset the connection can **write** (typically a scratch dataset).

package/skills/malloy-materialization-tuning/SKILL.md

@@ -93,7 +93,7 @@ Frame each recommendation with the evidence from Step 2 (the run IDs, timings, b
 
   `set` also sets `scope: version` (a schedule requires it); the server rejects an invalid cron or an illegal scope/freshness combination, so a rejection means the change was unsafe.
 
-- **Change the scope:** `scope` is declared at the root of the package's `publisher.json` (`"scope": "package"` or `"version"`). Edit it there, then reload/republish the package. Remember a schedule pins scope to `version`, so clear the schedule first if moving to `package`.
+- **Change the scope:** `scope` is declared in the package's `publisher.json` under `materialization` (`"scope": "package"` or `"version"`; the root-level form is deprecated but still read). Edit it there, then reload/republish the package. Remember a schedule pins scope to `version`, so clear the schedule first if moving to `package`.
 
 After applying, re-run Step 2 on the next few builds to confirm the change did what you predicted (more reuse, shorter builds, or a table that is actually read).
 

package/skills/malloy-model/SKILL.md

@@ -293,7 +293,7 @@ source: orders is duckdb.table('orders.parquet') extend {
 ```
 
 - **Source-level** `#(authorize) "<expr>"` gates that one source. **File-level** `##(authorize) "<expr>"` applies to every source in the file. Multiple gates combine as an OR, access is granted if any one is true, so a permissive file-level gate is a **model-wide override**, not an added restriction.
-- **Not inherited, not joined.** The gate applies only to the source a query directly runs against. A source that `extend`s a locked base does **not** inherit the base's gate, and a gate on a source reached only via `join_*` never fires. Pair a locked base (`#(authorize) "false"`) with curated extension sources, using access modifiers (`include { public: …, private: * }`), so an extension re-exposes only a curated column surface instead of leaking the base's data through the join or extend.
+- **Entry point only: not joined, but inherited through `extend`.** The gate applies to the source a query enters through. A gate on a source reached only via `join_*` **never fires**, at any depth, so anything ungated that joins a locked base hands the base's rows to every caller. A source that `extend`s a locked base and declares no gate of its own **does** carry the base's gate; declaring its own replaces it. Pair a locked base (`#(authorize) "false"`) with curated extension sources, using access modifiers (`include { public: …, private: * }`), so an extension re-exposes only a curated column surface, and keep sensitive sources out of ungated joins.
 - The expression may reference only givens and literals, never a column of the gated source; the check runs against a synthetic probe row, not your data.
 
 > **Trust caveat.** Givens are **caller-asserted**, anyone who can reach the query API can claim a favorable given, e.g. `{"ROLE":"admin"}`. `#(authorize)` is only a real boundary when it sits behind a trusted tier that sets givens from its own verified context, never directly from an untrusted caller. It is not, on its own, end-user authentication.

package/skills/malloy-modeling/SKILL.md

@@ -9,7 +9,10 @@ description: Build semantic models with Malloy for the Malloy Publisher. Read th
 
 ## Pre-Flight Checklist
 
-1. **Discover first**: ground yourself with `malloy_getContext` before writing ANY code. It returns the package's sources, views, and fields (with their docs), so you build on what actually exists. Never guess field names.
+1. **Discover first**: ground yourself before writing ANY code, with the tool that matches what you are modelling.
+   - Modelling data **already in a package**: `malloy_getContext` returns that package's sources, views, and fields (with their docs).
+   - Modelling **a database with no package yet**: `malloy_getContext` has nothing to return, so use `malloy_searchDatabaseSchema` instead. It walks the connection's schemas and tables, ranks them against a plain-English description, and gives you each table's columns plus the `source:` line to start from. Take those names verbatim into step 5.
+   Never guess field names either way.
 2. **Search docs proactively**: call `malloy_searchDocs` BEFORE writing unfamiliar patterns (window functions, query-based sources, pipelines). Don't guess. Malloy syntax is specific and SQL intuition is often wrong.
 3. **Use `skill:malloy-patterns`** to discover available doc topics (YoY, cohorts, rendering, window functions).
 4. **Check diagnostics** after writing: fix the FIRST error first, errors cascade.
@@ -43,7 +46,7 @@ DISCOVER → SCOPE → SOURCES → DEFINITIONS → BUILD BASE → BUILD JOINED
 
 | Step | Skill | What Happens |
 |------|-------|-------------|
-| 1. Discover | `skill:malloy-discover` | Read the model and data; scan sources, fields, distributions; detect prior art |
+| 1. Discover | `skill:malloy-discover` | Read the model and data; scan sources, fields, distributions; detect prior art. With no package yet, start from `malloy_searchDatabaseSchema` to find the tables in the connection |
 | 2. Propose Scope | `skill:malloy-scope` | Present findings, user selects focus |
 | 3. Propose Sources | `skill:malloy-define` | Propose source plan, user confirms architecture |
 | 4. Propose Definitions | `skill:malloy-define` | Propose fields per base source, user confirms logic |
@@ -102,6 +105,7 @@ Ensure the Publisher MCP tools are configured before modeling.
 | `malloy_compile` | Compile-check a change and get diagnostics back without running a query |
 | `malloy_reloadPackage` | Recompile a package from disk so a saved edit becomes queryable by name |
 | `malloy_searchDocs` | Search Malloy docs (call BEFORE unfamiliar patterns) |
+| `malloy_searchDatabaseSchema` | Find the tables in a database connection by plain-English description, when modelling data that is not in a package yet. Returns each table's columns and the `source:` line to start from. Names and types only: no row value is returned |
 
 Never guess field names. Ground yourself with `malloy_getContext` to see the sources and fields a package defines.