@malloy-publisher/skills 0.1.0

package/skills/malloy-materialization-tuning/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: malloy-materialization-tuning
+description: Optimize a package's Malloy Persistence materializations for cost and performance using the malloy-pub CLI and the materialization history. Recommend what to persist, what to stop persisting, and how to schedule/scope it. Use when the user asks to make a package cheaper or faster, tune persistence, decide what to materialize, or review persist/schedule choices.
+---
+
+# Tuning materializations for cost and performance
+
+This skill turns the signals the open-source Publisher already records (the materialization history, per-run timings, and which sources were built vs reused) into concrete, **recommendations-only** advice: which sources to persist, which to stop persisting, and how to schedule and scope them. It is the local counterpart to the platform's usage-driven optimization: the Publisher has the raw signals, and you read them with the `malloy-pub` CLI.
+
+> **Recommendations only. Never change a model, schedule, or scope without the user's explicit go-ahead.** Present the findings and the proposed edits, then apply them only when asked. Persisting the wrong source wastes storage and rebuild time; unpersisting a hot one makes queries slow. Let the user decide.
+
+Assumes the `malloy-pub` CLI is on PATH and points at the server (`--url` or `MALLOY_PUBLISHER_URL`, default `http://localhost:4000`). Substitute the real environment and package for `<env>` / `<pkg>`.
+
+## Step 1: Take inventory
+
+Establish what the package persists today and how it is governed.
+
+- **Persist sources:** the sources annotated `#@ persist name="…"` in the package's `.malloy` files. Read the models (or `malloy_getContext` the package) to list them.
+- **Schedule + scope:**
+
+  ```bash
+  malloy-pub schedule view --environment <env> --package <pkg>
+  ```
+
+  This prints the cron (or `none`, meaning publish / on-demand only), the persist **scope** (`package` = artifacts reused across versions; `version` = per published version), and whether a freshness policy is set. A control-plane-managed package (manifestLocation set) is refreshed by the control plane, not the standalone scheduler, so leave its cadence alone.
+
+## Step 2: Read the materialization history
+
+The history is where cost lives. Each run records its trigger, timing, and how many sources were built vs reused.
+
+- **Across the whole environment** (all packages, newest first; the rows are interleaved and labeled by package, not grouped into contiguous per-package blocks):
+
+  ```bash
+  malloy-pub list materialization --environment <env>
+  ```
+
+  Columns: Package, ID, Status, **Trigger** (`SCHEDULER` vs `ON_DEMAND`), Started, Completed, Error.
+
+- **For one package:**
+
+  ```bash
+  malloy-pub list materialization --environment <env> --package <pkg>
+  ```
+
+- **A single run's detail** (the cost signals):
+
+  ```bash
+  malloy-pub get materialization <id> --environment <env> --package <pkg>
+  ```
+
+  In the JSON, read:
+  - `metadata.durationMs`: how long the build took.
+  - `metadata.sourcesBuilt` vs `metadata.sourcesReused`: how much work each run actually did. A run that is nearly all _reused_ is cheap; one that is nearly all _built_ every time is where cost accumulates.
+  - `metadata.trigger`: `SCHEDULER` (a cron fired it) or `ON_DEMAND`.
+  - `manifest.entries[*]`: the persisted sources: `sourceName`, `physicalTableName`, `realization`.
+
+Look across several runs, not one: the pattern over the recent history (how often it rebuilds, how much it reuses, how long it takes) is the signal.
+
+## Step 3: Analyze and recommend
+
+Weigh rebuild cost against query benefit. Common findings:
+
+- **Persist candidate:** an expensive, frequently-queried source that is _not_ persisted (recomputed on every query). Recommend adding `#@ persist name="…"`. Strongest when the source is a heavy aggregate/join reused by many queries and its inputs change slowly.
+- **Removal candidate:** a persisted source that is cheap to compute, rarely queried, or rebuilt far more often than it is read. Recommend dropping the `#@ persist` annotation (and its table): the storage + rebuild cost is not buying anything.
+- **Cadence mismatch:** a `SCHEDULER` cadence out of step with how fast the data changes or how long a build takes. If most scheduled runs are all-reused (nothing changed), the cron is too frequent, so loosen it. If queries routinely read stale data, tighten it. If the build's `durationMs` approaches the interval, the cadence is too aggressive.
+- **Scope mismatch:** `scope: version` re-materializes per published version (right when versions must be isolated, e.g. a schedule); `scope: package` reuses one lineage across versions (cheaper when versions can share). A schedule _requires_ `version`. If a package carries a schedule it does not need, clearing the schedule frees it to use the cheaper package scope.
+
+Frame each recommendation with the evidence from Step 2 (the run IDs, timings, built/reused counts) so the user can judge it.
+
+## Step 4: Apply (only once approved)
+
+- **Add a persist source:** add the `#@ persist` annotation in the `.malloy` file (use the modeling workflow to validate and reload), then rebuild so it is materialized:
+
+  ```bash
+  malloy-pub materialize --environment <env> --package <pkg> --wait
+  ```
+
+- **Remove a persist source:** delete the `#@ persist` annotation, then drop the old run's tables **before** rebuilding what remains:
+
+  ```bash
+  malloy-pub delete materialization <id> --environment <env> --package <pkg> --drop-tables
+  malloy-pub materialize --environment <env> --package <pkg> --wait
+  ```
+
+  > `--drop-tables` drops **every** physical table in that run's manifest, not just the removed source's: auto-run assigns stable table names and carries unchanged sources forward, so an old run's manifest names tables a newer manifest still serves. Do the drop **first, then rebuild**: `materialize --wait` re-creates every source that is still persisted, ending in the desired state. Dropping a run whose tables the current serving manifest depends on, without an immediate rebuild, breaks queries (`Table ... does not exist`).
+
+- **Change the schedule / cadence:**
+
+  ```bash
+  malloy-pub schedule set "0 6 * * *" --environment <env> --package <pkg>   # 5-field UTC cron
+  malloy-pub schedule clear --environment <env> --package <pkg>
+  ```
+
+  `set` also sets `scope: version` (a schedule requires it); the server rejects an invalid cron or an illegal scope/freshness combination, so a rejection means the change was unsafe.
+
+- **Change the scope:** `scope` is declared at the root of the package's `publisher.json` (`"scope": "package"` or `"version"`). Edit it there, then reload/republish the package. Remember a schedule pins scope to `version`, so clear the schedule first if moving to `package`.
+
+After applying, re-run Step 2 on the next few builds to confirm the change did what you predicted (more reuse, shorter builds, or a table that is actually read).
+
+## What this skill does not do
+
+- It does not decide for the user or apply changes silently; every edit needs an explicit go-ahead.
+- It cannot see per-query read counts (the open-source Publisher records build history, not query-level table usage), so "rarely queried" is a judgment from the model and the user's knowledge, not a measured metric. Say so when it matters.

package/skills/malloy-model/SKILL.md

@@ -0,0 +1,320 @@
+---
+name: malloy-model
+description: Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to "create a malloy model", "add dimensions", "add measures", "create a source", or any Malloy model authoring task.
+---
+
+# Building Malloy Models
+
+> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
+
+## Getting Started (New Projects)
+
+If no `.malloy` files exist yet, do discovery and propose a structure first, then return here to build base source and joined source files. Keep proposals and the analysis behind them in the conversation.
+
+**File structure convention** (a flat layout at the package root is the simplest default):
+```
+<package-name>/
+  publisher.json              # Required for publishing (name, version, description)
+  customers.malloy            # Base source: one per table
+  products.malloy
+  orders.malloy
+  user_order_facts.malloy     # Computed source
+  order_analysis.malloy       # Source: one per analytical domain
+  customer_health.malloy
+```
+
+Versions for new packages should start at "0.0.1".
+
+**Before creating any files**, check for an existing `publisher.json` in the target directory. If one exists for a different package, create a new subdirectory for your package, don't overwrite another package's config.
+
+In Publisher an environment is a project, and Publisher is single-tenant, so there is no org/tenant layer to model around: one environment holds one set of packages.
+
+## Prior Art Dispatch
+
+If your discovery turned up existing modeling patterns to mirror (a derived table, UNNEST joins, a review or curation pass), read the relevant reference before building.
+
+| Pattern found in prior art | Reference to read |
+|---------------------|-------------------|
+| Derived table (PDT/NDT) | `skill:malloy-lookml-review` build-derived-tables guidance |
+| UNNEST joins or struct access | `skill:malloy-lookml-review` build-unnest guidance |
+| Review pass for coverage | `skill:malloy-lookml-review` review-coverage guidance |
+| Curate pass with visibility seeds | `skill:malloy-lookml-review` curate-visibility guidance |
+
+## Base Source Templates
+
+### Base Source (Simple Mode)
+
+```malloy
+source: customers is my_conn.table('sales.customers')
+extend {
+  primary_key: customer_id
+
+  dimension:
+    // Use dimension (not rename:) for cleaner column names
+    order_type is `Type`
+    full_name is concat(first_name, ' ', last_name)
+    segment is lifetime_value ?
+      pick 'enterprise' when >= 100000
+      pick 'mid-market' when >= 10000
+      else 'SMB'
+
+  measure:
+    customer_count is count()
+}
+```
+
+> **Every `dimension:` needs `name is expr`.** A bare column name like `dimension: species` is a parse error (e.g. `missing IS at ...`). Raw columns are already usable in `group_by` / `select` without any declaration, so only add a `dimension:` when deriving or renaming a field (e.g. `revenue is price * quantity`).
+
+### Base Source (Curated Mode with Access Modifiers)
+
+```malloy
+##! experimental.access_modifiers
+
+source: orders is my_conn.table('sales.orders')
+include {
+  public:
+    #(doc) Order identifier
+    order_id
+
+    #(doc) Customer who placed the order
+    user_id
+
+    #(doc) Total sale price in USD
+    sale_price
+
+    #(doc) Order creation timestamp
+    created_at
+
+  internal:
+    raw_payload_json  // Verified empty via index query + user confirmation
+}
+extend {
+  primary_key: order_id
+
+  dimension:
+    #(doc) Date the order was placed
+    order_date is created_at::date
+
+  measure:
+    #(doc) Total number of orders
+    order_count is count()
+
+    #(doc) Total revenue in USD
+    # currency
+    revenue is sum(sale_price)
+}
+```
+
+### Computed Source (from Query)
+
+```malloy
+import "orders.malloy"
+
+source: user_order_facts is from(
+  orders -> {
+    group_by: customer_id
+    aggregate:
+      total_orders is count()
+      total_revenue is sum(sale_price)
+      first_order_date is min(created_at)
+      last_order_date is max(created_at)
+  }
+) extend {
+  primary_key: customer_id
+
+  dimension:
+    days_since_last_order is days(now - last_order_date)
+    is_repeat_buyer is total_orders > 1
+
+  measure:
+    buyer_count is count()
+    avg_customer_ltv is avg(total_revenue)
+}
+```
+
+For advanced query-based source patterns (window functions, pipelines), see `reference/query-sources.md`.
+
+## Joined Source File Template
+
+```malloy
+import "customers.malloy"
+import "orders.malloy"
+import "user_order_facts.malloy"
+
+#(doc) Customer health analysis. Use for retention, segmentation, and churn risk.
+source: customer_health is customers extend {
+  join_one: user_order_facts with customer_id
+  join_many: orders on customer_id = orders.customer_id
+
+  dimension:
+    is_at_risk is user_order_facts.days_since_last_order > 90
+      and user_order_facts.total_orders > 1
+
+  measure:
+    revenue_per_customer is orders.sale_price.sum() / nullif(customer_count, 0)
+    at_risk_count is count() { where: is_at_risk = true }
+}
+```
+
+## Base vs Joined Sources
+
+| | Base Joined Source File | Joined Source File |
+|---|---|---|
+| **Contains** | One table's fields | Joins between base sources |
+| **Dimensions** | Intrinsic to this table only | Cross-source (require joins) |
+| **Measures** | Single-table aggregations | Cross-source aggregations |
+| **Joins** | None (or only lookup joins intrinsic to the source) | Defines relationships between base sources |
+| **Views** | None (views belong in analysis) | None |
+| **One per** | Physical table or computed source | Analytical domain |
+
+## Key Rules
+
+- **Every `dimension:` needs `name is expr`**: a bare `dimension: species` is a parse error. Raw columns are queryable directly in `group_by` / `select`; only declare a `dimension:` to derive or rename a field.
+- **Define joined tables before referencing them**, use `import` statements in multi-file architecture
+- **Use `nullif(denominator, 0)` for all division**
+- **Alias joined fields before using in `order_by`**: `group_by: yr is table.year`
+- **Verify join paths** exist before referencing `a.b.field` (each hop needs explicit join)
+- **Pick syntax**: value BEFORE condition, `pick 'Small' when size < 10`
+- **`where:` vs `having:`**: Use `where:` for row filters, `having:` for aggregate filters
+- **Never use `rename:`**, it's incompatible with `include {}`. Always use `internal:` + `dimension:` for cleaner column names (e.g., mark `` `Type` `` as `internal`, add `dimension: order_type is \`Type\``)
+- **Mark raw columns `internal` when a derived dimension replaces them**
+- **Check for duplicate rows** before building measures
+- When both a combined table (all types) and filtered/split tables exist, prefer the split tables
+- **DRY: define measures/dimensions in base source files, not inline in views**
+
+## Parameterizable Filters with `#(filter)`
+
+`#(filter)` annotations declare filterable dimensions on a source. Publisher parses them, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. This gives you a clean way to expose tunable knobs (date range, region, manufacturer) without hand-rolling parameterization in every query.
+
+Filters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an analysis needs a knob the source doesn't expose, the right move is to add a `#(filter)` to the source itself, not to wedge filtering into the consumer.
+
+### Syntax
+
+```malloy
+#(filter) [name=NAME] dimension=DIMENSION type=TYPE [implicit] [required]
+```
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `name` | No | Unique identifier for the filter; defaults to the dimension name. Used as the API parameter key. |
+| `dimension` | Yes | The source dimension this filter targets. Quote with `"..."` if the name contains spaces. |
+| `type` | Yes | Comparator (see below). |
+| `implicit` | No | Hides the filter from the UI and API summaries. Used for infrastructure concerns the system injects rather than the user. |
+| `required` | No | Server returns 400 if a required filter has no value at query time. Use this for governance, latency, and correctness, see below. |
+
+### Filter types
+
+| Type | Malloy clause | Use case |
+|------|---------------|----------|
+| `equal` | `dimension = 'value'` | Exact match on a single value |
+| `in` | `dimension ? 'a' \| 'b' \| 'c'` | Match any of multiple values |
+| `like` | `dimension ~ '%value%'` | Substring / pattern matching |
+| `greater_than` | `dimension > value` | Range floor (after, minimum) |
+| `less_than` | `dimension < value` | Range ceiling (before, maximum) |
+
+### Example
+
+```malloy
+#(filter) name=Manufacturer dimension=Manufacturer type=in
+#(filter) name=Subject dimension=Subject type=like
+#(filter) name=Major_Recall dimension="Major Recall" type=equal
+#(filter) name=Recall_After dimension="Report Received Date" type=greater_than
+#(filter) name=Recall_Before dimension="Report Received Date" type=less_than
+source: recalls is duckdb.table('data/auto_recalls.csv') extend {
+  measure:
+    recall_count is count()
+}
+```
+
+For date-range filters, declare two filters with distinct `name` values targeting the same dimension (one `greater_than`, one `less_than`).
+
+### When to use `required`
+
+`required` filters are a correctness, latency, and governance mechanism, not just UX. Mark a filter `required` when:
+
+1. **Modeling correctness, the source's `primary_key:` is only unique under a filter.** If a high-cardinality key is not unique across the whole table but is unique within a scoping dimension, then that scoping dimension MUST be supplied for symmetric aggregation to produce correct numbers. For example, if `events.id` repeats across days but is unique within a single `event_date`, queries that don't pin the date can fan out and return hash-collision-sized garbage (~10²¹). Declare `#(filter) name=Event_Date dimension=event_date type=equal required` so the server refuses queries that don't provide it.
+2. **Query latency, the source spans more data than any single query should scan.** A multi-year, multi-region table where every reasonable analysis is scoped to a date range or region: making the date filter required prevents accidental full-table scans.
+3. **Partial views** that are only meaningful inside a date range, region, or business segment.
+4. **Governance**, an analyst should never query the raw source without a scoping filter applied.
+
+For (1), pair the required filter with a comment explaining the cardinality dependency, and consider also declaring `#(doc)` on the source noting the constraint.
+
+### When to use `implicit`
+
+Use `implicit` for filters the *system* must inject but users should not see. The filter applies; it just doesn't appear in the UI or API filter list.
+
+### Type-aware literals
+
+Publisher formats values based on the dimension's data type, `string` → `'value'`, `boolean` → bare `true`/`false`, `date` → `@YYYY-MM-DD`. You don't quote values yourself in the API call; Publisher handles formatting.
+
+### Bypass
+
+Pass `bypass_filters=true` (REST) or `bypassFilters: true` (POST body) to skip filter injection entirely. Use sparingly, required-filter governance only works if bypass is restricted to trusted callers.
+
+## Access Control: Source Gating with `#(authorize)`
+
+Gate query access to a source with `#(authorize)` over declared `given:` values (`given:` is Malloy's native runtime-parameter mechanism, the going-forward replacement for `#(filter)`). Publisher evaluates a source's in-scope `#(authorize)` expressions against the request's supplied givens before running the query; if **any** expression returns `true` the request proceeds, otherwise it is denied with **403**. A source with no in-scope `#(authorize)` annotations is unrestricted.
+
+```malloy
+##! experimental.givens
+
+given:
+  ROLE :: string
+
+#(authorize) "$ROLE = 'analyst'"
+source: orders is duckdb.table('orders.parquet') extend {
+  measure: order_count is count()
+}
+```
+
+- **Source-level** `#(authorize) "<expr>"` gates that one source. **File-level** `##(authorize) "<expr>"` applies to every source in the file. Multiple gates combine as an OR, access is granted if any one is true, so a permissive file-level gate is a **model-wide override**, not an added restriction.
+- **Not inherited, not joined.** The gate applies only to the source a query directly runs against. A source that `extend`s a locked base does **not** inherit the base's gate, and a gate on a source reached only via `join_*` never fires. Pair a locked base (`#(authorize) "false"`) with curated extension sources, using access modifiers (`include { public: …, private: * }`), so an extension re-exposes only a curated column surface instead of leaking the base's data through the join or extend.
+- The expression may reference only givens and literals, never a column of the gated source; the check runs against a synthetic probe row, not your data.
+
+> **Trust caveat.** Givens are **caller-asserted**, anyone who can reach the query API can claim a favorable given, e.g. `{"ROLE":"admin"}`. `#(authorize)` is only a real boundary when it sits behind a trusted tier that sets givens from its own verified context, never directly from an untrusted caller. It is not, on its own, end-user authentication.
+>
+> **Forward direction.** Givens are how access control is built here, and the planned next step is **identity-bound ("secure") givens** - reserved values a trusted tier populates from a verified token or proxy header, which the caller cannot override - turning `#(authorize)` into a standalone boundary. Model access on `given:` + `#(authorize)` now; it is the surface that carries forward.
+
+Full syntax, OR/override semantics, validation, and the error contract are covered in your deployment's `#(authorize)` reference documentation.
+
+## Join Syntax
+
+- Simple join: `join_one: users with user_id`
+- Expression join: `join_one: origin is airports on origin_code = origin.code`
+- Composite key: `join_one: items on order_id = items.order_id and product_id = items.product_id`
+- Multiple joins to same table: `join_one: origin_airport is airports with origin`
+
+**Join Types:** `join_one:` (many-to-one, efficient) | `join_many:` (one-to-many, always safe) | `join_cross:` (many-to-many)
+
+**Verify cardinality** before writing joins: `run: target -> { group_by: fk_col, aggregate: n is count(), having: n > 1, limit: 5 }`. 0 results → `join_one`. Any results → `join_many`.
+
+## After Writing: Check & Review
+
+Check diagnostics after writing. Errors cascade, fix the FIRST error only, then re-check. If errors persist, use the debugging strategy: look at first error, search docs if unsure, fix, repeat.
+
+**Validate with `execute_query`:** Run queries, check distributions, verify measures, confirm joins (no fan-out).
+
+To inspect the sources and fields a model already defines, ground yourself with `get_context`. It returns the package's sources, views, and fields, so there is no separate schema-search step. When you're unsure of Malloy syntax, call `search_malloy_docs` rather than guessing.
+
+## Advanced Patterns
+
+Load the relevant reference file when you encounter these scenarios:
+
+| Scenario | Read |
+|----------|------|
+| Need pre-aggregated or windowed source | `reference/query-sources.md` |
+| Curating access modifiers | `reference/access-modifiers.md` |
+| Normalized/ER-style schema (4+ tables, no clear fact table) | `reference/normalized-schemas.md` |
+| Formalizing analysis into a model | `reference/analysis-to-model.md` |
+| Many-to-many / bridge tables / composite keys | `reference/bridge-tables.md` |
+
+## Done
+
+Step complete. Output: base source files (`.malloy`, one per table) and joined source files (`.malloy`, one per analytical domain).
+
+**Suggest next steps to the user:**
+
+- Open the model in the browser to see it live: `http://localhost:4000/<environmentName>/<packageName>` for the package, or `http://localhost:4000/<environmentName>/<packageName>/<modelPath>` for a single model file. First confirm the running server actually serves this package (it is in the loaded `publisher.config.json`, or mounted live with `--server_root . --watch-env <env>`); a package the server has not loaded returns a 404, so do not hand over a link to a package that was just authored but never loaded.
+- Build a notebook with interactive filters over the model (see `skill:malloy-notebooks`).
+- Run analysis questions against the model (see `skill:malloy-analysis`).
+- When you're ready to serve the model, publishing is out of scope for open-source Publisher v1: self-hosters commit the package to git and use their host's publish path.

package/skills/malloy-model/reference/access-modifiers.md

@@ -0,0 +1,21 @@
+# Access Modifiers: Curate the Interface
+
+Add `##! experimental.access_modifiers` at the top of base source files.
+
+**Every column must be explicitly declared** in `include {}`, listed under a single `public:` block with `#(doc)` tags, or under `internal:`. Do not use `public: *`.
+
+**`internal` fields are NOT accessible through join paths.** Most fields should stay public.
+
+**GOTCHA:** `include { except: a, b }` without a `public:` block makes remaining fields **private**, not public.
+
+**What to mark `internal` (verify via index query + confirm with user first):**
+- Empty/garbage columns
+- Raw JSON/array blobs (when all useful fields extracted as dimensions)
+- Duplicate columns where only one is correct
+- Raw columns replaced by a derived dimension
+
+**What to mark `private` (only after user confirms, very rare):** Highly sensitive data only (SSNs, raw credit cards, passwords).
+
+**Include positioning:**
+- `include { } extend { }`: New definitions in extend are PUBLIC
+- `extend { } include { }`: Include covers everything

package/skills/malloy-model/reference/analysis-to-model.md

@@ -0,0 +1,82 @@
+# Starting from Analysis
+
+When you've been doing analysis, writing queries, building views, creating notebooks, and want to formalize the work into a reusable semantic model, follow this process.
+
+**The model emerges from your analysis.** You don't design sources top-down, you identify what's worth encoding as reusable building blocks.
+
+## Step 1: IDENTIFY, what belongs in a model?
+
+Scan your analysis `.malloy` and `.malloynb` files. For each dimension, measure, join, or calculation, ask: **is this a reusable building block?**
+
+A measure or dimension belongs in a model if it meets ANY of these criteria:
+
+| Criteria | Example |
+|----------|---------|
+| **Reusable**, another query or user would benefit from this definition | `revenue is sum(sale_price)`, any analysis involving orders needs this |
+| **Composable**, can be combined with other concepts for future analysis | `customer_tier` dimension, useful for segmenting any metric |
+| **Encodes business logic**, standardizes a calculation that should be consistent | Regex that parses a messy column, status mapping, tier boundaries |
+| **Enables reproducibility**, someone repeating this analysis shouldn't re-derive it | Complex window function for first-touch attribution |
+
+**Include even if used once**, if a calculation encodes business logic (regex, tier boundaries, status mappings) or was hard to get right (window functions, multi-step derivations), it belongs in the model.
+
+**When uncertain, propose with reasoning and ask the user.** Show what you think should be modeled and why. The user confirms, adjusts, or defers.
+
+### What to scan for
+
+| Pattern in analysis | Promotes to |
+|---------------------|-------------|
+| Tables used in queries | Base source candidates |
+| Dimensions that define segments, categories, or time buckets | Dimensions in base sources |
+| Measures that compute business metrics | Measures in base sources |
+| Joins between tables | Joined source candidates |
+| Pre-aggregated patterns at different grains | Computed source candidates |
+| Complex calculations (regex, window functions) | Dimensions in base sources, especially worth modeling |
+| Repeated `where:` filters that define business segments | Filtered measures or dimensions |
+
+## Step 2: DECOMPOSE, break into composable building blocks
+
+Take the identified elements and decompose compound analysis into atomic, reusable parts.
+
+**Atomic measures:** Break compound insights into their building blocks.
+```
+Analysis: "Revenue grew 23% YoY for enterprise customers"
+→ Measure: revenue is sum(sale_price)
+→ Measure: enterprise_revenue is sum(sale_price) { where: segment = 'enterprise' }
+→ Dimension: segment (already exists or needs creation)
+```
+
+**Reusable dimensions:** Separate bucketing logic from one-off filters.
+```
+Analysis: "High-value users (>$500 LTV) churn less"
+→ Dimension: ltv_tier is lifetime_value ? pick 'high' when > 500 else 'standard'
+→ Not: a filtered measure that bakes in the $500 threshold
+```
+
+**What to leave behind:**
+- Ad-hoc calculations that answered a specific question but aren't reusable
+- Filtered aggregates tied to a specific finding (unless the filter defines a business segment)
+- Views that are specific to one analysis narrative
+
+## Step 3: STRUCTURE, build the model files
+
+Map the identified and decomposed elements to the file architecture:
+- Single-table measures/dimensions → base source files (one per table)
+- Cross-table measures/joins → joined source files (one per analytical domain)
+- Pre-aggregated patterns at different grains → computed source files (see `reference/query-sources.md`)
+
+**The analysis files remain as-is**, they're the record of your investigation. The model is the reusable layer extracted from them. Later, analysis files can be refactored to `import` the model and reference its sources instead of defining everything inline.
+
+## Step 4: VALIDATE, confirm the extraction
+
+Run key analysis queries against the new model to confirm results match. If numbers differ, a dimension or measure was extracted incorrectly.
+
+Then apply the same curate and document standards as the rest of this skill (access modifiers, `#(doc)` tags).
+
+## When to formalize
+
+Don't formalize too early. Good signals:
+- You've answered 3+ distinct questions and keep redefining the same measures
+- You find yourself copy-pasting dimensions or joins between queries
+- Someone else needs to use your analysis patterns
+- You want a notebook or dashboard that should survive data refreshes
+- You built something non-trivial (regex, window function, multi-step logic) that would be painful to recreate

package/skills/malloy-model/reference/bridge-tables.md

@@ -0,0 +1,51 @@
+# Bridge Tables & Composite Keys
+
+## Bridge Table Pattern (Many-to-Many)
+
+Bridge table gets `join_one` to each source; other sources get `join_many` to bridge:
+
+```malloy
+// Bridge source
+source: enrollments is conn.table('enrollments') extend {
+  join_one: students with student_id   // each enrollment → one student
+  join_one: courses with course_id     // each enrollment → one course
+}
+// Query source: students with their enrollments
+source: student_courses is students extend {
+  join_many: enrollments on student_id = enrollments.student_id
+}
+```
+
+## Composite Key Joins
+
+When no single column is unique (bridge tables, time-series snapshots), use `on` with `and`:
+
+```malloy
+// `with` is single-column only, composite keys MUST use `on` + `and`
+join_one: items on order_id = items.order_id and product_id = items.product_id
+```
+
+`primary_key` is single-column only, pick the highest-cardinality column.
+
+## Cardinality Verification
+
+Before writing any join, check FK uniqueness with `execute_query`:
+
+```malloy
+run: target_table -> { group_by: fk_col, aggregate: n is count(), having: n > 1, limit: 5 }
+```
+
+- 0 results = unique → `join_one` (more efficient)
+- Any results = not unique → `join_many` (always safe)
+
+For composite keys, test multi-column: `group_by: col_a, col_b`, same pattern.
+
+## Post-Join Verification
+
+After writing joins, verify row counts haven't inflated:
+
+```malloy
+run: source -> { aggregate: row_count is count() }
+```
+
+If higher than the raw table, a `join_one` target key isn't unique, switch to `join_many`.

package/skills/malloy-model/reference/normalized-schemas.md

@@ -0,0 +1,66 @@
+# Normalized Schemas (3-Stage Pattern)
+
+Use this pattern when data is in normalized/relational form, ER-diagram style schemas, application databases (CRM, ERP, OLTP), many tables linked by foreign keys, no clear single fact table, or many-to-many relationships.
+
+For most star/snowflake schemas, the base source + joined source layers are sufficient.
+
+**Why multiple sources?** Different sources answer different questions.
+- `deals` with `join_one: company` → counts only companies WITH deals
+- `companies` with `join_many: deals` → counts ALL companies
+
+## The 3 Stages
+
+| Stage | Purpose | Naming |
+|-------|---------|--------|
+| 1. Base | Raw tables + single-table dimensions | `_xxx_base` |
+| 2. Modeled | Forward joins (each relationship defined once) | `_xxx_modeled` |
+| 3. Query Sources | Reverse joins + measures (public) | `xxx` (no prefix) |
+
+## Example
+
+```malloy
+// Stage 1: Base sources
+source: _companies_base is duckdb.table('company') extend {
+  primary_key: _id
+  dimension:
+    company_name is name
+    size_bucket is employee_count ?
+      pick 'Small' when < 50 pick 'Medium' when < 500 else 'Large'
+}
+source: _deals_base is duckdb.table('deal') extend { primary_key: _id }
+
+// Stage 2: Modeled (forward joins, defined once)
+source: _companies_modeled is _companies_base extend {}
+source: _deals_modeled is _deals_base extend {
+  join_one: company is _companies_modeled on company_id = company._id
+}
+
+// Stage 3: Query sources (reverse joins + measures)
+source: companies is _companies_modeled extend {
+  join_many: deals is _deals_modeled on _id = deals.company_id
+  measure:
+    company_count is count()
+    deal_count is count(deals._id)
+}
+source: deals is _deals_modeled extend {
+  measure:
+    deal_count is count()
+    total_value is sum(amount)
+}
+```
+
+## CRITICAL: Reuse Modeled Sources
+
+In query sources, join to MODELED sources, not raw tables:
+
+```malloy
+// WRONG - joins raw table, loses dimensions/measures
+source: accounts is _accounts extend {
+  join_many: trans is conn.table('financial.trans') on ...
+}
+
+// RIGHT - joins the modeled source
+source: accounts is _accounts extend {
+  join_many: trans is transactions on account_id = trans.account_id
+}
+```