@malloy-publisher/skills 0.1.0

package/skills/malloy-debug/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: malloy-debug
+description: Fix Malloy compile errors and understand error messages. Use when encountering errors in .malloy files, user says "fix this error", "malloy error", "compile error", "syntax error", or sees 20+ cascading errors.
+---
+
+# Debugging Malloy Errors
+
+> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
+
+## Get Diagnostics
+
+**Claude Code (in VS Code terminal):** Call `mcp__ide__getDiagnostics` with the file URI.
+
+**VS Code Copilot / Cursor:** Use the `ReadLints` tool on the file path, or open the file and check lints in the editor.
+
+**Claude Code (standalone terminal):** No IDE diagnostics available. Ask the user to open the file in VS Code with the Malloy extension and report the errors.
+
+## Strategy
+
+**Errors cascade.** Later errors may be caused by or hidden behind earlier ones. Fix the FIRST error only, re-check diagnostics, repeat. Do not attempt to fix multiple errors at once.
+
+1. Look at FIRST error, ignore all others
+2. Call `search_malloy_docs` with the error message if unsure
+3. Fix that one issue, re-check diagnostics
+4. Repeat until clean. New errors may appear as earlier ones are resolved
+
+## Quick Fixes
+
+| Error | Fix |
+|-------|-----|
+| "Unknown field" | Check typo, source order, wrong source, or missing `import` |
+| "Can't use type string" | Cast: `field::number` |
+| "Aggregate not allowed in where" | Use `having:` instead |
+| 20+ random errors | Backtick reserved word (`` `Date` ``, `` `Hour` ``, `` `number` ``) |
+| "Can't find field" with `rename:` | Never use `rename:`. It's incompatible with `include {}`. Use `internal:` + `dimension:` instead |
+| Import path errors | Check paths: `import "orders.malloy"`. All files should be in the same directory (flat layout) |
+| `from()` errors | Verify the source query returns the expected columns, check that imported sources are defined |
+| "Cannot redefine 'X'" | Field already exists from query-based source (`-> { group_by, aggregate }`). Remove the dimension, add only NEW derived fields in `extend {}`. Use `include {}` to add `#(doc)` tags to existing fields. |
+
+## Gotchas Checklist
+
+### Backtick Reserved Words
+```malloy
+// WRONG                        // RIGHT
+dimension: d is Date::date      dimension: d is `Date`::date
+```
+Common reserved words: `Date`, `Timestamp`, `Type`, `Hour`, `source`, `year`, `month`, `day`, `week`, `quarter`, `count`, `sum`, `avg`, `min`, `max`, `number`, `string`, `boolean`, `true`, `false`, `order`, `select`, `from`, `where`, `index`, `table`, `time`, `now`, `today`, `range`, `window`, `row`, `current`
+
+**When in doubt, backtick it.** See the Malloy docs (https://docs.malloydata.dev) for the full categorized list of reserved words.
+
+**Note on `number`:** Only the bare word needs backticking. Compound names like `account_number` are fine.
+
+### Use `is not null` for NULL Checks
+```malloy
+// WRONG                        // RIGHT
+is_sold is sold_at != null      is_sold is sold_at is not null
+```
+
+### Call Date Functions, Don't Access as Properties
+```malloy
+// WRONG                        // RIGHT
+dow is created_at.day_of_week   dow is day_of_week(created_at)
+```
+Properties: `.month`, `.year`, `.day` | Functions: `day_of_week()`, `week()`, `hour()`
+
+### Use `having:` for Aggregate Filters
+```malloy
+// WRONG                        // RIGHT
+where: order_count > 10         having: order_count > 10
+```
+
+### Cast Strings Before Aggregating
+```malloy
+// WRONG                        // RIGHT
+avg(score)                      avg(score::number)
+```
+
+### Use Boolean Literals Without Quotes
+```malloy
+// WRONG                        // RIGHT
+where: active = 'true'          where: active = true
+```
+
+### Alias Joined Fields Before Using in order_by
+```malloy
+// WRONG                        // RIGHT
+group_by: races.year            group_by: yr is races.year
+order_by: races.year            order_by: yr
+```
+
+### Use Method Syntax for Joined Aggregates
+```malloy
+// WRONG                        // RIGHT
+sum(items.cost)                 items.cost.sum()
+```
+
+### Define Lookup Tables First (or Use Imports)
+```malloy
+// Single file: define lookup tables before referencing them
+// Multi-file: use import statements
+import "customers.malloy"
+
+source: orders is conn.table('orders') extend {
+  join_one: customers with customer_id  // Works because customers imported
+}
+```
+
+### Use `nullif` for Division
+```malloy
+// WRONG                        // RIGHT
+a / b                           a / nullif(b, 0)
+```
+
+## Cross-File Errors
+
+| Error | Fix |
+|-------|-----|
+| "Can't find source X" | Add `import "X.malloy"` at top of file (all files in same directory) |
+| Wrong import path | All `.malloy` files should be in the package root (flat layout). Use `import "orders.malloy"`, not `import "../sources/orders.malloy"` |
+| Circular imports | Source A imports Source B which imports Source A. Restructure to break the cycle |
+| `from()` "Can't find field" | Verify the source query's GROUP BY and aggregate fields match what you reference in `extend {}` |

package/skills/malloy-define/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: malloy-define
+description: Propose a source plan and field definitions for a Malloy semantic model. Covers picking which sources to model and at what grain, then proposing the specific renames, dimensions, and measures per source, every proposal backed by querying the data.
+---
+
+# Propose sources and definitions
+
+This skill covers two consecutive activities when building or extending a Malloy semantic model:
+
+> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
+
+- **Propose sources**: the architectural blueprint (which sources, what grain).
+- **Propose definitions**: the specific fields per base source (renames, dimensions, measures).
+
+Both happen in conversation. Propose, let the user confirm or adjust, then carry the confirmed plan forward into the actual `.malloy` model. There is no separate plan-file store: keep the source plan and field proposals in the conversation, and write the model itself when the user has confirmed. See your modeling workflow for the broader picture.
+
+Read the existing model first so you propose against what is really there. Use `get_context` with a plain-English description to inspect the current sources and fields and find the most relevant existing sources. Confirm the scope (which tables are in play) before proposing the source plan.
+
+## Propose a source plan
+
+**Goal:** Propose the full source architecture for the tables in scope.
+
+### Base sources
+
+One base source per table in scope. For each, specify:
+
+| Source | Table | Grain | Primary Key | Role |
+|--------|-------|-------|-------------|------|
+| orders | sales.orders | one row per order | order_id | Fact, transactions |
+| customers | sales.customers | one row per customer | customer_id | Dimension, who |
+| products | sales.products | one row per product | product_id | Dimension, what |
+
+### Computed sources
+
+Computed sources are created from queries, not physical tables. Propose them when:
+
+1. **Grain mismatch**: the analytical scope requires a grain that no physical table provides (e.g., customer-level metrics from an order-grain table).
+2. **Repeated aggregation patterns**: the same group-by plus aggregate pattern would be used in multiple places.
+3. **Cross-entity aggregations**: inspecting the model and querying the data shows that an aggregate rolled up to a different entity would be reused.
+
+For each computed source, explain:
+
+| Source | Source Query | Grain | Rationale |
+|--------|-------------|-------|-----------|
+| user_order_facts | orders grouped by customer_id | one row per customer | Need customer-level order metrics (LTV, order count, recency) for customer health analysis |
+
+### Dependencies
+
+Show which sources depend on which:
+
+```
+customers (physical) ← user_order_facts (derived, sources from orders)
+orders (physical) → user_order_facts (derived)
+products (physical): independent
+```
+
+### Deferred sources
+
+List sources considered but not included, with reasoning:
+
+- **order_items**: bridge table, defer until line-item analysis is needed.
+- **monthly_product_facts**: derived, defer until product trend analysis is requested.
+
+### User interaction
+
+The user will:
+- **Confirm** the source plan as-is.
+- **Add** missing sources (physical or derived).
+- **Remove** unnecessary sources.
+- **Validate** grain assignments.
+- **Defer** sources to later iterations.
+
+Once the source plan is confirmed, carry it forward into the definitions step below. Keep the confirmed map in the conversation rather than persisting it to a separate file.
+
+## Propose definitions
+
+**Goal:** Propose specific fields per base source with data evidence, working from the confirmed source plan.
+
+### For each base source
+
+Present a table of proposed fields.
+
+**Renames (schema cleanup):**
+
+| Raw Column | Proposed Name | Reason |
+|-----------|---------------|--------|
+| `Order Date` | order_date | Whitespace in column name |
+| `Type` | order_type | Reserved word |
+| `number` | item_number | Reserved word |
+
+**Dimensions:**
+
+| Field | Logic | Data Evidence | Priority |
+|-------|-------|---------------|----------|
+| order_status | status column | 5 distinct values: pending, processing, shipped, delivered, cancelled | must-have |
+| order_month | submitted_at.month | Time trending | must-have |
+| order_size | total buckets (data-driven) | Distribution: min $5, p25 $35, median $85, p75 $150, p95 $450, max $2,400. Proposed breaks at p25/p75: <$35, $35-$150, >$150 | nice-to-have |
+| is_returned | returned_at is not null | 8% of orders have non-null returned_at | nice-to-have |
+
+**Data-driven tiers:** For bucketed dimensions like `order_size`, always derive boundaries from the actual data distribution (percentiles, natural breaks, clustering). Query `min`, `max`, `p25`, `p50`, `p75`, `p95` and propose boundaries based on the distribution. Show the evidence so the user can confirm or adjust. Never use arbitrary hardcoded thresholds unless the user explicitly provides them.
+
+**Measures:**
+
+| Field | Logic | Data Evidence | Priority |
+|-------|-------|---------------|----------|
+| order_count | count() | Basic metric | must-have |
+| revenue | sum(total) | Total column includes tax. Range: $5 - $2,400 | must-have |
+| avg_order_value | revenue / nullif(order_count, 0) | Derived from above | must-have |
+| return_rate | returned_count / nullif(order_count, 0) | 8% overall return rate | nice-to-have |
+
+### For each computed source
+
+Show the source query and additional fields.
+
+**`user_order_facts`**, derived from `orders` grouped by `customer_id`:
+
+| Aggregated Field | Logic |
+|-----------------|-------|
+| total_orders | count() |
+| total_revenue | sum(total_price) |
+| first_order_date | min(submitted_at) |
+| last_order_date | max(submitted_at) |
+
+**Additional dimensions on top:**
+
+| Field | Logic | Evidence |
+|-------|-------|----------|
+| days_since_last_order | days(now - last_order_date) | Recency metric |
+| is_repeat_buyer | total_orders > 1 | 62% of customers are repeat |
+| buyer_frequency | total_orders buckets | Distribution: 1 (38%), 2-4 (35%), 5-19 (22%), 20+ (5%) |
+
+### Business logic questions
+
+Flag decisions the agent can't make from data alone. Be specific and data-grounded:
+
+> **Q1:** Your `orders` table has both `created_at` and `submitted_at`. 87% of rows have them within 1 minute, but 13% differ by 1-3 days. Which should be the canonical order date?
+>
+> **Q2:** I'm proposing `order_size` tiers based on the data distribution: small (<$35, below p25), medium ($35-$150, p25-p75), large (>$150, above p75). Do these data-driven breaks work for you, or do you have specific business thresholds?
+>
+> **Q3:** The `status` column has 5 values. Should "cancelled" orders be excluded from revenue calculations, or included with a separate measure?
+
+### Priority ranking
+
+Group proposals into:
+- **Must-have**: core metrics that every analyst needs (counts, sums, primary dimensions).
+- **Nice-to-have**: useful but not critical (bucketed dimensions, rates).
+- **Value-add**: new insights the data supports but may not be asked for yet (computed sources, complex measures).
+
+### User interaction
+
+The user will:
+- **Confirm** business logic decisions.
+- **Adjust** thresholds and bucket boundaries.
+- **Add** missing fields.
+- **Remove** fields they don't need.
+- **Change** priorities.
+
+Once the definitions are confirmed, write them into the `.malloy` model (see your modeling workflow). Use `#(doc)` annotations to document sources and fields, and `#(filter)` annotations to declare server-side filterable dimensions where appropriate. Keep the confirmed definitions in the conversation; there is no separate plan-file store.
+
+## Data-driven proposals
+
+**Every recommendation must be backed by a query result.** Do not propose based on column names or schema structure alone. Always run `execute_query` to check the actual data before presenting. To learn what sources and fields exist, ground yourself with `get_context`: it returns the model's sources, views, and fields, so there is no separate schema-search step.
+
+| Proposal Type | What to query first |
+|--------------|---------------------|
+| Dimension (bucketed) | Distribution: min, p25, median, p75, p95, max. Propose boundaries from natural breaks, not arbitrary values. |
+| Dimension (categorical) | Distinct values and frequencies. Show the actual categories and their counts. |
+| Measure (sum/avg) | Sample values: min, max, avg. Verify the column contains what you think (e.g., is `total` gross or net?). |
+| Measure (rate/ratio) | Query both numerator and denominator. Verify they make sense together. |
+| Denormalized field vs join | Compare the pre-computed column against the joined aggregate. Report match rate. Recommend whichever is more reliable. |
+| Computed source | Run the proposed group-by plus aggregation. Verify the grain collapses as expected and the result is useful. |
+| Date field selection | Query all candidate date columns. Show % of rows where they differ and by how much. |
+| Column rename | Verify the column has data worth exposing (not 100% NULL). |
+
+**Example, denormalized vs joined:**
+
+> "Your `customers` table has an `order_count` column. I compared it against `count()` from the `orders` table:
+> - 94% of customers match exactly
+> - 6% have stale counts (the denormalized value is lower than the actual count)
+> - The max discrepancy is 12 orders
+>
+> I'd recommend using the joined count from `orders` rather than the denormalized `order_count`. Want to keep the denormalized column as internal, or drop it?"
+
+## Tips
+
+- **Show data, not assumptions:** every proposed dimension or measure should have evidence (distinct values, distributions, ranges).
+- **Use `execute_query`** to verify any data questions before presenting to the user.
+- **Don't over-propose:** 5-8 dimensions and 4-6 measures per base source is usually enough to start.
+- **Rank everything:** users appreciate knowing what's essential vs. optional.
+- **Business logic questions must be specific.** "What date should I use?" is bad. "Your table has `created_at` and `submitted_at` that differ by 1-3 days in 13% of rows, which is canonical?" is good.
+
+## Output
+
+A confirmed source architecture and a confirmed set of field definitions (renames, dimensions, measures, business decisions), held in the conversation and ready to write into the `.malloy` model via your modeling workflow.

package/skills/malloy-discover/SKILL.md

@@ -0,0 +1,182 @@
+---
+name: malloy-discover
+description: Silent data discovery for Malloy modeling. Used at Step 1 of the modeling workflow. Scans tables, columns, distributions, and relationships without user interaction. The agent builds an internal picture before presenting anything.
+---
+
+# Data Discovery (Step 1, Silent)
+
+> **CRITICAL**: Read the model before writing ANY Malloy code. The model defines the sources, connection names, and fields. Never guess connection names.
+
+> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
+
+> **PREREQUISITE:** Make sure the Malloy MCP tools (`get_context`, `execute_query`, `search_malloy_docs`) are configured and reachable. If they are not, stop and resolve the MCP connection before continuing.
+
+**This step is silent.** The agent does not present findings to the user yet. That happens in the next step (PROPOSE SCOPE).
+
+## Tools
+
+- **`get_context`**: Ground yourself in the package's sources, views, and fields (with their docs). Call FIRST. The sources and their join paths are the schema you build on.
+- **`execute_query`**: Run ad-hoc queries to preview data, verify values, check NULLs, validate assumptions.
+- **`search_malloy_docs`**: Get Malloy syntax help when needed.
+
+## Workflow
+
+```
+1. Check for prior art signals                 → If found, ask user: "I found [LookML/dbt] files, use as prior art?"
+2. If user confirms: read adapter reference        → Follow skill:malloy-lookml-review, keep prior-art notes in-conversation
+3. get_context                           → Ground yourself: sources, views, fields
+4. Inspect source definitions                  → See ALL fields and join paths for key sources
+5. Derive candidate joins/dimensions/measures  → Read them off the model and the data, not a suggestion tool
+6. Define a minimal source if one is missing   → Just enough to run execute_query for previews
+7. execute_query(query)                  → Preview data, verify values, check NULLs, check duplicates
+8. search_malloy_docs(query)                    → Get syntax help when needed
+9. Proceed to Step 2 (PROPOSE SCOPE)
+```
+
+**If the model has no sources defined** and no LookML files are present, do NOT silently retry or proceed without data. Tell the user: "No model sources were found. Please check that the package points at a connected data source, then try again."
+
+**If the model has no sources defined** but LookML files ARE present (LookML-only mode), skip steps 3-7. Use connection name and table paths from the LookML review. Flag all proposals as unvalidated.
+
+**Key principle:** Query data to verify assumptions. Don't ask the user to confirm values you can check yourself.
+
+**Search docs proactively.** If you discover patterns that need derived/pre-aggregated sources, window functions, or unfamiliar features, call `search_malloy_docs` BEFORE writing code, not just when you hit errors.
+
+## Query File for Discovery
+
+**In the schema-first workflow:** Run ad-hoc queries with `execute_query`. If the source you want to preview is not yet defined in the model, define a minimal one against the connection and table so you can run previews. The real model fields are built in later steps.
+
+```malloy
+// minimal source for previewing data during discovery
+source: explore is my_conn.table('schema.table') extend {}
+```
+
+**In analysis-first mode:** There is no temp file. The analysis `.malloy` file IS your working file. It grows throughout the session and becomes the input for formalizing into a model. See `skill:malloy-analyze` for that workflow.
+
+## What to Capture
+
+When reviewing tables and columns, capture:
+
+### Table-Level
+- All tables with row counts
+- Connection name and schema (CRITICAL, never guess)
+- Table roles: fact, dimension, bridge, lookup, staging, operational
+- Join relationships (FK → PK mappings)
+
+### Column-Level
+- Primary key and foreign key columns
+- Data types (watch for string dates, arrays, JSON)
+- Reserved word columns that need backticking (`Date`, `Type`, `number`, `source`, etc.)
+- Column cardinality and NULL rates (via `execute_query`)
+- Data distributions for key numeric and categorical columns
+
+### Data Quality
+- **Check for duplicate rows** on primary keys. Run `group_by: pk, aggregate: count(), having: count() > 1` on each key table. Duplicates cause `sum()` to return nonsensical values.
+- **Denormalized count columns**: beware pre-aggregated fields (e.g., `order_count` in a customer table) that may conflict with joined counts.
+- **Delimited list columns**: flag string columns containing comma-separated values.
+
+### Data-Driven Validation
+
+**Every recommendation must be grounded in queried data, not schema inference.** During discovery, run `execute_query` to validate assumptions before proposing anything in later steps.
+
+| What to validate | Query to run |
+|-----------------|-------------|
+| **Denormalized vs joined values** | Compare pre-computed columns (e.g., `customers.order_count`) against the actual joined aggregate (`count()` from `orders`). Report discrepancy rate. If >0%, flag for user decision. |
+| **Candidate date fields** | When multiple date/timestamp columns exist, query both. What % of rows differ? By how much? This informs which is canonical. |
+| **Numeric column distributions** | Query min, max, avg, percentiles (p25, p50, p75, p95). These inform tier boundaries and detect outliers. |
+| **Categorical column cardinality** | Query distinct values. A `status` column with 5 values behaves differently from one with 500. |
+| **Column usefulness** | Query NULL rates. Columns that are >95% NULL are candidates for `internal`. |
+| **Join cardinality** | Query FK uniqueness: `group_by: fk_col, aggregate: row_count is count(), having: row_count > 1`. Determines `join_one` vs `join_many`. |
+| **Revenue/amount columns** | When multiple money columns exist (`total`, `subtotal`, `amount`, `price`), query a sample to understand how they relate (does `total = subtotal + tax`?). |
+| **Join key value compatibility** | For every proposed join, sample 5-10 actual values from each side. Check for format mismatches: abbreviations ("4th Av" vs "4 Avenue"), ordinals ("23 St" vs "23rd St"), casing, prefixes. Mismatched values mean the join won't work even if column names match. |
+| **Mixed-grain rows** | For each key table, run top-N and bottom-N by primary metric. Look for summary/aggregate rows mixed with detail data (e.g., "System Total" rows in a station-level table). These corrupt measures if not filtered out. |
+
+**Never assume from column names.** Always query the data to confirm. A column named `total` could include or exclude tax. A `status` column could have unexpected values. A FK could have orphaned references.
+
+### Example Queries
+
+**Tier boundaries**: query distribution, propose breaks from percentiles:
+```malloy
+run: orders -> {
+  aggregate:
+    min_val is min(sale_price), p25 is sale_price.percentile(25)
+    median_val is sale_price.percentile(50), p75 is sale_price.percentile(75)
+    p95 is sale_price.percentile(95), max_val is max(sale_price)
+}
+```
+
+**Denormalized vs joined**: compare pre-computed column against real aggregate, report match rate:
+```malloy
+run: customers -> {
+  join_many: orders on customer_id = orders.customer_id
+  aggregate:
+    total is count()
+    match is count() { where: order_count = count(orders.order_id) }
+}
+```
+
+**Canonical date**: when multiple date columns exist, check how often they differ:
+```malloy
+run: orders -> {
+  aggregate:
+    total is count()
+    same_date is count() { where: created_at::date = submitted_at::date }
+    max_gap_days is max(days(submitted_at - created_at))
+}
+```
+
+**Revenue columns**: when multiple money columns exist, verify their relationship:
+```malloy
+run: orders -> {
+  aggregate:
+    total_eq_parts is count() { where: abs(sale_price - (subtotal + tax)) < 0.01 }
+    total is count()
+}
+```
+
+### Schema Shape
+- Is this a star/snowflake schema (use base + joined source layers) or normalized/ER-style (may need 3-stage pattern)?
+- Combined vs split tables: prefer filtered/split tables over combined when both exist.
+
+## Computed Source Detection
+
+Flag potential computed sources when:
+
+1. **Grain mismatch**: the analytical scope requires a grain that no physical table provides (e.g., customer-level metrics from an order-grain table)
+2. **Repeated aggregation patterns**: the same GROUP BY + aggregate pattern would be needed in multiple analyses
+3. **Cross-entity aggregations**: the model or the data implies cross-entity aggregations that require a pre-aggregated entity
+
+## Prior Art Detection
+
+Check for prior art signals at the start of discovery. If a signal is found and the user confirms, **you MUST read** the corresponding reference skill and follow its instructions.
+
+| Signal | Source Type | Reference to Read |
+|--------|------------|-------------------|
+| `.lkml` files in project or subdirectories | lookml | `skill:malloy-lookml-review` |
+| `dbt_project.yml` in project or parent dirs | dbt | dbt review (future) |
+
+The reference handles inventory, classification, and produces prior-art notes. Keep those notes in-conversation, then continue with normal discovery below.
+
+**If DB connection available (LookML + DB mode):**
+- Read the model and run `execute_query` as normal
+- Use prior art as additional context, not a replacement for data validation
+- **The LookML connection name is NOT the Malloy connection name.** Always use the connection name from the model.
+
+**If no DB connection (LookML-only mode):**
+- Skip the model-read and `execute_query` steps
+- Use connection name and table paths extracted from prior art source files
+- Flag all proposals in Steps 2-4 as **unvalidated**
+- Proceed directly to Step 2 (PROPOSE SCOPE)
+
+**Prior art findings enhance discovery, they don't replace it.** When a DB connection is available, always validate assumptions against the actual data.
+
+## After Discovery
+
+Do NOT present findings to the user yet.
+
+## Done
+
+Step complete. Output: discovery findings (internal: tables, columns, relationships, data quality, prior art). Continue to the next modeling step (see your modeling workflow).
+
+## Verify Source Joins
+
+When reading joins off the model or the data, watch for `join_many` where the actual relationship is many-to-one. Always verify cardinality. Prefer `join_one` when each row in the primary table matches at most one row in the joined table.

package/skills/malloy-document/SKILL.md

@@ -0,0 +1,165 @@
+---
+name: malloy-document
+description: Add documentation with #(doc) tags to Malloy models so fields and sources are described in plain language. Use when user asks to "add documentation", "add doc tags", "document the model", or wants fields and sources described for natural-language search and discovery. For declaring parameterizable filters with #(filter), see the malloy-model skill. Filters are a runtime/modeling construct (governance, latency, correctness), not a documentation tag.
+---
+
+# Documenting a Malloy Model
+
+Add `#(doc)` tags to describe sources and fields in plain language so they are easy to find and understand:
+
+| Tag | Purpose | Goes on |
+|-----|---------|---------|
+| `#(doc)` | Plain-language description for natural-language search | source, dimension, measure, view, join |
+| `#(filter)` | Declare a parameterizable filter (runtime/modeling concern, see `malloy-model`) | source |
+
+`#(doc)` is a standard Malloy annotation. It documents a field or source with a human-readable description that downstream tools can surface and search against.
+
+## #(doc) Tag
+
+Add before any source, dimension, measure, view, or join. When multiple fields share a keyword, use it once as a block header. Tags and field names are indented under the keyword; tags go on the line(s) directly above the field they annotate.
+
+**Tag ordering** (when a field has multiple tags): `#(doc)` → render tags (`# currency`, `# label`, etc.) → field name. Separate each field group with a blank line:
+
+```malloy
+#(doc) Customer who placed the order
+join_one: users with user_id
+
+dimension:
+  #(doc) Date the order was placed (UTC)
+  order_date is created_at::date
+
+measure:
+  #(doc) Total revenue from all orders in USD
+  # currency
+  revenue is sum(total)
+```
+
+### Writing Doc Strings for Retrieval
+
+Doc strings power natural-language search: users type plain-English questions and the system matches against your `#(doc)` strings. Write descriptions that match how analysts would search:
+
+- **Include business meaning**, not code mechanics: what it represents, not how it's implemented
+- **Include units** (USD, count, percentage) and valid values for categorical fields
+- **Avoid Malloy jargon**: never use "filterable", "groupable", "dimension", "measure", "aggregation"
+
+**Good examples:**
+- `#(doc) Total revenue from completed orders in USD` matches "what was our revenue?"
+- `#(doc) Customer signup date (UTC)` matches "when did the customer join?"
+- `#(doc) Order status: pending, processing, shipped, delivered, cancelled` matches "what are the order statuses?"
+
+**Bad examples:**
+- `#(doc) Filterable dimension for order status`: no analyst searches for "filterable"
+- `#(doc) Groupable by region`: "groupable" is a system concept
+- `#(doc) Aggregation of total sales`: "aggregation" doesn't match natural queries
+
+## #(filter): see `malloy-model`
+
+`#(filter)` is also a `#(...)`-shaped annotation, but unlike `#(doc)` it's a **runtime/modeling construct**: it shapes governance, query latency, and correctness, not discoverability. The full reference (syntax, filter types, `required` / `implicit` flags, and when each applies) lives in `malloy-model` § Parameterizable Filters with `#(filter)` alongside the other source-authoring constructs.
+
+One rule worth knowing here: filters live on the source, never on the consumer. Ad-hoc reports and notebooks that import a source inherit its filters automatically; they do not (and cannot) declare new ones.
+
+## `internal:` and `private:`: column-level access in a source
+
+`#(doc)` describes what's exposed. Two access modifiers control what's exposed in the first place, and both live **inside** a source's `include {}` block. They are about the source's public API and data sensitivity, not about documentation, so reach for them when curating which columns callers can pick.
+
+| Mechanism | Layer | Why you reach for it |
+|---|---|---|
+| `internal:` | Inside a source (one column in `include {}`) | The column **isn't part of your model's public API**. Common reasons: data is messy (empty/garbage, raw JSON, duplicates), or a documented derived dimension already supersedes it, or the raw column exists only to be joined on / referenced internally and shouldn't appear as a dimension callers can pick. The data may be perfectly fine, it's just not what you want exposed. |
+| `private:` | Inside a source (one column in `include {}`) | The **data is sensitive**: SSN, raw credit card, password. Governance / security concern; a harder block than `internal:`. |
+
+In one sentence: **`internal:` and `private:` shape what's inside a source's public API; `#(doc)` describes the fields you do expose.**
+
+### Example
+
+A base source pulled from a messy raw table often uses `internal:` to drop raw fields from the public API, while documenting the curated columns with `#(doc)`.
+
+```malloy
+// orders_base.malloy
+#(doc) Raw orders. Use orders.malloy as the entry point for analysis.
+source: orders_base is conn.table('orders_raw')
+  include {
+    public: id, customer_id, order_date, total
+    internal: raw_json_payload, deprecated_status_code, _temp_dedup_marker
+  }
+  extend {
+    primary_key: id
+  }
+```
+
+```malloy
+// orders.malloy
+import "orders_base.malloy"
+
+#(doc) Order analysis. Use for revenue, fulfillment, and customer-order joins.
+source: orders is orders_base extend {
+  // joins, measures, curated dimensions
+}
+```
+
+The base source stays fully queryable (`run: orders_base -> { ... }` still works); `internal:` only governs which columns appear as public dimensions callers can pick.
+
+## Annotating Columns in Include (Experimental)
+
+With `##! experimental.access_modifiers`, you can add `#(doc)` tags to raw table columns inside `include` blocks. This documents columns without redefining them as dimensions.
+
+```malloy
+##! experimental.access_modifiers
+
+source: orders is conn.table('orders') include {
+  public:
+    #(doc) Order line item identifier
+    id
+
+    #(doc) Customer email address
+    email
+
+    #(doc) Order status: pending, shipped, delivered
+    status
+
+  // internal: only for verified noise (empty cols, raw JSON blobs, duplicates)
+}
+extend {
+  // ... dimensions and measures
+}
+```
+
+**When to use:**
+- Documenting raw columns without creating explicit dimensions
+- Curating which columns are public vs internal
+
+## Source-Level Documentation
+
+Document **when to use** a source, not what it contains. Dimensions and measures can already be searched directly, so the source-level `#(doc)` should describe what questions/analyses this source answers.
+
+**Base source files:** Document what the table represents.
+```malloy
+#(doc) Customer records with demographics and segmentation. One row per customer.
+source: customers is conn.table('sales.customers') extend { ... }
+```
+
+**Source files:** Document what analytical questions the source answers.
+```malloy
+#(doc) Customer health analysis. Use for retention, segmentation, churn risk, and lifetime value. For order-level analysis, use order_analysis instead.
+source: customer_health is customers extend { ... }
+```
+
+**Best practices:**
+- Add `#(doc)` to all base source and joined source definitions
+- Base source docs: describe what the table is (one row per what)
+- Source docs: describe what questions/analyses the source answers
+- Documentation happens per-source-file, not in one monolithic file
+
+## Flag Ambiguous Descriptions
+
+After writing `#(doc)` tags, present any that required judgment to the user for confirmation:
+
+| Field | Proposed doc | Confidence | Uncertainty |
+|-------|-------------|------------|-------------|
+| `total` | "Total order amount in USD" | Medium | Could be gross or net, verified with sample query |
+| `status` | "Order status: pending, shipped, delivered" | High | Values confirmed via a query of distinct values |
+
+Only flag fields where the description required assumptions about business meaning, units, or valid values. When in doubt about valid values, run a quick query against the data to confirm them before writing the description. Use `malloy_getContext` to ground yourself in the package's sources and fields and `malloy_executeQuery` to check distinct values, for example `run: source -> { group_by: status }`.
+
+## Done
+
+Step complete. Output: `#(doc)` tags added to all public fields and sources.