coalesce-transform-mcp 0.1.0

package/dist/resources/context/node-payloads.md

@@ -0,0 +1,114 @@
+# Node Payloads
+
+Use this resource when reading or writing full workspace node bodies.
+
+## Core Rule
+
+Treat workspace node bodies as structured objects with important differences between top-level fields, `metadata`, `config`, and arrays.
+
+## Common Payload Areas
+
+### Top-Level Fields
+
+Common top-level fields include:
+- `id`
+- `name`
+- `description`
+- `nodeType`
+- `database`
+- `schema`
+- `locationName`
+- `storageLocations`
+- `config`
+- `metadata`
+
+Not every node type uses every field.
+
+Project policy:
+- Do not send `overrideSQL`
+- Do not send `override.*`
+- If a node definition mentions `overrideSQLToggle`, treat it as disallowed and omit it from writes
+
+### `metadata`
+
+`metadata` usually contains node-shape details such as:
+- `columns`
+- lineage/source structures
+- mapping-related structures
+
+This is where many node-specific editing mistakes happen.
+
+### `config`
+
+`config` is node-type-specific operational configuration.
+
+Examples include:
+- `preSQL`
+- `postSQL`
+- `insertStrategy`
+- node-type-specific dropdown or tabular settings
+
+Do not assume the same config keys exist across node types.
+
+### `storageLocations` and top-level location fields
+
+Some nodes use `storageLocations`.
+Some also expose location fields at the top level such as:
+- `database`
+- `schema`
+- `locationName`
+
+Verify the saved node body instead of assuming one location representation is always present.
+
+## Array Safety
+
+Arrays are replace-on-write unless the tool says otherwise.
+
+This is especially important for:
+- `metadata.columns`
+- other metadata arrays
+- `storageLocations`
+
+If you send only the new array items, you will usually replace the old array.
+
+## Tool Guidance
+
+For tool selection and helper preference guidance, see `coalesce://context/node-creation-decision-tree`.
+
+### Prefer `update-workspace-node`
+
+Use it for most partial node edits because it:
+- reads the current node first
+- deep-merges object fields
+- writes the merged full body back
+
+### Use `set-workspace-node` carefully
+
+Use `set-workspace-node` only when:
+- you intend a full replacement
+- you already have the complete final node body
+
+### Scratch creation
+
+For scratch creation guidance (use cases, completion levels, expected fields), see `coalesce://context/pipeline-workflows`.
+
+## Validation Rules
+
+After helper-based writes, inspect:
+- `validation`
+- `warning`
+
+Do not assume a node is fully ready unless the helper says the requested completion was satisfied.
+
+## Rename Safety
+
+If a node name changes:
+- verify the saved node body still looks correct
+- verify related mapping/source fields still make sense
+- do not assume downstream references were updated automatically
+
+## Related Resources
+
+- `coalesce://context/hydrated-metadata`
+- `coalesce://context/storage-mappings`
+- `coalesce://context/node-creation-decision-tree`

package/dist/resources/context/node-type-corpus.md

@@ -0,0 +1,166 @@
+# Node Type Discovery and Corpus
+
+## Prefer Repo-Backed Discovery First
+
+**BEFORE creating or editing nodes:**
+1. Determine which node types are already observed in the workspace
+2. If a local committed repo is available, use repo-backed discovery with explicit `repoPath` or the `COALESCE_REPO_PATH` fallback
+3. Use the corpus only when the repo is unavailable or lacks the committed definition
+
+Repo-backed workflow:
+- Install the package in Coalesce
+- Commit the workspace branch
+- Update the local repo clone to that branch
+- Use `list-repo-packages`, `list-repo-node-types`, `get-repo-node-type-definition`, or `generate-set-workspace-node-template`
+- Prefer explicit `repoPath` on repo-aware calls; when omitted, tools fall back to `COALESCE_REPO_PATH`
+- There is no server-wide repo mapping in v1
+
+## Repo-Aware Tools
+
+### `list-repo-packages`
+Inspect committed `packages/*.yml`
+Use this to discover exact package aliases and see which enabled node-type IDs have committed definitions
+
+### `list-repo-node-types`
+List exact resolvable node-type identifiers from committed `nodeTypes/`
+Use this to confirm the exact direct identifier or `alias:::id` value before generating templates
+
+### `get-repo-node-type-definition`
+Resolve one exact node type from the committed repo
+Returns the parsed outer definition plus raw and parsed `metadata.nodeMetadataSpec`
+
+### `generate-set-workspace-node-template`
+Generate a `set-workspace-node` body template from either:
+- a raw definition object
+- a committed repo definition resolved by `repoPath` + `nodeType`
+
+In repo mode, this preserves the exact resolved node type, including package-backed identifiers like `alias:::id`
+
+## Corpus Tools
+
+### `search-node-type-variants`
+Search for node type families (e.g., "stage", "dimension")
+Returns variants with their config schema and metadata examples
+
+### `get-node-type-variant`
+Get exact variant definition by key
+Use this to get the authoritative structure for a node type
+
+### `list-workspace-node-types` *(NEW)*
+Scan workspace nodes and return distinct observed node types
+Use this to inspect current workspace usage and exact identifiers before recommending
+The response includes `basis: "observed_nodes"` to make the contract explicit
+
+## Node-Type-Specific Patterns
+
+Project policy:
+- Never recommend or set SQL override fields
+- Ignore or remove `overrideSQLToggle`, `overrideSQL`, and `override.*` when reading node-type definitions
+- Prefer native node configuration and metadata patterns instead of SQL override
+
+The corpus contains node-type-specific patterns and configurations. These vary by:
+- Package source (Coalesce built-in vs community packages)
+- Node type family (Stage, Dimension, Fact, View, etc.)
+- Package version and variant
+
+**When to consult corpus patterns:**
+- Creating nodes with complex metadata (sources, joins, config)
+- Applying node-type-specific logic (materialization, caching, partitioning)
+- Understanding required vs optional config fields
+- Adapting SQL patterns for specific node types
+
+The corpus provides real-world examples from actual node type source code, including:
+- Metadata structure (columns, sources, mappings)
+- Config field schemas and valid values
+- SQL patterns and Jinja syntax specific to that node type
+- Storage location and reference patterns
+
+## Column-Level Attributes (columnSelector)
+
+Node type definitions contain config items with `"type": "columnSelector"`. These are **column-level attributes** — boolean flags set directly on individual column objects in `metadata.columns`, NOT in the node-level `config` object.
+
+**How it works:**
+
+1. The node type definition (in `nodeTypes/` in the local repo) has a config item like:
+   ```json
+   { "displayName": "Business Key", "attributeName": "isBusinessKey", "type": "columnSelector" }
+   ```
+2. To activate it, set the `attributeName` as a boolean on the column:
+   ```json
+   { "name": "CUSTOMER_ID", "dataType": "NUMBER(38,0)", "isBusinessKey": true, "transform": "..." }
+   ```
+3. The node type's Jinja templates reference it: `columns | selectattr('isBusinessKey')`
+
+**Discovery workflow:**
+
+1. Use `get-repo-node-type-definition` to read the node type definition from the local repo
+2. Find config items where `type` is `"columnSelector"`
+3. The `attributeName` field tells you what property to set on columns
+4. Set `attributeName: true` on the appropriate columns via `update-workspace-node` or `replace-workspace-node-columns`
+
+**Important:** Attribute names vary by node type and package. Always look them up in the actual node type definition — do not guess or hardcode attribute names.
+
+## Workflow
+
+1. **Discover observed types:** `list-workspace-node-types`
+2. **Check if recommended type is already observed:** Compare to workspace types
+3. **If unobserved:** Do not claim it is unavailable; confirm installation in the UI before proceeding
+4. **If a committed local repo is available:** Use repo-aware tools with `repoPath` or `COALESCE_REPO_PATH`
+5. **If repo resolution fails or the definition is missing:** Use `search-node-type-variants` and `get-node-type-variant`
+6. **Adapt example:** Replace placeholder values with user-specific data
+7. **Create/update:** Use appropriate tool with correct structure
+
+## Node Type Availability
+
+If a recommended node type is not observed in current workspace nodes:
+1. Do not claim the type is unavailable
+2. Explain that the workspace scan only reflects existing nodes, not a true installed-type registry
+3. Confirm package installation in the Coalesce UI when the exact availability is uncertain
+4. Prefer repo-backed discovery to recover the exact identifier before proceeding
+
+## Node Type Format
+
+Node types can appear in two formats:
+
+1. **Simple format:** Direct node type names without package prefix
+   - Examples: `"Stage"`, `"persistentStage"`, `"View"`
+   - Used for built-in node types or custom node types in the repo
+
+2. **Package-prefixed format:** Package name followed by `:::` and node type ID
+   - Examples: `"IncrementalLoading:::230"`, `"Databricks-Incremental-nodes:::278"`
+   - Used for package-installed node types from published packages
+   - The format is: `"PackageName:::NodeTypeID"`
+
+### Creating Nodes with Package-Prefixed Types
+
+When creating nodes, you can use **either format**:
+
+- Full format: `nodeType: "IncrementalLoading:::230"`
+- Bare ID: `nodeType: "230"` (will match any package with that ID)
+
+Prefer the full package-prefixed format when you know it. Using just the numeric ID (e.g., `"230"`) is safest when a matching package-prefixed type is already observed in workspace nodes.
+
+### Discovering Node Type Format
+
+Use `list-workspace-node-types` to see the exact format of observed node types. The response will show package-prefixed types like `"IncrementalLoading:::230"` when those identifiers are already present in current workspace nodes.
+
+## Template Usage
+
+### Template Generation Hierarchy
+
+1. **For committed local repo definitions:** Use `generate-set-workspace-node-template` in repo mode
+   - Resolves the exact committed definition from `repoPath` or `COALESCE_REPO_PATH`
+   - Preserves exact direct or package-backed node-type identifiers
+   - Best choice when the local repo contains the definition
+
+2. **For known corpus variants:** Use `get-node-type-variant`
+   - Returns exact variant definition from the committed corpus snapshot
+   - Best fallback when repo-backed resolution is unavailable
+
+3. **For generating templates from corpus variants:** Use `generate-set-workspace-node-template-from-variant`
+   - Converts a corpus variant into an editable YAML-friendly template
+   - Use this when the repo does not contain the committed definition
+
+4. **For discovery:** Search first, then get variant
+   - Use `search-node-type-variants` to find the right fallback variant
+   - Then use `get-node-type-variant` or `generate-set-workspace-node-template-from-variant`

package/dist/resources/context/node-type-selection-guide.md

@@ -0,0 +1,96 @@
+# Node Type Selection Guide
+
+When creating pipeline nodes, choose the node type based on the actual purpose of the node — not the SQL pattern.
+
+## General-Purpose Node Types (use for most transforms)
+
+### Stage / Work
+- **Purpose**: General-purpose intermediate processing. Handles single-source, multi-source joins, GROUP BY, UNION, filters, transforms. These are interchangeable for most patterns.
+- **Materialization**: Table or View
+- **Use for**: Column renames, type casts, WHERE filters, GROUP BY aggregations, multi-table JOINs, UNION/UNION ALL, any transformation without special requirements
+- **Default choice**: When in doubt, use Stage or Work
+
+### View
+- **Purpose**: Virtual table with no physical storage. Recalculates on every query.
+- **Materialization**: View only
+- **Use for**: Lightweight projections, secure views, cost savings when recomputation is OK
+- **Avoid when**: Downstream queries are performance-critical or aggregations are expensive
+
+## Dimensional Modeling Node Types (only when explicitly building a dimensional model)
+
+### Dimension
+- **Purpose**: Descriptive business context (customers, products, locations). Requires business keys. Supports SCD Type 1/2.
+- **Materialization**: Table or View
+- **Use ONLY when**: Building a star/snowflake schema, node is named dim_ or dimension_, SCD tracking is needed
+- **NOT for**: Generic GROUP BY, aggregations, staging, or CTE decomposition
+
+### Fact
+- **Purpose**: Business measures (revenue, quantity, cost) at a defined grain. Requires business keys.
+- **Materialization**: Table or View
+- **Use ONLY when**: Building a fact table in a dimensional model, node is named fct_ or fact_
+- **NOT for**: Any GROUP BY or SUM — those are transforms, not fact tables
+
+### Factless Fact
+- **Purpose**: Record events without numeric measures (attendance, eligibility)
+- **Use ONLY when**: Event tracking without measures in a dimensional model
+
+## Change Tracking Node Types
+
+### Persistent Stage
+- **Purpose**: CDC / change tracking with business keys. Type 1/Type 2 history.
+- **Materialization**: Table only
+- **Use ONLY when**: Goal explicitly mentions CDC, change tracking, or history tracking
+- **NOT for**: Simple staging, general transforms, CTE decomposition
+
+## Data Vault Node Types
+
+### Hub / Satellite / Link
+- **Use ONLY when**: Explicitly building a Data Vault model
+- **NOT for**: General-purpose joins or transforms
+
+## Specialized Materialization Patterns (only when explicitly requested)
+
+### Dynamic Tables
+- **Purpose**: Snowflake-managed declarative refresh with lag-based orchestration
+- **Use when**: Near-real-time / continuous refresh, streaming-like pipelines, replacing complex Streams+Tasks
+- **NOT for**: Batch ETL, scheduled runs, CTE decomposition, cost-sensitive workloads
+- **Key difference**: Snowflake manages the refresh DAG automatically. You specify a lag (e.g., 5 minutes) and Snowflake keeps data fresh within that window. This adds continuous compute cost.
+
+### Incremental Load
+- **Purpose**: Process only new/modified records via high-water mark comparison
+- **Use when**: Large tables where full refresh is too expensive, append-only sources
+- **NOT for**: Full-refresh staging, CTE decomposition, small-to-medium tables
+
+### Materialized View
+- **Purpose**: Pre-computed aggregations that auto-refresh when base data changes
+- **Use when**: Expensive aggregations that need to stay current, single-source only
+- **NOT for**: Multi-source joins, standard transforms
+
+### Deferred Merge
+- **Purpose**: Snowflake Streams + scheduled merge tasks for high-frequency ingestion
+- **Use when**: High-frequency data ingestion where immediate merge is too expensive
+- **NOT for**: Batch ETL, standard staging
+
+### Tasks / DAG
+- **Purpose**: Snowflake scheduled or DAG-based orchestration
+- **Use when**: Building scheduled task workflows
+- **NOT for**: Data transformation nodes
+
+## Decision Rules
+
+1. **CTE decomposition**: Each CTE becomes a Stage or Work node. Never Dimension, Fact, or specialized types unless the user explicitly names it that way.
+2. **GROUP BY / SUM / COUNT**: These are transforms. Use Stage/Work. Only use Fact/Dimension if building a dimensional model.
+3. **Multi-source JOIN**: Use Stage or Work — both handle joins via sourceMapping.
+4. **Name prefix**: `stg_` → Stage, `wrk_` → Work, `dim_` → Dimension, `fct_` → Fact, `vw_` → View. Follow the prefix.
+5. **No explicit requirement**: Default to Stage for single-source, Work for multi-source.
+6. **Specialized types require explicit user request**: Dynamic Tables, Incremental Load, Materialized View, Deferred Merge, and Tasks are specialized materialization patterns. **NEVER auto-select these.** The user must explicitly ask for continuous refresh, incremental processing, or the specific pattern by name.
+
+## Common Mistakes
+
+| Mistake | Why It's Wrong | Correct Approach |
+|---------|---------------|-----------------|
+| Using Dynamic Tables for batch ETL | Dynamic Tables add continuous compute cost and are for near-real-time refresh | Use Stage or Work with table materialization |
+| Picking node type by ID (e.g., "65") without `plan-pipeline` | The agent doesn't know what the ID maps to or if it's appropriate | Always call `plan-pipeline` first |
+| Using Dimension/Fact for GROUP BY | GROUP BY is a transform, not a dimensional model | Use Stage or Work |
+| Skipping `plan-pipeline` and guessing "Stage" | May miss better options or select an inappropriate type | Always call `plan-pipeline` with `repoPath` |
+| Using Incremental Load for small tables | Incremental overhead isn't worth it for fast full refreshes | Use Stage with full refresh |

package/dist/resources/context/overview.md

@@ -0,0 +1,135 @@
+# Coalesce MCP Server Overview
+
+## How This Server Works
+
+This MCP server provides tools and resources for working with Coalesce Transform workspaces and environments via the Coalesce API.
+
+### Available Resources
+
+Consult these resources for specific guidance. Load only what the current task needs.
+
+- **coalesce://context/pipeline-workflows** — Building pipelines, node type selection, multi-node sequences, incremental setup
+- **coalesce://context/node-operations** — Editing nodes: join conditions, columns, config, renames, SQL conversion
+- **coalesce://context/node-creation-decision-tree** — Routing: which tool to use for creation vs update
+- **coalesce://context/data-engineering-principles** — Architecture, platforms, methodology, materialization, packages
+- **coalesce://context/aggregation-patterns** — GROUP BY, datatype inference, common aggregation patterns
+- **coalesce://context/node-type-corpus** — Node type discovery, corpus search, metadata patterns
+- **coalesce://context/tool-usage** — Core tool rules, discovery patterns, pagination, parallelization
+- **coalesce://context/sql-platform-selection** — Determine the active SQL platform
+- **coalesce://context/sql-snowflake** — Snowflake SQL conventions
+- **coalesce://context/sql-databricks** — Databricks SQL conventions
+- **coalesce://context/sql-bigquery** — BigQuery SQL conventions
+- **coalesce://context/storage-mappings** — `{{ ref() }}` syntax and storage locations
+- **coalesce://context/id-discovery** — Resolving project, workspace, environment, and node IDs
+- **coalesce://context/node-payloads** — Full node body editing guidance
+- **coalesce://context/hydrated-metadata** — Hydrated metadata structures
+- **coalesce://context/run-operations** — Start, retry, diagnose, and cancel runs
+- **coalesce://context/intelligent-node-configuration** — Intelligent config completion
+
+### Response Guidelines
+
+- Answer questions directly without preamble
+- Use fenced code blocks with language tags
+- Provide detail when requested, brevity otherwise
+- For multi-step pipeline creation, report progress after each node
+
+## How Coalesce Nodes Work
+
+**CRITICAL**: Coalesce nodes are NOT SQL scripts. They are declarative configurations with these components:
+
+| Component | Where it lives | What it does |
+|-----------|---------------|--------------|
+| **Columns** | `metadata.columns[].transform` | Each column has a SQL expression (e.g., `"ORDERS"."CUSTOMER_ID"`, `SUM("ORDERS"."TOTAL")`) |
+| **Join condition** | `metadata.sourceMapping[].join.joinCondition` | The FROM/JOIN/WHERE/GROUP BY clause using `{{ ref() }}` syntax |
+| **Dependencies** | `metadata.sourceMapping[].dependencies` | Which upstream nodes this node reads from |
+| **Config** | `config` | Node-type-specific settings (truncate, business keys, SCD, etc.) |
+
+The node type's Jinja template combines these into final SQL at compile time. You configure **columns and joins separately** — you never write a complete SQL query.
+
+**Example — a CLV aggregation node:**
+
+```text
+Columns:
+  - CUSTOMER_ID: transform = "CUSTOMER_LOYALTY"."CUSTOMER_ID"
+  - TOTAL_ORDERS: transform = COUNT(DISTINCT "ORDER_HEADER"."ORDER_ID")
+  - LIFETIME_VALUE: transform = SUM("ORDER_HEADER"."ORDER_TOTAL")
+
+joinCondition:
+  FROM {{ ref('STAGING', 'CUSTOMER_LOYALTY') }} "CUSTOMER_LOYALTY"
+  LEFT JOIN {{ ref('STAGING', 'ORDER_HEADER') }} "ORDER_HEADER"
+    ON "CUSTOMER_LOYALTY"."CUSTOMER_ID" = "ORDER_HEADER"."CUSTOMER_ID"
+  GROUP BY "CUSTOMER_LOYALTY"."CUSTOMER_ID"
+```
+
+**Key rules:**
+
+- Column transforms reference **predecessor table aliases** from the joinCondition
+- GROUP BY goes inside the joinCondition, after JOIN clauses
+- Aggregate functions (COUNT, SUM, AVG) go in column transforms, NOT in joinCondition
+- CASE expressions go in column transforms
+- CTEs are NOT supported — break into separate upstream nodes
+- Do NOT write `overrideSQL` or pass raw SQL — use the column/joinCondition model
+
+**Required workflow for creating nodes:**
+
+1. **Always call `plan-pipeline` first** — it discovers all available node types from the repo and ranks them for your use case. Do NOT guess node types like "Stage" or "View".
+2. **Use `create-workspace-node-from-predecessor`** (or `create-workspace-node-from-scratch` for nodes with no upstream). Pass `repoPath` for automatic config completion.
+3. **Use `create-workspace-node-from-predecessor`** (or `create-workspace-node-from-scratch` for nodes with no upstream) — they handle validation, config completion, and column-level attributes automatically.
+
+Config completion is automatic when `repoPath` is provided — the response includes `configCompletion` showing what node-level config and column-level attributes were applied.
+
+**Post-creation verification (required before moving to the next node):**
+
+1. Check `nextSteps` in the creation response — follow all required steps (especially join setup for multi-predecessor nodes)
+2. Check `validation.allPredecessorsRepresented` — if false, the join is incomplete
+3. For multi-predecessor nodes: set up the join condition via `convert-join-to-aggregation` (aggregation), `apply-join-condition` (row-level joins), or `update-workspace-node` (manual)
+4. Verify the final node with `get-workspace-node` — confirm columns, joinCondition, and config are correct
+5. Follow naming conventions: STG_ for staging, DIM_ for dimensions, FACT_ for facts, INT_ for intermediate (e.g., `STG_LOCATION`, `FACT_ORDERS`). Default to UPPERCASE for Snowflake, but **respect the user's chosen casing**
+
+**Anti-pattern — writing SQL and passing it to the planner:**
+
+Do NOT author SQL yourself to pass to `plan-pipeline` or `create-pipeline-from-sql`. The `sql` parameter exists solely for converting SQL that the **user** provided. When building a pipeline, use declarative tools:
+
+1. `create-workspace-node-from-predecessor` to create nodes
+2. `update-workspace-node` to set joinCondition
+3. `replace-workspace-node-columns` or `convert-join-to-aggregation` for column transforms
+
+### Writing SQL for Nodes
+
+Determine the platform first, then load exactly one dialect resource:
+
+1. **Detect**: `get-project` for warehouse type, or check existing node SQL (see `coalesce://context/sql-platform-selection`)
+2. **Load one**: Snowflake -> `coalesce://context/sql-snowflake`, Databricks -> `coalesce://context/sql-databricks`, BigQuery -> `coalesce://context/sql-bigquery`
+3. **Follow that dialect's rules** for the entire edit
+
+## Operational Scope
+
+### In Scope
+
+- Creating/updating workspace nodes
+- Reasoning from project metadata and workspace patterns
+- Writing SQL transforms and joins
+- Running jobs and monitoring runs
+- Managing environments and projects
+
+### Out of Scope
+
+- **Previewing compiled SQL** — compilation happens at deploy/run time
+- **Changing node type** — set at creation time, cannot be changed via API. Create a new node of the desired type instead
+- **Data preview / row counts** — this server manages node definitions, not live warehouse data
+- **Cross-workspace replication** — no clone tool; recreate each node bottom-up in the target workspace
+- Creating/modifying node type templates, source nodes, or macros
+
+### Description Generation
+
+Only generate descriptions when explicitly requested. Focus on disambiguation — what makes this column the one the user is looking for? If you lack context, ask.
+
+Apply all descriptions in one call using `replace-workspace-node-columns`.
+
+## Documentation
+
+- [General Coalesce Docs](https://docs.coalesce.io/docs)
+- [API Documentation](https://docs.coalesce.io/docs/api)
+- [Snowflake Base Node Types](https://docs.coalesce.io/docs/marketplace/package/coalesce_snowflake_base-node-types)
+- [Incremental Loading](https://docs.coalesce.io/docs/marketplace/package/coalesce_snowflake_incremental-loading)
+- [Dynamic Tables](https://docs.coalesce.io/docs/marketplace/package/coalesce_snowflake_dynamic-tables)