coalesce-transform-mcp 0.1.0

package/dist/resources/context/tool-usage.md

@@ -0,0 +1,98 @@
+# Tool Usage Patterns
+
+## Core Rules
+
+- Only call tools that actually exist in this MCP server. Do not invent tool names.
+- Prefer the smallest sufficient tool: list for discovery, get for detail, workflow helpers for multi-step operations.
+- If the request is exploratory or ambiguous, ask clarifying questions before mutating anything.
+- Before mutating a node, read the current state unless a helper already does that for you.
+- Treat arrays as full-replacement fields unless a tool explicitly documents merge semantics. This is especially important for `metadata.columns`.
+- When a helper returns `validation`, `warning`, `resultsError`, or `incomplete`, inspect those fields before continuing.
+- Before declaring work complete, verify real outcomes by re-reading the saved node and checking `validation` fields.
+
+## Discovery Patterns
+
+### Find Projects and Workspaces
+
+- `list-projects` or `get-project` to discover project structure
+- For workspace IDs: `list-projects({ includeWorkspaces: true })` or `get-project({ projectID, includeWorkspaces: true })`
+- For broader ID resolution guidance, see `coalesce://context/id-discovery`
+
+### List vs Get
+
+- **List** endpoints for discovery and broad state: `list-environments`, `list-projects`, `list-runs`, `list-workspace-nodes`, `list-environment-nodes`
+- **Get** endpoints when you have an ID and need the full object: `get-environment`, `get-project`, `get-run`, `get-workspace-node`, `get-environment-node`
+
+### Pagination
+
+- Large list endpoints may be paginated. Do not assume one page is complete if `next` or paging cursors are present.
+- Prefer discovery first, then targeted reads.
+- Large JSON responses may be auto-cached to `coalesce_transform_mcp_data_cache/auto-cache/`; the tool returns cache metadata plus a `coalesce://cache/...` resource URI and `resource_link` instead of the full payload.
+- Use explicit cache tools for large lists: `cache-workspace-nodes`, `cache-environment-nodes`, `cache-runs`, `cache-org-users`.
+
+### Large Workspace Analysis
+
+- `analyze-workspace-patterns` for a compact workspace profile (naming, packages, methodology conventions)
+- `cache-workspace-nodes` when the full payload should be written to `coalesce_transform_mcp_data_cache/nodes/` for reuse
+- For architecture guidance, see `coalesce://context/data-engineering-principles`
+
+## Mandatory: Plan Before Creating Nodes
+
+**NEVER skip `plan-pipeline` before creating nodes.** Do not hardcode node types like "Stage", "65", or any other ID.
+
+1. Call `plan-pipeline` with `goal`, `sourceNodeIDs`, and `repoPath` — it discovers and ranks all node types
+2. Use the `nodeType` from the plan result when calling creation tools — the planner already excludes specialized types (Dynamic Table, Incremental, Materialized View, etc.) unless your context explicitly requires them
+
+**Common mistake**: Picking node types by ID or name without calling `plan-pipeline`. This leads to wrong types (e.g., Dynamic Tables for batch ETL) because the agent has no visibility into what types are available or appropriate.
+
+## Repo-Backed and Corpus Workflows
+
+Before creating nodes with complex metadata:
+
+1. Check observed types: `list-workspace-node-types` (see `coalesce://context/node-type-corpus`)
+2. If a local repo is available: install the package, commit, update clone, use `list-repo-node-types` / `get-repo-node-type-definition` / `generate-set-workspace-node-template` with `repoPath` or `COALESCE_REPO_PATH`
+3. If the repo lacks the definition: use `search-node-type-variants` / `get-node-type-variant` / `generate-set-workspace-node-template-from-variant`
+4. Adapt the pattern with user-specific data
+5. Create with `create-workspace-node-from-predecessor` or pipeline tools
+
+## Parallel vs Sequential Tool Use
+
+### Safe to Parallelize
+
+- Independent reads
+- Multiple list/get calls for unrelated objects
+- Discovery reads across several predecessor nodes
+
+### Usually Sequential
+
+- Dependent node creation (upstream before downstream)
+- Create then validate flows
+- Update flows where the next step depends on the saved body
+- Run lifecycle steps
+
+### Independent Writes
+
+Parallelize only when writes truly do not depend on each other and you verify each result afterward. When in doubt, prefer sequential.
+
+## Storage and SQL Context
+
+- Use the actual node body to determine storage/location fields when writing SQL references
+- Determine the platform with `coalesce://context/sql-platform-selection`, then read the matching dialect resource
+- Cross-check `{{ ref() }}` assumptions against `coalesce://context/storage-mappings`
+- For full-body edits, cross-check `coalesce://context/node-payloads` and `coalesce://context/hydrated-metadata`
+
+## Capabilities and Boundaries
+
+Your capabilities are defined by the tools in this MCP server:
+
+- **Tool names**: Use exact tool names as listed. If an operation is not available as a tool, it falls outside your scope.
+- **Source nodes**: Read-only. You can read them and use as predecessors, but cannot modify their columns or config.
+- **Array fields**: Always send the full accumulated array. The server replaces — it does not merge.
+- **Helper warnings**: `warning`, `validation`, and `resultsError` fields are actionable. A successful call with warnings may not be fully correct.
+- **Node readiness**: A node is fully usable only when `validation` confirms coverage and correctness.
+
+## Related Resources
+
+- `coalesce://context/pipeline-workflows` — building pipelines end-to-end
+- `coalesce://context/node-operations` — editing nodes after creation
+- `coalesce://context/node-creation-decision-tree` — which tool to use

package/dist/resources/index.d.ts

@@ -0,0 +1,5 @@
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+/**
+ * Register all Coalesce context resources with the MCP server
+ */
+export declare function registerResources(server: McpServer): void;

package/dist/resources/index.js

@@ -0,0 +1,254 @@
+import { ResourceTemplate } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { existsSync, readFileSync, readdirSync } from "fs";
+import { basename, join, dirname } from "path";
+import { fileURLToPath } from "url";
+import { buildCacheResourceUri, getCacheDir, getCacheResourceMimeType, resolveCacheResourceUri, } from "../cache-dir.js";
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = dirname(__filename);
+// Resource URIs
+const RESOURCES = {
+    OVERVIEW: "coalesce://context/overview",
+    SQL_PLATFORM_SELECTION: "coalesce://context/sql-platform-selection",
+    SQL_SNOWFLAKE: "coalesce://context/sql-snowflake",
+    SQL_DATABRICKS: "coalesce://context/sql-databricks",
+    SQL_BIGQUERY: "coalesce://context/sql-bigquery",
+    DATA_ENGINEERING_PRINCIPLES: "coalesce://context/data-engineering-principles",
+    STORAGE_MAPPINGS: "coalesce://context/storage-mappings",
+    TOOL_USAGE: "coalesce://context/tool-usage",
+    ID_DISCOVERY: "coalesce://context/id-discovery",
+    NODE_CREATION_DECISION_TREE: "coalesce://context/node-creation-decision-tree",
+    NODE_PAYLOADS: "coalesce://context/node-payloads",
+    HYDRATED_METADATA: "coalesce://context/hydrated-metadata",
+    RUN_OPERATIONS: "coalesce://context/run-operations",
+    NODE_TYPE_CORPUS: "coalesce://context/node-type-corpus",
+    AGGREGATION_PATTERNS: "coalesce://context/aggregation-patterns",
+    INTELLIGENT_NODE_CONFIGURATION: "coalesce://context/intelligent-node-configuration",
+    PIPELINE_WORKFLOWS: "coalesce://context/pipeline-workflows",
+    NODE_OPERATIONS: "coalesce://context/node-operations",
+    NODE_TYPE_SELECTION_GUIDE: "coalesce://context/node-type-selection-guide",
+};
+// Map URIs to file paths
+const RESOURCE_FILES = {
+    [RESOURCES.OVERVIEW]: "context/overview.md",
+    [RESOURCES.SQL_PLATFORM_SELECTION]: "context/sql-platform-selection.md",
+    [RESOURCES.SQL_SNOWFLAKE]: "context/sql-snowflake.md",
+    [RESOURCES.SQL_DATABRICKS]: "context/sql-databricks.md",
+    [RESOURCES.SQL_BIGQUERY]: "context/sql-bigquery.md",
+    [RESOURCES.DATA_ENGINEERING_PRINCIPLES]: "context/data-engineering-principles.md",
+    [RESOURCES.STORAGE_MAPPINGS]: "context/storage-mappings.md",
+    [RESOURCES.TOOL_USAGE]: "context/tool-usage.md",
+    [RESOURCES.ID_DISCOVERY]: "context/id-discovery.md",
+    [RESOURCES.NODE_CREATION_DECISION_TREE]: "context/node-creation-decision-tree.md",
+    [RESOURCES.NODE_PAYLOADS]: "context/node-payloads.md",
+    [RESOURCES.HYDRATED_METADATA]: "context/hydrated-metadata.md",
+    [RESOURCES.RUN_OPERATIONS]: "context/run-operations.md",
+    [RESOURCES.NODE_TYPE_CORPUS]: "context/node-type-corpus.md",
+    [RESOURCES.AGGREGATION_PATTERNS]: "context/aggregation-patterns.md",
+    [RESOURCES.INTELLIGENT_NODE_CONFIGURATION]: "context/intelligent-node-configuration.md",
+    [RESOURCES.PIPELINE_WORKFLOWS]: "context/pipeline-workflows.md",
+    [RESOURCES.NODE_OPERATIONS]: "context/node-operations.md",
+    [RESOURCES.NODE_TYPE_SELECTION_GUIDE]: "context/node-type-selection-guide.md",
+};
+// Resource metadata
+const RESOURCE_METADATA = {
+    [RESOURCES.OVERVIEW]: {
+        name: "Coalesce Overview",
+        description: "General Coalesce concepts, response guidelines, and operational constraints for AI assistants",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.SQL_PLATFORM_SELECTION]: {
+        name: "SQL Platform Selection",
+        description: "How to determine the active SQL platform from project metadata and existing node SQL before choosing a dialect-specific resource",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.SQL_SNOWFLAKE]: {
+        name: "SQL Rules: Snowflake",
+        description: "Snowflake-specific SQL conventions for Coalesce node SQL",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.SQL_DATABRICKS]: {
+        name: "SQL Rules: Databricks",
+        description: "Databricks-specific SQL conventions for Coalesce node SQL",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.SQL_BIGQUERY]: {
+        name: "SQL Rules: BigQuery",
+        description: "BigQuery-specific SQL conventions for Coalesce node SQL",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.DATA_ENGINEERING_PRINCIPLES]: {
+        name: "Data Engineering Principles",
+        description: "Data engineering best practices for node type selection, layered architecture, methodology detection, materialization strategies, and dependency management",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.STORAGE_MAPPINGS]: {
+        name: "Storage Locations and References",
+        description: "Storage location concepts, {{ ref() }} syntax, and reference patterns in Coalesce SQL",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.TOOL_USAGE]: {
+        name: "Tool Usage Patterns",
+        description: "Best practices for tool batching, parallelization, SQL conversion, and node operations",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.ID_DISCOVERY]: {
+        name: "ID Discovery",
+        description: "How to resolve project, workspace, environment, job, run, node, and org IDs before calling Coalesce tools",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.NODE_CREATION_DECISION_TREE]: {
+        name: "Node Creation Decision Tree",
+        description: "How to choose between predecessor-based creation, updates, and full replacements for workspace nodes",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.NODE_PAYLOADS]: {
+        name: "Node Payloads",
+        description: "Practical guidance for working with workspace node bodies, including top-level fields, metadata, config, and array-replacement risks",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.HYDRATED_METADATA]: {
+        name: "Hydrated Metadata",
+        description: "Practical summary of Coalesce hydrated metadata structures for advanced node payload editing",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.RUN_OPERATIONS]: {
+        name: "Run Operations",
+        description: "Guidance for starting, retrying, polling, diagnosing, and canceling Coalesce runs",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.NODE_TYPE_CORPUS]: {
+        name: "Node Type Corpus",
+        description: "Node type discovery, corpus search, metadata patterns (consult BEFORE creating or editing nodes)",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.AGGREGATION_PATTERNS]: {
+        name: "Aggregation Patterns",
+        description: "Automatic JOIN ON generation, GROUP BY detection, datatype inference, and patterns for converting joins to aggregations",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.INTELLIGENT_NODE_CONFIGURATION]: {
+        name: "Intelligent Node Configuration",
+        description: "How intelligent config completion works for workspace nodes, including schema resolution, intelligence rules, and automatic field detection",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.PIPELINE_WORKFLOWS]: {
+        name: "Pipeline Workflows",
+        description: "Building pipelines end-to-end: node type selection, multi-node sequences, incremental setup, and pipeline execution",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.NODE_OPERATIONS]: {
+        name: "Node Operations",
+        description: "Editing existing nodes: join conditions, column operations, config fields, rename safety, SQL-to-graph conversion, and debugging",
+        mimeType: "text/markdown",
+    },
+    [RESOURCES.NODE_TYPE_SELECTION_GUIDE]: {
+        name: "Node Type Selection Guide",
+        description: "When to use each Coalesce node type: Stage/Work for general transforms, Dimension/Fact only for dimensional modeling, and when to avoid Dynamic Tables, Incremental Loads, and other specialized patterns",
+        mimeType: "text/markdown",
+    },
+};
+/**
+ * Read a resource content file
+ */
+function readResourceContent(relativePath) {
+    const fullPath = join(__dirname, relativePath);
+    return readFileSync(fullPath, "utf-8");
+}
+function listCacheFilePaths(directory) {
+    if (!existsSync(directory)) {
+        return [];
+    }
+    const entries = readdirSync(directory, { withFileTypes: true });
+    const filePaths = [];
+    for (const entry of entries) {
+        const entryPath = join(directory, entry.name);
+        if (entry.isDirectory()) {
+            filePaths.push(...listCacheFilePaths(entryPath));
+            continue;
+        }
+        if (entry.isFile()) {
+            filePaths.push(entryPath);
+        }
+    }
+    return filePaths.sort();
+}
+function listCacheResources(baseDir) {
+    const cacheDir = getCacheDir(baseDir);
+    return listCacheFilePaths(cacheDir).flatMap((filePath) => {
+        const uri = buildCacheResourceUri(filePath, baseDir);
+        if (!uri) {
+            return [];
+        }
+        return [
+            {
+                uri,
+                name: basename(filePath),
+                description: "Cached Coalesce MCP artifact stored on disk and exposed through MCP resources.",
+                mimeType: getCacheResourceMimeType(filePath),
+            },
+        ];
+    });
+}
+/**
+ * Register all Coalesce context resources with the MCP server
+ */
+export function registerResources(server) {
+    for (const [uri, metadata] of Object.entries(RESOURCE_METADATA)) {
+        server.resource(metadata.name, uri, {
+            description: metadata.description,
+            mimeType: metadata.mimeType,
+        }, async (resourceUri) => {
+            const filePath = RESOURCE_FILES[uri];
+            if (!filePath) {
+                throw new Error(`Unknown resource: ${uri}`);
+            }
+            try {
+                const content = readResourceContent(filePath);
+                return {
+                    contents: [
+                        {
+                            uri: resourceUri.toString(),
+                            mimeType: metadata.mimeType,
+                            text: content,
+                        },
+                    ],
+                };
+            }
+            catch (error) {
+                throw new Error(`Failed to read resource ${uri}: ${error instanceof Error ? error.message : String(error)}`);
+            }
+        });
+    }
+    server.resource("Coalesce Cache Artifact", new ResourceTemplate("coalesce://cache/{cacheKey}", {
+        list: async () => ({
+            resources: listCacheResources(),
+        }),
+        complete: {
+            cacheKey: async (value) => listCacheResources()
+                .map((resource) => resource.uri.split("/").pop() ?? "")
+                .filter((cacheKey) => cacheKey.startsWith(value))
+                .slice(0, 50),
+        },
+    }), {
+        description: "Dynamic resources for cached tool responses, cache snapshots, and pipeline summaries.",
+    }, async (resourceUri) => {
+        const resolved = resolveCacheResourceUri(resourceUri.toString());
+        if (!resolved) {
+            throw new Error(`Unknown cache resource: ${resourceUri.toString()}`);
+        }
+        try {
+            return {
+                contents: [
+                    {
+                        uri: resourceUri.toString(),
+                        mimeType: getCacheResourceMimeType(resolved.filePath),
+                        text: readFileSync(resolved.filePath, "utf8"),
+                    },
+                ],
+            };
+        }
+        catch (error) {
+            throw new Error(`Failed to read cache resource ${resourceUri.toString()}: ${error instanceof Error ? error.message : String(error)}`);
+        }
+    });
+}