@pgsql/transform 18.1.1 → 18.3.0

package/README.md

@@ -12,25 +12,9 @@
    <a href="https://www.npmjs.com/package/@pgsql/transform"><img height="20" src="https://img.shields.io/github/package-json/v/constructive-io/pgsql-parser?filename=packages%2Ftransform%2Fpackage.json"/></a>
 </p>
 
-`@pgsql/transform` is a TypeScript library for transforming PostgreSQL ASTs between different PostgreSQL versions. It serves as a crucial component for building a single source of truth deparser that can handle SQL from multiple PostgreSQL versions while maintaining backward compatibility.
+AST-based SQL transformation, qualification, classification, and dependency-closure analysis for PostgreSQL. Works on plain SQL **and inside PL/pgSQL function bodies** (via `plpgsql-parser` hydration) — no regexes.
 
-## Transforming ASTs Between PG Versions
-
-The transform package enables you to:
-
-- **Transform legacy ASTs**: Convert ASTs from PostgreSQL versions 13-16 to version 17
-- **Build unified deparsers**: Create a single deparser pipeline that works with multiple PostgreSQL versions
-- **Maintain backward compatibility**: Support legacy codebases while leveraging the latest PostgreSQL features
-
-## Key Limitation
-
-This package only supports ASTs derived from SQL that is parseable by PostgreSQL 17. This means:
-
-- ✅ **Supported**: SQL from PG13-16 that remains valid in PG17
-- ❌ **Not supported**: Deprecated syntax from older versions that was removed
-- ❌ **Not supported**: SQL that cannot be parsed by the PG17 parser
-
-This design ensures all transformed ASTs can be reliably deparsed using the latest PostgreSQL grammar.
+> Looking for the PG13→17/18 version-upgrade AST transformer previously published under this name? It now lives at [`@pgsql/transform-ast`](../transform-ast).
 
 ## Installation
 
@@ -38,82 +22,54 @@ This design ensures all transformed ASTs can be reliably deparsed using the late
 npm install @pgsql/transform
 ```
 
-## 🚀 Quick Start
-
-### Multi-Version Transformer
-
-```typescript
-import { ASTTransformer } from '@pgsql/transform';
-
-const transformer = new ASTTransformer();
-
-// Transform any version to PG17
-const pg17Ast = transformer.transformToLatest(pg13Ast, 13);
-
-// Transform between specific versions
-const pg15Ast = transformer.transform(pg13Ast, 13, 15);
+The parser runs on a WASM build of the real PostgreSQL parser; call `loadModule()` from `plpgsql-parser` once before using any synchronous API.
 
-// Convenience methods
-const result = transformer.transform13To17(pg13Ast);
-```
+## Features
 
-### Direct Transformers
+### `transformSql` — schema-name rewriting
 
-For better performance when you know source and target versions:
+Rewrite schema names everywhere they can appear (DDL, DML, function bodies, trigger definitions, grants, policies, comments, type casts, string-embedded types inside PL/pgSQL, ...):
 
 ```typescript
-import { PG13ToPG17Transformer } from '@pgsql/transform';
+import { loadModule } from 'plpgsql-parser';
+import { transformSql } from '@pgsql/transform';
 
-const transformer = new PG13ToPG17Transformer();
-const pg17Ast = transformer.transform(pg13Ast);
+await loadModule();
+const mapping = new Map([['my-schema', 'my_schema']]);
+const { sql } = transformSql(inputSql, mapping);
 ```
 
-### Integration Example
+### `classifyStatements` — per-statement AST facts
 
 ```typescript
-import { parse } from '@pgsql/parser';
-import { deparse } from 'pgsql-deparser';
-import { PG13ToPG17Transformer } from '@pgsql/transform';
-
-// Parse with older version
-const pg13Ast = await parse('SELECT * FROM users', { version: 13 });
+import { classifyStatements } from '@pgsql/transform';
 
-// Transform to latest
-const transformer = new PG13ToPG17Transformer();
-const pg17Ast = transformer.transform(pg13Ast);
-
-// Deparse with latest grammar
-const sql = await deparse(pg17Ast);
+const facts = classifyStatements(sql);
+// per statement: kind (schema|table|view|index|type|function|trigger|policy|grant|...),
+// creates, references (incl. inside PL/pgSQL bodies), referencedSchemas, roles,
+// fkTargets, securityRelevant, securityDefiner, dynamicSql
 ```
 
-## 🔄 Transformation Chain
+### `qualifyUnqualified` — add schema qualification
+
+Qualify unqualified object references against an inventory of known objects, with multi-schema routing support.
 
-**Incremental**: PG13 → PG14 → PG15 → PG16 → PG17
-- Step-by-step version upgrades
-- Useful for debugging transformation issues
+### `resolveFixtureClosure` — transitive dependency closure
 
-**Direct**: PG13 → PG17, PG14 → PG17, etc.
-- Single-step transformations
-- Optimized for performance
+Given a set of changes (name + SQL + optional declared dependencies), compute the transitive closure of a selection: forward producers of every referenced object/schema/role, plus attached fixtures (policies/grants/RLS targeting closure members), with explicit unresolved-reference reporting.
 
-## 📋 Supported Transformations
+### `categorizeChange` / `buildCategoryOf` — change categorization
 
-| From | To | Transformer |
-|------|----|-----------| 
-| PG13 | PG14, PG15, PG16, PG17 | `V13ToV14Transformer`, `PG13ToPG17Transformer` |
-| PG14 | PG15, PG16, PG17 | `V14ToV15Transformer`, `PG14ToPG17Transformer` |
-| PG15 | PG16, PG17 | `V15ToV16Transformer`, `PG15ToPG17Transformer` |
-| PG16 | PG17 | `V16ToV17Transformer`, `PG16ToPG17Transformer` |
+Profile-driven categorization of migration changes (e.g. schema / functionality / security / fixtures) from their AST facts.
 
-## 🏗️ Architecture
+### Round-trip validation
 
-The transform package fits into the broader pgsql-parser ecosystem:
+`normalizeTree` / `cleanTree` / `validateRoundTrip` — dependency-free AST normalization and mutation-aware parse→deparse→re-parse validation.
 
-1. **Parse** legacy SQL with version-specific parsers
-2. **Transform** ASTs to the latest version
-3. **Deparse** using the unified, latest-version deparser
+## Scripts
 
-This enables a single source of truth for SQL generation while supporting legacy codebases.
+- `npm run fixtures` — regenerate `__fixtures__/output/` golden files from `__fixtures__/input/`
+- `npm run scan-corpus <dir> [dir...]` — audit node-type coverage over a SQL corpus
 
 ---
 

package/categorize.d.ts

@@ -0,0 +1,52 @@
+import { StatementFacts } from './facts';
+/**
+ * A category is a chunk-seam key a change is assigned to. Free-form string; the
+ * built-in {@link TIER_PROFILE} uses `'security' | 'functionality' | 'fixtures'
+ * | 'schema'`.
+ *
+ * This module is the classifier-driven counterpart to `@pgpmjs/core`'s
+ * `boundary: 'category'` rebundle mode: core stays classifier-agnostic and only
+ * consumes a `categoryOf(changeName)` seam function, while the rules that decide
+ * *what* each change is (the profile) live here, next to the AST facts.
+ */
+export type ChangeCategory = string;
+/**
+ * Decides a change's category from the AST facts of its (deploy) statements.
+ * A profile is a pure function of the facts — swap it to carve a monolith along
+ * different seams (e.g. per-role admin/users modules) without touching core.
+ */
+export interface CategoryProfile {
+    /** Identifier for the profile (for logging / provenance). */
+    name: string;
+    /** Map a change's statement facts to its category. */
+    categorize: (facts: StatementFacts[], changeName: string) => ChangeCategory;
+}
+/**
+ * Default tier profile: isolates the security surface and procedural code out
+ * of the schema base. Precedence, highest first — a change is placed in the
+ * most specialized tier any of its statements reaches:
+ *
+ *   security       any securityRelevant statement (policy/grant/RLS/role/owner)
+ *   functionality  functions and triggers
+ *   fixtures       seed DML (INSERT/UPDATE/DELETE)
+ *   schema         schemas, tables, views, indexes, types, constraints (base)
+ *
+ * Ordering/dependencies across the resulting chunks are still resolved by
+ * core's rebundle graph, so this only decides *naming/grouping*, never deploy
+ * order.
+ */
+export declare const TIER_PROFILE: CategoryProfile;
+/**
+ * Classify a single change's deploy SQL into its category under `profile`
+ * (defaults to {@link TIER_PROFILE}).
+ */
+export declare function categorizeChange(sql: string, changeName: string, profile?: CategoryProfile): ChangeCategory;
+/**
+ * Build a `categoryOf(changeName)` seam function from a map of change name to
+ * its deploy SQL — ready to pass straight to `@pgpmjs/core`'s rebundle
+ * `boundary: 'category'` / `categoryOf` options.
+ *
+ * Changes absent from `changeSql` return `undefined`, letting core fall back to
+ * its folder key for anything the classifier did not see.
+ */
+export declare function buildCategoryOf(changeSql: Record<string, string>, profile?: CategoryProfile): (changeName: string) => ChangeCategory | undefined;

package/categorize.js

@@ -0,0 +1,56 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.TIER_PROFILE = void 0;
+exports.categorizeChange = categorizeChange;
+exports.buildCategoryOf = buildCategoryOf;
+const facts_1 = require("./facts");
+/**
+ * Default tier profile: isolates the security surface and procedural code out
+ * of the schema base. Precedence, highest first — a change is placed in the
+ * most specialized tier any of its statements reaches:
+ *
+ *   security       any securityRelevant statement (policy/grant/RLS/role/owner)
+ *   functionality  functions and triggers
+ *   fixtures       seed DML (INSERT/UPDATE/DELETE)
+ *   schema         schemas, tables, views, indexes, types, constraints (base)
+ *
+ * Ordering/dependencies across the resulting chunks are still resolved by
+ * core's rebundle graph, so this only decides *naming/grouping*, never deploy
+ * order.
+ */
+exports.TIER_PROFILE = {
+    name: 'tier',
+    categorize(facts) {
+        if (facts.length === 0)
+            return 'schema';
+        if (facts.some(f => f.securityRelevant))
+            return 'security';
+        if (facts.some(f => f.kind === 'function' || f.kind === 'trigger'))
+            return 'functionality';
+        if (facts.some(f => f.kind === 'seed_dml'))
+            return 'fixtures';
+        return 'schema';
+    },
+};
+/**
+ * Classify a single change's deploy SQL into its category under `profile`
+ * (defaults to {@link TIER_PROFILE}).
+ */
+function categorizeChange(sql, changeName, profile = exports.TIER_PROFILE) {
+    return profile.categorize((0, facts_1.classifyStatements)(sql), changeName);
+}
+/**
+ * Build a `categoryOf(changeName)` seam function from a map of change name to
+ * its deploy SQL — ready to pass straight to `@pgpmjs/core`'s rebundle
+ * `boundary: 'category'` / `categoryOf` options.
+ *
+ * Changes absent from `changeSql` return `undefined`, letting core fall back to
+ * its folder key for anything the classifier did not see.
+ */
+function buildCategoryOf(changeSql, profile = exports.TIER_PROFILE) {
+    const categories = new Map();
+    for (const [name, sql] of Object.entries(changeSql)) {
+        categories.set(name, categorizeChange(sql, name, profile));
+    }
+    return (changeName) => categories.get(changeName);
+}

package/esm/categorize.js

@@ -0,0 +1,51 @@
+import { classifyStatements } from './facts';
+/**
+ * Default tier profile: isolates the security surface and procedural code out
+ * of the schema base. Precedence, highest first — a change is placed in the
+ * most specialized tier any of its statements reaches:
+ *
+ *   security       any securityRelevant statement (policy/grant/RLS/role/owner)
+ *   functionality  functions and triggers
+ *   fixtures       seed DML (INSERT/UPDATE/DELETE)
+ *   schema         schemas, tables, views, indexes, types, constraints (base)
+ *
+ * Ordering/dependencies across the resulting chunks are still resolved by
+ * core's rebundle graph, so this only decides *naming/grouping*, never deploy
+ * order.
+ */
+export const TIER_PROFILE = {
+    name: 'tier',
+    categorize(facts) {
+        if (facts.length === 0)
+            return 'schema';
+        if (facts.some(f => f.securityRelevant))
+            return 'security';
+        if (facts.some(f => f.kind === 'function' || f.kind === 'trigger'))
+            return 'functionality';
+        if (facts.some(f => f.kind === 'seed_dml'))
+            return 'fixtures';
+        return 'schema';
+    },
+};
+/**
+ * Classify a single change's deploy SQL into its category under `profile`
+ * (defaults to {@link TIER_PROFILE}).
+ */
+export function categorizeChange(sql, changeName, profile = TIER_PROFILE) {
+    return profile.categorize(classifyStatements(sql), changeName);
+}
+/**
+ * Build a `categoryOf(changeName)` seam function from a map of change name to
+ * its deploy SQL — ready to pass straight to `@pgpmjs/core`'s rebundle
+ * `boundary: 'category'` / `categoryOf` options.
+ *
+ * Changes absent from `changeSql` return `undefined`, letting core fall back to
+ * its folder key for anything the classifier did not see.
+ */
+export function buildCategoryOf(changeSql, profile = TIER_PROFILE) {
+    const categories = new Map();
+    for (const [name, sql] of Object.entries(changeSql)) {
+        categories.set(name, categorizeChange(sql, name, profile));
+    }
+    return (changeName) => categories.get(changeName);
+}

package/esm/facts.js

@@ -0,0 +1,289 @@
+import { walk as walkSql } from '@pgsql/traverse';
+import { transformSync, walk as walkPlpgsql } from 'plpgsql-parser';
+const SECURITY_TAGS = new Set([
+    'CreatePolicyStmt',
+    'AlterPolicyStmt',
+    'GrantStmt',
+    'GrantRoleStmt',
+    'AlterDefaultPrivilegesStmt',
+    'SecLabelStmt',
+    'AlterOwnerStmt',
+    'CreateRoleStmt',
+    'AlterRoleStmt'
+]);
+const KIND_BY_TAG = {
+    CreateSchemaStmt: 'schema',
+    CreateStmt: 'table',
+    ViewStmt: 'view',
+    IndexStmt: 'index',
+    CompositeTypeStmt: 'type',
+    CreateEnumStmt: 'type',
+    CreateDomainStmt: 'type',
+    CreateRangeStmt: 'type',
+    DefineStmt: 'type',
+    CreateFunctionStmt: 'function',
+    CreateTrigStmt: 'trigger',
+    CreateEventTrigStmt: 'trigger',
+    CreatePolicyStmt: 'policy',
+    AlterPolicyStmt: 'policy',
+    GrantStmt: 'grant',
+    GrantRoleStmt: 'grant',
+    AlterDefaultPrivilegesStmt: 'grant',
+    CommentStmt: 'comment',
+    InsertStmt: 'seed_dml',
+    UpdateStmt: 'seed_dml',
+    DeleteStmt: 'seed_dml'
+};
+function qn(schema, name) {
+    return { schema: schema ?? null, name };
+}
+function nameListToQualified(names) {
+    if (!Array.isArray(names) || names.length === 0)
+        return null;
+    const parts = names
+        .map((n) => n?.String?.sval)
+        .filter((s) => typeof s === 'string');
+    if (parts.length === 0)
+        return null;
+    if (parts.length === 1)
+        return qn(null, parts[0]);
+    return qn(parts[parts.length - 2], parts[parts.length - 1]);
+}
+function isCatalogSchema(schema) {
+    return schema === 'pg_catalog' || schema === 'information_schema';
+}
+function pushRef(refs, ref) {
+    if (!ref || !ref.schema || isCatalogSchema(ref.schema))
+        return;
+    if (refs.some(r => r.schema === ref.schema && r.name === ref.name))
+        return;
+    refs.push(ref);
+}
+function collectRoles(node, roles) {
+    const push = (role) => {
+        if (typeof role === 'string' && role.length > 0 && !roles.includes(role)) {
+            roles.push(role);
+        }
+    };
+    if (Array.isArray(node?.grantees)) {
+        for (const g of node.grantees)
+            push(g?.RoleSpec?.rolename);
+    }
+    if (Array.isArray(node?.roles)) {
+        for (const r of node.roles)
+            push(r?.RoleSpec?.rolename);
+    }
+    push(node?.newowner?.rolename);
+    push(node?.role?.rolename);
+}
+/**
+ * Create a read-only visitor that accumulates references, roles and FK
+ * targets into the provided facts object.
+ */
+function createFactsVisitor(facts, bodyRefs) {
+    const push = (ref) => {
+        pushRef(facts.references, ref);
+        if (bodyRefs)
+            pushRef(bodyRefs, ref);
+    };
+    return {
+        RangeVar: (path) => {
+            const node = path.node;
+            if (node.schemaname) {
+                push(qn(node.schemaname, node.relname));
+            }
+        },
+        FuncCall: (path) => {
+            push(nameListToQualified(path.node.funcname));
+        },
+        TypeName: (path) => {
+            push(nameListToQualified(path.node.names));
+        },
+        ColumnRef: (path) => {
+            // schema.table.column references carry cross-schema information
+            const fields = path.node.fields;
+            if (Array.isArray(fields) && fields.length >= 3) {
+                const parts = fields
+                    .map((f) => f?.String?.sval)
+                    .filter((s) => typeof s === 'string');
+                if (parts.length >= 3) {
+                    push(qn(parts[0], parts[1]));
+                }
+            }
+        },
+        Constraint: (path) => {
+            const node = path.node;
+            if (node.contype === 'CONSTR_FOREIGN' && node.pktable) {
+                const target = qn(node.pktable.schemaname ?? null, node.pktable.relname);
+                if (!facts.fkTargets.some(t => t.schema === target.schema && t.name === target.name)) {
+                    facts.fkTargets.push(target);
+                }
+                if (target.schema)
+                    pushRef(facts.references, target);
+            }
+        }
+    };
+}
+function classifyOne(nodeTag, node) {
+    const facts = {
+        kind: KIND_BY_TAG[nodeTag] ?? 'other',
+        nodeTag,
+        creates: [],
+        references: [],
+        referencedSchemas: [],
+        roles: [],
+        fkTargets: [],
+        bodyReferences: [],
+        securityRelevant: SECURITY_TAGS.has(nodeTag),
+        securityDefiner: false,
+        dynamicSql: false
+    };
+    switch (nodeTag) {
+        case 'CreateSchemaStmt':
+            facts.creates.push(qn(null, node.schemaname));
+            break;
+        case 'CreateStmt':
+        case 'ViewStmt': {
+            const rel = nodeTag === 'ViewStmt' ? node.view : node.relation;
+            if (rel)
+                facts.creates.push(qn(rel.schemaname ?? null, rel.relname));
+            break;
+        }
+        case 'IndexStmt':
+            if (node.relation) {
+                facts.creates.push(qn(node.relation.schemaname ?? null, node.idxname ?? node.relation.relname));
+            }
+            break;
+        case 'CreateFunctionStmt': {
+            const name = nameListToQualified(node.funcname);
+            if (name)
+                facts.creates.push(name);
+            for (const opt of node.options ?? []) {
+                const def = opt?.DefElem;
+                if (def?.defname === 'security' && def?.arg?.Boolean?.boolval === true) {
+                    facts.securityDefiner = true;
+                }
+            }
+            break;
+        }
+        case 'CreateTrigStmt':
+            if (node.relation) {
+                // Trigger names are only unique per table; qualify with the table.
+                facts.creates.push(qn(node.relation.schemaname ?? null, `${node.relation.relname}.${node.trigname}`));
+            }
+            pushRef(facts.references, nameListToQualified(node.funcname));
+            break;
+        case 'CreatePolicyStmt':
+        case 'AlterPolicyStmt':
+            if (node.table) {
+                // Policy names are only unique per table; qualify with the table.
+                facts.creates.push(qn(node.table.schemaname ?? null, `${node.table.relname}.${node.policy_name}`));
+                // The guarded table lives on `node.table`; the generic RangeVar walker
+                // does not descend into it, so capture the reference explicitly.
+                if (node.table.schemaname) {
+                    pushRef(facts.references, qn(node.table.schemaname, node.table.relname));
+                }
+            }
+            break;
+        case 'CreateSeqStmt':
+            if (node.sequence) {
+                facts.creates.push(qn(node.sequence.schemaname ?? null, node.sequence.relname));
+            }
+            break;
+        case 'CompositeTypeStmt':
+            if (node.typevar) {
+                facts.creates.push(qn(node.typevar.schemaname ?? null, node.typevar.relname));
+            }
+            break;
+        case 'CreateEnumStmt':
+        case 'CreateDomainStmt':
+        case 'CreateRangeStmt': {
+            const name = nameListToQualified(node.typeName ?? node.domainname);
+            if (name)
+                facts.creates.push(name);
+            break;
+        }
+        case 'AlterTableStmt': {
+            if (node.relation) {
+                facts.creates.push(qn(node.relation.schemaname ?? null, node.relation.relname));
+            }
+            const cmds = node.cmds ?? [];
+            for (const cmd of cmds) {
+                const subtype = cmd?.AlterTableCmd?.subtype;
+                if (subtype === 'AT_EnableRowSecurity' || subtype === 'AT_ForceRowSecurity' ||
+                    subtype === 'AT_DisableRowSecurity' || subtype === 'AT_NoForceRowSecurity') {
+                    facts.kind = 'rls_enable';
+                    facts.securityRelevant = true;
+                }
+                else if (subtype === 'AT_AddConstraint') {
+                    const contype = cmd?.AlterTableCmd?.def?.Constraint?.contype;
+                    facts.kind = contype === 'CONSTR_FOREIGN' ? 'fk_constraint' : 'constraint';
+                }
+                else if (subtype === 'AT_ChangeOwner') {
+                    facts.securityRelevant = true;
+                    const owner = cmd?.AlterTableCmd?.newowner?.rolename;
+                    if (owner && !facts.roles.includes(owner))
+                        facts.roles.push(owner);
+                }
+            }
+            break;
+        }
+        case 'InsertStmt':
+        case 'UpdateStmt':
+        case 'DeleteStmt':
+            if (node.relation) {
+                facts.creates.push(qn(node.relation.schemaname ?? null, node.relation.relname));
+            }
+            break;
+        default:
+            break;
+    }
+    collectRoles(node, facts.roles);
+    return facts;
+}
+/**
+ * Classify each top-level statement in a SQL script into {@link StatementFacts}.
+ *
+ * Uses the same parser stack as the schema transformer (pgsql AST walk plus
+ * hydrated PL/pgSQL body walk), so references inside function bodies are
+ * included. The input SQL is never modified.
+ */
+export function classifyStatements(sql) {
+    const allFacts = [];
+    transformSync(sql, (ctx) => {
+        const stmts = ctx.sql?.stmts ?? [];
+        for (const stmt of stmts) {
+            const stmtNode = stmt?.stmt;
+            const nodeTag = stmtNode ? Object.keys(stmtNode)[0] : 'other';
+            const node = stmtNode?.[nodeTag] ?? {};
+            const facts = classifyOne(nodeTag, node);
+            if (stmtNode) {
+                walkSql(stmtNode, createFactsVisitor(facts));
+            }
+            allFacts.push(facts);
+        }
+        for (const fn of ctx.functions ?? []) {
+            const facts = allFacts[fn.stmtIndex];
+            if (!facts || !fn.plpgsql?.hydrated)
+                continue;
+            // Refs already found outside the body (signature types, defaults)
+            // constrain deploy order and are not body-only.
+            const outer = new Set(facts.references.map(r => `${r.schema ?? '?'}.${r.name}`));
+            walkPlpgsql(fn.plpgsql.hydrated, {
+                PLpgSQL_stmt_dynexecute: () => { facts.dynamicSql = true; },
+                PLpgSQL_stmt_dynfors: () => { facts.dynamicSql = true; }
+            }, {
+                walkSqlExpressions: true,
+                sqlVisitor: createFactsVisitor(facts, facts.bodyReferences)
+            });
+            facts.bodyReferences = facts.bodyReferences.filter(r => !outer.has(`${r.schema ?? '?'}.${r.name}`));
+        }
+    }, { hydrate: true });
+    for (const facts of allFacts) {
+        const selfRef = (r) => facts.creates.some(c => c.schema === r.schema && c.name === r.name);
+        facts.references = facts.references.filter(r => !selfRef(r));
+        facts.bodyReferences = facts.bodyReferences.filter(r => !selfRef(r));
+        facts.referencedSchemas = [...new Set(facts.references.map(r => r.schema).filter(Boolean))];
+    }
+    return allFacts;
+}