clickhouse-migrations 1.4.0 → 1.5.0

package/README.md

@@ -14,7 +14,9 @@ Create a directory, where migrations will be stored. It will be used as the valu
 
 In the directory, create migration files, which should be named like this: `1_some_text.sql`, `2_other_text.sql`, `10_more_test.sql`. What's important here is that the migration version number should come first, followed by an underscore (`_`), and then any text can follow. The version number should increase for every next migration. Please note that once a migration file has been applied to the database, it cannot be modified or removed.
 
-For migrations' content should be used correct SQL ClickHouse queries. Multiple queries can be used in a single migration file, and each query should be terminated with a semicolon (;). The queries could be idempotent - for example: `CREATE TABLE IF NOT EXISTS table ...;` Clickhouse settings, that can be included at the query level, can be added like `SET allow_experimental_object_type = 1;`. For adding comments should be used `--`, `# `, `#!`.
+Migration files contain ClickHouse SQL statements separated by semicolons (`;`). Semicolons and comment markers inside single-quoted strings, quoted identifiers (`"..."`, `` `...` ``), and dollar-quoted strings (`$tag$...$tag$` or `$$...$$`) are preserved. Whitespace inside quoted text is also preserved. In SQL, comments (`--`, `//`, `# `, `#!`, and nested `/* ... */`) are removed, and whitespace outside quoted text is collapsed. Inline formatted data is preserved separately as described below.
+
+See [SQL execution and settings](#sql-execution-and-settings) below for details on settings, supported migration content, and error handling.
 
 If the database provided in the `--db` option (or in `CH_MIGRATIONS_DB`) doesn't exist, it will be created automatically. To disable this behavior (for example, when the user has no privileges to create databases), use the `--skip-db-creation` option (or set `CH_MIGRATIONS_SKIP_DB_CREATION` to `'true'`); in that case the database must already exist.
 
@@ -150,3 +152,28 @@ LAYOUT(COMPLEX_KEY_HASHED());
 ```
 
 _Please note:_ When substitution is enabled, the checksum is taken from the **raw file**, before substitution. This keeps a migration stable across environments and keeps secrets out of `_migrations`. Two consequences follow: an already-applied migration is not re-run when a variable changes (ship a new migration instead), and the substituted SQL still reaches ClickHouse, so it may appear in `system.query_log` or server logs.
+
+## SQL execution and settings
+
+Queries execute in file order. Assignment-style `SET name = value` statements are collected before executing the file: the last assignment to each setting applies to **every query in that file**, including queries written before the `SET`. This preserves the package's existing behavior. Settings do not carry over to the next migration file.
+
+Settings are sent with each HTTP request, overriding defaults supplied in the host URL. This works without shared session state when a load balancer routes requests to different servers, and also supports HTTP parameters such as `wait_end_of_query`.
+
+Use assignment-style `SET` commands for settings and query parameters. Quoted string values, multiline assignments and compound parameter values are supported; the available setting names and values depend on your ClickHouse version:
+
+```sql
+SET max_threads = 2,
+    log_comment = 'migration; keep this text';
+SET wait_end_of_query = 1;
+SET session_timezone = 'UTC';
+SET param_d = {'10': [11, 12], '13': [14, 15]};
+SELECT {d:Map(String, Array(UInt8))};
+```
+
+For a setting specific to one query, use that query's `SETTINGS` clause where ClickHouse supports it. Commands requiring shared session state, such as `SET ROLE` followed by another query, are not supported. Use `SET session_timezone = 'UTC'` to configure the timezone for the file.
+
+Inline `INSERT ... FORMAT` data is accepted, including `INSERT ... SELECT ... FROM input(...) FORMAT ...`. `FORMAT Values` uses the same quote handling as `INSERT ... VALUES`. For other formats, everything after the format name is preserved verbatim up to the next semicolon or the end of the file: tabs, newlines, quotes and comment markers remain data.
+
+Raw formats retain the legacy semicolon delimiter, even inside quoted data. Payloads containing semicolons should use `INSERT ... VALUES` or be loaded separately. ClickHouse validates the data format; this package does not parse CSV, JSON or other raw formats.
+
+Unterminated SQL quotes/comments and malformed setting assignments cause an error identifying the migration file before any query in that file executes. ClickHouse validates SQL syntax, setting names, setting values and inline data when each query runs. Database setup and earlier files may already have run, and server-side errors can still leave a file partially applied. Where appropriate, use idempotent statements such as `CREATE TABLE IF NOT EXISTS ...`.

package/lib/migrate.js

@@ -201,24 +201,27 @@ const apply_migrations = async (client, migrations, migrations_home) => {
             // Skip if a migration is already applied.
             continue;
         }
-        // Substitute ${VAR} from the environment (opt-in) at apply time only. The
-        // checksum above is over the raw file, so it stays stable across environments.
-        let sql;
+        // Parse the entire file before executing its statements. Substitution is
+        // opt-in; the checksum above always uses the original file content.
+        let queries;
+        let settings;
         try {
-            sql = (0, sql_parse_1.substitute_env)(content);
+            ({ queries, settings } = (0, sql_parse_1.parse_migration_sql)((0, sql_parse_1.substitute_env)(content)));
         }
         catch (e) {
             log('error', `the migration ${migration.file} has an error.`, e instanceof Error ? e.message : String(e));
             process.exit(1);
         }
-        // Extract sql from the migration.
-        const queries = (0, sql_parse_1.sql_queries)(sql);
-        const sets = (0, sql_parse_1.sql_sets)(sql);
+        // Keep the established file-wide SET behavior: the last assignment wins,
+        // including for queries earlier in the file. Send settings on every request
+        // instead of using sessions, whose state is lost if a load balancer routes
+        // the next query to another server. Per-query settings also override host
+        // URL defaults and support HTTP parameters such as wait_end_of_query.
         for (const query of queries) {
             try {
                 await client.command({
                     query: query,
-                    clickhouse_settings: sets,
+                    clickhouse_settings: settings,
                 });
             }
             catch (e) {

package/lib/sql-parse.d.ts

@@ -1,6 +1,8 @@
 declare const substitute_env: (content: string) => string;
-declare const sql_queries: (content: string) => string[];
-declare const sql_sets: (content: string) => {
-    [key: string]: string;
+declare const parse_migration_sql: (content: string) => {
+    queries: string[];
+    settings: Record<string, string>;
 };
-export { sql_queries, sql_sets, substitute_env };
+declare const sql_queries: (content: string) => string[];
+declare const sql_sets: (content: string) => Record<string, string>;
+export { parse_migration_sql, sql_queries, sql_sets, substitute_env };

package/lib/sql-parse.js

@@ -1,6 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.substitute_env = exports.sql_sets = exports.sql_queries = void 0;
+exports.substitute_env = exports.sql_sets = exports.sql_queries = exports.parse_migration_sql = void 0;
+const sql_split_1 = require("./sql-split");
 // Replace ${NAME} in migration content with env vars. Opt-in via
 // CH_MIGRATIONS_SUBSTITUTE_ENV=true; otherwise returns content unchanged.
 // Use $${NAME} to keep a literal ${NAME}. Throws on an unset variable or a
@@ -51,35 +52,86 @@ const substitute_env = (content) => {
     return result;
 };
 exports.substitute_env = substitute_env;
-// Extract sql queries from migrations.
-const sql_queries = (content) => {
-    const queries = content
-        .replace(/(--|#!|#\s).*(\n|\r\n|\r|$)/gm, '\n')
-        .replace(/^\s*(SET\s).*(\n|\r\n|\r|$)/gm, '')
-        .replace(/(\n|\r\n|\r)/gm, ' ')
-        .replace(/\s+/g, ' ')
-        .split(';')
-        .map((el) => el.trim())
-        .filter((el) => el.length != 0);
-    return queries;
+// HTTP settings take values, not SQL string-literal syntax. Decode quoted
+// strings once; leave numbers and compound parameter values as text so large
+// integers never lose precision through JavaScript Number conversion.
+const setting_value = (raw) => {
+    const tag = /^\$[A-Za-z0-9_]*\$/.exec(raw)?.[0];
+    if (tag && raw.endsWith(tag))
+        return raw.slice(tag.length, -tag.length);
+    const quote = raw[0];
+    if (!["'", '"', '`'].includes(quote) || !raw.endsWith(quote))
+        return raw;
+    // Hex escapes represent bytes; decode UTF-8 after unescaping so \xC3\xA9 is é.
+    const input = Buffer.from(raw.slice(1, -1));
+    const bytes = [];
+    const escapes = {
+        '0': 0,
+        a: 7,
+        b: 8,
+        e: 27,
+        f: 12,
+        n: 10,
+        r: 13,
+        t: 9,
+        v: 11,
+        "'": 39,
+        '"': 34,
+        '`': 96,
+        '\\': 92,
+        '/': 47,
+        '=': 61,
+    };
+    for (let i = 0; i < input.length; i += 1) {
+        const byte = input[i];
+        if (byte === quote.charCodeAt(0) && input[i + 1] === byte) {
+            i += 1;
+        }
+        else if (byte === 92 && i + 1 < input.length) {
+            const next = String.fromCharCode(input[i + 1]);
+            const hex = input.subarray(i + 2, i + 4).toString();
+            if (next === 'x' && /^[0-9a-f]{2}$/i.test(hex)) {
+                bytes.push(parseInt(hex, 16));
+                i += 3;
+                continue;
+            }
+            if (Object.hasOwn(escapes, next)) {
+                bytes.push(escapes[next]);
+                i += 1;
+                continue;
+            }
+            if (next === 'N') {
+                i += 1; // ClickHouse's \N escape is an empty string inside a literal.
+                continue;
+            }
+        }
+        bytes.push(byte);
+    }
+    return Buffer.from(bytes).toString('utf8');
 };
-exports.sql_queries = sql_queries;
-// Extract query settings from migrations.
-const sql_sets = (content) => {
-    const sets = {};
-    const sets_arr = content
-        .replace(/(--|#!|#\s).*(\n|\r\n|\r|$)/gm, '\n')
-        .replace(/^\s*(?!SET\s).*(\n|\r\n|\r|$)/gm, '')
-        .replace(/^\s*(SET\s)/gm, '')
-        .replace(/(\n|\r\n|\r)/gm, ' ')
-        .replace(/\s+/g, '')
-        .split(';');
-    sets_arr.forEach((set_full) => {
-        const set = set_full.split('=');
-        if (set[0]) {
-            sets[set[0]] = set[1];
+const parse_migration_sql = (content) => {
+    const queries = [];
+    const settings = {};
+    for (const statement of (0, sql_split_1.split_statements)(content)) {
+        // Only assignment-style SET belongs to file-wide settings. Administrative
+        // statements such as SET DEFAULT ROLE remain SQL for ClickHouse to execute.
+        const setting = /^SET\s+([A-Za-z_][A-Za-z0-9_]*\s*=[\s\S]*)$/i.exec(statement);
+        if (!setting) {
+            queries.push(statement);
+            continue;
+        }
+        for (const assignment of (0, sql_split_1.split_assignments)(setting[1])) {
+            const match = /^([A-Za-z_][A-Za-z0-9_]*)\s*=\s*([\s\S]+)$/.exec(assignment);
+            if (!match)
+                throw new Error(`invalid SET assignment: ${assignment}`);
+            settings[match[1]] = setting_value(match[2].trim());
         }
-    });
-    return sets;
+    }
+    return { queries, settings };
 };
+exports.parse_migration_sql = parse_migration_sql;
+// Preserve the existing helper exports for callers importing lib/sql-parse.
+const sql_queries = (content) => parse_migration_sql(content).queries;
+exports.sql_queries = sql_queries;
+const sql_sets = (content) => parse_migration_sql(content).settings;
 exports.sql_sets = sql_sets;

package/lib/sql-split.d.ts

@@ -0,0 +1,3 @@
+declare const split_statements: (content: string) => string[];
+declare const split_assignments: (content: string) => string[];
+export { split_statements, split_assignments };

package/lib/sql-split.js

@@ -0,0 +1,213 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.split_assignments = exports.split_statements = void 0;
+// Return a location for error messages, for example: "line 12, column 7".
+const describe_position = (content, index) => {
+    const before = content.slice(0, index);
+    const line = before.split(/\r\n|\r|\n/).length;
+    const lineStart = Math.max(before.lastIndexOf('\n'), before.lastIndexOf('\r')) + 1;
+    // Count code points, not UTF-16 units, so supplementary characters do not shift the column.
+    const column = Array.from(before.slice(lineStart)).length + 1;
+    return `line ${line}, column ${column}`;
+};
+const isSpace = (ch) => ch === ' ' || ch === '\t' || ch === '\n' || ch === '\r' || ch === '\f' || ch === '\v';
+// Both statements and SET assignments use the same quoting/comment rules.
+const split_sql = (content, separator) => {
+    const statements = [];
+    let current = '';
+    let pendingSpace = false; // whitespace seen in plain SQL, emitted lazily as one space
+    let bracketDepth = 0;
+    let insertStatement = false;
+    let insertHeader = false;
+    let inputSource = false;
+    let inputFunctionExpected = false;
+    let formatExpected = false;
+    let previousToken = '';
+    const rememberToken = (token, quoted = false) => {
+        inputFunctionExpected = token === 'INPUT' && ['FROM', 'JOIN'].includes(previousToken);
+        previousToken = quoted ? 'quoted' : token;
+    };
+    const emitPlain = (text) => {
+        if (pendingSpace && current.length > 0) {
+            current += ' ';
+        }
+        pendingSpace = false;
+        current += text;
+    };
+    const flush = (preserveData = false) => {
+        const statement = preserveData ? current : current.trim();
+        if (statement || separator === ',')
+            statements.push(statement);
+        current = '';
+        pendingSpace = false;
+        bracketDepth = 0;
+        insertStatement = false;
+        insertHeader = false;
+        inputSource = false;
+        inputFunctionExpected = false;
+        formatExpected = false;
+        previousToken = '';
+    };
+    const finishFormat = (name, start) => {
+        formatExpected = false;
+        insertHeader = false;
+        if (name === 'VALUES')
+            return start; // VALUES uses SQL literals, including quoted semicolons.
+        // Other formats contain data, not SQL: preserve tabs, newlines, quotes and
+        // comment markers verbatim. Keep the legacy boundary: the next ';' or EOF.
+        // Semicolons inside raw data require a format-aware loader, outside this runner.
+        const semicolon = content.indexOf(';', start);
+        const end = semicolon === -1 ? content.length : semicolon;
+        current += content.slice(start, end);
+        flush(true);
+        return semicolon === -1 ? end : end + 1;
+    };
+    const n = content.length;
+    let i = 0;
+    while (i < n) {
+        const ch = content[i];
+        const next = i + 1 < n ? content[i + 1] : '';
+        // Line comments end at the next newline; markers inside quotes are copied below.
+        if ((ch === '-' && next === '-') ||
+            (ch === '/' && next === '/') ||
+            (ch === '#' && (next === '!' || next === '' || isSpace(next)))) {
+            while (i < n && content[i] !== '\n' && content[i] !== '\r') {
+                i += 1;
+            }
+            pendingSpace = true; // the comment stood between tokens; keep them apart
+            continue;
+        }
+        // block comments, nested
+        if (ch === '/' && next === '*') {
+            const start = i;
+            let depth = 1;
+            i += 2;
+            while (i < n && depth > 0) {
+                if (content[i] === '/' && content[i + 1] === '*') {
+                    depth += 1;
+                    i += 2;
+                }
+                else if (content[i] === '*' && content[i + 1] === '/') {
+                    depth -= 1;
+                    i += 2;
+                }
+                else {
+                    i += 1;
+                }
+            }
+            if (depth > 0) {
+                throw new Error(`unterminated block comment starting at ${describe_position(content, start)}`);
+            }
+            pendingSpace = true;
+            continue;
+        }
+        // quoted strings and identifiers: copied verbatim, including the quotes
+        if (ch === "'" || ch === '"' || ch === '`') {
+            const quote = ch;
+            let j = i + 1;
+            while (j < n) {
+                const c = content[j];
+                if (c === '\\' && j + 1 < n) {
+                    j += 2; // backslash escape: skip the escaped character whatever it is
+                    continue;
+                }
+                if (c === quote) {
+                    if (content[j + 1] === quote) {
+                        j += 2; // doubled quote inside the literal
+                        continue;
+                    }
+                    break; // closing quote
+                }
+                j += 1;
+            }
+            if (j >= n) {
+                const what = quote === "'" ? 'string literal' : 'quoted identifier';
+                throw new Error(`unterminated ${what} starting at ${describe_position(content, i)}`);
+            }
+            const end = j + 1; // include the closing quote
+            emitPlain(content.slice(i, end));
+            if (formatExpected && quote !== "'") {
+                i = finishFormat(content.slice(i + 1, j).toUpperCase(), end);
+                continue;
+            }
+            rememberToken(quote === "'" ? 'quoted' : content.slice(i + 1, j).toUpperCase(), true);
+            formatExpected = false;
+            i = end;
+            continue;
+        }
+        // dollar-quoted strings: $tag$ ... $tag$ (tag may be empty)
+        if (ch === '$' && (i === 0 || !/[A-Za-z0-9_$]/.test(content[i - 1]))) {
+            const tagMatch = /^\$[A-Za-z0-9_]*\$/.exec(content.slice(i));
+            if (tagMatch) {
+                const opener = tagMatch[0];
+                const close = content.indexOf(opener, i + opener.length);
+                if (close === -1) {
+                    throw new Error(`unterminated dollar-quoted string ${opener} starting at ${describe_position(content, i)}`);
+                }
+                const end = close + opener.length;
+                emitPlain(content.slice(i, end));
+                rememberToken('quoted');
+                formatExpected = false;
+                i = end;
+                continue;
+            }
+        }
+        // Commas inside arrays, maps and tuples belong to the setting value.
+        if (ch === separator && (separator === ';' || bracketDepth === 0)) {
+            flush();
+            i += 1;
+            continue;
+        }
+        // plain SQL
+        if (isSpace(ch)) {
+            pendingSpace = true;
+            i += 1;
+            continue;
+        }
+        // Recognize data only in an INSERT header or after input(). WITH/SELECT
+        // expressions and identifiers such as "AS format" must stay ordinary SQL.
+        const word = /^[A-Za-z_][A-Za-z0-9_$]*/.exec(content.slice(i));
+        if (word) {
+            const keyword = word[0].toUpperCase();
+            if (formatExpected) {
+                emitPlain(word[0]);
+                i = finishFormat(keyword, i + word[0].length);
+                continue;
+            }
+            if (separator === ';' && bracketDepth === 0) {
+                const identifier = ['INTO', 'TABLE', 'FUNCTION', '.', 'AS'].includes(previousToken);
+                if (!identifier) {
+                    if (keyword === 'INSERT' && (!current || /^WITH\b/i.test(current))) {
+                        insertStatement = true;
+                        insertHeader = true;
+                    }
+                    if (['WITH', 'SELECT', 'VALUES'].includes(keyword))
+                        insertHeader = false;
+                    formatExpected = insertStatement && (insertHeader || inputSource) && keyword === 'FORMAT';
+                }
+            }
+            rememberToken(keyword);
+            emitPlain(word[0]);
+            i += word[0].length;
+            continue;
+        }
+        // input() also accepts raw FORMAT data after an INSERT SELECT, even in a subquery.
+        if (ch === '(' && inputFunctionExpected)
+            inputSource = true;
+        inputFunctionExpected = false;
+        if ('([{'.includes(ch))
+            bracketDepth += 1;
+        if (')]}'.includes(ch))
+            bracketDepth -= 1;
+        previousToken = ch;
+        formatExpected = false;
+        emitPlain(ch);
+        i += 1;
+    }
+    flush();
+    return statements;
+};
+const split_statements = (content) => split_sql(content, ';');
+exports.split_statements = split_statements;
+const split_assignments = (content) => split_sql(content, ',');
+exports.split_assignments = split_assignments;

package/lib/version.d.ts

@@ -1 +1 @@
-export declare const VERSION = "1.4.0";
+export declare const VERSION = "1.5.0";

package/lib/version.js

@@ -1,4 +1,4 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.VERSION = void 0;
-exports.VERSION = '1.4.0'; // x-release-please-version
+exports.VERSION = '1.5.0'; // x-release-please-version

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clickhouse-migrations",
-  "version": "1.4.0",
+  "version": "1.5.0",
   "description": "ClickHouse Migrations",
   "bin": {
     "clickhouse-migrations": "lib/cli.js"
@@ -49,6 +49,7 @@
   "overrides": {
     "js-yaml": "^4.2.0",
     "@babel/core": "^7.29.7",
+    "brace-expansion": "^5.0.12",
     "test-exclude": "^8.0.0",
     "glob": "^13.0.6"
   },