clickhouse-migrations 1.1.4 → 1.3.0

package/README.md

@@ -1,4 +1,3 @@
-
 # clickhouse-migrations
 
 > ClickHouse Migrations CLI
@@ -13,11 +12,11 @@ npm install clickhouse-migrations
 
 Create a directory, where migrations will be stored. It will be used as the value for the `--migrations-home` option (or for environment variable `CH_MIGRATIONS_HOME`).
 
-In the directory, create migration files, which should be named like this: `1_some_text.sql`, `2_other_text.sql`, `10_more_test.sql`. What's important here is that the migration version number should come first, followed by an underscore (`_`), and then any text can follow. The version number should increase for every next migration. Please note that once a migration file has been applied to the database, it cannot be modified or removed. 
+In the directory, create migration files, which should be named like this: `1_some_text.sql`, `2_other_text.sql`, `10_more_test.sql`. What's important here is that the migration version number should come first, followed by an underscore (`_`), and then any text can follow. The version number should increase for every next migration. Please note that once a migration file has been applied to the database, it cannot be modified or removed.
 
-For migrations' content should be used correct SQL ClickHouse queries. Multiple queries can be used in a single migration file, and each query should be terminated with a semicolon (;). The queries could be idempotent - for example: `CREATE TABLE IF NOT EXISTS table ...;` Clickhouse settings, that can be included at the query level, can be added like `SET allow_experimental_object_type = 1;`. For adding comments should be used `--`, `# `, `#!`. 
+For migrations' content should be used correct SQL ClickHouse queries. Multiple queries can be used in a single migration file, and each query should be terminated with a semicolon (;). The queries could be idempotent - for example: `CREATE TABLE IF NOT EXISTS table ...;` Clickhouse settings, that can be included at the query level, can be added like `SET allow_experimental_object_type = 1;`. For adding comments should be used `--`, `# `, `#!`.
 
-If the database provided in the `--db` option (or in `CH_MIGRATIONS_DB`) doesn't exist, it will be created automatically.
+If the database provided in the `--db` option (or in `CH_MIGRATIONS_DB`) doesn't exist, it will be created automatically. To disable this behavior (for example, when the user has no privileges to create databases), use the `--skip-db-creation` option (or set `CH_MIGRATIONS_SKIP_DB_CREATION` to `'true'`); in that case the database must already exist.
 
 For TLS/HTTPS connections, you can provide a custom CA certificate and optional client certificate/key via the `--ca-cert`, `--cert`, and `--key` options (or the `CH_MIGRATIONS_CA_CERT`, `CH_MIGRATIONS_CERT`, and `CH_MIGRATIONS_KEY` environment variables).
 
@@ -26,7 +25,7 @@ For TLS/HTTPS connections, you can provide a custom CA certificate and optional
     $ clickhouse-migrations migrate <options>
 
   Required options
-      --host=<name>             Clickhouse hostname 
+      --host=<name>             Clickhouse hostname
                                   (ex. https://clickhouse:8123)
       --user=<name>             Username
       --password=<password>     Password
@@ -36,11 +35,14 @@ For TLS/HTTPS connections, you can provide a custom CA certificate and optional
   Optional options
       --db-engine=<value>       ON CLUSTER and/or ENGINE for DB
                                   (default: 'ENGINE=Atomic')
-      --timeout=<value>         Client request timeout 
+      --table-engine=<value>    Engine for the _migrations table
+                                  (default: 'MergeTree')
+      --timeout=<value>         Client request timeout
                                   (milliseconds, default: 30000)
       --ca-cert=<path>          CA certificate file path
       --cert=<path>             Client certificate file path
-      --key=<path>              Client key file path                            
+      --key=<path>              Client key file path
+      --skip-db-creation        Skip database creation
 
   Environment variables
       Instead of options can be used environment variables.
@@ -51,24 +53,30 @@ For TLS/HTTPS connections, you can provide a custom CA certificate and optional
       CH_MIGRATIONS_HOME        Migrations' directory (--migrations-home)
 
       CH_MIGRATIONS_DB_ENGINE   (optional) DB engine (--db-engine)
-      CH_MIGRATIONS_TIMEOUT     (optional) Client request timeout 
+      CH_MIGRATIONS_TABLE_ENGINE  (optional) Migrations table engine (--table-engine)
+      CH_MIGRATIONS_TIMEOUT     (optional) Client request timeout
                                   (--timeout)
       CH_MIGRATIONS_CA_CERT     (optional) CA certificate file path
       CH_MIGRATIONS_CERT        (optional) Client certificate file path
       CH_MIGRATIONS_KEY         (optional) Client key file path
+      CH_MIGRATIONS_SKIP_DB_CREATION
+                                (optional) Skip database creation,
+                                  set to 'true' to enable
+                                  (--skip-db-creation)
 
 
   CLI executions examples
     settings are passed as command-line options
       clickhouse-migrations migrate --host=http://localhost:8123
-      --user=default --password='' --db=analytics 
+      --user=default --password='' --db=analytics
       --migrations-home=/app/clickhouse/migrations
 
     settings provided as options, including timeout and db-engine
-      clickhouse-migrations migrate --host=http://localhost:8123 
-      --user=default --password='' --db=analytics 
-      --migrations-home=/app/clickhouse/migrations --timeout=60000 
-      --db-engine="ON CLUSTER default ENGINE=Replicated('{replica}')"    
+      clickhouse-migrations migrate --host=http://localhost:8123
+      --user=default --password='' --db=analytics
+      --migrations-home=/app/clickhouse/migrations --timeout=60000
+      --db-engine="ON CLUSTER default ENGINE=Replicated('{replica}')"
+      --table-engine="ReplicatedMergeTree('/clickhouse/tables/{database}/migrations', '{replica}')"
 
     settings provided as environment variables
       clickhouse-migrations migrate
@@ -81,10 +89,16 @@ For TLS/HTTPS connections, you can provide a custom CA certificate and optional
       --user=default --password='' --db=analytics
       --migrations-home=/app/clickhouse/migrations
       --ca-cert=/app/certs/ca.pem --cert=/app/certs/client.crt --key=/app/certs/client.key
+
+    skip database creation (assumes database already exists)
+      clickhouse-migrations migrate --host=http://localhost:8123
+      --user=default --password='' --db=analytics
+      --migrations-home=/app/clickhouse/migrations --skip-db-creation
 ```
 
 Migration file example:
 (e.g., located at /app/clickhouse/migrations/1_init.sql)
+
 ```
 -- an example of migration file 1_init.sql
 
@@ -96,8 +110,8 @@ CREATE TABLE IF NOT EXISTS events (
   event JSON
 )
 ENGINE=AggregatingMergeTree
-PARTITION BY toYYYYMM(timestamp) 
-SAMPLE BY session_id 
-ORDER BY (session_id) 
+PARTITION BY toYYYYMM(timestamp)
+SAMPLE BY session_id
+ORDER BY (session_id)
 SETTINGS index_granularity = 8192;
-```      
+```

package/lib/migrate.d.ts

@@ -1,3 +1,3 @@
-declare const migration: (migrations_home: string, host: string, username: string, password: string, db_name: string, db_engine?: string, timeout?: string, ca_cert?: string | undefined, cert?: string | undefined, key?: string | undefined) => Promise<void>;
+declare const migration: (migrations_home: string, host: string, username: string, password: string, db_name: string, db_engine?: string, table_engine?: string, timeout?: string, ca_cert?: string | undefined, cert?: string | undefined, key?: string | undefined, skip_db_creation?: boolean) => Promise<void>;
 declare const migrate: () => void;
 export { migrate, migration };

package/lib/migrate.js

@@ -35,6 +35,15 @@ var __generator = (this && this.__generator) || function (thisArg, body) {
         if (op[0] & 5) throw op[1]; return { value: op[0] ? op[1] : void 0, done: true };
     }
 };
+var __spreadArray = (this && this.__spreadArray) || function (to, from, pack) {
+    if (pack || arguments.length === 2) for (var i = 0, l = from.length, ar; i < l; i++) {
+        if (ar || !(i in from)) {
+            if (!ar) ar = Array.prototype.slice.call(from, 0, i);
+            ar[i] = from[i];
+        }
+    }
+    return to.concat(ar || Array.prototype.slice.call(from));
+};
 var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
@@ -90,8 +99,8 @@ var connect = function (url, username, password, db_name, timeout, ca_cert, cert
     }
     return (0, client_1.createClient)(db_params);
 };
-var create_db = function (host, username, password, db_name, db_engine, timeout, ca_cert, cert, key) { return __awaiter(void 0, void 0, void 0, function () {
-    var client, e_1, q, e_2;
+var create_db = function (host, username, password, db_name, db_engine, timeout, ca_cert, cert, key, skip_db_creation) { return __awaiter(void 0, void 0, void 0, function () {
+    var client, e_1, exists, resultSet, rows, e_2, q, e_3;
     return __generator(this, function (_a) {
         switch (_a.label) {
             case 0:
@@ -109,60 +118,105 @@ var create_db = function (host, username, password, db_name, db_engine, timeout,
                 process.exit(1);
                 return [3 /*break*/, 4];
             case 4:
-                q = db_engine
-                    ? "CREATE DATABASE IF NOT EXISTS \"".concat(db_name, "\" ").concat(db_engine)
-                    : "CREATE DATABASE IF NOT EXISTS \"".concat(db_name, "\"");
+                if (!skip_db_creation) return [3 /*break*/, 14];
+                exists = false;
                 _a.label = 5;
             case 5:
-                _a.trys.push([5, 7, , 8]);
-                return [4 /*yield*/, client.command({
-                        query: q,
-                        clickhouse_settings: {
-                            wait_end_of_query: 1,
-                        },
+                _a.trys.push([5, 8, , 10]);
+                return [4 /*yield*/, client.query({
+                        query: "SELECT 1 AS ok FROM system.databases WHERE name = {db_name:String}",
+                        query_params: { db_name: db_name },
+                        format: 'JSONEachRow',
                     })];
             case 6:
-                _a.sent();
-                return [3 /*break*/, 8];
+                resultSet = _a.sent();
+                return [4 /*yield*/, resultSet.json()];
             case 7:
+                rows = (_a.sent());
+                exists = rows.length > 0;
+                return [3 /*break*/, 10];
+            case 8:
                 e_2 = _a.sent();
-                log('error', "can't create the database ".concat(db_name, "."), e_2.message);
-                process.exit(1);
-                return [3 /*break*/, 8];
-            case 8: return [4 /*yield*/, client.close()];
+                log('error', "can't verify that the database ".concat(db_name, " exists."), e_2.message);
+                return [4 /*yield*/, client.close()];
             case 9:
+                _a.sent();
+                process.exit(1);
+                return [3 /*break*/, 10];
+            case 10:
+                if (!!exists) return [3 /*break*/, 12];
+                log('error', "the database ".concat(db_name, " doesn't exist and --skip-db-creation is set. Please create the database manually or remove --skip-db-creation."));
+                return [4 /*yield*/, client.close()];
+            case 11:
+                _a.sent();
+                process.exit(1);
+                _a.label = 12;
+            case 12: return [4 /*yield*/, client.close()];
+            case 13:
                 _a.sent();
                 return [2 /*return*/];
-        }
-    });
-}); };
-var init_migration_table = function (client) { return __awaiter(void 0, void 0, void 0, function () {
-    var q, e_3;
-    return __generator(this, function (_a) {
-        switch (_a.label) {
-            case 0:
-                q = "CREATE TABLE IF NOT EXISTS _migrations (\n      uid UUID DEFAULT generateUUIDv4(),\n      version UInt32,\n      checksum String,\n      migration_name String,\n      applied_at DateTime DEFAULT now()\n    )\n    ENGINE = MergeTree\n    ORDER BY tuple(applied_at)";
-                _a.label = 1;
-            case 1:
-                _a.trys.push([1, 3, , 4]);
+            case 14:
+                q = db_engine
+                    ? "CREATE DATABASE IF NOT EXISTS \"".concat(db_name, "\" ").concat(db_engine)
+                    : "CREATE DATABASE IF NOT EXISTS \"".concat(db_name, "\"");
+                _a.label = 15;
+            case 15:
+                _a.trys.push([15, 17, , 18]);
                 return [4 /*yield*/, client.command({
                         query: q,
                         clickhouse_settings: {
                             wait_end_of_query: 1,
                         },
                     })];
-            case 2:
+            case 16:
                 _a.sent();
-                return [3 /*break*/, 4];
-            case 3:
+                return [3 /*break*/, 18];
+            case 17:
                 e_3 = _a.sent();
-                log('error', "can't create the _migrations table.", e_3.message);
+                log('error', "can't create the database ".concat(db_name, "."), e_3.message);
                 process.exit(1);
-                return [3 /*break*/, 4];
-            case 4: return [2 /*return*/];
+                return [3 /*break*/, 18];
+            case 18: return [4 /*yield*/, client.close()];
+            case 19:
+                _a.sent();
+                return [2 /*return*/];
         }
     });
 }); };
+var init_migration_table = function (client_2) {
+    var args_1 = [];
+    for (var _i = 1; _i < arguments.length; _i++) {
+        args_1[_i - 1] = arguments[_i];
+    }
+    return __awaiter(void 0, __spreadArray([client_2], args_1, true), void 0, function (client, table_engine) {
+        var q, e_4;
+        if (table_engine === void 0) { table_engine = 'MergeTree'; }
+        return __generator(this, function (_a) {
+            switch (_a.label) {
+                case 0:
+                    q = "CREATE TABLE IF NOT EXISTS _migrations (\n      uid UUID DEFAULT generateUUIDv4(),\n      version UInt32,\n      checksum String,\n      migration_name String,\n      applied_at DateTime DEFAULT now()\n    )\n    ENGINE = ".concat(table_engine, "\n    ORDER BY tuple(applied_at)");
+                    _a.label = 1;
+                case 1:
+                    _a.trys.push([1, 3, , 4]);
+                    return [4 /*yield*/, client.command({
+                            query: q,
+                            clickhouse_settings: {
+                                wait_end_of_query: 1,
+                            },
+                        })];
+                case 2:
+                    _a.sent();
+                    return [3 /*break*/, 4];
+                case 3:
+                    e_4 = _a.sent();
+                    log('error', "can't create the _migrations table.", e_4.message);
+                    process.exit(1);
+                    return [3 /*break*/, 4];
+                case 4: return [2 /*return*/];
+            }
+        });
+    });
+};
 var get_migrations = function (migrations_home) {
     var files;
     try {
@@ -196,7 +250,7 @@ var get_migrations = function (migrations_home) {
     return migrations;
 };
 var apply_migrations = function (client, migrations, migrations_home) { return __awaiter(void 0, void 0, void 0, function () {
-    var migration_query_result, resultSet, e_4, migrations_applied, applied_migrations, _i, migrations_1, migration_1, content, checksum, queries, sets, _a, queries_1, query, e_5, e_6;
+    var migration_query_result, resultSet, e_5, migrations_applied, applied_migrations, _i, migrations_1, migration_1, content, checksum, queries, sets, _a, queries_1, query, e_6, e_7;
     return __generator(this, function (_b) {
         switch (_b.label) {
             case 0:
@@ -215,8 +269,8 @@ var apply_migrations = function (client, migrations, migrations_home) { return _
                 migration_query_result = _b.sent();
                 return [3 /*break*/, 5];
             case 4:
-                e_4 = _b.sent();
-                log('error', "can't select data from the _migrations table.", e_4.message);
+                e_5 = _b.sent();
+                log('error', "can't select data from the _migrations table.", e_5.message);
                 process.exit(1);
                 return [3 /*break*/, 5];
             case 5:
@@ -272,11 +326,11 @@ var apply_migrations = function (client, migrations, migrations_home) { return _
                 _b.sent();
                 return [3 /*break*/, 11];
             case 10:
-                e_5 = _b.sent();
+                e_6 = _b.sent();
                 if (applied_migrations) {
                     log('info', "The migration(s) ".concat(applied_migrations, " was successfully applied!"));
                 }
-                log('error', "the migrations ".concat(migration_1.file, " has an error. Please, fix it (be sure that already executed parts of the migration would not be run second time) and re-run migration script."), e_5.message);
+                log('error', "the migrations ".concat(migration_1.file, " has an error. Please, fix it (be sure that already executed parts of the migration would not be run second time) and re-run migration script."), e_6.message);
                 process.exit(1);
                 return [3 /*break*/, 11];
             case 11:
@@ -293,8 +347,8 @@ var apply_migrations = function (client, migrations, migrations_home) { return _
                 _b.sent();
                 return [3 /*break*/, 15];
             case 14:
-                e_6 = _b.sent();
-                log('error', "can't insert a data into the table _migrations.", e_6.message);
+                e_7 = _b.sent();
+                log('error', "can't insert a data into the table _migrations.", e_7.message);
                 process.exit(1);
                 return [3 /*break*/, 15];
             case 15:
@@ -314,17 +368,17 @@ var apply_migrations = function (client, migrations, migrations_home) { return _
         }
     });
 }); };
-var migration = function (migrations_home, host, username, password, db_name, db_engine, timeout, ca_cert, cert, key) { return __awaiter(void 0, void 0, void 0, function () {
+var migration = function (migrations_home, host, username, password, db_name, db_engine, table_engine, timeout, ca_cert, cert, key, skip_db_creation) { return __awaiter(void 0, void 0, void 0, function () {
     var migrations, client;
     return __generator(this, function (_a) {
         switch (_a.label) {
             case 0:
                 migrations = get_migrations(migrations_home);
-                return [4 /*yield*/, create_db(host, username, password, db_name, db_engine, timeout, ca_cert, cert, key)];
+                return [4 /*yield*/, create_db(host, username, password, db_name, db_engine, timeout, ca_cert, cert, key, skip_db_creation)];
             case 1:
                 _a.sent();
                 client = connect(host, username, password, db_name, timeout, ca_cert, cert, key);
-                return [4 /*yield*/, init_migration_table(client)];
+                return [4 /*yield*/, init_migration_table(client, table_engine)];
             case 2:
                 _a.sent();
                 return [4 /*yield*/, apply_migrations(client, migrations, migrations_home)];
@@ -350,14 +404,16 @@ var migrate = function () {
         .requiredOption('--db <name>', 'Database name', process.env.CH_MIGRATIONS_DB)
         .requiredOption('--migrations-home <dir>', "Migrations' directory", process.env.CH_MIGRATIONS_HOME)
         .option('--db-engine <value>', 'ON CLUSTER and/or ENGINE clauses for database (default: "ENGINE=Atomic")', process.env.CH_MIGRATIONS_DB_ENGINE)
+        .option('--table-engine <value>', 'Engine for the _migrations table (default: "MergeTree")', process.env.CH_MIGRATIONS_TABLE_ENGINE)
         .option('--timeout <value>', 'Client request timeout (milliseconds, default value 30000)', process.env.CH_MIGRATIONS_TIMEOUT)
         .option('--ca-cert <path>', 'CA certificate file path', process.env.CH_MIGRATIONS_CA_CERT)
         .option('--cert <path>', 'Client certificate file path', process.env.CH_MIGRATIONS_CERT)
         .option('--key <path>', 'Client key file path', process.env.CH_MIGRATIONS_KEY)
+        .option('--skip-db-creation', 'Skip database creation', process.env.CH_MIGRATIONS_SKIP_DB_CREATION === 'true')
         .action(function (options) { return __awaiter(void 0, void 0, void 0, function () {
         return __generator(this, function (_a) {
             switch (_a.label) {
-                case 0: return [4 /*yield*/, migration(options.migrationsHome, options.host, options.user, options.password, options.db, options.dbEngine, options.timeout, options.caCert, options.cert, options.key)];
+                case 0: return [4 /*yield*/, migration(options.migrationsHome, options.host, options.user, options.password, options.db, options.dbEngine, options.tableEngine, options.timeout, options.caCert, options.cert, options.key, options.skipDbCreation)];
                 case 1:
                     _a.sent();
                     return [2 /*return*/];

package/lib/version.d.ts

@@ -1 +1 @@
-export declare const VERSION = "1.1.4";
+export declare const VERSION = "1.3.0";

package/lib/version.js

@@ -1,4 +1,4 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.VERSION = void 0;
-exports.VERSION = '1.1.4'; // x-release-please-version
+exports.VERSION = '1.3.0'; // x-release-please-version

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "clickhouse-migrations",
-  "version": "1.1.4",
+  "version": "1.3.0",
   "description": "ClickHouse Migrations",
   "bin": {
     "clickhouse-migrations": "lib/cli.js"