@verdnatura/myt 1.8.6 → 2.0.2

package/README.md

@@ -1,6 +1,6 @@
-# Myt - MySQL version control
+# Myt - MariaDB and MySQL version control
 
-Utilities to ease the maintenance of MySQL or MariaDB database versioning using
+Utilities to ease the maintenance of MariaDB or MySQL database versioning using
 a Git repository.
 
 This project is just to bring an idea to life and is still in an early stage of
@@ -9,7 +9,7 @@ development, so any help is welcomed! Feel free to contribute.
 ## Requirements
 
 * Git
-* Docker (Optional, used to manage local server)
+* Docker >= 23 with BuildKit (Optional, used to manage local server)
 
 ## Installation
 
@@ -57,11 +57,36 @@ Local server management commands:
 
  * **dump**: Export database structure and fixtures.
  * **fixtures**: Export local database fixtures.
+ * **build**: Builds local database server image.
+ * **apply**: Applies structure, fixtures and changes to database.
  * **run**: Build and start local database server container.
  * **start**: Start local database server container.
 
 Each command can have its own specific commandline options.
 
+### Usage as library
+
+It is also possible to use myt as a JavaScript library, not only from the
+command line. Below is a reference example.
+
+```js
+const Myt = require('@verdnatura/myt/myt');
+const Run = require('@verdnatura/myt/myt-run');
+
+async function main() {
+   const myt = new Myt();
+   await myt.init({
+      workspace: path.join(__dirname, '..')
+   });
+   const server = await myt.run(Run, {
+      random: true,
+      persist: process.platform != 'linux'
+   });
+   console.log(server.dbConfig);
+   await myt.deinit();
+}
+```
+
 ## Basic information
 
 First of all you have to initialize the workspace.
@@ -143,38 +168,6 @@ Don't place your PL/SQL objects here, use the routines folder!
      `- 00-sameNumbers.sql
 ```
 
-### Environment-Specific Versioning with Realms
-
-#### Overview
-
-We have introduced a new feature that allows users to apply version-specific changes based on their configured environment, or "realm". This ensures that certain changes are only applied when the user is operating within a specific realm, providing an additional layer of customization and control.
-
-#### Configuration
-
-To make use of this feature, you need to configure your realm in the `versionConfig` table. Set your realm by inserting or updating a record in this table. The `realm` field should contain the identifier of your environment.
-
-#### File Naming Convention
-
-To designate a file as realm-specific, include a `.your_realm.` segment before the file extension. For example, if your realm is set to 'ab', the file should be named like this: `filename.ab.sql`.
-
-#### How It Works
-
-1. **Set your Realm**: Configure your realm in the `versionConfig` table.
-  
-2. **Add Files**: Place your realm-specific files in the `versions` folder. Make sure to follow the naming convention.
-
-3. **Version Order**: Files are applied in the existing version order. In addition to that, the realm is validated.
-
-4. **Apply Changes**: Run the usual versioning commands. The realm-specific files will only be applied if your configured realm matches the realm in the file name.
-
-#### Important Notes
-
-- If no realm is configured, realm-specific files will be ignored.
-  
-- If you have a realm configured but the realm-specific files belong to a different realm, those files will also be ignored.
-
-This feature allows for greater flexibility when working in different environments, making it easier to manage realm-specific changes in a shared repository.
-
 ### Local server
 
 The local server is created as a MariaDB Docker container using the base dump 
@@ -185,29 +178,46 @@ routines.
 
 You can create your local fixture and structure files.
 
-* *dump/dump.before.sql*
-* *dump/dump.after.sql*
-* *dump/fixtures.before.sql*
-* *dump/fixtures.after.sql*
-* *dump/fixtures.local.sql*
+* *structure/before.sql*
+* *structure/after.sql*
+* *fixtures/before.sql*
+* *fixtures/after.sql*
+* *fixtures/local.sql*
 
 ### Realms
 
-You can create your local realms folders.
+Realms allows users to apply version-specific changes based on their configured
+environment, or "realm". This ensures that certain changes are only applied
+when the user is operating within a specific realm, providing an additional
+layer of customization and control.
+
+Realm folders structure.
 
 ```text
-   dump
-   `- realms
-      |- marvel
-      |  `- 00-spiderman.sql
-      |  `- 01-hulk.sql
-      `- dc
-         `- 00-superman.sql
-         `- 01-batman.sql
-         `- 02-wonder_woman.sql
-         `- 03-flash.sql
+   realms
+   |- marvel
+   |  `- 00-spiderman.sql
+   |  `- 01-hulk.sql
+   `- dc
+      `- 00-superman.sql
+      `- 01-batman.sql
+      `- 02-wonder_woman.sql
+      `- 03-flash.sql
 ```
 
+#### Configuration
+
+To make use of this feature, you need to configure your realm in the
+`versionConfig` table. Set your realm by inserting or updating a record in
+this table. The `realm` field should contain the identifier of your
+environment.
+
+#### Realm-specific version files
+
+To designate a version file as realm-specific, include a `.your_realm.` segment
+before the file extension. For example, if your realm is set to 'foo', the file 
+should be named like this: `filename.foo.sql`.
+
 ## Versioning commands
 
 ### init
@@ -290,13 +300,31 @@ specified *local* is used.
 $ myt fixtures [<remote>]
 ```
 
+### build
+
+Builds local database server image. It only rebuilds the image when sources
+have been modified.
+
+```text
+$ myt build [-n|--name] [-t|--tag] [-f|--force] [-m|--realm <string>]
+```
+
+### apply
+
+Applies structure, fixtures and changes to database. Primarily used by the
+build command to rebuild the local database.
+
+```text
+$ myt apply [-s|--structure] [-c|--changes] [-m|--realm <string>] [-l|--load <string>]
+```
+
 ### run
 
 Builds and starts local database server container. It only rebuilds the image
 when dump have been modified.
 
 ```text
-$ myt run [<realm>|-m|--realm <string>] [-c|--ci] [-r|--random] [-p|--persist] [-n|--network <string>] [-i|--ip]
+$ myt run [<realm>|-m|--realm <string>] [-c|--ci] [-r|--random] [-p|--persist] [-k|--keep] [-n|--network <string>] [-i|--ip]
 ```
 
 ### start

package/assets/myt.default.yml

@@ -7,9 +7,12 @@ mockFunctions:
   - mockUtcTime
 sumViews: true
 defaultDefiner: root@localhost
+protectedRemotes:
+  - production
 localRemotes:
   - local
   - docker
+  - socket
 deprecMarkRegex: __$
 deprecCommentRegex: ^@deprecated [0-9]{4}-[0-9]{2}-[0-9]{2}
 deprecDateRegex: '[0-9]{4}-[0-9]{2}-[0-9]{2}'
@@ -28,6 +31,10 @@ privileges:
     - procs_priv
   where: >-
     `Host` = ''
+dbImageTag: mariadb:10.11.16-jammy
+rootPassword: root
+triggersImport: after
+gitShortLen: 12
 schemas:
   - myt
 fixtures:

package/assets/structure.sql

@@ -23,7 +23,7 @@ CREATE TABLE `versionLog` (
 ALTER TABLE `versionLog`
     ADD PRIMARY KEY (`code`,`number`,`file`);
 
-CREATE TABLE `util`.`versionConfig` (
+CREATE TABLE `versionConfig` (
     `id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
     `realm` VARCHAR(16) NULL DEFAULT NULL COMMENT 'Data set on which the project runs'
 ) ENGINE=InnoDB;

package/{server/Dockerfile.client → docker/client/Dockerfile}

@@ -1,4 +1,4 @@
-FROM debian:bullseye-slim
+FROM debian:11.11-slim
 
 ARG DEBIAN_FRONTEND=noninteractive
 RUN apt-get update \
@@ -8,7 +8,7 @@ RUN apt-get update \
     && rm -rf /var/lib/apt/lists/*
 
 COPY \
-    server/docker-dump.sh \
-    server/docker-fixtures.sh \
+    docker-dump.sh \
+    docker-fixtures.sh \
     /usr/local/bin/
 WORKDIR /workspace

package/docker/server/Dockerfile

@@ -0,0 +1,37 @@
+ARG MYT_TAG=myt/app
+ARG BASE_TAG=myt/base
+
+FROM ${MYT_TAG} AS dump
+
+ARG ROOT_PASS
+ENV ROOT_PASS=${ROOT_PASS}
+
+RUN mkdir -p /mysql-template \
+    && chown mysql:mysql /mysql-template
+
+WORKDIR /workspace
+
+COPY myt.config.yml .
+COPY remotes/socket.ini ./remotes/
+COPY structure structure
+COPY fixtures fixtures
+COPY versions versions
+COPY routines routines
+COPY realms realms
+
+ARG MYT_REALM
+ARG MYT_COMMIT
+RUN --mount=type=tmpfs,target=/var/lib/mysql \
+    chown mysql:mysql /var/lib/mysql \
+    && gosu mysql docker-init.sh
+
+FROM ${BASE_TAG}
+
+COPY --from=dump --chown=mysql:mysql \
+    /mysql-template /mysql-template
+
+ENTRYPOINT ["docker-start.sh"]
+CMD ["mariadbd"]
+
+HEALTHCHECK --interval=2s --timeout=10s --retries=200 \
+    CMD gosu mysql mariadb-admin ping || exit 1

package/docker/server/Dockerfile.base

@@ -0,0 +1,6 @@
+ARG DB_TAG=mariadb:10.11.16-jammy
+FROM ${DB_TAG}
+
+ARG SERVER_DIR=docker/server
+COPY ${SERVER_DIR}/docker.cnf /etc/mysql/conf.d/
+COPY ${SERVER_DIR}/docker-start.sh /usr/local/bin/

package/docker/server/Dockerfile.myt

@@ -0,0 +1,35 @@
+ARG BASE_TAG=myt/base
+FROM ${BASE_TAG}
+
+# Myt
+
+ARG DEBIAN_FRONTEND=noninteractive
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        curl \
+        ca-certificates \
+        gnupg2 \
+    && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && npm install -g corepack@0.34.6 \
+    && corepack enable \
+    && rm -rf /var/lib/apt/lists/*
+
+# RUN apt-get update \
+#     && apt-get install -y --no-install-recommends \
+#         libkrb5-dev \
+#         libssl-dev \
+#         build-essential \
+#     && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /myt
+COPY package.json ./
+RUN pnpm install --prod --ignore-scripts
+
+COPY . ./
+RUN npm install -g /myt
+
+# Server
+
+ARG SERVER_DIR=docker/server
+COPY ${SERVER_DIR}/docker-init.sh /usr/local/bin/

package/docker/server/docker-init.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+set -e
+
+export MYSQL_ROOT_PASSWORD="$ROOT_PASS"
+export MYSQL_ALLOW_EMPTY_PASSWORD='true'
+export MARIADB_MYSQL_LOCALHOST_USER='yes'
+
+. /usr/local/bin/docker-entrypoint.sh
+
+ARGS=(mariadbd --datadir=/var/lib/mysql)
+
+mysql_check_config ${ARGS[@]}
+docker_setup_env ${ARGS[@]}
+docker_create_db_directories
+docker_verify_minimum_env
+docker_init_database_dir ${ARGS[@]}
+
+docker_temp_server_start ${ARGS[@]}
+docker_setup_db
+
+MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb -u root \
+    -e "GRANT ALL PRIVILEGES ON *.* TO mysql@localhost WITH GRANT OPTION"
+
+docker_process_init_files /docker-entrypoint-initdb.d/*
+
+myt apply \
+    --debug \
+    --structure \
+    --changes \
+    --remote socket \
+    --docker \
+    --load "$MYT_COMMIT" \
+    --realm "$MYT_REALM"
+
+docker_temp_server_stop
+cp -a /var/lib/mysql/. /mysql-template

package/{server → docker/server}/docker-start.sh

@@ -4,7 +4,7 @@ set -e
 # XXX: Workaround to avoid OverlayFS bug on MacOs
 # https://docs.docker.com/storage/storagedriver/overlayfs-driver/#limitations-on-overlayfs-compatibility
 
-if [ ! -d /var/lib/mysql/mysql ]; then
+if [[ ! -d /var/lib/mysql/mysql && -d /mysql-template ]]; then
 	chown mysql:mysql /var/lib/mysql
 	cp -a /mysql-template/. /var/lib/mysql
 fi

package/lib/command.js

@@ -3,12 +3,16 @@
  */
 module.exports = class MytCommand{
     constructor(myt, opts) {
-        this.myt = myt;
-        this.opts = opts;
+        Object.assign(this, {
+            myt,
+            ctx: myt.ctx,
+            cfg: myt.cfg,
+            opts: opts || {}
+        });
         this.handlers = {};
     }
 
-    async cli(myt, opts) {
+    async cli() {
         const reporter = this.constructor.reporter;
         if (reporter)
         for (const event in reporter) {
@@ -19,11 +23,15 @@ module.exports = class MytCommand{
                 this.on(event, handler);
         }
 
-        await this.run(myt, opts);
+        return await this.run();
     }
 
-    async run(myt, opts) {
-        throw new Error('run command not defined');
+    async run() {
+        return await this._run(this.myt, this.ctx, this.cfg, this.opts);
+    }
+
+    async _run(myt, ctx, cfg, opts) {
+        throw new Error('run method not defined');
     }
 
     on(event, handler) {

package/lib/docker.js

@@ -3,7 +3,7 @@ const execFile = require('child_process').execFile;
 const camelToSnake = require('./util').camelToSnake;
 
 const docker = {
-    async run(image, commandArgs, options, execOptions) {
+    async run(image, commandArgs, options, execOptions, debug) {
         const args = commandArgs
             ? [image].concat(commandArgs)
             : image;
@@ -13,19 +13,24 @@ const docker = {
             args,
             options,
             execMode,
-            execOptions
+            execOptions,
+            debug
         );
         return options.detach
             ? new Container(child.stdout.trim())
             : child;
     },
 
-    async build(url, options, execOptions) {
+    async build(url, options, debug) {
+        if (debug === true && options.progress == null)
+            options = Object.assign({progress: 'plain'}, options);
+
         return await this.command('build',
             url,
             options,
             'spawn',
-            execOptions
+            undefined,
+            debug
         );
     },
 
@@ -50,51 +55,72 @@ const docker = {
         return await ct.inspect(options);
     },
 
-    async command(command, args, options, execMode, execOptions) {
+    async images(tag, options) {
+        options = Object.assign({format: 'json'}, options);
+        const child = await docker.command('images', tag, options);
+        return JSON.parse(child.stdout || null);
+    },
+
+    async command(command, args, options, execMode, execOptions, debug) {
         const execArgs = [command];
 
         if (options)
-            for (const option in options) {
-                execArgs.push(`--${camelToSnake(option)}`);
-                if (typeof options[option] !== 'boolean')
-                    execArgs.push(options[option]);
+        for (const option in options) {
+            const value = options[option];
+            const param = `--${camelToSnake(option)}`;
+
+            if (typeof value == 'boolean') {
+                execArgs.push (param)
+            } else if (Array.isArray(value)) {
+                for (const val of value)
+                    execArgs.push(param, val)
+            } else {
+                execArgs.push(param, value);
             }
+        }
 
         if (Array.isArray(args))
             Array.prototype.push.apply(execArgs, args);
         else if (args)
             execArgs.push(args);
 
-        return await new Promise((resolve, reject) => {
-            if (execMode == 'spawn') {
-                if (execOptions === true)
-                    execOptions = {
-                        stdio: [
-                            process.stdin,
-                            process.stdout,
-                            process.stderr
-                        ] 
-                    };
-
-                const child = spawn('docker', execArgs, execOptions || undefined);
+        const dockerBin = 'docker';
+
+        if (debug === true) {
+            const quotedArgs = execArgs
+                .map(x => /\s/g.test(x) ? `"${x}"` : x)
+                .join(' ');
+            console.debug('Command:', `${dockerBin} ${quotedArgs}`.yellow);
+        }
+
+        if (execMode == 'spawn') {
+            if (debug === true && !execOptions)
+                execOptions = {
+                    stdio: [
+                        process.stdin,
+                        process.stdout,
+                        process.stderr
+                    ] 
+                };
+            const child = spawn(dockerBin, execArgs, execOptions);
+            return await new Promise((resolve, reject) => {
                 child.on('exit', code => {
-                    if (code !== 0) {
-                        const quotedArgs = execArgs
-                            .map(x => /\s/g.test(x) ? `"${x}"` : x)
-                            .join(' ');
-                        reject(new Error(`Docker exit code ${code}: 'docker ${quotedArgs}'`));
-                    } else
+                    if (code !== 0)
+                        reject(new Error(`${dockerBin} exit code ${code}`));
+                    else
                         resolve(code);
                 });
-            } else {
-                execFile('docker', execArgs, (err, stdout, stderr) => {
+            });
+        } else {
+            return await new Promise((resolve, reject) => {
+                execFile(dockerBin, execArgs, (err, stdout, stderr) => {
                     if (err)
                         reject(err);
                     else
                         resolve({stdout, stderr});
                 });
-            }
-        });
+            });
+        }
     }
 };
 
@@ -119,13 +145,13 @@ class Container {
 
     async inspect(options) {
         const child = await docker.command('inspect', this.id, options);
-        return JSON.parse(child.stdout);
+        return JSON.parse(child.stdout || null);
     }
 
-    async exec(options, command, commandArgs, execMode, execOptions) {
+    async exec(options, command, commandArgs, execMode, execOptions, debug) {
         let args = [this.id, command];
         if (commandArgs) args = args.concat(commandArgs);
-        await docker.command('exec', args, options, execMode, execOptions);
+        await docker.command('exec', args, options, execMode, execOptions, debug);
     }
 }
 

package/lib/dumper.js

@@ -4,8 +4,10 @@ const path = require('path');
 const SqlString = require('sqlstring');
 
 module.exports = class Dumper {
-    constructor(opts) {
-        this.opts = opts;
+    constructor(myt) {
+        this.myt = myt;
+        this.cfg = myt.cfg;
+        this.ctx = myt.ctx;
     }
 
     async init(dumpDir, dumpFile) {
@@ -18,11 +20,10 @@ module.exports = class Dumper {
         // "invalid" WriteStream
         const dumpStream = await fs.createWriteStream(dumpPath);
 
-        const buidDir = path.join(__dirname, '..',)
+        const buidDir = path.join(__dirname, '../docker/client');
         await docker.build(buidDir, {
-            tag: 'myt/client',
-            file: path.join(buidDir, 'server', 'Dockerfile.client')
-        }, this.opts.debug);
+            tag: 'myt/client'
+        }, this.cfg.debug);
 
         this.dumpStream = dumpStream;
     }
@@ -83,7 +84,7 @@ module.exports = class Dumper {
     }
 
     async runDump(command, args) {
-        const iniPath = path.join('remotes', this.opts.iniFile);
+        const iniPath = path.join('remotes', this.ctx.iniFile);
         const myArgs = [
             `--defaults-file=${iniPath}`
         ];
@@ -97,8 +98,9 @@ module.exports = class Dumper {
         const commandArgs = [command].concat(myArgs, args);
         await docker.run('myt/client', commandArgs, {
             addHost: 'host.docker.internal:host-gateway',
-            volume: `${this.opts.mytDir}:/workspace`,
+            volume: `${this.ctx.mytDir}:/workspace`,
+            network: 'host',
             rm: true
-        }, execOptions);
+        }, execOptions, this.cfg.debug);
     }
 }

package/lib/exporter-engine.js

@@ -3,10 +3,10 @@ const fs = require('fs-extra');
 const Exporter = require('./exporter');
 
 module.exports = class ExporterEngine {
-    constructor(conn, opts) {
+    constructor(conn, myt) {
         this.conn = conn;
-        this.opts = opts;
-        this.pullFile = `${opts.mytDir}/.pullinfo.json`;
+        this.myt = myt;
+        this.pullFile = `${myt.ctx.mytDir}/.pullinfo.json`;
         this.exporters = [];
         this.exporterMap = {};
     }
@@ -36,7 +36,7 @@ module.exports = class ExporterEngine {
         ];
 
         for (const type of types) {
-            const exporter = new Exporter(type, this.opts.replace);
+            const exporter = new Exporter(type, this.myt.cfg.replace);
             await exporter.init();
 
             this.exporters.push(exporter);

package/lib/server.js

@@ -17,13 +17,7 @@ module.exports = class Server {
             let elapsedTime = 0;
             let maxInterval = 4 * 60 * 1000;
 
-            const dbConfig = this.dbConfig;
-            let myConf = {
-                user: dbConfig.user,
-                password: dbConfig.password,
-                host: dbConfig.host,
-                port: dbConfig.port
-            };
+            const dbConfig = Object.assign({}, this.dbConfig);
 
             async function checker() {
                 elapsedTime += interval;
@@ -40,7 +34,7 @@ module.exports = class Server {
                 if (status === 'exited')
                     return reject(new Error('Docker exited, please see the docker logs for more info'));
 
-                const conn = mysql.createConnection(myConf);
+                const conn = mysql.createConnection(dbConfig);
                 conn.on('error', () => {});
                 conn.connect(err => {
                     conn.destroy();