percona_migrator 0.1.0.rc.5 → 0.1.0.rc.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e7301e478b66a555e4abe723cd9429914fba7820
-  data.tar.gz: fbf40c010ac9272f913b1e79285cb5f61fb11839
+  metadata.gz: 7c92cf9bc3f2fd48fbe1277d9196037bb9d92fec
+  data.tar.gz: 8df8aceabfb8822fe30c3c1cae0d9055b58e2834
 SHA512:
-  metadata.gz: 39543206254ddc6f383446adca091ead90f3ac4a879dad0f7b246b8443d3bbeadc5eb95500fa26040fb8c28ef6182ad94e0620a1a4e8dd593f885f057555ed41
-  data.tar.gz: 011386f3f3e8866fb9db4c19c98f9e050c4899168bb5c0eb9d78ddd56451ad5eeb89caaa3973f7574f21038b89dbcda4f511353103c760f577007043e8f82af0
+  metadata.gz: a1ee2cb57ce1d7062f34d6a3c604081f2c7c7863c76e2acccf7cdadb010239e4eb94350e0c28a17eb44a90f61c7c940239682dbec5e5d2bddcd18239386b1259
+  data.tar.gz: e9fb460fb9eaaecee4ef5a6e5d424a23da11b9717522fd4e4914fa01d27230c5c813dac1160c460f161414077e41f64f4348a94619f0c7e43f8071abc6a57100

data/CHANGELOG.md

@@ -6,6 +6,18 @@ Please follow the format in [Keep a Changelog](http://keepachangelog.com/)
 
 ## [Unreleased]
 
+## [0.1.0.rc.6] - 2016-04-07
+
+### Added
+
+- Support non-ddl migrations by implementing the methods for the ActiveRecord
+    quering to work.
+
+### Changed
+
+- Refactor the PerconaAdapter to use the Runner as connection client, as all the
+    other adapters.
+
 ## [0.1.0.rc.5] - 2016-03-29
 
 ### Changed
@@ -22,7 +34,7 @@ Please follow the format in [Keep a Changelog](http://keepachangelog.com/)
 - Support for foreing keys in db/schema.rb when using [Foreigner
 gem](https://github.com/matthuhiggins/foreigner) in Rails 3 apps. This allows to
 define foreign keys with #execute, but does not provide support for
-#add_foreign_key yet.
+add_foreign_key yet.
 
 ## [0.1.0.rc.3] - 2016-03-10
 

data/README.md

@@ -1,26 +1,36 @@
-# PerconaMigrator [![Build Status](https://travis-ci.org/redbooth/percona_migrator.svg?branch=master)](https://travis-ci.org/redbooth/percona_migrator) [![Code Climate](https://codeclimate.com/github/redbooth/percona_migrator/badges/gpa.svg)](https://codeclimate.com/github/redbooth/percona_migrator)
+# Percona Migrator [![Build Status](https://travis-ci.org/redbooth/percona_migrator.svg?branch=master)](https://travis-ci.org/redbooth/percona_migrator) [![Code Climate](https://codeclimate.com/github/redbooth/percona_migrator/badges/gpa.svg)](https://codeclimate.com/github/redbooth/percona_migrator)
 
-Percona Migrator is a tool for running online and non-blocking
-DDL `ActiveRecord::Migrations` using `pt-online-schema-change` command-line tool of
+Percona Migrator is an **ActiveRecord connection adapter** that allows running
+**MySQL online and non-blocking DDL** `ActiveRecord::Migration` without needing
+    to use a different DSL other than Rails' migrations DSL.
+
+It uses `pt-online-schema-change` command-line tool of
 [Percona
 Toolkit](https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html)
-which supports foreign key constraints.
-
-It adds a `db:percona_migrate:up` runs your migration using the
-`pt-online-schema-change` command.  It will apply exactly the same changes as
-if you run it with `db:migrate:up` avoiding deadlocks and without the need to
-change how you write regular rails migrations.
-
-It also disables `rake db:migrate:up` for the ddl migrations on envs with
-PERCONA_TOOLKIT var set to ensure all these migrations use Percona in production.
+which runs MySQL alter table statements without downtime.
 
 ## Installation
 
-Percona Migrator relies on `pt-online-schema-change` from  [Percona
+Percona Migrator relies on `pt-online-schema-change` from [Percona
 Toolkit](https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html)
 
-For mac, you can install it with homebrew typing `brew install percona-toolkit`. For
-linux machines check out the [Percona Toolkit download
+### Mac
+
+`brew install percona-toolkit`
+
+### Linux
+
+#### Ubuntu/Debian based
+
+`apt-get install percona-toolkit`
+
+#### Arch Linux
+
+`pacman -S percona-toolkit perl-dbd-mysql`
+
+#### Other distros
+
+For other Linux distributions check out the [Percona Toolkit download
 page](https://www.percona.com/downloads/percona-toolkit/) to find the package
 that fits your distribution.
 
@@ -42,20 +52,46 @@ Or install it yourself as:
 
 ## Usage
 
-Percona Migrator is meant to be used only on production or production-like
-environments. To that end, it will only run if the `PERCONA_TOOLKIT`
-environment variable is present.
+Once you added it to your app's Gemfile, you can create and run Rails migrations
+as usual.
+
+All the `ALTER TABLE` statements will be executed with
+`pt-online-schema-change`, which will provide additional output to the
+migration.
+
+### LHM support
+
+If you moved to Soundcloud's [Lhm](https://github.com/soundcloud/lhm) already,
+we got you covered. Percona Migrator overrides Lhm's DSL so that all the alter
+statements also go through `pt-online-schema-change` as well.
+
+You can keep your Lhm migrations and start using Rails migration's DSL back
+again in your next migration.
+
+## How it works
+
+When booting your Rails app, Percona Migrator extends the
+`ActiveRecord::Migration#migrate` method to reset the connection and reestablish
+it using the `PerconaAdapter` instead of the one you defined in your
+`config/database.yml`.
 
-From that same environment where you added the variable, execute the following:
+Then, when any migration DSL methods such as `add_column` or `create_table` are
+executed, they all go to the
+[PerconaAdapter](https://github.com/redbooth/percona_migrator/blob/master/lib/active_record/connection_adapters/percona_adapter.rb).
+There, the methods that require `ALTER TABLE` SQL statements, like `add_column`,
+are overriden to get executed with
+[PerconaMigrator::Runner](https://github.com/redbooth/percona_migrator/blob/master/lib/percona_migrator/runner.rb),
+which deals with the `pt-online-schema-change` binary. All the others, like
+`create_table`, are delegated to the ActiveRecord's built in Mysql2Adapter and
+so they follow the regular path.
 
-1. `bundle exec rake db:migrate:status` to find out your migration's version
-number
-2. `rake db:percona_migrate:up VERSION=<version>`.  This will run the migration
-and mark it as up. Otherwise, if the migration fails, it will still be listed as down
+[PerconaMigrator::Runner](https://github.com/redbooth/percona_migrator/blob/master/lib/percona_migrator/runner.rb)
+spawns a new process that runs the `pt-online-schema-change` binary present in
+the system, with the apropriate arguments for the generated SQL.
 
-You can also mark the migration as run manually, by executing `bundle exec rake
-db:migrate:mark_as_up VERSION=<version>`. Likewise, there's a `bundle exec rake
-db:migrate:mark_as_down VERSION=<version>` that may be of help.
+When an any error occurs, an `ActiveRecord::StatementInvalid` exception is
+raised and the migration is aborted, as all other ActiveRecord connection
+adapters.
 
 ## Development
 

data/lib/active_record/connection_adapters/percona_adapter.rb

@@ -9,19 +9,19 @@ module ActiveRecord
     # Establishes a connection to the database that's used by all Active
     # Record objects.
     def self.percona_connection(config)
-      connection = mysql2_connection(config)
-      client = connection.raw_connection
+      mysql2_connection = mysql2_connection(config)
 
-      config.merge!(
-        logger: logger,
-        runner: PerconaMigrator::Runner.new(logger),
-        cli_generator: PerconaMigrator::CliGenerator.new(config)
+      cli_generator = PerconaMigrator::CliGenerator.new(config)
+      runner = PerconaMigrator::Runner.new(
+        logger,
+        cli_generator,
+        mysql2_connection
       )
 
-      connection_options = { mysql_adapter: connection }
+      connection_options = { mysql_adapter: mysql2_connection }
 
       ConnectionAdapters::PerconaMigratorAdapter.new(
-        client,
+        runner,
         logger,
         connection_options,
         config
@@ -42,58 +42,48 @@ module ActiveRecord
 
       ADAPTER_NAME = 'Percona'.freeze
 
-      def_delegators :mysql_adapter, :tables, :select_values, :exec_delete,
-        :exec_insert, :exec_query, :last_inserted_id, :select, :create_table,
-        :drop_table
+      def_delegators :mysql_adapter, :last_inserted_id, :each_hash
 
-      def initialize(connection, logger, connection_options, config)
+      def initialize(connection, _logger, connection_options, _config)
         super
+        @visitor = BindSubstitution.new(self)
         @mysql_adapter = connection_options[:mysql_adapter]
-        @logger = logger
-        @runner = config[:runner]
-        @cli_generator = config[:cli_generator]
       end
 
-      # Returns true, as this adapter supports migrations
-      def supports_migrations?
-        true
+      def exec_delete(sql, name, binds)
+        execute(to_sql(sql, binds), name)
+        @connection.affected_rows
       end
+      alias :exec_update :exec_delete
 
-      # Delegates #each_hash to the mysql adapter
-      #
-      # @param result [Mysql2::Result]
-      def each_hash(result)
-        if block_given?
-          mysql_adapter.each_hash(result, &Proc.new)
-        else
-          mysql_adapter.each_hash(result)
-        end
+      def exec_insert(sql, name, binds)
+        execute(to_sql(sql, binds), name)
       end
 
-      def new_column(field, default, type, null, collation)
-        Column.new(field, default, type, null, collation)
+      def exec_query(sql, name = 'SQL', _binds = [])
+        result = execute(sql, name)
+        ActiveRecord::Result.new(result.fields, result.to_a)
       end
 
-      # Adds a new column to the named table
-      #
-      # @param table_name [String, Symbol]
-      # @param column_name [String, Symbol]
-      # @param type [Symbol]
-      # @param options [Hash] optional
-      def add_column(table_name, column_name, type, options = {})
-        super
-        command = cli_generator.generate(table_name, @sql)
-        log(@sql, nil) { runner.execute(command) }
+      # Executes a SELECT query and returns an array of rows. Each row is an
+      # array of field values.
+      def select_rows(sql, name = nil)
+        execute(sql, name).to_a
       end
 
-      # Removes the column(s) from the table definition
-      #
-      # @param table_name [String, Symbol]
-      # @param column_names [String, Symbol, Array<String>, Array<Symbol>]
-      def remove_column(table_name, *column_names)
-        super
-        command = cli_generator.generate(table_name, @sql)
-        log(@sql, nil) { runner.execute(command) }
+      # Executes a SELECT query and returns an array of record hashes with the
+      # column names as keys and column values as values.
+      def select(sql, name = nil, binds = [])
+        exec_query(sql, name, binds).to_a
+      end
+
+      # Returns true, as this adapter supports migrations
+      def supports_migrations?
+        true
+      end
+
+      def new_column(field, default, type, null, collation)
+        Column.new(field, default, type, null, collation)
       end
 
       # Adds a new index to the table
@@ -103,10 +93,7 @@ module ActiveRecord
       # @param options [Hash] optional
       def add_index(table_name, column_name, options = {})
         index_name, index_type, index_columns, index_options = add_index_options(table_name, column_name, options)
-        execute "ADD #{index_type} INDEX #{quote_column_name(index_name)} (#{index_columns})#{index_options}"
-
-        command = cli_generator.generate(table_name, @sql)
-        log(@sql, nil) { runner.execute(command) }
+        execute "ALTER TABLE #{quote_table_name(table_name)} ADD #{index_type} INDEX #{quote_column_name(index_name)} (#{index_columns})#{index_options}"
       end
 
       # Remove the given index from the table.
@@ -115,59 +102,17 @@ module ActiveRecord
       # @param options [Hash] optional
       def remove_index(table_name, options = {})
         index_name = index_name_for_remove(table_name, options)
-        execute "DROP INDEX #{quote_column_name(index_name)}"
-
-        command = cli_generator.generate(table_name, @sql)
-        log(@sql, nil) { runner.execute(command) }
-      end
-
-      # Records the SQL statement to be executed. This is used to then delegate
-      # the execution to Percona's pt-online-schema-change.
-      #
-      # @param sql [String]
-      # @param _name [String] optional
-      def execute(sql, _name = nil)
-        @sql = sql
-        true
+        execute "ALTER TABLE #{quote_table_name(table_name)} DROP INDEX #{quote_column_name(index_name)}"
       end
 
-      # Executes the passed statement through pt-online-schema-change if it's
-      # an alter statement, or through the mysql adapter otherwise
-      #
-      # @param sql [String]
-      # @param name [String]
-      def percona_execute(sql, name)
-        if alter_statement?(sql)
-          command = cli_generator.parse_statement(sql)
-          log(sql, nil) { runner.execute(command) }
-        else
-          mysql_adapter.execute(sql, name)
-        end
-      end
-
-      # This abstract method leaves up to the connection adapter freeing the
-      # result, if it needs to. Check out: https://github.com/rails/rails/blob/330c6af05c8b188eb072afa56c07d5fe15767c3c/activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb#L247
-      #
-      # @param sql [String]
-      # @param name [String] optional
-      def execute_and_free(sql, name = nil)
-        yield mysql_adapter.execute(sql, name)
-      end
-
-      def error_number(exception)
+      # Returns the MySQL error number from the exception. The
+      # AbstractMysqlAdapter requires it to be implemented
+      def error_number(_exception)
       end
 
       private
 
-      attr_reader :mysql_adapter, :logger, :runner, :cli_generator
-
-      # Checks whether the sql statement is an ALTER TABLE
-      #
-      # @param sql [String]
-      # @return [Boolean]
-      def alter_statement?(sql)
-        sql =~ /alter table/i
-      end
+      attr_reader :mysql_adapter
     end
   end
 end

data/lib/percona_migrator/alter_argument.rb

@@ -1,30 +1,32 @@
 module PerconaMigrator
+  class InvalidAlterStatement < StandardError; end
 
   # Represents the '--alter' argument of Percona's pt-online-schema-change
   # See https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html
   class AlterArgument
-    ALTER_TABLE_REGEX = /ALTER TABLE `(\w+)` /
+    ALTER_TABLE_REGEX = /\AALTER TABLE `(\w+)` /
+
+    attr_reader :table_name
 
     # Constructor
     #
     # @param statement [String]
+    # @raise [InvalidAlterStatement] if the statement is not an ALTER TABLE
     def initialize(statement)
       @statement = statement
+
+      match = statement.match(ALTER_TABLE_REGEX)
+      raise InvalidAlterStatement unless match
+
+      @table_name = match.captures[0]
     end
 
-    # Returns the '--alter' pt-online-schema-change argumment as a string. See
+    # Returns the '--alter' pt-online-schema-change argument as a string. See
     # https://www.percona.com/doc/percona-toolkit/2.0/pt-online-schema-change.html
     def to_s
       "--alter \"#{parsed_statement}\""
     end
 
-    # Returns the name of the table the alter statement refers to
-    #
-    # @return [String]
-    def table_name
-      statement.match(ALTER_TABLE_REGEX).captures[0]
-    end
-
     private
 
     attr_reader :statement

data/lib/percona_migrator/railtie.rb

@@ -48,18 +48,6 @@ module PerconaMigrator
               connection_config.merge(adapter: 'percona')
             )
           end
-
-          # It executes the passed statement through the PerconaMigratorAdapter
-          # if it's an alter statement. It uses the mysql adapter otherwise.
-          #
-          # This is because +pt-online-schema-change+ is intended for alter
-          # statements only.
-          #
-          # @param sql [String]
-          # @param name [String] optional
-          def execute(sql, name = nil)
-            percona_execute(sql, name)
-          end
         end
       end
     end

data/lib/percona_migrator/runner.rb

@@ -47,11 +47,35 @@ module PerconaMigrator
     # Constructor
     #
     # @param logger [IO]
-    def initialize(logger)
+    def initialize(logger, cli_generator, mysql_adapter)
       @logger = logger
+      @cli_generator = cli_generator
+      @mysql_adapter = mysql_adapter
       @status = nil
     end
 
+    # Executes the passed sql statement using pt-online-schema-change for ALTER
+    # TABLE statements, or the specified mysql adapter otherwise.
+    #
+    # @param sql [String]
+    def query(sql)
+      if alter_statement?(sql)
+        command = cli_generator.parse_statement(sql)
+        execute(command)
+      else
+        mysql_adapter.execute(sql)
+      end
+    end
+
+    # Returns the number of rows affected by the last UPDATE, DELETE or INSERT
+    # statements
+    #
+    # @return [Integer]
+    def affected_rows
+      mysql_adapter.raw_connection.affected_rows
+    end
+
+    # TODO: rename it so we don't confuse it with AR's #execute
     # Runs and logs the given command
     #
     # @param command [String]
@@ -64,7 +88,15 @@ module PerconaMigrator
 
     private
 
-    attr_reader :command, :logger, :status
+    attr_reader :command, :logger, :status, :cli_generator, :mysql_adapter
+
+    # Checks whether the sql statement is an ALTER TABLE
+    #
+    # @param sql [String]
+    # @return [Boolean]
+    def alter_statement?(sql)
+      sql =~ /\Aalter table/i
+    end
 
     # Logs the start and end of the execution
     #
@@ -84,7 +116,9 @@ module PerconaMigrator
 
     # Executes the command outputing any errors
     #
-    # @raise [Errno::ENOENT] if pt-online-schema-change can't be found
+    # @raise [NoStatusError] if the spawned process' status can't be retrieved
+    # @raise [SignalError] if the spawned process receives a signal
+    # @raise [CommandNotFoundError] if pt-online-schema-change can't be found
     def run_command
       message = nil
       Open3.popen3(command) do |_stdin, stdout, stderr, waith_thr|

data/lib/percona_migrator/version.rb

@@ -1,3 +1,3 @@
 module PerconaMigrator
-  VERSION = '0.1.0.rc.5'.freeze
+  VERSION = '0.1.0.rc.6'.freeze
 end

data/percona_migrator.gemspec

@@ -7,8 +7,8 @@ require 'percona_migrator/version'
 Gem::Specification.new do |spec|
   spec.name          = 'percona_migrator'
   spec.version       = PerconaMigrator::VERSION
-  spec.authors       = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas']
-  spec.email         = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com']
+  spec.authors       = ['Ilya Zayats', 'Pau Pérez', 'Fran Casas', 'Jorge Morante']
+  spec.email         = ['ilya.zayats@redbooth.com', 'pau.perez@redbooth.com', 'fran.casas@redbooth.com', 'jorge.morante@redbooth.com']
 
   spec.summary       = %q{pt-online-schema-change runner for ActiveRecord migrations}
   spec.description   = %q{Execute your ActiveRecord migrations with Percona's pt-online-schema-change}

metadata

@@ -1,16 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: percona_migrator
 version: !ruby/object:Gem::Version
-  version: 0.1.0.rc.5
+  version: 0.1.0.rc.6
 platform: ruby
 authors:
 - Ilya Zayats
 - Pau Pérez
 - Fran Casas
+- Jorge Morante
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-03-29 00:00:00.000000000 Z
+date: 2016-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -127,6 +128,7 @@ email:
 - ilya.zayats@redbooth.com
 - pau.perez@redbooth.com
 - fran.casas@redbooth.com
+- jorge.morante@redbooth.com
 executables: []
 extensions: []
 extra_rdoc_files: []