mysql_online_migrations 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8b8c18f26ec8b8f57c2d469d2e11e3837de24130
-  data.tar.gz: 3a65090310d0a049c85ad22da8099ea46939a913
+  metadata.gz: 1577a541999a6a5ba4675641f7ce752819a79253
+  data.tar.gz: fa2c478e55d72bc8d78e386a36f493fe98d58f30
 SHA512:
-  metadata.gz: 3bb7753d1b88a3a77cddf0f23a2a2b6cfe3717821dc625b0fb5024bd4cce25fc45aecd66a75cbb1e8b16464fb8cdd11da1e6b548e57a2c38adc7ec0dd2e77c6a
-  data.tar.gz: a631129c1c90cddd1528658c37a815749a0954882fdf8ddcf387bdf1f7c555fd63dd5ecf18ca2b60443833a9df33fdd96f98a2bfe6a08f7c3e8691b09d544f76
+  metadata.gz: 24384b66dfa4347dffa0c38469937eea5097ea2b1a90b4dbf64fc2be96f1f6b52ce95204c0035e68150503a422fbfab5ad0bed2244368ec0f8d0e3bf8cecd02e
+  data.tar.gz: 4ec8b0d6cf8558b4a1e1c36d6f7bd46eaabc805d2226acc793593b9722c4d9de94c0bb64b9a51e744eedb2163a872dea2c6e525002c17954fef19baa68326d71

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mysql_online_migrations (0.1.2)
+    mysql_online_migrations (0.1.4)
       activerecord (~> 3.2.15)
       activesupport (~> 3.2.15)
       mysql2

data/README.md

@@ -1,21 +1,22 @@
 mysql_online_migrations
 =======================
 
-Patch Rails migrations to support MySQL 5.6 online migrations capabilities.  
-Prior to MySQL 5.6, when adding / removing / renaming indexes and columns, MySQL would lock the writes of the whole table.  
-MySQL 5.6 adds a way to append `LOCK=NONE` to those alter table statements to allow online migrations.
+Patch Rails migrations to enforce MySQL 5.6 online migrations.
+Prior to MySQL 5.6, when adding / removing / renaming indexes and columns, MySQL would lock the writes of the whole table.
+MySQL 5.6 by default will try to apply the least locking possible. You however don't know what kind of locking it applies and there's situations where it can't allow writes during a migration (See Caveats).
+This gem enforces `LOCK=NONE` in all migration statements of Rails. Therefore, you're getting an error when MySQL cannot write during the migration so there's no surprise when rolling out in production.
 
 
 Requirements
 =======================
-Built for Rails 3.2.15, may be compatible with Rails 4.0 but you'd lose the new features introduced.  
+Built for Rails 3.2.15, may be compatible with Rails 4.0 but you'd lose the new features introduced.
 This gem actually just requires ActiveRecord and ActiveSupport, the full Rails gem should not be required.
 
 List of requirements :
 
 - Use mysql2 adapter
 - Use ActiveRecord "~> 3.2.15"
-- Use MySQL or Percona Server 5.6.X
+- Use MySQL or Percona Server 5.6.X with InnoDB
 
 Scope of this gem
 =======================
@@ -25,36 +26,39 @@ Patch Rails migrations to automatically add `LOCK=NONE` in the following context
 - Index management : `add_index`, `remove_index`, `rename_index`
 - Add column : `add_column`, `add_timestamps`
 - Remove column : `remove_column`, `remove_timestamps`
+- Change column : `change_column`, `change_column_null`, `change_column_default`
 
 Usage
 =======================
-In a typical Rails app, just add it to your gem file :  
+In a typical Rails app, just add it to your Gemfile :
+
 `gem 'mysql_online_migrations'`
 
-And you're ready for online migrations!
+Then run `bundle install`
+
+You're ready for online migrations! Please read the caveats section though.
 
 ### Turn it off for a whole environment
-Example for environment test (your CI might not use MySQL 5.6 yet), add the following to `config/environments/test.rb`:  
+Example for environment test (your CI might not use MySQL 5.6 yet), add the following to `config/environments/test.rb`:
 `config.active_record.mysql_online_migrations = false`
 
 ### Turn it off for a specific statement
-Add `lock: true` to any of the method calls mentioned above. Example :  
+Add `lock: true` to any of the method calls mentioned above. Example :
 `add_index :users, :name, lock: true`
 
-The `lock: none` will be useful when hitting the caveats of `LOCK=NONE`. Please read the following section.
+The `lock: true` will be useful when hitting the caveats of `LOCK=NONE`. Please read the following section.
 
 Caveats
 =======================
 
-Here's a list of things you can't do with LOCK=NONE and therefore you should provide `lock: true`:  
+The MySQL manual contains a list of which DDL statements can be run with `LOCK=NONE` under [Table 14.5 Summary of Online Status for DDL Operations](http://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html).  The short version is that __you can not yet__:
 
 - Index a column of type text
 - Change the type of a column
 - Change the length of a column
-- Change the nullable value of a column
-- When adding an AUTO_INCREMENT column,
-- Other stuff found on https://blogs.oracle.com/mysqlinnodb/entry/online_alter_table_in_mysql :
-  - when the table contains FULLTEXT indexes or a hidden FTS_DOC_ID column, or
-  - when there are FOREIGN KEY constraints referring to the table, with ON…CASCADE or ON…SET NULL option.
+- Set a column to NOT NULL (at least not with the default SQL_MODE)
+- Adding an AUTO_INCREMENT column,
 
-If you don't use `lock: true` when it's not supported, you'll get a MySQL exception. No risk to lock the table by accident.
+If you don't use `lock: true` when online migration is not supported, you'll get a MySQL exception. No risk to lock the table by accident.
+It's therefore highly recommended to use it in development/test/staging environment before running migrations in production.
+If you have to perform such a migration without locking the table, tools such as [pt-online-schema-change](http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html) and [LHM](https://github.com/soundcloud/lhm) are viable options

data/lib/mysql_online_migrations/columns.rb

@@ -15,6 +15,21 @@ module MysqlOnlineMigrations
       execute("ALTER TABLE #{quote_table_name(table_name)} #{change_column_sql(table_name, column_name, type, options)} #{lock_statement(lock, true)}")
     end
 
+    def change_column_default(table_name, column_name, default, options = {})
+      column = column_for(table_name, column_name)
+      change_column table_name, column_name, column.sql_type, options.merge(:default => default)
+    end
+
+    def change_column_null(table_name, column_name, null, default = nil, options = {})
+      column = column_for(table_name, column_name)
+
+      unless null || default.nil?
+        execute("UPDATE #{quote_table_name(table_name)} SET #{quote_column_name(column_name)}=#{quote(default)} WHERE #{quote_column_name(column_name)} IS NULL")
+      end
+
+      change_column table_name, column_name, column.sql_type, options.merge(:null => null)
+    end
+
     def rename_column(table_name, column_name, new_column_name, options = {})
       lock, options = extract_lock_from_options(options)
       execute("ALTER TABLE #{quote_table_name(table_name)} #{rename_column_sql(table_name, column_name, new_column_name)} #{lock_statement(lock, true)}")

data/mysql_online_migrations.gemspec

@@ -1,8 +1,8 @@
 Gem::Specification.new do |s|
   s.name        = 'mysql_online_migrations'
-  s.version     = '0.1.3'
-  s.summary     = "Use MySQL 5.6+ capacities to perform online migrations"
-  s.description = "MySQL 5.6 adds a way to append `LOCK=NONE` to alter table statements to allow online migrations. Let's use it."
+  s.version     = '0.1.4'
+  s.summary     = "Use MySQL 5.6+ capacities to enforce online migrations"
+  s.description = "MySQL 5.6 adds a `LOCK=NONE` option to make sure migrations are done with no locking. Let's use it."
   s.authors     = ["Anthony Alberto"]
   s.email       = 'alberto.anthony@gmail.com'
   s.homepage    = 'https://github.com/anthonyalberto/mysql_online_migrations'
@@ -18,4 +18,5 @@ Gem::Specification.new do |s|
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
-end
+  s.license = 'MIT'
+end

data/spec/lib/mysql_online_migrations/columns_spec.rb

@@ -2,6 +2,7 @@ require "spec_helper"
 
 describe MysqlOnlineMigrations::Columns do
   let(:comma_before_lock_none) { true }
+  let(:queries_with_lock) { {} }
 
   context "#add_column" do
     let(:queries) do
@@ -81,10 +82,31 @@ describe MysqlOnlineMigrations::Columns do
 
   context "#change_column" do
     let(:queries) do
-      # Unsupported with lock=none : change column type, change limit, change null
+      # Unsupported with lock=none : change column type, change limit, set NOT NULL.
       {
         [:testing, :foo, :string, { default: 'def', limit: 100 }] =>
           "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(100) DEFAULT 'def'",
+        [:testing, :foo, :string, { null: true, limit: 100 }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(100) DEFAULT NULL",
+      }
+    end
+
+    let(:queries_with_lock) do
+      {
+        [:testing, :foo, :string, { limit: 200 }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(200) DEFAULT NULL",
+        [:testing, :foo, :string, { default: 'def' }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(255) DEFAULT 'def'",
+        [:testing, :foo, :string, { null: false }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(255) NOT NULL",
+        [:testing, :foo, :string, { null: false, default: 'def', limit: 200 }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(200) DEFAULT 'def' NOT NULL",
+        [:testing, :foo, :string, { null: true }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(255) DEFAULT NULL",
+        [:testing, :foo, :integer, { null: true, limit: 6 }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` bigint DEFAULT NULL",
+        [:testing, :foo, :integer, { null: true, limit: 1 }] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` tinyint DEFAULT NULL"
       }
     end
 
@@ -92,6 +114,54 @@ describe MysqlOnlineMigrations::Columns do
 
     it_behaves_like "a method that adds LOCK=NONE when needed"
     it_behaves_like "a request with LOCK=NONE that doesn't crash in MySQL"
+    it_behaves_like "queries_with_lock that executes properly"
+  end
+
+  context "#change_column_default" do
+    let(:queries) do
+      {
+        [:testing, :foo, 'def', {}] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(100) DEFAULT 'def'",
+        [:testing, :foo, nil, {}] =>
+          "ALTER TABLE `testing` CHANGE `foo` `foo` varchar(100) DEFAULT NULL"
+      }
+    end
+
+    let(:method_name) { :change_column_default }
+
+    it_behaves_like "a method that adds LOCK=NONE when needed"
+    it_behaves_like "a request with LOCK=NONE that doesn't crash in MySQL"
+    it_behaves_like "queries_with_lock that executes properly"
+  end
+
+  context "#change_column_null" do
+    let(:queries) do
+      #change_column_null doesn't set DEFAULT in sql. It just issues an update statement before setting the NULL value if setting NULL to false
+      {
+        [:testing, :bam, true, nil, {}] =>
+          "ALTER TABLE `testing` CHANGE `bam` `bam` varchar(100) DEFAULT 'test'",
+        [:testing, :bam, true, 'def', {}] =>
+          "ALTER TABLE `testing` CHANGE `bam` `bam` varchar(100) DEFAULT 'test'"
+      }
+    end
+
+    let(:queries_with_lock) do
+      {
+        [:testing, :bam, false, nil, {}] =>
+          "ALTER TABLE `testing` CHANGE `bam` `bam` varchar(100) DEFAULT 'test' NOT NULL",
+        [:testing, :bam, false, 'def', {}] =>
+          [
+            "UPDATE `testing` SET `bam`='def' WHERE `bam` IS NULL",
+            "ALTER TABLE `testing` CHANGE `bam` `bam` varchar(100) DEFAULT 'test' NOT NULL"
+          ]
+      }
+    end
+
+    let(:method_name) { :change_column_null }
+
+    it_behaves_like "a method that adds LOCK=NONE when needed"
+    it_behaves_like "a request with LOCK=NONE that doesn't crash in MySQL"
+    it_behaves_like "queries_with_lock that executes properly"
   end
 
   context "#rename_column" do

data/spec/lib/mysql_online_migrations/indexes_spec.rb

@@ -2,6 +2,7 @@ require "spec_helper"
 
 describe MysqlOnlineMigrations::Indexes do
   let(:comma_before_lock_none) { false }
+  let(:queries_with_lock) { {} }
 
   context "#add_index" do
     let(:queries) do

data/spec/support/helpers.rb

@@ -1,4 +1,7 @@
 module Helpers
+  CATCH_STATEMENT_REGEX = /^(alter|create|drop|update) /i
+  DDL_STATEMENT_REGEX  = /^(alter|create|drop) /i
+
   def execute(statement)
   end
 
@@ -9,7 +12,7 @@ module Helpers
   def stub_execute
     original_execute = @adapter.method(:execute)
     @adapter.stub(:execute) do |statement|
-      if statement =~ /^(alter|create|drop) /i
+      if statement =~ CATCH_STATEMENT_REGEX
         execute(statement.squeeze(' ').strip)
       else
         original_execute.call(statement)
@@ -18,7 +21,11 @@ module Helpers
   end
 
   def add_lock_none(str, with_comma)
-    "#{str}#{with_comma ? ' ,' : ''} LOCK=NONE"
+    if str =~ DDL_STATEMENT_REGEX
+      "#{str}#{with_comma ? ' ,' : ''} LOCK=NONE"
+    else
+      str
+    end
   end
 
   def rebuild_table
@@ -28,6 +35,7 @@ module Helpers
       t.column :foo, :string, :limit => 100
       t.column :bar, :string, :limit => 100
       t.column :baz, :string, :limit => 100
+      t.column :bam, :string, :limit => 100, default: "test", null: false
       t.column :extra, :string, :limit => 100
       t.timestamps
     end

data/spec/support/shared_examples/mysql_online_migrations.rb

@@ -1,10 +1,11 @@
 shared_examples_for "a method that adds LOCK=NONE when needed" do
   before(:each) do
     stub_execute
+    @queries = queries.merge(queries_with_lock)
   end
 
   it "adds LOCK=NONE at the end of the query" do
-    queries.each do |arguments, output|
+    @queries.each do |arguments, output|
       Array.wrap(output).each { |out| should_receive(:execute).with(add_lock_none(out, comma_before_lock_none)) }
       @adapter.public_send(method_name, *arguments)
     end
@@ -14,7 +15,7 @@ shared_examples_for "a method that adds LOCK=NONE when needed" do
     before(:each) { set_ar_setting(false) }
 
     it "doesn't add lock none" do
-      queries.each do |arguments, output|
+      @queries.each do |arguments, output|
         Array.wrap(output).each { |out| should_receive(:execute).with(out) }
         @adapter.public_send(method_name, *arguments)
       end
@@ -23,7 +24,7 @@ shared_examples_for "a method that adds LOCK=NONE when needed" do
 
   context "with lock: true option" do
     it "doesn't add lock none" do
-      queries.each do |arguments, output|
+      @queries.each do |arguments, output|
         Array.wrap(output).each { |out| should_receive(:execute).with(out) }
         arguments[-1] = arguments.last.merge(lock: true)
         @adapter.public_send(method_name, *arguments)
@@ -41,4 +42,15 @@ shared_examples_for "a request with LOCK=NONE that doesn't crash in MySQL" do
       end
     end
   end
+end
+
+shared_examples_for "queries_with_lock that executes properly" do
+  it "succeeds without exception" do
+    queries.each do |_, output|
+      Array.wrap(output).each do |out|
+        @adapter.execute(out)
+        rebuild_table
+      end
+    end
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mysql_online_migrations
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - Anthony Alberto
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-11-24 00:00:00.000000000 Z
+date: 2013-11-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -94,8 +94,8 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-description: MySQL 5.6 adds a way to append `LOCK=NONE` to alter table statements
-  to allow online migrations. Let's use it.
+description: MySQL 5.6 adds a `LOCK=NONE` option to make sure migrations are done
+  with no locking. Let's use it.
 email: alberto.anthony@gmail.com
 executables: []
 extensions: []
@@ -117,7 +117,8 @@ files:
 - spec/support/helpers.rb
 - spec/support/shared_examples/mysql_online_migrations.rb
 homepage: https://github.com/anthonyalberto/mysql_online_migrations
-licenses: []
+licenses:
+- MIT
 metadata: {}
 post_install_message: 
 rdoc_options: []
@@ -138,7 +139,7 @@ rubyforge_project:
 rubygems_version: 2.0.2
 signing_key: 
 specification_version: 4
-summary: Use MySQL 5.6+ capacities to perform online migrations
+summary: Use MySQL 5.6+ capacities to enforce online migrations
 test_files:
 - spec/lib/mysql_online_migrations/columns_spec.rb
 - spec/lib/mysql_online_migrations/indexes_spec.rb