large-hadron-migrator 0.1.2 → 0.1.3

data/CHANGES.md

@@ -0,0 +1,9 @@
+# 0.1.3
+ * code cleanup
+ * Merged [Pullrequest #8](https://github.com/soundcloud/large-hadron-migrator/pull/8)
+ * Merged [Pullrequest #7](https://github.com/soundcloud/large-hadron-migrator/pull/7)
+ * Merged [Pullrequest #4](https://github.com/soundcloud/large-hadron-migrator/pull/4)
+ * Merged [Pullrequest #1](https://github.com/soundcloud/large-hadron-migrator/pull/1)
+
+# 0.1.2
+ * Initial Release

data/{README.markdown → README.md}

@@ -5,7 +5,7 @@ an agile manner. Most Rails projects start like this, and at first, making
 changes is fast and easy.
 
 That is until your tables grow to millions of records. At this point, the
-locking nature of `ALTER TABLE` may take your site down for an hour our more
+locking nature of `ALTER TABLE` may take your site down for an hour or more
 while critical tables are migrated. In order to avoid this, developers begin
 to design around the problem by introducing join tables or moving the data
 into another layer. Development gets less and less agile as tables grow and
@@ -89,7 +89,7 @@ there can only ever be one version of the record in the journal table.
 
 If the journalling trigger hits an already persisted record, it will be
 replaced with the latest data and action. `ON DUPLICATE KEY` comes in handy
-here. This insures that all journal records will be consistent with the
+here. This ensures that all journal records will be consistent with the
 original table.
 
 ### Perform alter statement on new table
@@ -100,20 +100,20 @@ indexes at the end of the copying process.
 
 ### Copy in chunks up to max primary key value to new table
 
-Currently InnoDB aquires a read lock on the source rows in `INSERT INTO...
+Currently InnoDB acquires a read lock on the source rows in `INSERT INTO...
 SELECT`. LHM reads 35K ranges and pauses for a specified number of milliseconds
 so that contention can be minimized.
 
 ### Switch new and original table names and remove triggers
 
-The orignal and copy table are now atomically switched with `RENAME TABLE
+The original and copy table are now atomically switched with `RENAME TABLE
 original TO archive_original, copy_table TO original`. The triggers are removed
 so that journalling stops and all mutations and reads now go against the
 original table.
 
 ### Replay journal: insert, update, deletes
 
-Because the chunked copy stops at the intial maximum id, we can simply replay
+Because the chunked copy stops at the initial maximum id, we can simply replay
 all inserts in the journal table without worrying about collisions.
 
 Updates and deletes are then replayed.
@@ -131,8 +131,8 @@ pass, so this will be quite short compared to the copy phase. The
 inconsistency during replay is similar in effect to a slave which is slightly
 behind master.
 
-There is also caveat with the current journalling scheme; stale journal
-'update' entries are still replayed. Imagine an update to the a record in the
+There is also a caveat with the current journalling scheme; stale journal
+'update' entries are still replayed. Imagine an update to a record in the
 migrated table while the journal is replaying. The journal may already contain
 an update for this record, which becomes stale now. When it is replayed, the
 second change will be lost. So if a record is updated twice, once before and
@@ -162,9 +162,9 @@ Several hours into the migration, a critical fix had to be deployed to the
 site. We rolled out the fix and restarted the app servers in mid migration.
 This was not a good idea.
 
-TL;DR: Never restart during migrations when removing columns with large
-hadron. You can restart while adding migrations as long as active record reads
-column definitions from the slave.
+TL;DR: Never restart during migrations when removing columns with LHM.
+You can restart while adding migrations as long as active record reads column 
+definitions from the slave.
 
 The information below is only relevant if you want to restart your app servers
 while migrating in a master slave setup.

data/VERSION

@@ -1 +1 @@
-0.1.2
+0.1.3

data/lib/large_hadron_migration.rb

@@ -100,16 +100,17 @@ class LargeHadronMigration < ActiveRecord::Migration
 
     raise "chunk_size must be >= 1" unless chunk_size >= 1
 
-    new_table      = "new_#{curr_table}"
-    old_table      = "%s_#{curr_table}" % Time.now.strftime("%Y_%m_%d_%H_%M_%S_%3N")
-    journal_table  = "#{old_table}_changes"
+    started = Time.now.strftime("%Y_%m_%d_%H_%M_%S_%3N")
+    new_table      = "lhmn_%s" % curr_table
+    old_table      = "lhmo_%s_%s" % [started, curr_table]
+    journal_table  = "lhmj_%s_%s" % [started, curr_table]
 
     last_insert_id = last_insert_id(curr_table)
     say "last inserted id in #{curr_table}: #{last_insert_id}"
 
     begin
       # clean tables. old tables are never deleted to guard against rollbacks.
-      execute %Q/drop table if exists %s/ % new_table
+      execute %Q{drop table if exists %s} % new_table
 
       clone_table(curr_table, new_table, id_window)
       clone_table_for_changes(curr_table, journal_table)
@@ -253,8 +254,8 @@ class LargeHadronMigration < ActiveRecord::Migration
     end
   end
 
-  def self.clone_table(source, dest, window = 0)
-    execute schema_sql(source, dest, window)
+  def self.clone_table(source, dest, window = 0, add_action_column = false)
+    execute schema_sql(source, dest, window, add_action_column)
   end
 
   def self.common_columns(t1, t2)
@@ -262,11 +263,7 @@ class LargeHadronMigration < ActiveRecord::Migration
   end
 
   def self.clone_table_for_changes(table, journal_table)
-    clone_table(table, journal_table)
-    execute %Q{
-      alter table %s
-      add column hadron_action varchar(15);
-    } % journal_table
+    clone_table(table, journal_table, 0, true)
   end
 
   def self.rename_tables(tables = {})
@@ -333,11 +330,15 @@ class LargeHadronMigration < ActiveRecord::Migration
   end
 
   def self.replay_delete_changes(table, journal_table)
-    execute %Q{
-      delete from #{table} where id in (
-        select id from #{journal_table} where hadron_action = 'delete'
-      )
-    }
+    with_master do
+      if connection.select_values("select id from #{journal_table} where hadron_action = 'delete' LIMIT 1").any?
+        execute %Q{
+          delete from #{table} where id in (
+            select id from #{journal_table} where hadron_action = 'delete'
+          )
+        }
+      end
+    end
   end
 
   def self.replay_update_changes(table, journal_table, chunk_size = 10000, wait = 0.2)
@@ -359,12 +360,17 @@ class LargeHadronMigration < ActiveRecord::Migration
   #  behavior with the latter where the auto_increment of the source table
   #  got modified when updating the destination.
   #
-  def self.schema_sql(source, dest, window)
+  def self.schema_sql(source, dest, window, add_action_column = false)
     show_create(source).tap do |schema|
       schema.gsub!(/auto_increment=(\d+)/i) do
         "auto_increment=#{  $1.to_i + window }"
       end
 
+      if add_action_column
+        schema.sub!(/\) ENGINE=/,
+          ", hadron_action ENUM('update', 'insert', 'delete'), INDEX hadron_action (hadron_action) USING BTREE) ENGINE=")
+      end
+
       schema.gsub!('CREATE TABLE `%s`' % source, 'CREATE TABLE `%s`' % dest)
     end
   end

data/spec/large_hadron_migration_spec.rb

@@ -99,7 +99,8 @@ describe "LargeHadronMigration", "triggers" do
   end
 
   it "should create a table for triggered changes" do
-    truthiness_column "triggerme_changes", "hadron_action", "varchar"
+    truthiness_column "triggerme_changes", "hadron_action", "enum"
+    truthiness_index "triggerme_changes", "hadron_action", [ "hadron_action" ], false
   end
 
   it "should trigger on insert" do
@@ -341,6 +342,11 @@ describe "LargeHadronMigration", "replaying changes" do
     end
   end
 
+  it "doesn't replay delete if there are any" do
+    LargeHadronMigration.should_receive(:execute).never
+    LargeHadronMigration.replay_delete_changes("source", "source_changes")
+  end
+
 end
 
 describe "LargeHadronMigration", "units" do

data/spec/spec_helper.rb

@@ -90,6 +90,21 @@ module SpecHelper
 
   end
 
+  def truthiness_index(table, expected_index_name, indexed_columns, unique)
+    index = sql("SHOW INDEXES FROM #{table}").all_hashes.inject({}) do |a, part|
+      index_name = part['Key_name']
+      a[index_name] ||= { 'unique' => '0' == part['Non_unique'], 'columns' => [] }
+      column_index = part['Seq_in_index'].to_i - 1
+      a[index_name]['columns'][column_index] = part['Column_name']
+      a
+    end[expected_index_name]
+
+    flunk("no index named #{expected_index_name} found on #{table}") unless index
+
+    index['columns'].should == indexed_columns
+    index['unique'].should == unique
+  end
+
 end
 
 # Mock Rails Environment

metadata

@@ -1,12 +1,8 @@
 --- !ruby/object:Gem::Specification 
 name: large-hadron-migrator
 version: !ruby/object:Gem::Version 
-  prerelease: false
-  segments: 
-  - 0
-  - 1
-  - 2
-  version: 0.1.2
+  prerelease: 
+  version: 0.1.3
 platform: ruby
 authors: 
 - SoundCloud
@@ -16,8 +12,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-05-04 00:00:00 +02:00
-default_executable: 
+date: 2011-09-28 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: activerecord
@@ -27,10 +22,6 @@ dependencies:
     requirements: 
     - - ~>
       - !ruby/object:Gem::Version 
-        segments: 
-        - 2
-        - 3
-        - 8
         version: 2.3.8
   type: :runtime
   version_requirements: *id001
@@ -42,10 +33,6 @@ dependencies:
     requirements: 
     - - ~>
       - !ruby/object:Gem::Version 
-        segments: 
-        - 2
-        - 3
-        - 8
         version: 2.3.8
   type: :runtime
   version_requirements: *id002
@@ -57,10 +44,6 @@ dependencies:
     requirements: 
     - - "="
       - !ruby/object:Gem::Version 
-        segments: 
-        - 2
-        - 8
-        - 1
         version: 2.8.1
   type: :runtime
   version_requirements: *id003
@@ -72,10 +55,6 @@ dependencies:
     requirements: 
     - - "="
       - !ruby/object:Gem::Version 
-        segments: 
-        - 1
-        - 3
-        - 1
         version: 1.3.1
   type: :development
   version_requirements: *id004
@@ -89,11 +68,11 @@ extra_rdoc_files: []
 
 files: 
 - .gitignore
-- CHANGES.markdown
+- CHANGES.md
 - Gemfile
 - Gemfile.lock
 - LICENSE
-- README.markdown
+- README.md
 - Rakefile
 - VERSION
 - large-hadron-migrator.gemspec
@@ -101,7 +80,6 @@ files:
 - spec/large_hadron_migration_spec.rb
 - spec/migrate/add_new_column.rb
 - spec/spec_helper.rb
-has_rdoc: true
 homepage: http://github.com/soundcloud/large-hadron-migrator
 licenses: []
 
@@ -115,21 +93,17 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      segments: 
-      - 0
       version: "0"
 required_rubygems_version: !ruby/object:Gem::Requirement 
   none: false
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      segments: 
-      - 0
       version: "0"
 requirements: []
 
 rubyforge_project: 
-rubygems_version: 1.3.7
+rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: online schema changer for mysql