lhm 1.3.0 → 2.0.0

data/.gitignore

@@ -4,3 +4,5 @@ Gemfile.lock
 gemfiles/*.lock
 pkg/*
 .rvmrc
+.ruby-version
+.ruby-gemset

data/.travis.yml

@@ -2,8 +2,11 @@ language: ruby
 before_script:
   - "mysql -e 'create database lhm;'"
 rvm:
-  - 1.8.7
   - 1.9.3
+  - 2.0.0
+matrix:
+  allow_failures:
+    - rvm: 2.0.0
 gemfile:
   - gemfiles/ar-2.3_mysql.gemfile
   - gemfiles/ar-3.2_mysql.gemfile

data/README.md

@@ -1,4 +1,4 @@
-# Large Hadron Migrator [![Build Status](https://secure.travis-ci.org/soundcloud/large-hadron-migrator.png?branch=master)][4]
+# Large Hadron Migrator [![Build Status][5]][4]
 
 Rails style database migrations are a useful way to evolve your data schema in
 an agile manner. Most Rails projects start like this, and at first, making
@@ -45,13 +45,16 @@ the [twitter solution][2], it does not require the presence of an indexed
 Lhm currently only works with MySQL databases and requires an established
 ActiveRecord or DataMapper connection.
 
-It is compatible and [continuously tested][4] with Ruby 1.8.7 and Ruby 1.9.x,
+It is compatible and [continuously tested][4] with MRI 1.9.x,
 ActiveRecord 2.3.x and 3.x (mysql and mysql2 adapters), as well as DataMapper
 1.2 (dm-mysql-adapter).
 
 Lhm also works with dm-master-slave-adapter, it'll bind to the master before
 running the migrations.
 
+The test suite is also run against MRI 2.0.0 in Continuous Integration, but
+there are a few bugs left to fix.
+
 ## Limitations
 
 Lhm requires a monotonically increasing numeric Primary Key on the table, due to how
@@ -143,13 +146,13 @@ to prevent accidental data loss.
 There are two different table rename strategies available: LockedSwitcher and
 AtomicSwitcher.
 
-The LockedSwitcher strategy locks the table being migrated and issues two ALTER TABLE statements. 
+The LockedSwitcher strategy locks the table being migrated and issues two ALTER TABLE statements.
 The AtomicSwitcher uses a single atomic RENAME TABLE query and is the favored solution.
 
-Lhm chooses AtomicSwitcher if no strategy is specified, **unless** your version of MySQL is 
-affected by [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675). If your version is 
-affected, Lhm will raise an error if you don't specify a strategy. You're recommended 
-to use the LockedSwitcher in these cases to avoid replication issues. 
+Lhm chooses AtomicSwitcher if no strategy is specified, **unless** your version of MySQL is
+affected by [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675). If your version is
+affected, Lhm will raise an error if you don't specify a strategy. You're recommended
+to use the LockedSwitcher in these cases to avoid replication issues.
 
 To specify the strategy in your migration:
 
@@ -159,6 +162,41 @@ Lhm.change_table :users, :atomic_switch => true do |m|
 end
 ```
 
+## Limiting the data that is migrated
+
+For instances where you want to limit the data that is migrated to the new
+table by some conditions, you may tell the migration to filter by a set of
+conditions:
+
+```ruby
+Lhm.change_table(:sounds) do |m|
+  m.filter("inner join users on users.`id` = sounds.`user_id` and sounds.`public` = 1")
+end
+```
+
+Note that this SQL will be inserted into the copy directly after the "from"
+statement - so be sure to use inner/outer join syntax and not cross joins. These
+conditions will not affect the triggers, so any modifications to the table
+during the run will happen on the new table as well.
+
+## Cleaning up after an interrupted Lhm run
+
+If an Lhm migration is interrupted, it may leave behind the temporary tables
+used in the migration. If the migration is re-started, the unexpected presence
+of these tables will cause an error. In this case, `Lhm.cleanup` can be used
+to drop any orphaned Lhm temporary tables.
+
+To see what Lhm tables are found:
+
+```ruby
+Lhm.cleanup
+```
+
+To remove any Lhm tables found:
+```ruby
+Lhm.cleanup(true)
+```
+
 ## Contributing
 
 We'll check out your contribution if you:
@@ -184,4 +222,5 @@ The license is included as LICENSE in this directory.
 [1]: http://www.facebook.com/note.php?note\_id=430801045932
 [2]: https://github.com/freels/table_migrator
 [3]: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html
-[4]: http://travis-ci.org/soundcloud/large-hadron-migrator
+[4]: https://travis-ci.org/soundcloud/lhm
+[5]: https://travis-ci.org/soundcloud/lhm.png?branch=master

data/gemfiles/ar-2.3_mysql.gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source "https://rubygems.org"
 
 gem "mysql", "~> 2.8.1"
 gem "activerecord", "~> 2.3.14"

data/gemfiles/ar-3.2_mysql.gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source "https://rubygems.org"
 
 gem "mysql", "~> 2.8.1"
 gem "activerecord", "~> 3.2.2"

data/gemfiles/ar-3.2_mysql2.gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source "https://rubygems.org"
 
 gem "mysql2", "~> 0.3.11"
 gem "activerecord", "~> 3.2.2"

data/gemfiles/dm_mysql.gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source "https://rubygems.org"
 
 gem 'dm-core'
 gem 'dm-mysql-adapter'

data/lhm.gemspec

@@ -13,7 +13,7 @@ Gem::Specification.new do |s|
   s.email         = %q{rany@soundcloud.com, tobi@soundcloud.com, ts@soundcloud.com}
   s.summary       = %q{online schema changer for mysql}
   s.description   = %q{Migrate large tables without downtime by copying to a temporary table in chunks. The old table is not dropped. Instead, it is moved to timestamp_table_name for verification.}
-  s.homepage      = %q{http://github.com/soundcloud/large-hadron-migrator}
+  s.homepage      = %q{http://github.com/soundcloud/lhm}
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   s.require_paths = ["lib"]

data/lib/lhm.rb

@@ -26,6 +26,10 @@ module Lhm
   #   Size of a chunk (defaults to: 40,000)
   # @option options [Fixnum] :throttle
   #   Time to wait between chunks in milliseconds (defaults to: 100)
+  # @option options [Fixnum] :start
+  #   Primary Key position at which to start copying chunks
+  # @option options [Fixnum] :limit
+  #   Primary Key position at which to stop copying chunks
   # @option options [Boolean] :atomic_switch
   #   Use atomic switch to rename tables (defaults to: true)
   #   If using a version of mysql affected by atomic switch bug, LHM forces user
@@ -38,7 +42,6 @@ module Lhm
     invoker = Invoker.new(origin, connection)
     block.call(invoker.migrator)
     invoker.run(options)
-
     true
   end
 
@@ -72,8 +75,8 @@ module Lhm
   end
 
   protected
+
   def self.connection
     Connection.new(adapter)
   end
-
 end

data/lib/lhm/chunker.rb

@@ -43,8 +43,8 @@ module Lhm
 
     def copy(lowest, highest)
       "insert ignore into `#{ destination_name }` (#{ columns }) " +
-      "select #{ columns } from `#{ origin_name }` " +
-      "where `id` between #{ lowest } and #{ highest }"
+      "select #{ select_columns } from `#{ origin_name }` " +
+      "#{ conditions } #{ origin_name }.`id` between #{ lowest } and #{ highest }"
     end
 
     def select_start
@@ -63,6 +63,10 @@ module Lhm
 
   private
 
+    def conditions
+      @migration.conditions ? "#{@migration.conditions} and" : "where"
+    end
+
     def destination_name
       @migration.destination.name
     end
@@ -75,6 +79,10 @@ module Lhm
       @columns ||= @migration.intersection.joined
     end
 
+    def select_columns
+      @select_columns ||= @migration.intersection.typed(origin_name)
+    end
+
     def validate
       if @start && @limit && @start > @limit
         error("impossible chunk options (limit must be greater than start)")

data/lib/lhm/migration.rb

@@ -5,11 +5,12 @@ require 'lhm/intersection'
 
 module Lhm
   class Migration
-    attr_reader :origin, :destination
+    attr_reader :origin, :destination, :conditions
 
-    def initialize(origin, destination, time = Time.now)
+    def initialize(origin, destination, conditions = nil, time = Time.now)
       @origin = origin
       @destination = destination
+      @conditions = conditions
       @start = time
     end
 

data/lib/lhm/migrator.rb

@@ -13,7 +13,7 @@ module Lhm
     include Command
     include SqlHelper
 
-    attr_reader :name, :statements, :connection
+    attr_reader :name, :statements, :connection, :conditions
 
     def initialize(table, connection = nil)
       @connection = connection
@@ -135,10 +135,29 @@ module Lhm
     # @param [String, Symbol] index_name
     #   Optional name of the index to be removed
     def remove_index(columns, index_name = nil)
+      columns = [columns].flatten.map(&:to_sym)
+      from_origin = @origin.indices.find {|name, cols| cols.map(&:to_sym) == columns}
+      index_name ||= from_origin[0] unless from_origin.nil?
       index_name ||= idx_name(@origin.name, columns)
       ddl("drop index `%s` on `%s`" % [index_name, @name])
     end
 
+    # Filter the data that is copied into the new table by the provided SQL.
+    # This SQL will be inserted into the copy directly after the "from"
+    # statement - so be sure to use inner/outer join syntax and not cross joins.
+    #
+    # @example Add a conditions filter to the migration.
+    #   Lhm.change_table(:sounds) do |m|
+    #     m.filter("inner join users on users.`id` = sounds.`user_id` and sounds.`public` = 1")
+    #   end
+    #
+    # @param [ String ] sql The sql filter.
+    #
+    # @return [ String ] The sql filter.
+    def filter(sql)
+      @conditions = sql
+    end
+
   private
 
     def validate
@@ -160,7 +179,7 @@ module Lhm
     def execute
       destination_create
       @connection.sql(@statements)
-      Migration.new(@origin, destination_read)
+      Migration.new(@origin, destination_read, conditions)
     end
 
     def destination_create

data/lib/lhm/version.rb

@@ -2,5 +2,5 @@
 # Schmidt
 
 module Lhm
-  VERSION = "1.3.0"
+  VERSION = "2.0.0"
 end

data/spec/fixtures/permissions.ddl

@@ -0,0 +1,5 @@
+CREATE TABLE `permissions` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `track_id` int(11) DEFAULT NULL,
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8

data/spec/fixtures/tracks.ddl

@@ -0,0 +1,5 @@
+CREATE TABLE `tracks` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `public` int(4) DEFAULT 0,
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8

data/spec/fixtures/users.ddl

@@ -8,5 +8,7 @@ CREATE TABLE `users` (
   `description` text,
   PRIMARY KEY (`id`),
   UNIQUE KEY `index_users_on_reference` (`reference`),
-  KEY `index_users_on_username_and_created_at` (`username`,`created_at`)
+  KEY `index_users_on_username_and_created_at` (`username`,`created_at`),
+  KEY `index_with_a_custom_name` (`username`,`group`)
+
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8

data/spec/integration/lhm_spec.rb

@@ -13,6 +13,46 @@ describe Lhm do
   describe "changes" do
     before(:each) do
       table_create(:users)
+      table_create(:tracks)
+      table_create(:permissions)
+    end
+
+    describe "when providing a subset of data to copy" do
+
+      before do
+        execute("insert into tracks set id = 13, public = 0")
+        11.times { |n| execute("insert into tracks set id = #{n + 1}, public = 1") }
+        11.times { |n| execute("insert into permissions set track_id = #{n + 1}") }
+
+        Lhm.change_table(:permissions, :atomic_switch => false) do |t|
+          t.filter("inner join tracks on tracks.`id` = permissions.`track_id` and tracks.`public` = 1")
+        end
+      end
+
+      describe "when no additional data is inserted into the table" do
+
+        it "migrates the existing data" do
+          slave do
+            count_all(:permissions).must_equal(11)
+          end
+        end
+      end
+
+      describe "when additional data is inserted" do
+
+        before do
+          execute("insert into tracks set id = 14, public = 0")
+          execute("insert into tracks set id = 15, public = 1")
+          execute("insert into permissions set track_id = 14")
+          execute("insert into permissions set track_id = 15")
+        end
+
+        it "migrates all data" do
+          slave do
+            count_all(:permissions).must_equal(13)
+          end
+        end
+      end
     end
 
     it "should add a column" do
@@ -103,11 +143,21 @@ describe Lhm do
 
     it "should remove an index with a custom name" do
       Lhm.change_table(:users, :atomic_switch => false) do |t|
-        t.remove_index(:reference, :index_users_on_reference)
+        t.remove_index([:username, :group])
+      end
+
+      slave do
+        index?(:users, :index_with_a_custom_name).must_equal(false)
+      end
+    end
+
+    it "should remove an index with a custom name by name" do
+      Lhm.change_table(:users, :atomic_switch => false) do |t|
+        t.remove_index(:irrelevant_column_name, :index_with_a_custom_name)
       end
 
       slave do
-        index?(:users, :index_users_on_reference).must_equal(false)
+        index?(:users, :index_with_a_custom_name).must_equal(false)
       end
     end
 

data/spec/unit/chunker_spec.rb

@@ -26,8 +26,8 @@ describe Lhm::Chunker do
     it "should copy the correct range and column" do
       @chunker.copy(from = 1, to = 100).must_equal(
         "insert ignore into `destination` (`secret`) " +
-        "select `secret` from `origin` " +
-        "where `id` between 1 and 100"
+        "select origin.`secret` from `origin` " +
+        "where origin.`id` between 1 and 100"
       )
     end
   end

data/spec/unit/migration_spec.rb

@@ -13,7 +13,7 @@ describe Lhm::Migration do
     @start = Time.now
     @origin = Lhm::Table.new("origin")
     @destination = Lhm::Table.new("destination")
-    @migration = Lhm::Migration.new(@origin, @destination, @start)
+    @migration = Lhm::Migration.new(@origin, @destination, nil, @start)
   end
 
   it "should name archive" do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: lhm
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 2.0.0
   prerelease: 
 platform: ruby
 authors:
@@ -12,7 +12,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-28 00:00:00.000000000 Z
+date: 2013-07-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -90,7 +90,9 @@ files:
 - spec/bootstrap.rb
 - spec/fixtures/destination.ddl
 - spec/fixtures/origin.ddl
+- spec/fixtures/permissions.ddl
 - spec/fixtures/small_table.ddl
+- spec/fixtures/tracks.ddl
 - spec/fixtures/users.ddl
 - spec/integration/atomic_switcher_spec.rb
 - spec/integration/chunker_spec.rb
@@ -113,7 +115,7 @@ files:
 - spec/unit/sql_helper_spec.rb
 - spec/unit/table_spec.rb
 - spec/unit/unit_helper.rb
-homepage: http://github.com/soundcloud/large-hadron-migrator
+homepage: http://github.com/soundcloud/lhm
 licenses: []
 post_install_message: 
 rdoc_options: []
@@ -137,4 +139,33 @@ rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: online schema changer for mysql
-test_files: []
+test_files:
+- spec/README.md
+- spec/bootstrap.rb
+- spec/fixtures/destination.ddl
+- spec/fixtures/origin.ddl
+- spec/fixtures/permissions.ddl
+- spec/fixtures/small_table.ddl
+- spec/fixtures/tracks.ddl
+- spec/fixtures/users.ddl
+- spec/integration/atomic_switcher_spec.rb
+- spec/integration/chunker_spec.rb
+- spec/integration/cleanup_spec.rb
+- spec/integration/entangler_spec.rb
+- spec/integration/integration_helper.rb
+- spec/integration/lhm_spec.rb
+- spec/integration/locked_switcher_spec.rb
+- spec/integration/table_spec.rb
+- spec/unit/active_record_connection_spec.rb
+- spec/unit/atomic_switcher_spec.rb
+- spec/unit/chunker_spec.rb
+- spec/unit/connection_spec.rb
+- spec/unit/datamapper_connection_spec.rb
+- spec/unit/entangler_spec.rb
+- spec/unit/intersection_spec.rb
+- spec/unit/locked_switcher_spec.rb
+- spec/unit/migration_spec.rb
+- spec/unit/migrator_spec.rb
+- spec/unit/sql_helper_spec.rb
+- spec/unit/table_spec.rb
+- spec/unit/unit_helper.rb