lhm 1.0.0.rc.1

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+.rvmrc

data/.travis.yml

@@ -0,0 +1,10 @@
+rvm:
+  - 1.8.7
+  - 1.9.2
+  - 1.9.3
+before_script:
+  - "mysql -e 'create database large_hadron_migrator;'"
+branches:
+  only:
+    - master
+    - stable

data/CHANGELOG.md

@@ -0,0 +1,32 @@
+# 1.0.0-RC1
+
+* rewrite.
+
+# 0.9.1
+
+# 0.2.1 (November 26, 2011)
+
+* Include changelog in gem
+
+# 0.2.0 (November 26, 2011)
+
+* Add Ruby 1.8 compatibility
+* Setup travis continuous integration
+* Fix record lose issue
+* Fix and speed up specs
+
+# 0.1.4
+
+* Merged [Pullrequest #9](https://github.com/soundcloud/large-hadron-migrator/pull/9)
+
+# 0.1.3
+
+* code cleanup
+* Merged [Pullrequest #8](https://github.com/soundcloud/large-hadron-migrator/pull/8)
+* Merged [Pullrequest #7](https://github.com/soundcloud/large-hadron-migrator/pull/7)
+* Merged [Pullrequest #4](https://github.com/soundcloud/large-hadron-migrator/pull/4)
+* Merged [Pullrequest #1](https://github.com/soundcloud/large-hadron-migrator/pull/1)
+
+# 0.1.2
+
+* Initial Release

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2011, SoundCloud, Rany Keddo, Tobias Bielohlawek, Tobias Schmidt
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+- Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+- Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+- Neither the name of the SoundCloud nor the names of its contributors may be
+  used to endorse or promote products derived from this software without
+  specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README.md

@@ -0,0 +1,98 @@
+# Large Hadron Migrator [![Build Status](https://secure.travis-ci.org/soundcloud/large-hadron-migrator.png)](http://travis-ci.org/soundcloud/large-hadron-migrator)
+
+Rails style database migrations are a useful way to evolve your data schema in
+an agile manner. Most Rails projects start like this, and at first, making
+changes is fast and easy.
+
+That is until your tables grow to millions of records. At this point, the
+locking nature of `ALTER TABLE` may take your site down for an hour or more
+while critical tables are migrated. In order to avoid this, developers begin
+to design around the problem by introducing join tables or moving the data
+into another layer. Development gets less and less agile as tables grow and
+grow. To make the problem worse, adding or changing indices to optimize data
+access becomes just as difficult.
+
+> Side effects may include black holes and universe implosion.
+
+There are few things that can be done at the server or engine level. It is
+possible to change default values in an `ALTER TABLE` without locking the
+table.  The InnoDB Plugin provides facilities for online index creation, which
+is great if you are using this engine, but only solves half the problem.
+
+At SoundCloud we started having migration pains quite a while ago, and after
+looking around for third party solutions [0] [1] [2], we decided to create our
+own. We called it Large Hadron Migrator, and it is a gem for online
+ActiveRecord migrations.
+
+![LHC](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)
+
+[The Large Hadron collider at CERN](http://en.wikipedia.org/wiki/Large_Hadron_Collider)
+
+## The idea
+
+The basic idea is to perform the migration online while the system is live,
+without locking the table. Similar to OAK (online alter table) [2] and the
+facebook tool [0], we use a copy table, triggers and a journal table.
+
+The Large Hadron is a test driven Ruby solution which can easily be dropped
+into an ActiveRecord migration. It presumes a single auto incremented
+numerical primary key called id as per the Rails convention. Unlike the
+twitter solution [1], it does not require the presence of an indexed
+`updated_at` column.
+
+## Usage
+
+After including Lhm, `hadron_change_table` becomes available
+with the following methods:
+
+    class MigrateArbitrary < ActiveRecord::Migration
+      include Lhm
+
+      def self.up
+        hadron_change_table(:users) do |t|
+          t.add_column(:arbitrary, "INT(12)")
+          t.add_index([:arbitrary, :created_at])
+          t.ddl("alter table %s add column flag tinyint(1)" % t.name)
+        end
+      end
+
+      def self.down
+        hadron_change_table(:users) do |t|
+          t.remove_index([:arbitrary, :created_at])
+          t.remove_column(:arbitrary)
+        end
+      end
+    end
+
+## Migration phases
+
+_TODO_
+
+### When adding a column
+
+_TODO_
+
+### When removing a column
+
+_TODO_
+
+## Contributing
+
+We'll check out your contribution if you:
+
+- Provide a comprehensive suite of tests for your fork.
+- Have a clear and documented rationale for your changes.
+- Package these up in a pull request.
+
+We'll do our best to help you out with any contribution issues you may have.
+
+## License
+
+The license is included as LICENSE in this directory.
+
+## Footnotes
+
+[0]: http://www.facebook.com/note.php?note\_id=430801045932 "Facebook"
+[1]: https://github.com/freels/table\_migrator "Twitter"
+[2]: http://openarkkit.googlecode.com "OAK online alter table"
+

data/Rakefile

@@ -0,0 +1,16 @@
+require 'rake/testtask'
+
+Rake::TestTask.new("unit") do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['spec/unit/*_spec.rb']
+  t.verbose = true
+end
+
+Rake::TestTask.new("integration") do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['spec/integration/*_spec.rb']
+  t.verbose = true
+end
+
+task :default => [:unit, :integration]
+

data/TODO

@@ -0,0 +1,11 @@
+todo
+
+. locked_switcher
+. schema_creator
+. chunker
+
+checklist
+
+. consider entanglement epoch
+  - should not lose changes due to low epoch
+

data/lhm.gemspec

@@ -0,0 +1,29 @@
+# -*- encoding: utf-8 -*-
+
+lib = File.expand_path('../lib', __FILE__)
+$:.unshift(lib) unless $:.include?(lib)
+
+puts $:.inspect
+
+require 'lhm'
+
+Gem::Specification.new do |s|
+  s.name          = "lhm"
+  s.version       = Lhm::VERSION
+  s.platform      = Gem::Platform::RUBY
+  s.authors       = ["SoundCloud", "Rany Keddo", "Tobias Bielohlawek", "Tobias Schmidt"]
+  s.email         = %q{rany@soundcloud.com, tobi@soundcloud.com, ts@soundcloud.com}
+  s.summary       = %q{online schema changer for mysql}
+  s.description   = %q{Migrate large tables without downtime by copying to a temporary table in chunks. The old table is not dropped. Instead, it is moved to timestamp_table_name for verification.}
+  s.homepage      = %q{http://github.com/soundcloud/large-hadron-migrator}
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # this should be a real dependency, but we're using a different gem in our code
+  s.add_development_dependency "mysql", "~> 2.8.1"
+  s.add_development_dependency "rspec", "=1.3.1"
+  s.add_development_dependency "rake"
+end
+

data/lib/lhm.rb

@@ -0,0 +1,20 @@
+#
+#  Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
+#  Schmidt
+#
+
+require 'lhm/table'
+require 'lhm/invoker'
+require 'lhm/migration'
+
+module Lhm
+  VERSION = "1.0.0.rc.1"
+
+  def hadron_change_table(table_name, chunk_options = {}, &block)
+    origin = Table.parse(table_name, connection)
+    invoker = Invoker.new(origin, connection)
+    block.call(invoker.migrator)
+    invoker.run(chunk_options)
+  end
+end
+

data/lib/lhm/chunker.rb

@@ -0,0 +1,73 @@
+#
+#  Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
+#  Schmidt
+#
+
+require 'lhm/migration'
+require 'lhm/command'
+
+module Lhm
+  class Chunker
+    include Command
+
+    #
+    # Copy from origin to destination in chunks of size `stride`. Sleeps for
+    # `throttle` milliseconds between each stride.
+    #
+
+    def initialize(migration, limit = 1, connection = nil, options = {})
+      @stride = options[:stride] || 40_000
+      @throttle = options[:throttle] || 100
+      @limit = limit
+      @connection = connection
+      @migration = migration
+    end
+
+    #
+    # Copies chunks of size `stride`, starting from id 1 up to id `limit`.
+    #
+
+    def up_to(limit)
+      traversable_chunks_up_to(limit).times do |n|
+        yield(bottom(n + 1), top(n + 1, limit)) && sleep(throttle_seconds)
+      end
+    end
+
+    def traversable_chunks_up_to(limit)
+      (limit / @stride.to_f).ceil
+    end
+
+    def bottom(chunk)
+      (chunk - 1) * @stride + 1
+    end
+
+    def top(chunk, limit)
+      [chunk * @stride, limit].min
+    end
+
+    def copy(lowest, highest)
+      "insert ignore into `#{ @migration.destination.name }` (#{ cols.joined  }) " +
+      "select #{ cols.joined } from `#{ @migration.origin.name }` " +
+      "where `id` between #{ lowest } and #{ highest }"
+    end
+
+  private
+
+    def cols
+      @cols ||= @migration.intersection
+    end
+
+    def execute
+      up_to(@limit) do |lowest, highest|
+        print "."
+
+        sql copy(lowest, highest)
+      end
+    end
+
+    def throttle_seconds
+      @throttle / 100.0
+    end
+  end
+end
+

data/lib/lhm/command.rb

@@ -0,0 +1,70 @@
+#
+#  Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
+#  Schmidt
+#
+#  Apply a change to the database.
+#
+
+module Lhm
+  module Command
+    def self.included(base)
+      base.send :attr_reader, :connection
+    end
+
+    #
+    # Command Interface
+    #
+
+    def validate; end
+
+    def revert
+      raise NotImplementedError.new(self.class.name)
+    end
+
+    def run(&block)
+      validate
+
+      if(block_given?)
+        before
+        block.call(self)
+        after
+      else
+        execute
+      end
+    end
+
+  private
+
+    def execute
+      raise NotImplementedError.new(self.class.name)
+    end
+
+    def before
+      raise NotImplementedError.new(self.class.name)
+    end
+
+    def after
+      raise NotImplementedError.new(self.class.name)
+    end
+
+    def table?(table_name)
+      @connection.table_exists?(table_name)
+    end
+
+    def error(msg)
+      raise Exception.new("#{ self.class }: #{ msg }")
+    end
+
+    def sql(statements)
+      [*statements].each do |statement|
+        begin
+          @connection.execute(statement)
+        rescue Mysql::Error => e
+          revert
+          error "#{ statement } failed: #{ e.inspect }"
+        end
+      end
+    end
+  end
+end
+

data/lib/lhm/entangler.rb

@@ -0,0 +1,105 @@
+#
+#  Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
+#  Schmidt
+#
+#  Creates entanglement between two tables. All creates, updates and deletes
+#  to origin will be repeated on the the destination table.
+#
+
+require 'lhm/command'
+
+module Lhm
+  class Entangler
+    include Command
+
+    attr_reader :epoch
+
+    def initialize(migration, connection = nil)
+      @common = migration.intersection
+      @origin = migration.origin
+      @destination = migration.destination
+      @connection = connection
+    end
+
+    def entangle
+      [
+        create_trigger_del,
+        create_trigger_ins,
+        create_trigger_upd
+      ]
+    end
+
+    def untangle
+      [
+        "drop trigger if exists `#{ trigger(:del) }`",
+        "drop trigger if exists `#{ trigger(:ins) }`",
+        "drop trigger if exists `#{ trigger(:upd) }`"
+      ]
+    end
+
+    def create_trigger_ins
+      strip %Q{
+        create trigger `#{ trigger(:ins) }`
+        after insert on `#{ @origin.name }` for each row
+        replace into `#{ @destination.name }` (#{ @common.joined })
+        values (#{ @common.typed("NEW") })
+      }
+    end
+
+    def create_trigger_upd
+      strip %Q{
+        create trigger `#{ trigger(:upd) }`
+        after update on `#{ @origin.name }` for each row
+        replace into `#{ @destination.name }` (#{ @common.joined })
+        values (#{ @common.typed("NEW") })
+      }
+    end
+
+    def create_trigger_del
+      strip %Q{
+        create trigger `#{ trigger(:del) }`
+        after delete on `#{ @origin.name }` for each row
+        delete ignore from `#{ @destination.name }`
+        where `#{ @destination.name }`.`id` = OLD.`id`
+      }
+    end
+
+    def trigger(type)
+      "lhmt_#{ type }_#{ @origin.name }"
+    end
+
+    #
+    # Command implementation
+    #
+
+    def validate
+      unless table?(@origin.name)
+        error("#{ @origin.name } does not exist")
+      end
+
+      unless table?(@destination.name)
+        error("#{ @destination.name } does not exist")
+      end
+    end
+
+    def before
+      sql(entangle)
+      @epoch = connection.select_value("select max(id) from #{ @origin.name }").to_i
+    end
+
+    def after
+      sql(untangle)
+    end
+
+    def revert
+      after
+    end
+
+  private
+
+    def strip(sql)
+      sql.strip.gsub(/\n */, "\n")
+    end
+  end
+end
+