sbader-lhm 1.1.0

data/.gitignore

@@ -0,0 +1,6 @@
+*.gem
+.bundle
+Gemfile.lock
+gemfiles/*.lock
+pkg/*
+.rvmrc

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+before_script:
+  - "mysql -e 'create database lhm;'"
+rvm:
+  - 1.8.7
+  - 1.9.3
+gemfile:
+  - gemfiles/ar-2.3_mysql.gemfile
+  - gemfiles/ar-3.2_mysql.gemfile
+  - gemfiles/ar-3.2_mysql2.gemfile

data/CHANGELOG.md

@@ -0,0 +1,99 @@
+# 1.1.0 (April 29, 2012)
+
+* Add option to specify custom index name
+* Add mysql2 compatibility
+* Add AtomicSwitcher
+
+# 1.0.3 (February 23, 2012)
+
+* Improve change_column
+
+# 1.0.2 (February 17, 2012)
+
+* closes https://github.com/soundcloud/large-hadron-migrator/issues/11
+  this critical bug could cause data loss. table parser was replaced with
+  an implementation that reads directly from information_schema.
+
+# 1.0.1 (February 09, 2012)
+
+* released to rubygems
+
+# 1.0.0 (February 09, 2012)
+
+* added change_column
+* final 1.0 release
+
+# 1.0.0.rc8 (February 09, 2012)
+
+* removed spec binaries from gem bins
+
+# 1.0.0.rc7 (January 31, 2012)
+
+* added SqlHelper.annotation into the middle of trigger statements. this
+  is for the benefit of the killer script which should not kill trigger
+  statements.
+
+# 1.0.0.rc6 (January 30, 2012)
+
+* added --confirm to kill script; fixes to kill script
+
+# 1.0.0.rc5 (January 30, 2012)
+
+* moved scripts into bin, renamed, added to gem binaries
+
+# 1.0.0.rc4 (January 29, 2012)
+
+* added '-- lhm' to the end of statements for more visibility
+
+# 1.0.0.rc3 (January 19, 2012)
+
+* Speedup migrations for tables with large minimum id
+* Add a bit yard documentation
+* Fix issues with index creation on reserved column names
+* Improve error handling
+* Add tests for replication
+* Rename public API method from `hadron_change_table` to `change_table`
+* Add tests for ActiveRecord 2.3 and 3.1 compatibility
+
+# 1.0.0.rc2 (January 18, 2012)
+
+* Speedup migrations for tables with large ids
+* Fix conversion of milliseconds to seconds
+* Fix handling of sql errors
+* Add helper to create unique index
+* Allow index creation on prefix of column
+* Quote column names on index creation
+* Remove ambiguous method signature
+* Documentation fix
+* 1.8.7 compatibility
+
+# 1.0.0.rc1 (January 15, 2012)
+
+* rewrite.
+
+# 0.2.1 (November 26, 2011)
+
+* Include changelog in gem
+
+# 0.2.0 (November 26, 2011)
+
+* Add Ruby 1.8 compatibility
+* Setup travis continuous integration
+* Fix record lose issue
+* Fix and speed up specs
+
+# 0.1.4
+
+* Merged [Pullrequest #9](https://github.com/soundcloud/large-hadron-migrator/pull/9)
+
+# 0.1.3
+
+* code cleanup
+* Merged [Pullrequest #8](https://github.com/soundcloud/large-hadron-migrator/pull/8)
+* Merged [Pullrequest #7](https://github.com/soundcloud/large-hadron-migrator/pull/7)
+* Merged [Pullrequest #4](https://github.com/soundcloud/large-hadron-migrator/pull/4)
+* Merged [Pullrequest #1](https://github.com/soundcloud/large-hadron-migrator/pull/1)
+
+# 0.1.2
+
+* Initial Release

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2011, SoundCloud, Rany Keddo, Tobias Bielohlawek, Tobias Schmidt
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+- Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+- Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+- Neither the name of the SoundCloud nor the names of its contributors may be
+  used to endorse or promote products derived from this software without
+  specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README.md

@@ -0,0 +1,146 @@
+# Large Hadron Migrator [![Build Status](https://secure.travis-ci.org/soundcloud/large-hadron-migrator.png)][4]
+
+Rails style database migrations are a useful way to evolve your data schema in
+an agile manner. Most Rails projects start like this, and at first, making
+changes is fast and easy.
+
+That is until your tables grow to millions of records. At this point, the
+locking nature of `ALTER TABLE` may take your site down for an hour or more
+while critical tables are migrated. In order to avoid this, developers begin
+to design around the problem by introducing join tables or moving the data
+into another layer. Development gets less and less agile as tables grow and
+grow. To make the problem worse, adding or changing indices to optimize data
+access becomes just as difficult.
+
+> Side effects may include black holes and universe implosion.
+
+There are few things that can be done at the server or engine level. It is
+possible to change default values in an `ALTER TABLE` without locking the
+table. The InnoDB Plugin provides facilities for online index creation, which
+is great if you are using this engine, but only solves half the problem.
+
+At SoundCloud we started having migration pains quite a while ago, and after
+looking around for third party solutions, we decided to create our
+own. We called it Large Hadron Migrator, and it is a gem for online
+ActiveRecord migrations.
+
+![LHC](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)
+
+[The Large Hadron collider at CERN](http://en.wikipedia.org/wiki/Large_Hadron_Collider)
+
+## The idea
+
+The basic idea is to perform the migration online while the system is live,
+without locking the table. In contrast to [OAK][0] and the
+[facebook tool][1], we only use a copy table and triggers.
+
+The Large Hadron is a test driven Ruby solution which can easily be dropped
+into an ActiveRecord migration. It presumes a single auto incremented
+numerical primary key called id as per the Rails convention. Unlike the
+[twitter solution][2], it does not require the presence of an indexed
+`updated_at` column.
+
+## Requirements
+
+Lhm currently only works with MySQL databases and requires an established
+ActiveRecord connection.
+
+It is compatible and [continuously tested][4] with Ruby 1.8.7 and Ruby 1.9.x,
+ActiveRecord 2.3.x and 3.x as well as mysql and mysql2 adapters.
+
+## Installation
+
+Install it via `gem install lhm` or add `gem "lhm"` to your Gemfile.
+
+## Usage
+
+You can invoke Lhm directly from a plain ruby file after connecting ActiveRecord
+to your mysql instance:
+
+```ruby
+require 'lhm'
+
+ActiveRecord::Base.establish_connection(
+  :adapter => 'mysql',
+  :host => '127.0.0.1',
+  :database => 'lhm'
+)
+
+Lhm.change_table :users do |m|
+  m.add_column :arbitrary, "INT(12)"
+  m.add_index  [:arbitrary_id, :created_at]
+  m.ddl("alter table %s add column flag tinyint(1)" % m.name)
+end
+```
+
+To use Lhm from an ActiveRecord::Migration in a Rails project, add it to your
+Gemfile, then invoke as follows:
+
+```ruby
+require 'lhm'
+
+class MigrateUsers < ActiveRecord::Migration
+  def self.up
+    Lhm.change_table :users do |m|
+      m.add_column :arbitrary, "INT(12)"
+      m.add_index  [:arbitrary_id, :created_at]
+      m.ddl("alter table %s add column flag tinyint(1)" % m.name)
+    end
+  end
+
+  def self.down
+    Lhm.change_table :users do |m|
+      m.remove_index  [:arbitrary_id, :created_at]
+      m.remove_column :arbitrary)
+    end
+  end
+end
+```
+
+## Table rename strategies
+
+There are two different table rename strategies available: LockedSwitcher and
+AtomicSwitcher.
+
+For all setups which use replication and a MySQL version
+affected by the the [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675),
+we recommend the LockedSwitcher strategy to avoid replication issues. This
+strategy locks the table being migrated and issues two ALTER TABLE statements.
+The AtomicSwitcher uses a single atomic RENAME TABLE query and should be favored
+in setups which do not suffer from the mentioned replication bug.
+
+Lhm chooses the strategy automatically based on the used MySQL server version,
+but you can override the behavior with an option:
+
+```ruby
+Lhm.change_table :users, :atomic_switch => true do |m|
+  # ...
+end
+```
+
+## Contributing
+
+We'll check out your contribution if you:
+
+  * Provide a comprehensive suite of tests for your fork.
+  * Have a clear and documented rationale for your changes.
+  * Package these up in a pull request.
+
+We'll do our best to help you out with any contribution issues you may have.
+
+## License
+
+The license is included as LICENSE in this directory.
+
+## Similar solutions
+
+  * [OAK: online alter table][0]
+  * [Facebook][1]
+  * [Twitter][2]
+  * [pt-online-schema-change][3]
+
+[0]: http://openarkkit.googlecode.com
+[1]: http://www.facebook.com/note.php?note\_id=430801045932
+[2]: https://github.com/freels/table_migrator
+[3]: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html
+[4]: http://travis-ci.org/soundcloud/large-hadron-migrator

data/Rakefile

@@ -0,0 +1,20 @@
+require 'rake/testtask'
+require 'bundler'
+
+Bundler::GemHelper.install_tasks
+
+Rake::TestTask.new("unit") do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['spec/unit/*_spec.rb']
+  t.verbose = true
+end
+
+Rake::TestTask.new("integration") do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['spec/integration/*_spec.rb']
+  t.verbose = true
+end
+
+task :specs => [:unit, :integration]
+task :default => :specs
+

data/bin/lhm-kill-queue

@@ -0,0 +1,172 @@
+#!/usr/bin/env ruby
+
+require 'active_record'
+require 'lhm/sql_helper'
+require 'optparse'
+
+module Lhm
+  class KillQueue
+
+    def initialize
+      @port = 3306
+      @grace = 10
+      @tiny = 0.1
+      @marker = "%#{ SqlHelper.annotation }%"
+
+      OptionParser.new do |opts|
+        opts.on("-h", "--hostname HOSTNAME") { |v| @hostname = v }
+        opts.on("-u", "--username USERNAME") { |v| @username = v }
+        opts.on("-p", "--password PASSWORD") { |v| @password = v }
+        opts.on("-d", "--database DATABASE") { |v| @database = v }
+        opts.on("-m", "--mode MODE") { |v| @mode = v.to_sym }
+        opts.on("-y", "--confirm") { |v| @confirm = true }
+      end.parse!
+
+      unless(@hostname && @username && @password && @database)
+        abort usage
+      end
+
+      unless([:kill, :master, :slave].include?(@mode))
+        abort "specify -m kill OR -m master OR -m slave"
+      end
+
+      connect
+    end
+
+    def usage
+      <<-desc.gsub(/^      /, '')
+        kills queries on the given server after detecting 'lock table#{ @marker }'.
+        usage:
+          lhm-kill-queue -h hostname -u username -p password -d database \\
+          (-m kill | -m master | -m slave) [--confirm]
+
+      desc
+    end
+
+    def run
+      case @mode
+      when :kill then kill
+      when :master then master
+      when :slave then slave
+      end
+    end
+
+    def kill
+      lock = trip
+      kill_process(lock)
+    end
+
+    def master
+      lock = trip
+      puts "starting to kill non lhm processes in #{ @grace } seconds"
+      sleep(@grace + @tiny)
+
+      [list_non_lhm].flatten.each do |process|
+        kill_process(process)
+        sleep(@tiny)
+      end
+    end
+
+    def slave
+      lock = trip
+      puts "starting to kill non lhm SELECT processes in #{ @grace } seconds"
+      sleep(@grace + @tiny)
+
+      [list_non_lhm].flatten.each do |process|
+        if(select?(process))
+          kill_process(process)
+          sleep(@tiny)
+        end
+      end
+    end
+
+  private
+
+    def connect
+      ActiveRecord::Base.establish_connection({
+        :adapter => 'mysql',
+        :host => @hostname,
+        :port => @port,
+        :username => @username,
+        :password => @password,
+        :database => @database
+      })
+    end
+
+    def connection
+      ActiveRecord::Base.connection
+    end
+
+    def list_non_lhm
+      select_processes %Q(
+        info not like '#{ @marker }' and time > #{ @grace } and command = 'Query'
+      )
+    end
+
+    def trip
+      until res = select_processes("info like 'lock table#{ @marker }'").first
+        sleep @tiny
+        print '.'
+      end
+
+      res
+    end
+
+    def kill_process(process_id)
+      puts "killing #{ select_statement(process_id) }"
+
+      if(@confirm)
+        print "confirm ('y' to confirm): "
+
+        if(gets.strip != 'y')
+          puts "skipped."
+          return
+        end
+      end
+
+      connection.execute("kill #{ process_id }")
+      puts "killed #{ process_id }"
+    end
+
+    def select?(process)
+      if statement = select_statement(process)
+        case statement
+        when /delete/i then false
+        when /update/i then false
+        when /insert/i then false
+        else
+          !!statement.match(/select/i)
+        end
+      end
+    end
+
+    def select_statement(process)
+      if process
+        value %Q(
+          select info from information_schema.processlist where id = #{ process }
+        )
+      end
+    end
+
+    def select_processes(predicate)
+      values %Q(
+        select id from information_schema.processlist
+         where db = '#{ @database }'
+           and user = '#{ @username }'
+           and #{ predicate }
+      )
+    end
+
+    def value(statement)
+      connection.select_value(statement)
+    end
+
+    def values(statement)
+      connection.select_values(statement)
+    end
+  end
+end
+
+killer = Lhm::KillQueue.new
+killer.run
+