lhm-shopify 3.3.5

data/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in shopify_lhm.gemspec
+gemspec
+

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2011 - 2013, SoundCloud, Rany Keddo, Tobias Bielohlawek, Tobias Schmidt
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+- Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+- Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+- Neither the name of the SoundCloud nor the names of its contributors may be
+  used to endorse or promote products derived from this software without
+  specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README.md

@@ -0,0 +1,284 @@
+# Large Hadron Migrator
+
+[![Tests](https://github.com/Shopify/lhm/actions/workflows/test.yml/badge.svg)](https://github.com/Shopify/lhm/actions/workflows/test.yml)
+
+This is the Shopify fork of [SoundCloud's LHM](https://github.com/soundcloud/lhm). The 
+following description, originally from SoundCloud (with minor updates by Shopify),
+gives some of the flavor around its original creation, and its choice of name...
+
+Rails-style database migrations are a useful way to evolve your database schema in
+an agile manner. Most Rails projects start like this, and at first, making
+changes is fast and easy.
+
+That is, until your tables grow to millions or billions of records. At this point,
+the locking nature of `ALTER TABLE` may take your site down for hours or more
+while critical tables are migrated. In order to avoid this, developers begin
+to design around the problem by introducing join tables or moving the data
+into another layer. Development gets less and less agile as tables grow and
+grow. To make the problem worse, adding or changing indices to optimize data
+access becomes just as difficult.
+
+*Side effects may include black holes and universe implosion.*
+
+There are few things that can be done at the server or engine level. It is
+possible to change default values in an `ALTER TABLE` without locking the
+table. InnoDB provides facilities for online index creation, but that only
+solves half the problem.
+
+At SoundCloud we started having migration pains quite a while ago, and after
+looking around for third party solutions, we decided to create our
+own. We called it **Large Hadron Migrator**, and it is a Ruby Gem that provides
+facilities for online ActiveRecord migrations.
+
+![The Large Hadron Collider at CERN](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)
+
+The [Large Hadron Collider](http://en.wikipedia.org/wiki/Large_Hadron_Collider) at [CERN](https://en.wikipedia.org/wiki/CERN) near Geneva, Switzerland.
+
+## The idea
+
+The basic idea is to perform the migration online while the system is live,
+without locking the table. In contrast to [OAK][0] and the [facebook tool][1], we
+only use a copy table and triggers.
+
+LHM is a test-driven Ruby solution which can easily be dropped into an ActiveRecord
+migration. It presumes a single auto-incremented numeric primary key called `id` as
+per the Rails convention. Unlike [Matt Freels's `table_migrator` solution][2],
+it does not require the presence of an indexed `updated_at` column.
+
+## Requirements
+
+LHM currently only works with MySQL databases and requires an established
+ActiveRecord connection.
+
+## Limitations
+
+Due to the Chunker implementation, LHM requires that the table to migrate has a
+a single integer numeric key column named `id`.
+
+## Installation
+
+Install it via `gem install lhm` or by adding `gem "lhm"` to your `Gemfile`.
+
+## Usage
+
+You can invoke LHM directly from a plain Ruby file after connecting ActiveRecord
+to your MySQL instance:
+
+```ruby
+require 'lhm'
+
+ActiveRecord::Base.establish_connection(
+  :adapter => 'mysql',
+  :host => '127.0.0.1',
+  :database => 'lhm'
+)
+
+# and migrate
+Lhm.change_table :users do |m|
+  m.add_column :arbitrary, "INT(12)"
+  m.add_index  [:arbitrary_id, :created_at]
+  m.ddl("alter table %s add column flag tinyint(1)" % m.name)
+end
+```
+
+To use LHM from an `ActiveRecord::Migration` in a Rails project, add it to your
+`Gemfile`, then invoke as follows:
+
+```ruby
+require 'lhm'
+
+class MigrateUsers < ActiveRecord::Migration
+  def self.up
+    Lhm.change_table :users do |m|
+      m.add_column :arbitrary, "INT(12)"
+      m.add_index  [:arbitrary_id, :created_at]
+      m.ddl("alter table %s add column flag tinyint(1)" % m.name)
+    end
+  end
+
+  def self.down
+    Lhm.change_table :users do |m|
+      m.remove_index  [:arbitrary_id, :created_at]
+      m.remove_column :arbitrary
+    end
+  end
+end
+```
+
+**Note:** LHM does not delete the old, leftover table. This is intentional, in order
+to prevent accidental data loss. After successful or failed LHM migrations, these leftover
+tables must be cleaned up.
+
+## Throttler
+
+LHM uses a throttling mechanism to read data in your original table. By default, 2,000 rows are read each 0.1 second. If you want to change that behaviour, you can pass an instance of a throttler with the `throttler` option. In this example, 1,000 rows will be read with a 10 second delay between each processing:
+
+```ruby
+my_throttler = Lhm::Throttler::Time.new(stride: 1000, delay: 10)
+
+Lhm.change_table :users, throttler: my_throttler  do |m|
+  ...
+end
+```
+
+### SlaveLag Throttler
+
+Lhm uses by default the time throttler, however a better solution is to throttle the copy of the data
+depending on the time that the slaves are behind. To use the SlaveLag throttler:
+
+```ruby
+Lhm.change_table :users, throttler: :slave_lag_throttler  do |m|
+  ...
+end
+```
+
+Or to set that as default throttler, use the following (for instance in a Rails initializer):
+
+```ruby
+Lhm.setup_throttler(:slave_lag_throttler)
+```
+
+### ThreadsRunning Throttler
+
+If you don't have access to connect directly to your replicas, you can also
+throttle based on the number of threads running in MySQL, as a proxy for "is
+this operation causing excessive load":
+
+```ruby
+Lhm.change_table :users, throttler: :threads_running_throttler do |m|
+  ...
+end
+```
+
+Or to set that as default throttler, use the following (for instance in a Rails initializer):
+
+```ruby
+Lhm.setup_throttler(:threads_running_throttler)
+```
+
+## Table rename strategies
+
+There are two different table rename strategies available: `LockedSwitcher` and
+`AtomicSwitcher`.
+
+The `LockedSwitcher` strategy locks the table being migrated and issues two `ALTER TABLE` statements. The `AtomicSwitcher` uses a single atomic `RENAME TABLE` query and is the favored solution.
+
+LHM chooses `AtomicSwitcher` if no strategy is specified, **unless** your version of MySQL is
+affected by [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675). If your version is
+affected, LHM will raise an error if you don't specify a strategy. You're recommended
+to use the `LockedSwitcher` in these cases to avoid replication issues.
+
+To specify the strategy in your migration:
+
+```ruby
+Lhm.change_table :users, :atomic_switch => true do |m|
+  ...
+end
+```
+
+## Limiting the data that is migrated
+
+For instances where you want to limit the data that is migrated to the new
+table by some conditions, you may tell the migration to filter by a set of
+conditions:
+
+```ruby
+Lhm.change_table(:sounds) do |m|
+  m.filter("inner join users on users.`id` = sounds.`user_id` and sounds.`public` = 1")
+end
+```
+
+Note that this SQL will be inserted into the copy directly after the `FROM` clause
+so be sure to use `INNER JOIN` or `OUTER JOIN` syntax and not comma-joins. These
+conditions will not affect the triggers, so any modifications to the table
+during the run will happen on the new table as well.
+
+## Cleaning up after an interrupted Lhm run
+
+If an LHM migration is interrupted, it may leave behind the temporary tables
+and/or triggers used in the migration. If the migration is re-started, the
+unexpected presence of these tables will cause an error.
+
+In this case, `Lhm.cleanup` can be used to drop any orphaned LHM temporary tables or triggers.
+
+To see what LHM tables/triggers are found:
+
+```ruby
+Lhm.cleanup
+```
+
+To remove any LHM tables/triggers found:
+
+```ruby
+Lhm.cleanup(true)
+```
+
+Optionally, only remove tables up to a specific time, if you want to retain previous migrations.
+
+Rails:
+
+```ruby
+Lhm.cleanup(true, until: 1.day.ago)
+```
+
+Ruby:
+
+```ruby
+Lhm.cleanup(true, until: Time.now - 86400)
+```
+
+## Contributing
+
+To run the tests:
+
+```bash
+bundle exec rake unit # unit tests
+bundle exec rake integration # integration tests
+bundle exec rake unit # all tests
+```
+
+You can run an individual test as follows:
+
+```bash
+bundle exec rake unit TEST=spec/integration/atomic_switcher_spec.rb
+```
+
+You can check the code coverage reporting for an individual test as follows:
+
+```bash
+rm -rf coverage
+COV=1 bundle exec rake unit TEST=spec/integration/atomic_switcher_spec.rb
+open coverage/index.html
+```
+
+To check the code coverage for all tests:
+
+```bash
+rm -rf coverage
+COV=1 bundle exec rake unit && bundle exec rake integration
+open coverage/index.html
+```
+
+### dbdeployer
+
+The integration tests rely on a replicated configuration for MySQL. We're using 
+[`dbdeployer`](https://github.com/datacharmer/dbdeployer) to manage the test nodes
+required to set up these configurations.
+
+## License
+
+The license is included as [LICENSE](LICENSE) in this directory.
+
+## Similar solutions
+
+  * [OAK: online alter table][0]
+  * [Facebook][1]
+  * [Twitter][2]
+  * [pt-online-schema-change][3]
+
+[0]: http://openarkkit.googlecode.com
+[1]: http://www.facebook.com/note.php?note\_id=430801045932
+[2]: https://github.com/freels/table_migrator
+[3]: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html
+[4]: https://travis-ci.org/soundcloud/lhm
+[5]: https://travis-ci.org/soundcloud/lhm.svg?branch=master

data/Rakefile

@@ -0,0 +1,22 @@
+require 'rake/testtask'
+require 'bundler'
+require 'bundler/gem_tasks'
+
+Bundler::GemHelper.install_tasks
+
+Rake::TestTask.new('unit') do |t|
+  t.libs << 'lib'
+  t.libs << 'spec'
+  t.test_files = FileList['spec/unit/**/*_spec.rb']
+  t.verbose = true
+end
+
+Rake::TestTask.new('integration') do |t|
+  t.libs << 'lib'
+  t.libs << 'spec'
+  t.test_files = FileList['spec/integration/**/*_spec.rb']
+  t.verbose = true
+end
+
+task :specs => [:unit, :integration]
+task :default => :specs

data/dbdeployer/config.json

@@ -0,0 +1,32 @@
+{
+ 	"version": "1.8.0",
+ 	"sandbox-home": "./dbdeployer/sandboxes",
+ 	"sandbox-binary": "./dbdeployer/binaries",
+ 	"use-sandbox-catalog": true,
+ 	"master-slave-base-port": 11000,
+ 	"group-replication-base-port": 12000,
+ 	"group-replication-sp-base-port": 13000,
+ 	"fan-in-replication-base-port": 14000,
+ 	"all-masters-replication-base-port": 15000,
+ 	"multiple-base-port": 16000,
+ 	"group-port-delta": 125,
+ 	"mysqlx-port-delta": 10000,
+ 	"master-name": "master",
+ 	"master-abbr": "m",
+ 	"node-prefix": "node",
+ 	"slave-prefix": "slave",
+ 	"slave-abbr": "s",
+ 	"sandbox-prefix": "msb_",
+ 	"master-slave-prefix": "rsandbox_",
+ 	"group-prefix": "group_msb_",
+ 	"group-sp-prefix": "group_sp_msb_",
+ 	"multiple-prefix": "multi_msb_",
+ 	"fan-in-prefix": "fan_in_msb_",
+ 	"all-masters-prefix": "all_masters_msb_",
+ 	"reserved-ports": [
+ 		1186,
+ 		3306,
+ 		33060
+ 	],
+ 	"timestamp": "Mon Jul 16 17:36:55 AST 2018"
+ }

data/dbdeployer/install.sh

@@ -0,0 +1,64 @@
+set -e
+mkdir -p ./dbdeployer/sandboxes
+mkdir -p ./dbdeployer/binaries
+
+if [ -z "$(uname | grep Darwin)" ]; then
+  OS=linux
+  set -x
+else
+  OS=osx
+fi
+
+echo "Checking if dbdeployer is installed"
+if ! [ -x "$(command -v ./bin/dbdeployer)" ]; then
+  echo "Not installed...starting install"
+  VERSION=1.56.0
+  origin=https://github.com/datacharmer/dbdeployer/releases/download/v$VERSION
+  filename=dbdeployer-$VERSION.$OS
+  wget -q $origin/$filename.tar.gz
+  tar -xzf $filename.tar.gz
+  chmod +x $filename
+  sudo mv $filename ./bin/dbdeployer
+  rm $filename.tar.gz
+else
+  echo "Installation found!"
+fi
+
+
+echo "Checking if mysql 5.7.22 is available for dbdeployer"
+if [ -z "$(./bin/dbdeployer --config ./dbdeployer/config.json --sandbox-binary "./dbdeployer/binaries" available | grep 5.7.22)" ]; then
+  echo "Not found..."
+
+  if [ "$OS" = "linux" ]; then
+    MYSQL_FILE=mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz
+  else
+    MYSQL_FILE=mysql-5.7.22-macos10.13-x86_64.tar.gz
+  fi
+
+  if [ ! -f $MYSQL_FILE ]; then
+    echo "Downloading $MYSQL_FILE...(this may take a while)"
+    wget -q "https://dev.mysql.com/get/Downloads/MySQL-5.7/$MYSQL_FILE"
+  fi
+
+  echo "Setting up..."
+  ./bin/dbdeployer unpack $MYSQL_FILE --verbosity 0 --config ./dbdeployer/config.json --sandbox-binary "./dbdeployer/binaries"
+  rm $MYSQL_FILE
+else
+  echo "mysql 5.7.22 found!"
+fi
+
+echo "Forcing new replication setup..."
+./bin/dbdeployer deploy replication 5.7.22 --nodes 2 --force --config ./dbdeployer/config.json --sandbox-binary "./dbdeployer/binaries" --sandbox-home "./dbdeployer/sandboxes"
+./bin/dbdeployer global status --config ./dbdeployer/config.json --sandbox-binary "./dbdeployer/binaries" --sandbox-home "./dbdeployer/sandboxes"
+
+echo "Setting up database.yml"
+DATABASE_YML=spec/integration/database.yml
+echo "master:" > $DATABASE_YML
+cat ./dbdeployer/sandboxes/rsandbox_5_7_22/master/my.sandbox.cnf | grep -A 4 client | tail -n 4 | sed -e 's/  * = /: /' -e 's/^/  /' >> $DATABASE_YML
+
+echo "slave:" >> $DATABASE_YML
+cat ./dbdeployer/sandboxes/rsandbox_5_7_22/node1/my.sandbox.cnf | grep -A 4 client | tail -n 4 | sed -e 's/  * = /: /' -e 's/^/  /' >> $DATABASE_YML
+
+cat $DATABASE_YML
+
+echo "You are ready to run the integration test suite..."

data/dev.yml

@@ -0,0 +1,20 @@
+name: lhm
+up:
+  - homebrew:
+      - mysql-client@5.7:
+          or: [mysql@5.7]
+          conflicts: [shopify/shopify/mysql-client, mysql-connector-c, mysql, mysql-client]
+      - wget
+  - ruby: 2.6.5
+  - bundler
+  - custom:
+      name: Database
+      met?: test -f spec/integration/database.yml && test "$(./dbdeployer/sandboxes/rsandbox_5_7_22/status_all | grep on\  | wc -l | xargs echo)" = "2"
+      meet: ./dbdeployer/install.sh
+      down: ./dbdeployer/sandboxes/rsandbox_5_7_22/stop_all
+
+commands:
+  unit: bundle exec rake unit
+  int: bundle exec rake integration
+  test: bundle exec rake unit && bundle exec rake integration
+  cov: rm -rf coverage; COV=1 bundle exec rake unit && bundle exec rake integration; open coverage/index.html