lhm 1.1.0 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.travis.yml +1 -0
- data/CHANGELOG.md +6 -0
- data/LICENSE +1 -1
- data/README.md +52 -16
- data/Rakefile +0 -1
- data/bin/lhm-spec-clobber.sh +3 -3
- data/gemfiles/dm_mysql.gemfile +5 -0
- data/lhm.gemspec +0 -2
- data/lib/lhm.rb +16 -3
- data/lib/lhm/atomic_switcher.rb +4 -4
- data/lib/lhm/chunker.rb +2 -2
- data/lib/lhm/command.rb +1 -1
- data/lib/lhm/connection.rb +143 -0
- data/lib/lhm/entangler.rb +5 -5
- data/lib/lhm/intersection.rb +1 -1
- data/lib/lhm/invoker.rb +1 -1
- data/lib/lhm/locked_switcher.rb +5 -4
- data/lib/lhm/migration.rb +1 -1
- data/lib/lhm/migrator.rb +5 -8
- data/lib/lhm/sql_helper.rb +14 -22
- data/lib/lhm/table.rb +29 -15
- data/lib/lhm/version.rb +2 -2
- data/spec/bootstrap.rb +1 -1
- data/spec/integration/atomic_switcher_spec.rb +3 -3
- data/spec/integration/chunker_spec.rb +1 -1
- data/spec/integration/entangler_spec.rb +1 -1
- data/spec/integration/integration_helper.rb +35 -18
- data/spec/integration/lhm_spec.rb +4 -1
- data/spec/integration/locked_switcher_spec.rb +1 -1
- data/spec/integration/table_spec.rb +1 -1
- data/spec/unit/active_record_connection_spec.rb +40 -0
- data/spec/unit/atomic_switcher_spec.rb +1 -1
- data/spec/unit/chunker_spec.rb +1 -1
- data/spec/unit/connection_spec.rb +11 -0
- data/spec/unit/datamapper_connection_spec.rb +49 -0
- data/spec/unit/entangler_spec.rb +1 -1
- data/spec/unit/intersection_spec.rb +1 -1
- data/spec/unit/locked_switcher_spec.rb +1 -1
- data/spec/unit/migration_spec.rb +1 -1
- data/spec/unit/migrator_spec.rb +1 -1
- data/spec/unit/sql_helper_spec.rb +1 -1
- data/spec/unit/table_spec.rb +1 -1
- data/spec/unit/unit_helper.rb +12 -1
- metadata +9 -43
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,9 @@
|
|
1
|
+
# 1.2.0 (February 22, 2013)
|
2
|
+
|
3
|
+
* Added DataMapper support, no API changes for current users. Refer to the
|
4
|
+
README for information.
|
5
|
+
* Documentation updates. Thanks @tiegz and @vinbarnes.
|
6
|
+
|
1
7
|
# 1.1.0 (April 29, 2012)
|
2
8
|
|
3
9
|
* Add option to specify custom index name
|
data/LICENSE
CHANGED
data/README.md
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Large Hadron Migrator [![Build Status](https://secure.travis-ci.org/soundcloud/large-hadron-migrator.png)][4]
|
1
|
+
# Large Hadron Migrator [![Build Status](https://secure.travis-ci.org/soundcloud/large-hadron-migrator.png?branch=master)][4]
|
2
2
|
|
3
3
|
Rails style database migrations are a useful way to evolve your data schema in
|
4
4
|
an agile manner. Most Rails projects start like this, and at first, making
|
@@ -22,7 +22,7 @@ is great if you are using this engine, but only solves half the problem.
|
|
22
22
|
At SoundCloud we started having migration pains quite a while ago, and after
|
23
23
|
looking around for third party solutions, we decided to create our
|
24
24
|
own. We called it Large Hadron Migrator, and it is a gem for online
|
25
|
-
ActiveRecord migrations.
|
25
|
+
ActiveRecord and DataMapper migrations.
|
26
26
|
|
27
27
|
![LHC](http://farm4.static.flickr.com/3093/2844971993_17f2ddf2a8_z.jpg)
|
28
28
|
|
@@ -35,18 +35,22 @@ without locking the table. In contrast to [OAK][0] and the
|
|
35
35
|
[facebook tool][1], we only use a copy table and triggers.
|
36
36
|
|
37
37
|
The Large Hadron is a test driven Ruby solution which can easily be dropped
|
38
|
-
into an ActiveRecord migration. It presumes a single auto
|
39
|
-
numerical primary key called id as per the Rails convention. Unlike
|
40
|
-
[twitter solution][2], it does not require the presence of an indexed
|
38
|
+
into an ActiveRecord or DataMapper migration. It presumes a single auto
|
39
|
+
incremented numerical primary key called id as per the Rails convention. Unlike
|
40
|
+
the [twitter solution][2], it does not require the presence of an indexed
|
41
41
|
`updated_at` column.
|
42
42
|
|
43
43
|
## Requirements
|
44
44
|
|
45
45
|
Lhm currently only works with MySQL databases and requires an established
|
46
|
-
ActiveRecord connection.
|
46
|
+
ActiveRecord or DataMapper connection.
|
47
47
|
|
48
48
|
It is compatible and [continuously tested][4] with Ruby 1.8.7 and Ruby 1.9.x,
|
49
|
-
ActiveRecord 2.3.x and 3.x
|
49
|
+
ActiveRecord 2.3.x and 3.x (mysql and mysql2 adapters), as well as DataMapper
|
50
|
+
1.2 (dm-mysql-adapter).
|
51
|
+
|
52
|
+
Lhm also works with dm-master-slave-adapter, it'll bind to the master before
|
53
|
+
running the migrations.
|
50
54
|
|
51
55
|
## Installation
|
52
56
|
|
@@ -66,6 +70,10 @@ ActiveRecord::Base.establish_connection(
|
|
66
70
|
:database => 'lhm'
|
67
71
|
)
|
68
72
|
|
73
|
+
# or with DataMapper
|
74
|
+
Lhm.setup(DataMapper.setup(:default, 'mysql://127.0.0.1/lhm'))
|
75
|
+
|
76
|
+
# and migrate
|
69
77
|
Lhm.change_table :users do |m|
|
70
78
|
m.add_column :arbitrary, "INT(12)"
|
71
79
|
m.add_index [:arbitrary_id, :created_at]
|
@@ -91,26 +99,54 @@ class MigrateUsers < ActiveRecord::Migration
|
|
91
99
|
def self.down
|
92
100
|
Lhm.change_table :users do |m|
|
93
101
|
m.remove_index [:arbitrary_id, :created_at]
|
94
|
-
m.remove_column :arbitrary
|
102
|
+
m.remove_column :arbitrary
|
103
|
+
end
|
104
|
+
end
|
105
|
+
end
|
106
|
+
```
|
107
|
+
|
108
|
+
Using dm-migrations, you'd define all your migrations as follows, and then call
|
109
|
+
`migrate_up!` or `migrate_down!` as normal.
|
110
|
+
|
111
|
+
```ruby
|
112
|
+
require 'dm-migrations/migration_runner'
|
113
|
+
require 'lhm'
|
114
|
+
|
115
|
+
migration 1, :migrate_users do
|
116
|
+
up do
|
117
|
+
Lhm.change_table :users do |m|
|
118
|
+
m.add_column :arbitrary, "INT(12)"
|
119
|
+
m.add_index [:arbitrary_id, :created_at]
|
120
|
+
m.ddl("alter table %s add column flag tinyint(1)" % m.name)
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
124
|
+
down do
|
125
|
+
Lhm.change_table :users do |m|
|
126
|
+
m.remove_index [:arbitrary_id, :created_at]
|
127
|
+
m.remove_column :arbitrary
|
95
128
|
end
|
96
129
|
end
|
97
130
|
end
|
98
131
|
```
|
99
132
|
|
133
|
+
**Note:** Lhm won't delete the old, leftover table. This is on purpose, in order
|
134
|
+
to prevent accidental data loss.
|
135
|
+
|
100
136
|
## Table rename strategies
|
101
137
|
|
102
138
|
There are two different table rename strategies available: LockedSwitcher and
|
103
139
|
AtomicSwitcher.
|
104
140
|
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
141
|
+
The LockedSwitcher strategy locks the table being migrated and issues two ALTER TABLE statements.
|
142
|
+
The AtomicSwitcher uses a single atomic RENAME TABLE query and is the favored solution.
|
143
|
+
|
144
|
+
Lhm chooses AtomicSwitcher if no strategy is specified, **unless** your version of MySQL is
|
145
|
+
affected by [binlog bug #39675](http://bugs.mysql.com/bug.php?id=39675). If your version is
|
146
|
+
affected, Lhm will raise an error if you don't specify a strategy. You're recommended
|
147
|
+
to use the LockedSwitcher in these cases to avoid replication issues.
|
111
148
|
|
112
|
-
|
113
|
-
but you can override the behavior with an option:
|
149
|
+
To specify the strategy in your migration:
|
114
150
|
|
115
151
|
```ruby
|
116
152
|
Lhm.change_table :users, :atomic_switch => true do |m|
|
data/Rakefile
CHANGED
data/bin/lhm-spec-clobber.sh
CHANGED
@@ -6,15 +6,15 @@ set -u
|
|
6
6
|
source ~/.lhm
|
7
7
|
|
8
8
|
lhmkill() {
|
9
|
-
|
10
|
-
|
9
|
+
echo killing lhm-cluster
|
10
|
+
ps -ef | sed -n "/[m]ysqld.*lhm-cluster/p" | awk '{ print $2 }' | xargs kill
|
11
|
+
sleep 2
|
11
12
|
}
|
12
13
|
|
13
14
|
echo stopping other running mysql instance
|
14
15
|
launchctl remove com.mysql.mysqld || { echo launchctl did not remove mysqld; }
|
15
16
|
"$mysqldir"/bin/mysqladmin shutdown || { echo mysqladmin did not shut down anything; }
|
16
17
|
|
17
|
-
echo killing lhm-cluster
|
18
18
|
lhmkill
|
19
19
|
|
20
20
|
echo removing $basedir
|
data/lhm.gemspec
CHANGED
data/lib/lhm.rb
CHANGED
@@ -1,9 +1,9 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
|
-
require 'active_record'
|
5
4
|
require 'lhm/table'
|
6
5
|
require 'lhm/invoker'
|
6
|
+
require 'lhm/connection'
|
7
7
|
require 'lhm/version'
|
8
8
|
|
9
9
|
# Large hadron migrator - online schema change tool
|
@@ -34,7 +34,8 @@ module Lhm
|
|
34
34
|
# @return [Boolean] Returns true if the migration finishes
|
35
35
|
# @raise [Error] Raises Lhm::Error in case of a error and aborts the migration
|
36
36
|
def self.change_table(table_name, options = {}, &block)
|
37
|
-
connection =
|
37
|
+
connection = Connection.new(adapter)
|
38
|
+
|
38
39
|
origin = Table.parse(table_name, connection)
|
39
40
|
invoker = Invoker.new(origin, connection)
|
40
41
|
block.call(invoker.migrator)
|
@@ -42,4 +43,16 @@ module Lhm
|
|
42
43
|
|
43
44
|
true
|
44
45
|
end
|
46
|
+
|
47
|
+
def self.setup(adapter)
|
48
|
+
@@adapter = adapter
|
49
|
+
end
|
50
|
+
|
51
|
+
def self.adapter
|
52
|
+
@@adapter ||=
|
53
|
+
begin
|
54
|
+
raise 'Please call Lhm.setup' unless defined?(ActiveRecord)
|
55
|
+
ActiveRecord::Base.connection
|
56
|
+
end
|
57
|
+
end
|
45
58
|
end
|
data/lib/lhm/atomic_switcher.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/command'
|
@@ -13,7 +13,6 @@ module Lhm
|
|
13
13
|
# Lhm::SqlHelper.supports_atomic_switch?.
|
14
14
|
class AtomicSwitcher
|
15
15
|
include Command
|
16
|
-
include SqlHelper
|
17
16
|
|
18
17
|
attr_reader :connection
|
19
18
|
|
@@ -36,14 +35,15 @@ module Lhm
|
|
36
35
|
end
|
37
36
|
|
38
37
|
def validate
|
39
|
-
unless
|
38
|
+
unless @connection.table_exists?(@origin.name) &&
|
39
|
+
@connection.table_exists?(@destination.name)
|
40
40
|
error "`#{ @origin.name }` and `#{ @destination.name }` must exist"
|
41
41
|
end
|
42
42
|
end
|
43
43
|
|
44
44
|
private
|
45
45
|
def execute
|
46
|
-
sql
|
46
|
+
@connection.sql(statements)
|
47
47
|
end
|
48
48
|
end
|
49
49
|
end
|
data/lib/lhm/chunker.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/command'
|
@@ -83,7 +83,7 @@ module Lhm
|
|
83
83
|
|
84
84
|
def execute
|
85
85
|
up_to do |lowest, highest|
|
86
|
-
affected_rows = update(copy(lowest, highest))
|
86
|
+
affected_rows = @connection.update(copy(lowest, highest))
|
87
87
|
|
88
88
|
if affected_rows > 0
|
89
89
|
sleep(throttle_seconds)
|
data/lib/lhm/command.rb
CHANGED
@@ -0,0 +1,143 @@
|
|
1
|
+
module Lhm
|
2
|
+
require 'lhm/sql_helper'
|
3
|
+
|
4
|
+
class Connection
|
5
|
+
def self.new(adapter)
|
6
|
+
if defined?(DataMapper) && adapter.is_a?(DataMapper::Adapters::AbstractAdapter)
|
7
|
+
DataMapperConnection.new(adapter)
|
8
|
+
elsif defined?(ActiveRecord)
|
9
|
+
ActiveRecordConnection.new(adapter)
|
10
|
+
else
|
11
|
+
raise 'Neither DataMapper nor ActiveRecord found.'
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
class DataMapperConnection
|
16
|
+
include SqlHelper
|
17
|
+
|
18
|
+
def initialize(adapter)
|
19
|
+
@adapter = adapter
|
20
|
+
@database_name = adapter.options['database'] || adapter.options['path'][1..-1]
|
21
|
+
end
|
22
|
+
|
23
|
+
def sql(statements)
|
24
|
+
[statements].flatten.each do |statement|
|
25
|
+
execute(tagged(statement))
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
def show_create(table_name)
|
30
|
+
sql = "show create table `#{ table_name }`"
|
31
|
+
select_values(sql).last
|
32
|
+
end
|
33
|
+
|
34
|
+
def current_database
|
35
|
+
@database_name
|
36
|
+
end
|
37
|
+
|
38
|
+
def update(statements)
|
39
|
+
[statements].flatten.inject(0) do |memo, statement|
|
40
|
+
result = @adapter.execute(tagged(statement))
|
41
|
+
memo += result.affected_rows
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def select_all(sql)
|
46
|
+
@adapter.select(sql).to_a
|
47
|
+
end
|
48
|
+
|
49
|
+
def select_one(sql)
|
50
|
+
select_all(sql).first
|
51
|
+
end
|
52
|
+
|
53
|
+
def select_values(sql)
|
54
|
+
select_one(sql).values
|
55
|
+
end
|
56
|
+
|
57
|
+
def select_value(sql)
|
58
|
+
select_one(sql)
|
59
|
+
end
|
60
|
+
|
61
|
+
def destination_create(origin)
|
62
|
+
original = %{CREATE TABLE "#{ origin.name }"}
|
63
|
+
replacement = %{CREATE TABLE "#{ origin.destination_name }"}
|
64
|
+
|
65
|
+
sql(origin.ddl.gsub(original, replacement))
|
66
|
+
end
|
67
|
+
|
68
|
+
def execute(sql)
|
69
|
+
@adapter.execute(sql)
|
70
|
+
end
|
71
|
+
|
72
|
+
def table_exists?(table_name)
|
73
|
+
!!select_one(%Q{
|
74
|
+
select *
|
75
|
+
from information_schema.tables
|
76
|
+
where table_schema = '#{ @database_name }'
|
77
|
+
and table_name = '#{ table_name }'
|
78
|
+
})
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
class ActiveRecordConnection
|
83
|
+
include SqlHelper
|
84
|
+
|
85
|
+
def initialize(adapter)
|
86
|
+
@adapter = adapter
|
87
|
+
@database_name = @adapter.current_database
|
88
|
+
end
|
89
|
+
|
90
|
+
def sql(statements)
|
91
|
+
[statements].flatten.each do |statement|
|
92
|
+
execute(tagged(statement))
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
def show_create(table_name)
|
97
|
+
sql = "show create table `#{ table_name }`"
|
98
|
+
specification = nil
|
99
|
+
execute(sql).each { |row| specification = row.last }
|
100
|
+
specification
|
101
|
+
end
|
102
|
+
|
103
|
+
def current_database
|
104
|
+
@database_name
|
105
|
+
end
|
106
|
+
|
107
|
+
def update(sql)
|
108
|
+
@adapter.update(sql)
|
109
|
+
end
|
110
|
+
|
111
|
+
def select_all(sql)
|
112
|
+
@adapter.select_all(sql)
|
113
|
+
end
|
114
|
+
|
115
|
+
def select_one(sql)
|
116
|
+
@adapter.select_one(sql)
|
117
|
+
end
|
118
|
+
|
119
|
+
def select_values(sql)
|
120
|
+
@adapter.select_values(sql)
|
121
|
+
end
|
122
|
+
|
123
|
+
def select_value(sql)
|
124
|
+
@adapter.select_value(sql)
|
125
|
+
end
|
126
|
+
|
127
|
+
def destination_create(origin)
|
128
|
+
original = %{CREATE TABLE `#{ origin.name }`}
|
129
|
+
replacement = %{CREATE TABLE `#{ origin.destination_name }`}
|
130
|
+
|
131
|
+
sql(origin.ddl.gsub(original, replacement))
|
132
|
+
end
|
133
|
+
|
134
|
+
def execute(sql)
|
135
|
+
@adapter.execute(sql)
|
136
|
+
end
|
137
|
+
|
138
|
+
def table_exists?(table_name)
|
139
|
+
@adapter.table_exists?(table_name)
|
140
|
+
end
|
141
|
+
end
|
142
|
+
end
|
143
|
+
end
|
data/lib/lhm/entangler.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/command'
|
@@ -68,21 +68,21 @@ module Lhm
|
|
68
68
|
end
|
69
69
|
|
70
70
|
def validate
|
71
|
-
unless
|
71
|
+
unless @connection.table_exists?(@origin.name)
|
72
72
|
error("#{ @origin.name } does not exist")
|
73
73
|
end
|
74
74
|
|
75
|
-
unless
|
75
|
+
unless @connection.table_exists?(@destination.name)
|
76
76
|
error("#{ @destination.name } does not exist")
|
77
77
|
end
|
78
78
|
end
|
79
79
|
|
80
80
|
def before
|
81
|
-
sql(entangle)
|
81
|
+
@connection.sql(entangle)
|
82
82
|
end
|
83
83
|
|
84
84
|
def after
|
85
|
-
sql(untangle)
|
85
|
+
@connection.sql(untangle)
|
86
86
|
end
|
87
87
|
|
88
88
|
def revert
|
data/lib/lhm/intersection.rb
CHANGED
data/lib/lhm/invoker.rb
CHANGED
data/lib/lhm/locked_switcher.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/command'
|
@@ -53,7 +53,8 @@ module Lhm
|
|
53
53
|
end
|
54
54
|
|
55
55
|
def validate
|
56
|
-
unless
|
56
|
+
unless @connection.table_exists?(@origin.name) &&
|
57
|
+
@connection.table_exists?(@destination.name)
|
57
58
|
error "`#{ @origin.name }` and `#{ @destination.name }` must exist"
|
58
59
|
end
|
59
60
|
end
|
@@ -61,11 +62,11 @@ module Lhm
|
|
61
62
|
private
|
62
63
|
|
63
64
|
def revert
|
64
|
-
sql
|
65
|
+
@connection.sql("unlock tables")
|
65
66
|
end
|
66
67
|
|
67
68
|
def execute
|
68
|
-
sql
|
69
|
+
@connection.sql(statements)
|
69
70
|
end
|
70
71
|
end
|
71
72
|
end
|
data/lib/lhm/migration.rb
CHANGED
data/lib/lhm/migrator.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/command'
|
@@ -142,7 +142,7 @@ module Lhm
|
|
142
142
|
private
|
143
143
|
|
144
144
|
def validate
|
145
|
-
unless
|
145
|
+
unless @connection.table_exists?(@origin.name)
|
146
146
|
error("could not find origin table #{ @origin.name }")
|
147
147
|
end
|
148
148
|
|
@@ -152,22 +152,19 @@ module Lhm
|
|
152
152
|
|
153
153
|
dest = @origin.destination_name
|
154
154
|
|
155
|
-
if
|
155
|
+
if @connection.table_exists?(dest)
|
156
156
|
error("#{ dest } should not exist; not cleaned up from previous run?")
|
157
157
|
end
|
158
158
|
end
|
159
159
|
|
160
160
|
def execute
|
161
161
|
destination_create
|
162
|
-
sql(@statements)
|
162
|
+
@connection.sql(@statements)
|
163
163
|
Migration.new(@origin, destination_read)
|
164
164
|
end
|
165
165
|
|
166
166
|
def destination_create
|
167
|
-
|
168
|
-
replacement = "CREATE TABLE `#{ @origin.destination_name }`"
|
169
|
-
|
170
|
-
sql(@origin.ddl.gsub(original, replacement))
|
167
|
+
@connection.destination_create(@origin)
|
171
168
|
end
|
172
169
|
|
173
170
|
def destination_read
|
data/lib/lhm/sql_helper.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
module Lhm
|
@@ -20,28 +20,10 @@ module Lhm
|
|
20
20
|
end.join(', ')
|
21
21
|
end
|
22
22
|
|
23
|
-
def table?(table_name)
|
24
|
-
connection.table_exists?(table_name)
|
25
|
-
end
|
26
|
-
|
27
|
-
def sql(statements)
|
28
|
-
[statements].flatten.each do |statement|
|
29
|
-
connection.execute(tagged(statement))
|
30
|
-
end
|
31
|
-
rescue ActiveRecord::StatementInvalid => e
|
32
|
-
error e.message
|
33
|
-
end
|
34
|
-
|
35
|
-
def update(statements)
|
36
|
-
[statements].flatten.inject(0) do |memo, statement|
|
37
|
-
memo += connection.update(tagged(statement))
|
38
|
-
end
|
39
|
-
rescue ActiveRecord::StatementInvalid => e
|
40
|
-
error e.message
|
41
|
-
end
|
42
|
-
|
43
23
|
def version_string
|
44
|
-
connection.select_one("show variables like 'version'")
|
24
|
+
row = connection.select_one("show variables like 'version'")
|
25
|
+
value = struct_key(row, "Value")
|
26
|
+
row[value]
|
45
27
|
end
|
46
28
|
|
47
29
|
private
|
@@ -81,5 +63,15 @@ module Lhm
|
|
81
63
|
end
|
82
64
|
return true
|
83
65
|
end
|
66
|
+
|
67
|
+
def struct_key(struct, key)
|
68
|
+
keys = if struct.is_a? Hash
|
69
|
+
struct.keys
|
70
|
+
else
|
71
|
+
struct.members
|
72
|
+
end
|
73
|
+
|
74
|
+
keys.find {|k| k.to_s.downcase == key.to_s.downcase }
|
75
|
+
end
|
84
76
|
end
|
85
77
|
end
|
data/lib/lhm/table.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require 'lhm/sql_helper'
|
@@ -37,10 +37,7 @@ module Lhm
|
|
37
37
|
end
|
38
38
|
|
39
39
|
def ddl
|
40
|
-
|
41
|
-
specification = nil
|
42
|
-
@connection.execute(sql).each { |row| specification = row.last }
|
43
|
-
specification
|
40
|
+
@connection.show_create(@table_name)
|
44
41
|
end
|
45
42
|
|
46
43
|
def parse
|
@@ -48,10 +45,14 @@ module Lhm
|
|
48
45
|
|
49
46
|
Table.new(@table_name, extract_primary_key(schema), ddl).tap do |table|
|
50
47
|
schema.each do |defn|
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
48
|
+
column_name = struct_key(defn, "COLUMN_NAME")
|
49
|
+
column_type = struct_key(defn, "COLUMN_TYPE")
|
50
|
+
is_nullable = struct_key(defn, "IS_NULLABLE")
|
51
|
+
column_default = struct_key(defn, "COLUMN_DEFAULT")
|
52
|
+
table.columns[defn[column_name]] = {
|
53
|
+
:type => defn[column_type],
|
54
|
+
:is_nullable => defn[is_nullable],
|
55
|
+
:column_default => defn[column_default]
|
55
56
|
}
|
56
57
|
end
|
57
58
|
|
@@ -67,20 +68,25 @@ module Lhm
|
|
67
68
|
@connection.select_all %Q{
|
68
69
|
select *
|
69
70
|
from information_schema.columns
|
70
|
-
where table_name =
|
71
|
-
and table_schema =
|
71
|
+
where table_name = '#{ @table_name }'
|
72
|
+
and table_schema = '#{ @schema_name }'
|
72
73
|
}
|
73
74
|
end
|
74
75
|
|
75
76
|
def read_indices
|
76
77
|
@connection.select_all %Q{
|
77
78
|
show indexes from `#{ @schema_name }`.`#{ @table_name }`
|
78
|
-
where key_name !=
|
79
|
+
where key_name != 'PRIMARY'
|
79
80
|
}
|
80
81
|
end
|
81
82
|
|
82
83
|
def extract_indices(indices)
|
83
|
-
indices.
|
84
|
+
indices.
|
85
|
+
map do |row|
|
86
|
+
key_name = struct_key(row, "Key_name")
|
87
|
+
column_name = struct_key(row, "COLUMN_NAME")
|
88
|
+
[row[key_name], row[column_name]]
|
89
|
+
end.
|
84
90
|
inject(Hash.new { |h, k| h[k] = []}) do |memo, (idx, column)|
|
85
91
|
memo[idx] << column
|
86
92
|
memo
|
@@ -88,8 +94,16 @@ module Lhm
|
|
88
94
|
end
|
89
95
|
|
90
96
|
def extract_primary_key(schema)
|
91
|
-
cols = schema.select
|
92
|
-
|
97
|
+
cols = schema.select do |defn|
|
98
|
+
column_key = struct_key(defn, "COLUMN_KEY")
|
99
|
+
defn[column_key] == "PRI"
|
100
|
+
end
|
101
|
+
|
102
|
+
keys = cols.map do |defn|
|
103
|
+
column_name = struct_key(defn, "COLUMN_NAME")
|
104
|
+
defn[column_name]
|
105
|
+
end
|
106
|
+
|
93
107
|
keys.length == 1 ? keys.first : keys
|
94
108
|
end
|
95
109
|
end
|
data/lib/lhm/version.rb
CHANGED
data/spec/bootstrap.rb
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require File.expand_path(File.dirname(__FILE__)) + '/integration_helper'
|
@@ -14,9 +14,9 @@ describe Lhm::AtomicSwitcher do
|
|
14
14
|
|
15
15
|
describe "switching" do
|
16
16
|
before(:each) do
|
17
|
-
@origin
|
17
|
+
@origin = table_create("origin")
|
18
18
|
@destination = table_create("destination")
|
19
|
-
@migration
|
19
|
+
@migration = Lhm::Migration.new(@origin, @destination)
|
20
20
|
end
|
21
21
|
|
22
22
|
it "rename origin to archive" do
|
@@ -1,24 +1,29 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require File.expand_path(File.dirname(__FILE__)) + "/../bootstrap"
|
5
5
|
|
6
|
-
require 'active_record'
|
7
6
|
begin
|
8
|
-
require '
|
7
|
+
require 'active_record'
|
8
|
+
begin
|
9
|
+
require 'mysql2'
|
10
|
+
rescue LoadError
|
11
|
+
require 'mysql'
|
12
|
+
end
|
9
13
|
rescue LoadError
|
10
|
-
require '
|
14
|
+
require 'dm-core'
|
15
|
+
require 'dm-mysql-adapter'
|
11
16
|
end
|
12
17
|
require 'lhm/table'
|
13
18
|
require 'lhm/sql_helper'
|
19
|
+
require 'lhm/connection'
|
14
20
|
|
15
21
|
module IntegrationHelper
|
16
22
|
#
|
17
23
|
# Connectivity
|
18
24
|
#
|
19
|
-
|
20
25
|
def connection
|
21
|
-
|
26
|
+
@connection
|
22
27
|
end
|
23
28
|
|
24
29
|
def connect_master!
|
@@ -30,28 +35,37 @@ module IntegrationHelper
|
|
30
35
|
end
|
31
36
|
|
32
37
|
def connect!(port)
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
38
|
+
adapter = nil
|
39
|
+
if defined?(ActiveRecord)
|
40
|
+
ActiveRecord::Base.establish_connection(
|
41
|
+
:adapter => defined?(Mysql2) ? 'mysql2' : 'mysql',
|
42
|
+
:host => '127.0.0.1',
|
43
|
+
:database => 'lhm',
|
44
|
+
:username => 'root',
|
45
|
+
:port => port
|
46
|
+
)
|
47
|
+
adapter = ActiveRecord::Base.connection
|
48
|
+
elsif defined?(DataMapper)
|
49
|
+
adapter = DataMapper.setup(:default, "mysql://root@localhost:#{port}/lhm")
|
50
|
+
end
|
51
|
+
|
52
|
+
Lhm.setup(adapter)
|
53
|
+
@connection = Lhm::Connection.new(adapter)
|
40
54
|
end
|
41
55
|
|
42
56
|
def select_one(*args)
|
43
|
-
connection.select_one(*args)
|
57
|
+
@connection.select_one(*args)
|
44
58
|
end
|
45
59
|
|
46
60
|
def select_value(*args)
|
47
|
-
connection.select_value(*args)
|
61
|
+
@connection.select_value(*args)
|
48
62
|
end
|
49
63
|
|
50
64
|
def execute(*args)
|
51
65
|
retries = 10
|
52
66
|
begin
|
53
|
-
connection.execute(*args)
|
54
|
-
rescue
|
67
|
+
@connection.execute(*args)
|
68
|
+
rescue => e
|
55
69
|
if (retries -= 1) > 0 && e.message =~ /Table '.*?' doesn't exist/
|
56
70
|
sleep 0.1
|
57
71
|
retry
|
@@ -69,8 +83,11 @@ module IntegrationHelper
|
|
69
83
|
# check the master binlog position and wait for the slave to catch up
|
70
84
|
# to that position.
|
71
85
|
sleep 1
|
86
|
+
elsif
|
87
|
+
connect_master!
|
72
88
|
end
|
73
89
|
|
90
|
+
|
74
91
|
yield block
|
75
92
|
|
76
93
|
if master_slave_mode?
|
@@ -93,7 +110,7 @@ module IntegrationHelper
|
|
93
110
|
end
|
94
111
|
|
95
112
|
def table_read(fixture_name)
|
96
|
-
Lhm::Table.parse(fixture_name, connection)
|
113
|
+
Lhm::Table.parse(fixture_name, @connection)
|
97
114
|
end
|
98
115
|
|
99
116
|
def table_exists?(table)
|
@@ -1,4 +1,4 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require File.expand_path(File.dirname(__FILE__)) + '/integration_helper'
|
@@ -161,10 +161,12 @@ describe Lhm do
|
|
161
161
|
|
162
162
|
insert = Thread.new do
|
163
163
|
10.times do |n|
|
164
|
+
connect_master!
|
164
165
|
execute("insert into users set reference = '#{ 100 + n }'")
|
165
166
|
sleep(0.17)
|
166
167
|
end
|
167
168
|
end
|
169
|
+
sleep 2
|
168
170
|
|
169
171
|
options = { :stride => 10, :throttle => 97, :atomic_switch => false }
|
170
172
|
Lhm.change_table(:users, options) do |t|
|
@@ -187,6 +189,7 @@ describe Lhm do
|
|
187
189
|
sleep(0.17)
|
188
190
|
end
|
189
191
|
end
|
192
|
+
sleep 2
|
190
193
|
|
191
194
|
options = { :stride => 10, :throttle => 97, :atomic_switch => false }
|
192
195
|
Lhm.change_table(:users, options) do |t|
|
@@ -0,0 +1,40 @@
|
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd.
|
2
|
+
|
3
|
+
require File.expand_path(File.dirname(__FILE__)) + '/unit_helper'
|
4
|
+
require 'lhm/connection'
|
5
|
+
|
6
|
+
if defined?(ActiveRecord)
|
7
|
+
describe Lhm::Connection::ActiveRecordConnection do
|
8
|
+
let(:active_record) { MiniTest::Mock.new }
|
9
|
+
|
10
|
+
before do
|
11
|
+
active_record.expect :current_database, 'the db'
|
12
|
+
end
|
13
|
+
|
14
|
+
after do
|
15
|
+
active_record.verify
|
16
|
+
end
|
17
|
+
|
18
|
+
it 'creates an ActiveRecord connection when the DM classes are not there' do
|
19
|
+
connection.must_be_instance_of(Lhm::Connection::ActiveRecordConnection)
|
20
|
+
end
|
21
|
+
|
22
|
+
it 'initializes the db name from the connection' do
|
23
|
+
connection.current_database.must_equal('the db')
|
24
|
+
end
|
25
|
+
|
26
|
+
it 'backticks the table names' do
|
27
|
+
table_name = 'my_table'
|
28
|
+
|
29
|
+
active_record.expect :execute,
|
30
|
+
[['returned sql']],
|
31
|
+
["show create table `#{table_name}`"]
|
32
|
+
|
33
|
+
connection.show_create(table_name)
|
34
|
+
end
|
35
|
+
|
36
|
+
def connection
|
37
|
+
Lhm::Connection.new(active_record)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
data/spec/unit/chunker_spec.rb
CHANGED
@@ -0,0 +1,49 @@
|
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd.
|
2
|
+
|
3
|
+
require File.expand_path(File.dirname(__FILE__)) + '/unit_helper'
|
4
|
+
require 'lhm/connection'
|
5
|
+
|
6
|
+
if defined?(DataMapper)
|
7
|
+
describe Lhm::Connection::DataMapperConnection do
|
8
|
+
let(:data_mapper) { MiniTest::Mock.new }
|
9
|
+
let(:options) { { 'database' => 'the db' } }
|
10
|
+
|
11
|
+
before do
|
12
|
+
data_mapper.expect :is_a?, true, [DataMapper::Adapters::AbstractAdapter]
|
13
|
+
data_mapper.expect :options, options
|
14
|
+
end
|
15
|
+
|
16
|
+
after do
|
17
|
+
data_mapper.verify
|
18
|
+
end
|
19
|
+
|
20
|
+
it 'creates a DataMapperConnection when the adapter is from DM' do
|
21
|
+
connection.must_be_instance_of(Lhm::Connection::DataMapperConnection)
|
22
|
+
end
|
23
|
+
|
24
|
+
it 'initializes the db name from the database option' do
|
25
|
+
connection.current_database.must_equal('the db')
|
26
|
+
end
|
27
|
+
|
28
|
+
it 'initializes the db name form the path if the database option is not available' do
|
29
|
+
options['database'] = nil
|
30
|
+
options['path'] = '/still the db'
|
31
|
+
|
32
|
+
connection.current_database.must_equal('still the db')
|
33
|
+
end
|
34
|
+
|
35
|
+
it 'backticks the table names' do
|
36
|
+
table_name = 'my_table'
|
37
|
+
|
38
|
+
data_mapper.expect :select,
|
39
|
+
[{ :sql => 'returned sql' }],
|
40
|
+
["show create table `#{table_name}`"]
|
41
|
+
|
42
|
+
connection.show_create(table_name)
|
43
|
+
end
|
44
|
+
|
45
|
+
def connection
|
46
|
+
Lhm::Connection.new(data_mapper)
|
47
|
+
end
|
48
|
+
end
|
49
|
+
end
|
data/spec/unit/entangler_spec.rb
CHANGED
data/spec/unit/migration_spec.rb
CHANGED
data/spec/unit/migrator_spec.rb
CHANGED
data/spec/unit/table_spec.rb
CHANGED
data/spec/unit/unit_helper.rb
CHANGED
@@ -1,8 +1,19 @@
|
|
1
|
-
# Copyright (c) 2011, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
1
|
+
# Copyright (c) 2011 - 2013, SoundCloud Ltd., Rany Keddo, Tobias Bielohlawek, Tobias
|
2
2
|
# Schmidt
|
3
3
|
|
4
4
|
require File.expand_path(File.dirname(__FILE__)) + "/../bootstrap"
|
5
5
|
|
6
|
+
begin
|
7
|
+
require 'active_record'
|
8
|
+
begin
|
9
|
+
require 'mysql2'
|
10
|
+
rescue LoadError
|
11
|
+
require 'mysql'
|
12
|
+
end
|
13
|
+
rescue LoadError
|
14
|
+
require 'dm-core'
|
15
|
+
end
|
16
|
+
|
6
17
|
module UnitHelper
|
7
18
|
def fixture(name)
|
8
19
|
File.read $fixtures.join(name)
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: lhm
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.2.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -12,7 +12,7 @@ authors:
|
|
12
12
|
autorequire:
|
13
13
|
bindir: bin
|
14
14
|
cert_chain: []
|
15
|
-
date:
|
15
|
+
date: 2013-02-22 00:00:00.000000000 Z
|
16
16
|
dependencies:
|
17
17
|
- !ruby/object:Gem::Dependency
|
18
18
|
name: minitest
|
@@ -46,22 +46,6 @@ dependencies:
|
|
46
46
|
- - ! '>='
|
47
47
|
- !ruby/object:Gem::Version
|
48
48
|
version: '0'
|
49
|
-
- !ruby/object:Gem::Dependency
|
50
|
-
name: activerecord
|
51
|
-
requirement: !ruby/object:Gem::Requirement
|
52
|
-
none: false
|
53
|
-
requirements:
|
54
|
-
- - ! '>='
|
55
|
-
- !ruby/object:Gem::Version
|
56
|
-
version: '0'
|
57
|
-
type: :runtime
|
58
|
-
prerelease: false
|
59
|
-
version_requirements: !ruby/object:Gem::Requirement
|
60
|
-
none: false
|
61
|
-
requirements:
|
62
|
-
- - ! '>='
|
63
|
-
- !ruby/object:Gem::Version
|
64
|
-
version: '0'
|
65
49
|
description: Migrate large tables without downtime by copying to a temporary table
|
66
50
|
in chunks. The old table is not dropped. Instead, it is moved to timestamp_table_name
|
67
51
|
for verification.
|
@@ -85,11 +69,13 @@ files:
|
|
85
69
|
- gemfiles/ar-2.3_mysql.gemfile
|
86
70
|
- gemfiles/ar-3.2_mysql.gemfile
|
87
71
|
- gemfiles/ar-3.2_mysql2.gemfile
|
72
|
+
- gemfiles/dm_mysql.gemfile
|
88
73
|
- lhm.gemspec
|
89
74
|
- lib/lhm.rb
|
90
75
|
- lib/lhm/atomic_switcher.rb
|
91
76
|
- lib/lhm/chunker.rb
|
92
77
|
- lib/lhm/command.rb
|
78
|
+
- lib/lhm/connection.rb
|
93
79
|
- lib/lhm/entangler.rb
|
94
80
|
- lib/lhm/intersection.rb
|
95
81
|
- lib/lhm/invoker.rb
|
@@ -113,8 +99,11 @@ files:
|
|
113
99
|
- spec/integration/lhm_spec.rb
|
114
100
|
- spec/integration/locked_switcher_spec.rb
|
115
101
|
- spec/integration/table_spec.rb
|
102
|
+
- spec/unit/active_record_connection_spec.rb
|
116
103
|
- spec/unit/atomic_switcher_spec.rb
|
117
104
|
- spec/unit/chunker_spec.rb
|
105
|
+
- spec/unit/connection_spec.rb
|
106
|
+
- spec/unit/datamapper_connection_spec.rb
|
118
107
|
- spec/unit/entangler_spec.rb
|
119
108
|
- spec/unit/intersection_spec.rb
|
120
109
|
- spec/unit/locked_switcher_spec.rb
|
@@ -143,31 +132,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
143
132
|
version: '0'
|
144
133
|
requirements: []
|
145
134
|
rubyforge_project:
|
146
|
-
rubygems_version: 1.8.
|
135
|
+
rubygems_version: 1.8.24
|
147
136
|
signing_key:
|
148
137
|
specification_version: 3
|
149
138
|
summary: online schema changer for mysql
|
150
|
-
test_files:
|
151
|
-
- spec/README.md
|
152
|
-
- spec/bootstrap.rb
|
153
|
-
- spec/fixtures/destination.ddl
|
154
|
-
- spec/fixtures/origin.ddl
|
155
|
-
- spec/fixtures/small_table.ddl
|
156
|
-
- spec/fixtures/users.ddl
|
157
|
-
- spec/integration/atomic_switcher_spec.rb
|
158
|
-
- spec/integration/chunker_spec.rb
|
159
|
-
- spec/integration/entangler_spec.rb
|
160
|
-
- spec/integration/integration_helper.rb
|
161
|
-
- spec/integration/lhm_spec.rb
|
162
|
-
- spec/integration/locked_switcher_spec.rb
|
163
|
-
- spec/integration/table_spec.rb
|
164
|
-
- spec/unit/atomic_switcher_spec.rb
|
165
|
-
- spec/unit/chunker_spec.rb
|
166
|
-
- spec/unit/entangler_spec.rb
|
167
|
-
- spec/unit/intersection_spec.rb
|
168
|
-
- spec/unit/locked_switcher_spec.rb
|
169
|
-
- spec/unit/migration_spec.rb
|
170
|
-
- spec/unit/migrator_spec.rb
|
171
|
-
- spec/unit/sql_helper_spec.rb
|
172
|
-
- spec/unit/table_spec.rb
|
173
|
-
- spec/unit/unit_helper.rb
|
139
|
+
test_files: []
|