ar_mysql_flexmaster 1.0.2 → 1.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +9 -10
- data/README.md +60 -54
- data/Rakefile +1 -1
- data/ar_mysql_flexmaster.gemspec +11 -8
- data/bin/master_cut +6 -9
- data/gemfiles/rails3.2.gemfile.lock +42 -42
- data/gemfiles/rails4.2.gemfile +2 -2
- data/gemfiles/rails4.2.gemfile.lock +65 -64
- data/gemfiles/rails5.0.gemfile +2 -2
- data/gemfiles/rails5.0.gemfile.lock +76 -123
- data/lib/active_record/connection_adapters/mysql_flexmaster_adapter.rb +12 -10
- data/test/ar_flexmaster_test.rb +10 -7
- data/test/boot_mysql_env.rb +5 -5
- data/test/boot_slave +4 -4
- data/test/integration/no_traffic_test.rb +0 -1
- data/test/integration/with_queries_to_be_killed_test.rb +1 -1
- data/test/integration/wrong_setup_test.rb +1 -1
- data/test/integration_helper.rb +1 -2
- metadata +32 -15
- data/gemfiles/rails4.0.gemfile +0 -6
- data/gemfiles/rails4.0.gemfile.lock +0 -104
- data/gemfiles/rails4.1.gemfile +0 -6
- data/gemfiles/rails4.1.gemfile.lock +0 -109
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 003ae600df9f49fa3ed61a3f24b50be9a1c1578f
|
4
|
+
data.tar.gz: 964eea239d02d3ac8984f690e73b0d9dcda4f4a8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 54d905d3643055ddd23984cb162519c7dac4f06613b63f2e0bee48e0f4b742983039f543244af8a425582b3a37fa7a65b6c163df586e802ab3ba13405e83f34e
|
7
|
+
data.tar.gz: 5d10d47653118fb716cf3af3146270e96c403eaf9041ccb1dec81dea699e481d3244eb3ef85917e4ebc6989acf3d8f13968881f382dcedaa236551c7c4ceccf5
|
data/.travis.yml
CHANGED
@@ -1,26 +1,25 @@
|
|
1
1
|
language: ruby
|
2
2
|
|
3
3
|
rvm:
|
4
|
-
- 2.
|
5
|
-
- 2.
|
6
|
-
- 2.
|
4
|
+
- 2.2.6
|
5
|
+
- 2.3.3
|
6
|
+
- 2.4.1
|
7
7
|
|
8
8
|
gemfile:
|
9
9
|
- gemfiles/rails3.2.gemfile
|
10
|
-
- gemfiles/rails4.0.gemfile
|
11
|
-
- gemfiles/rails4.1.gemfile
|
12
10
|
- gemfiles/rails4.2.gemfile
|
13
11
|
- gemfiles/rails5.0.gemfile
|
14
12
|
|
15
|
-
sudo:
|
13
|
+
sudo: required
|
16
14
|
|
17
15
|
bundler_args: --no-deployment
|
18
16
|
|
17
|
+
before_script:
|
18
|
+
- sudo cp /usr/share/doc/mysql-server-5.6/examples/my-default.cnf /usr/share/mysql/my-default.cnf
|
19
|
+
|
19
20
|
script: bundle exec rake test
|
20
21
|
|
21
22
|
matrix:
|
22
23
|
exclude:
|
23
|
-
- rvm: 2.
|
24
|
-
gemfile: gemfiles/
|
25
|
-
- rvm: 2.1.8
|
26
|
-
gemfile: gemfiles/rails5.0.gemfile
|
24
|
+
- rvm: 2.4.1
|
25
|
+
gemfile: gemfiles/rails3.2.gemfile
|
data/README.md
CHANGED
@@ -1,19 +1,20 @@
|
|
1
|
-
[![Build Status](https://travis-ci.org/
|
1
|
+
[![Build Status](https://travis-ci.org/zendesk/ar_mysql_flexmaster.svg?branch=master)](https://travis-ci.org/zendesk/ar_mysql_flexmaster)
|
2
2
|
|
3
3
|
# Flexmaster
|
4
4
|
|
5
|
-
Flexmaster is an adapter for ActiveRecord and
|
6
|
-
among a list of potential masters at runtime.
|
7
|
-
HA solution (load balancing, middleware)
|
5
|
+
Flexmaster is an adapter for ActiveRecord and MySQL that allows an application
|
6
|
+
node to find a master among a list of potential masters at runtime. It trades
|
7
|
+
some properties of a more traditional HA solution (load balancing, middleware)
|
8
|
+
for simplicity of operation.
|
8
9
|
|
9
10
|
## Configuration:
|
10
11
|
|
11
|
-
Your environment should be configured with 1 active master and N replicas.
|
12
|
-
global
|
13
|
-
depends on it).
|
12
|
+
Your environment should be configured with 1 active master and N replicas. Each
|
13
|
+
replica should have MySQL’s global `READ_ONLY` flag set to true (this is really
|
14
|
+
best practices for your replicas anyway, but Flexmaster depends on it).
|
14
15
|
|
15
|
-
database.yml should contain a list of hosts
|
16
|
-
It should look like this:
|
16
|
+
database.yml should contain a list of hosts – all of them potential masters, all
|
17
|
+
of them potential replicas. It should look like this:
|
17
18
|
|
18
19
|
```
|
19
20
|
production:
|
@@ -28,79 +29,86 @@ production_slave:
|
|
28
29
|
hosts: ["db01:3306", "db02:3306"]
|
29
30
|
```
|
30
31
|
|
31
|
-
In this example, we
|
32
|
-
|
32
|
+
In this example, we’ve configured two different connections for Rails to use.
|
33
|
+
Note that they’re identical except for the `slave: true` key in the
|
34
|
+
`production_slave` YAML block. Adding `slave: true` indicates to Flexmaster that
|
35
|
+
this connection should prefer a read-only slave wherever possible.
|
33
36
|
|
34
37
|
## How it works
|
35
38
|
|
36
39
|
### Overview
|
37
40
|
|
38
|
-
The
|
39
|
-
of the replication chain may be read-write at any given
|
40
|
-
on the
|
41
|
+
The MySQL `READ_ONLY` flag is used to indicate a current master amongst the
|
42
|
+
cluster. Only one member of the replication chain may be read-write at any given
|
43
|
+
time. The application picks in run time, based on the `READ_ONLY` flag, which
|
44
|
+
host is correct.
|
41
45
|
|
42
|
-
###
|
46
|
+
### Boot time
|
43
47
|
|
44
|
-
Your
|
45
|
-
it finds the correct host.
|
48
|
+
Your ActiveRecord application will pick a correct MySQL host for the given
|
49
|
+
configuration by probing hosts until it finds the correct host.
|
46
50
|
|
47
|
-
For master configurations (slave: true is not specified):
|
51
|
+
For master configurations (`slave: true` is not specified):
|
48
52
|
|
49
|
-
The application will probe each host in turn, and find the
|
50
|
-
that is read-write (SET GLOBAL
|
53
|
+
The application will probe each host in turn, and find the MySQL candidate among
|
54
|
+
these nodes that is read-write (`SET GLOBAL READ_ONLY=0`).
|
51
55
|
|
52
|
-
If it finds more than one node where READ_ONLY == 0
|
56
|
+
If it finds more than one node where `READ_ONLY == 0`, it will abort.
|
53
57
|
|
54
|
-
For slave configurations (slave: true specified):
|
58
|
+
For slave configurations (`slave: true` specified):
|
55
59
|
|
56
|
-
The application will choose a replica at random from amongst those where
|
57
|
-
it will fall back to the
|
60
|
+
The application will choose a replica at random from amongst those where
|
61
|
+
`READ_ONLY == 1`. If no active replicas are found, it will fall back to the
|
62
|
+
master.
|
58
63
|
|
59
|
-
###
|
64
|
+
### Run time
|
60
65
|
|
61
|
-
Before each transaction is opened on the master, the application checks the
|
62
|
-
If
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
+
Before each transaction is opened on the master, the application checks the
|
67
|
+
status of the `READ_ONLY` variable. If `READ_ONLY == 0` (our active connection
|
68
|
+
is still to the current master), it will proceed with the transaction as normal.
|
69
|
+
If `READ_ONLY == 1` (the master has been demoted), it will drop the current
|
70
|
+
connection and re-poll the cluster, sleeping for up to a default of 5 seconds
|
71
|
+
for a new master to be promoted. When it finds the new master, it will continue
|
72
|
+
playing the transaction on it.
|
66
73
|
|
67
|
-
###
|
74
|
+
### Promoting a new master
|
68
75
|
|
69
|
-
*The bin/master_cut script in this project will perform steps 3
|
76
|
+
*The `bin/master_cut` script in this project will perform steps 3–5 for you.*
|
70
77
|
|
71
78
|
The process of promoting a new master to head the cluster should be as follows:
|
72
79
|
|
73
|
-
1.
|
74
|
-
1.
|
75
|
-
chain to look like this:
|
80
|
+
1. Identify a new candidate master.
|
81
|
+
1. Ensure that all other replicas in the cluster are chained off the candidate
|
82
|
+
master; you want the chain to look like this:
|
76
83
|
|
77
84
|
```
|
78
85
|
<existing master> -> <candidate master> -> <other replicas>
|
79
|
-
-> <other replicas>
|
80
|
-
|
81
|
-
```
|
82
|
-
|
83
|
-
1. set the old master to READ_ONLY = 1
|
84
|
-
1. record the master-bin-log position of the candidate master (if you want to re-use the old master as a replica)
|
85
|
-
1. set the new master to READ_ONLY = 0
|
86
|
+
-> <other replicas>
|
86
87
|
|
87
|
-
|
88
|
-
|
88
|
+
```
|
89
|
+
1. Set the old master to `READ_ONLY = 1`.
|
90
|
+
1. Record the master-bin-log position of the candidate master (if you want to
|
91
|
+
re-use the old master as a replica).
|
92
|
+
1. Set the new master to `READ_ONLY = 0`.
|
89
93
|
|
90
|
-
The application will
|
94
|
+
The application nodes will, in time, find that the old master is inactive and
|
95
|
+
will move their connections to the new master.
|
91
96
|
|
92
|
-
|
97
|
+
The application will also eventually shift slave traffic to another node in the
|
98
|
+
cluster.
|
93
99
|
|
94
|
-
|
95
|
-
will crash. In theory there's a workaround for this problem, in pratice it's rather unwieldy due
|
96
|
-
to a lack of shared global variables in mysql.
|
100
|
+
### Caveats and gotchas
|
97
101
|
|
98
|
-
-
|
102
|
+
- Any explicit (`BEGIN` … `END`) transaction that are in-flight when the old
|
103
|
+
master goes `READ_ONLY` will crash. In theory there’s a workaround for this
|
104
|
+
problem, in pratice it’s rather unwieldy due to a lack of shared global
|
105
|
+
variables in MySQL.
|
106
|
+
- Connection variables are unsupported, due to the connection being able to go
|
107
|
+
away at any time.
|
99
108
|
|
100
109
|
## Installation
|
101
110
|
|
102
|
-
|
103
|
-
Add this line to your application's Gemfile:
|
111
|
+
Add this line to your application’s Gemfile:
|
104
112
|
|
105
113
|
gem 'ar_mysql_flexmaster'
|
106
114
|
|
@@ -119,5 +127,3 @@ Or install it yourself as:
|
|
119
127
|
3. Commit your changes (`git commit -am 'Added some feature'`)
|
120
128
|
4. Push to the branch (`git push origin my-new-feature`)
|
121
129
|
5. Create new Pull Request
|
122
|
-
|
123
|
-
|
data/Rakefile
CHANGED
data/ar_mysql_flexmaster.gemspec
CHANGED
@@ -1,27 +1,30 @@
|
|
1
1
|
# -*- encoding: utf-8 -*-
|
2
2
|
|
3
3
|
Gem::Specification.new do |gem|
|
4
|
-
gem.authors = ["Ben Osheroff"]
|
5
|
-
gem.email = ["
|
6
|
-
gem.description =
|
7
|
-
gem.summary =
|
8
|
-
gem.homepage = "http://github.com/
|
4
|
+
gem.authors = ["Ben Osheroff", "Benjamin Quorning", "Gabe Martin-Dempesy", "Michael Grosser", "Pierre Schambacher"]
|
5
|
+
gem.email = ["bquorning@zendesk.com", "gabe@zendesk.com", "mgrosser@zendesk.com", "pschambacher@zendesk.com"]
|
6
|
+
gem.description = "ar_mysql_flexmaster allows configuring N mysql servers in database.yml and auto-selects which is a master at runtime"
|
7
|
+
gem.summary = "select a master at runtime from a list"
|
8
|
+
gem.homepage = "http://github.com/zendesk/ar_mysql_flexmaster"
|
9
9
|
|
10
10
|
gem.files = `git ls-files`.split($\)
|
11
|
-
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
11
|
+
gem.executables = gem.files.grep(%r{^bin/}).map { |f| File.basename(f) }
|
12
12
|
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
13
13
|
gem.name = "ar_mysql_flexmaster"
|
14
14
|
gem.require_paths = ["lib"]
|
15
|
-
gem.version = "1.0.
|
15
|
+
gem.version = "1.0.3"
|
16
|
+
|
17
|
+
gem.required_ruby_version = ">= 2.2"
|
16
18
|
|
17
19
|
gem.add_runtime_dependency("mysql2")
|
18
20
|
gem.add_runtime_dependency("activerecord")
|
19
21
|
gem.add_runtime_dependency("activesupport")
|
22
|
+
gem.add_development_dependency("bundler")
|
20
23
|
gem.add_development_dependency("rake")
|
21
24
|
gem.add_development_dependency("wwtd")
|
22
25
|
gem.add_development_dependency("minitest")
|
23
26
|
gem.add_development_dependency("mocha", "~> 1.1.0")
|
24
27
|
gem.add_development_dependency("bump")
|
25
28
|
gem.add_development_dependency("pry")
|
26
|
-
gem.add_development_dependency("
|
29
|
+
gem.add_development_dependency("isolated_server")
|
27
30
|
end
|
data/bin/master_cut
CHANGED
@@ -10,8 +10,8 @@ Thread.abort_on_exception = false
|
|
10
10
|
|
11
11
|
opts = GetoptLong.new(
|
12
12
|
["--password", "-p", GetoptLong::REQUIRED_ARGUMENT],
|
13
|
-
["--rehome-master",
|
14
|
-
["--start-slave", "-s", GetoptLong::NO_ARGUMENT]
|
13
|
+
["--rehome-master", "-r", GetoptLong::NO_ARGUMENT],
|
14
|
+
["--start-slave", "-s", GetoptLong::NO_ARGUMENT]
|
15
15
|
)
|
16
16
|
|
17
17
|
opts.each do |opt, arg|
|
@@ -39,7 +39,6 @@ unless $old_master && $new_master && $username
|
|
39
39
|
usage
|
40
40
|
end
|
41
41
|
|
42
|
-
|
43
42
|
def open_cx(host)
|
44
43
|
host, port = host.split(":")
|
45
44
|
port = port.to_i if port
|
@@ -121,7 +120,7 @@ def process_kill_thread
|
|
121
120
|
end
|
122
121
|
|
123
122
|
def wait_for_slave_catchup(master, slave)
|
124
|
-
|
123
|
+
loop do
|
125
124
|
master_info = master.query("show master status").first
|
126
125
|
slave_info = slave.query("show slave status").first
|
127
126
|
break if master_info['Position'] <= slave_info['Exec_Master_Log_Pos']
|
@@ -129,11 +128,9 @@ def wait_for_slave_catchup(master, slave)
|
|
129
128
|
end
|
130
129
|
|
131
130
|
def kill_query!(cx, id)
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
raise e unless e.errno == 1094 # unknown thread id error
|
136
|
-
end
|
131
|
+
cx.query("kill #{id}")
|
132
|
+
rescue Mysql2::Error => e
|
133
|
+
raise e unless e.errno == 1094 # unknown thread id error
|
137
134
|
end
|
138
135
|
|
139
136
|
def swap_thread
|
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
|
-
remote:
|
2
|
+
remote: ..
|
3
3
|
specs:
|
4
|
-
ar_mysql_flexmaster (0.
|
4
|
+
ar_mysql_flexmaster (1.0.2)
|
5
5
|
activerecord
|
6
6
|
activesupport
|
7
7
|
mysql2
|
@@ -9,12 +9,12 @@ PATH
|
|
9
9
|
GEM
|
10
10
|
remote: https://rubygems.org/
|
11
11
|
specs:
|
12
|
-
actionmailer (3.2.22)
|
13
|
-
actionpack (= 3.2.22)
|
12
|
+
actionmailer (3.2.22.5)
|
13
|
+
actionpack (= 3.2.22.5)
|
14
14
|
mail (~> 2.5.4)
|
15
|
-
actionpack (3.2.22)
|
16
|
-
activemodel (= 3.2.22)
|
17
|
-
activesupport (= 3.2.22)
|
15
|
+
actionpack (3.2.22.5)
|
16
|
+
activemodel (= 3.2.22.5)
|
17
|
+
activesupport (= 3.2.22.5)
|
18
18
|
builder (~> 3.0.0)
|
19
19
|
erubis (~> 2.7.0)
|
20
20
|
journey (~> 1.0.4)
|
@@ -22,69 +22,69 @@ GEM
|
|
22
22
|
rack-cache (~> 1.2)
|
23
23
|
rack-test (~> 0.6.1)
|
24
24
|
sprockets (~> 2.2.1)
|
25
|
-
activemodel (3.2.22)
|
26
|
-
activesupport (= 3.2.22)
|
25
|
+
activemodel (3.2.22.5)
|
26
|
+
activesupport (= 3.2.22.5)
|
27
27
|
builder (~> 3.0.0)
|
28
|
-
activerecord (3.2.22)
|
29
|
-
activemodel (= 3.2.22)
|
30
|
-
activesupport (= 3.2.22)
|
28
|
+
activerecord (3.2.22.5)
|
29
|
+
activemodel (= 3.2.22.5)
|
30
|
+
activesupport (= 3.2.22.5)
|
31
31
|
arel (~> 3.0.2)
|
32
32
|
tzinfo (~> 0.3.29)
|
33
|
-
activeresource (3.2.22)
|
34
|
-
activemodel (= 3.2.22)
|
35
|
-
activesupport (= 3.2.22)
|
36
|
-
activesupport (3.2.22)
|
33
|
+
activeresource (3.2.22.5)
|
34
|
+
activemodel (= 3.2.22.5)
|
35
|
+
activesupport (= 3.2.22.5)
|
36
|
+
activesupport (3.2.22.5)
|
37
37
|
i18n (~> 0.6, >= 0.6.4)
|
38
38
|
multi_json (~> 1.0)
|
39
39
|
arel (3.0.3)
|
40
40
|
builder (3.0.4)
|
41
|
-
bump (0.5.
|
42
|
-
coderay (1.1.
|
41
|
+
bump (0.5.4)
|
42
|
+
coderay (1.1.1)
|
43
43
|
erubis (2.7.0)
|
44
44
|
hike (1.2.3)
|
45
|
-
i18n (0.
|
45
|
+
i18n (0.8.6)
|
46
|
+
isolated_server (0.4.12)
|
46
47
|
journey (1.0.4)
|
47
|
-
json (1.8.
|
48
|
-
mail (2.5.
|
48
|
+
json (1.8.6)
|
49
|
+
mail (2.5.5)
|
49
50
|
mime-types (~> 1.16)
|
50
51
|
treetop (~> 1.4.8)
|
51
52
|
metaclass (0.0.4)
|
52
53
|
method_source (0.8.2)
|
53
54
|
mime-types (1.25.1)
|
54
|
-
minitest (5.
|
55
|
+
minitest (5.10.3)
|
55
56
|
mocha (1.1.0)
|
56
57
|
metaclass (~> 0.0.1)
|
57
|
-
multi_json (1.
|
58
|
-
mysql2 (0.3.
|
59
|
-
mysql_isolated_server (0.5.3)
|
58
|
+
multi_json (1.12.1)
|
59
|
+
mysql2 (0.3.21)
|
60
60
|
polyglot (0.3.5)
|
61
|
-
pry (0.10.
|
61
|
+
pry (0.10.4)
|
62
62
|
coderay (~> 1.1.0)
|
63
63
|
method_source (~> 0.8.1)
|
64
64
|
slop (~> 3.4)
|
65
65
|
rack (1.4.7)
|
66
|
-
rack-cache (1.
|
66
|
+
rack-cache (1.7.0)
|
67
67
|
rack (>= 0.4)
|
68
68
|
rack-ssl (1.3.4)
|
69
69
|
rack
|
70
70
|
rack-test (0.6.3)
|
71
71
|
rack (>= 1.0)
|
72
|
-
rails (3.2.22)
|
73
|
-
actionmailer (= 3.2.22)
|
74
|
-
actionpack (= 3.2.22)
|
75
|
-
activerecord (= 3.2.22)
|
76
|
-
activeresource (= 3.2.22)
|
77
|
-
activesupport (= 3.2.22)
|
72
|
+
rails (3.2.22.5)
|
73
|
+
actionmailer (= 3.2.22.5)
|
74
|
+
actionpack (= 3.2.22.5)
|
75
|
+
activerecord (= 3.2.22.5)
|
76
|
+
activeresource (= 3.2.22.5)
|
77
|
+
activesupport (= 3.2.22.5)
|
78
78
|
bundler (~> 1.0)
|
79
|
-
railties (= 3.2.22)
|
80
|
-
railties (3.2.22)
|
81
|
-
actionpack (= 3.2.22)
|
82
|
-
activesupport (= 3.2.22)
|
79
|
+
railties (= 3.2.22.5)
|
80
|
+
railties (3.2.22.5)
|
81
|
+
actionpack (= 3.2.22.5)
|
82
|
+
activesupport (= 3.2.22.5)
|
83
83
|
rack-ssl (~> 1.3.2)
|
84
84
|
rake (>= 0.8.7)
|
85
85
|
rdoc (~> 3.4)
|
86
86
|
thor (>= 0.14.6, < 2.0)
|
87
|
-
rake (
|
87
|
+
rake (12.0.0)
|
88
88
|
rdoc (3.12.2)
|
89
89
|
json (~> 1.4)
|
90
90
|
slop (3.6.0)
|
@@ -93,12 +93,12 @@ GEM
|
|
93
93
|
multi_json (~> 1.0)
|
94
94
|
rack (~> 1.0)
|
95
95
|
tilt (~> 1.1, != 1.3.0)
|
96
|
-
thor (0.
|
96
|
+
thor (0.20.0)
|
97
97
|
tilt (1.4.1)
|
98
98
|
treetop (1.4.15)
|
99
99
|
polyglot
|
100
100
|
polyglot (>= 0.3.1)
|
101
|
-
tzinfo (0.3.
|
101
|
+
tzinfo (0.3.53)
|
102
102
|
wwtd (1.3.0)
|
103
103
|
|
104
104
|
PLATFORMS
|
@@ -107,14 +107,14 @@ PLATFORMS
|
|
107
107
|
DEPENDENCIES
|
108
108
|
ar_mysql_flexmaster!
|
109
109
|
bump
|
110
|
+
isolated_server
|
110
111
|
minitest
|
111
112
|
mocha (~> 1.1.0)
|
112
113
|
mysql2 (~> 0.3.0)
|
113
|
-
mysql_isolated_server (~> 0.5)
|
114
114
|
pry
|
115
115
|
rails (~> 3.2.0)
|
116
116
|
rake
|
117
117
|
wwtd
|
118
118
|
|
119
119
|
BUNDLED WITH
|
120
|
-
1.
|
120
|
+
1.15.4
|