mixed_gauge 0.2.1 → 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +1 -0
- data/CHANGELOG.md +5 -0
- data/README.md +127 -22
- data/lib/mixed_gauge.rb +1 -1
- data/lib/mixed_gauge/all_shards_in_parallel.rb +4 -4
- data/lib/mixed_gauge/cluster_config.rb +83 -9
- data/lib/mixed_gauge/config.rb +4 -14
- data/lib/mixed_gauge/model.rb +12 -13
- data/lib/mixed_gauge/routing.rb +1 -0
- data/lib/mixed_gauge/{sub_model_repository.rb → shard_repository.rb} +16 -15
- data/lib/mixed_gauge/version.rb +1 -1
- data/mixed_gauge.gemspec +2 -2
- metadata +8 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b52a83a9e2dcfa88c05591683f5f3f8ca6cdc25e
|
4
|
+
data.tar.gz: 2ebdd85105cf34b3d4cf09485293953e85894be0
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 85c09e39e2a6963dda8afab08b8bad16e7ec4be4ac506f8b208295d84842f9d4b92344e099072cfdf3443af2ce7a7e51d7f629d0940aadeb42ff691541d43948
|
7
|
+
data.tar.gz: 8c014307fa8eae21d653ebed9264f01ba8408db1364a158fcec682fdab9a618f81a9aeff5dbc22e25404228992ed191485e96e26510a33ad9c35d5081a9cd9b9
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,4 +1,9 @@
|
|
1
1
|
# CHANGELOG for mixed_gauge
|
2
|
+
## 1.0.0
|
3
|
+
- Change cluster slots definition interface: does not specify entire slots,
|
4
|
+
but specify only slots size.
|
5
|
+
- Add config validation.
|
6
|
+
- Change class name of generated model: SubModel -> Shard
|
2
7
|
|
3
8
|
## 0.2.1
|
4
9
|
- Fix performance issue that `MixedGauge::Model.shard_for` is very slow.
|
data/README.md
CHANGED
@@ -2,19 +2,123 @@
|
|
2
2
|
[![Build Status](https://travis-ci.org/taiki45/mixed_gauge.svg?branch=master)](https://travis-ci.org/taiki45/mixed_gauge) [![Coverage Status](https://coveralls.io/repos/taiki45/mixed_gauge/badge.svg?branch=master)](https://coveralls.io/r/taiki45/mixed_gauge?branch=master) [![Code Climate](https://codeclimate.com/github/taiki45/mixed_gauge/badges/gpa.svg)](https://codeclimate.com/github/taiki45/mixed_gauge) [![Gem Version](https://badge.fury.io/rb/mixed_gauge.svg)](http://badge.fury.io/rb/mixed_gauge)
|
3
3
|
|
4
4
|
A simple and robust ActiveRecord extension for database sharding.
|
5
|
-
mixed_gauge offers shards management with hash slots
|
6
|
-
|
5
|
+
mixed_gauge offers shards management with hash slots and re-sharding support.
|
6
|
+
It enable you to execute efficient queries to single node with KVS-like interface.
|
7
|
+
And you can even execute limited RDB queries to all nodes with ActiveRecord interface,
|
8
|
+
'course in-parallel.
|
7
9
|
|
8
|
-
##
|
9
|
-
|
10
|
+
## Goal and concept
|
11
|
+
- Simple
|
12
|
+
- No downtime migrations
|
13
|
+
- Rollback-able operations
|
10
14
|
|
11
|
-
|
15
|
+
Database sharding tend to be over-complexed. There are cases which need these complex database sharding but in some cases database sharding can be more simple. The large data set which is enoght big to partition should be designed to be distributed, or should be re-design if it wasn't. Design to be distributed uses key based relation or reverse indexes to fits its limitation. In that case, the data set is almost design to be distributed, mixed_gauge strongly encourages your database sharding by its simplicity.
|
12
16
|
|
13
|
-
|
14
|
-
TODO
|
17
|
+
We, offer 24/7 services, must keep our services running. mixed_gauge supports online migrations: adding new nodes to cluster or removing some existing nodes from cluster. It comes with "key distibution model with hash slots" and database replication and multi-master replication. In sharding we need re-sharding, move data from node to anoter node in cluster, when adding or removing new nodes from cluster. But by setting some rule to node management and using replication, we can finish moving data before adding or removing nodes. The detail operations are specified later chapter of this document.
|
15
18
|
|
16
|
-
|
19
|
+
All operaions should be rollback-able in case of any failures. mixed_gauge's node management can rollback adding and removing nodes operation. The detail operations are specified later chapter of this document.
|
20
|
+
|
21
|
+
|
22
|
+
## Main components of sharding teqnique
|
23
|
+
### Distribution model
|
24
|
+
mixed_gauge's database sharding is based on keys distribution model with hash slots. The key space is split into arbitrary size of slots. `hash(v) mod N` determines which slot is used where `N` is size of configured hash slots. Hash slot is a virtual node and it is assigned to real node.
|
25
|
+
|
26
|
+
The default hash function is CRC32 which has better perfomance for this kind of cases. You can use other hash function.
|
27
|
+
|
28
|
+
### Node management
|
29
|
+
mixed_gauge's database sharding sets a rule to both adding nodes and removing nodes. The node size must be incresed by multiple of 2. At first, the node size is 1. Then the node size is incresed to 2, next is 4, and next of next is 8.
|
30
|
+
|
31
|
+
By setting this rule, we can move (copy) data from node to node before adding or removing nodes by "database replication". For example, when we have `cluster(A)`, which has single node A and node A is assigned (0..1023) hash slots, and plan to migrate to `cluster(A, B)`, which has 2 nodes A and B and node A is assigned (0..511) slots and node B is assigned (512..1023) slots, we can copy and replicate from A to B before migration then just balance hash slots to node B.
|
32
|
+
|
33
|
+
```
|
34
|
+
(1) (2) (3)
|
35
|
+
┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
|
36
|
+
│ │ │ │ │ │ │ │ │ │
|
37
|
+
│ A │ │ A │──────▶│ B │ │ A │ │ B │
|
38
|
+
│ │ │ │ │ │ │ │ │ │
|
39
|
+
└───────┘ └───────┘ └───────┘ └───────┘ └───────┘
|
40
|
+
|
41
|
+
0..1023 0..1023 0..511 512..1023
|
42
|
+
```
|
43
|
+
|
44
|
+
### Migration operations
|
45
|
+
```
|
46
|
+
(1) From 1 node cluster
|
47
|
+
|
48
|
+
0..1023
|
49
|
+
┌───────┐
|
50
|
+
│ │
|
51
|
+
│ │
|
52
|
+
│ A │
|
53
|
+
│ │
|
54
|
+
│ │
|
55
|
+
└───────┘
|
56
|
+
|
57
|
+
(2) Copy data and start replication
|
58
|
+
|
59
|
+
0..1023
|
60
|
+
┌───────┐ ┌───────┐
|
61
|
+
│ │ │ │
|
62
|
+
│ │ │ │
|
63
|
+
│ A │──────▶│ B │
|
64
|
+
│ │ │ │
|
65
|
+
│ │ │ │
|
66
|
+
└───────┘ └───────┘
|
67
|
+
|
68
|
+
(3) Change auto_increment config
|
69
|
+
not to conflict id column
|
70
|
+
|
71
|
+
0..1023
|
72
|
+
1 ┌───────┐ ┌───────┐ 2
|
73
|
+
3 │ │ │ │ 4
|
74
|
+
5 │ │ │ │ 6
|
75
|
+
. │ A │──────▶│ B │ .
|
76
|
+
. │ │ │ │ .
|
77
|
+
│ │ │ │
|
78
|
+
└───────┘ └───────┘
|
79
|
+
increment=2 increment=2
|
80
|
+
offset=1 offset=2
|
81
|
+
|
82
|
+
(4) Start Multi-master replication
|
83
|
+
|
84
|
+
0..1023
|
85
|
+
┌───────┐ ┌───────┐
|
86
|
+
│ │──────▶│ │
|
87
|
+
│ │ │ │
|
88
|
+
│ A │ │ B │
|
89
|
+
│ │ │ │
|
90
|
+
│ │◀──────│ │
|
91
|
+
└───────┘ └───────┘
|
92
|
+
|
93
|
+
(5) Deploy app and apply new cluster
|
94
|
+
configuration
|
95
|
+
|
96
|
+
0..511 512..1023
|
97
|
+
┌───────┐ ┌───────┐
|
98
|
+
│ │──────▶│ │
|
99
|
+
│ │ │ │
|
100
|
+
│ A │ │ B │
|
101
|
+
│ │ │ │
|
102
|
+
│ │◀──────│ │
|
103
|
+
└───────┘ └───────┘
|
104
|
+
|
105
|
+
(6) Stop Multi-master replication
|
106
|
+
|
107
|
+
0..511 512..1023
|
108
|
+
┌───────┐ ┌───────┐
|
109
|
+
│ │ │ │
|
110
|
+
│ │ │ │
|
111
|
+
│ A │ │ B │
|
112
|
+
│ │ │ │
|
113
|
+
│ │ │ │
|
114
|
+
└───────┘ └───────┘
|
115
|
+
```
|
116
|
+
|
117
|
+
In step 3, we set enough big offset not to conflict auto increment value
|
118
|
+
on applying config.
|
17
119
|
|
120
|
+
|
121
|
+
## Usage
|
18
122
|
Add additional database connection config to `database.yml`.
|
19
123
|
|
20
124
|
```yaml
|
@@ -43,17 +147,17 @@ Configure slots (virtual node for cluster) then assign slots to real node.
|
|
43
147
|
MixedGauge.configure do |config|
|
44
148
|
config.define_cluster(:user) do |cluster|
|
45
149
|
# When slots per node * max nodes per cluster = (2 ** 10) * (2 ** 10)
|
46
|
-
cluster.
|
47
|
-
cluster.register(
|
48
|
-
cluster.register(
|
49
|
-
cluster.register(
|
50
|
-
cluster.register(
|
150
|
+
cluster.define_slot_size(1048576)
|
151
|
+
cluster.register(0..262143, :production_user_001)
|
152
|
+
cluster.register(262144..524287, :production_user_002)
|
153
|
+
cluster.register(524288..786431, :production_user_003)
|
154
|
+
cluster.register(786432..1048575, :production_user_004)
|
51
155
|
end
|
52
156
|
end
|
53
157
|
```
|
54
158
|
|
55
159
|
Include `MixedGauge::Model` to your model class, specify cluster name for the
|
56
|
-
model, specify distkey which
|
160
|
+
model, specify distkey which determines node to store.
|
57
161
|
|
58
162
|
```ruby
|
59
163
|
class User < ActiveRecord::Base
|
@@ -63,11 +167,11 @@ class User < ActiveRecord::Base
|
|
63
167
|
end
|
64
168
|
```
|
65
169
|
|
66
|
-
Use `.get` to retrive single
|
170
|
+
Use `.get` to retrive single record which is connected to proper
|
67
171
|
database node. Use `.put!` to create new record to proper database node.
|
68
172
|
|
69
|
-
`.all_shards`
|
70
|
-
database
|
173
|
+
`.all_shards` returns each model class which is connected to proper
|
174
|
+
database node. You can query with these models and aggregate result.
|
71
175
|
|
72
176
|
```ruby
|
73
177
|
User.put!(email: 'alice@example.com', name: 'alice')
|
@@ -79,7 +183,7 @@ alice.save!
|
|
79
183
|
User.all_shards.flat_map {|m| m.find_by(name: 'alice') }.compact
|
80
184
|
```
|
81
185
|
|
82
|
-
When you want to execute queries in parallel, use `.all_shards_in_parallel`.
|
186
|
+
When you want to execute queries in all nodes in parallel, use `.all_shards_in_parallel`.
|
83
187
|
It returns `Mixedgauge::AllShardsInParallel` and it offers some collection
|
84
188
|
actions which runs in parallel. It is aliased to `.parallel`.
|
85
189
|
|
@@ -89,7 +193,7 @@ User.parallel.flat_map {|m| m.where(age: 1) }.size #=> 1
|
|
89
193
|
```
|
90
194
|
|
91
195
|
When you want find by non-distkey, not recomended though, you can define finder
|
92
|
-
methods to model class.
|
196
|
+
methods to model class for convenience.
|
93
197
|
|
94
198
|
```ruby
|
95
199
|
class User < ActiveRecord::Base
|
@@ -110,7 +214,7 @@ alice.save!
|
|
110
214
|
```
|
111
215
|
|
112
216
|
Sometimes you want to generates distkey value before validation. Since mixed_gauge
|
113
|
-
generates sub class of your models, AR's callback is
|
217
|
+
generates sub class of your models, AR's callback is usesless for this usecase,
|
114
218
|
so mixed_gauge offers its own callback method.
|
115
219
|
|
116
220
|
```ruby
|
@@ -136,6 +240,7 @@ access_token = AccessToken.put!
|
|
136
240
|
access_token.token #=> a generated token
|
137
241
|
```
|
138
242
|
|
243
|
+
|
139
244
|
## Advanced configuration
|
140
245
|
### Hash fucntion
|
141
246
|
Default hash fucntion is CRC32, which has better perfomance for this kind of
|
@@ -159,8 +264,8 @@ Suggested hash functions are:
|
|
159
264
|
- FNV Hash
|
160
265
|
- SuperFastHash
|
161
266
|
|
162
|
-
## Installation
|
163
267
|
|
268
|
+
## Installation
|
164
269
|
Add this line to your application's Gemfile:
|
165
270
|
|
166
271
|
```ruby
|
@@ -175,6 +280,6 @@ Or install it yourself as:
|
|
175
280
|
|
176
281
|
$ gem install mixed_gauge
|
177
282
|
|
178
|
-
## Contributing
|
179
283
|
|
284
|
+
## Contributing
|
180
285
|
Feel free to pull request and issue :)
|
data/lib/mixed_gauge.rb
CHANGED
@@ -6,7 +6,7 @@ require 'mixed_gauge/errors'
|
|
6
6
|
require 'mixed_gauge/cluster_config'
|
7
7
|
require 'mixed_gauge/config'
|
8
8
|
require 'mixed_gauge/routing'
|
9
|
-
require 'mixed_gauge/
|
9
|
+
require 'mixed_gauge/shard_repository'
|
10
10
|
require 'mixed_gauge/all_shards_in_parallel'
|
11
11
|
require 'mixed_gauge/model'
|
12
12
|
|
@@ -2,12 +2,12 @@ module MixedGauge
|
|
2
2
|
# Support parallel execution with each shard and deal with AR connection
|
3
3
|
# management in parallel execution.
|
4
4
|
class AllShardsInParallel
|
5
|
-
# @param [Array<Class>] An array of
|
5
|
+
# @param [Array<Class>] An array of shard model class
|
6
6
|
def initialize(shards)
|
7
7
|
@shards = shards
|
8
8
|
end
|
9
9
|
|
10
|
-
# @yield [Class] A
|
10
|
+
# @yield [Class] A shard model class
|
11
11
|
# @return [Array] A result
|
12
12
|
# @example
|
13
13
|
# User.all_shards_in_parallel.map(&:count).reduce(&:+)
|
@@ -19,7 +19,7 @@ module MixedGauge
|
|
19
19
|
commands.map(&:get)
|
20
20
|
end
|
21
21
|
|
22
|
-
# @yield [Class] A
|
22
|
+
# @yield [Class] A shard model class
|
23
23
|
# @return [Array] A result
|
24
24
|
# @example
|
25
25
|
# User.all_shards_in_parallel.flat_map {|m| m.where(age: 1) }
|
@@ -27,7 +27,7 @@ module MixedGauge
|
|
27
27
|
map(&block).flatten
|
28
28
|
end
|
29
29
|
|
30
|
-
# @yield [Class] A
|
30
|
+
# @yield [Class] A shard model class
|
31
31
|
# @return [MixedGauge::AllShardsInParallel]
|
32
32
|
# @example
|
33
33
|
# User.all_shards_in_parallel.each {|m| puts m.count }
|
@@ -9,25 +9,26 @@ module MixedGauge
|
|
9
9
|
@connection_registry = {}
|
10
10
|
end
|
11
11
|
|
12
|
-
# @param [
|
13
|
-
def
|
14
|
-
@slots =
|
12
|
+
# @param [Integer] size The slot size of this cluster.
|
13
|
+
def define_slot_size(n)
|
14
|
+
@slots = 0..(n - 1)
|
15
15
|
end
|
16
16
|
|
17
|
-
# @param [Range] slots
|
17
|
+
# @param [Range] assigned_slots The assigned range of slots of given
|
18
|
+
# connection (shard).
|
18
19
|
# @param [Symbol] connection connection name
|
19
|
-
def register(
|
20
|
-
@connection_registry[
|
20
|
+
def register(assigned_slots, connection)
|
21
|
+
@connection_registry[assigned_slots] = connection
|
21
22
|
end
|
22
23
|
|
24
|
+
# @raise [RuntimeError]
|
23
25
|
def validate_config!
|
24
|
-
|
25
|
-
# validate non Fixnum slots.
|
26
|
+
Validator.new(slot_size, @connection_registry).validate!
|
26
27
|
end
|
27
28
|
|
28
29
|
# @return [Integer]
|
29
30
|
def slot_size
|
30
|
-
@slots.size
|
31
|
+
defined?(@slot_size) ? @slot_size : @slot_size = @slots.size
|
31
32
|
end
|
32
33
|
|
33
34
|
# @param [Integer] slot
|
@@ -40,5 +41,78 @@ module MixedGauge
|
|
40
41
|
def connections
|
41
42
|
@connection_registry.values
|
42
43
|
end
|
44
|
+
|
45
|
+
class Validator
|
46
|
+
# @param [Integer] slot_size
|
47
|
+
# @param [Hash{Range => Symbol}] connection_registry
|
48
|
+
def initialize(slot_size, connection_registry)
|
49
|
+
@slot_size = slot_size
|
50
|
+
@connection_registry = connection_registry
|
51
|
+
end
|
52
|
+
|
53
|
+
# @raise [RuntimeError]
|
54
|
+
def validate!
|
55
|
+
all_start_points = @connection_registry.keys.map(&:min).sort
|
56
|
+
all_end_points = @connection_registry.keys.map(&:max).sort
|
57
|
+
|
58
|
+
check_first_start_point(all_start_points.min)
|
59
|
+
check_coverage(all_start_points, all_end_points)
|
60
|
+
check_last_end_point(all_end_points.max)
|
61
|
+
end
|
62
|
+
|
63
|
+
private
|
64
|
+
|
65
|
+
# @param [Integer] first_start_point
|
66
|
+
def check_first_start_point(first_start_point)
|
67
|
+
unless first_start_point == 0
|
68
|
+
report_invalid_first_start_point(first_start_point)
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
# @param [Array<Integer>] all_start_points
|
73
|
+
# @param [Array<Integer>] all_end_points
|
74
|
+
def check_coverage(all_start_points, all_end_points)
|
75
|
+
all_end_points.each_with_index do |end_point, i|
|
76
|
+
break if all_end_points.size == i + 1
|
77
|
+
|
78
|
+
next_start_point = all_start_points[i + 1]
|
79
|
+
unless end_point.succ == next_start_point
|
80
|
+
report_invalid_coverage(end_point, all_start_points[i + 1])
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
# @param [Integer] last_end_point
|
86
|
+
def check_last_end_point(last_end_point)
|
87
|
+
unless last_end_point == @slot_size - 1
|
88
|
+
report_invalid_last_end_point(last_end_point)
|
89
|
+
end
|
90
|
+
end
|
91
|
+
|
92
|
+
# @param [Integer] point
|
93
|
+
def report_invalid_first_start_point(point)
|
94
|
+
r = @connection_registry.keys.find {|range| range.min == point }
|
95
|
+
connection = @connection_registry[r]
|
96
|
+
raise "First start point must be `0` but given `#{point}`: invalid slot configuration for #{connection}"
|
97
|
+
end
|
98
|
+
|
99
|
+
# @param [Integer] end_point
|
100
|
+
# @param [Integer] next_start_point
|
101
|
+
def report_invalid_coverage(end_point, next_start_point)
|
102
|
+
end_point_slot = @connection_registry.keys.find {|range| range.max == end_point }
|
103
|
+
end_point_connection = @connection_registry[end_point_slot]
|
104
|
+
start_point_slot = @connection_registry.keys.
|
105
|
+
find {|range| range.min == next_start_point && range.max != end_point }
|
106
|
+
start_point_connection = @connection_registry[start_point_slot]
|
107
|
+
raise %!End point `#{end_point}` of "#{end_point_connection}" or start point `#{next_start_point}` of "#{start_point_connection}" is invalid. Next start point must be "previous end point + 1".!
|
108
|
+
end
|
109
|
+
|
110
|
+
# @param [Integer] point
|
111
|
+
def report_invalid_last_end_point(point)
|
112
|
+
r = @connection_registry.keys.find {|range| range.max == point }
|
113
|
+
connection = @connection_registry[r]
|
114
|
+
raise "Last end point must be `#{@slot_size - 1}` but given `#{point}`: invalid slot configuration for #{connection}"
|
115
|
+
end
|
116
|
+
end
|
43
117
|
end
|
44
118
|
end
|
data/lib/mixed_gauge/config.rb
CHANGED
@@ -12,17 +12,14 @@ module MixedGauge
|
|
12
12
|
end
|
13
13
|
|
14
14
|
# Define config for specific cluster.
|
15
|
+
# See README.md for example.
|
15
16
|
# @param [Symbol] cluster_name
|
16
17
|
# @yield [MixedGauge::ClusterConfig]
|
17
|
-
# @
|
18
|
-
# config.define_cluster(:user) do |cluster|
|
19
|
-
# cluster.define_slots(1..1048576)
|
20
|
-
# cluster.register(1..524288, :production_user_001)
|
21
|
-
# cluster.register(524289..1048576, :production_user_002)
|
22
|
-
# end
|
18
|
+
# @raise [RuntimeError] When this cluster config is invalid.
|
23
19
|
def define_cluster(cluster_name, &block)
|
24
20
|
cluster_config = ClusterConfig.new(cluster_name)
|
25
21
|
cluster_config.instance_eval(&block)
|
22
|
+
cluster_config.validate_config!
|
26
23
|
@cluster_configs[cluster_name] = cluster_config
|
27
24
|
end
|
28
25
|
|
@@ -34,14 +31,7 @@ module MixedGauge
|
|
34
31
|
|
35
32
|
# Register arbitrary hash function. Hash function must be a proc and
|
36
33
|
# must return integer.
|
37
|
-
#
|
38
|
-
# # gem install fnv
|
39
|
-
# require "fnv"
|
40
|
-
# Mixedgauge.configure do |config|
|
41
|
-
# config.register_hash_function do |key|
|
42
|
-
# FNV.new.fnv1a_64(key)
|
43
|
-
# end
|
44
|
-
# end
|
34
|
+
# See README.md for example.
|
45
35
|
def register_hash_function(&block)
|
46
36
|
raise ArgumentError if block.arity != 1
|
47
37
|
raise ArgumentError unless block.call('test value').is_a? Integer
|
data/lib/mixed_gauge/model.rb
CHANGED
@@ -20,7 +20,7 @@ module MixedGauge
|
|
20
20
|
|
21
21
|
included do
|
22
22
|
class_attribute :cluster_routing, instance_writer: false
|
23
|
-
class_attribute :
|
23
|
+
class_attribute :shard_repository, instance_writer: false
|
24
24
|
class_attribute :distkey, instance_writer: false
|
25
25
|
end
|
26
26
|
|
@@ -30,7 +30,7 @@ module MixedGauge
|
|
30
30
|
def use_cluster(name)
|
31
31
|
config = MixedGauge.config.fetch_cluster_config(name)
|
32
32
|
self.cluster_routing = MixedGauge::Routing.new(config)
|
33
|
-
self.
|
33
|
+
self.shard_repository = MixedGauge::ShardRepository.new(config, self)
|
34
34
|
self.abstract_class = true
|
35
35
|
end
|
36
36
|
|
@@ -45,7 +45,7 @@ module MixedGauge
|
|
45
45
|
# When distkey value is empty, raises MixedGauge::MissingDistkeyAttribute
|
46
46
|
# error.
|
47
47
|
# @param [Hash] attributes
|
48
|
-
# @return [ActiveRecord::Base] A
|
48
|
+
# @return [ActiveRecord::Base] A shard model instance
|
49
49
|
# @raise [MixedGauge::MissingDistkeyAttribute]
|
50
50
|
def put!(attributes)
|
51
51
|
raise '`distkey` is not defined. Use `def_distkey`.' unless distkey
|
@@ -60,7 +60,7 @@ module MixedGauge
|
|
60
60
|
|
61
61
|
# Returns nil when not found. Except that, is same as `.get!`.
|
62
62
|
# @param [String] key
|
63
|
-
# @return [ActiveRecord::Base, nil] A
|
63
|
+
# @return [ActiveRecord::Base, nil] A shard model instance
|
64
64
|
def get(key)
|
65
65
|
raise 'key must be a String' unless key.is_a?(String)
|
66
66
|
shard_for(key.to_s).find_by(distkey => key)
|
@@ -70,7 +70,7 @@ module MixedGauge
|
|
70
70
|
# `ActiveRecord::RecordNotFound` so you can rescue that exception as same
|
71
71
|
# as AR's RecordNotFound.
|
72
72
|
# @param [String] key
|
73
|
-
# @return [ActiveRecord::Base] A
|
73
|
+
# @return [ActiveRecord::Base] A shard model instance
|
74
74
|
# @raise [MixedGauge::RecordNotFound]
|
75
75
|
def get!(key)
|
76
76
|
get(key) or raise MixedGauge::RecordNotFound
|
@@ -93,22 +93,21 @@ module MixedGauge
|
|
93
93
|
@before_put_callback = block
|
94
94
|
end
|
95
95
|
|
96
|
-
# Returns a generated
|
97
|
-
# shard for given key.
|
96
|
+
# Returns a generated model class of included model class which has proper
|
97
|
+
# connection config for the shard for given key.
|
98
98
|
# @param [String] key A value of distkey
|
99
|
-
# @return [Class] A
|
99
|
+
# @return [Class] A generated model class for given distkey value
|
100
100
|
def shard_for(key)
|
101
101
|
connection_name = cluster_routing.route(key.to_s)
|
102
|
-
|
102
|
+
shard_repository.fetch(connection_name)
|
103
103
|
end
|
104
104
|
|
105
|
-
# Returns all generated
|
106
|
-
#
|
107
|
-
# @return [Array<Class>] An array of sub models
|
105
|
+
# Returns all generated shard model class. Useful to query to all shards.
|
106
|
+
# @return [Array<Class>] An array of shard models
|
108
107
|
# @example
|
109
108
|
# User.all_shards.flat_map {|m| m.find_by(name: 'alice') }.compact
|
110
109
|
def all_shards
|
111
|
-
|
110
|
+
shard_repository.all
|
112
111
|
end
|
113
112
|
|
114
113
|
# @return [Mixedgauge::AllShardsInParallel]
|
data/lib/mixed_gauge/routing.rb
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
module MixedGauge
|
2
|
-
class
|
2
|
+
class ShardRepository
|
3
3
|
attr_reader :base_class
|
4
4
|
|
5
5
|
# @param [ClusterConfig] cluster_config
|
@@ -7,32 +7,33 @@ module MixedGauge
|
|
7
7
|
def initialize(cluster_config, base_class)
|
8
8
|
@base_class = base_class
|
9
9
|
|
10
|
-
|
11
|
-
[connection_name,
|
10
|
+
shards = cluster_config.connections.map do |connection_name|
|
11
|
+
[connection_name, generate_model_for_shard(connection_name)]
|
12
12
|
end
|
13
|
-
@
|
13
|
+
@shards = Hash[shards]
|
14
14
|
end
|
15
15
|
|
16
16
|
# @param [Symbol] connection_name
|
17
|
-
# @return [Class] A
|
17
|
+
# @return [Class] A model class for this shard
|
18
18
|
def fetch(connection_name)
|
19
|
-
@
|
19
|
+
@shards.fetch(connection_name)
|
20
20
|
end
|
21
21
|
|
22
22
|
# @return [Array<Class>]
|
23
23
|
def all
|
24
|
-
@
|
24
|
+
@shards.values
|
25
25
|
end
|
26
26
|
|
27
27
|
private
|
28
28
|
|
29
29
|
# @param [Symbol] connection_name
|
30
|
-
# @return [Class] A
|
31
|
-
|
30
|
+
# @return [Class] A sub class of given AR model.
|
31
|
+
# A sub class has connection setting for specific shard.
|
32
|
+
def generate_model_for_shard(connection_name)
|
32
33
|
base_class_name = @base_class.name
|
33
34
|
class_name = generate_class_name(connection_name)
|
34
35
|
|
35
|
-
|
36
|
+
model = Class.new(base_class) do
|
36
37
|
self.table_name = base_class.table_name
|
37
38
|
module_eval <<-RUBY, __FILE__, __LINE__ + 1
|
38
39
|
def self.name
|
@@ -40,14 +41,14 @@ module MixedGauge
|
|
40
41
|
end
|
41
42
|
RUBY
|
42
43
|
end
|
43
|
-
|
44
|
-
|
44
|
+
model.class_eval { establish_connection(connection_name) }
|
45
|
+
model
|
45
46
|
end
|
46
47
|
|
47
|
-
# @param [Symbol]
|
48
|
+
# @param [Symbol] connection_name
|
48
49
|
# @return [String]
|
49
|
-
def generate_class_name(
|
50
|
-
"
|
50
|
+
def generate_class_name(connection_name)
|
51
|
+
"ShardFor#{connection_name.to_s.gsub('-', '_').classify}"
|
51
52
|
end
|
52
53
|
end
|
53
54
|
end
|
data/lib/mixed_gauge/version.rb
CHANGED
data/mixed_gauge.gemspec
CHANGED
@@ -9,8 +9,8 @@ Gem::Specification.new do |spec|
|
|
9
9
|
spec.authors = ['Taiki Ono']
|
10
10
|
spec.email = ['taiks.4559@gmail.com']
|
11
11
|
|
12
|
-
spec.summary = %{
|
13
|
-
spec.description = %{#{spec.summary}
|
12
|
+
spec.summary = %{A simple and robust ActiveRecord extension for database sharding.}
|
13
|
+
spec.description = %{#{spec.summary} Supports shards management with hash slots, re-sharding support, efficient KVS queries, limited RDB queries.}
|
14
14
|
spec.homepage = 'https://github.com/taiki45/mixed_gauge'
|
15
15
|
spec.license = 'MIT'
|
16
16
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: mixed_gauge
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Taiki Ono
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-06-
|
11
|
+
date: 2015-06-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|
@@ -164,9 +164,9 @@ dependencies:
|
|
164
164
|
- - ">="
|
165
165
|
- !ruby/object:Gem::Version
|
166
166
|
version: '0'
|
167
|
-
description:
|
168
|
-
|
169
|
-
|
167
|
+
description: A simple and robust ActiveRecord extension for database sharding. Supports
|
168
|
+
shards management with hash slots, re-sharding support, efficient KVS queries, limited
|
169
|
+
RDB queries.
|
170
170
|
email:
|
171
171
|
- taiks.4559@gmail.com
|
172
172
|
executables: []
|
@@ -199,7 +199,7 @@ files:
|
|
199
199
|
- lib/mixed_gauge/model.rb
|
200
200
|
- lib/mixed_gauge/railtie.rb
|
201
201
|
- lib/mixed_gauge/routing.rb
|
202
|
-
- lib/mixed_gauge/
|
202
|
+
- lib/mixed_gauge/shard_repository.rb
|
203
203
|
- lib/mixed_gauge/version.rb
|
204
204
|
- lib/tasks/mixed_gauge.rake
|
205
205
|
- mixed_gauge.gemspec
|
@@ -223,9 +223,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
223
223
|
version: '0'
|
224
224
|
requirements: []
|
225
225
|
rubyforge_project:
|
226
|
-
rubygems_version: 2.
|
226
|
+
rubygems_version: 2.4.5
|
227
227
|
signing_key:
|
228
228
|
specification_version: 4
|
229
|
-
summary:
|
229
|
+
summary: A simple and robust ActiveRecord extension for database sharding.
|
230
230
|
test_files: []
|
231
231
|
has_rdoc:
|