rails-sharding 1.0.3 → 1.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.ruby-version +1 -1
- data/.travis.yml +1 -1
- data/README.md +24 -8
- data/lib/rails/sharding/connection_handler.rb +25 -6
- data/lib/rails/sharding/version.rb +1 -1
- data/lib/tasks/rails-sharding.rake +22 -13
- data/rails-sharding.gemspec +6 -6
- metadata +17 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 57602c5018eb3d77a8a684949ffab67a73fed32f4a606e7c918f1cef8181e3b5
|
4
|
+
data.tar.gz: bae0f706bbbc0cdc301451630cb781d6f63b39ab23a54fad8ec12d3e9178de25
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 134c2e5e7f8967167aef8e923c114b203b39e3ad0799de21d503346ac69b358495e52dec898934b322e2baef977c5bd16a2a989e68471cb0972c90ff3a4b07a4
|
7
|
+
data.tar.gz: 44b6700d349482e1d0fbd8d716299f821edd9fc9750bdffab746a949e0c80732e3747e8864f2594bdc79aa0abe40d73a9894c2712edacd12c33c454797be9235
|
data/.ruby-version
CHANGED
@@ -1 +1 @@
|
|
1
|
-
ruby-2.
|
1
|
+
ruby-2.5.7
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -8,28 +8,34 @@
|
|
8
8
|
|
9
9
|
Simple and robust sharding gem for Rails, including Migrations and ActiveRecord extensions
|
10
10
|
|
11
|
-
This gems allows you to easily create extra databases to your rails application, and freely allocate ActiveRecord instances to any of the databases.
|
11
|
+
This gems allows you to easily create extra databases to your rails application, and freely allocate ActiveRecord instances to any of the databases.
|
12
12
|
|
13
|
-
|
13
|
+
Accessing shards is as simple as:
|
14
14
|
```ruby
|
15
|
+
# creating a user to a specific shard
|
15
16
|
new_user = User.using_shard(:shard_group1, :shard1).create(username: 'x')
|
17
|
+
|
18
|
+
# retrieving a user from a specific shard
|
16
19
|
loaded_user = User.using_shard(:shard_group1, :shard1).where(username: 'x').first
|
17
20
|
```
|
18
21
|
|
19
|
-
You can also use the block syntax
|
22
|
+
You can also use the block syntax:
|
20
23
|
```ruby
|
21
24
|
Rails::Sharding.using_shard(:shard_group1, :shard1) do
|
25
|
+
# All statements inside this block will go to the selected shard
|
26
|
+
|
27
|
+
# Do some queries
|
22
28
|
new_user = User.create(username: 'x')
|
23
29
|
loaded_user = User.where(username: 'x').first
|
24
30
|
billing_infos = loaded_user.billing_infos.all
|
25
31
|
end
|
26
32
|
```
|
27
33
|
|
28
|
-
You can also pick and choose which models will be shardable
|
34
|
+
You can also pick and choose which models will be shardable. Non shardable models will be retrieved from the master database, even if inside a `using_shard` block.
|
29
35
|
|
30
36
|
## Compatibility
|
31
37
|
Gem version 1.x.x:
|
32
|
-
* Rails 5.0
|
38
|
+
* Rails 5.0 and 5.1
|
33
39
|
* Databases: MySQL, MariaDB, Postgres
|
34
40
|
|
35
41
|
Gem version 0.x.x:
|
@@ -51,7 +57,6 @@ bundle
|
|
51
57
|
```
|
52
58
|
|
53
59
|
## Creating Shards
|
54
|
-
This gem helps you create shards that are additional and completely separate from your master database. The master database is the one that is created and managed through rails, and is the default storage for all your models.
|
55
60
|
|
56
61
|
To start with the rails-sharding gem, run the command
|
57
62
|
```
|
@@ -80,7 +85,9 @@ development:
|
|
80
85
|
...
|
81
86
|
```
|
82
87
|
|
83
|
-
Rename it to `config/shards.yml` and change it to your database configuration. This example file defines a single shard group (named `shard_group1`) containing two shards (`shard1` and `shard2`).
|
88
|
+
Rename it to `config/shards.yml` and change it to your database configuration. This example file defines a single shard group (named `shard_group1`) containing two shards (`shard1` and `shard2`).
|
89
|
+
|
90
|
+
**A shard group is a set of shards that should have the same schema.**
|
84
91
|
|
85
92
|
When you're ready to create the shards run
|
86
93
|
```
|
@@ -88,7 +95,13 @@ rake shards:create
|
|
88
95
|
```
|
89
96
|
|
90
97
|
## Migrating Shards
|
91
|
-
Go to the directory `db/shards_migrations/shard_group1` and add all migrations that you want to run on the shards of `shard_group1`. By design, all shards in a same group should always have the same schema.
|
98
|
+
Go to the directory `db/shards_migrations/shard_group1` and add all migrations that you want to run on the shards of `shard_group1`. By design, all shards in a same group should always have the same schema.
|
99
|
+
|
100
|
+
|
101
|
+
As of now, there is no generator for migrations. You can use the regular rails generator and move the migrations to the `shards_migration` folder.
|
102
|
+
|
103
|
+
|
104
|
+
For example, add the following migration to your `db/shards_migrations/shard_group1`:
|
92
105
|
```ruby
|
93
106
|
# 20160808000000_create_users.rb
|
94
107
|
class CreateClients < ActiveRecord::Migration[5.0]
|
@@ -175,6 +188,9 @@ Rails::Sharding.setup do |config|
|
|
175
188
|
end
|
176
189
|
```
|
177
190
|
|
191
|
+
## Wiki
|
192
|
+
Want to know more? How to integrate with RSpec, Capistrano, etc? Take a look at our [wiki](https://github.com/hsgubert/rails-sharding/wiki).
|
193
|
+
|
178
194
|
## Development and Contributing
|
179
195
|
|
180
196
|
After checking out the repo:
|
@@ -26,13 +26,30 @@ module Rails::Sharding
|
|
26
26
|
|
27
27
|
resolver = ActiveRecord::ConnectionAdapters::ConnectionSpecification::Resolver.new(shard_group_configurations)
|
28
28
|
begin
|
29
|
+
connection_spec = resolver.spec(shard_name.to_sym)
|
30
|
+
|
31
|
+
# since Rails 5.1 connection_spec already comes with :name set to whatever
|
32
|
+
# key you used to retrieve it from the resolver, in this case the shard_name.
|
33
|
+
# We don't want that, so we overwrite it with our connection name formed
|
34
|
+
# by shard_group:shard_name
|
29
35
|
connection_name = connection_name(shard_group, shard_name)
|
30
|
-
connection_spec
|
36
|
+
connection_spec.instance_variable_set(:@name, connection_name)
|
31
37
|
rescue ActiveRecord::AdapterNotSpecified
|
32
|
-
raise Errors::ConfigNotFoundError, "Cannot find configuration for shard '#{shard_group}:#{shard_name}' in environment '#{environment}' in #{Config.shards_config_file}"
|
38
|
+
raise Errors::ConfigNotFoundError, "Cannot find configuration for shard '#{shard_group}:#{shard_name}' in environment '#{environment}' in #{Config.shards_config_file}, or it does not specify :adapter"
|
33
39
|
end
|
34
40
|
|
35
|
-
connection_handler.establish_connection
|
41
|
+
# Since Rails 5.1 we cannot use connection_handler.establish_connection anymore,
|
42
|
+
# because it does more than establishing_connection. It does a second
|
43
|
+
# connection specification lookup on Base.configurations (where our spec
|
44
|
+
# is not, because we define it in a shards.yml instead of the regular
|
45
|
+
# database.yml) and it notifies subscribers of a event that does not concern
|
46
|
+
# us. Because of this we directly create the connection_pool and inject to
|
47
|
+
# the handler.
|
48
|
+
# Note: we should consider writting our connection handler from scratch, since
|
49
|
+
# it has become simpler than reusing the one from rails. And it will not break
|
50
|
+
# as long as the ConnectionPool interface is stable.
|
51
|
+
connection_handler.remove_connection(connection_spec.name)
|
52
|
+
connection_handler.send(:owner_to_pool)[connection_spec.name] = ActiveRecord::ConnectionAdapters::ConnectionPool.new(connection_spec)
|
36
53
|
end
|
37
54
|
|
38
55
|
def self.connection_pool(shard_group, shard_name)
|
@@ -61,7 +78,7 @@ module Rails::Sharding
|
|
61
78
|
end
|
62
79
|
|
63
80
|
def self.with_connection(shard_group, shard_name, &block)
|
64
|
-
|
81
|
+
connection_pool(shard_group, shard_name).with_connection do |connection|
|
65
82
|
if connection && Config.add_shard_tag_to_query_logs
|
66
83
|
connection_name = connection_name(shard_group, shard_name)
|
67
84
|
add_shard_tag_to_connection_log(connection, connection_name)
|
@@ -91,6 +108,8 @@ module Rails::Sharding
|
|
91
108
|
end
|
92
109
|
|
93
110
|
# Adds a shard tag to the log of all queries executed through this connection
|
111
|
+
# Obs: connection inherits from ActiveRecord::ConnectionAdapters::AbstractAdapter
|
112
|
+
# but its class depends on the database adapter used.
|
94
113
|
def self.add_shard_tag_to_connection_log(connection, shard_tag)
|
95
114
|
# avoids modifing connection twice
|
96
115
|
if connection.respond_to? :shard_tag
|
@@ -107,9 +126,9 @@ module Rails::Sharding
|
|
107
126
|
|
108
127
|
# defines a new #log that adds a tag to the log
|
109
128
|
class << connection
|
110
|
-
def log(sql, name="SQL", binds=[], statement_name=nil, &block)
|
129
|
+
def log(sql, name="SQL", binds=[], type_casted_binds=[], statement_name=nil, &block)
|
111
130
|
name = (name.to_s + " (#{shard_tag})").strip
|
112
|
-
|
131
|
+
original_log(sql, name, binds, type_casted_binds, statement_name, &block)
|
113
132
|
end
|
114
133
|
end
|
115
134
|
|
@@ -274,22 +274,31 @@ shards_namespace = namespace :shards do
|
|
274
274
|
|
275
275
|
desc "Empty the test shards (drops all tables) (options: SHARD_GROUP=x, SHARD=x)"
|
276
276
|
task :purge => [:_make_activerecord_base_shardable] do
|
277
|
-
|
278
|
-
|
279
|
-
|
280
|
-
|
281
|
-
should_reconnect = Rails::Sharding::ConnectionHandler.connection_pool(shard_group, shard).active_connection?
|
282
|
-
Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard, 'test')
|
277
|
+
begin
|
278
|
+
# saves the current RAILS_ENV (we must change it so the environment is set correcly on the metadata table)
|
279
|
+
initial_rails_env = Rails.env
|
280
|
+
Rails.env = 'test'
|
283
281
|
|
284
|
-
|
285
|
-
|
286
|
-
|
287
|
-
|
288
|
-
|
289
|
-
|
290
|
-
|
282
|
+
Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, configuration|
|
283
|
+
puts "== Purging test shard #{shard_group}:#{shard}"
|
284
|
+
begin
|
285
|
+
# establishes connection with test shard, saving if it was connected before (rails 4.2 doesn't do this, but should)
|
286
|
+
should_reconnect = Rails::Sharding::ConnectionHandler.connection_pool(shard_group, shard).active_connection?
|
287
|
+
Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard, 'test')
|
288
|
+
|
289
|
+
Rails::Sharding.using_shard(shard_group, shard) do
|
290
|
+
ActiveRecord::Tasks::DatabaseTasks.purge(configuration)
|
291
|
+
end
|
292
|
+
ensure
|
293
|
+
if should_reconnect
|
294
|
+
# reestablishes connection for RAILS_ENV environment (whatever that is)
|
295
|
+
Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard)
|
296
|
+
end
|
291
297
|
end
|
292
298
|
end
|
299
|
+
ensure
|
300
|
+
# restores rails env
|
301
|
+
Rails.env = initial_rails_env
|
293
302
|
end
|
294
303
|
end
|
295
304
|
end
|
data/rails-sharding.gemspec
CHANGED
@@ -23,14 +23,14 @@ Gem::Specification.new do |spec|
|
|
23
23
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
24
24
|
spec.require_paths = ["lib"]
|
25
25
|
|
26
|
-
spec.add_runtime_dependency 'rails', '5.0
|
26
|
+
spec.add_runtime_dependency 'rails', '~> 5.1.0'
|
27
27
|
|
28
|
-
spec.add_development_dependency "bundler", "~> 1.
|
29
|
-
spec.add_development_dependency "rake", "~>
|
28
|
+
spec.add_development_dependency "bundler", "~> 1.17"
|
29
|
+
spec.add_development_dependency "rake", "~> 13.0"
|
30
30
|
spec.add_development_dependency "rspec", "~> 3.0"
|
31
|
-
spec.add_development_dependency "byebug", '~>
|
31
|
+
spec.add_development_dependency "byebug", '~> 11'
|
32
32
|
spec.add_development_dependency "mysql2", '~> 0'
|
33
|
-
spec.add_development_dependency "pg" # postgres driver
|
33
|
+
spec.add_development_dependency "pg", '~> 0' # postgres driver
|
34
34
|
spec.add_development_dependency "codeclimate-test-reporter", '~> 1'
|
35
|
-
spec.add_development_dependency "simplecov"
|
35
|
+
spec.add_development_dependency "simplecov", '~> 0'
|
36
36
|
end
|
metadata
CHANGED
@@ -1,57 +1,57 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rails-sharding
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Henrique Gubert
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2019-11-08 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rails
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- -
|
17
|
+
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: 5.0
|
19
|
+
version: 5.1.0
|
20
20
|
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- -
|
24
|
+
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: 5.0
|
26
|
+
version: 5.1.0
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: bundler
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
31
|
- - "~>"
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '1.
|
33
|
+
version: '1.17'
|
34
34
|
type: :development
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - "~>"
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '1.
|
40
|
+
version: '1.17'
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: rake
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
45
|
- - "~>"
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '
|
47
|
+
version: '13.0'
|
48
48
|
type: :development
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '
|
54
|
+
version: '13.0'
|
55
55
|
- !ruby/object:Gem::Dependency
|
56
56
|
name: rspec
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -72,14 +72,14 @@ dependencies:
|
|
72
72
|
requirements:
|
73
73
|
- - "~>"
|
74
74
|
- !ruby/object:Gem::Version
|
75
|
-
version: '
|
75
|
+
version: '11'
|
76
76
|
type: :development
|
77
77
|
prerelease: false
|
78
78
|
version_requirements: !ruby/object:Gem::Requirement
|
79
79
|
requirements:
|
80
80
|
- - "~>"
|
81
81
|
- !ruby/object:Gem::Version
|
82
|
-
version: '
|
82
|
+
version: '11'
|
83
83
|
- !ruby/object:Gem::Dependency
|
84
84
|
name: mysql2
|
85
85
|
requirement: !ruby/object:Gem::Requirement
|
@@ -98,14 +98,14 @@ dependencies:
|
|
98
98
|
name: pg
|
99
99
|
requirement: !ruby/object:Gem::Requirement
|
100
100
|
requirements:
|
101
|
-
- - "
|
101
|
+
- - "~>"
|
102
102
|
- !ruby/object:Gem::Version
|
103
103
|
version: '0'
|
104
104
|
type: :development
|
105
105
|
prerelease: false
|
106
106
|
version_requirements: !ruby/object:Gem::Requirement
|
107
107
|
requirements:
|
108
|
-
- - "
|
108
|
+
- - "~>"
|
109
109
|
- !ruby/object:Gem::Version
|
110
110
|
version: '0'
|
111
111
|
- !ruby/object:Gem::Dependency
|
@@ -126,14 +126,14 @@ dependencies:
|
|
126
126
|
name: simplecov
|
127
127
|
requirement: !ruby/object:Gem::Requirement
|
128
128
|
requirements:
|
129
|
-
- - "
|
129
|
+
- - "~>"
|
130
130
|
- !ruby/object:Gem::Version
|
131
131
|
version: '0'
|
132
132
|
type: :development
|
133
133
|
prerelease: false
|
134
134
|
version_requirements: !ruby/object:Gem::Requirement
|
135
135
|
requirements:
|
136
|
-
- - "
|
136
|
+
- - "~>"
|
137
137
|
- !ruby/object:Gem::Version
|
138
138
|
version: '0'
|
139
139
|
description: |-
|
@@ -193,8 +193,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
193
193
|
- !ruby/object:Gem::Version
|
194
194
|
version: '0'
|
195
195
|
requirements: []
|
196
|
-
|
197
|
-
rubygems_version: 2.6.11
|
196
|
+
rubygems_version: 3.0.6
|
198
197
|
signing_key:
|
199
198
|
specification_version: 4
|
200
199
|
summary: Simple and robust sharding for Rails, including Migrations and ActiveRecord
|