active_record_shards 3.11.0 → 3.19.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/README.md +120 -57
- data/lib/active_record_shards/association_collection_connection_selection.rb +23 -16
- data/lib/active_record_shards/configuration_parser.rb +19 -5
- data/lib/active_record_shards/connection_handler.rb +3 -8
- data/lib/active_record_shards/connection_pool.rb +1 -0
- data/lib/active_record_shards/connection_specification.rb +7 -16
- data/lib/active_record_shards/connection_switcher-4-2.rb +56 -0
- data/lib/active_record_shards/connection_switcher-5-0.rb +1 -1
- data/lib/active_record_shards/connection_switcher-5-1.rb +1 -1
- data/lib/active_record_shards/connection_switcher.rb +89 -61
- data/lib/active_record_shards/default_replica_patches.rb +278 -0
- data/lib/active_record_shards/default_slave_patches.rb +3 -148
- data/lib/active_record_shards/deprecation.rb +12 -0
- data/lib/active_record_shards/migration.rb +35 -31
- data/lib/active_record_shards/model.rb +26 -10
- data/lib/active_record_shards/patches-4-2.rb +1 -5
- data/lib/active_record_shards/schema_dumper_extension.rb +6 -5
- data/lib/active_record_shards/shard_selection.rb +28 -26
- data/lib/active_record_shards/shard_support.rb +1 -0
- data/lib/active_record_shards/sql_comments.rb +11 -4
- data/lib/active_record_shards/tasks.rb +40 -35
- data/lib/active_record_shards.rb +111 -7
- metadata +61 -58
- data/lib/active_record_shards/connection_switcher-4-0.rb +0 -68
- data/lib/active_record_shards/patches-3-2.rb +0 -11
- data/lib/active_record_shards/patches-5-0.rb +0 -13
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 24b77105aa8df10b1e79313f01b54fe87dba293938c9d41dc103f41cac64ec84
|
4
|
+
data.tar.gz: ec1da959a6dc2adbb6f287f73a3346c68fd49db30f7922dbc9adca7368532af2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5ccae049b9c4a450f04cc2ae6b0cfca9eba75ac72d5f795c0c49ae5fc19e37fad1341d14f3ba34f75c684696131586a663fb306bcd35c2dad9fc625ad36955f3
|
7
|
+
data.tar.gz: 0b46d1cafa4fecae447a01106bfc4de592b7271fc17fef4c254a44a80cc22b22f5155e460d6aba036baad1a229f6f0b6935c3d108c935c5fd5cec9e6271eab83
|
data/README.md
CHANGED
@@ -1,11 +1,20 @@
|
|
1
|
-
[![Build Status](https://
|
1
|
+
[![Build Status](https://github.com/zendesk/active_record_shards/workflows/CI/badge.svg)](https://github.com/zendesk/active_record_shards/actions?query=workflow%3ACI)
|
2
2
|
|
3
3
|
# ActiveRecord Shards
|
4
4
|
|
5
|
-
ActiveRecord Shards is an extension for ActiveRecord that provides support for sharded database and
|
5
|
+
ActiveRecord Shards is an extension for ActiveRecord that provides support for sharded database and replicas. Basically it is just a nice way to
|
6
6
|
switch between database connections. We've made the implementation very small, and have tried not to reinvent any wheels already present in ActiveRecord.
|
7
7
|
|
8
|
-
ActiveRecord Shards has been used and tested on Rails
|
8
|
+
ActiveRecord Shards has been used and tested on Rails 4.2, 5.x and 6.0, and has in some form or another been used in production on large Rails apps for several years.
|
9
|
+
|
10
|
+
- [Installation](#installation)
|
11
|
+
- [Configuration](#configuration)
|
12
|
+
- [Migrations](#migrations)
|
13
|
+
- [Example](#example)
|
14
|
+
- [Shared Model](#create-a-table-for-the-shared-not-sharded-model)
|
15
|
+
- [Sharded Model](#create-a-table-for-the-sharded-model)
|
16
|
+
- [Usage](#usage)
|
17
|
+
- [Debugging](#debugging)
|
9
18
|
|
10
19
|
## Installation
|
11
20
|
|
@@ -15,85 +24,139 @@ and make sure to require 'active\_record\_shards' in some way.
|
|
15
24
|
|
16
25
|
## Configuration
|
17
26
|
|
18
|
-
Add the
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
27
|
+
Add the replica and shard configuration to config/database.yml:
|
28
|
+
|
29
|
+
```yaml
|
30
|
+
production:
|
31
|
+
adapter: mysql
|
32
|
+
encoding: utf8
|
33
|
+
database: my_app_main
|
34
|
+
pool: 5
|
35
|
+
host: db1
|
36
|
+
username: root
|
37
|
+
password:
|
38
|
+
replica:
|
39
|
+
host: db1_replica
|
40
|
+
shards:
|
41
|
+
1:
|
42
|
+
host: db_shard1
|
43
|
+
database: my_app_shard
|
44
|
+
replica:
|
45
|
+
host: db_shard1_replica
|
46
|
+
2:
|
47
|
+
host: db_shard2
|
48
|
+
database: my_app_shard
|
49
|
+
replica:
|
50
|
+
host: db_shard2_replica
|
51
|
+
```
|
41
52
|
|
42
53
|
basically connections inherit configuration from the parent configuration file.
|
43
54
|
|
55
|
+
## Migrations
|
56
|
+
|
57
|
+
ActiveRecord Shards also patches migrations to support running migrations on a shared (not sharded) or a sharded database.
|
58
|
+
Each migration class has to specify a shard spec indicating where to run the migration.
|
59
|
+
|
60
|
+
Valid shard specs:
|
61
|
+
|
62
|
+
* `:none` - Run this migration on the shared database, not any shards
|
63
|
+
* `:all` - Run this migration on all of the shards, not the shared database
|
64
|
+
|
65
|
+
#### Example
|
66
|
+
|
67
|
+
###### Create a table for the shared (not sharded) model
|
68
|
+
|
69
|
+
```ruby
|
70
|
+
class CreateAccounts < ActiveRecord::Migration
|
71
|
+
shard :none
|
72
|
+
|
73
|
+
def change
|
74
|
+
create_table :accounts do |t|
|
75
|
+
# This is NOT necessary for the gem to work, we just use it in the examples below demonstrating one way to switch shards
|
76
|
+
t.integer :shard_id, null: false
|
77
|
+
|
78
|
+
t.string :name
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
82
|
+
```
|
83
|
+
|
84
|
+
###### Create a table for the sharded model
|
85
|
+
|
86
|
+
```ruby
|
87
|
+
class CreateProjects < ActiveRecord::Migration
|
88
|
+
shard :all
|
89
|
+
|
90
|
+
def change
|
91
|
+
create_table :projects do |t|
|
92
|
+
t.references :account
|
93
|
+
t.string :name
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
97
|
+
```
|
98
|
+
|
44
99
|
## Usage
|
45
100
|
|
46
101
|
Normally you have some models that live on a shared database, and you might need to query this data in order to know what shard to switch to.
|
47
102
|
All the models that live on the shared database must be marked as not\_sharded:
|
48
103
|
|
49
|
-
|
50
|
-
|
104
|
+
```ruby
|
105
|
+
class Account < ActiveRecord::Base
|
106
|
+
not_sharded
|
51
107
|
|
52
|
-
|
53
|
-
|
108
|
+
has_many :projects
|
109
|
+
end
|
54
110
|
|
55
|
-
|
56
|
-
|
57
|
-
|
111
|
+
class Project < ActiveRecord::Base
|
112
|
+
belongs_to :account
|
113
|
+
end
|
114
|
+
```
|
58
115
|
|
59
116
|
So in this setup the accounts live on the shared database, but the projects are sharded. If accounts have a shard\_id column, you could lookup the account
|
60
117
|
in a rack middleware and switch to the right shard:
|
61
118
|
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
def call(env)
|
68
|
-
account = lookup_account(env)
|
119
|
+
```ruby
|
120
|
+
class AccountMiddleware
|
121
|
+
def initialize(app)
|
122
|
+
@app = app
|
123
|
+
end
|
69
124
|
|
70
|
-
|
71
|
-
|
72
|
-
@app.call(env)
|
73
|
-
end
|
74
|
-
else
|
75
|
-
@app.call(env)
|
76
|
-
end
|
77
|
-
end
|
125
|
+
def call(env)
|
126
|
+
account = lookup_account(env)
|
78
127
|
|
79
|
-
|
80
|
-
|
128
|
+
if account
|
129
|
+
ActiveRecord::Base.on_shard(account.shard_id) do
|
130
|
+
@app.call(env)
|
81
131
|
end
|
132
|
+
else
|
133
|
+
@app.call(env)
|
82
134
|
end
|
135
|
+
end
|
83
136
|
|
84
|
-
|
137
|
+
def lookup_account(env)
|
138
|
+
# ...
|
139
|
+
end
|
140
|
+
end
|
141
|
+
```
|
85
142
|
|
86
|
-
|
87
|
-
Account.find_by_big_expensive_query
|
88
|
-
end
|
143
|
+
You can switch to the replica databases at any point by wrapping your code in an on\_replica block:
|
89
144
|
|
90
|
-
|
145
|
+
```ruby
|
146
|
+
ActiveRecord::Base.on_replica do
|
147
|
+
Account.find_by_big_expensive_query
|
148
|
+
end
|
149
|
+
```
|
150
|
+
|
151
|
+
This will perform the query on the replica, and mark the returned instances as read only. There is also a shortcut for this:
|
91
152
|
|
92
|
-
|
153
|
+
```ruby
|
154
|
+
Account.on_replica.find_by_big_expensive_query
|
155
|
+
```
|
93
156
|
|
94
157
|
## Debugging
|
95
158
|
|
96
|
-
Show if a query went to
|
159
|
+
Show if a query went to primary or replica in the logs:
|
97
160
|
|
98
161
|
```Ruby
|
99
162
|
require 'active_record_shards/sql_comments'
|
@@ -1,42 +1,49 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
module ActiveRecordShards
|
3
4
|
module AssociationCollectionConnectionSelection
|
4
|
-
def
|
5
|
-
condition ?
|
5
|
+
def on_replica_if(condition)
|
6
|
+
condition ? on_replica : self
|
6
7
|
end
|
8
|
+
alias_method :on_slave_if, :on_replica_if
|
7
9
|
|
8
|
-
def
|
9
|
-
|
10
|
+
def on_replica_unless(condition)
|
11
|
+
on_replica_if(!condition)
|
10
12
|
end
|
13
|
+
alias_method :on_slave_unless, :on_replica_unless
|
11
14
|
|
12
|
-
def
|
13
|
-
condition ?
|
15
|
+
def on_primary_if(condition)
|
16
|
+
condition ? on_primary : self
|
14
17
|
end
|
18
|
+
alias_method :on_master_if, :on_primary_if
|
15
19
|
|
16
|
-
def
|
17
|
-
|
20
|
+
def on_primary_unless(condition)
|
21
|
+
on_primary_if(!condition)
|
18
22
|
end
|
23
|
+
alias_method :on_master_unless, :on_primary_unless
|
19
24
|
|
20
|
-
def
|
21
|
-
|
25
|
+
def on_replica
|
26
|
+
PrimaryReplicaProxy.new(self, :replica)
|
22
27
|
end
|
28
|
+
alias_method :on_slave, :on_replica
|
23
29
|
|
24
|
-
def
|
25
|
-
|
30
|
+
def on_primary
|
31
|
+
PrimaryReplicaProxy.new(self, :primary)
|
26
32
|
end
|
33
|
+
alias_method :on_master, :on_primary
|
27
34
|
|
28
|
-
class
|
35
|
+
class PrimaryReplicaProxy
|
29
36
|
def initialize(association_collection, which)
|
30
37
|
@association_collection = association_collection
|
31
38
|
@which = which
|
32
39
|
end
|
33
40
|
|
34
|
-
def method_missing(method, *args, &block) # rubocop:disable Style/
|
35
|
-
# would love to not rely on version here, unfortunately @association_collection
|
36
|
-
# is a sensitive little bitch of an object.
|
41
|
+
def method_missing(method, *args, &block) # rubocop:disable Style/MethodMissingSuper, Style/MissingRespondToMissing
|
37
42
|
reflection = @association_collection.proxy_association.reflection
|
38
43
|
reflection.klass.on_cx_switch_block(@which) { @association_collection.send(method, *args, &block) }
|
39
44
|
end
|
40
45
|
end
|
46
|
+
|
47
|
+
MasterSlaveProxy = PrimaryReplicaProxy
|
41
48
|
end
|
42
49
|
end
|
@@ -1,4 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
require 'active_support/core_ext'
|
3
4
|
|
4
5
|
module ActiveRecordShards
|
@@ -6,10 +7,15 @@ module ActiveRecordShards
|
|
6
7
|
module_function
|
7
8
|
|
8
9
|
def explode(conf)
|
9
|
-
conf = conf.deep_dup
|
10
|
+
conf = conf.to_h.deep_dup
|
10
11
|
|
11
12
|
conf.to_a.each do |env_name, env_config|
|
12
13
|
next unless shards = env_config.delete('shards')
|
14
|
+
|
15
|
+
unless shards.keys.all? { |shard_name| shard_name.is_a?(Integer) }
|
16
|
+
raise "All shard names must be integers: #{shards.keys.inspect}."
|
17
|
+
end
|
18
|
+
|
13
19
|
env_config['shard_names'] = shards.keys
|
14
20
|
shards.each do |shard_name, shard_conf|
|
15
21
|
expand_child!(env_config, shard_conf)
|
@@ -18,10 +24,18 @@ module ActiveRecordShards
|
|
18
24
|
end
|
19
25
|
|
20
26
|
conf.to_a.each do |env_name, env_config|
|
21
|
-
if
|
22
|
-
expand_child!(env_config,
|
23
|
-
conf["#{env_name}
|
27
|
+
if replica_conf = env_config.delete('replica')
|
28
|
+
expand_child!(env_config, replica_conf)
|
29
|
+
conf["#{env_name}_replica"] = replica_conf
|
30
|
+
end
|
31
|
+
|
32
|
+
# rubocop:disable Style/Next
|
33
|
+
if legacy_replica_conf = env_config.delete('slave')
|
34
|
+
ActiveRecordShards::Deprecation.warn('`slave` configuration keys should be replaced with `replica` keys!')
|
35
|
+
expand_child!(env_config, legacy_replica_conf)
|
36
|
+
conf["#{env_name}_replica"] = legacy_replica_conf
|
24
37
|
end
|
38
|
+
# rubocop:enable Style/Next
|
25
39
|
end
|
26
40
|
|
27
41
|
conf
|
@@ -29,7 +43,7 @@ module ActiveRecordShards
|
|
29
43
|
|
30
44
|
def expand_child!(parent, child)
|
31
45
|
parent.each do |key, value|
|
32
|
-
unless ['slave', 'shards'].include?(key) || value.is_a?(Hash)
|
46
|
+
unless ['slave', 'replica', 'shards'].include?(key) || value.is_a?(Hash)
|
33
47
|
child[key] ||= value
|
34
48
|
end
|
35
49
|
end
|
@@ -1,13 +1,8 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
ActiveRecord::ConnectionAdapters::ConnectionHandler.class_eval do
|
3
4
|
remove_method :retrieve_connection_pool
|
4
|
-
|
5
|
-
|
6
|
-
class_to_pool[klass.connection_pool_name] ||= pool_for(klass)
|
7
|
-
end
|
8
|
-
else
|
9
|
-
def retrieve_connection_pool(klass)
|
10
|
-
(@class_to_pool || @connection_pools)[klass.connection_pool_name]
|
11
|
-
end
|
5
|
+
def retrieve_connection_pool(klass)
|
6
|
+
class_to_pool[klass.connection_pool_name] ||= pool_for(klass)
|
12
7
|
end
|
13
8
|
end
|
@@ -1,16 +1,12 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
class << ActiveRecord::Base
|
3
|
-
remove_method :establish_connection
|
4
|
+
remove_method :establish_connection if ActiveRecord::VERSION::MAJOR >= 5
|
4
5
|
def establish_connection(spec = ENV["DATABASE_URL"])
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
spec = resolver.spec(spec)
|
10
|
-
else
|
11
|
-
resolver = ActiveRecordShards::ConnectionSpecification::Resolver.new spec, configurations
|
12
|
-
spec = resolver.spec
|
13
|
-
end
|
6
|
+
spec ||= ActiveRecord::ConnectionHandling::DEFAULT_ENV.call
|
7
|
+
spec = spec.to_sym if spec.is_a?(String)
|
8
|
+
resolver = ActiveRecordShards::ConnectionSpecification::Resolver.new configurations
|
9
|
+
spec = resolver.spec(spec)
|
14
10
|
|
15
11
|
unless respond_to?(spec.adapter_method)
|
16
12
|
raise AdapterNotFound, "database configuration specifies nonexistent #{spec.config[:adapter]} adapter"
|
@@ -18,11 +14,6 @@ class << ActiveRecord::Base
|
|
18
14
|
|
19
15
|
remove_connection
|
20
16
|
specification_cache[connection_pool_name] = spec
|
21
|
-
|
22
|
-
if ActiveRecord::VERSION::MAJOR >= 4
|
23
|
-
connection_handler.establish_connection self, spec
|
24
|
-
else
|
25
|
-
connection_handler.establish_connection connection_pool_name, spec
|
26
|
-
end
|
17
|
+
connection_handler.establish_connection self, spec
|
27
18
|
end
|
28
19
|
end
|
@@ -0,0 +1,56 @@
|
|
1
|
+
module ActiveRecordShards
|
2
|
+
module ConnectionSwitcher
|
3
|
+
# Name of the connection pool. Used by ConnectionHandler to retrieve the current connection pool.
|
4
|
+
def connection_pool_name # :nodoc:
|
5
|
+
name = current_shard_selection.shard_name(self)
|
6
|
+
|
7
|
+
# e.g. if "production_replica" is not defined in `Configuration`, fall back to "production"
|
8
|
+
if configurations[name].nil? && on_replica?
|
9
|
+
current_shard_selection.shard_name(self, false)
|
10
|
+
else
|
11
|
+
name
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
private
|
16
|
+
|
17
|
+
def ensure_shard_connection
|
18
|
+
establish_shard_connection unless connected_to_shard?
|
19
|
+
end
|
20
|
+
|
21
|
+
def establish_shard_connection
|
22
|
+
name = connection_pool_name
|
23
|
+
spec = configurations[name]
|
24
|
+
|
25
|
+
if spec.nil?
|
26
|
+
raise ActiveRecord::AdapterNotSpecified, "No database defined by #{name} in your database config. (configurations: #{configurations.keys.inspect})"
|
27
|
+
end
|
28
|
+
|
29
|
+
specification_cache[name] ||= begin
|
30
|
+
resolver = ActiveRecordShards::ConnectionSpecification::Resolver.new configurations
|
31
|
+
resolver.spec(spec)
|
32
|
+
end
|
33
|
+
|
34
|
+
connection_handler.establish_connection(self, specification_cache[name])
|
35
|
+
end
|
36
|
+
|
37
|
+
def specification_cache
|
38
|
+
@@specification_cache ||= {}
|
39
|
+
end
|
40
|
+
|
41
|
+
# Helper method to clear global state when testing.
|
42
|
+
def clear_specification_cache
|
43
|
+
@@specification_cache = {}
|
44
|
+
end
|
45
|
+
|
46
|
+
def connection_pool_key
|
47
|
+
specification_cache[connection_pool_name]
|
48
|
+
end
|
49
|
+
|
50
|
+
def connected_to_shard?
|
51
|
+
specs_to_pools = Hash[connection_handler.connection_pool_list.map { |pool| [pool.spec, pool] }]
|
52
|
+
|
53
|
+
specs_to_pools.key?(connection_pool_key)
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
@@ -4,7 +4,7 @@ module ActiveRecordShards
|
|
4
4
|
name = current_shard_selection.resolve_connection_name(sharded: is_sharded?, configurations: configurations)
|
5
5
|
|
6
6
|
unless configurations[name] || name == "primary"
|
7
|
-
raise ActiveRecord::AdapterNotSpecified, "No database defined by #{name} in database.
|
7
|
+
raise ActiveRecord::AdapterNotSpecified, "No database defined by #{name} in your database config. (configurations: #{configurations.keys.inspect})"
|
8
8
|
end
|
9
9
|
|
10
10
|
name
|
@@ -4,7 +4,7 @@ module ActiveRecordShards
|
|
4
4
|
name = current_shard_selection.resolve_connection_name(sharded: is_sharded?, configurations: configurations)
|
5
5
|
|
6
6
|
unless configurations[name] || name == "primary"
|
7
|
-
raise ActiveRecord::AdapterNotSpecified, "No database defined by #{name} in database.
|
7
|
+
raise ActiveRecord::AdapterNotSpecified, "No database defined by #{name} in your database config. (configurations: #{configurations.keys.inspect})"
|
8
8
|
end
|
9
9
|
|
10
10
|
name
|