active_record_replica 2.0.1 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/README.md +381 -3
- data/lib/active_record_replica/active_record_replica.rb +63 -62
- data/lib/active_record_replica/extensions.rb +1 -1
- data/lib/active_record_replica/version.rb +1 -1
- data/test/active_record_replica_test.rb +91 -81
- data/test/database.yml +1 -1
- data/test/test_helper.rb +38 -0
- metadata +11 -12
- data/test/test.sqlite3 +0 -0
- data/test/test_replica.sqlite3 +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: ee1b932bf5ba07710e4ecdcc0257b8447a98a16f
|
4
|
+
data.tar.gz: 06742d6ba57b35da5b60c0772dce3ceaecd3b2fe
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3b87ec244163be9129dbcfa7cf1f8ae0c7447c644560ee8646d1666bee28cc2a45790b81477bdd59fbcdd33d20cf1d0740d4dcafb0cf36d61fb383c115d2abae
|
7
|
+
data.tar.gz: 10cdb1ef5e8d5b12a2277c346e336dc72b99ba4606e6e3d80dc08999b6bf180be24899210c10146ed51e5ef42f9e26f94389ed0aaa3f87d6d2e45ceeb77ad217
|
data/README.md
CHANGED
@@ -1,10 +1,388 @@
|
|
1
1
|
# Active Record Replica
|
2
|
-
|
2
|
+
[](https://rubygems.org/gems/active_record_replica)
|
3
|
+
[](https://travis-ci.org/teespring/active_record_replica)
|
4
|
+
[](http://opensource.org/licenses/Apache-2.0)
|
5
|
+

|
3
6
|
|
4
7
|
Redirect ActiveRecord (Rails) reads to replica databases while ensuring all writes go to the primary database.
|
5
8
|
|
6
9
|
## Status
|
10
|
+
This is a slight modification of Rocket Job's original library, simply renaming it from `active_record_slave` to `active_record_replica`.
|
7
11
|
|
8
|
-
|
12
|
+
In order to more clearly distinguish the library from `active_record_slave`, we also incremented the major version – it is, however, functionally equivalent.
|
9
13
|
|
10
|
-
|
14
|
+
## Introduction
|
15
|
+
|
16
|
+
`active_record_replica` redirects all database reads to replica instances while ensuring
|
17
|
+
that all writes go to the primary database. `active_record_replica` ensures that
|
18
|
+
any reads that are performed within a database transaction are by default directed to the primary
|
19
|
+
database to ensure data consistency.
|
20
|
+
|
21
|
+
## Status
|
22
|
+
|
23
|
+
Production Ready. Actively used in large production environments.
|
24
|
+
|
25
|
+
## Features
|
26
|
+
|
27
|
+
* Redirecting reads to a single replica database.
|
28
|
+
* Works with any database driver that works with ActiveRecord.
|
29
|
+
* Supports all Rails 3, 4, or 5 read apis.
|
30
|
+
* Including dynamic finders, AREL, and ActiveRecord::Base.select.
|
31
|
+
* **NOTE**: In Rails 3 and 4, QueryCache is only enabled for BaseConnection by default. In Rails 5, it's enabled for all connections. [(PR)](https://github.com/rails/rails/pull/28869)
|
32
|
+
* Transaction aware
|
33
|
+
* Detects when a query is inside of a transaction and sends those reads to the primary by default.
|
34
|
+
* Can be configured to send reads in a transaction to replica databases.
|
35
|
+
* Lightweight footprint.
|
36
|
+
* No overhead whatsoever when a replica is _not_ configured.
|
37
|
+
* Negligible overhead when redirecting reads to the replica.
|
38
|
+
* Connection Pools to both databases are retained and maintained independently by ActiveRecord.
|
39
|
+
* The primary and replica databases do not have to be of the same type.
|
40
|
+
* For example Oracle could be the primary with MySQL as the replica database.
|
41
|
+
* Debug logs include a prefix of `Replica: ` to indicate which SQL statements are going
|
42
|
+
to the replica database.
|
43
|
+
|
44
|
+
### Example showing Replica redirected read
|
45
|
+
|
46
|
+
```ruby
|
47
|
+
# Read from the replica database
|
48
|
+
r = Role.where(name: 'manager').first
|
49
|
+
r.description = 'Manager'
|
50
|
+
|
51
|
+
# Save changes back to the primary database
|
52
|
+
r.save!
|
53
|
+
```
|
54
|
+
|
55
|
+
Log file output:
|
56
|
+
|
57
|
+
03-13-12 05:56:05 pm,[2608],b[0],[0], Replica: Role Load (3.0ms) SELECT `roles`.* FROM `roles` WHERE `roles`.`name` = 'manager' LIMIT 1
|
58
|
+
03-13-12 05:56:22 pm,[2608],b[0],[0], AREL (12.0ms) UPDATE `roles` SET `description` = 'Manager' WHERE `roles`.`id` = 5
|
59
|
+
|
60
|
+
### Example showing how reads within a transaction go to the primary
|
61
|
+
|
62
|
+
```ruby
|
63
|
+
Role.transaction do
|
64
|
+
r = Role.where(name: 'manager').first
|
65
|
+
r.description = 'Manager'
|
66
|
+
r.save!
|
67
|
+
end
|
68
|
+
```
|
69
|
+
|
70
|
+
Log file output:
|
71
|
+
|
72
|
+
03-13-12 06:02:09 pm,[2608],b[0],[0], Role Load (2.0ms) SELECT `roles`.* FROM `roles` WHERE `roles`.`name` = 'manager' LIMIT 1
|
73
|
+
03-13-12 06:02:09 pm,[2608],b[0],[0], AREL (2.0ms) UPDATE `roles` SET `description` = 'Manager' WHERE `roles`.`id` = 4
|
74
|
+
|
75
|
+
### Forcing a read against the primary
|
76
|
+
|
77
|
+
Sometimes it is necessary to read from the primary:
|
78
|
+
|
79
|
+
```ruby
|
80
|
+
ActiveRecordReplica.read_from_primary do
|
81
|
+
r = Role.where(name: 'manager').first
|
82
|
+
end
|
83
|
+
```
|
84
|
+
|
85
|
+
## Usage Notes
|
86
|
+
|
87
|
+
### delete_all
|
88
|
+
|
89
|
+
Delete all executes against the primary database since it is only a delete:
|
90
|
+
|
91
|
+
```
|
92
|
+
D, [2012-11-06T19:47:29.125932 #89772] DEBUG -- : SQL (1.0ms) DELETE FROM "users"
|
93
|
+
```
|
94
|
+
|
95
|
+
### destroy_all
|
96
|
+
|
97
|
+
First performs a read against the replica database and then deletes the corresponding
|
98
|
+
data from the primary
|
99
|
+
|
100
|
+
```
|
101
|
+
D, [2012-11-06T19:43:26.890674 #89002] DEBUG -- : Replica: User Load (0.1ms) SELECT "users".* FROM "users"
|
102
|
+
D, [2012-11-06T19:43:26.890972 #89002] DEBUG -- : (0.0ms) begin transaction
|
103
|
+
D, [2012-11-06T19:43:26.891667 #89002] DEBUG -- : SQL (0.4ms) DELETE FROM "users" WHERE "users"."id" = ? [["id", 3]]
|
104
|
+
D, [2012-11-06T19:43:26.892697 #89002] DEBUG -- : (0.9ms) commit transaction
|
105
|
+
```
|
106
|
+
|
107
|
+
## Transactions
|
108
|
+
|
109
|
+
By default ActiveRecordReplica detects when a call is inside a transaction and will
|
110
|
+
send all reads to the _primary_ when a transaction is active.
|
111
|
+
|
112
|
+
It is now possible to send reads to database replicas and ignore whether currently
|
113
|
+
inside a transaction:
|
114
|
+
|
115
|
+
In file config/application.rb:
|
116
|
+
|
117
|
+
```ruby
|
118
|
+
# Read from replica even when in an active transaction
|
119
|
+
config.active_record_replica.ignore_transactions = true
|
120
|
+
```
|
121
|
+
|
122
|
+
It is important to identify any code in the application that depends on being
|
123
|
+
able to read any changes already part of the transaction, but not yet committed
|
124
|
+
and wrap those reads with `ActiveRecordReplica.read_from_primary`
|
125
|
+
|
126
|
+
```ruby
|
127
|
+
Inquiry.transaction do
|
128
|
+
# Create a new inquiry
|
129
|
+
Inquiry.create
|
130
|
+
|
131
|
+
# The above inquiry is not visible yet if already in a Rails transaction.
|
132
|
+
# Use `read_from_primary` to ensure it is included in the count below:
|
133
|
+
ActiveRecordReplica.read_from_primary do
|
134
|
+
count = Inquiry.count
|
135
|
+
end
|
136
|
+
|
137
|
+
end
|
138
|
+
```
|
139
|
+
|
140
|
+
## Note
|
141
|
+
|
142
|
+
`active_record_replica` is a very simple layer that inserts itself into the call chain whenever a replica is configured.
|
143
|
+
By observation we noticed that all reads are made to a select set of methods and
|
144
|
+
all writes are made directly to one method: `execute`.
|
145
|
+
|
146
|
+
Using this observation `active_record_replica` only needs to intercept calls to the known select apis:
|
147
|
+
* select_all
|
148
|
+
* select_one
|
149
|
+
* select_rows
|
150
|
+
* select_value
|
151
|
+
* select_values
|
152
|
+
|
153
|
+
Calls to the above methods are redirected to the replica active record model `ActiveRecordReplica::Replica`.
|
154
|
+
This model is 100% managed by the regular Active Record mechanisms such as connection pools etc.
|
155
|
+
|
156
|
+
This lightweight approach ensures that all calls to the above API's are redirected to the replica without impacting:
|
157
|
+
* Transactions
|
158
|
+
* Writes
|
159
|
+
* Any SQL calls directly to `execute`
|
160
|
+
|
161
|
+
One of the limitations with this approach is that any code that performs a query by calling `execute` direct will not
|
162
|
+
be redirected to the replica instance. In this case replace the use of `execute` with one of the the above select methods.
|
163
|
+
|
164
|
+
|
165
|
+
## Note when using `dependent: destroy`
|
166
|
+
|
167
|
+
When performing in-memory only model assignments Active Record will create a transaction against the primary even though
|
168
|
+
the transaction may never be used.
|
169
|
+
|
170
|
+
Even though the transaction is unused it sends the following messages to the primary database:
|
171
|
+
~~
|
172
|
+
SET autocommit=0
|
173
|
+
commit
|
174
|
+
SET autocommit=1
|
175
|
+
~~
|
176
|
+
|
177
|
+
This will impact the primary database if sufficient calls are made, such as in batch workers.
|
178
|
+
|
179
|
+
For Example:
|
180
|
+
|
181
|
+
~~ruby
|
182
|
+
class Parent < ActiveRecord::Base
|
183
|
+
has_one :child, dependent: :destroy
|
184
|
+
end
|
185
|
+
|
186
|
+
class Child < ActiveRecord::Base
|
187
|
+
belongs_to :parent
|
188
|
+
end
|
189
|
+
|
190
|
+
# The following code will create an unused transaction against the primary, even when reads are going to replicas:
|
191
|
+
parent = Parent.new
|
192
|
+
parent.child = Child.new
|
193
|
+
~~
|
194
|
+
|
195
|
+
If the `dependent: :destroy` is removed it no longer creates a transaction, but it also means dependents are not
|
196
|
+
destroyed when a parent is destroyed.
|
197
|
+
|
198
|
+
For this scenario when we are 100% confident no writes are being performed the following can be performed to
|
199
|
+
ignore any attempt Active Record makes at creating the transaction:
|
200
|
+
|
201
|
+
~~ruby
|
202
|
+
ActiveRecordReplica.skip_transactions do
|
203
|
+
parent = Parent.new
|
204
|
+
parent.child = Child.new
|
205
|
+
end
|
206
|
+
~~
|
207
|
+
|
208
|
+
To help identify any code within a block that is creating transactions, wrap the code with
|
209
|
+
`ActiveRecordReplica.block_transactions` to make it raise an exception anytime a transaction is attempted:
|
210
|
+
|
211
|
+
~~ruby
|
212
|
+
ActiveRecordReplica.block_transactions do
|
213
|
+
parent = Parent.new
|
214
|
+
parent.child = Child.new
|
215
|
+
end
|
216
|
+
~~
|
217
|
+
|
218
|
+
## Install
|
219
|
+
|
220
|
+
Add to `Gemfile`
|
221
|
+
|
222
|
+
```ruby
|
223
|
+
gem 'active_record_replica'
|
224
|
+
```
|
225
|
+
|
226
|
+
Run bundler to install:
|
227
|
+
|
228
|
+
```
|
229
|
+
bundle
|
230
|
+
```
|
231
|
+
|
232
|
+
Or, without Bundler:
|
233
|
+
|
234
|
+
```
|
235
|
+
gem install active_record_replica
|
236
|
+
```
|
237
|
+
|
238
|
+
## Configuration
|
239
|
+
|
240
|
+
To enable replica reads for any environment just add a _replica:_ entry to database.yml
|
241
|
+
along with all the usual ActiveRecord database configuration options.
|
242
|
+
|
243
|
+
For Example:
|
244
|
+
|
245
|
+
```yaml
|
246
|
+
production:
|
247
|
+
database: production
|
248
|
+
username: username
|
249
|
+
password: password
|
250
|
+
encoding: utf8
|
251
|
+
adapter: mysql
|
252
|
+
host: primary1
|
253
|
+
pool: 50
|
254
|
+
replica:
|
255
|
+
database: production
|
256
|
+
username: username
|
257
|
+
password: password
|
258
|
+
encoding: utf8
|
259
|
+
adapter: mysql
|
260
|
+
host: replica1
|
261
|
+
pool: 50
|
262
|
+
```
|
263
|
+
|
264
|
+
Sometimes it is useful to turn on replica reads per host, for example to activate
|
265
|
+
replica reads only on the linux host 'batch':
|
266
|
+
|
267
|
+
```yaml
|
268
|
+
production:
|
269
|
+
database: production
|
270
|
+
username: username
|
271
|
+
password: password
|
272
|
+
encoding: utf8
|
273
|
+
adapter: mysql
|
274
|
+
host: primary1
|
275
|
+
pool: 50
|
276
|
+
<% if `hostname`.strip == 'batch' %>
|
277
|
+
replica:
|
278
|
+
database: production
|
279
|
+
username: username
|
280
|
+
password: password
|
281
|
+
encoding: utf8
|
282
|
+
adapter: mysql
|
283
|
+
host: replica1
|
284
|
+
pool: 50
|
285
|
+
<% end %>
|
286
|
+
```
|
287
|
+
|
288
|
+
If there are multiple replicas, it is possible to randomly select a replica on startup
|
289
|
+
to balance the load across the replicas:
|
290
|
+
|
291
|
+
```yaml
|
292
|
+
production:
|
293
|
+
database: production
|
294
|
+
username: username
|
295
|
+
password: password
|
296
|
+
encoding: utf8
|
297
|
+
adapter: mysql
|
298
|
+
host: primary1
|
299
|
+
pool: 50
|
300
|
+
replica:
|
301
|
+
database: production
|
302
|
+
username: username
|
303
|
+
password: password
|
304
|
+
encoding: utf8
|
305
|
+
adapter: mysql
|
306
|
+
host: <%= %w(replica1 replica2 replica3).sample %>
|
307
|
+
pool: 50
|
308
|
+
```
|
309
|
+
|
310
|
+
Replicas can also be assigned to specific hosts by using the hostname:
|
311
|
+
|
312
|
+
```yaml
|
313
|
+
production:
|
314
|
+
database: production
|
315
|
+
username: username
|
316
|
+
password: password
|
317
|
+
encoding: utf8
|
318
|
+
adapter: mysql
|
319
|
+
host: primary1
|
320
|
+
pool: 50
|
321
|
+
replica:
|
322
|
+
database: production
|
323
|
+
username: username
|
324
|
+
password: password
|
325
|
+
encoding: utf8
|
326
|
+
adapter: mysql
|
327
|
+
host: <%= `hostname`.strip == 'app1' ? 'replica1' : 'replica2' %>
|
328
|
+
pool: 50
|
329
|
+
```
|
330
|
+
|
331
|
+
## Set primary as default for Read
|
332
|
+
|
333
|
+
The default behavior can also set to read/write operations against primary database.
|
334
|
+
|
335
|
+
Create an initializer file config/initializer/active_record_replica.rb to force read from primary:
|
336
|
+
|
337
|
+
```yaml
|
338
|
+
ActiveRecordReplica.read_from_primary!
|
339
|
+
```
|
340
|
+
|
341
|
+
Then use this method and supply block to read from the replica database:
|
342
|
+
|
343
|
+
```yaml
|
344
|
+
ActiveRecordReplica.read_from_replica do
|
345
|
+
User.count
|
346
|
+
end
|
347
|
+
```
|
348
|
+
|
349
|
+
## Dependencies
|
350
|
+
|
351
|
+
See [.travis.yml](https://github.com/reidmorrison/active_record_replica/blob/master/.travis.yml) for the list of tested Ruby platforms
|
352
|
+
|
353
|
+
## Versioning
|
354
|
+
|
355
|
+
This project uses [Semantic Versioning](http://semver.org/).
|
356
|
+
|
357
|
+
## Contributing
|
358
|
+
|
359
|
+
1. Fork repository in Github.
|
360
|
+
|
361
|
+
2. Checkout your forked repository:
|
362
|
+
|
363
|
+
```bash
|
364
|
+
git clone https://github.com/your_github_username/active_record_replica.git
|
365
|
+
cd active_record_replica
|
366
|
+
```
|
367
|
+
|
368
|
+
3. Create branch for your contribution:
|
369
|
+
|
370
|
+
```bash
|
371
|
+
git co -b your_new_branch_name
|
372
|
+
```
|
373
|
+
|
374
|
+
4. Make code changes.
|
375
|
+
|
376
|
+
5. Ensure tests pass.
|
377
|
+
|
378
|
+
6. Push to your fork origin.
|
379
|
+
|
380
|
+
```bash
|
381
|
+
git push origin
|
382
|
+
```
|
383
|
+
|
384
|
+
7. Submit PR from the branch on your fork in Github.
|
385
|
+
|
386
|
+
## Author
|
387
|
+
|
388
|
+
[Reid Morrison](https://github.com/reidmorrison) :: @reidmorrison
|
@@ -2,13 +2,6 @@
|
|
2
2
|
# ActiveRecord read from a replica
|
3
3
|
#
|
4
4
|
module ActiveRecordReplica
|
5
|
-
# Select Methods
|
6
|
-
SELECT_METHODS = [:select, :select_all, :select_one, :select_rows, :select_value, :select_values]
|
7
|
-
|
8
|
-
# In case in the future we are forced to intercept connection#execute if the
|
9
|
-
# above select methods are not sufficient
|
10
|
-
# SQL_READS = /\A\s*(SELECT|WITH|SHOW|CALL|EXPLAIN|DESCRIBE)/i
|
11
|
-
|
12
5
|
# Install ActiveRecord::Replica into ActiveRecord to redirect reads to the replica
|
13
6
|
# Parameters:
|
14
7
|
# adapter_class:
|
@@ -19,35 +12,32 @@ module ActiveRecordReplica
|
|
19
12
|
# In a non-Rails environment, supply the environment such as
|
20
13
|
# 'development', 'production'
|
21
14
|
def self.install!(adapter_class = nil, environment = nil)
|
22
|
-
replica_config =
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
base = adapter_class || ActiveRecord::Base.connection.class
|
34
|
-
base.include(Extensions)
|
35
|
-
else
|
36
|
-
ActiveRecord::Base.logger.info "ActiveRecordReplica not installed since no replica database defined"
|
15
|
+
replica_config = ActiveRecord::Base.configurations[environment || Rails.env]["replica"]
|
16
|
+
unless replica_config
|
17
|
+
ActiveRecord::Base.logger.info("ActiveRecordReplica not installed since no replica database defined")
|
18
|
+
return
|
19
|
+
end
|
20
|
+
|
21
|
+
# When the DBMS is not available, an exception (e.g. PG::ConnectionBad) is raised
|
22
|
+
active_db_connection = ActiveRecord::Base.connection.active? rescue false
|
23
|
+
unless active_db_connection
|
24
|
+
ActiveRecord::Base.logger.info("ActiveRecord not connected so not installing ActiveRecordReplica")
|
25
|
+
return
|
37
26
|
end
|
27
|
+
|
28
|
+
version = ActiveRecordReplica::VERSION
|
29
|
+
ActiveRecord::Base.logger.info("ActiveRecordReplica.install! v#{version} Establishing connection to replica database")
|
30
|
+
Replica.establish_connection(replica_config)
|
31
|
+
|
32
|
+
# Inject a new #select method into the ActiveRecord Database adapter
|
33
|
+
base = adapter_class || ActiveRecord::Base.connection.class
|
34
|
+
base.include(Extensions)
|
38
35
|
end
|
39
36
|
|
40
37
|
# Force reads for the supplied block to read from the primary database
|
41
38
|
# Only applies to calls made within the current thread
|
42
|
-
def self.read_from_primary
|
43
|
-
|
44
|
-
begin
|
45
|
-
previous = Thread.current.thread_variable_get(:active_record_replica)
|
46
|
-
read_from_primary!
|
47
|
-
yield
|
48
|
-
ensure
|
49
|
-
Thread.current.thread_variable_set(:active_record_replica, previous)
|
50
|
-
end
|
39
|
+
def self.read_from_primary(&block)
|
40
|
+
thread_variable_yield(:active_record_replica, :primary, &block)
|
51
41
|
end
|
52
42
|
|
53
43
|
#
|
@@ -56,71 +46,63 @@ module ActiveRecordReplica
|
|
56
46
|
# and set ActiveRecordReplica.read_from_primary! to force read from primary.
|
57
47
|
# Then use this method and supply block to read from the replica database
|
58
48
|
# Only applies to calls made within the current thread
|
59
|
-
def self.read_from_replica
|
60
|
-
|
61
|
-
begin
|
62
|
-
previous = Thread.current.thread_variable_get(:active_record_replica)
|
63
|
-
read_from_replica!
|
64
|
-
yield
|
65
|
-
ensure
|
66
|
-
Thread.current.thread_variable_set(:active_record_replica, previous)
|
67
|
-
end
|
49
|
+
def self.read_from_replica(&block)
|
50
|
+
thread_variable_yield(:active_record_replica, :replica, &block)
|
68
51
|
end
|
69
52
|
|
70
53
|
# When only reading from a replica it is important to prevent entering any into
|
71
54
|
# a transaction since the transaction still sends traffic to the primary
|
72
55
|
# that will cause the primary database to slow down processing empty transactions.
|
73
56
|
def self.block_transactions
|
74
|
-
|
75
|
-
previous = Thread.current.thread_variable_get(:active_record_replica_transaction)
|
76
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, :block)
|
77
|
-
yield
|
78
|
-
ensure
|
79
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, previous)
|
80
|
-
end
|
57
|
+
thread_variable_yield(:active_record_replica_transaction, :block, &block)
|
81
58
|
end
|
82
59
|
|
83
60
|
# During this block any attempts to start or end transactions will be ignored.
|
84
61
|
# This extreme action should only be taken when 100% certain no writes are going to be
|
85
62
|
# performed.
|
86
63
|
def self.skip_transactions
|
87
|
-
|
88
|
-
previous = Thread.current.thread_variable_get(:active_record_replica_transaction)
|
89
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, :skip)
|
90
|
-
yield
|
91
|
-
ensure
|
92
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, previous)
|
93
|
-
end
|
64
|
+
thread_variable_yield(:active_record_replica_transaction, :skip, &block)
|
94
65
|
end
|
95
66
|
|
96
67
|
# Whether this thread is currently forcing all reads to go against the primary database
|
97
68
|
def self.read_from_primary?
|
98
|
-
|
69
|
+
!read_from_replica?
|
99
70
|
end
|
100
71
|
|
101
72
|
# Whether this thread is currently forcing all reads to go against the replica database
|
102
73
|
def self.read_from_replica?
|
103
|
-
Thread.current.thread_variable_get(:active_record_replica)
|
74
|
+
case Thread.current.thread_variable_get(:active_record_replica)
|
75
|
+
when :primary
|
76
|
+
false
|
77
|
+
when :replica
|
78
|
+
true
|
79
|
+
else
|
80
|
+
@read_from_replica
|
81
|
+
end
|
104
82
|
end
|
105
83
|
|
106
|
-
# Force all subsequent reads
|
84
|
+
# Force all subsequent reads in this process to read from the primary database.
|
85
|
+
#
|
86
|
+
# The default behavior can be set to read/write operations against primary.
|
87
|
+
# Create an initializer file config/initializer/active_record_replica.rb
|
88
|
+
# and set ActiveRecordReplica.read_from_primary! to force read from primary.
|
107
89
|
def self.read_from_primary!
|
108
|
-
|
90
|
+
@read_from_replica = false
|
109
91
|
end
|
110
92
|
|
111
|
-
#
|
93
|
+
# Force all subsequent reads in this process to read from the replica database.
|
112
94
|
def self.read_from_replica!
|
113
|
-
|
95
|
+
@read_from_replica = true
|
114
96
|
end
|
115
97
|
|
116
98
|
# Whether any attempt to start a transaction should result in an exception
|
117
99
|
def self.block_transactions?
|
118
|
-
|
100
|
+
thread_variable_equals(:active_record_replica_transaction, :block)
|
119
101
|
end
|
120
102
|
|
121
103
|
# Whether any attempt to start a transaction should be skipped.
|
122
104
|
def self.skip_transactions?
|
123
|
-
|
105
|
+
thread_variable_equals(:active_record_replica_transaction, :skip)
|
124
106
|
end
|
125
107
|
|
126
108
|
# Returns whether replica reads are ignoring transactions
|
@@ -135,5 +117,24 @@ module ActiveRecordReplica
|
|
135
117
|
|
136
118
|
private
|
137
119
|
|
120
|
+
def self.thread_variable_equals(key, value)
|
121
|
+
Thread.current.thread_variable_get(key) == value
|
122
|
+
end
|
123
|
+
|
124
|
+
# Sets the thread variable for the duration of the supplied block.
|
125
|
+
# Restores the previous value on completion of the block.
|
126
|
+
def self.thread_variable_yield(key, new_value)
|
127
|
+
previous = Thread.current.thread_variable_get(key)
|
128
|
+
return yield if previous == new_value
|
129
|
+
|
130
|
+
begin
|
131
|
+
Thread.current.thread_variable_set(key, new_value)
|
132
|
+
yield
|
133
|
+
ensure
|
134
|
+
Thread.current.thread_variable_set(key, previous)
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
138
|
@ignore_transactions = false
|
139
|
+
@read_from_replica = true
|
139
140
|
end
|
@@ -3,7 +3,7 @@ module ActiveRecordReplica
|
|
3
3
|
module Extensions
|
4
4
|
extend ActiveSupport::Concern
|
5
5
|
|
6
|
-
|
6
|
+
[:select, :select_all, :select_one, :select_rows, :select_value, :select_values].each do |select_method|
|
7
7
|
class_eval <<-RUBY, __FILE__, __LINE__ + 1
|
8
8
|
def #{select_method}(sql, name = nil, *args)
|
9
9
|
return super if active_record_replica_read_from_primary?
|
@@ -1,93 +1,44 @@
|
|
1
|
-
|
2
|
-
require 'logger'
|
3
|
-
require 'erb'
|
4
|
-
|
5
|
-
l = Logger.new('test.log')
|
6
|
-
l.level = ::Logger::DEBUG
|
7
|
-
ActiveRecord::Base.logger = l
|
8
|
-
ActiveRecord::Base.configurations = YAML::load(ERB.new(IO.read('test/database.yml')).result)
|
9
|
-
|
10
|
-
# Define Schema in second database (replica)
|
11
|
-
# Note: This is not be required when the primary database is being replicated to the replica db
|
12
|
-
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['test']['replica'])
|
13
|
-
|
14
|
-
# Create table users in database active_record_replica_test
|
15
|
-
ActiveRecord::Schema.define :version => 0 do
|
16
|
-
create_table :users, :force => true do |t|
|
17
|
-
t.string :name
|
18
|
-
t.string :address
|
19
|
-
end
|
20
|
-
end
|
21
|
-
|
22
|
-
# Define Schema in primary database
|
23
|
-
ActiveRecord::Base.establish_connection(:test)
|
1
|
+
# frozen_string_literal: true
|
24
2
|
|
25
|
-
|
26
|
-
ActiveRecord::Schema.define :version => 0 do
|
27
|
-
create_table :users, :force => true do |t|
|
28
|
-
t.string :name
|
29
|
-
t.string :address
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
|
-
# AR Model
|
34
|
-
class User < ActiveRecord::Base
|
35
|
-
end
|
36
|
-
|
37
|
-
# Install ActiveRecord replica. Done automatically by railtie in a Rails environment
|
38
|
-
# Also tell it to use the test environment since Rails.env is not available
|
39
|
-
ActiveRecordReplica.install!(nil, 'test')
|
3
|
+
require_relative "test_helper"
|
40
4
|
|
41
5
|
#
|
42
6
|
# Unit Test for active_record_replica
|
43
7
|
#
|
44
|
-
# Since
|
8
|
+
# Since there is no database replication in this test environment, it will
|
45
9
|
# use 2 separate databases. Writes go to the first database and reads to the second.
|
46
10
|
# As a result any writes to the first database will not be visible when trying to read from
|
47
11
|
# the second test database.
|
12
|
+
#
|
13
|
+
# The tests verify that reads going to the replica database do not find data written to the primary.
|
48
14
|
class ActiveRecordReplicaTest < Minitest::Test
|
49
|
-
describe
|
15
|
+
describe ActiveRecordReplica do
|
16
|
+
let(:user_name) { "Joe Bloggs" }
|
17
|
+
let(:address) { "Somewhere" }
|
18
|
+
let(:user) { User.new(name: user_name, address: address) }
|
50
19
|
|
51
20
|
before do
|
52
21
|
ActiveRecordReplica.ignore_transactions = false
|
53
|
-
|
54
|
-
User.delete_all
|
55
|
-
|
56
|
-
@name = "Joe Bloggs"
|
57
|
-
@address = "Somewhere"
|
58
|
-
@user = User.new(
|
59
|
-
:name => @name,
|
60
|
-
:address => @address
|
61
|
-
)
|
62
|
-
end
|
63
|
-
|
64
|
-
after do
|
22
|
+
ActiveRecordReplica.read_from_replica!
|
65
23
|
User.delete_all
|
66
24
|
end
|
67
25
|
|
68
|
-
it
|
69
|
-
|
26
|
+
it "saves to primary" do
|
27
|
+
user.save!
|
70
28
|
end
|
71
29
|
|
72
|
-
|
73
|
-
# NOTE:
|
74
|
-
#
|
75
|
-
# There is no automated replication between the SQL lite databases
|
76
|
-
# so the tests will be verifying that reads going to the "replica" (second)
|
77
|
-
# database do not find data written to the primary.
|
78
|
-
#
|
79
|
-
it 'saves to primary, read from replica' do
|
30
|
+
it "saves to primary, read from replica" do
|
80
31
|
# Read from replica
|
81
|
-
assert_equal 0, User.where(:
|
32
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
82
33
|
|
83
34
|
# Write to primary
|
84
|
-
|
35
|
+
user.save!
|
85
36
|
|
86
37
|
# Read from replica
|
87
|
-
assert_equal 0, User.where(:
|
38
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
88
39
|
end
|
89
40
|
|
90
|
-
it
|
41
|
+
it "save to primary, read from primary when in a transaction" do
|
91
42
|
assert_equal false, ActiveRecordReplica.ignore_transactions?
|
92
43
|
|
93
44
|
User.transaction do
|
@@ -95,56 +46,115 @@ class ActiveRecordReplicaTest < Minitest::Test
|
|
95
46
|
assert_equal 0, User.count
|
96
47
|
|
97
48
|
# Read from Primary
|
98
|
-
assert_equal 0, User.where(:
|
49
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
99
50
|
|
100
51
|
# Write to primary
|
101
|
-
|
52
|
+
user.save!
|
102
53
|
|
103
54
|
# Read from Primary
|
104
|
-
assert_equal 1, User.where(:
|
55
|
+
assert_equal 1, User.where(name: user_name, address: address).count
|
105
56
|
end
|
106
57
|
|
107
58
|
# Read from Non-replicated replica
|
108
|
-
assert_equal 0, User.where(:
|
59
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
109
60
|
end
|
110
61
|
|
111
|
-
it
|
62
|
+
it "save to primary, read from replica when ignoring transactions" do
|
112
63
|
ActiveRecordReplica.ignore_transactions = true
|
113
|
-
|
64
|
+
assert ActiveRecordReplica.ignore_transactions?
|
114
65
|
|
115
66
|
User.transaction do
|
116
67
|
# The delete_all in setup should have cleared the table
|
117
68
|
assert_equal 0, User.count
|
118
69
|
|
119
70
|
# Read from Primary
|
120
|
-
assert_equal 0, User.where(:
|
71
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
121
72
|
|
122
73
|
# Write to primary
|
123
|
-
|
74
|
+
user.save!
|
124
75
|
|
125
76
|
# Read from Non-replicated replica
|
126
|
-
assert_equal 0, User.where(:
|
77
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
127
78
|
end
|
128
79
|
|
129
80
|
# Read from Non-replicated replica
|
130
|
-
assert_equal 0, User.where(:
|
81
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
131
82
|
end
|
132
83
|
|
133
|
-
it
|
84
|
+
it "saves to primary, force a read from primary even when _not_ in a transaction" do
|
134
85
|
# Read from replica
|
135
|
-
assert_equal 0, User.where(:
|
86
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
136
87
|
|
137
88
|
# Write to primary
|
138
|
-
|
89
|
+
user.save!
|
139
90
|
|
140
91
|
# Read from replica
|
141
|
-
assert_equal 0, User.where(:
|
92
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
142
93
|
|
143
94
|
# Read from Primary
|
144
95
|
ActiveRecordReplica.read_from_primary do
|
145
|
-
assert_equal 1, User.where(:
|
96
|
+
assert_equal 1, User.where(name: user_name, address: address).count
|
146
97
|
end
|
147
98
|
end
|
148
99
|
|
100
|
+
describe ".read_from_replica?" do
|
101
|
+
it "is true when global replica flag is set" do
|
102
|
+
ActiveRecordReplica.read_from_replica!
|
103
|
+
assert ActiveRecordReplica.read_from_replica?
|
104
|
+
end
|
105
|
+
|
106
|
+
it "is false when reading from replica" do
|
107
|
+
ActiveRecordReplica.read_from_primary!
|
108
|
+
refute ActiveRecordReplica.read_from_replica?
|
109
|
+
end
|
110
|
+
end
|
111
|
+
|
112
|
+
describe ".read_from_primary?" do
|
113
|
+
it "is true when global primary flag is set" do
|
114
|
+
ActiveRecordReplica.read_from_primary!
|
115
|
+
assert ActiveRecordReplica.read_from_primary?
|
116
|
+
end
|
117
|
+
|
118
|
+
it "is false when reading from replica" do
|
119
|
+
ActiveRecordReplica.read_from_replica!
|
120
|
+
refute ActiveRecordReplica.read_from_primary?
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
124
|
+
describe ".read_from_replica" do
|
125
|
+
it "works with global replica flag" do
|
126
|
+
ActiveRecordReplica.read_from_replica!
|
127
|
+
ActiveRecordReplica.read_from_replica do
|
128
|
+
assert ActiveRecordReplica.read_from_replica?
|
129
|
+
refute ActiveRecordReplica.read_from_primary?
|
130
|
+
end
|
131
|
+
end
|
132
|
+
|
133
|
+
it "overwrites global replica flag" do
|
134
|
+
ActiveRecordReplica.read_from_primary!
|
135
|
+
ActiveRecordReplica.read_from_replica do
|
136
|
+
assert ActiveRecordReplica.read_from_replica?
|
137
|
+
refute ActiveRecordReplica.read_from_primary?
|
138
|
+
end
|
139
|
+
end
|
140
|
+
end
|
141
|
+
|
142
|
+
describe ".read_from_primary" do
|
143
|
+
it "works with global replica flag" do
|
144
|
+
ActiveRecordReplica.read_from_primary!
|
145
|
+
ActiveRecordReplica.read_from_primary do
|
146
|
+
assert ActiveRecordReplica.read_from_primary?
|
147
|
+
refute ActiveRecordReplica.read_from_replica?
|
148
|
+
end
|
149
|
+
end
|
150
|
+
|
151
|
+
it "overwrites global replica flag" do
|
152
|
+
ActiveRecordReplica.read_from_replica!
|
153
|
+
ActiveRecordReplica.read_from_primary do
|
154
|
+
assert ActiveRecordReplica.read_from_primary?
|
155
|
+
refute ActiveRecordReplica.read_from_replica?
|
156
|
+
end
|
157
|
+
end
|
158
|
+
end
|
149
159
|
end
|
150
160
|
end
|
data/test/database.yml
CHANGED
@@ -3,7 +3,7 @@ test:
|
|
3
3
|
database: test/test.sqlite3
|
4
4
|
pool: 5
|
5
5
|
timeout: 5000
|
6
|
-
# Make the replica a separate database that is not
|
6
|
+
# Make the replica a separate database that is not replicated to ensure reads
|
7
7
|
# and writes go to the appropriate databases
|
8
8
|
replica:
|
9
9
|
adapter: sqlite3
|
data/test/test_helper.rb
CHANGED
@@ -4,3 +4,41 @@ require 'active_record'
|
|
4
4
|
require 'minitest/autorun'
|
5
5
|
require 'active_record_replica'
|
6
6
|
require 'awesome_print'
|
7
|
+
require 'logger'
|
8
|
+
require 'erb'
|
9
|
+
|
10
|
+
l = Logger.new('test.log')
|
11
|
+
l.level = ::Logger::DEBUG
|
12
|
+
ActiveRecord::Base.logger = l
|
13
|
+
ActiveRecord::Base.configurations = YAML::load(ERB.new(IO.read('test/database.yml')).result)
|
14
|
+
|
15
|
+
# Define Schema in second database (replica)
|
16
|
+
# Note: This is not be required when the primary database is being replicated to the replica db
|
17
|
+
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['test']['replica'])
|
18
|
+
|
19
|
+
# Create table users in database active_record_replica_test
|
20
|
+
ActiveRecord::Schema.define :version => 0 do
|
21
|
+
create_table :users, :force => true do |t|
|
22
|
+
t.string :name
|
23
|
+
t.string :address
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
# Define Schema in primary database
|
28
|
+
ActiveRecord::Base.establish_connection(:test)
|
29
|
+
|
30
|
+
# Create table users in database active_record_replica_test
|
31
|
+
ActiveRecord::Schema.define :version => 0 do
|
32
|
+
create_table :users, :force => true do |t|
|
33
|
+
t.string :name
|
34
|
+
t.string :address
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
# AR Model
|
39
|
+
class User < ActiveRecord::Base
|
40
|
+
end
|
41
|
+
|
42
|
+
# Install ActiveRecord replica. Done automatically by railtie in a Rails environment
|
43
|
+
# Also tell it to use the test environment since Rails.env is not available
|
44
|
+
ActiveRecordReplica.install!(nil, 'test')
|
metadata
CHANGED
@@ -1,14 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: active_record_replica
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 3.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Reid Morrison
|
8
|
-
|
8
|
+
- James Brady
|
9
|
+
autorequire:
|
9
10
|
bindir: bin
|
10
11
|
cert_chain: []
|
11
|
-
date:
|
12
|
+
date: 2021-03-18 00:00:00.000000000 Z
|
12
13
|
dependencies:
|
13
14
|
- !ruby/object:Gem::Dependency
|
14
15
|
name: activerecord
|
@@ -30,9 +31,10 @@ dependencies:
|
|
30
31
|
- - "<"
|
31
32
|
- !ruby/object:Gem::Version
|
32
33
|
version: 6.0.0
|
33
|
-
description:
|
34
|
+
description:
|
34
35
|
email:
|
35
36
|
- reidmo@gmail.com
|
37
|
+
- james@spri.ng
|
36
38
|
executables: []
|
37
39
|
extensions: []
|
38
40
|
extra_rdoc_files: []
|
@@ -49,14 +51,12 @@ files:
|
|
49
51
|
- lib/active_record_replica/version.rb
|
50
52
|
- test/active_record_replica_test.rb
|
51
53
|
- test/database.yml
|
52
|
-
- test/test.sqlite3
|
53
54
|
- test/test_helper.rb
|
54
|
-
|
55
|
-
homepage: https://github.com/rocketjob/active_record_replica
|
55
|
+
homepage: https://github.com/teespring/active_record_replica
|
56
56
|
licenses:
|
57
57
|
- Apache-2.0
|
58
58
|
metadata: {}
|
59
|
-
post_install_message:
|
59
|
+
post_install_message:
|
60
60
|
rdoc_options: []
|
61
61
|
require_paths:
|
62
62
|
- lib
|
@@ -71,14 +71,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
71
71
|
- !ruby/object:Gem::Version
|
72
72
|
version: '0'
|
73
73
|
requirements: []
|
74
|
-
|
75
|
-
|
74
|
+
rubyforge_project:
|
75
|
+
rubygems_version: 2.6.14.3
|
76
|
+
signing_key:
|
76
77
|
specification_version: 4
|
77
78
|
summary: Redirect ActiveRecord (Rails) reads to replica databases while ensuring all
|
78
79
|
writes go to the primary database.
|
79
80
|
test_files:
|
80
81
|
- test/database.yml
|
81
82
|
- test/test_helper.rb
|
82
|
-
- test/test.sqlite3
|
83
83
|
- test/active_record_replica_test.rb
|
84
|
-
- test/test_replica.sqlite3
|
data/test/test.sqlite3
DELETED
Binary file
|
data/test/test_replica.sqlite3
DELETED
Binary file
|