active_record_replica 2.0.1 → 3.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/README.md +381 -3
- data/lib/active_record_replica/active_record_replica.rb +63 -62
- data/lib/active_record_replica/extensions.rb +1 -1
- data/lib/active_record_replica/version.rb +1 -1
- data/test/active_record_replica_test.rb +91 -81
- data/test/database.yml +1 -1
- data/test/test_helper.rb +38 -0
- metadata +11 -12
- data/test/test.sqlite3 +0 -0
- data/test/test_replica.sqlite3 +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: ee1b932bf5ba07710e4ecdcc0257b8447a98a16f
|
4
|
+
data.tar.gz: 06742d6ba57b35da5b60c0772dce3ceaecd3b2fe
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3b87ec244163be9129dbcfa7cf1f8ae0c7447c644560ee8646d1666bee28cc2a45790b81477bdd59fbcdd33d20cf1d0740d4dcafb0cf36d61fb383c115d2abae
|
7
|
+
data.tar.gz: 10cdb1ef5e8d5b12a2277c346e336dc72b99ba4606e6e3d80dc08999b6bf180be24899210c10146ed51e5ef42f9e26f94389ed0aaa3f87d6d2e45ceeb77ad217
|
data/README.md
CHANGED
@@ -1,10 +1,388 @@
|
|
1
1
|
# Active Record Replica
|
2
|
-
|
2
|
+
[![Gem Version](https://img.shields.io/gem/v/active_record_replica.svg)](https://rubygems.org/gems/active_record_replica)
|
3
|
+
[![Build Status](https://travis-ci.org/teespring/active_record_replica.svg?branch=master)](https://travis-ci.org/teespring/active_record_replica)
|
4
|
+
[![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0)
|
5
|
+
![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
|
3
6
|
|
4
7
|
Redirect ActiveRecord (Rails) reads to replica databases while ensuring all writes go to the primary database.
|
5
8
|
|
6
9
|
## Status
|
10
|
+
This is a slight modification of Rocket Job's original library, simply renaming it from `active_record_slave` to `active_record_replica`.
|
7
11
|
|
8
|
-
|
12
|
+
In order to more clearly distinguish the library from `active_record_slave`, we also incremented the major version – it is, however, functionally equivalent.
|
9
13
|
|
10
|
-
|
14
|
+
## Introduction
|
15
|
+
|
16
|
+
`active_record_replica` redirects all database reads to replica instances while ensuring
|
17
|
+
that all writes go to the primary database. `active_record_replica` ensures that
|
18
|
+
any reads that are performed within a database transaction are by default directed to the primary
|
19
|
+
database to ensure data consistency.
|
20
|
+
|
21
|
+
## Status
|
22
|
+
|
23
|
+
Production Ready. Actively used in large production environments.
|
24
|
+
|
25
|
+
## Features
|
26
|
+
|
27
|
+
* Redirecting reads to a single replica database.
|
28
|
+
* Works with any database driver that works with ActiveRecord.
|
29
|
+
* Supports all Rails 3, 4, or 5 read apis.
|
30
|
+
* Including dynamic finders, AREL, and ActiveRecord::Base.select.
|
31
|
+
* **NOTE**: In Rails 3 and 4, QueryCache is only enabled for BaseConnection by default. In Rails 5, it's enabled for all connections. [(PR)](https://github.com/rails/rails/pull/28869)
|
32
|
+
* Transaction aware
|
33
|
+
* Detects when a query is inside of a transaction and sends those reads to the primary by default.
|
34
|
+
* Can be configured to send reads in a transaction to replica databases.
|
35
|
+
* Lightweight footprint.
|
36
|
+
* No overhead whatsoever when a replica is _not_ configured.
|
37
|
+
* Negligible overhead when redirecting reads to the replica.
|
38
|
+
* Connection Pools to both databases are retained and maintained independently by ActiveRecord.
|
39
|
+
* The primary and replica databases do not have to be of the same type.
|
40
|
+
* For example Oracle could be the primary with MySQL as the replica database.
|
41
|
+
* Debug logs include a prefix of `Replica: ` to indicate which SQL statements are going
|
42
|
+
to the replica database.
|
43
|
+
|
44
|
+
### Example showing Replica redirected read
|
45
|
+
|
46
|
+
```ruby
|
47
|
+
# Read from the replica database
|
48
|
+
r = Role.where(name: 'manager').first
|
49
|
+
r.description = 'Manager'
|
50
|
+
|
51
|
+
# Save changes back to the primary database
|
52
|
+
r.save!
|
53
|
+
```
|
54
|
+
|
55
|
+
Log file output:
|
56
|
+
|
57
|
+
03-13-12 05:56:05 pm,[2608],b[0],[0], Replica: Role Load (3.0ms) SELECT `roles`.* FROM `roles` WHERE `roles`.`name` = 'manager' LIMIT 1
|
58
|
+
03-13-12 05:56:22 pm,[2608],b[0],[0], AREL (12.0ms) UPDATE `roles` SET `description` = 'Manager' WHERE `roles`.`id` = 5
|
59
|
+
|
60
|
+
### Example showing how reads within a transaction go to the primary
|
61
|
+
|
62
|
+
```ruby
|
63
|
+
Role.transaction do
|
64
|
+
r = Role.where(name: 'manager').first
|
65
|
+
r.description = 'Manager'
|
66
|
+
r.save!
|
67
|
+
end
|
68
|
+
```
|
69
|
+
|
70
|
+
Log file output:
|
71
|
+
|
72
|
+
03-13-12 06:02:09 pm,[2608],b[0],[0], Role Load (2.0ms) SELECT `roles`.* FROM `roles` WHERE `roles`.`name` = 'manager' LIMIT 1
|
73
|
+
03-13-12 06:02:09 pm,[2608],b[0],[0], AREL (2.0ms) UPDATE `roles` SET `description` = 'Manager' WHERE `roles`.`id` = 4
|
74
|
+
|
75
|
+
### Forcing a read against the primary
|
76
|
+
|
77
|
+
Sometimes it is necessary to read from the primary:
|
78
|
+
|
79
|
+
```ruby
|
80
|
+
ActiveRecordReplica.read_from_primary do
|
81
|
+
r = Role.where(name: 'manager').first
|
82
|
+
end
|
83
|
+
```
|
84
|
+
|
85
|
+
## Usage Notes
|
86
|
+
|
87
|
+
### delete_all
|
88
|
+
|
89
|
+
Delete all executes against the primary database since it is only a delete:
|
90
|
+
|
91
|
+
```
|
92
|
+
D, [2012-11-06T19:47:29.125932 #89772] DEBUG -- : SQL (1.0ms) DELETE FROM "users"
|
93
|
+
```
|
94
|
+
|
95
|
+
### destroy_all
|
96
|
+
|
97
|
+
First performs a read against the replica database and then deletes the corresponding
|
98
|
+
data from the primary
|
99
|
+
|
100
|
+
```
|
101
|
+
D, [2012-11-06T19:43:26.890674 #89002] DEBUG -- : Replica: User Load (0.1ms) SELECT "users".* FROM "users"
|
102
|
+
D, [2012-11-06T19:43:26.890972 #89002] DEBUG -- : (0.0ms) begin transaction
|
103
|
+
D, [2012-11-06T19:43:26.891667 #89002] DEBUG -- : SQL (0.4ms) DELETE FROM "users" WHERE "users"."id" = ? [["id", 3]]
|
104
|
+
D, [2012-11-06T19:43:26.892697 #89002] DEBUG -- : (0.9ms) commit transaction
|
105
|
+
```
|
106
|
+
|
107
|
+
## Transactions
|
108
|
+
|
109
|
+
By default ActiveRecordReplica detects when a call is inside a transaction and will
|
110
|
+
send all reads to the _primary_ when a transaction is active.
|
111
|
+
|
112
|
+
It is now possible to send reads to database replicas and ignore whether currently
|
113
|
+
inside a transaction:
|
114
|
+
|
115
|
+
In file config/application.rb:
|
116
|
+
|
117
|
+
```ruby
|
118
|
+
# Read from replica even when in an active transaction
|
119
|
+
config.active_record_replica.ignore_transactions = true
|
120
|
+
```
|
121
|
+
|
122
|
+
It is important to identify any code in the application that depends on being
|
123
|
+
able to read any changes already part of the transaction, but not yet committed
|
124
|
+
and wrap those reads with `ActiveRecordReplica.read_from_primary`
|
125
|
+
|
126
|
+
```ruby
|
127
|
+
Inquiry.transaction do
|
128
|
+
# Create a new inquiry
|
129
|
+
Inquiry.create
|
130
|
+
|
131
|
+
# The above inquiry is not visible yet if already in a Rails transaction.
|
132
|
+
# Use `read_from_primary` to ensure it is included in the count below:
|
133
|
+
ActiveRecordReplica.read_from_primary do
|
134
|
+
count = Inquiry.count
|
135
|
+
end
|
136
|
+
|
137
|
+
end
|
138
|
+
```
|
139
|
+
|
140
|
+
## Note
|
141
|
+
|
142
|
+
`active_record_replica` is a very simple layer that inserts itself into the call chain whenever a replica is configured.
|
143
|
+
By observation we noticed that all reads are made to a select set of methods and
|
144
|
+
all writes are made directly to one method: `execute`.
|
145
|
+
|
146
|
+
Using this observation `active_record_replica` only needs to intercept calls to the known select apis:
|
147
|
+
* select_all
|
148
|
+
* select_one
|
149
|
+
* select_rows
|
150
|
+
* select_value
|
151
|
+
* select_values
|
152
|
+
|
153
|
+
Calls to the above methods are redirected to the replica active record model `ActiveRecordReplica::Replica`.
|
154
|
+
This model is 100% managed by the regular Active Record mechanisms such as connection pools etc.
|
155
|
+
|
156
|
+
This lightweight approach ensures that all calls to the above API's are redirected to the replica without impacting:
|
157
|
+
* Transactions
|
158
|
+
* Writes
|
159
|
+
* Any SQL calls directly to `execute`
|
160
|
+
|
161
|
+
One of the limitations with this approach is that any code that performs a query by calling `execute` direct will not
|
162
|
+
be redirected to the replica instance. In this case replace the use of `execute` with one of the the above select methods.
|
163
|
+
|
164
|
+
|
165
|
+
## Note when using `dependent: destroy`
|
166
|
+
|
167
|
+
When performing in-memory only model assignments Active Record will create a transaction against the primary even though
|
168
|
+
the transaction may never be used.
|
169
|
+
|
170
|
+
Even though the transaction is unused it sends the following messages to the primary database:
|
171
|
+
~~
|
172
|
+
SET autocommit=0
|
173
|
+
commit
|
174
|
+
SET autocommit=1
|
175
|
+
~~
|
176
|
+
|
177
|
+
This will impact the primary database if sufficient calls are made, such as in batch workers.
|
178
|
+
|
179
|
+
For Example:
|
180
|
+
|
181
|
+
~~ruby
|
182
|
+
class Parent < ActiveRecord::Base
|
183
|
+
has_one :child, dependent: :destroy
|
184
|
+
end
|
185
|
+
|
186
|
+
class Child < ActiveRecord::Base
|
187
|
+
belongs_to :parent
|
188
|
+
end
|
189
|
+
|
190
|
+
# The following code will create an unused transaction against the primary, even when reads are going to replicas:
|
191
|
+
parent = Parent.new
|
192
|
+
parent.child = Child.new
|
193
|
+
~~
|
194
|
+
|
195
|
+
If the `dependent: :destroy` is removed it no longer creates a transaction, but it also means dependents are not
|
196
|
+
destroyed when a parent is destroyed.
|
197
|
+
|
198
|
+
For this scenario when we are 100% confident no writes are being performed the following can be performed to
|
199
|
+
ignore any attempt Active Record makes at creating the transaction:
|
200
|
+
|
201
|
+
~~ruby
|
202
|
+
ActiveRecordReplica.skip_transactions do
|
203
|
+
parent = Parent.new
|
204
|
+
parent.child = Child.new
|
205
|
+
end
|
206
|
+
~~
|
207
|
+
|
208
|
+
To help identify any code within a block that is creating transactions, wrap the code with
|
209
|
+
`ActiveRecordReplica.block_transactions` to make it raise an exception anytime a transaction is attempted:
|
210
|
+
|
211
|
+
~~ruby
|
212
|
+
ActiveRecordReplica.block_transactions do
|
213
|
+
parent = Parent.new
|
214
|
+
parent.child = Child.new
|
215
|
+
end
|
216
|
+
~~
|
217
|
+
|
218
|
+
## Install
|
219
|
+
|
220
|
+
Add to `Gemfile`
|
221
|
+
|
222
|
+
```ruby
|
223
|
+
gem 'active_record_replica'
|
224
|
+
```
|
225
|
+
|
226
|
+
Run bundler to install:
|
227
|
+
|
228
|
+
```
|
229
|
+
bundle
|
230
|
+
```
|
231
|
+
|
232
|
+
Or, without Bundler:
|
233
|
+
|
234
|
+
```
|
235
|
+
gem install active_record_replica
|
236
|
+
```
|
237
|
+
|
238
|
+
## Configuration
|
239
|
+
|
240
|
+
To enable replica reads for any environment just add a _replica:_ entry to database.yml
|
241
|
+
along with all the usual ActiveRecord database configuration options.
|
242
|
+
|
243
|
+
For Example:
|
244
|
+
|
245
|
+
```yaml
|
246
|
+
production:
|
247
|
+
database: production
|
248
|
+
username: username
|
249
|
+
password: password
|
250
|
+
encoding: utf8
|
251
|
+
adapter: mysql
|
252
|
+
host: primary1
|
253
|
+
pool: 50
|
254
|
+
replica:
|
255
|
+
database: production
|
256
|
+
username: username
|
257
|
+
password: password
|
258
|
+
encoding: utf8
|
259
|
+
adapter: mysql
|
260
|
+
host: replica1
|
261
|
+
pool: 50
|
262
|
+
```
|
263
|
+
|
264
|
+
Sometimes it is useful to turn on replica reads per host, for example to activate
|
265
|
+
replica reads only on the linux host 'batch':
|
266
|
+
|
267
|
+
```yaml
|
268
|
+
production:
|
269
|
+
database: production
|
270
|
+
username: username
|
271
|
+
password: password
|
272
|
+
encoding: utf8
|
273
|
+
adapter: mysql
|
274
|
+
host: primary1
|
275
|
+
pool: 50
|
276
|
+
<% if `hostname`.strip == 'batch' %>
|
277
|
+
replica:
|
278
|
+
database: production
|
279
|
+
username: username
|
280
|
+
password: password
|
281
|
+
encoding: utf8
|
282
|
+
adapter: mysql
|
283
|
+
host: replica1
|
284
|
+
pool: 50
|
285
|
+
<% end %>
|
286
|
+
```
|
287
|
+
|
288
|
+
If there are multiple replicas, it is possible to randomly select a replica on startup
|
289
|
+
to balance the load across the replicas:
|
290
|
+
|
291
|
+
```yaml
|
292
|
+
production:
|
293
|
+
database: production
|
294
|
+
username: username
|
295
|
+
password: password
|
296
|
+
encoding: utf8
|
297
|
+
adapter: mysql
|
298
|
+
host: primary1
|
299
|
+
pool: 50
|
300
|
+
replica:
|
301
|
+
database: production
|
302
|
+
username: username
|
303
|
+
password: password
|
304
|
+
encoding: utf8
|
305
|
+
adapter: mysql
|
306
|
+
host: <%= %w(replica1 replica2 replica3).sample %>
|
307
|
+
pool: 50
|
308
|
+
```
|
309
|
+
|
310
|
+
Replicas can also be assigned to specific hosts by using the hostname:
|
311
|
+
|
312
|
+
```yaml
|
313
|
+
production:
|
314
|
+
database: production
|
315
|
+
username: username
|
316
|
+
password: password
|
317
|
+
encoding: utf8
|
318
|
+
adapter: mysql
|
319
|
+
host: primary1
|
320
|
+
pool: 50
|
321
|
+
replica:
|
322
|
+
database: production
|
323
|
+
username: username
|
324
|
+
password: password
|
325
|
+
encoding: utf8
|
326
|
+
adapter: mysql
|
327
|
+
host: <%= `hostname`.strip == 'app1' ? 'replica1' : 'replica2' %>
|
328
|
+
pool: 50
|
329
|
+
```
|
330
|
+
|
331
|
+
## Set primary as default for Read
|
332
|
+
|
333
|
+
The default behavior can also set to read/write operations against primary database.
|
334
|
+
|
335
|
+
Create an initializer file config/initializer/active_record_replica.rb to force read from primary:
|
336
|
+
|
337
|
+
```yaml
|
338
|
+
ActiveRecordReplica.read_from_primary!
|
339
|
+
```
|
340
|
+
|
341
|
+
Then use this method and supply block to read from the replica database:
|
342
|
+
|
343
|
+
```yaml
|
344
|
+
ActiveRecordReplica.read_from_replica do
|
345
|
+
User.count
|
346
|
+
end
|
347
|
+
```
|
348
|
+
|
349
|
+
## Dependencies
|
350
|
+
|
351
|
+
See [.travis.yml](https://github.com/reidmorrison/active_record_replica/blob/master/.travis.yml) for the list of tested Ruby platforms
|
352
|
+
|
353
|
+
## Versioning
|
354
|
+
|
355
|
+
This project uses [Semantic Versioning](http://semver.org/).
|
356
|
+
|
357
|
+
## Contributing
|
358
|
+
|
359
|
+
1. Fork repository in Github.
|
360
|
+
|
361
|
+
2. Checkout your forked repository:
|
362
|
+
|
363
|
+
```bash
|
364
|
+
git clone https://github.com/your_github_username/active_record_replica.git
|
365
|
+
cd active_record_replica
|
366
|
+
```
|
367
|
+
|
368
|
+
3. Create branch for your contribution:
|
369
|
+
|
370
|
+
```bash
|
371
|
+
git co -b your_new_branch_name
|
372
|
+
```
|
373
|
+
|
374
|
+
4. Make code changes.
|
375
|
+
|
376
|
+
5. Ensure tests pass.
|
377
|
+
|
378
|
+
6. Push to your fork origin.
|
379
|
+
|
380
|
+
```bash
|
381
|
+
git push origin
|
382
|
+
```
|
383
|
+
|
384
|
+
7. Submit PR from the branch on your fork in Github.
|
385
|
+
|
386
|
+
## Author
|
387
|
+
|
388
|
+
[Reid Morrison](https://github.com/reidmorrison) :: @reidmorrison
|
@@ -2,13 +2,6 @@
|
|
2
2
|
# ActiveRecord read from a replica
|
3
3
|
#
|
4
4
|
module ActiveRecordReplica
|
5
|
-
# Select Methods
|
6
|
-
SELECT_METHODS = [:select, :select_all, :select_one, :select_rows, :select_value, :select_values]
|
7
|
-
|
8
|
-
# In case in the future we are forced to intercept connection#execute if the
|
9
|
-
# above select methods are not sufficient
|
10
|
-
# SQL_READS = /\A\s*(SELECT|WITH|SHOW|CALL|EXPLAIN|DESCRIBE)/i
|
11
|
-
|
12
5
|
# Install ActiveRecord::Replica into ActiveRecord to redirect reads to the replica
|
13
6
|
# Parameters:
|
14
7
|
# adapter_class:
|
@@ -19,35 +12,32 @@ module ActiveRecordReplica
|
|
19
12
|
# In a non-Rails environment, supply the environment such as
|
20
13
|
# 'development', 'production'
|
21
14
|
def self.install!(adapter_class = nil, environment = nil)
|
22
|
-
replica_config =
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
base = adapter_class || ActiveRecord::Base.connection.class
|
34
|
-
base.include(Extensions)
|
35
|
-
else
|
36
|
-
ActiveRecord::Base.logger.info "ActiveRecordReplica not installed since no replica database defined"
|
15
|
+
replica_config = ActiveRecord::Base.configurations[environment || Rails.env]["replica"]
|
16
|
+
unless replica_config
|
17
|
+
ActiveRecord::Base.logger.info("ActiveRecordReplica not installed since no replica database defined")
|
18
|
+
return
|
19
|
+
end
|
20
|
+
|
21
|
+
# When the DBMS is not available, an exception (e.g. PG::ConnectionBad) is raised
|
22
|
+
active_db_connection = ActiveRecord::Base.connection.active? rescue false
|
23
|
+
unless active_db_connection
|
24
|
+
ActiveRecord::Base.logger.info("ActiveRecord not connected so not installing ActiveRecordReplica")
|
25
|
+
return
|
37
26
|
end
|
27
|
+
|
28
|
+
version = ActiveRecordReplica::VERSION
|
29
|
+
ActiveRecord::Base.logger.info("ActiveRecordReplica.install! v#{version} Establishing connection to replica database")
|
30
|
+
Replica.establish_connection(replica_config)
|
31
|
+
|
32
|
+
# Inject a new #select method into the ActiveRecord Database adapter
|
33
|
+
base = adapter_class || ActiveRecord::Base.connection.class
|
34
|
+
base.include(Extensions)
|
38
35
|
end
|
39
36
|
|
40
37
|
# Force reads for the supplied block to read from the primary database
|
41
38
|
# Only applies to calls made within the current thread
|
42
|
-
def self.read_from_primary
|
43
|
-
|
44
|
-
begin
|
45
|
-
previous = Thread.current.thread_variable_get(:active_record_replica)
|
46
|
-
read_from_primary!
|
47
|
-
yield
|
48
|
-
ensure
|
49
|
-
Thread.current.thread_variable_set(:active_record_replica, previous)
|
50
|
-
end
|
39
|
+
def self.read_from_primary(&block)
|
40
|
+
thread_variable_yield(:active_record_replica, :primary, &block)
|
51
41
|
end
|
52
42
|
|
53
43
|
#
|
@@ -56,71 +46,63 @@ module ActiveRecordReplica
|
|
56
46
|
# and set ActiveRecordReplica.read_from_primary! to force read from primary.
|
57
47
|
# Then use this method and supply block to read from the replica database
|
58
48
|
# Only applies to calls made within the current thread
|
59
|
-
def self.read_from_replica
|
60
|
-
|
61
|
-
begin
|
62
|
-
previous = Thread.current.thread_variable_get(:active_record_replica)
|
63
|
-
read_from_replica!
|
64
|
-
yield
|
65
|
-
ensure
|
66
|
-
Thread.current.thread_variable_set(:active_record_replica, previous)
|
67
|
-
end
|
49
|
+
def self.read_from_replica(&block)
|
50
|
+
thread_variable_yield(:active_record_replica, :replica, &block)
|
68
51
|
end
|
69
52
|
|
70
53
|
# When only reading from a replica it is important to prevent entering any into
|
71
54
|
# a transaction since the transaction still sends traffic to the primary
|
72
55
|
# that will cause the primary database to slow down processing empty transactions.
|
73
56
|
def self.block_transactions
|
74
|
-
|
75
|
-
previous = Thread.current.thread_variable_get(:active_record_replica_transaction)
|
76
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, :block)
|
77
|
-
yield
|
78
|
-
ensure
|
79
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, previous)
|
80
|
-
end
|
57
|
+
thread_variable_yield(:active_record_replica_transaction, :block, &block)
|
81
58
|
end
|
82
59
|
|
83
60
|
# During this block any attempts to start or end transactions will be ignored.
|
84
61
|
# This extreme action should only be taken when 100% certain no writes are going to be
|
85
62
|
# performed.
|
86
63
|
def self.skip_transactions
|
87
|
-
|
88
|
-
previous = Thread.current.thread_variable_get(:active_record_replica_transaction)
|
89
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, :skip)
|
90
|
-
yield
|
91
|
-
ensure
|
92
|
-
Thread.current.thread_variable_set(:active_record_replica_transaction, previous)
|
93
|
-
end
|
64
|
+
thread_variable_yield(:active_record_replica_transaction, :skip, &block)
|
94
65
|
end
|
95
66
|
|
96
67
|
# Whether this thread is currently forcing all reads to go against the primary database
|
97
68
|
def self.read_from_primary?
|
98
|
-
|
69
|
+
!read_from_replica?
|
99
70
|
end
|
100
71
|
|
101
72
|
# Whether this thread is currently forcing all reads to go against the replica database
|
102
73
|
def self.read_from_replica?
|
103
|
-
Thread.current.thread_variable_get(:active_record_replica)
|
74
|
+
case Thread.current.thread_variable_get(:active_record_replica)
|
75
|
+
when :primary
|
76
|
+
false
|
77
|
+
when :replica
|
78
|
+
true
|
79
|
+
else
|
80
|
+
@read_from_replica
|
81
|
+
end
|
104
82
|
end
|
105
83
|
|
106
|
-
# Force all subsequent reads
|
84
|
+
# Force all subsequent reads in this process to read from the primary database.
|
85
|
+
#
|
86
|
+
# The default behavior can be set to read/write operations against primary.
|
87
|
+
# Create an initializer file config/initializer/active_record_replica.rb
|
88
|
+
# and set ActiveRecordReplica.read_from_primary! to force read from primary.
|
107
89
|
def self.read_from_primary!
|
108
|
-
|
90
|
+
@read_from_replica = false
|
109
91
|
end
|
110
92
|
|
111
|
-
#
|
93
|
+
# Force all subsequent reads in this process to read from the replica database.
|
112
94
|
def self.read_from_replica!
|
113
|
-
|
95
|
+
@read_from_replica = true
|
114
96
|
end
|
115
97
|
|
116
98
|
# Whether any attempt to start a transaction should result in an exception
|
117
99
|
def self.block_transactions?
|
118
|
-
|
100
|
+
thread_variable_equals(:active_record_replica_transaction, :block)
|
119
101
|
end
|
120
102
|
|
121
103
|
# Whether any attempt to start a transaction should be skipped.
|
122
104
|
def self.skip_transactions?
|
123
|
-
|
105
|
+
thread_variable_equals(:active_record_replica_transaction, :skip)
|
124
106
|
end
|
125
107
|
|
126
108
|
# Returns whether replica reads are ignoring transactions
|
@@ -135,5 +117,24 @@ module ActiveRecordReplica
|
|
135
117
|
|
136
118
|
private
|
137
119
|
|
120
|
+
def self.thread_variable_equals(key, value)
|
121
|
+
Thread.current.thread_variable_get(key) == value
|
122
|
+
end
|
123
|
+
|
124
|
+
# Sets the thread variable for the duration of the supplied block.
|
125
|
+
# Restores the previous value on completion of the block.
|
126
|
+
def self.thread_variable_yield(key, new_value)
|
127
|
+
previous = Thread.current.thread_variable_get(key)
|
128
|
+
return yield if previous == new_value
|
129
|
+
|
130
|
+
begin
|
131
|
+
Thread.current.thread_variable_set(key, new_value)
|
132
|
+
yield
|
133
|
+
ensure
|
134
|
+
Thread.current.thread_variable_set(key, previous)
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
138
|
@ignore_transactions = false
|
139
|
+
@read_from_replica = true
|
139
140
|
end
|
@@ -3,7 +3,7 @@ module ActiveRecordReplica
|
|
3
3
|
module Extensions
|
4
4
|
extend ActiveSupport::Concern
|
5
5
|
|
6
|
-
|
6
|
+
[:select, :select_all, :select_one, :select_rows, :select_value, :select_values].each do |select_method|
|
7
7
|
class_eval <<-RUBY, __FILE__, __LINE__ + 1
|
8
8
|
def #{select_method}(sql, name = nil, *args)
|
9
9
|
return super if active_record_replica_read_from_primary?
|
@@ -1,93 +1,44 @@
|
|
1
|
-
|
2
|
-
require 'logger'
|
3
|
-
require 'erb'
|
4
|
-
|
5
|
-
l = Logger.new('test.log')
|
6
|
-
l.level = ::Logger::DEBUG
|
7
|
-
ActiveRecord::Base.logger = l
|
8
|
-
ActiveRecord::Base.configurations = YAML::load(ERB.new(IO.read('test/database.yml')).result)
|
9
|
-
|
10
|
-
# Define Schema in second database (replica)
|
11
|
-
# Note: This is not be required when the primary database is being replicated to the replica db
|
12
|
-
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['test']['replica'])
|
13
|
-
|
14
|
-
# Create table users in database active_record_replica_test
|
15
|
-
ActiveRecord::Schema.define :version => 0 do
|
16
|
-
create_table :users, :force => true do |t|
|
17
|
-
t.string :name
|
18
|
-
t.string :address
|
19
|
-
end
|
20
|
-
end
|
21
|
-
|
22
|
-
# Define Schema in primary database
|
23
|
-
ActiveRecord::Base.establish_connection(:test)
|
1
|
+
# frozen_string_literal: true
|
24
2
|
|
25
|
-
|
26
|
-
ActiveRecord::Schema.define :version => 0 do
|
27
|
-
create_table :users, :force => true do |t|
|
28
|
-
t.string :name
|
29
|
-
t.string :address
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
|
-
# AR Model
|
34
|
-
class User < ActiveRecord::Base
|
35
|
-
end
|
36
|
-
|
37
|
-
# Install ActiveRecord replica. Done automatically by railtie in a Rails environment
|
38
|
-
# Also tell it to use the test environment since Rails.env is not available
|
39
|
-
ActiveRecordReplica.install!(nil, 'test')
|
3
|
+
require_relative "test_helper"
|
40
4
|
|
41
5
|
#
|
42
6
|
# Unit Test for active_record_replica
|
43
7
|
#
|
44
|
-
# Since
|
8
|
+
# Since there is no database replication in this test environment, it will
|
45
9
|
# use 2 separate databases. Writes go to the first database and reads to the second.
|
46
10
|
# As a result any writes to the first database will not be visible when trying to read from
|
47
11
|
# the second test database.
|
12
|
+
#
|
13
|
+
# The tests verify that reads going to the replica database do not find data written to the primary.
|
48
14
|
class ActiveRecordReplicaTest < Minitest::Test
|
49
|
-
describe
|
15
|
+
describe ActiveRecordReplica do
|
16
|
+
let(:user_name) { "Joe Bloggs" }
|
17
|
+
let(:address) { "Somewhere" }
|
18
|
+
let(:user) { User.new(name: user_name, address: address) }
|
50
19
|
|
51
20
|
before do
|
52
21
|
ActiveRecordReplica.ignore_transactions = false
|
53
|
-
|
54
|
-
User.delete_all
|
55
|
-
|
56
|
-
@name = "Joe Bloggs"
|
57
|
-
@address = "Somewhere"
|
58
|
-
@user = User.new(
|
59
|
-
:name => @name,
|
60
|
-
:address => @address
|
61
|
-
)
|
62
|
-
end
|
63
|
-
|
64
|
-
after do
|
22
|
+
ActiveRecordReplica.read_from_replica!
|
65
23
|
User.delete_all
|
66
24
|
end
|
67
25
|
|
68
|
-
it
|
69
|
-
|
26
|
+
it "saves to primary" do
|
27
|
+
user.save!
|
70
28
|
end
|
71
29
|
|
72
|
-
|
73
|
-
# NOTE:
|
74
|
-
#
|
75
|
-
# There is no automated replication between the SQL lite databases
|
76
|
-
# so the tests will be verifying that reads going to the "replica" (second)
|
77
|
-
# database do not find data written to the primary.
|
78
|
-
#
|
79
|
-
it 'saves to primary, read from replica' do
|
30
|
+
it "saves to primary, read from replica" do
|
80
31
|
# Read from replica
|
81
|
-
assert_equal 0, User.where(:
|
32
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
82
33
|
|
83
34
|
# Write to primary
|
84
|
-
|
35
|
+
user.save!
|
85
36
|
|
86
37
|
# Read from replica
|
87
|
-
assert_equal 0, User.where(:
|
38
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
88
39
|
end
|
89
40
|
|
90
|
-
it
|
41
|
+
it "save to primary, read from primary when in a transaction" do
|
91
42
|
assert_equal false, ActiveRecordReplica.ignore_transactions?
|
92
43
|
|
93
44
|
User.transaction do
|
@@ -95,56 +46,115 @@ class ActiveRecordReplicaTest < Minitest::Test
|
|
95
46
|
assert_equal 0, User.count
|
96
47
|
|
97
48
|
# Read from Primary
|
98
|
-
assert_equal 0, User.where(:
|
49
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
99
50
|
|
100
51
|
# Write to primary
|
101
|
-
|
52
|
+
user.save!
|
102
53
|
|
103
54
|
# Read from Primary
|
104
|
-
assert_equal 1, User.where(:
|
55
|
+
assert_equal 1, User.where(name: user_name, address: address).count
|
105
56
|
end
|
106
57
|
|
107
58
|
# Read from Non-replicated replica
|
108
|
-
assert_equal 0, User.where(:
|
59
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
109
60
|
end
|
110
61
|
|
111
|
-
it
|
62
|
+
it "save to primary, read from replica when ignoring transactions" do
|
112
63
|
ActiveRecordReplica.ignore_transactions = true
|
113
|
-
|
64
|
+
assert ActiveRecordReplica.ignore_transactions?
|
114
65
|
|
115
66
|
User.transaction do
|
116
67
|
# The delete_all in setup should have cleared the table
|
117
68
|
assert_equal 0, User.count
|
118
69
|
|
119
70
|
# Read from Primary
|
120
|
-
assert_equal 0, User.where(:
|
71
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
121
72
|
|
122
73
|
# Write to primary
|
123
|
-
|
74
|
+
user.save!
|
124
75
|
|
125
76
|
# Read from Non-replicated replica
|
126
|
-
assert_equal 0, User.where(:
|
77
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
127
78
|
end
|
128
79
|
|
129
80
|
# Read from Non-replicated replica
|
130
|
-
assert_equal 0, User.where(:
|
81
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
131
82
|
end
|
132
83
|
|
133
|
-
it
|
84
|
+
it "saves to primary, force a read from primary even when _not_ in a transaction" do
|
134
85
|
# Read from replica
|
135
|
-
assert_equal 0, User.where(:
|
86
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
136
87
|
|
137
88
|
# Write to primary
|
138
|
-
|
89
|
+
user.save!
|
139
90
|
|
140
91
|
# Read from replica
|
141
|
-
assert_equal 0, User.where(:
|
92
|
+
assert_equal 0, User.where(name: user_name, address: address).count
|
142
93
|
|
143
94
|
# Read from Primary
|
144
95
|
ActiveRecordReplica.read_from_primary do
|
145
|
-
assert_equal 1, User.where(:
|
96
|
+
assert_equal 1, User.where(name: user_name, address: address).count
|
146
97
|
end
|
147
98
|
end
|
148
99
|
|
100
|
+
describe ".read_from_replica?" do
|
101
|
+
it "is true when global replica flag is set" do
|
102
|
+
ActiveRecordReplica.read_from_replica!
|
103
|
+
assert ActiveRecordReplica.read_from_replica?
|
104
|
+
end
|
105
|
+
|
106
|
+
it "is false when reading from replica" do
|
107
|
+
ActiveRecordReplica.read_from_primary!
|
108
|
+
refute ActiveRecordReplica.read_from_replica?
|
109
|
+
end
|
110
|
+
end
|
111
|
+
|
112
|
+
describe ".read_from_primary?" do
|
113
|
+
it "is true when global primary flag is set" do
|
114
|
+
ActiveRecordReplica.read_from_primary!
|
115
|
+
assert ActiveRecordReplica.read_from_primary?
|
116
|
+
end
|
117
|
+
|
118
|
+
it "is false when reading from replica" do
|
119
|
+
ActiveRecordReplica.read_from_replica!
|
120
|
+
refute ActiveRecordReplica.read_from_primary?
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
124
|
+
describe ".read_from_replica" do
|
125
|
+
it "works with global replica flag" do
|
126
|
+
ActiveRecordReplica.read_from_replica!
|
127
|
+
ActiveRecordReplica.read_from_replica do
|
128
|
+
assert ActiveRecordReplica.read_from_replica?
|
129
|
+
refute ActiveRecordReplica.read_from_primary?
|
130
|
+
end
|
131
|
+
end
|
132
|
+
|
133
|
+
it "overwrites global replica flag" do
|
134
|
+
ActiveRecordReplica.read_from_primary!
|
135
|
+
ActiveRecordReplica.read_from_replica do
|
136
|
+
assert ActiveRecordReplica.read_from_replica?
|
137
|
+
refute ActiveRecordReplica.read_from_primary?
|
138
|
+
end
|
139
|
+
end
|
140
|
+
end
|
141
|
+
|
142
|
+
describe ".read_from_primary" do
|
143
|
+
it "works with global replica flag" do
|
144
|
+
ActiveRecordReplica.read_from_primary!
|
145
|
+
ActiveRecordReplica.read_from_primary do
|
146
|
+
assert ActiveRecordReplica.read_from_primary?
|
147
|
+
refute ActiveRecordReplica.read_from_replica?
|
148
|
+
end
|
149
|
+
end
|
150
|
+
|
151
|
+
it "overwrites global replica flag" do
|
152
|
+
ActiveRecordReplica.read_from_replica!
|
153
|
+
ActiveRecordReplica.read_from_primary do
|
154
|
+
assert ActiveRecordReplica.read_from_primary?
|
155
|
+
refute ActiveRecordReplica.read_from_replica?
|
156
|
+
end
|
157
|
+
end
|
158
|
+
end
|
149
159
|
end
|
150
160
|
end
|
data/test/database.yml
CHANGED
@@ -3,7 +3,7 @@ test:
|
|
3
3
|
database: test/test.sqlite3
|
4
4
|
pool: 5
|
5
5
|
timeout: 5000
|
6
|
-
# Make the replica a separate database that is not
|
6
|
+
# Make the replica a separate database that is not replicated to ensure reads
|
7
7
|
# and writes go to the appropriate databases
|
8
8
|
replica:
|
9
9
|
adapter: sqlite3
|
data/test/test_helper.rb
CHANGED
@@ -4,3 +4,41 @@ require 'active_record'
|
|
4
4
|
require 'minitest/autorun'
|
5
5
|
require 'active_record_replica'
|
6
6
|
require 'awesome_print'
|
7
|
+
require 'logger'
|
8
|
+
require 'erb'
|
9
|
+
|
10
|
+
l = Logger.new('test.log')
|
11
|
+
l.level = ::Logger::DEBUG
|
12
|
+
ActiveRecord::Base.logger = l
|
13
|
+
ActiveRecord::Base.configurations = YAML::load(ERB.new(IO.read('test/database.yml')).result)
|
14
|
+
|
15
|
+
# Define Schema in second database (replica)
|
16
|
+
# Note: This is not be required when the primary database is being replicated to the replica db
|
17
|
+
ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['test']['replica'])
|
18
|
+
|
19
|
+
# Create table users in database active_record_replica_test
|
20
|
+
ActiveRecord::Schema.define :version => 0 do
|
21
|
+
create_table :users, :force => true do |t|
|
22
|
+
t.string :name
|
23
|
+
t.string :address
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
# Define Schema in primary database
|
28
|
+
ActiveRecord::Base.establish_connection(:test)
|
29
|
+
|
30
|
+
# Create table users in database active_record_replica_test
|
31
|
+
ActiveRecord::Schema.define :version => 0 do
|
32
|
+
create_table :users, :force => true do |t|
|
33
|
+
t.string :name
|
34
|
+
t.string :address
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
# AR Model
|
39
|
+
class User < ActiveRecord::Base
|
40
|
+
end
|
41
|
+
|
42
|
+
# Install ActiveRecord replica. Done automatically by railtie in a Rails environment
|
43
|
+
# Also tell it to use the test environment since Rails.env is not available
|
44
|
+
ActiveRecordReplica.install!(nil, 'test')
|
metadata
CHANGED
@@ -1,14 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: active_record_replica
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 3.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Reid Morrison
|
8
|
-
|
8
|
+
- James Brady
|
9
|
+
autorequire:
|
9
10
|
bindir: bin
|
10
11
|
cert_chain: []
|
11
|
-
date:
|
12
|
+
date: 2021-03-18 00:00:00.000000000 Z
|
12
13
|
dependencies:
|
13
14
|
- !ruby/object:Gem::Dependency
|
14
15
|
name: activerecord
|
@@ -30,9 +31,10 @@ dependencies:
|
|
30
31
|
- - "<"
|
31
32
|
- !ruby/object:Gem::Version
|
32
33
|
version: 6.0.0
|
33
|
-
description:
|
34
|
+
description:
|
34
35
|
email:
|
35
36
|
- reidmo@gmail.com
|
37
|
+
- james@spri.ng
|
36
38
|
executables: []
|
37
39
|
extensions: []
|
38
40
|
extra_rdoc_files: []
|
@@ -49,14 +51,12 @@ files:
|
|
49
51
|
- lib/active_record_replica/version.rb
|
50
52
|
- test/active_record_replica_test.rb
|
51
53
|
- test/database.yml
|
52
|
-
- test/test.sqlite3
|
53
54
|
- test/test_helper.rb
|
54
|
-
|
55
|
-
homepage: https://github.com/rocketjob/active_record_replica
|
55
|
+
homepage: https://github.com/teespring/active_record_replica
|
56
56
|
licenses:
|
57
57
|
- Apache-2.0
|
58
58
|
metadata: {}
|
59
|
-
post_install_message:
|
59
|
+
post_install_message:
|
60
60
|
rdoc_options: []
|
61
61
|
require_paths:
|
62
62
|
- lib
|
@@ -71,14 +71,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
71
71
|
- !ruby/object:Gem::Version
|
72
72
|
version: '0'
|
73
73
|
requirements: []
|
74
|
-
|
75
|
-
|
74
|
+
rubyforge_project:
|
75
|
+
rubygems_version: 2.6.14.3
|
76
|
+
signing_key:
|
76
77
|
specification_version: 4
|
77
78
|
summary: Redirect ActiveRecord (Rails) reads to replica databases while ensuring all
|
78
79
|
writes go to the primary database.
|
79
80
|
test_files:
|
80
81
|
- test/database.yml
|
81
82
|
- test/test_helper.rb
|
82
|
-
- test/test.sqlite3
|
83
83
|
- test/active_record_replica_test.rb
|
84
|
-
- test/test_replica.sqlite3
|
data/test/test.sqlite3
DELETED
Binary file
|
data/test/test_replica.sqlite3
DELETED
Binary file
|