thanx-knockoff 1.1.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +10 -0
- data/.rspec +2 -0
- data/.travis.yml +8 -0
- data/CHANGELOG.md +10 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +21 -0
- data/README.md +204 -0
- data/Rakefile +6 -0
- data/bin/console +14 -0
- data/bin/setup +7 -0
- data/ci/install_modern_sqlite.sh +13 -0
- data/knockoff.gemspec +28 -0
- data/lib/knockoff.rb +65 -0
- data/lib/knockoff/active_record/base.rb +29 -0
- data/lib/knockoff/active_record/relation.rb +28 -0
- data/lib/knockoff/base.rb +43 -0
- data/lib/knockoff/config.rb +113 -0
- data/lib/knockoff/error.rb +4 -0
- data/lib/knockoff/replica_connection_pool.rb +58 -0
- data/lib/knockoff/version.rb +3 -0
- metadata +148 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 2e4a9c23826fcfe014d7664d2b77b099356b4c8e1fefff23f2c50a90311c224c
|
4
|
+
data.tar.gz: 37d81e202e622fcd79d203339e4f0c4c705f52beb2d53ab73c9235fdf2833d33
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: b589d8997c9ce7f627c82ffa05f147448d8a19f95650c5b53f0b287ec17a7efa39b83e918de0cbcd57a3b312151e877d4f0797c847b4ec8691fb3461dc9344a5
|
7
|
+
data.tar.gz: a33d6341c20bb278a0357e0eabbc14f6639ba6a5bf5af3f1c387402ba207745cc4d091e8a5cab7d75968bc0949d284f250db199f267006468fb66221c516b028
|
data/.gitignore
ADDED
data/.rspec
ADDED
data/.travis.yml
ADDED
data/CHANGELOG.md
ADDED
@@ -0,0 +1,10 @@
|
|
1
|
+
## Unreleased
|
2
|
+
|
3
|
+
*None*
|
4
|
+
|
5
|
+
## 1.1.0
|
6
|
+
|
7
|
+
- Allow for not checking `inside_transaction?` on primary database (https://github.com/joinhandshake/knockoff/pull/13)
|
8
|
+
- Allow setting `Knockoff.default_target` to set the default target other than `:primary` (https://github.com/joinhandshake/knockoff/pull/11)
|
9
|
+
- Drop Ruby 2.2 support
|
10
|
+
- Add Ruby 2.5 support
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2016 Scott Ringwelski
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,204 @@
|
|
1
|
+
# Knockoff
|
2
|
+
|
3
|
+
[![Build Status](https://travis-ci.org/joinhandshake/knockoff.svg?branch=master)](https://travis-ci.org/joinhandshake/knockoff)
|
4
|
+
[![Gem Version](https://badge.fury.io/rb/knockoff.svg)](https://badge.fury.io/rb/knockoff)
|
5
|
+
|
6
|
+
A gem for easily using read replicas.
|
7
|
+
|
8
|
+
:handshake: Battle tested at [Handshake](https://joinhandshake.com/engineering/)
|
9
|
+
|
10
|
+
## Library Goals
|
11
|
+
|
12
|
+
* Minimal ActiveRecord monkey-patching
|
13
|
+
* Easy run-time configuration using ENV variables
|
14
|
+
* Opt-in usage of replicas
|
15
|
+
* No need to change code when adding/removing replicas
|
16
|
+
* Be thread safe
|
17
|
+
|
18
|
+
## Supported Versions
|
19
|
+
|
20
|
+
Knockoff supports Rails 4 and 5
|
21
|
+
|
22
|
+
## Installation
|
23
|
+
|
24
|
+
Add this line to your application's Gemfile:
|
25
|
+
|
26
|
+
```ruby
|
27
|
+
gem 'knockoff'
|
28
|
+
```
|
29
|
+
|
30
|
+
And then execute:
|
31
|
+
|
32
|
+
$ bundle
|
33
|
+
|
34
|
+
Or install it yourself as:
|
35
|
+
|
36
|
+
$ gem install knockoff
|
37
|
+
|
38
|
+
## Usage
|
39
|
+
|
40
|
+
### Initializer
|
41
|
+
|
42
|
+
Add an initializer at config/knockoff.rb with the below contents
|
43
|
+
|
44
|
+
```
|
45
|
+
Knockoff.enabled = true # NOTE: Consider adding ENV based disabling
|
46
|
+
```
|
47
|
+
|
48
|
+
### Configuration
|
49
|
+
|
50
|
+
Configuration is done using ENV properties. This makes it easy to add and remove replicas at runtime (or to fully disable if needed). First, set up ENV variables pointing to your replica databases. Consider using the [dotenv](https://github.com/bkeepers/dotenv) gem for manging ENV variables.
|
51
|
+
|
52
|
+
```
|
53
|
+
# .env
|
54
|
+
|
55
|
+
REPLICA_1=postgres://username:password@localhost:5432/database_name
|
56
|
+
```
|
57
|
+
|
58
|
+
The second ENV variable to set is `KNOCKOFF_REPLICA_ENVS` which is a comma-separated list of ENVS holding database URLs to use as replicas. In this case, the ENV would be set as follows.
|
59
|
+
|
60
|
+
```
|
61
|
+
# .env
|
62
|
+
|
63
|
+
KNOCKOFF_REPLICA_ENVS=REPLICA_1
|
64
|
+
```
|
65
|
+
|
66
|
+
Note that it can be multiple replicas, and `knockoff` will use both evenly:
|
67
|
+
|
68
|
+
```
|
69
|
+
KNOCKOFF_REPLICA_ENVS=REPLICA_1,REPLICA_2
|
70
|
+
```
|
71
|
+
|
72
|
+
Lastly, knockoff will read the `'knockoff_replicas'` database.yml config for specifying additional params:
|
73
|
+
|
74
|
+
```
|
75
|
+
# database.yml
|
76
|
+
|
77
|
+
knockoff_replicas:
|
78
|
+
<<: *common
|
79
|
+
prepared_statements: false
|
80
|
+
```
|
81
|
+
|
82
|
+
### Basics
|
83
|
+
|
84
|
+
To use one of the replica databases, use
|
85
|
+
|
86
|
+
```
|
87
|
+
Knockoff.on_replica { User.count }
|
88
|
+
```
|
89
|
+
|
90
|
+
To force primary, use
|
91
|
+
|
92
|
+
```
|
93
|
+
Knockoff.on_primary { User.create(name: 'Bob') }
|
94
|
+
```
|
95
|
+
|
96
|
+
### Using in Controllers
|
97
|
+
|
98
|
+
A common use case is to use replicas for GET requests and otherwise use primary. A simplified use case might look something like this:
|
99
|
+
|
100
|
+
```
|
101
|
+
# application_controller.rb
|
102
|
+
|
103
|
+
around_action :choose_database
|
104
|
+
|
105
|
+
def choose_database(&block)
|
106
|
+
if should_use_primary_database?
|
107
|
+
Knockoff.on_primary(&block)
|
108
|
+
else
|
109
|
+
Knockoff.on_replica(&block)
|
110
|
+
end
|
111
|
+
end
|
112
|
+
|
113
|
+
def should_use_primary_database?
|
114
|
+
request.method_symbol != :get
|
115
|
+
end
|
116
|
+
|
117
|
+
```
|
118
|
+
|
119
|
+
#### Replication Lag
|
120
|
+
|
121
|
+
Replicas will often be slightly behind the primary database. To compensate, consider "sticking" a user who has recently made changes to the primary for a small duration of time to the primary database. This will avoid cases where a user creates a record on primary, is redirected to view that record, and receives a 404 error since the record is not yet in the replica. A simple implementation for this could look like:
|
122
|
+
|
123
|
+
```
|
124
|
+
# application_record.rb
|
125
|
+
|
126
|
+
after_commit :track_commit_occurred_in_request
|
127
|
+
|
128
|
+
# If any commit happens in a request, we record that and have the logged_in_user
|
129
|
+
# read from primary for a short period of time.
|
130
|
+
def track_commit_occurred_in_request
|
131
|
+
RequestLocals.store['commit_occurred_in_current_request'] = true
|
132
|
+
end
|
133
|
+
|
134
|
+
# application_controller.rb
|
135
|
+
|
136
|
+
after_action :force_leader_if_commit
|
137
|
+
|
138
|
+
def force_leader_if_commit
|
139
|
+
if RequestLocals.store['commit_occurred_in_current_request'].to_b
|
140
|
+
session[:use_leader_until] = Time.current + FORCE_PRIMARY_DURATION
|
141
|
+
end
|
142
|
+
end
|
143
|
+
|
144
|
+
```
|
145
|
+
|
146
|
+
Then, in your `should_use_primary_database?` method, consult `session[:use_leader_until]` for the decision.
|
147
|
+
|
148
|
+
### Run-time Configuration
|
149
|
+
|
150
|
+
Knockoff can be configured during runtime. This is done through the `establish_new_connections!` method which takes in a hash of new configurations to apply to each replica before re-connecting.
|
151
|
+
|
152
|
+
```
|
153
|
+
Knockoff.establish_new_connections!({ 'pool' => db_pool })
|
154
|
+
```
|
155
|
+
|
156
|
+
For example, to specify a puma connection pool at bootup your code might look something like
|
157
|
+
|
158
|
+
```
|
159
|
+
# puma.rb
|
160
|
+
|
161
|
+
db_pool = Integer(ENV['PUMA_WORKER_DB_POOL'] || threads_count)
|
162
|
+
# Configure the database connection to have the new pool size and re-establish connection
|
163
|
+
database_config = ActiveRecord::Base.configurations[Rails.env] || Rails.application.config.database_configuration[Rails.env]
|
164
|
+
database_config['pool'] = db_pool
|
165
|
+
ActiveRecord::Base.establish_connection(database_config)
|
166
|
+
Knockoff.establish_new_connections!({ 'pool' => db_pool })
|
167
|
+
|
168
|
+
```
|
169
|
+
|
170
|
+
#### Forking
|
171
|
+
|
172
|
+
For forking servers, you may disconnect all replicas before forking with `Knockoff.disconnect_all!`.
|
173
|
+
|
174
|
+
```
|
175
|
+
# puma.rb
|
176
|
+
|
177
|
+
before_fork do
|
178
|
+
ActiveRecord::Base.connection_pool.disconnect!
|
179
|
+
Knockoff.disconnect_all!
|
180
|
+
end
|
181
|
+
```
|
182
|
+
|
183
|
+
### Other Cases
|
184
|
+
|
185
|
+
There are likely other cases specific to each application where it makes sense to force primary database and avoid replication lag. Good candidates are time-based pages (a live calendar, for example), forms, and payments.
|
186
|
+
|
187
|
+
## Usage Notes
|
188
|
+
|
189
|
+
* Do not use prepared statements with this gem
|
190
|
+
|
191
|
+
## Development
|
192
|
+
|
193
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
194
|
+
|
195
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
196
|
+
|
197
|
+
## Contributing
|
198
|
+
|
199
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/joinhandshake/knockoff.
|
200
|
+
|
201
|
+
## License
|
202
|
+
|
203
|
+
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
204
|
+
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "knockoff"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start
|
data/bin/setup
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
#!/usr/bin/env bash
|
2
|
+
set -e # halt script on error
|
3
|
+
|
4
|
+
sudo apt-get autoremove sqlite3
|
5
|
+
sudo apt-get install python-software-properties
|
6
|
+
sudo apt-add-repository -y ppa:travis-ci/sqlite3
|
7
|
+
sudo apt-get -y update
|
8
|
+
sudo apt-cache show sqlite3
|
9
|
+
sudo apt-get install sqlite3=3.7.15.1-1~travis1
|
10
|
+
sudo sqlite3 -version
|
11
|
+
sudo psql --version
|
12
|
+
sudo mysql --version
|
13
|
+
gem update bundler
|
data/knockoff.gemspec
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
# coding: utf-8
|
2
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require 'knockoff/version'
|
5
|
+
|
6
|
+
Gem::Specification.new do |spec|
|
7
|
+
spec.name = "thanx-knockoff"
|
8
|
+
spec.version = Knockoff::VERSION
|
9
|
+
spec.authors = ["Scott Ringwelski"]
|
10
|
+
spec.email = ["me@sgringwe.com"]
|
11
|
+
|
12
|
+
spec.summary = %q{A gem for easily using read replicas}
|
13
|
+
spec.homepage = "https://github.com/sgringwe/knockoff"
|
14
|
+
spec.license = "MIT"
|
15
|
+
|
16
|
+
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
17
|
+
spec.bindir = "exe"
|
18
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
19
|
+
spec.require_paths = ["lib"]
|
20
|
+
|
21
|
+
spec.add_runtime_dependency 'activerecord', '>= 4.0.0'
|
22
|
+
spec.add_runtime_dependency 'activesupport', '>= 4.0.0'
|
23
|
+
|
24
|
+
spec.add_development_dependency "bundler"
|
25
|
+
spec.add_development_dependency "rake", "~> 10.0"
|
26
|
+
spec.add_development_dependency "rspec"
|
27
|
+
spec.add_development_dependency 'sqlite3'
|
28
|
+
end
|
data/lib/knockoff.rb
ADDED
@@ -0,0 +1,65 @@
|
|
1
|
+
require 'active_record'
|
2
|
+
require 'knockoff/version'
|
3
|
+
require 'knockoff/base'
|
4
|
+
require 'knockoff/config'
|
5
|
+
require 'knockoff/error'
|
6
|
+
require 'knockoff/replica_connection_pool'
|
7
|
+
require 'knockoff/active_record/base'
|
8
|
+
require 'knockoff/active_record/relation'
|
9
|
+
|
10
|
+
module Knockoff
|
11
|
+
class << self
|
12
|
+
attr_accessor :enabled
|
13
|
+
attr_reader :default_target
|
14
|
+
|
15
|
+
def on_replica(check_transaction: true, &block)
|
16
|
+
Base.new(:replica, check_transaction: check_transaction).run(&block)
|
17
|
+
end
|
18
|
+
|
19
|
+
def on_primary(&block)
|
20
|
+
Base.new(:primary).run(&block)
|
21
|
+
end
|
22
|
+
|
23
|
+
def default_target=(target)
|
24
|
+
@default_target = Base.new(target).target
|
25
|
+
end
|
26
|
+
|
27
|
+
def replica_pool
|
28
|
+
@replica_pool ||= ReplicaConnectionPool.new(config.replica_database_keys)
|
29
|
+
end
|
30
|
+
|
31
|
+
def clear_all_active_connections!
|
32
|
+
replica_pool.clear_all_active_connections!
|
33
|
+
end
|
34
|
+
|
35
|
+
# Iterates through the replica pool and calls disconnect on each one's connection.
|
36
|
+
def disconnect_all!
|
37
|
+
replica_pool.disconnect_all_replicas!
|
38
|
+
end
|
39
|
+
|
40
|
+
# Updates the config (both internal representation and the ActiveRecord::Base.configuration)
|
41
|
+
# with the new config, and then reconnects each replica connection in the replica
|
42
|
+
# pool.
|
43
|
+
def establish_new_connections!(new_config)
|
44
|
+
config.update_replica_configs(new_config)
|
45
|
+
replica_pool.reconnect_all_replicas!
|
46
|
+
end
|
47
|
+
|
48
|
+
def config
|
49
|
+
@config ||= Config.new
|
50
|
+
end
|
51
|
+
|
52
|
+
def base_transaction_depth
|
53
|
+
@base_transaction_depth ||= begin
|
54
|
+
testcase = ActiveSupport::TestCase
|
55
|
+
if defined?(testcase) &&
|
56
|
+
testcase.respond_to?(:use_transactional_fixtures) &&
|
57
|
+
testcase.try(:use_transactional_fixtures)
|
58
|
+
1
|
59
|
+
else
|
60
|
+
0
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
@@ -0,0 +1,29 @@
|
|
1
|
+
module ActiveRecord
|
2
|
+
class Base
|
3
|
+
class << self
|
4
|
+
alias_method :original_connection, :connection
|
5
|
+
|
6
|
+
def connection
|
7
|
+
target = Thread.current[:knockoff] || Knockoff.default_target
|
8
|
+
case target
|
9
|
+
when :replica
|
10
|
+
# Attempts to use a random replica connection, but otherwise falls back to primary
|
11
|
+
Knockoff.replica_pool.random_replica_connection.original_connection
|
12
|
+
when :primary, NilClass
|
13
|
+
original_connection
|
14
|
+
else
|
15
|
+
raise Knockoff::Error, "Invalid target: #{Thread.current[:knockoff]}"
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
# Generate scope at top level e.g. User.on_replica
|
20
|
+
def on_replica
|
21
|
+
# Why where(nil)?
|
22
|
+
# http://stackoverflow.com/questions/18198963/with-rails-4-model-scoped-is-deprecated-but-model-all-cant-replace-it
|
23
|
+
context = where(nil)
|
24
|
+
context.knockoff_target = :replica
|
25
|
+
context
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
@@ -0,0 +1,28 @@
|
|
1
|
+
module ActiveRecord
|
2
|
+
class Relation
|
3
|
+
attr_accessor :knockoff_target
|
4
|
+
|
5
|
+
# Supports queries like User.on_replica.to_a
|
6
|
+
alias_method :exec_queries_without_knockoff, :exec_queries
|
7
|
+
|
8
|
+
def exec_queries
|
9
|
+
if knockoff_target == :replica
|
10
|
+
Knockoff.on_replica { exec_queries_without_knockoff }
|
11
|
+
else
|
12
|
+
exec_queries_without_knockoff
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
|
17
|
+
# Supports queries like User.on_replica.count
|
18
|
+
alias_method :calculate_without_knockoff, :calculate
|
19
|
+
|
20
|
+
def calculate(*args)
|
21
|
+
if knockoff_target == :replica
|
22
|
+
Knockoff.on_replica { calculate_without_knockoff(*args) }
|
23
|
+
else
|
24
|
+
calculate_without_knockoff(*args)
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
module Knockoff
|
2
|
+
class Base
|
3
|
+
attr_reader :target
|
4
|
+
|
5
|
+
def initialize(target, check_transaction: true)
|
6
|
+
@target = decide_with(target, check_transaction)
|
7
|
+
end
|
8
|
+
|
9
|
+
def run(&block)
|
10
|
+
run_on @target, &block
|
11
|
+
end
|
12
|
+
|
13
|
+
private
|
14
|
+
|
15
|
+
def decide_with(target, check_transaction)
|
16
|
+
calculated_target =
|
17
|
+
if Knockoff.enabled
|
18
|
+
target
|
19
|
+
else
|
20
|
+
:primary
|
21
|
+
end
|
22
|
+
|
23
|
+
# Don't allow setting the target to anything other than primary if we are already in a transaction
|
24
|
+
if calculated_target != :primary && check_transaction && inside_transaction?
|
25
|
+
raise Knockoff::Error.new('on_replica cannot be used inside transaction block!')
|
26
|
+
end
|
27
|
+
calculated_target
|
28
|
+
end
|
29
|
+
|
30
|
+
def inside_transaction?
|
31
|
+
open_transactions = run_on(:primary) { ActiveRecord::Base.connection.open_transactions }
|
32
|
+
open_transactions > Knockoff.base_transaction_depth
|
33
|
+
end
|
34
|
+
|
35
|
+
def run_on(target)
|
36
|
+
backup = Thread.current[:knockoff] # Save for recursive nested calls
|
37
|
+
Thread.current[:knockoff] = target
|
38
|
+
yield
|
39
|
+
ensure
|
40
|
+
Thread.current[:knockoff] = backup
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,113 @@
|
|
1
|
+
module Knockoff
|
2
|
+
class Config
|
3
|
+
# The current environment. Normally set to Rails.env, but
|
4
|
+
# will default to 'development' outside of Rails apps.
|
5
|
+
attr_reader :environment
|
6
|
+
|
7
|
+
# An array of configs to use for the replica pool.
|
8
|
+
attr_reader :replica_configs
|
9
|
+
|
10
|
+
# A hash of replica configs to their config hash.
|
11
|
+
attr_reader :replicas_configurations
|
12
|
+
|
13
|
+
def initialize
|
14
|
+
@environment = 'development'
|
15
|
+
@replicas_configurations = {}
|
16
|
+
set_replica_configs
|
17
|
+
|
18
|
+
if !properly_configured? && Knockoff.enabled
|
19
|
+
puts "[Knockoff] WARNING: Detected enabled Knockoff without proper replica pool configuration. Setting Knockoff.enabled to false."
|
20
|
+
Knockoff.enabled = false
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
def replica_database_keys
|
25
|
+
@replicas_configurations.keys
|
26
|
+
end
|
27
|
+
|
28
|
+
def replica_env_keys
|
29
|
+
if ENV['KNOCKOFF_REPLICA_ENVS'].nil?
|
30
|
+
[]
|
31
|
+
else
|
32
|
+
ENV['KNOCKOFF_REPLICA_ENVS'].split(',').map(&:strip)
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
def update_replica_configs(new_configs)
|
37
|
+
ActiveRecord::Base.configurations['knockoff_replicas'].merge(new_configs) if ActiveRecord::Base.configurations['knockoff_replicas'].present?
|
38
|
+
@replicas_configurations.each do |key, _config|
|
39
|
+
update_replica_config(key, new_configs)
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
# If replica_configs actually containts some configuration information, then
|
44
|
+
# we know it was properly configured. Improper URI's will be ignored during the
|
45
|
+
# initialization step.
|
46
|
+
def properly_configured?
|
47
|
+
!@replica_configs.empty?
|
48
|
+
end
|
49
|
+
|
50
|
+
private
|
51
|
+
|
52
|
+
def update_replica_config(key, new_configs)
|
53
|
+
@replicas_configurations[key].merge!(new_configs)
|
54
|
+
ActiveRecord::Base.configurations[key].merge!(new_configs)
|
55
|
+
end
|
56
|
+
|
57
|
+
def set_replica_configs
|
58
|
+
@replica_configs ||= parse_knockoff_replica_envs_to_configs
|
59
|
+
end
|
60
|
+
|
61
|
+
def parse_knockoff_replica_envs_to_configs
|
62
|
+
# As a basic prevention of crashes, attempt to parse each DB uri
|
63
|
+
# and don't add the uri to the final list if it can't be parsed
|
64
|
+
replica_env_keys.map.with_index(0) do |env_key, index|
|
65
|
+
begin
|
66
|
+
uri = URI.parse(ENV[env_key])
|
67
|
+
|
68
|
+
# Configure parameters such as prepared_statements, pool, reaping_frequency for all replicas.
|
69
|
+
replica_config = ActiveRecord::Base.configurations['knockoff_replicas'] || {}
|
70
|
+
|
71
|
+
adapter =
|
72
|
+
if uri.scheme == "postgres"
|
73
|
+
'postgresql'
|
74
|
+
else
|
75
|
+
uri.scheme
|
76
|
+
end
|
77
|
+
|
78
|
+
# Base config from the ENV uri. Sqlite is a special case
|
79
|
+
# and all others follow 'normal' config
|
80
|
+
uri_config =
|
81
|
+
if uri.scheme == 'sqlite3'
|
82
|
+
{
|
83
|
+
'adapter' => adapter,
|
84
|
+
'database' => uri.to_s.split(':')[1]
|
85
|
+
}
|
86
|
+
else
|
87
|
+
{
|
88
|
+
'adapter' => adapter,
|
89
|
+
'database' => (uri.path || "").split("/")[1],
|
90
|
+
'username' => uri.user,
|
91
|
+
'password' => uri.password,
|
92
|
+
'host' => uri.host,
|
93
|
+
'port' => uri.port
|
94
|
+
}.merge(Rack::Utils.parse_query(uri.query))
|
95
|
+
end
|
96
|
+
|
97
|
+
# Store the hash in configuration and use it when we establish the connection later.
|
98
|
+
key = "knockoff_replica_#{index}"
|
99
|
+
full_config = replica_config.merge(uri_config)
|
100
|
+
|
101
|
+
ActiveRecord::Base.configurations[key] = full_config
|
102
|
+
@replicas_configurations[key] = full_config
|
103
|
+
rescue URI::InvalidURIError
|
104
|
+
Rails.logger.info "LOG NOTIFIER: Invalid URL specified in follower_env_keys. Not including URI, which may result in no followers used." # URI is purposely not printed to logs
|
105
|
+
# Return a 'nil' which will be removed from
|
106
|
+
# configs with `compact`, resulting in no configs and no followers,
|
107
|
+
# therefore disabled since this env will not be in environments list.
|
108
|
+
nil
|
109
|
+
end
|
110
|
+
end.compact
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
@@ -0,0 +1,58 @@
|
|
1
|
+
module Knockoff
|
2
|
+
class ReplicaConnectionPool
|
3
|
+
attr_reader :pool
|
4
|
+
|
5
|
+
def initialize(config_keys)
|
6
|
+
@pool = Concurrent::Hash.new
|
7
|
+
|
8
|
+
config_keys.each do |config_key|
|
9
|
+
@pool[config_key] = connection_class(config_key)
|
10
|
+
end
|
11
|
+
end
|
12
|
+
|
13
|
+
def clear_all_active_connections!
|
14
|
+
@pool.each do |_name, klass|
|
15
|
+
klass.clear_active_connections!
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
def disconnect_all_replicas!
|
20
|
+
@pool.each do |_name, klass|
|
21
|
+
klass.connection.disconnect!
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
# Assumes that the config has been updated to something new, and
|
26
|
+
# simply reconnects with the config.
|
27
|
+
def reconnect_all_replicas!
|
28
|
+
@pool.each do |database_key, klass|
|
29
|
+
klass.establish_connection database_key.to_sym
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
def random_replica_connection
|
34
|
+
@pool[@pool.keys.sample]
|
35
|
+
end
|
36
|
+
|
37
|
+
# Based off of code from replica_pools gem
|
38
|
+
# generates a unique ActiveRecord::Base subclass for a single replica
|
39
|
+
def connection_class(config_key)
|
40
|
+
# Config key is of schema 'knockoff_replica_n'
|
41
|
+
class_name = "KnockoffReplica#{config_key.split('_').last}"
|
42
|
+
|
43
|
+
# TODO: Hardcoding the uri string feels meh. Either set the database config
|
44
|
+
# or reference ENV instead
|
45
|
+
Knockoff.module_eval %Q{
|
46
|
+
class #{class_name} < ActiveRecord::Base
|
47
|
+
self.abstract_class = true
|
48
|
+
establish_connection :#{config_key}
|
49
|
+
def self.connection_config
|
50
|
+
configurations[#{config_key.to_s.inspect}]
|
51
|
+
end
|
52
|
+
end
|
53
|
+
}, __FILE__, __LINE__
|
54
|
+
|
55
|
+
Knockoff.const_get(class_name)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
metadata
ADDED
@@ -0,0 +1,148 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: thanx-knockoff
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.1.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Scott Ringwelski
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2019-09-18 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: activerecord
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 4.0.0
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: 4.0.0
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: activesupport
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 4.0.0
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 4.0.0
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: bundler
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rake
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '10.0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '10.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: rspec
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: sqlite3
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
description:
|
98
|
+
email:
|
99
|
+
- me@sgringwe.com
|
100
|
+
executables: []
|
101
|
+
extensions: []
|
102
|
+
extra_rdoc_files: []
|
103
|
+
files:
|
104
|
+
- ".gitignore"
|
105
|
+
- ".rspec"
|
106
|
+
- ".travis.yml"
|
107
|
+
- CHANGELOG.md
|
108
|
+
- Gemfile
|
109
|
+
- LICENSE.txt
|
110
|
+
- README.md
|
111
|
+
- Rakefile
|
112
|
+
- bin/console
|
113
|
+
- bin/setup
|
114
|
+
- ci/install_modern_sqlite.sh
|
115
|
+
- knockoff.gemspec
|
116
|
+
- lib/knockoff.rb
|
117
|
+
- lib/knockoff/active_record/base.rb
|
118
|
+
- lib/knockoff/active_record/relation.rb
|
119
|
+
- lib/knockoff/base.rb
|
120
|
+
- lib/knockoff/config.rb
|
121
|
+
- lib/knockoff/error.rb
|
122
|
+
- lib/knockoff/replica_connection_pool.rb
|
123
|
+
- lib/knockoff/version.rb
|
124
|
+
homepage: https://github.com/sgringwe/knockoff
|
125
|
+
licenses:
|
126
|
+
- MIT
|
127
|
+
metadata: {}
|
128
|
+
post_install_message:
|
129
|
+
rdoc_options: []
|
130
|
+
require_paths:
|
131
|
+
- lib
|
132
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
133
|
+
requirements:
|
134
|
+
- - ">="
|
135
|
+
- !ruby/object:Gem::Version
|
136
|
+
version: '0'
|
137
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
138
|
+
requirements:
|
139
|
+
- - ">="
|
140
|
+
- !ruby/object:Gem::Version
|
141
|
+
version: '0'
|
142
|
+
requirements: []
|
143
|
+
rubyforge_project:
|
144
|
+
rubygems_version: 2.7.8
|
145
|
+
signing_key:
|
146
|
+
specification_version: 4
|
147
|
+
summary: A gem for easily using read replicas
|
148
|
+
test_files: []
|