knockoff 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e0025a70e9bc0d93706b972c83a1bf27ed62a836
-  data.tar.gz: 7eada472ea6751980a6109d047f7600cce5590a3
+  metadata.gz: f6249cf98485220a497bcfe3ae9fc7056606451f
+  data.tar.gz: ee8347e81037c6a8641849473fb08ddc304eff2c
 SHA512:
-  metadata.gz: 7743e8ea20c39af558a3425d4b2e565dad7b9f38967833eeaece006874aba0662d364b8383b87c699448750feca896eced1e4bd54de7b9831c55f82cc2fcbcad
-  data.tar.gz: 6d1a25ea1b2578529e59b22ac11668a1d13355b2857000a83a169f91eb28cc2e3bcdff94356e0c5544df9de81503fb100acffc27503bae3168b38d41a9f04e22
+  metadata.gz: 1b2ca3c16d58470005cea91b6f356f540335de2b461f34367a43b8a545dae477cdee607fab03a318915fc638d93001e3774f35616e5211053379dd057d8a201d
+  data.tar.gz: 92e4702b63d1378d8cacf0f8a02f423880b665c6d6bf19930c30ff0d0d174fdb2e94be4c5b39b3d717ccdedaf5265cd18a54cae05c444b810c0cd00023ff2a70

data/README.md

@@ -1,6 +1,7 @@
 # Knockoff (WIP)
 
 [![Build Status](https://travis-ci.org/sgringwe/knockoff.svg?branch=master)](https://travis-ci.org/sgringwe/knockoff)
+[![Gem Version](https://badge.fury.io/rb/knockoff.svg)](https://badge.fury.io/rb/knockoff)
 
 A gem for easily using read replicas. Heavily based off of https://github.com/kenn/slavery and https://github.com/kickstarter/replica_pools gem.
 
@@ -30,9 +31,145 @@ Or install it yourself as:
 
 ## Usage
 
-TODO
+### Initializer
 
-### Usage Notes
+Add an initializer at config/knockoff.rb with the below contents
+
+```
+Knockoff.enabled = true # NOTE: Consider adding ENV based disabling
+```
+
+### Configuration
+
+Configuration is done using ENV properties. This makes it easy to add and remove replicas at runtime (or to fully disable if needed). First, set up ENV variables pointing to your replica databases. Consider using the [dotenv](https://github.com/bkeepers/dotenv) gem for manging ENV variables.
+
+```
+# .env
+
+REPLICA_1=postgres://username:password@localhost:5432/database_name
+```
+
+The second ENV variable to set is `KNOCKOFF_REPLICA_ENVS` which is a comma-separated list of ENVS holding database URLs to use as replicas. In this case, the ENV would be set as follows.
+
+```
+# .env
+
+KNOCKOFF_REPLICA_ENVS=REPLICA_1
+```
+
+Note that it can be multiple replicas, and `knockoff` will use both evenly:
+
+```
+KNOCKOFF_REPLICA_ENVS=REPLICA_1,REPLICA_2
+```
+
+Lastly, knockoff will read the `'knockoff_replicas'` database.yml config for specifying additional params:
+
+```
+# database.yml
+
+knockoff_replicas:
+  <<: *common
+  prepared_statements: false
+```
+
+### Basics
+
+To use one of the replica databases, use
+
+```
+Knockoff.on_replica { User.count }
+```
+
+To force primary, use
+
+```
+Knockoff.on_primary { User.create(name: 'Bob') }
+```
+
+### Using in Controllers
+
+A common use case is to use replicas for GET requests and otherwise use primary. A simplified use case might look something like this:
+
+```
+# application_controller.rb
+
+around_action :choose_database
+
+def choose_database(&block)
+  if should_use_primary_database?
+    Knockoff.on_primary(&block)
+  else
+    Knockoff.on_replica(&block)
+  end
+end
+
+def should_use_primary_database?
+  request.method_symbol != :get
+end
+
+```
+
+#### Replication Lag
+
+Replicas will often be slightly behind the primary database. To compensate, consider "sticking" a user who has recently made changes to the primary for a small duration of time to the primary database. This will avoid cases where a user creates a record on primary, is redirected to view that record, and receives a 404 error since the record is not yet in the replica. A simple implementation for this could look like:
+
+```
+# application_record.rb
+
+after_commit :track_commit_occurred_in_request
+
+# If any commit happens in a request, we record that and have the logged_in_user
+# read from primary for a short period of time.
+def track_commit_occurred_in_request
+  RequestLocals.store['commit_occurred_in_current_request'] = true
+end
+
+# application_controller.rb
+
+after_action :force_leader_if_commit
+
+def force_leader_if_commit
+  if RequestLocals.store['commit_occurred_in_current_request'].to_b
+    session[:use_leader_until] = Time.current + FORCE_PRIMARY_DURATION
+  end
+end
+
+```
+
+Then, in your `should_use_primary_database?` method, consult `RequestLocals['commit_occurred_in_current_request']` for the decision.
+
+### Run-time Configuration
+
+Knockoff can be configured during runtime. This is done through the `establish_new_connections!` method which takes in a hash of new configurations to apply to each replica before re-connecting.
+
+```
+Knockoff.establish_new_connections!({ 'pool' => db_pool })
+```
+
+For example, to specify a puma connection pool at bootup your code might look something like
+
+```
+# puma.rb
+
+db_pool = Integer(ENV['PUMA_WORKER_DB_POOL'] || threads_count)
+# Configure the database connection to have the new pool size and re-establish connection
+database_config = ActiveRecord::Base.configurations[Rails.env] || Rails.application.config.database_configuration[Rails.env]
+database_config['pool'] = db_pool
+ActiveRecord::Base.establish_connection(database_config)
+Knockoff.establish_new_connections!({ 'pool' => db_pool })
+
+```
+
+#### Forking
+
+For forking servers, you may disconnect all replicas before forking with `Knockoff.disconnect_all!`.
+
+### Other Cases
+
+There are likely other cases specific to each application where it makes sense to force primary database and avoid replication lag. Good candidates are time-based pages (a live calendar, for example), forms, and payments.
+
+## Usage Notes
 
 * Do not use prepared statements with this gem
 
@@ -44,7 +181,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/knockoff.
+Bug reports and pull requests are welcome on GitHub at https://github.com/sgringwe/knockoff.
 
 
 ## License

data/knockoff.gemspec

@@ -19,6 +19,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_runtime_dependency 'activerecord', '>= 4.0.0'
+  spec.add_runtime_dependency 'activesupport', '>= 4.0.0'
   spec.add_runtime_dependency 'request_store_rails', '>= 1.0.0'
 
   spec.add_development_dependency "bundler", "~> 1.10"

data/lib/knockoff.rb

@@ -11,14 +11,6 @@ require 'knockoff/active_record/relation'
 module Knockoff
   class << self
     attr_accessor :enabled
-    attr_writer :spec_key
-
-    def spec_key
-      case @spec_key
-      when String   then @spec_key
-      when NilClass then @spec_key = "#{ActiveRecord::ConnectionHandling::RAILS_ENV.call}_replica"
-      end
-    end
 
     def on_replica(&block)
       Base.new(:replica).run(&block)
@@ -29,7 +21,20 @@ module Knockoff
     end
 
     def replica_pool
-      @replica_pool ||= ReplicaConnectionPool.new(config.replica_uris)
+      @replica_pool ||= ReplicaConnectionPool.new(config.replica_database_keys)
+    end
+
+    # Iterates through the replica pool and calls disconnect on each one's connection.
+    def disconnect_all!
+      replica_pool.disconnect_all_replicas!
+    end
+
+    # Updates the config (both internal representation and the ActiveRecord::Base.configuration)
+    # with the new config, and then reconnects each replica connection in the replica
+    # pool.
+    def establish_new_connections!(new_config)
+      config.update_replica_configs(new_config)
+      replica_pool.reconnect_all_replicas!
     end
 
     def config

data/lib/knockoff/config.rb

@@ -2,15 +2,22 @@ module Knockoff
   class Config
     # The current environment. Normally set to Rails.env, but
     # will default to 'development' outside of Rails apps.
-    attr_accessor :environment
+    attr_reader :environment
 
-    # An array of URIs to use for the replica pool.
-    # TODO: Add support for inheriting from database.yml
-    attr_accessor :replica_uris
+    # An array of configs to use for the replica pool.
+    attr_reader :replica_configs
+
+    # A hash of replica configs to their config hash.
+    attr_reader :replicas_configurations
 
     def initialize
       @environment = 'development'
-      set_replica_uris
+      @replicas_configurations = {}
+      set_replica_configs
+    end
+
+    def replica_database_keys
+      @replicas_configurations.keys
     end
 
     def replica_env_keys
@@ -21,18 +28,66 @@ module Knockoff
       end
     end
 
+    def update_replica_configs(new_configs)
+      ActiveRecord::Base.configurations['knockoff_replicas'].merge(new_configs) if ActiveRecord::Base.configurations['knockoff_replicas'].present?
+      @replicas_configurations.each do |key, _config|
+        update_replica_config(key, new_configs)
+      end
+    end
+
     private
 
-    def set_replica_uris
-      @replica_uris ||= parse_knockoff_replica_envs_to_uris
+    def update_replica_config(key, new_configs)
+      @replicas_configurations[key].merge!(new_configs)
+      ActiveRecord::Base.configurations[key].merge!(new_configs)
     end
 
-    def parse_knockoff_replica_envs_to_uris
+    def set_replica_configs
+      @replica_configs ||= parse_knockoff_replica_envs_to_configs
+    end
+
+    def parse_knockoff_replica_envs_to_configs
       # As a basic prevention of crashes, attempt to parse each DB uri
       # and don't add the uri to the final list if it can't be parsed
-      replica_env_keys.map do |env_key|
+      replica_env_keys.map.with_index(0) do |env_key, index|
         begin
-          URI.parse(ENV[env_key])
+          uri = URI.parse(ENV[env_key])
+
+          # Configure parameters such as prepared_statements, pool, reaping_frequency for all replicas.
+          replica_config = ActiveRecord::Base.configurations['knockoff_replicas'] || {}
+
+          adapter =
+            if uri.scheme == "postgres"
+              'postgresql'
+            else
+              uri.scheme
+            end
+
+          # Base config from the ENV uri. Sqlite is a special case
+          # and all others follow 'normal' config
+          uri_config =
+            if uri.scheme == 'sqlite3'
+              {
+                'adapter' => adapter,
+                'database' => uri.to_s.split(':')[1]
+              }
+            else
+              {
+                'adapter' => adapter,
+                'database' => (uri.path || "").split("/")[1],
+                'username' => uri.user,
+                'password' => uri.password,
+                'host' => uri.host,
+                'port' => uri.port
+              }
+            end
+
+          # Store the hash in configuration and use it when we establish the connection later.
+          key = "knockoff_replica_#{index}"
+          full_config = replica_config.merge(uri_config)
+
+          ActiveRecord::Base.configurations[key] = full_config
+          @replicas_configurations[key] = full_config
         rescue URI::InvalidURIError
           Rails.logger.info "LOG NOTIFIER: Invalid URL specified in follower_env_keys. Not including URI, which may result in no followers used." # URI is purposely not printed to logs
           # Return a 'nil' which will be removed from

data/lib/knockoff/replica_connection_pool.rb

@@ -2,11 +2,25 @@ module Knockoff
   class ReplicaConnectionPool
     attr_reader :pool
 
-    def initialize(uris)
+    def initialize(config_keys)
       @pool = Concurrent::Hash.new
 
-      uris.each_with_index do |uri, index|
-        @pool["replica_#{index}"] = connection_class(index, uri)
+      config_keys.each do |config_key|
+        @pool[config_key] = connection_class(config_key)
+      end
+    end
+
+    def disconnect_all_replicas!
+      @pool.each do |_name, klass|
+        klass.connection.disconnect!
+      end
+    end
+
+    # Assumes that the config has been updated to something new, and
+    # simply reconnects with the config.
+    def reconnect_all_replicas!
+      @pool.each do |database_key, klass|
+        klass.establish_connection database_key.to_sym
       end
     end
 
@@ -16,15 +30,16 @@ module Knockoff
 
     # Based off of code from replica_pools gem
     # generates a unique ActiveRecord::Base subclass for a single replica
-    def connection_class(replica_index, uri)
-      class_name = "Replica#{replica_index}"
+    def connection_class(config_key)
+      # Config key is of schema 'knockoff_replica_n'
+      class_name = "KnockoffReplica#{config_key.split('_').last}"
 
       # TODO: Hardcoding the uri string feels meh. Either set the database config
       # or reference ENV instead
       Knockoff.module_eval %Q{
         class #{class_name} < ActiveRecord::Base
           self.abstract_class = true
-          establish_connection '#{uri}'
+          establish_connection :#{config_key}
         end
       }, __FILE__, __LINE__
 

data/lib/knockoff/version.rb

@@ -1,3 +1,3 @@
 module Knockoff
-  VERSION = '0.1.0'.freeze
+  VERSION = '0.2.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: knockoff
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Scott Ringwelski
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-11-27 00:00:00.000000000 Z
+date: 2016-12-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -24,6 +24,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 4.0.0
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.0.0
 - !ruby/object:Gem::Dependency
   name: request_store_rails
   requirement: !ruby/object:Gem::Requirement
@@ -145,3 +159,4 @@ signing_key:
 specification_version: 4
 summary: A gem for easily using read replicas
 test_files: []
+has_rdoc: