slave_pools 0.1.0

data/LICENSE

@@ -0,0 +1,20 @@
+The MIT license:
+Copyright (c) 2012 Kickstarter
+ 
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+ 
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,259 @@
+= SlavePools
+
+== Easy Single Master/ Multiple Slave Setup for use in Ruby/Rails projects
+
+SlavePools builds a base layer of master/slave query splitting, by overwriting ActiveRecord's connection (with connection_proxy). With this in place, you can easily add a second layer of traffic splitting, by wrapping requests in the provided helper methods (examples below), and have a manageable master/slave solution for a standard rails application
+
+Overview
+* Sends only whitelisted SELECT-type queries to the Slaves
+* Sends all other queries to the Master
+* Works with query caching and transactions
+* Easy to separate types of read traffic into different collections of slaves (e.g. separating admin and user traffic)
+* Minimalist approach
+  * doesn't include sharding
+  * doesn't create a new ActiveRecord adapter
+  * doesn't weight slave db's
+  * Builds onto a standard database.yml file (gem doesn't initialize if no slaves are specified)
+  * doesn't switch slaves on its own (the user specifies when to switch in their code)
+
+The SlavePools GEM started as a fork of Maximilian Sch\303\266fmann's https://github.com/schoefmax/multi_db
+The MultiDB gem was inspired by Rick Olson's "masochism"-Plugin
+
+== Usage
+
+Toggle to next slave:
+  SlavePools.next_slave!
+
+Specify a different slave pool than the default:
+  SlavePools.with_pool('other_pool') { #do stuff }
+
+Specifically use the master for a call:
+  SlavePools.with_master { #do stuff }
+
+Determine if there are slaves:
+  SlavePools.active?
+
+The gem, by default, sends writes and reads to the master and slave databases, respectfully. But in your app, if you write to the master during a request, you will probably want to read from the master in that request as well, in case there is replication. You will also probably want to read from the master on the next request (after a write to the master) to cover redirects.
+
+Using a standard rails application setup, you can achieve this by adding these example methods to your application controller (some of these may be folded in the gem, but leaving out for now):
+
+  class ApplicationController < ActionController::Base
+
+    around_filter   :stick_to_master_for_updates
+    around_filter   :use_master_for_redirect #goes with above
+    after_filter    :switch_to_next_slave
+
+    def switch_to_next_slave
+      SlavePools.next_slave! if slaves?
+    end
+
+    def use_admin_slave_pool
+      SlavePools.with_pool('admin') { yield } if slaves?
+    end
+
+    def stick_to_master_for_updates
+      if slaves? && (request.post? || request.put? || request.delete?)
+        SlavePools.with_master { yield }
+        session[:stick_to_master] = 1
+      else
+        yield
+      end
+    end
+
+    def use_master_for_redirect
+      if slaves? && session[:stick_to_master]
+        session[:stick_to_master] = nil
+        SlavePools.with_master { yield }
+      else
+        yield
+      end
+    end
+
+    def use_master
+      if slaves?
+        SlavePools.with_master { yield }
+        session[:stick_to_master] = 1
+      else
+        yield
+      end
+    end
+
+    def slaves?
+      SlavePools.active?
+    end
+  end
+
+For other cases where you use the master for writes, you should wrap the request in a 'use_master' block
+
+  class PostsController < ApplicationController
+    around_filter :use_master, :only=>:index
+
+    def index
+      Activity.create()
+      # index is a GET call, but we've decided to record something, so we want to wrap it in a use_master block
+    end
+  end
+
+* works with activerecord 3.2.12 (not tested with Rails 2)
+
+=== Install
+
+Add to your Gemfile
+
+  gem 'slave_pools'
+
+=== Setup
+
+slave_pools identifies slave databases by looking for entries of the form
+"<tt><environment>_pool_<pool_name>_name_<db_name></tt>".
+
+In your database.yml, add sections for the slaves, e.g.:
+
+  development: # that would be the master
+    adapter: mysql
+    database: myapp_production
+    username: root
+    password:
+    host: localhost
+
+  development_pool_default_name_slave1: # that would be a slave named 'slave1' in the 'default' pool
+    adapter: mysql
+    database: slave_db1
+    username: root
+    password:
+    host: 10.0.0.2
+
+  development_pool_default_name_slave2: # that would be a slave named 'slave2' in the 'default' pool
+    ...
+  development_pool_admin_name_slave1: # that would be a slave named 'slave1' in the 'admin' pool (db names can be reused across pools)
+    ...
+  development_pool_admin_name_another_slave: # that would be a slave named 'another_slave' in the 'admin' pool
+
+This also creates an abstract classes named <tt>SlavePools::DefaultDb1</tt> for each db of the form <tt>SlavePools::<PoolName><DbName></tt>etc. If no slaves are specified, the SlavePools setup does not run, and the development DB would be used as normal.
+
+For development testing, I recommend creating a read-only mysql user and just point all of your slave DB's to the your development DB using the read-only user.
+
+The Default SlavePool will be used for all requests, so you should name on of the pools 'default' (if there isn't a 'default' slave_pool, the first slave_pool specified becomes the default)
+
+To enable the proxy globally, add this to a config/initializers:
+
+  SlavePools.setup!
+
+If you only want to enable it for specific environments, add this to
+the corresponding file in config/environments:
+
+  config.after_initialize do
+    SlavePools.setup!
+  end
+
+
+=== Using with Phusion Passenger
+
+(this is a note from MultiDB gem and has not been verified)
+
+With Passengers smart spawning method, child processes forked by the ApplicationSpawner
+won't have the connection proxy set up properly (this is a note from ).
+
+To make it work, add this to your <tt>environment.rb</tt> or an initializer script
+(e.g. <tt>config/initializers/connection_proxy.rb</tt>):
+
+  if defined?(PhusionPassenger)
+    PhusionPassenger.on_event(:starting_worker_process) do |forked|
+      if forked
+        # ... set configuration options, if any ...
+        SlavePools::ConnectionProxy.setup!
+      end
+    end
+  else # not using passenger (e.g. development/testing)
+    # ... set configuration options, if any ...
+    SlavePools::ConnectionProxy.setup!
+  end
+
+=== Using with ThinkingSphinx
+
+ThinkingSphinx looks for an adapter type and
+  SlavePools::ConnectionProxy.setup!
+
+  if ActiveRecord::Base.respond_to?('connection_proxy')
+    ThinkingSphinx::AbstractAdapter.class_eval do
+      def self.standard_adapter_for_model(model)
+        :mysql
+      end
+    end
+  end
+
+=== Forcing the master for certain actions
+
+Just add this to your controller:
+
+  around_filter(:only => :foo_action) { |c,a| ActiveRecord::Base.connection_proxy.with_master { a.call } }
+
+=== Forcing the master for certain models
+
+In your environment.rb or an initializer, add this *before* the call to <tt>setup!</tt>:
+
+  SlavePoolsModule::ConnectionProxy.master_models = ['CGI::Session::ActiveRecordStore::Session', 'PaymentTransaction', ...]
+  SlavePoolsModule::ConnectionProxy.setup!
+
+*NOTE*: You cannot safely add more master_models after calling <tt>setup!</tt>.
+=== Features
+* Minimalist implementation - does include sharding, doesn't creation a new adapter (so if you don't specify slaves for
+  an environment, the connection is not overwritten, and the DB works as normal), doesn't blacklist/remove slaves,
+* It sends everything except "select ..." queries to the master, instead of
+  sending only specific things to the master and anything "else" to the slave.
+  This avoids accidental writes to the master when there are API changes in
+  ActiveRecord which haven't been picked up by multi_db yet.
+  Note that this behavior will also always send helper methods like "+quote+" or
+  "<tt>add_limit!</tt>" to the master connection object, which doesn't add any
+  more load on the master, as these methods don't communicate with the db server
+  itself.
+
+
+=== Differences to "multi_db":
+
+* Supports multiple separate pools of slave databases
+* query caching is fixed
+* tries a slave once and immediately reverts to the master afterwards (does not cycle through slaves)
+* stays with the same slave DB until explicitly told to change. In practical usage, it didn't make sense to us
+  to have it cycle through slaves in the same web request, so I made the 'sticky slave' feature permanent
+* removed weighted slave rotation for now (didn't need it)
+* Currently not using Threaded variables (left this commented out in the code for now, may revisit)
+* Added with_pool method
+* does not blacklist slaves for timing out (we want other more robust monitoring software to take care of this)
+* better default case handling - if no slave DB's are specified, the regular Environment database is used, and the gem is
+  not initialized
+* added a wrapper class for shorter calls
+
+=== See also
+
+===== Masochism
+
+The original master/slave plugin:
+
+* http://github.com/technoweenie/masochism
+
+===== MultiDb
+
+The project is based on:
+
+* https://github.com/schoefmax/multi_db
+
+=== Running specs
+
+If you haven't already, install the rspec gem, then set up your database
+with a test database and a read_only user.
+
+To match spec/config/database.yml, you can:
+
+  mysql>
+    create database test_db;
+    create user 'read_only'@'localhost' identified by 'readme';
+    grant select on db_test.* to 'read_only'@'localhost';
+
+From the plugin directory, run:
+
+  rspec spec
+
+Author: Dan Drabik
+Copyright (c) 2012, Kickstarter
+Released under the MIT license

data/lib/slave_pools.rb

@@ -0,0 +1,43 @@
+require 'active_record'
+require 'slave_pools/slave_pool'
+require 'slave_pools/active_record_extensions'
+require 'slave_pools/observer_extensions'
+require 'slave_pools/query_cache_compat'
+require 'slave_pools/connection_proxy'
+
+#wrapper class to make the calls more succinct
+
+class SlavePools
+
+  def self.setup!
+    SlavePoolsModule::ConnectionProxy.setup!
+  end
+
+  def self.active?
+    ActiveRecord::Base.respond_to?('connection_proxy')
+  end
+
+  def self.next_slave!
+    ActiveRecord::Base.connection_proxy.next_slave! if active?
+  end
+
+  def self.with_pool(pool_name = 'default')
+    if active?
+      ActiveRecord::Base.connection_proxy.with_pool(pool_name) { yield }
+    else
+      yield
+    end
+  end
+
+  def self.with_master
+    if active?
+      ActiveRecord::Base.connection_proxy.with_master { yield }
+    else
+      yield
+    end
+  end
+
+  def self.current
+    ActiveRecord::Base.connection_proxy.current if active?
+  end
+end

data/lib/slave_pools/active_record_extensions.rb

@@ -0,0 +1,54 @@
+module SlavePoolsModule
+  module ActiveRecordExtensions
+    def self.included(base)
+      base.send :include, InstanceMethods
+      base.send :extend, ClassMethods
+      base.cattr_accessor :connection_proxy
+      # handle subclasses which were defined by the framework or plugins
+      base.send(:descendants).each do |child|
+        child.hijack_connection
+      end
+    end
+
+    module InstanceMethods
+      def reload(options = nil)
+        self.connection_proxy.with_master { super }
+      end
+    end
+
+    module ClassMethods
+      # Make sure transactions always switch to the master
+      def transaction(options = {}, &block)
+        if self.connection.kind_of?(ConnectionProxy)
+          super
+        else
+          self.connection_proxy.with_master { super }
+        end
+      end
+
+      # Make sure caching always uses master connection
+      def cache(&block)
+        if ActiveRecord::Base.configurations.blank?
+          yield
+        else
+          ActiveRecord::Base.connection.cache(&block)
+        end
+      end
+
+      def inherited(child)
+        super
+        child.hijack_connection
+      end
+
+      def hijack_connection
+        return if ConnectionProxy.master_models.include?(self.to_s)
+        # logger.info "[SlavePools] hijacking connection for #{self.to_s}" # commenting out noisy logging
+        class << self
+          def connection
+            self.connection_proxy
+          end
+        end
+      end      
+    end
+  end
+end

data/lib/slave_pools/connection_proxy.rb

@@ -0,0 +1,280 @@
+require 'active_record/connection_adapters/abstract/query_cache'
+require 'set'
+
+module SlavePoolsModule
+  class ConnectionProxy
+    include ActiveRecord::ConnectionAdapters::QueryCache
+    include QueryCacheCompat
+
+    # Safe methods are those that should either go to the slave ONLY or go
+    # to the current active connection.
+    SAFE_METHODS = Set.new([ :select_all, :select_one, :select_value, :select_values,
+      :select_rows, :select, :verify!, :raw_connection, :active?, :reconnect!,
+      :disconnect!, :reset_runtime, :log, :log_info ])
+
+    if ActiveRecord.const_defined?(:SessionStore) # >= Rails 2.3
+      DEFAULT_MASTER_MODELS = ['ActiveRecord::SessionStore::Session']
+    else # =< Rails 2.3
+      DEFAULT_MASTER_MODELS = ['CGI::Session::ActiveRecordStore::Session']
+    end
+
+    attr_accessor :master
+    attr_accessor :master_depth, :current, :current_pool
+
+    class << self
+
+      # defaults to Rails.env if multi_db is used with Rails
+      # defaults to 'development' when used outside Rails
+      attr_accessor :environment
+
+      # a list of models that should always go directly to the master
+      #
+      # Example:
+      #
+      #  SlavePool::ConnectionProxy.master_models = ['MySessionStore', 'PaymentTransaction']
+      attr_accessor :master_models
+
+      #true or false - whether you want to include the ActionController helpers or not
+      #this allow
+
+      # if master should be the default db
+      attr_accessor :defaults_to_master
+
+      # #setting a config instance variable so that thinking sphinx,and other gems that use the connection.instance_variable_get(:@config), work correctly
+      attr_accessor :config
+
+      # Replaces the connection of ActiveRecord::Base with a proxy and
+      # establishes the connections to the slaves.
+      def setup!
+        self.master_models ||= DEFAULT_MASTER_MODELS
+        self.environment   ||= (defined?(Rails.env) ? Rails.env : 'development')
+
+        slave_pools = init_slave_pools
+        # if there are no slave pools, we just want to silently exit and not edit the ActiveRecord::Base.connection
+        if !slave_pools.empty?
+          master = ActiveRecord::Base
+          master.send :include, SlavePoolsModule::ActiveRecordExtensions
+          ActiveRecord::Observer.send :include, SlavePoolsModule::ObserverExtensions
+
+          master.connection_proxy = new(master, slave_pools)
+          master.logger.info("** slave_pools with master and #{slave_pools.length} slave_pool#{"s" if slave_pools.length > 1} (#{slave_pools.keys}) loaded.")
+        else
+          ActiveRecord::Base.logger.info(" No Slave Pools specified for this environment") #this is currently not logging
+        end
+      end
+
+      protected
+
+      def init_slave_pools
+        slave_pools = {}
+        ActiveRecord::Base.configurations.each do |name, db_config|
+          # look for dbs matching the slave_pool format and verify a test connection before adding it to the pools
+          if name.to_s =~ /#{self.environment}_pool_(.*)_name_(.*)/ && connection_valid?(db_config)
+            slave_pools = add_to_pool(slave_pools, $1, $2, name, db_config)
+          end
+        end
+        return slave_pools
+      end
+
+      private :new
+
+    end # end class << self
+
+    def initialize(master, slave_pools)
+      @slave_pools = {}
+      slave_pools.each do |pool_name, slaves|
+        @slave_pools[pool_name.to_sym] = SlavePool.new(pool_name, slaves)
+      end
+      @master    = master
+      @reconnect = false
+      @current_pool = default_pool
+      if self.class.defaults_to_master
+        @current = @master
+        @master_depth = 1
+        @config = master.connection.instance_variable_get(:@config)
+      else
+        @current = slave
+        @master_depth = 0
+        @config = @current.config_hash #setting this
+      end
+
+    end
+
+    def default_pool
+      @slave_pools[:default] || @slave_pools.values.first #if there is no default specified, use the first pool found
+    end
+
+    def slave_pools
+      @slave_pools
+    end
+
+    def slave
+      @current_pool.current
+    end
+
+    def with_pool(pool_name = 'default')
+      @current_pool = @slave_pools[pool_name.to_sym] || default_pool
+      @current = slave unless within_master_block?
+      yield
+    ensure
+      @current_pool = default_pool
+      @current = slave unless within_master_block?
+    end
+
+    def with_master
+      @current = @master
+      @master_depth += 1
+      yield
+    ensure
+      @master_depth -= 1
+      @master_depth = 0 if @master_depth < 0 # ensure that master depth never gets below 0
+      @current = slave if !within_master_block?
+    end
+
+    def within_master_block?
+      @master_depth > 0
+    end
+
+    def transaction(start_db_transaction = true, &block)
+      with_master { @master.retrieve_connection.transaction(start_db_transaction, &block) }
+    end
+
+    # Calls the method on master/slave and dynamically creates a new
+    # method on success to speed up subsequent calls
+    def method_missing(method, *args, &block)
+      send(target_method(method), method, *args, &block).tap do
+        create_delegation_method!(method)
+      end
+    end
+
+    # Switches to the next slave database for read operations.
+    # Fails over to the master database if all slaves are unavailable.
+    def next_slave!
+      return if within_master_block? # don't if in with_master block
+      @current = @current_pool.next
+    rescue
+      @current = @master
+    end
+
+   protected
+
+    def create_delegation_method!(method)
+      self.instance_eval %Q{
+        def #{method}(*args, &block)
+          #{target_method(method)}(:#{method}, *args, &block)
+        end
+      }, __FILE__, __LINE__
+    end
+
+    def target_method(method)
+      unsafe?(method) ? :send_to_master : :send_to_current
+    end
+
+    def send_to_master(method, *args, &block)
+      reconnect_master! if @reconnect
+      @master.retrieve_connection.send(method, *args, &block)
+    rescue => e
+      log_errors(e, 'send_to_master', method)
+      raise_master_error(e)
+    end
+
+    def send_to_current(method, *args, &block)
+      reconnect_master! if @reconnect && master?
+      # logger.debug "[SlavePools] Using #{@current.name}"
+      @current = @master if unsafe?(method) #failsafe to avoid sending dangerous method to master
+      @current.retrieve_connection.send(method, *args, &block)
+    rescue Mysql2::Error, ActiveRecord::StatementInvalid => e
+      log_errors(e, 'send_to_current', method)
+      raise_master_error(e) if master?
+      logger.warn "[SlavePools] Error reading from slave database"
+      logger.error %(#{e.message}\n#{e.backtrace.join("\n")})
+      if e.message.match(/Timeout waiting for a response from the last query/)
+        # Verify that the connection is active & re-raise
+        @current.retrieve_connection.verify!
+        raise e
+      else
+        send_to_master(method, *args, &block) # if cant connect, send the query to master
+      end
+    end
+
+    def reconnect_master!
+      @master.retrieve_connection.reconnect!
+      @reconnect = false
+    end
+
+    def raise_master_error(error)
+      logger.fatal "[SlavePools] Error accessing master database. Scheduling reconnect"
+      @reconnect = true
+      raise error
+    end
+
+    def unsafe?(method)
+      !SAFE_METHODS.include?(method)
+    end
+
+    def master?
+      @current == @master
+    end
+
+    def logger
+      ActiveRecord::Base.logger
+    end
+
+    private
+
+    def self.add_to_pool(slave_pools, pool_name, slave_name, full_db_name, db_config)
+      slave_pools[pool_name] ||= []
+      db_config_with_symbols = db_config.inject({}){|memo,(k,v)| memo[k.to_sym] = v; memo}
+      SlavePoolsModule.module_eval %Q{
+        class #{pool_name.camelize}#{slave_name.camelize} < ActiveRecord::Base
+          self.abstract_class = true
+          establish_connection :#{full_db_name}
+          def self.config_hash
+            #{db_config_with_symbols.inspect}
+          end
+        end
+      }, __FILE__, __LINE__
+      slave_pools[pool_name] << "SlavePoolsModule::#{pool_name.camelize}#{slave_name.camelize}".constantize
+      return slave_pools
+    end
+
+    # method to verify whether DB connection is active?
+    def self.connection_valid?(db_config = nil)
+      is_connected = false
+      if db_config
+        begin
+          ActiveRecord::Base.establish_connection(db_config)
+          ActiveRecord::Base.connection
+          is_connected = ActiveRecord::Base.connected?
+        rescue => e
+          log_errors(e, 'self.connection_valid?')
+        ensure
+          ActiveRecord::Base.establish_connection(environment) #rollback to the current environment to avoid issues
+        end
+      end
+      return is_connected
+    end
+
+    # logging instance errors
+    def log_errors(error, sp_method, db_method)
+      logger.error "[SlavePools] - Error: #{error}"
+      logger.error "[SlavePools] - SlavePool Method: #{sp_method}"
+      logger.error "[SlavePools] - Master Value: #{@master}"
+      logger.error "[SlavePools] - Master Depth: #{@master_depth}"
+      logger.error "[SlavePools] - Current Value: #{@current}"
+      logger.error "[SlavePools] - Current Pool: #{@current_pool}"
+      logger.error "[SlavePools] - Current Pool Slaves: #{@current_pool.slaves}" if @current_pool
+      logger.error "[SlavePools] - Current Pool Name: #{@current_pool.name}" if @current_pool
+      logger.error "[SlavePools] - Reconnect Value: #{@reconnect}"
+      logger.error "[SlavePools] - Default Pool: #{default_pool}"
+      logger.error "[SlavePools] - DB Method: #{db_method}"
+    end
+
+    # logging class errors
+    def self.log_errors(error, sp_method)
+      logger = ActiveRecord::Base.logger
+      logger.error "[SlavePools] - Error: #{error}"
+      logger.error "[SlavePools] - SlavePool Method: #{sp_method}"
+    end
+  end
+end