methodmissing-mysqlplus_adapter 1.0.1

data/README.textile

@@ -0,0 +1,97 @@
+h1. Mysqlplus ActiveRecord Adapter
+
+h2. Installation
+
+Grab mysqlplus  ....
+
+  git clone git://github.com/oldmoe/mysqlplus.git
+  cd mysqlplus
+  rake  
+
+... and mysqlplus_adapter ...
+
+  sudo gem install methodmissing-mysqlplus_adapter -s http://gems.github.com
+
+... then update config/database.yml
+
+  production:
+    adapter:  mysqlplus
+    database: myapp_production
+    username: root
+    password:  
+    host: localhost
+    pool: 10
+
+h2. Why Bother ?
+
+The mysqlplus gem ( a fork of mysql-2.8.4pre ) exposes an asynchronous API that lends itself very well to decoupling the stock Mysql protocol into full duplex in a multi-threaded environment.
+
+The ever popular "mysql ruby":http://www.tmtm.org/en/mysql/ruby/ do *not* schedule Threads and thus lock the whole MRI interpreter for any database I/O from within a given process.
+
+Rails since release 2.2 support connection pooling and Thread safety ( at the framework level mind you, plugins, gems and user code aside ) through :
+
+  config.threadsafe!
+
+This configuration hook removes the global Dispatcher lock and yields one connection from the pool per request / response cycle. 
+
+You'll need mysqplus and this adapter to get around the MRI lock with mysql ruby.
+
+h2. Configuration Options
+
+An additional connection specification element, *warmup* is available that attempts to establish the pooled connections in advance.This may be useful for high traffic environments where it makes sense to setup connections when bouncing the App and not let initial incoming requests be hogged with Mysql connection overheads.
+
+  production:
+    adapter:  mysqlplus
+    database: myapp_production
+    username: root
+    password:  
+    host: localhost
+    pool: 10  
+    warmup: true
+
+h3. Deferrable Results
+
+Deferred results simulate lazy loading in a background Thread, through another Mysql connection, other than the one the current Thread has acquired.This type of workflow assumes a decent Connection Pool size of 5 to 10 connections.
+
+  # Immediate yield control back to the current Thread as the query is pushed to the background.
+  # Yields an instance of ActiveRecord::Deferrable::Result
+  #
+  Post.find( :first, :defer => true )
+
+A deferred result blocks when any method's invoked on the result set right away.
+
+  Post.find( :first, :defer => true ).title
+
+This concept is quite useful in an MVC context, allowing the controller to fetch results, defer fetching them to the background and reference them in the view, allowing an undefined period / time slice during which rendering, template setup etc. may occur. 
+
+  class PostsController
+   
+    def index
+      # No blocking, executes in the background, yields a deferrable result.
+      #
+      @posts = Posts.published.find(:all, :defer => true )
+    end
+
+  end
+
+Since ActiveRecord 2.1 preloading favors multiple efficient queries to cumbersome and mostly slow JOINs.Those secondary queries can easily be pushed down different connections.
+
+  # Use 3 connections from the pool : 1 x Post, 1 x Comment and 1 x Vote 
+  #
+  Post.find(:all, :limit => 10, :include => [:comments, :votes], :defer => true )
+
+h2. Garbage Collection
+
+There's some experimental GC patches "available":http://github.com/oldmoe/mysqlplus/tree/with_async_validation - the mysql ruby gem forces GC every 20 queries, that's a guaranteed GC cycle every 5th request for a request with a 4 query overhead.
+
+h2. Stability
+
+In (pre)-production use at a handful of sites and the test suite is designed to run against the existing ActiveRecord suite.
+
+h2. TODO
+
+* Experiment with turning off query_with_result for certain queries.
+
+* Deferred inserts / updates - *dangerous* INSERT DELAYED for Innodb
+
+* 1.9 compatibility testing

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 1
+:major: 1
+:minor: 0

data/lib/active_record/connection_adapters/mysqlplus_adapter.rb

@@ -0,0 +1,112 @@
+begin
+  require_library_or_gem('mysqlplus')
+rescue LoadError
+  $stderr.puts <<-EOS
+        The mysqlplus gem is required!
+        'git clone git@github.com:oldmoe/mysqlplus.git && cd mysqlplus && rake'
+        There's some experimental GC patches available @ http://github.com/oldmoe/mysqlplus/tree/with_async_validation - the mysql gem forces GC every 20 queries, that's a guaranteed GC cycle every 5th request for a request with a 4 query overhead.
+      EOS
+  exit
+end
+
+begin 
+  require_library_or_gem('fastthread')
+rescue => LoadError
+  $stderr.puts "'gem install fastthread' for better performance"
+end  
+
+[ "mysqlplus_adapter/connection_pool",
+  "mysql_adapter",
+  "mysqlplus_adapter/deferrable/result",
+  "mysqlplus_adapter/deferrable/macro" ].each{|l| require "active_record/connection_adapters/#{l}" }
+
+module ActiveRecord
+  module ConnectionAdapters
+    class MysqlplusAdapter < ActiveRecord::ConnectionAdapters::MysqlAdapter
+      
+      DEFERRABLE_SQL = /^(INSERT|UPDATE|ALTER|DROP|SELECT|DELETE|RENAME|REPLACE|TRUNCATE)/i.freeze
+
+      # Accessor for the raw connection socket for integration with EventMachine etc.
+      # 
+      def socket
+        @connection.socket
+      end
+      
+      def execute(sql, name = nil, skip_logging = false) #:nodoc:
+        if skip_logging
+          @connection.c_async_query( sql )
+        else  
+          log("(Socket #{socket.to_s}) #{sql}",name) do 
+            @connection.c_async_query( sql )
+          end
+        end  
+      end
+      
+      # Determine if a given SQL snippet is deferrable to another Thread.
+      #
+      def deferrable?( sql )
+        !open_transactions? && 
+        initialized? && 
+        deferrable_sql?( sql )
+      end
+      
+      # Determine if the raw SQL can be deferred.This excludes changing users,
+      # retrieving column information, per connection configuration etc.
+      #
+      def deferrable_sql?( sql )
+        sql =~ DEFERRABLE_SQL
+      end
+      
+      # Only support deferring connections once the framework has been initialized.
+      #
+      def initialized?
+        Object.const_defined?( 'Rails' ) && ::Rails.initialized?
+      end
+      
+      # Are there any open transactions ? 
+      #
+      def open_transactions?
+        open_transactions != 0
+      end
+
+      private 
+
+        def configure_connection #:nodoc:
+          super 
+          @connection.disable_gc = true if disable_gc? 
+        end
+      
+        # See http://github.com/oldmoe/mysqlplus/tree/with_async_validation
+        def disable_gc? #:nodoc:
+          @connection.respond_to?( :disable_gc= )
+        end
+ 
+    end
+  end
+end
+
+module ActiveRecord
+  class << Base
+    
+    def mysqlplus_connection(config)
+      config = config.symbolize_keys
+      host     = config[:host]
+      port     = config[:port]
+      socket   = config[:socket]
+      username = config[:username] ? config[:username].to_s : 'root'
+      password = config[:password].to_s
+
+      if config.has_key?(:database)
+        database = config[:database]
+      else
+        raise ArgumentError, "No database specified. Missing argument: database."
+      end      
+          
+      mysql = Mysql.init
+      mysql.ssl_set(config[:sslkey], config[:sslcert], config[:sslca], config[:sslcapath], config[:sslcipher]) if config[:sslca] || config[:sslkey]
+
+      ConnectionAdapters::MysqlplusAdapter.new(mysql, logger, [host, username, password, database, port, socket], config)    
+    end
+    
+  end
+end    

data/lib/active_record/connection_adapters/mysqlplus_adapter/connection_pool.rb

@@ -0,0 +1,40 @@
+module ActiveRecord
+  module ConnectionAdapters
+    class ConnectionPool
+      
+      attr_reader :connections, 
+                  :checked_out,
+                  :reserved_connections
+
+      def initialize(spec)
+        @spec = spec
+        # The cache of reserved connections mapped to threads
+        @reserved_connections = {}
+        # The mutex used to synchronize pool access
+        @connection_mutex = Monitor.new
+        @queue = @connection_mutex.new_cond
+        # default 5 second timeout
+        @timeout = spec.config[:wait_timeout] || 5
+        # default max pool size to 5
+        @size = (spec.config[:pool] && spec.config[:pool].to_i) || 5
+        @connections = []
+        @checked_out = []
+        # warmup hook
+        warmup! if spec.config[:warmup]
+      end
+
+      private
+      
+        # Establish ( warmup ) all connections for this pool in advance.
+        #
+        def warmup!
+          @connection_mutex.synchronize do
+            1.upto(@size) do
+              @connections << new_connection
+            end
+          end  
+        end  
+
+    end  
+  end    
+end

data/lib/active_record/connection_adapters/mysqlplus_adapter/deferrable/macro.rb

@@ -0,0 +1,99 @@
+module ActiveRecord
+  module Deferrable
+    module Macro
+      
+      class << self
+
+        def install!
+          ActiveRecord::Base.send :extend, SingletonMethods      
+          ar_eigenclass::VALID_FIND_OPTIONS << :defer
+          alias_deferred :find, :find_by_sql, :preload_associations
+        end
+
+        private
+
+          def ar_eigenclass #:nodoc:
+            @@ar_eigneclass ||= (class << ActiveRecord::Base; self; end)
+          end
+
+          def alias_deferred( *method_signatures ) #:nodoc:
+            method_signatures.each do |method_signature| 
+              ar_eigenclass.alias_method_chain method_signature, :defer
+            end
+          end
+
+      end
+
+    end
+
+    module SingletonMethods
+
+      # !! EXPERIMENTAL !!
+      # Since ActiveRecord 2.1, multiple lightweight queries is preferred to expensive
+      # JOINS when eager loading related models.In some use cases it's more performant 
+      # to distribute those over multiple connections versus dispatching them on the same
+      # connection.
+      # 
+      # ....
+      # Record.find(:first, :include => [:other. :another], :defer => true)
+      # .... 
+      #
+      def preload_associations_with_defer(records, associations, preload_options={})
+        if preload_options.key?(:defer)
+          ActiveRecord::Deferrable::Result.new do
+            preload_associations_without_defer(records, associations, preload_options)
+          end
+        else
+          preload_associations_without_defer(records, associations, preload_options)
+        end    
+      end
+
+      # Execute raw SQL in another Thread.Blocks only when the result is immediately
+      # referenced.
+      # ...
+      # Record.find_by_sql( "SELECT SLEEP (1)", true ) 
+      # ...
+      #
+      def find_by_sql_with_defer( sql, defer = false )
+        if defer
+          ActiveRecord::Deferrable::Result.new do 
+            find_by_sql_without_defer( sql )
+          end
+        else
+          find_by_sql_without_defer( sql )
+        end    
+      end
+
+      # Executes a query in another background Thread.Blocks only when the result is
+      # immediately referenced.
+      # ...
+      # Record.find( :first, :conditions => ['records.some_id >= ?', 100], :defer => true ) 
+      # ...
+      #
+      def find_with_defer( *args )
+        options = args.dup.extract_options!
+        if options.key?(:defer)
+          with_deferred_scope do
+            ActiveRecord::Deferrable::Result.new do 
+              find_without_defer(*args)
+            end
+          end  
+        else
+          find_without_defer(*args)
+        end    
+      end  
+
+      private
+      
+        # Deferred finder scope
+        #
+        def with_deferred_scope( &block ) #:nodoc:
+          with_scope( { :find => { :defer => true } }, :merge, &block )
+        end
+
+    end
+
+  end
+end
+
+ActiveRecord::Deferrable::Macro.install!

data/lib/active_record/connection_adapters/mysqlplus_adapter/deferrable/result.rb

@@ -0,0 +1,46 @@
+module ActiveRecord
+  module Deferrable
+    class Result < ActiveSupport::BasicObject
+       
+      # Represents a Lazy Loaded resultset.
+      # Any method calls would block if the result hasn't yet been processed in a background
+      # Thread. 
+      #
+      def initialize( &deferrable )
+        defer!( deferrable )
+      end
+
+      # Calls a given procedure in a background Thread, on another
+      # connection from the pool.Guarantees that said connection is checked
+      # back in on completion.
+      #
+      def defer!( deferrable )
+        @result = Thread.new( deferrable ) do |deferrable|
+          begin
+            deferrable.call 
+          rescue => exception
+            exception  
+          ensure
+            ::ActiveRecord::Base.connection_pool.release_connection            
+          end
+        end
+        self
+      end
+      
+      # Delegates to the background Thread.
+      #
+      def method_missing(*args, &block)
+        @_result ||= @result.value
+        validate!
+        @_result.send(*args, &block)
+      end 
+      
+      # Re-raise any Exceptions from the background Thread.
+      #
+      def validate!
+        raise @_result if @_result.is_a?( Exception )
+      end
+
+    end    
+  end
+end  

data/test/connections/mysqlplus/connection.rb

@@ -0,0 +1,30 @@
+print "Using mysqplus\n"
+require "#{File.dirname(__FILE__)}/../../helper"
+require_dependency "#{AR_TEST_SUITE}/models/course"
+require 'logger'
+
+ActiveRecord::Base.logger = Logger.new(StringIO.new)
+
+# GRANT ALL PRIVILEGES ON activerecord_unittest.* to 'rails'@'localhost';
+# GRANT ALL PRIVILEGES ON activerecord_unittest2.* to 'rails'@'localhost';
+
+ActiveRecord::Base.configurations = {
+  'arunit' => {
+    :adapter  => 'mysqlplus',
+    :username => 'rails',
+    :encoding => 'utf8',
+    :database => 'activerecord_unittest',
+    :pool => 5,
+    :warmup => false
+  },
+  'arunit2' => {
+    :adapter  => 'mysqlplus',
+    :username => 'rails',
+    :database => 'activerecord_unittest2',
+    :pool => 5,
+    :warmup => false
+  }
+}
+
+ActiveRecord::Base.establish_connection 'arunit'
+Course.establish_connection 'arunit2'

data/test/deferrable_test.rb

@@ -0,0 +1,15 @@
+require "#{File.dirname(__FILE__)}/helper"
+Mysqlplus::Test.prepare!
+
+class DeferrableTest < Test::Unit::TestCase
+  
+  def test_should_be_able_to_find_records_in_a_background_thread
+    assert_equal MysqlUser.find(:first, :defer => true), MysqlUser.find(:first) 
+    assert_instance_of MysqlUser, MysqlUser.find(:first, :defer => true)
+  end
+
+  def test_should_be_able_to_find_records_by_sql_background_thread
+    assert_equal MysqlUser.find_by_sql("SELECT * FROM mysql.user WHERE User = 'root'", true), MysqlUser.find(:all, :conditions => ['user.User = ?', 'root']) 
+  end
+    
+end

data/test/helper.rb

@@ -0,0 +1,90 @@
+require 'rubygems'
+require 'active_support'
+require 'activerecord'
+ActiveRecord.load_all!
+
+module Mysqlplus
+  class Test
+    
+    MODELS_DIR = "#{File.dirname(__FILE__)}/models".freeze
+    
+    class << self
+      
+      def prepare!
+        require 'test/unit'
+        connect!
+        require_models()
+      end
+            
+      def setup!
+        setup_constants!
+        setup_config!
+      end
+        
+      def mysqlplus_connection
+        "#{File.dirname(__FILE__)}/connections/mysqlplus"
+      end
+
+      def active_record_test_files
+        returning([]) do |files|
+          files << glob( "#{AR_TEST_SUITE}/cases/**/*_test{,_mysqlplus}.rb" )
+        end.sort
+      end
+
+      def test_files
+        glob( "#{File.dirname(__FILE__)}/*_test.rb" )
+      end
+
+      private
+
+      def connect!
+        ::ActiveRecord::Base.establish_connection( :adapter  => 'mysqlplus',
+                                                   :username => 'root',
+                                                   :database => 'mysql',
+                                                   :pool => 5,
+                                                   :warmup => true )
+      end
+
+      def require_models
+        Dir.entries( MODELS_DIR ).grep(/.rb/).each do |model|
+          require_model( model )
+        end  
+      end  
+
+      def require_model( model )
+        require "#{MODELS_DIR}/#{model}"
+      end
+
+      def setup_constants!
+        set_constant( 'MYSQL_DB_USER' ){ 'rails' }
+        set_constant( 'AR_TEST_SUITE' ) do 
+          ENV['AR_TEST_SUITE'] || find_active_record_test_suite()
+        end
+      end
+      
+      def setup_config!
+        unless Object.const_defined?( 'MIGRATIONS_ROOT' )
+          require "#{::AR_TEST_SUITE}/config"
+        end
+      end
+
+      def set_constant( constant )
+        Object.const_set(constant, yield ) unless Object.const_defined?( constant )
+      end
+ 
+      def find_active_record_test_suite
+        returning( ($:).grep( /activerecord/ ).last.split('/') ) do |ar_ts|
+          ar_ts.pop
+          ar_ts << 'test'
+        end.join('/') 
+      end
+
+      def glob( pattern )
+        Dir.glob( pattern )
+      end        
+            
+    end
+  end
+end
+
+Mysqlplus::Test.setup!

data/test/models/mysql_user.rb

@@ -0,0 +1,3 @@
+class MysqlUser < ActiveRecord::Base  
+  set_table_name 'user'
+end

metadata

@@ -0,0 +1,70 @@
+--- !ruby/object:Gem::Specification 
+name: methodmissing-mysqlplus_adapter
+version: !ruby/object:Gem::Version 
+  version: 1.0.1
+platform: ruby
+authors: 
+- "Lourens Naud\xC3\xA9"
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-02-07 00:00:00 -08:00
+default_executable: 
+dependencies: []
+
+description: ActiveRecord Mysqlplus Adapter
+email: lourens@methodmissing.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- README.textile
+- VERSION.yml
+- lib/active_record
+- lib/active_record/connection_adapters
+- lib/active_record/connection_adapters/mysqlplus_adapter
+- lib/active_record/connection_adapters/mysqlplus_adapter/connection_pool.rb
+- lib/active_record/connection_adapters/mysqlplus_adapter/deferrable
+- lib/active_record/connection_adapters/mysqlplus_adapter/deferrable/macro.rb
+- lib/active_record/connection_adapters/mysqlplus_adapter/deferrable/result.rb
+- lib/active_record/connection_adapters/mysqlplus_adapter.rb
+- test/connections
+- test/connections/mysqlplus
+- test/connections/mysqlplus/connection.rb
+- test/deferrable_test.rb
+- test/helper.rb
+- test/models
+- test/models/mysql_user.rb
+has_rdoc: true
+homepage: http://github.com/methodmissing/mysqplus_adapter
+post_install_message: 
+rdoc_options: 
+- --inline-source
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: ActiveRecord Mysqlplus Adapter
+test_files: []
+