connection_manager 0.2.6 → 0.3.0

data/.gitignore

@@ -3,4 +3,5 @@ pkg/*
 *.sqlite3
 .bundle
 .DS_Store
-
+log/*
+Gemfile.lock

data/Gemfile

@@ -1,4 +1,3 @@
 source "http://rubygems.org"
 
-# Specify your gem's dependencies in rails-connection_manager.gemspec
 gemspec

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2011 Joshua T. Mckinney
+Copyright (c) 2011-2012 Joshua T. Mckinney
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,12 +1,22 @@
 # ConnectionManager
-Replication and Multi-Database ActiveRecord add on.
+Multi-Database, Replication and Sharding for ActiveRecord.
 
-## Goals
-* Take the lib I've been using finally make something out of it ;)
-* Use connection classes, instead of establish_connection on every model, to ensure connection pooling
-* Use non-adapter specific code.
-* Use the default database.yml as single point for all database configurations (no extra .yml files)
-* When slave objects are used in html helpers like link_to and form_for the created urls match those created using a master object
+## Background
+ActiveRecord, for quite some time now, has supported multiple database connections 
+through the use of #establish_connection and connection classes [more info](http://api.rubyonrails.org/classes/ActiveRecord/Base.html)
+Multiple databases, replication and shards can be implemented directly in rails without
+patching, but a gem helps to reduce redundant code and ensure consistency. 
+ConnectionManager replaces all the connection classes and subclasses required 
+for multiple database support in Rails with a few class methods and simple 
+database.yml configuration. Since ConnectionManager does not alter
+ActiveRecord's connection pool, thread safety is not a concern.
+
+## Upgrading to 0.3
+
+0.3 is a complete overhaul and will cause compatibility issues for folks who upgrade using the previous replication setup.
+Fortunately, for most folks the only change they have to do is specify the their slaves
+and masters in the database.yml and set build_connection_class to true to have
+ActiveRecord build their connection classes. See the example database.yml below.
 
 ## Installation
 
@@ -16,119 +26,169 @@ ConnectionManager is available through [Rubygems](https://rubygems.org/gems/conn
 
 ## Rails 3 setup (No Rails 2 at this time)
 
-ConnectionManager assumes the primary connection for the model is the master. For standard
-models using the default connection this means the main Rails database connection is the master.
+Add connection_manager to you gemfile:
+    
+    gem 'connection_manager'
 
-Example database.yml
+Run bundle install:
+    
+    bundle install
+
+### Example database.yml
 
     common: &common
     adapter: mysql2
     username: root
     password: *****
-    database_timezone: local
-    pool: 100
+    pool: 20
     connect_timeout: 20
     timeout: 900
     socket: /tmp/mysql.sock
+    build_connection_class: true
   
     development:
       <<: *common
       database: test_app
+      slaves: [slave_1_test_app_development, slave_2_test_app_development]
 
     slave_1_test_app_development:
       <<: *common
       database: test_app
+      readonly: true
   
     slave_2_test_app_development:
       <<: *common
       database: test_app
+      readonly: true
 
     user_data_development
       <<: *common
       database: user_data
+      slaves: [slave_1_user_data_development, slave_2_user_data_development]
 
     slave_1_user_data_development
       <<: *common
       database: user_data
+      readonly: true
 
     slave_2_user_data_development
       <<: *common
       database: user_data
+      readonly: true
 
 In the above database.yml the Master databases are listed as "development" and "user_data_development".
-As you can see the replication database name follow a strict standard for the connection names. 
-For slave_1_test_app_development, "slave" is the name of the replication, "1" is the count, "test_app"
-is the databases name and finally the "development" is the environment. (Of course in your database.yml
-each slave would have a different connection to is replication :)
+Replication databases are defined as normally connections and are added to the 'replications:' option for
+their master. The readonly option ensures all ActiveRecord objects returned from this connection are ALWAYS readonly.
 
 
-## Multiple Databases
+## Building Connection Classes
 
-At startup ConnectionManager builds connection classes  to ConnectionManager::Connections
-using the connections described in your database.yml based on the current rails environment.
+### Manually   
+ConnectionManager provides establish_managed_connection for build connection 
+classes and connection to multiple databases.
 
-You can use a different master by having the model inherit from one of your ConnectionManager::Connections.
+    class MyConnection < ActiveRecord::Base
+      establish_managed_connection("my_database_#{Rails.env}", :readonly => true)
+    end
+    
+    class User < MyConnection
+    end
+    
+    MyConnection    => MyConnection(abstract)
+    @user = User.first
+    @user.readonly? => true
 
-To view your ConnectionManager::Connections, at the Rails console type:
+The establish_managed_connection method, runs establish_connection with the supplied
+database.yml key, sets abstract_class to true, and (since :readonly is set to true) ensures
+all ActiveRecord objects build using this connection class are readonly. If readonly is set
+to true in the database.yml, passing the readonly option is not necessary.  
 
-   ConnectionManager::Connections.all => ["TestAppConnection", "Slave1TestAppConnection", "Slave2TestAppConnection"]
 
-If your using the example database.yml your array would look like this:
-    ["TestAppConnection", "Slave1TestAppConnection", "Slave2TestAppConnection", 
-    "UserDataConnection", "Slave1UserDataConnection", "Slave2UserDataConnection"]
+### Automatically   
+ActiveRecord can build all your connection classes for you. 
+The connection class names will be based on the database.yml keys.ActiveRecord will
+build connection classes for all the entries in the database.yml where 
+"build_connection_class" is true, and match the current environment settings
 
+## Using 
 
-To use one of your ConnectionManager::Connections for your models default/master database
-setup your model like the following
+The using method allows you specify the connection class to use
+for query. The return objects will have the correct model name, but the instance's
+class's superclass will be the connection class and all database actions performed 
+on the instance will use the connection class's connection.
     
-    class User < ConnectionManager::Connections::UserDataConnection
-        # model code ...
-    end
+    User.using("Slave1Connection").first
+
+    search = User.where(disabled => true)
+    @legacy_users = search.using("Shard1Connection").all #=> [<User::Shard1ConnectionDup...>,<User::Shard1ConnectionDup..]
+    @legacy_users.first.save #=> uses the Shard1Connection connection
+
+    @new_users = search.page(params[:page]).all => [<User...>,<User...>]
 
 ## Replication
 
-Simply add 'replicated' to your model beneath any defined associations
+Simply add 'replicated' to your model.
     
-    class User < ConnectionManager::Connections::UserDataConnection
+    class User < UserDataConnection
         has_one :job
         has_many :teams
         replicated # implement replication        
         # model code ...
     end
 
-The replicated method addeds subclass whose names match the replication connection name and count.
-Based on the above example database.yml User class would now have User::Slave1 and User::Slave2. 
+The replicated method builds models who inherit from the main model.
+    User::Slave1UserDataConnectionDup.superclass => Slave1UserDataConnection(abstract)
+    User::Slave1UserDataDup.first => returns results from slave_1_user_data_development 
+    User::Slave2UserDataDup.where(['created_at BETWEEN ? and ?',Time.now - 3.hours, Time.now]).all => returns results from slave_2_user_data_development
 
-You can treat your subclass like normal activerecord objects.
+Finally, ConnectionManager creates an additional class method that shifts through your 
+available slave connections each time it is called using a different connection on each action.
     
-    User::Slave1.first => returns results from slave_1_user_data_development 
-    User::Slave2.where(['created_at BETWEEN ? and ?',Time.now - 3.hours, Time.now]).all => returns results from slave_2_user_data_development
+    User.slaves.first  => returns results from slave_1_use_data_development 
+    User.slaves.last =>  => returns results from slave_2_use_data_development 
+    User.slaves.where(['created_at BETWEEN ? and ?',Time.now - 3.hours, Time.now]).all  => returns results from slave_1_user_data_development 
+    User.slaves.where(['created_at BETWEEN ? and ?',Time.now - 5.days, Time.now]).all  => returns results from slave_2_user_data_development 
 
-For a more elegant implementation, ConnectionManager also add class methods to your main model following the
-same naming standard as the subclass creation.
-    
-    User.slave_1.first  => returns results from slave_1_user_data_development 
-    User.slave_2.where(['created_at BETWEEN ? and ?',Time.now - 3.hours, Time.now]).all  => returns results from slave_2_user_data_development 
+Replicated defaults to the slaves replication type,so if you have only masters and a combination
+of masters and slaves for replication, you have set the replication type to masters
 
-Finally, ConnectionManager creates an addional class method that shifts through your 
-available slave connections each time it is called using a different connection on each action.
-    
-    User.slave.first  => returns results from slave_1_use_data_development 
-    User.slave.last =>  => returns results from slave_2_use_data_development 
-    User.slave.where(['created_at BETWEEN ? and ?',Time.now - 3.hours, Time.now]).all  => returns results from slave_1_user_data_development 
-    User.slave.where(['created_at BETWEEN ? and ?',Time.now - 5.days, Time.now]).all  => returns results from slave_2_user_data_development 
+    class User < UserDataConnection
+        replicated #slaves replication
+        replicated :type => :masters, :name => 'masters' # masters replication
+    end
+
+## Sharding
+
+After tinkering with some solutions for shards, I've come to a similar conclusion as [DataFabric] (https://github.com/mperham/data_fabric):
+"Sharding should be implemented at the application level". The #shards method is very basic and
+while it may be useful to most folks, it should really serve as an example of a possible solutions
+to your shard requirements.
+
+    class LegacyUser < UserShardConnection
+    end
+
+    class User < ActiveRecord::Base
+        self.shard_class_names = ["LegacyUser"]
+    end
+
+    # Calls the supplied block on all the shards available to User, including the User model itself.
+    User.shards{ |shard| shard.where(:user_name => "some_user").all} => [<User ...>,<LegacyUser ...>]
+
+## Migrations
+
+Nothing implement now to help but there are lots of potential solutions [here] (http://stackoverflow.com/questions/1404620/using-rails-migration-on-different-database-than-standard-production-or-devel)
 
-## TODO's
-* sharding - IN 2.0 AS BETA
-* cross schema joins - 2.2 AS BETA tested with Mysql2 ONLY
+## TODOs
+* Maybe add migration support for Rails AR implementations.
 
-## Other activerecord Connection gems
+## Other ActiveRecord Connection gems
+* [DataFabric] (https://github.com/mperham/data_fabric)
 * [Octopus](https://github.com/tchandy/octopus)
 
 ## Contributing to ConnectionManager
  
-* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
-* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Check out the latest master to make sure the feature has not been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already has not requested it and/or contributed it
 * Fork the project
 * Start a feature/bugfix branch
 * Commit and push until you are happy with your contribution

data/Rakefile

@@ -1 +1,3 @@
+require 'rake'
 require "bundler/gem_tasks"
+

data/connection_manager.gemspec

@@ -18,10 +18,16 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
   s.add_runtime_dependency 'activerecord', '~> 3.0'
-  s.add_development_dependency 'sqlite3'
   s.add_development_dependency 'rspec'
   s.add_development_dependency 'autotest'
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'factory_girl'
-  s.add_development_dependency 'mysql2'
-end
+  s.add_development_dependency 'activesupport', '~> 3.0'
+  
+  if(defined? RUBY_ENGINE and 'jruby' == RUBY_ENGINE)
+    s.add_development_dependency 'jruby-openssl'
+    s.add_development_dependency 'activerecord-jdbcmysql-adapter'
+  else
+    s.add_development_dependency "mysql2"
+  end
+end

data/lib/connection_manager/connection_builder.rb

@@ -0,0 +1,81 @@
+require 'active_support/core_ext/hash/indifferent_access'
+module ConnectionManager
+  module ConnectionBuilder         
+      
+    # Get the current environment if defined
+    # Check for Rails, check for RACK_ENV, default to 'development'
+    def ar_env
+      return Rails.env if defined?(Rails)
+      return RACK_ENV if defined?(RACK_ENV)
+      return ENV["AR_ENV"] if ENV["AR_ENV"]
+      "development"
+    end
+    
+    # Grab only thoses connections that correspond to the current env. If env
+    # is blank it grabs all the connection keys
+    # 
+    # If you current environment valid database keys can be:
+    # * development
+    # * other_database_development
+    # * slave_database_development  
+    def configuration_keys
+      ActiveRecord::Base.configurations.keys.select{|n| n.match(ar_env_regex)}
+    end
+      
+    def database_yml_attributes(name_from_yml)
+      found = ActiveRecord::Base.configurations[name_from_yml]
+      ActiveRecord::Base.configurations[name_from_yml].symbolize_keys if found
+    end
+        
+    # Returns currently loaded configuations where :build_connection is true
+    def database_keys_for_auto_build
+      ab_configs = []
+      configuration_keys.each do |key|    
+        ab_configs << key if (database_yml_attributes(key)[:build_connection_class] == true)
+      end
+      ab_configs
+    end
+    
+    # Builds connection classes using the database keys provided; expects an array.
+    def build_connection_classes(database_keys_to_use=database_keys_for_auto_build)      
+      database_keys_to_use.each do |key|
+        build_connection_class(connection_class_name(key),key) 
+      end
+    end 
+    
+    # Build connection classes base on the supplied class name and connection 
+    # key from database.yml
+    def build_connection_class(class_name,connection_key)
+      begin
+        class_name.constantize
+      rescue NameError 
+        klass = Class.new(ActiveRecord::Base)
+        new_connection_class = Object.const_set(class_name, klass)  
+        new_connection_class.establish_managed_connection(connection_key)
+      end
+    end
+    
+    private
+    def ar_env_regex
+      return @ar_env_regex if @ar_env_regex
+      s = "#{ar_env}$"
+      @ar_env_regex = Regexp.new("(#{s})")
+    end
+      
+    # Creates a string to be used for the class name. Removes the current env.
+    def clean_yml_key(name)
+      new_name = "#{name}".gsub(ar_env_regex,'')      
+      new_name = "Base"if new_name.blank?
+      new_name.gsub(/\_$/,'')
+    end
+      
+    # Given an connection key name from the database.yml, returns the string 
+    # equivelent of the class name for that entry.
+    def connection_class_name(name_from_yml)
+      new_class_name = clean_yml_key(name_from_yml)
+      new_class_name = new_class_name.gsub(/\_/,' ').titleize.gsub(/ /,'')
+      new_class_name << "Connection"
+      new_class_name
+    end
+  end
+end

data/lib/connection_manager/connection_manager_railtie.rb

@@ -1,7 +1,8 @@
 module ConnectionManager
   class ConnectionManagerRailtie < ::Rails::Railtie
-    initializer "connection_manager.setup" do |app|    
-      ConnectionManager::Connections.initialize  
+    initializer "connection_manager.build_connection_classes" do
+      ActiveRecord::Base.build_connection_classes
     end
   end
-end
+end
+

data/lib/connection_manager/helpers/abstract_adapter_helper.rb

@@ -0,0 +1,40 @@
+module ConnectionManager
+  module AbstractAdapterHelper
+    def config
+      @config
+    end
+    
+    def using_em_adapter?
+      (config[:adapter].match(/^em\_/) && defined?(EM) && EM::reactor_running?)
+    end
+    
+    def readonly?
+      (config[:readonly] == true)
+    end 
+    
+    def replicated?
+      (!slave_keys.blank? || !master_keys.blank?)
+    end
+   
+    def database_name
+      config[:database]
+    end
+   
+    def replication_keys(type=:slaves)
+      return slave_keys if type == :slaves
+      master_keys
+    end
+    
+    def slave_keys
+      slave_keys = []
+      slave_keys = config[:slaves].collect{|r| r.to_sym} if config[:slaves]
+      slave_keys  
+    end
+    
+    def master_keys
+      master_keys = []
+      master_keys = config[:masters].collect{|r| r.to_sym} if config[:masters]
+      master_keys
+    end
+  end
+end

data/lib/connection_manager/helpers/connection_helpers.rb

@@ -0,0 +1,105 @@
+require 'active_support/core_ext/hash/indifferent_access'
+module ConnectionManager
+  module ConnectionHelpers       
+    @@managed_connections = HashWithIndifferentAccess.new
+    
+    # Returns the database_name of the connection unless set otherwise
+    def database_name
+      @database_name = "#{connection.database_name.to_s}" unless @database_name
+      @database_name
+    end
+    
+    alias :schema_name :database_name
+    
+    # Returns true if this is a readonly only a readonly model
+    # If the connection.readonly? then the model that uses the connection 
+    # must be readonly.
+    def readonly?
+      ((@readonly == true)||connection.readonly?)
+    end
+    
+    # Allow setting of readonly at the model level
+    def readonly=readonly
+      @readonly = readonly
+    end
+    
+    # A place to store managed connections
+    def managed_connections
+      @@managed_connections
+    end
+    
+    def add_managed_connections(yml_key,value)
+      @@managed_connections[yml_key] ||= []
+      @@managed_connections[yml_key] << value unless @@managed_connections[yml_key].include?(value)
+      @@managed_connections
+    end
+    
+    def managed_connection_classes
+      managed_connections.values.flatten
+    end
+    
+    def yml_key
+      @yml_key
+    end
+    
+    # Tell Active Record to use a different database/schema on this model.
+    # You may call #use_database when your schemas reside on the same database server
+    # and you do not want to create extra connection class and database.yml entries.
+    # 
+    # Options:
+    # * :table_name_prefix - the prefix required for making cross database/schema
+    # joins for you database managmenet system. By default table_name_prefix is the
+    # database/schema name followed by a period EX: "my_database."
+    # * :table_name - the table name for the model if it does not match ActiveRecord
+    # naming conventions
+    # 
+    # EX: class LegacyUser < ActiveRecord::Base
+    #       use_database('DBUser', :table_name => 'UserData')
+    #     end
+    #     
+    #     LegacyUser.limit(1).to_sql => "SELECT * FROM `BDUser`.`UserData` LIMIT 1
+    #
+    def use_database(database_name,opts={})
+      @database_name = database_name
+      opts[:table_name_prefix] ||= "#{database_name}."
+      opts[:table_name] ||= self.table_name.to_s.split('.').last
+      self.table_name = opts[:table_name]
+      self.table_name_prefix = opts[:table_name_prefix]
+    end
+    
+    alias :use_schema :use_database
+   
+    # Establishes and checks in a connection, noramlly for abstract classes aka connection classes.
+    # 
+    # Options:
+    # * :abstract_class - used the set #abstract_class, default is true
+    # * :readonly - force all instances to readonly
+    # * :class_name - name of connection class name, default is current class's name
+    # * :table_name_prefix - prefix to append to table name for cross database joins,
+    #     default is the "#{self.database_name}."
+    # EX:
+    #   class MyConnection < ActiveRecord::Base
+    #     establish_managed_connection :key_from_db_yml,{:readonly => true}
+    #   end
+    #
+    def establish_managed_connection(yml_key,opts={})
+      @yml_key = yml_key
+      opts = {:class_name => self.name, 
+        :abstract_class => true}.merge(opts)   
+      establish_connection(yml_key)     
+      self.abstract_class = opts[:abstract_class]
+      set_to_readonly if (readonly? || opts[:readonly] || self.connection.readonly?)
+      add_managed_connections(yml_key,opts[:class_name])
+      use_database(self.database_name,opts)
+    end
+          
+    # Override ActiveRecord::Base instance method readonly? to force 
+    # readonly connections.
+    def set_to_readonly
+      self.readonly = true
+      define_method(:readonly?) do 
+        true
+      end  
+    end
+  end
+end

data/lib/connection_manager/{cross_schema_patch.rb → patches/cross_schema_patch.rb}

@@ -5,8 +5,7 @@ if ActiveRecord::VERSION::MAJOR == 3 && ActiveRecord::VERSION::MINOR == 0
   module ActiveRecord
     module ConnectionAdapters
       class Mysql2Adapter < AbstractAdapter     
-        
-        def real_tables(name = nil, database = nil) #:nodoc:
+        def new_tables(database = nil) #:nodoc:
           sql = ["SHOW TABLES", database].compact.join(' IN ')
           execute(sql, 'SCHEMA').collect do |field|
             field.first
@@ -21,7 +20,7 @@ if ActiveRecord::VERSION::MAJOR == 3 && ActiveRecord::VERSION::MINOR == 0
             table  = schema
             schema = nil
           end
-          real_tables(nil, schema).include? table
+          new_tables(schema).include? table
         end
       end
     end