decisiv-sharded_database 0.1.5.1 → 0.2.0.1

data/README.rdoc

@@ -9,7 +9,7 @@ The first step was solved by creating a SQL view to pool all production records
 
 == Uses
 
-Currently, sharded_database works by reflecting on a set of returned records, and taking the result of #determine_connection to delegate the connection to - for each instance. The path we chose had us creating a view that would return aggregate records, with an added 'source' column specifying the originating database. We then rigged our #determine_connection to return an abstract connection model depending on the value. Other ideas for implementation include an after_create callback that writes the source database / ID to a centralized table.
+Currently, sharded_database works by reflecting on a set of returned records, and taking the result of #sharded_connection_klass method to delegate the connection to - for each instance. The path we chose had us creating a view that would return aggregate records, with an added 'source' column specifying the originating database. We then rigged our #sharded_connection_klass to return an abstract connection model depending on the value. Other ideas for implementation include an after_create callback that writes the source database / ID to a centralized table.
 
 
 == Example and Usage
@@ -20,13 +20,16 @@ Setup your Aggregate model, inheriting from a class that establishes a connectio
 
   class AggregateFoo < ActiveRecord::Base
     include ShardedDatabase::Aggregate
+    source_class 'Foo'
 
-    def determine_connection
-      Connections[source.to_sym]
+    def sharded_connection_klass
+      "Connection::#{source.classify}".constantize
     end
   end
   
-The #determine_connection class above expects that there is a Connections constant that is a hash of key/value pairs of 'source' symbols and abstract models that connect to various databases. This method can be implemented in any way you desire to determine the connection source.
+The constant returned by #sharded_connection_klass is expected to be an subclass of ActiveRecord::Base and respond to #connection. It is suggested that you use the preserve_attributes class method in the aggregate class and use that attribute to augment and/or find the constant somehow.
+
+As of 0.2.0, the class method #source_class is required - sharded_database no longer guesses for the correct source class!
 
 
 === Loading Aggregate Records
@@ -43,12 +46,70 @@ Updating an attribute on the first record in the above array would update the co
 Associations are also taken into account, and any associations that are defined on a Foo model will be sourced correctly. I have only tested this one level deep, though.
 
 
-=== Accessing 'raw' AggregateFoo objects
+=== Accessing Non-Proxyable AggregateFoo Objects
+
+Easily done. Simply add a :aggregate_proxy option set to false to your finders.
+
+  AggreateFoo.all(:aggregate_proxy => false)
+
+
+== Working without an AggregateFoo table
+
+In the event that you need to load a Foo object directly, and you may or may not know the appropriate connection at runtime, sharded_database is able to apply itself to an ActiveRecord model directly.
+
+This is done by accepting a :connection key in the options hash for a #find method. The value of :connection can either be an abstract ActiveRecord class, a Proc, or a Symbol, which will attempt to call a method of the same name on the class doing the finding. The latter two options are also supplied the #find arguments, which is useful when wanting to dynamically determine a connection.
+
+
+=== Setup
+
+Any class that includes ShardedDatabase::Aggregate and defines a source_class will have that class be given this functionality. To add manually:
+
+  class Bar < ActiveRecord::Base
+    include ShardedDatabase::ModelWithConnection
+  end
+
+
+=== Usage
+
+As mentioned above, there are three acceptable values for the :connection key. Borrowing heavily from ActionController's callback setup, acceptable values can either be an ActiveRecord class, or a Proc or class method name (represented as a symbol) that will ultimately return an ActiveRecord class. Both the Proc and method will be passed a splat of the attributes passed to #find.
+
+The method implementation will be demonstrated in the Example, but the other implementations are as follows: 
+
+<b>Supplying a class</b>
+  Bar.find(:first, :order => 'created_at asc', :connection => BarConnection)
+
+<b>Supplying a Proc</b>
+  Bar.find(123, :connection => lambda { |*args| (args.first % 2) == 1 ? Connection::One : Connection::Two })
+
+...Which would use Connection::One, as % 2 of 123 is 1.
+
+
+=== Example
+
+A real world situation would be something akin to Flickr. Data is sharded over N number of databases, and the criteria sent to your finder method would be used to connect to the right datastore.
+
+Imagine you have a photo sharing application, and due to the amount of photos managed, you're forced to shard the 'photos' database across 3 database servers. To do this with sharded_database, you'd do something like:
+
+  class Photo < ActiveRecord::Base
+  
+    def self.determine_database(*attrs)   # [:all, { :conditions => { :name => 'mountain' } }]
+      case first_letter = attrs.last[:conditions][:name].first[0,1]  # get the first letter
+      when a..f   : Connection::AToF
+      when g..p   : Connection::GToP
+      when q..z   : Connection::QtoZ
+    end
+  
+  end
+  
+  Photo.all(:conditions => { :name => 'mountain' }, :connection => :determine_database)
+
+Which would return any photo named 'mountain' against the database Connection::GToP is connected to.
 
-Easily done. Simply add a :raw option set to true to your finder queries.
 
-  AggreateFoo.all(:raw => true)
+== TODO
 
+- Apply associations to objects found using the :connection key
+- Fix association application to regenerate a proper reflection/association
 
 
-Copyright &copy; 2008, Brennan Dunn, Decisiv Inc. Released under the MIT license.
+Copyright &copy; 2008, Brennan Dunn, Decisiv Inc. Released under the MIT license.

data/lib/sharded_database/aggregate.rb

@@ -7,55 +7,86 @@ module ShardedDatabase
       klass.extend         ClassMethods
       klass.send :include, InstanceMethods
       klass.class_eval do
-        cattr_accessor :connection_field, :source_class, :foreign_id
-        @connection_field = :oem
+        cattr_accessor :foreign_id
         @foreign_id = :other_id
 
         class << self
-          alias_method_chain :find, :raw
+          alias_method_chain :find, :aggregate_proxy
         end
       end
     end
 
     module ClassMethods
 
-      def find_with_raw(*args)
-        @raw = args.last.is_a?(Hash) && args.last.delete(:raw)
-        @raw ? temporarily_undef_method(:after_find) { find_without_raw(*args) } : find_without_raw(*args)
+      def find_with_aggregate_proxy(*args)
+        without_aggregate_proxy = args.last.is_a?(Hash) && args.last.delete(:aggregate_proxy).is_a?(FalseClass)
+        if without_aggregate_proxy
+          temporarily_undef_method(:after_find) { find_without_aggregate_proxy(*args) }
+        else
+          find_without_aggregate_proxy(*args)
+        end
       end
-
+      
       def preserve_attributes(*attrs)
         @preserved_attributes = attrs.map(&:to_s)
       end
+      
+      def source_class(klass_name)
+        @source_class = klass_name
+        apply_model_with_connection_to(klass_name)
+      end
+      
+      
+      private
+      
+      def apply_model_with_connection_to(klass_name)
+        require_dependency klass_name.underscore unless defined?(klass_name.constantize)  # ensure that the source class has been loaded
+        klass_name.constantize.send :include, ShardedDatabase::ModelWithConnection
+      end
 
     end
 
     module InstanceMethods
       
-      def determine_connection
-        # stub method - implement your own!
+      def sharded_connection_klass
+        raise NotImplementedError,
+          "You must implement your own #sharded_connection_klass method that returns an ActiveRecord::Base subclass which yeilds a connection."
       end
       
       def after_find
-        @klass      = determine_connection || raise(ShardedDatabase::NoConnectionError, 'Cannot determine connection class')
+        @klass      = sharded_connection_klass
         @connection = @klass.respond_to?(:connection) ? @klass.connection : raise(ShardedDatabase::NoConnectionError, 'Connection class does not respond to :connection')
         @foreign_id = self[self.class.foreign_id.to_sym]
 
         metaclass.delegate :connection, :to => @klass
-
+        
+        preserve_attributes
+        apply_proxy
+        channel_associations_to_proper_connection
+        
+      end
+      
+      
+      private
+      
+      def preserve_attributes
         (self.class.instance_variable_get("@preserved_attributes") || []).each do |attr|
-          metaclass.send :alias_method, "proxy_#{attr}", attr
+          metaclass.send :alias_method, "original_#{attr}", attr
         end
-
+      end
+      
+      def apply_proxy
         class << self
           alias_method :proxy_class, :class
-
+          
           include AggregateProxy
           instance_methods.each do |m|
-            undef_method(m) unless m =~ /^__|proxy_|inspect/
+            undef_method(m) unless m =~ /^__|proxy_|original_|inspect/
           end
         end
-
+      end
+      
+      def channel_associations_to_proper_connection
         self.class.reflect_on_all_associations.each do |a| 
           metaclass.send :alias_method, "proxy_#{a.name}".to_sym, a.name.to_sym
           metaclass.send :undef_method, a.name
@@ -83,7 +114,6 @@ module ShardedDatabase
             end
           }, __FILE__, __LINE__
         end
-        
       end
       
     end

data/lib/sharded_database/aggregate_proxy.rb

@@ -9,20 +9,21 @@ module ShardedDatabase
       load_target.inspect.gsub(/#<([\w\:]+)\s(.*?)>/) { "#<#{$1}(#{@klass.name}) #{$2}>" }
     end
 
+    def respond_to?(method)
+      load_target.respond_to?(method) || super
+    end
+
 
     private
 
     def load_target
       @target ||= begin
-                    klass = (self.proxy_class.source_class || self.proxy_class.name.gsub('Aggregate','')).constantize
-                    borrow_connection(klass, @klass) { |k| k.find(@foreign_id) }
+                    klass = (self.proxy_class.instance_variable_get('@source_class')).constantize
+                    ModelWithConnection.borrow_connection(klass, @klass) { |k| k.find(@foreign_id) }
                   end
     end
 
     def method_missing(method, *args, &block)
-      if association_method?(method)
-        #apply_connection_to_association(method)
-      end
       load_target.respond_to?(method) ? load_target.send(method, *args, &block) : super
     end
 
@@ -30,11 +31,5 @@ module ShardedDatabase
       load_target.class.reflect_on_all_associations.any? { |a| a.name == method.to_sym }
     end
 
-    def borrow_connection(requesting_class, target_class, &block)
-      eigen = requesting_class.metaclass
-      eigen.delegate :connection, :to => target_class
-      yield(requesting_class)
-    end
-
   end
 end

data/lib/sharded_database.rb

@@ -1,9 +1,10 @@
+require 'sharded_database/model_with_connection'
 require 'sharded_database/aggregate_proxy'
 require 'sharded_database/aggregate'
 require 'sharded_database/core_extensions'
 
 module ShardedDatabase
   
-  VERSION = '0.1.5'
+  VERSION = '0.2.0'
   
 end

data/test/lib/database.yml

@@ -5,16 +5,17 @@ master:
   password: 
   host: localhost
 
-one_db:
+shard_one:
   adapter: mysql
   database: sharded_database_one
   username: root
   password: 
   host: localhost
   
-two_db:
+shard_two:
   adapter: mysql
   database: sharded_database_two
   username: root
   password: 
-  host: localhost
+  host: localhost
+

data/test/lib/models.rb

@@ -1,42 +1,33 @@
 module Connection ; end
 
 class Connection::One < ActiveRecord::Base
-  establish_connection :one_db
+  establish_connection :shard_one
   self.abstract_class = true
 end
 
 class Connection::Two < ActiveRecord::Base
-  establish_connection :two_db
+  establish_connection :shard_two
   self.abstract_class = true
 end
 
-Connections = { :one => Connection::One, :two => Connection::Two }
-
 class GlobalConnection < ActiveRecord::Base
   establish_connection :master
   self.abstract_class = true
 end
 
-class AggregateEstimate < GlobalConnection
-  belongs_to :gun
-  include ShardedDatabase::Aggregate
-  self.foreign_id   = :other_id
-  preserve_attributes :source
-  
-  def determine_connection
-    Connections[source.to_sym]
-  end
-  
-end
-
 class Company < ActiveRecord::Base
   has_many :items
 end
 
-class Estimate < ActiveRecord::Base
+class Employee < ActiveRecord::Base
   belongs_to :company
   has_many :items
   
+  def self.pick_connection(*args)
+    id_to_find = args.first
+    (id_to_find % 2) == 1 ? Connection::One : Connection::Two
+  end
+  
   def call_company
     company
   end
@@ -44,5 +35,18 @@ class Estimate < ActiveRecord::Base
 end
 
 class Item < ActiveRecord::Base
-  belongs_to :estimate
+  belongs_to :employee
+end
+
+class AggregateEmployee < GlobalConnection
+  belongs_to :gun
+  include ShardedDatabase::Aggregate
+  self.foreign_id   = :other_id
+  source_class 'Employee'
+  preserve_attributes :source
+  
+  def sharded_connection_klass
+    "Connection::#{source.classify}".constantize
+  end
+  
 end

data/test/lib/test_case.rb

@@ -21,11 +21,11 @@ module ShardedDatabase
         SQLite3::Database.new(file)
       end
       
-      # setup aggregate table
+      # Setup master table
       ::ActiveRecord::Base.establish_connection :master
       ::ActiveRecord::Base.class_eval do
         silence do
-          connection.create_table :aggregate_estimates, :force => true do |t|
+          connection.create_table :aggregate_employees, :force => true do |t|
             t.string    :source
             t.integer   :other_id
             t.timestamp :created_at
@@ -33,18 +33,18 @@ module ShardedDatabase
         end
       end
       
-      # setup NUM_db.estimates
+      # Setup sharded DBs for employees
       %w(one two).each do |num|
-        ::ActiveRecord::Base.establish_connection "#{num}_db".to_sym
+        ::ActiveRecord::Base.establish_connection "shard_#{num}".to_sym
         ::ActiveRecord::Base.class_eval do
           silence do
-            connection.create_table :estimates, :force => true do |t|
+            connection.create_table :employees, :force => true do |t|
               t.belongs_to  :company
               t.string      :name
             end
             
             connection.create_table :items, :force => true do |t|
-              t.belongs_to  :estimate
+              t.belongs_to  :employee
               t.string      :name
             end
             
@@ -62,19 +62,19 @@ module ShardedDatabase
       one_company = Class.new(Connection::One) { set_table_name 'companies' }
       @company_1 = one_company.create :name => 'One Company'
       
-      one_estimate = Class.new(Connection::One) { set_table_name 'estimates' ; has_many(:items) ; belongs_to(:company) }
-      @one_1 = one_estimate.create :name => 'One Estimate', :company_id => @company_1.id
+      one_employee = Class.new(Connection::One) { set_table_name 'employees' ; has_many(:items) ; belongs_to(:company) }
+      @one_1 = one_employee.create :name => 'One Employee', :company_id => @company_1.id
       
-      two_estimate = Class.new(Connection::Two) { set_table_name 'estimates' ; has_many(:items) }
-      @two_1 = two_estimate.create :name => 'One Estimate 1'
-      @two_2 = two_estimate.create :name => 'Two Estimate 2'
+      two_employee = Class.new(Connection::Two) { set_table_name 'employees' ; has_many(:items) }
+      @two_1 = two_employee.create :name => 'One Employee 1'
+      @two_2 = two_employee.create :name => 'Two Employee 2'
       
-      one_item = Class.new(Connection::One) { set_table_name 'items' ; belongs_to(:estimate) }
-      one_item.create :name => 'One Test Item', :estimate_id => @one_1.id
+      one_item = Class.new(Connection::One) { set_table_name 'items' ; belongs_to(:employee) }
+      one_item.create :name => 'One Test Item', :employee_id => @one_1.id
             
-      AggregateEstimate.create :source => 'one', :other_id => @one_1.id
-      AggregateEstimate.create :source => 'two', :other_id => @two_1.id
-      AggregateEstimate.create :source => 'two', :other_id => @two_2.id
+      AggregateEmployee.create :source => 'one', :other_id => @one_1.id
+      AggregateEmployee.create :source => 'two', :other_id => @two_1.id
+      AggregateEmployee.create :source => 'two', :other_id => @two_2.id
     end
     
   end

data/test/sharded_database/association_test.rb

@@ -4,25 +4,25 @@ class AssociationTest < ShardedDatabase::TestCase
   
   def setup
     setup_environment
-    @parent = AggregateEstimate.find_by_source('one')
+    @parent = AggregateEmployee.find_by_source('one')
   end
   
   context 'Connection delegation on has_many associations' do
     
     should 'fetch items from the parent instance connection' do
       assert ! @parent.items.empty?
-      assert_connection :one_db, @parent.items.first
+      assert_connection :shard_one, @parent.items.first
     end
     
     should 'keep its connection when bubbling up to an associations parent' do
-      assert_equal @parent, @parent.items.first.estimate
+      assert_equal @parent, @parent.items.first.employee
     end
     
   end
   
   context '[UNFINISHED] Connection delegation on belongs_to associations' do
     
-    should 'fetch the associated company for an estimate from the respective connection' do
+    should 'fetch the associated company for an employee from the respective connection' do
       assert_instance_of Company, @parent.company
     end
     

data/test/sharded_database/connection_test.rb

@@ -5,18 +5,38 @@ class ConnectionTest < ShardedDatabase::TestCase
   
   
   should 'foundationally support instances having a different connection than their parent class' do
-    assert_not_equal AggregateEstimate.first.connection, AggregateEstimate.connection
+    assert_not_equal AggregateEmployee.first.connection, AggregateEmployee.connection
   end
   
   should 'have instances of the same source share the same connection' do
-    first, second = AggregateEstimate.find_all_by_source('two', :limit => 2)
+    first, second = AggregateEmployee.find_all_by_source('two', :limit => 2)
     
-    assert_connection :two_db, first, second
+    assert_connection :shard_two, first, second
     assert_equal first.connection.object_id, second.connection.object_id  # ensure that delegation is working correctly
   end
   
   should 'display instance connection when inspecting' do
-    assert_match %{(Connection::One)}, AggregateEstimate.find_by_source('one').inspect
+    assert_match %{(Connection::One)}, AggregateEmployee.find_by_source('one').inspect
+  end
+  
+  context 'loading records when given a :connection key in the options hash' do
+    
+    should 'allow for an ActiveRecord::Base class to be supplied' do
+      object = Employee.first(:connection => Connection::One)
+      assert_connection :shard_one, object      
+    end
+    
+    should 'allow for a Proc object to be supplied' do
+      proc = lambda { |*args| (args.first % 2) == 1 ? Connection::One : Connection::Two }
+      
+      assert_connection :shard_one, Employee.find(1, :connection => proc)
+      assert_connection :shard_two, Employee.find(2, :connection => proc)
+    end
+    
+    should 'allow for a symbol to be supplied (which calls a method)' do
+      assert_connection :shard_one, Employee.find(1, :connection => :pick_connection)
+    end
+    
   end
     
 end

data/test/sharded_database/instance_test.rb

@@ -4,14 +4,14 @@ class InstanceTest < ShardedDatabase::TestCase
   def setup ; setup_environment ; end
   
   
-  context 'Loading raw aggregate objects' do
+  context 'Loading non-proxyable aggregate objects' do
     
     setup do
-      @aggregates = AggregateEstimate.all(:raw => true)
+      @aggregates = AggregateEmployee.all(:aggregate_proxy => false)
     end
     
-    should 'all be AggregateEstimate instances' do
-      assert @aggregates.all? { |a| a.is_a?(AggregateEstimate) }
+    should 'all be AggregateEmployee instances' do
+      assert @aggregates.all? { |a| a.is_a?(AggregateEmployee) }
     end
     
   end
@@ -19,21 +19,25 @@ class InstanceTest < ShardedDatabase::TestCase
   context "A transformed aggregate instance" do
     
     setup do
-      @estimate = AggregateEstimate.first
+      @employee = AggregateEmployee.first
     end
-
+    
     should 'channel calls to #class to the proxy class' do
-      assert @estimate.is_a?(Estimate)
+      assert @employee.is_a?(Employee)
     end
     
     should 'have the same attribute fields as the proxy class' do
-      assert_same_elements @estimate.attributes.keys, Estimate.column_names
+      assert_same_elements @employee.attributes.keys, Employee.column_names
     end
     
     should 'preserve attributes supplied to #preserve_attributes' do
-      assert_equal 'one', @estimate.proxy_source
+      assert_equal 'one', @employee.original_source
     end
     
+    should 'have #respond_to? proxy to the instance' do
+      assert @employee.respond_to?(:call_company)
+    end
+
   end
   
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: decisiv-sharded_database
 version: !ruby/object:Gem::Version 
-  version: 0.1.5.1
+  version: 0.2.0.1
 platform: ruby
 authors: 
 - Brennan Dunn