db-charmer 1.5.5 → 1.6.0

data/CHANGES

@@ -1,3 +1,10 @@
+1.6.0 (2010-03-31):
+
+The major (and arguably the only noticeable) change in this version is our simple database
+sharding support. The feature is still in alpha stage and should not be used in production
+without complete understanding of the principles of its work.
+
+----------------------------------------------------------------------------------------
 1.5.5 (2010-03-15):
 
 Thanks to ngmoco.com (http://github.com/ngmoco) now DbCharmer supports one more use-case

data/README.rdoc

@@ -17,7 +17,7 @@ There are two options when approaching db-charmer installation:
 
 To install as a gem, add this to your environment.rb:
 
-  config.gem 'db-charmer', :lib => 'db_charmer', :source => 'http://gemcutter.org'
+  config.gem 'db-charmer', :lib => 'db_charmer'
 
 And then run the command:
 
@@ -249,6 +249,24 @@ and HABTM associations connection switching:
   @user.on_slave.posts
 
 
+=== Named Scopes Support
+
+To make it easier for +DbCharmer+ users to use connections switching methods with named scopes,
+we've added <tt>on_*</tt> methods support on the scopes as well. All the following scope chains
+would do exactly the same way (the query would be executed on the :foo database connection):
+
+  Post.on_db(:foo).published.with_comments.spam_marked.count
+  Post.published.on_db(:foo).with_comments.spam_marked.count
+  Post.published.with_comments.on_db(:foo).spam_marked.count
+  Post.published.with_comments.spam_marked.on_db(:foo).count
+
+And now, add this feature to our associations support and here is what we could do:
+
+  @user.on_db(:archive).posts.published.all
+  @user.posts.on_db(:olap).published.count
+  @user.posts.published.on_db(:foo).first
+
+
 === Bulk Connection Management
 
 Sometimes you want to run code where a large number of tables may be used, and you'd like
@@ -266,28 +284,178 @@ You can specify any number of remappings at once, and you can also use +:master+
 name that matches any model that has not had its connection set by +DbCharmer+ at all.
 
 *Note*: +DbCharmer+ works via +alias_method_chain+ in model classes. It is very careful
-to only patch the models it needs to. However, if you use +with_remapped_databases+ and 
+to only patch the models it needs to. However, if you use +with_remapped_databases+ and
 remap the default database (+:master+), then it has no choice but to patch all subclasses
 of +ActiveRecord::Base+. This should not cause any serious problems or any big performance
 impact, but it is worth noting.
 
 
-=== Named Scopes Support
+== Simple Sharding Support
 
-To make it easier for +DbCharmer+ users to use connections switching methods with named scopes,
-we've added <tt>on_*</tt> methods support on the scopes as well. All the following scope chains
-would do exactly the same way (the query would be executed on the :foo database connection):
+Starting with the release 1.6.0 of +DbCharmer+ we have added support for simple database sharding
+to our ActiveRecord extensions. The feature is still in alpha stage and should not be used in
+production without complete understanding of the principles of its work.
 
-  Post.on_db(:foo).published.with_comments.spam_marked.count
-  Post.published.on_db(:foo).with_comments.spam_marked.count
-  Post.published.with_comments.on_db(:foo).spam_marked.count
-  Post.published.with_comments.spam_marked.on_db(:foo).count
+At this point we support three sharding methods:
 
-And now, add this feature to our associations support and here is what we could do:
+1) range - really simple sharding method that allows you to take a table, slice is to a set of
+smaller tables with pre-defined ranges of primary keys and then put those smaller tables to
+different databases/servers. This could be useful for situations where you have a huge table that
+is slowly growing and you just want to keep it simple and split the table load into a few servers
+without building any complex sharding schemes.
 
-  @user.on_db(:archive).posts.published.all
-  @user.posts.on_db(:olap).published.count
-  @user.posts.published.on_db(:foo).first
+2) hash_map - pretty simple sharding method that allows you to take a table and slice it to a set
+of smaller tables by some key that has a pre-defined key of values. For example, list of US mailing
+addresses could be sharded by states, where you'd be able to define which states are stored in which
+databases/servers.
+
+3) db_block_map - this is a really complex sharding method that allows you to shard your table into a
+set of small fixed-size blocks that then would be assigned to a set of shards (databases/servers).
+Whenever you would need an additional blocks they would be allocated automatically and then balanced
+across the shards you have defined in your database. This method could be used to scale out huge
+tables with hundreds of millions to billions of rows and allows relatively easy re-sharding techniques
+to be implemented on top.
+
+
+=== How to enable sharding?
+
+To enable sharding extensions you need to take a few things:
+
+1) Create a Rails initializer (on run this code when you initialize your script/application) with a
+set of sharded connections defined. Each connection would have a name, sharding method and an optional
+set of parameters to initialize the sharding method of your choice.
+
+2) Specify sharding connection you want to use in your models.
+
+3) Specify the shard you want to use before doing any operations on your models.
+
+For more details please check out the following documentation sections.
+
+
+=== Sharded Connections
+
+Sharded connection is a simple abstractions that allows you to specify all sharding parameters for a
+cluster in one place and then use this centralized configuration in your models. Here are a few examples
+of sharded connections initizlization calls:
+
+1) Sample range-based sharded connection:
+
+  TEXTS_SHARDING_RANGES = {
+    0...100   => :shard1,
+    100..200  => :shard2,
+    :default  => :shard3
+  }
+
+  DbCharmer::Sharding.register_connection(
+    :name => :texts,
+    :method => :range,
+    :ranges => TEXTS_SHARDING_RANGES
+  )
+
+2) Sample hash map sharded connection:
+
+  SHARDING_MAP = {
+    'US'  => :us_users,
+    'CA'  => :ca_users,
+    :default  => :other_users
+  }
+
+  DbCharmer::Sharding.register_connection(
+    :name => :texts,
+    :method => :range,
+    :map => SHARDING_MAP
+  )
+
+3) Sample database block map sharded connection:
+
+  DbCharmer::Sharding.register_connection(
+    :name => :social,
+    :method => :db_block_map,
+    :block_size => 10000,                   # Number of keys per block
+    :map_table => :event_shards_map,        # Table with blocks to shards mapping
+    :shards_table => :event_shards_info,    # Shards connection information table
+    :connection => :social_shard_info       # What connection to use to read the map
+  )
+
+After your sharded connection is defined, you could use it in your models:
+
+  class Text < ActiveRecord::Base
+    db_magic :sharded => {
+      :key => :id,
+      :sharded_connection => :texts
+    }
+  end
+
+  class Event < ActiveRecord::Base
+    set_table_name :timeline_events
+
+    db_magic :sharded => {
+      :key => :to_uid,
+      :sharded_connection => :social
+    }
+  end
+
+
+=== Switching connections in sharded models
+
+Every time you need to perform an operation on a sharded model, you need to specify on which shard
+you want to do it. We have a method for this which would look familiar for the people that use
++DbCharmer+ for non-sharded environments since it looks and works just like those per-query
+connection management methods:
+
+  Event.shard_for(10).find(:conditions => { :to_uid => 123 }, :limit => 5)
+  Text.shard_for(123).find_by_id(123)
+
+There is another method that could be used with range and hash_map sharding methods, this method
+allows you to switch to the default shard:
+
+  Text.on_default_shard.create(:body => 'hello', :user_id => 123)
+
+And finally, there is a method that allows you to run your code on each shard in the system (at this
+point the method is supported in db_block_map method only):
+
+  Event.on_each_shard { |event| event.delete_all }
+
+
+=== Defining your own sharding methods
+
+It is possing with +DbCharmer+ for the users to define their own sharding methods. You need to do a
+few things to implement your very own sharding scheme:
+
+1) Create a class with a name DbCharmer::Sharding::Method::YourOwnName
+
+2) Implement at least a constructor <tt>initialize(config)</tt> and a lookup instance
+method <tt>shard_for_key(key)</tt> that would return either a connection name from <tt>database.yml</tt>
+file or just a hash of connection parameters for rails connection adapters.
+
+3) Register your sharded connection using the following call:
+
+  DbCharmer::Sharding.register_connection(
+    :name => :some_name,
+    :method => :your_own_name,    # your sharder class name in lower case
+    ... some additional parameters if needed ...
+  )
+
+4) Use your sharded connection as any standard one.
+
+
+=== Adding support for default shards in your custom sharding methods
+
+If you want to be able to use +on_default_shard+ method on your custom-sharded models, you
+need to do two things:
+
+1) implement <tt>support_default_shard?</tt> instance method on your sharded class that
+would return +true+ if you do support default shard specification and +false+ otherwise.
+
+2) implement <tt>:default</tt> symbol support as a key in your +shard_for_key+ method.
+
+
+=== Adding support for shards enumeration in your custom sharding methods
+
+To add shards enumeration support to your custom-sharded models you need to implement
+just one instance method +shard_connections+ on your sharded class. This method should
+return an array of sharding connection names or connection configurations to be used to
+establish shard connections in a loop.
 
 
 == Documentation

data/VERSION

@@ -1 +1 @@
-1.5.5
+1.6.0

data/db-charmer.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{db-charmer}
-  s.version = "1.5.5"
+  s.version = "1.6.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Alexey Kovyrin"]
-  s.date = %q{2010-03-15}
+  s.date = %q{2010-03-31}
   s.description = %q{ActiveRecord Connections Magic (slaves, multiple connections, etc)}
   s.email = %q{alexey@kovyrin.net}
   s.extra_rdoc_files = [
@@ -38,6 +38,12 @@ Gem::Specification.new do |s|
      "lib/db_charmer/multi_db_migrations.rb",
      "lib/db_charmer/multi_db_proxy.rb",
      "lib/db_charmer/scope_proxy.rb",
+     "lib/db_charmer/sharding.rb",
+     "lib/db_charmer/sharding/connection.rb",
+     "lib/db_charmer/sharding/method/db_block_map.rb",
+     "lib/db_charmer/sharding/method/hash_map.rb",
+     "lib/db_charmer/sharding/method/range.rb",
+     "lib/db_charmer/stub_connection.rb",
      "lib/tasks/databases.rake"
   ]
   s.homepage = %q{http://github.com/kovyrin/db-charmer}

data/lib/db_charmer.rb

@@ -1,5 +1,3 @@
-#puts "Loading DbCharmer..."
-
 module DbCharmer
   @@migration_connections_should_exist = Rails.env.production?
   mattr_accessor :migration_connections_should_exist
@@ -19,7 +17,7 @@ module DbCharmer
     return Rails.logger if defined?(Rails)
     @logger ||= Logger.new(STDERR)
   end
-  
+
   def self.with_remapped_databases(mappings, &proc)
     old_mappings = ActiveRecord::Base.db_charmer_database_remappings
     begin
@@ -33,18 +31,21 @@ module DbCharmer
       ActiveRecord::Base.db_charmer_database_remappings = old_mappings
     end
   end
-  
+
   def self.hijack_new_classes?
     @@hijack_new_classes
   end
-  
-  private
+
+private
+
   @@hijack_new_classes = false
   def self.with_all_hijacked
     old_hijack_new_classes = @@hijack_new_classes
     begin
       @@hijack_new_classes = true
-      ActiveRecord::Base.send(:subclasses).each { |s| s.hijack_connection! }
+      ActiveRecord::Base.send(:subclasses).each do |subclass|
+        subclass.hijack_connection!
+      end
       yield
     ensure
       @@hijack_new_classes = old_hijack_new_classes
@@ -52,6 +53,8 @@ module DbCharmer
   end
 end
 
+# These methods are added to all objects so we could call proxy? on anything
+# and figure if an object is a proxy w/o hitting method_missing or respond_to?
 class Object
   def self.proxy?
     false
@@ -65,8 +68,6 @@ end
 # We need blankslate for all the proxies we have
 require 'blankslate'
 
-#puts "Extending AR..."
-
 require 'db_charmer/active_record_extensions'
 require 'db_charmer/abstract_adapter_extensions'
 
@@ -116,8 +117,6 @@ module ActiveRecord
   end
 end
 
-#puts "Doing the magic..."
-
 require 'db_charmer/db_magic'
 require 'db_charmer/finder_overrides'
 require 'db_charmer/association_preload'
@@ -140,7 +139,7 @@ class ActiveRecord::Base
       hijack_connection! if DbCharmer.hijack_new_classes?
       out
     end
-    
+
     alias_method_chain :inherited, :hijacking
   end
 end

data/lib/db_charmer/active_record_extensions.rb

@@ -3,14 +3,16 @@ module DbCharmer
     module ClassMethods
 
       def establish_real_connection_if_exists(name, should_exist = false)
-        unless config = configurations[RAILS_ENV][name.to_s]
+        name = name.to_s
+        config = configurations[RAILS_ENV][name]
+        unless config
           if should_exist
             raise ArgumentError, "Invalid connection name (does not exist in database.yml): #{RAILS_ENV}/#{name}"
           end
           return # No need to establish connection - they do not want us to
         end
         # Pass connection name with config
-        config[:connection_name] = name.to_s
+        config[:connection_name] = name
         establish_connection(config)
       end
 
@@ -69,15 +71,15 @@ module DbCharmer
         return nil if (db_charmer_connection_level || 0) > 0
         name = db_charmer_connection_proxy.db_charmer_connection_name if db_charmer_connection_proxy
         name = (name || :master).to_sym
-        
+
         remapped = @@db_charmer_database_remappings[name]
         return DbCharmer::ConnectionFactory.connect(remapped, true) if remapped
       end
-      
+
       def db_charmer_database_remappings
         @@db_charmer_database_remappings
       end
-      
+
       def db_charmer_database_remappings=(mappings)
         raise "Mappings must be nil or respond to []" if mappings && (! mappings.respond_to?(:[]))
         @@db_charmer_database_remappings = mappings || { }

data/lib/db_charmer/connection_factory.rb

@@ -13,34 +13,64 @@ module DbCharmer
     end
 
     # Establishes connection or return an existing one from cache
-    def self.connect(db_name, should_exist = false)
-      db_name = db_name.to_s
-      @@connection_classes[db_name] ||= establish_connection(db_name, should_exist)
+    def self.connect(connection_name, should_exist = false)
+      connection_name = connection_name.to_s
+      @@connection_classes[connection_name] ||= establish_connection(connection_name, should_exist)
+    end
+
+    # Establishes connection or return an existing one from cache (not using AR database configs)
+    def self.connect_to_db(connection_name, config)
+      connection_name = connection_name.to_s
+      @@connection_classes[connection_name] ||= establish_connection_to_db(connection_name, config)
     end
 
     # Establish connection with a specified name
-    def self.establish_connection(db_name, should_exist = false)
-      abstract_class = generate_abstract_class(db_name, should_exist)
-      DbCharmer::ConnectionProxy.new(abstract_class, db_name)
+    def self.establish_connection(connection_name, should_exist = false)
+      abstract_class = generate_abstract_class(connection_name, should_exist)
+      DbCharmer::ConnectionProxy.new(abstract_class, connection_name)
+    end
+
+    # Establish connection with a specified name (not using AR database configs)
+    def self.establish_connection_to_db(connection_name, config)
+      abstract_class = generate_abstract_class_for_db(connection_name, config)
+      DbCharmer::ConnectionProxy.new(abstract_class, connection_name)
     end
 
     # Generate an abstract AR class with specified connection established
-    def self.generate_abstract_class(db_name, should_exist = false)
-      klass = abstract_connection_class_name(db_name)
+    def self.generate_abstract_class(connection_name, should_exist = false)
+      # Generate class
+      klass = generate_empty_abstract_ar_class(abstract_connection_class_name(connection_name))
+
+      # Establish connection
+      klass.establish_real_connection_if_exists(connection_name.to_sym, !!should_exist)
+
+      # Return the class
+      return klass
+    end
+
+    # Generate an abstract AR class with specified connection established (not using AR database configs)
+    def self.generate_abstract_class_for_db(connection_name, config)
       # Generate class
-      module_eval <<-EOF, __FILE__, __LINE__ + 1
-        class #{klass} < ActiveRecord::Base
-          self.abstract_class = true
-          establish_real_connection_if_exists(:#{db_name}, #{!!should_exist})
-        end
-      EOF
+      klass = generate_empty_abstract_ar_class(abstract_connection_class_name(connection_name))
+
+      # Establish connection
+      klass.establish_connection(config)
+
+      # Return the class
+      return klass
+    end
+
+    def self.generate_empty_abstract_ar_class(klass)
+      # Define class
+      module_eval "class #{klass} < ActiveRecord::Base; self.abstract_class = true; end"
+
       # Return class
       klass.constantize
     end
 
     # Generates unique names for our abstract AR classes
-    def self.abstract_connection_class_name(db_name)
-      "::AutoGeneratedAbstractConnectionClass#{db_name.to_s.camelize}"
+    def self.abstract_connection_class_name(connection_name)
+      "::AutoGeneratedAbstractConnectionClass#{connection_name.to_s.camelize}"
     end
   end
 end

data/lib/db_charmer/connection_proxy.rb

@@ -1,3 +1,4 @@
+# Simple proxy that sends all method calls to a real database connection
 module DbCharmer
   class ConnectionProxy < BlankSlate
     def initialize(abstract_class, db_name)

data/lib/db_charmer/connection_switch.rb

@@ -8,6 +8,12 @@ module DbCharmer
           return DbCharmer::ConnectionFactory.connect(conn, should_exist)
         end
 
+        if conn.kind_of?(Hash)
+          conn = conn.symbolize_keys
+          raise ArgumentError, "Missing required :name parameter" unless conn[:name]
+          return DbCharmer::ConnectionFactory.connect_to_db(conn[:name], conn)
+        end
+
         if conn.respond_to?(:db_charmer_connection_proxy)
           return conn.db_charmer_connection_proxy
         end

data/lib/db_charmer/db_magic.rb

@@ -9,15 +9,22 @@ module DbCharmer
         should_exist = opt[:should_exist] || DbCharmer.connections_should_exist?
 
         # Main connection management
-        db_magic_connection(opt[:connection], should_exist) if opt[:connection]
+        setup_connection_magic(opt[:connection], should_exist) if opt[:connection]
 
         # Set up slaves pool
         opt[:slaves] ||= []
         opt[:slaves] << opt[:slave] if opt[:slave]
-        db_magic_slaves(opt[:slaves], should_exist) if opt[:slaves].any?
+        setup_slaves_magic(opt[:slaves], should_exist) if opt[:slaves].any?
 
         # Setup inheritance magic
         setup_children_magic(opt)
+
+        # Setup sharding if needed
+        if opt[:sharded]
+          raise ArgumentError, "Can't use sharding on a model with slaves!" if opt[:slaves].any?
+          setup_sharding_magic(opt[:sharded])
+          setup_connection_magic(DbCharmer::StubConnection.new)
+        end
       end
 
     private
@@ -31,11 +38,17 @@ module DbCharmer
         end
       end
 
-      def db_magic_connection(conn, should_exist = false)
+      def setup_sharding_magic(config)
+        self.extend(DbCharmer::Sharding::ClassMethods)
+        name = config[:sharded_connection] or raise ArgumentError, "No :sharded_connection!"
+        self.sharded_connection = DbCharmer::Sharding.sharded_connection(name)
+      end
+
+      def setup_connection_magic(conn, should_exist = false)
         switch_connection_to(conn, should_exist)
       end
 
-      def db_magic_slaves(slaves, should_exist = false)
+      def setup_slaves_magic(slaves, should_exist = false)
         self.db_charmer_slaves = slaves.collect do |slave|
           coerce_to_connection_proxy(slave, should_exist)
         end
@@ -47,3 +60,4 @@ module DbCharmer
     end
   end
 end
+

data/lib/db_charmer/multi_db_migrations.rb

@@ -3,15 +3,23 @@ module DbCharmer
     @multi_db_names = nil
 
     def migrate_with_db_wrapper(direction)
-      @multi_db_names.each { |multi_db_name| on_db(multi_db_name) { migrate_without_db_wrapper(direction) } }
+      @multi_db_names.each do |multi_db_name|
+        on_db(multi_db_name) do
+          migrate_without_db_wrapper(direction)
+        end
+      end
     end
 
     def on_db(db_name)
-      announce "Switching connection to #{db_name}"
+      name = db_name.is_a?(Hash) ? db_name[:name] : db_name.inspect
+      announce "Switching connection to #{name}"
+      # Switch connection
       old_proxy = ActiveRecord::Base.db_charmer_connection_proxy
       ActiveRecord::Base.switch_connection_to(db_name, DbCharmer.migration_connections_should_exist?)
+      # Yield the block
       yield
     ensure
+      # Switch it back
       announce "Checking all database connections"
       ActiveRecord::Base.verify_active_connections!
       announce "Switching connection back to default"
@@ -19,12 +27,25 @@ module DbCharmer
     end
 
     def db_magic(opts = {})
-      opts[:connection] = opts[:connections] if opts[:connections]
-      raise ArgumentError, "No connection name - no magic!" unless opts[:connection]
-      @multi_db_names = [opts[:connection]].flatten
+      # Collect connections from all possible options
+      conns = [ opts[:connection], opts[:connections] ]
+      conns << shard_connections(opts[:sharded_connection]) if opts[:sharded_connection]
+
+      # Get a unique set of connections
+      conns = conns.flatten.compact.uniq
+      raise ArgumentError, "No connection name - no magic!" unless conns.any?
+
+      # Save connections
+      @multi_db_names = conns
       class << self
         alias_method_chain :migrate, :db_wrapper
       end
     end
+
+    # Return a list of connections to shards in a sharded connection
+    def shard_connections(conn_name)
+      conn = DbCharmer::Sharding.sharded_connection(conn_name)
+      conn.shard_connections
+    end
   end
 end

data/lib/db_charmer/multi_db_proxy.rb

@@ -1,5 +1,7 @@
 module DbCharmer
   module MultiDbProxy
+    # Simple proxy class that switches connections and then proxies all the calls
+    # This class is used to implement chained on_db calls
     class OnDbProxy < BlankSlate
       def initialize(proxy_target, slave)
         @proxy_target = proxy_target
@@ -10,8 +12,8 @@ module DbCharmer
 
       def method_missing(meth, *args, &block)
         # Switch connection and proxy the method call
-        @proxy_target.on_db(@slave) do |m|
-          res = m.__send__(meth, *args, &block)
+        @proxy_target.on_db(@slave) do |proxy_target|
+          res = proxy_target.__send__(meth, *args, &block)
 
           # If result is a scope/association, return a new proxy for it, otherwise return the result itself
           (res.proxy?) ? OnDbProxy.new(res, @slave) : res

data/lib/db_charmer/sharding.rb

@@ -0,0 +1,51 @@
+module DbCharmer
+  module Sharding
+    module ClassMethods
+      def self.extended(model)
+        model.cattr_accessor(:sharded_connection)
+      end
+
+      def shard_for(key, proxy_target = nil, &block)
+        raise ArgumentError, "No sharded connection configured!" unless sharded_connection
+        conn = sharded_connection.sharder.shard_for_key(key)
+        on_db(conn, proxy_target, &block)
+      end
+
+      # Run on default shard (if supported by the sharding method)
+      def on_default_shard(proxy_target = nil, &block)
+        raise ArgumentError, "No sharded connection configured!" unless sharded_connection
+
+        if sharded_connection.support_default_shard?
+          shard_for(:default, proxy_target, &block)
+        else
+          raise ArgumentError, "This model's sharding method does not support default shard"
+        end
+      end
+
+      # Enumerate shards
+      def on_each_shard(proxy_target = nil, &block)
+        raise ArgumentError, "No sharded connection configured!" unless sharded_connection
+
+        conns = sharded_connection.shard_connections
+        raise ArgumentError, "This model's sharding method does not support shards enumeration" unless conns
+
+        conns.each do |conn|
+          on_db(conn, proxy_target, &block)
+        end
+      end
+    end
+
+    #-------------------------------------------------------------
+    @@sharded_connections = {}
+
+    def self.register_connection(config)
+      name = config[:name] or raise ArgumentError, "No :name in connection!"
+      @@sharded_connections[name] = DbCharmer::Sharding::Connection.new(config)
+    end
+
+    def self.sharded_connection(name)
+      @@sharded_connections[name] or raise ArgumentError, "Invalid sharded connection name!"
+    end
+  end
+end
+

data/lib/db_charmer/sharding/connection.rb

@@ -0,0 +1,27 @@
+module DbCharmer
+  module Sharding
+    class Connection
+      attr_accessor :config, :sharder
+
+      def initialize(config)
+        @config = config
+        @sharder = self.instantiate_sharder
+      end
+
+      def instantiate_sharder
+        raise ArgumentError, "No :method passed!" unless config[:method]
+        sharder_class_name = "DbCharmer::Sharding::Method::#{config[:method].to_s.classify}"
+        sharder_class = sharder_class_name.constantize
+        sharder_class.new(config)
+      end
+
+      def shard_connections
+        sharder.respond_to?(:shard_connections) ? sharder.shard_connections : nil
+      end
+
+      def support_default_shard?
+        sharder.respond_to?(:support_default_shard?) && sharder.support_default_shard?
+      end
+    end
+  end
+end

data/lib/db_charmer/sharding/method/db_block_map.rb

@@ -0,0 +1,184 @@
+# This is a more sophisticated sharding method based on a database-backed
+# blocks map that holds block-shard associations. It automatically
+# creates new blocks for new keys and assigns them to shards.
+#
+module DbCharmer
+  module Sharding
+    module Method
+      class DbBlockMap
+        # Sharder name
+        attr_accessor :name
+
+        # Mapping db connection
+        attr_accessor :connection, :connection_name
+
+        # Mapping table name
+        attr_accessor :map_table
+
+        # Shards table name
+        attr_accessor :shards_table
+
+        # Sharding keys block size
+        attr_accessor :block_size
+
+        def initialize(config)
+          @name = config[:name] or raise(ArgumentError, "Missing required :name parameter!")
+          @connection = DbCharmer::ConnectionFactory.connect(config[:connection])
+          @block_size = (config[:block_size] || 10000).to_i
+
+          @map_table = config[:map_table] or raise(ArgumentError, "Missing required :map_table parameter!")
+          @shards_table = config[:shards_table] or raise(ArgumentError, "Missing required :shards_table parameter!")
+
+          # Local caches
+          @shard_info_cache = {}
+          @blocks_cache = {}
+        end
+
+        def shard_for_key(key)
+          block = block_for_key(key)
+
+          # Auto-allocate new blocks
+          block ||= allocate_new_block_for_key(key)
+          raise ArgumentError, "Invalid key value, no shards found for this key and could not create a new block!" unless block
+
+          # Bail if no shard found
+          shard_id = block['shard_id'].to_i
+          shard_info = shard_info_by_id(shard_id)
+          raise ArgumentError, "Invalid shard_id: #{shard_id}" unless shard_info
+
+          # Get config
+          shard_connection_config(shard_info)
+        end
+
+        class ShardInfo < ActiveRecord::Base
+          validates_presence_of :db_host
+          validates_presence_of :db_port
+          validates_presence_of :db_user
+          validates_presence_of :db_pass
+          validates_presence_of :db_name
+        end
+
+        # Returns a block for a key
+        def block_for_key(key, cache = true)
+          # Cleanup the cache if asked to
+          key_range = [ block_start_for_key(key), block_end_for_key(key) ]
+          block_cache_key = "%d-%d" % key_range
+          @blocks_cache[block_cache_key] = nil unless cache
+
+          # Fetch cached value or load from db
+          @blocks_cache[block_cache_key] ||= begin
+            sql = "SELECT * FROM #{map_table} WHERE start_id = #{key_range.first} AND end_id = #{key_range.last} LIMIT 1"
+            connection.select_one(sql, 'Find a shard block')
+          end
+        end
+
+        # Load shard info
+        def shard_info_by_id(shard_id, cache = true)
+          # Cleanup the cache if asked to
+          @shard_info_cache[shard_id] = nil unless cache
+
+          # Either load from cache or from db
+          @shard_info_cache[shard_id] ||= begin
+            prepare_shard_model
+            ShardInfo.find_by_id(shard_id)
+          end
+        end
+
+        def allocate_new_block_for_key(key)
+          # Can't find any shards to use for blocks allocation!
+          return nil unless shard = least_loaded_shard
+
+          # Figure out block limits
+          start_id = block_start_for_key(key)
+          end_id = block_end_for_key(key)
+
+          # Try to insert a new mapping (ignore duplicate key errors)
+          sql = <<-SQL
+            INSERT IGNORE INTO #{map_table}
+                           SET start_id = #{start_id},
+                               end_id = #{end_id},
+                               shard_id = #{shard.id},
+                               block_size = #{block_size},
+                               created_at = NOW(),
+                               updated_at = NOW()
+          SQL
+          connection.execute(sql, "Allocate new block")
+
+          # Retry block search after creation
+          block_for_key(key)
+        end
+
+        def least_loaded_shard
+          prepare_shard_model
+
+          # Select shard
+          shard = ShardInfo.all(:conditions => { :enabled => true, :open => true }, :order => 'blocks_count ASC', :limit => 1).first
+          raise "Can't find any shards to use for blocks allocation!" unless shard
+          return shard
+        end
+
+        def block_start_for_key(key)
+          block_size.to_i * (key.to_i / block_size.to_i)
+        end
+
+        def block_end_for_key(key)
+          block_size.to_i + block_start_for_key(key)
+        end
+
+        # Create configuration (use mapping connection as a template)
+        def shard_connection_config(shard)
+          # Format connection name
+          shard_name = "db_charmer_db_block_map_#{name}_shard_%05d" % shard.id
+
+          # Here we get the mapping connection's configuration
+          # They do not expose configs so we hack in and get the instance var
+          # FIXME: Find a better way, maybe move config method to our ar extenstions
+          connection.instance_variable_get(:@config).clone.merge(
+            # Name for the connection factory
+            :name => shard_name,
+            # Connection params
+            :host => shard.db_host,
+            :port => shard.db_port,
+            :username => shard.db_user,
+            :password => shard.db_pass,
+            :database => shard.db_name
+          )
+        end
+
+        def create_shard(params)
+          params = params.symbolize_keys
+          [ :db_host, :db_port, :db_user, :db_pass, :db_name ].each do |arg|
+            raise ArgumentError, "Missing required parameter: #{arg}" unless params[arg]
+          end
+
+          # Prepare model
+          prepare_shard_model
+
+          # Create the record
+          ShardInfo.create! do |shard|
+            shard.db_host = params[:db_host]
+            shard.db_port = params[:db_port]
+            shard.db_user = params[:db_user]
+            shard.db_pass = params[:db_pass]
+            shard.db_name = params[:db_name]
+          end
+        end
+
+        def shard_connections
+          # Find all shards
+          prepare_shard_model
+          shards = ShardInfo.all(:conditions => { :enabled => true })
+          # Map them to connections
+          shards.map { |shard| shard_connection_config(shard) }
+        end
+
+        # Prepare model for working with our shards table
+        def prepare_shard_model
+          ShardInfo.set_table_name(shards_table)
+          ShardInfo.switch_connection_to(connection)
+        end
+
+      end
+    end
+  end
+end

data/lib/db_charmer/sharding/method/hash_map.rb

@@ -0,0 +1,23 @@
+module DbCharmer
+  module Sharding
+    module Method
+      class HashMap
+        attr_accessor :map
+
+        def initialize(config)
+          @map = config[:map].clone or raise ArgumentError, "No :map defined!"
+        end
+
+        def shard_for_key(key)
+          res = map[key] || map[:default]
+          raise ArgumentError, "Invalid key value, no shards found for this key!" unless res
+          return res
+        end
+
+        def support_default_shard?
+          map.has_key?(:default)
+        end
+      end
+    end
+  end
+end

data/lib/db_charmer/sharding/method/range.rb

@@ -0,0 +1,29 @@
+module DbCharmer
+  module Sharding
+    module Method
+      class Range
+        attr_accessor :ranges
+
+        def initialize(config)
+          @ranges = config[:ranges] ? config[:ranges].clone : raise(ArgumentError, "No :ranges defined!")
+        end
+
+        def shard_for_key(key)
+          return ranges[:default] if key == :default
+
+          ranges.each do |range, shard|
+            next if range == :default
+            return shard if range.member?(key.to_i)
+          end
+
+          return ranges[:default] if ranges[:default]
+          raise ArgumentError, "Invalid key value, no shards found for this key!"
+        end
+
+        def support_default_shard?
+          ranges.has_key?(:default)
+        end
+      end
+    end
+  end
+end

data/lib/db_charmer/stub_connection.rb

@@ -0,0 +1,9 @@
+module DbCharmer
+  class StubConnection < ActiveRecord::ConnectionAdapters::AbstractAdapter
+    def initialize; end
+
+    def method_missing(*arg)
+      raise ActiveRecord::ConnectionNotEstablished, "You have to switch connection on your model before using it"
+    end
+  end
+end

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 1
-  - 5
-  - 5
-  version: 1.5.5
+  - 6
+  - 0
+  version: 1.6.0
 platform: ruby
 authors: 
 - Alexey Kovyrin
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-03-15 00:00:00 -04:00
+date: 2010-03-31 00:00:00 -04:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -74,6 +74,12 @@ files:
 - lib/db_charmer/multi_db_migrations.rb
 - lib/db_charmer/multi_db_proxy.rb
 - lib/db_charmer/scope_proxy.rb
+- lib/db_charmer/sharding.rb
+- lib/db_charmer/sharding/connection.rb
+- lib/db_charmer/sharding/method/db_block_map.rb
+- lib/db_charmer/sharding/method/hash_map.rb
+- lib/db_charmer/sharding/method/range.rb
+- lib/db_charmer/stub_connection.rb
 - lib/tasks/databases.rake
 has_rdoc: true
 homepage: http://github.com/kovyrin/db-charmer