mixed_gauge 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b52a83a9e2dcfa88c05591683f5f3f8ca6cdc25e
-  data.tar.gz: 2ebdd85105cf34b3d4cf09485293953e85894be0
+  metadata.gz: db4722c025b0f789f356bc263a7b5abd0e4824e9
+  data.tar.gz: 2b09c8ec6892123fdde93ccd227bf4e21cf65500
 SHA512:
-  metadata.gz: 85c09e39e2a6963dda8afab08b8bad16e7ec4be4ac506f8b208295d84842f9d4b92344e099072cfdf3443af2ce7a7e51d7f629d0940aadeb42ff691541d43948
-  data.tar.gz: 8c014307fa8eae21d653ebed9264f01ba8408db1364a158fcec682fdab9a618f81a9aeff5dbc22e25404228992ed191485e96e26510a33ad9c35d5081a9cd9b9
+  metadata.gz: 6f6b47b28eff87fa760c363acc7943200d2ff9fc12fce05aaf245acaa5a8890053b8cc8a698e4a9153740077bcbd8fcb01c8001f8d94b6e2fda80858c3461eef
+  data.tar.gz: a9c5e57c778f0be58568e237edfcea43db2e65b1452cbb5e4bcecb54e64bb61b59feb4052b8a5c09fef00f432213cb7cb4814c5f280ae2f75e7a471cb3e53e78

data/.travis.yml

@@ -1,6 +1,5 @@
 language: ruby
 sudo: false
-script: "bundle exec rake && bundle exec ruby spec/performance_test.rb"
 branches:
   only:
     - master

data/CHANGELOG.md

@@ -1,4 +1,7 @@
 # CHANGELOG for mixed_gauge
+## 1.1.0
+- Replication support. #5
+
 ## 1.0.0
 - Change cluster slots definition interface: does not specify entire slots,
     but specify only slots size.

data/README.md

@@ -1,11 +1,12 @@
-# MixedGauge
+# mixed_gauge
 [![Build Status](https://travis-ci.org/taiki45/mixed_gauge.svg?branch=master)](https://travis-ci.org/taiki45/mixed_gauge) [![Coverage Status](https://coveralls.io/repos/taiki45/mixed_gauge/badge.svg?branch=master)](https://coveralls.io/r/taiki45/mixed_gauge?branch=master) [![Code Climate](https://codeclimate.com/github/taiki45/mixed_gauge/badges/gpa.svg)](https://codeclimate.com/github/taiki45/mixed_gauge) [![Gem Version](https://badge.fury.io/rb/mixed_gauge.svg)](http://badge.fury.io/rb/mixed_gauge)
 
 A simple and robust ActiveRecord extension for database sharding.
 mixed_gauge offers shards management with hash slots and re-sharding support.
 It enable you to execute efficient queries to single node with KVS-like interface.
-And you can even execute limited RDB queries to all nodes with ActiveRecord interface,
-'course in-parallel.
+And you can even execute limited RDB queries to all nodes with ActiveRecord interface in-parallel.
+
+mixed_gauge is already used in production. [(blog post in Japanese)](http://techlife.cookpad.com/entry/2015/06/22/134108)
 
 ## Goal and concept
 - Simple
@@ -14,7 +15,7 @@ And you can even execute limited RDB queries to all nodes with ActiveRecord inte
 
 Database sharding tend to be over-complexed. There are cases which need these complex database sharding but in some cases database sharding can be more simple. The large data set which is enoght big to partition should be designed to be distributed, or should be re-design if it wasn't. Design to be distributed uses key based relation or reverse indexes to fits its limitation. In that case, the data set is almost design to be distributed, mixed_gauge strongly encourages your database sharding by its simplicity.
 
-We, offer 24/7 services, must keep our services running. mixed_gauge supports online migrations: adding new nodes to cluster or removing some existing nodes from cluster. It comes with "key distibution model with hash slots" and  database replication and multi-master replication. In sharding we need re-sharding, move data from node to anoter node in cluster, when adding or removing new nodes from cluster. But by setting some rule to node management and using replication, we can finish moving data before adding or removing nodes. The detail operations are specified later chapter of this document.
+We, offer 24/7 services, must keep our services running. mixed_gauge supports online migrations: adding new nodes to cluster or removing some existing nodes from cluster. It comes with "key distibution model with hash slots" and  database replication and multi-master replication. In sharding we need re-sharding, move data from node to another node in cluster, when adding or removing new nodes from cluster. But by setting some rule to node management and using replication, we can finish moving data before adding or removing nodes. The detail operations are specified later chapter of this document.
 
 All operaions should be rollback-able in case of any failures. mixed_gauge's node management can rollback adding and removing nodes operation. The detail operations are specified later chapter of this document.
 
@@ -240,6 +241,107 @@ access_token = AccessToken.put!
 access_token.token #=> a generated token
 ```
 
+## Sharding with Replication
+mixed_gauge also supports replication.
+
+In case you have 2 shards in cluster and each shard have read replica.
+
+- db-user-101 --replicated--> db-user-102
+- db-user-201 --replicated--> db-user-202
+
+Your database connection configuration might be like this:
+
+```yaml
+# database.yml
+production_user_001:
+  adapter: mysql2
+  username: user_writable
+  host: db-user-101
+production_user_002:
+  adapter: mysql2
+  username: user_writable
+  host: db-user-201
+production_user_readonly_001:
+  adapter: mysql2
+  username: user_readonly
+  host: db-user-102
+production_user_readonly_002:
+  adapter: mysql2
+  username: user_writable
+  host: db-user-202
+```
+
+Your initializer for mixed_gauge might be like this:
+
+```ruby
+MixedGauge.configure do |config|
+  config.define_cluster(:user) do |cluster|
+    cluster.define_slot_size(1048576)
+    cluster.register(0..524287, :production_user_001)
+    cluster.register(524288..1048575, :production_user_002)
+  end
+
+  config.define_cluster(:user_readonly) do |cluster|
+    cluster.define_slot_size(1048576)
+    cluster.register(0..524287, :production_user_readonly_001)
+    cluster.register(524288..1048575, :production_user_readonly_002)
+  end
+end
+```
+
+You can split read/write by defining AR model class for each connection:
+
+```ruby
+class User < ActiveRecord::Base
+  include MixedGauge::Model
+  use_cluster :user
+  def_distkey :email
+end
+
+class UserReadonly < ActiveRecord::Base
+  self.table_name = 'users'
+
+  include MixedGauge::Model
+  use_cluster :user_readonly
+  def_distkey :email
+end
+
+User.put!(name: 'Alice', email: 'alice@example.com')
+UserReadonly.get('alice@example.com')
+```
+
+If you want to switch specific shard to another shard in another cluster, define mapping between each model:
+
+```ruby
+class User < ActiveRecord::Base
+  include MixedGauge::Model
+  use_cluster :user
+  def_distkey :email
+
+  replicates_with slave: :UserReadonly
+end
+
+class UserReadonly < ActiveRecord::Base
+  self.table_name = 'users'
+
+  include MixedGauge::Model
+  use_cluster :user_readonly
+  def_distkey :email
+
+  replicates_with master: :User
+end
+```
+
+You can switch to another model which have connection to the shard by calling `.switch`:
+
+```ruby
+UserReadonly.all_shards do |readonly|
+  target_ids = readonly.where(age: 0).pluck(:id)
+  readonly.switch(:master) do |writable|
+    writable.where(id: target_ids).delete_all
+  end
+end
+```
 
 ## Advanced configuration
 ### Hash fucntion

data/Rakefile

@@ -6,6 +6,11 @@ require 'bundler/gem_tasks'
 begin
   require 'rspec/core/rake_task'
   RSpec::Core::RakeTask.new(:spec)
-  task :default => :spec
 rescue LoadError
 end
+
+task :performance_test do
+  ruby 'spec/performance_test.rb'
+end
+
+task :default => [:spec, :performance_test]

data/lib/mixed_gauge.rb

@@ -7,6 +7,7 @@ require 'mixed_gauge/cluster_config'
 require 'mixed_gauge/config'
 require 'mixed_gauge/routing'
 require 'mixed_gauge/shard_repository'
+require 'mixed_gauge/replication_mapping'
 require 'mixed_gauge/all_shards_in_parallel'
 require 'mixed_gauge/model'
 

data/lib/mixed_gauge/cluster_config.rb

@@ -1,7 +1,7 @@
 module MixedGauge
   # Mapping of slot -> connection_name.
   class ClusterConfig
-    attr_reader :name
+    attr_reader :name, :connection_registry
 
     # @param [Symbol] name
     def initialize(name)

data/lib/mixed_gauge/model.rb

@@ -6,6 +6,7 @@ module MixedGauge
   #     include MixedGauge::Model
   #     use_cluster :user
   #     def_distkey :email
+  #     replicates_with slave: :UserReadonly, backgroud: :UserBackground
   #   end
   #
   #   User.put!(email: 'alice@example.com', name: 'alice')
@@ -22,11 +23,12 @@ module MixedGauge
       class_attribute :cluster_routing, instance_writer: false
       class_attribute :shard_repository, instance_writer: false
       class_attribute :distkey, instance_writer: false
+      class_attribute :replication_mapping, instance_writer: false
     end
 
     module ClassMethods
       # The cluster config must be defined before `use_cluster`.
-      # @param [Symbol] A cluster name which is set by MixedGauge.configure
+      # @param [Symbol] name A cluster name which is set by MixedGauge.configure
       def use_cluster(name)
         config = MixedGauge.config.fetch_cluster_config(name)
         self.cluster_routing = MixedGauge::Routing.new(config)
@@ -41,6 +43,12 @@ module MixedGauge
         self.distkey = column.to_sym
       end
 
+      # @param [Hash{Symbol => Symbol}] mapping A pairs of role name and
+      #   AR model class name.
+      def replicates_with(mapping)
+        self.replication_mapping = MixedGauge::ReplicationMapping.new(mapping)
+      end
+
       # Create new record with given attributes in proper shard for given key.
       # When distkey value is empty, raises MixedGauge::MissingDistkeyAttribute
       # error.
@@ -118,6 +126,21 @@ module MixedGauge
       end
       alias_method :parallel, :all_shards_in_parallel
 
+      # See example definitions in `spec/models.rb`.
+      # @param [Symbol] A role name of target cluster.
+      # @return [Class, Object] if block given then yielded result else
+      #   target shard model.
+      # @example
+      #   UserReadonly.all_shards.each do |m|
+      #     target_ids = m.where(age: 1).pluck(:id)
+      #     m.switch(:master) do |master|
+      #       master.where(id: target_ids).delete_all
+      #     end
+      #   end
+      def switch(role_name, &block)
+        replication_mapping.switch(self, role_name, &block)
+      end
+
       # Define utility methods which uses all shards or specific shard.
       # These methods can be called from included model class.
       # @example

data/lib/mixed_gauge/replication_mapping.rb

@@ -0,0 +1,35 @@
+module MixedGauge
+  class ReplicationMapping
+    def initialize(mapping)
+      @mapping = mapping
+      @lock = Mutex.new
+    end
+
+    # @param [Class] A shard model having connection to specific shard
+    # @param [Symbol] A role name of target cluster.
+    # @return [Class, Object] if block given then yielded result else
+    #   target shard model.
+    def switch(from, role_name, &block)
+      @lock.synchronize { constantize! unless constantized? }
+
+      model = @mapping.fetch(role_name)
+      target_shard_model = model.shard_repository.fetch_by_slots(from.assigned_slots)
+
+      if block_given?
+        target_shard_model.connection_pool.with_connection { yield target_shard_model }
+      else
+        target_shard_model
+      end
+    end
+
+    private
+
+    def constantize!
+      @mapping = Hash[@mapping.map {|k, name| [k, name.to_s.constantize] }]
+    end
+
+    def constantized?
+      @mapping.values.first.is_a? Class
+    end
+  end
+end

data/lib/mixed_gauge/shard_repository.rb

@@ -7,8 +7,8 @@ module MixedGauge
     def initialize(cluster_config, base_class)
       @base_class = base_class
 
-      shards = cluster_config.connections.map do |connection_name|
-        [connection_name, generate_model_for_shard(connection_name)]
+      shards = cluster_config.connection_registry.map do |slot_range, connection_name|
+        [connection_name, generate_model_for_shard(connection_name, slot_range)]
       end
       @shards = Hash[shards]
     end
@@ -19,6 +19,12 @@ module MixedGauge
       @shards.fetch(connection_name)
     end
 
+    # @param [Range] slots
+    # @return [Class, nil] A AR model class.
+    def fetch_by_slots(assigned_slots)
+      @shards.find {|_, model| model.assigned_slots == assigned_slots }[1]
+    end
+
     # @return [Array<Class>]
     def all
       @shards.values
@@ -27,13 +33,19 @@ module MixedGauge
     private
 
     # @param [Symbol] connection_name
+    # @param [Range] slot_range
     # @return [Class] A sub class of given AR model.
     #   A sub class has connection setting for specific shard.
-    def generate_model_for_shard(connection_name)
+    def generate_model_for_shard(connection_name, slot_range)
       base_class_name = @base_class.name
       class_name = generate_class_name(connection_name)
 
       model = Class.new(base_class) do
+        class << self
+          attr_reader :assigned_slots
+        end
+        @assigned_slots = slot_range
+
         self.table_name = base_class.table_name
         module_eval <<-RUBY, __FILE__, __LINE__ + 1
           def self.name

data/lib/mixed_gauge/version.rb

@@ -1,3 +1,3 @@
 module MixedGauge
-  VERSION = '1.0.0'
+  VERSION = '1.1.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mixed_gauge
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Taiki Ono
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-06-24 00:00:00.000000000 Z
+date: 2016-05-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -198,6 +198,7 @@ files:
 - lib/mixed_gauge/errors.rb
 - lib/mixed_gauge/model.rb
 - lib/mixed_gauge/railtie.rb
+- lib/mixed_gauge/replication_mapping.rb
 - lib/mixed_gauge/routing.rb
 - lib/mixed_gauge/shard_repository.rb
 - lib/mixed_gauge/version.rb
@@ -223,7 +224,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.5
+rubygems_version: 2.6.3
 signing_key: 
 specification_version: 4
 summary: A simple and robust ActiveRecord extension for database sharding.