xbar 0.0.1 → 0.4.0

data/Appraisals

@@ -0,0 +1,25 @@
+# appraise "rails2" do
+#  gem 'activerecord', '~> 2.3'
+#  gem 'actionpack', '~> 2.3'
+# end
+
+appraise "rails3" do
+  gem 'activerecord', '~> 3.0.11'
+  gem 'actionpack', '~> 3.0.11'
+end
+
+appraise "rails31" do
+  gem 'activerecord', '~> 3.1.1'
+  gem 'actionpack', '~> 3.1.1'
+end
+
+
+appraise "rails32" do
+  gem 'rails', '3.2.1'
+end
+
+appraise "rails4" do
+  gem 'rails',     :path => '/opt/rails/rails-edge'
+  gem 'journey',   :git => 'https://github.com/rails/journey.git'
+  gem 'arel',      :git => 'https://github.com/rails/arel.git'
+end

data/README.mkdn

@@ -0,0 +1,215 @@
+# XBar  - MongoDB Style Sharding for ActiveRecord
+
+## Supported versions
+*Caution* -- only ActiveRecord 3.2 is currently supported.  This 
+will soon be improved.
+
+## General Design
+
+The XBar project is derived from *Octopus*.  Octopus showed that the
+implementation technique of using a proxy for the
+`ActiveRecord::ConnectionAdapters::AbstractAdapter` object instances
+is possible.  This proxy implements a sort of *late binding*,
+returning a real abstract adapter object that depends on the current
+state of the proxy, especially on the value of the `current_shard`.
+Many of the tricky pieces of code, especially those for managing
+associations, come from Octopus.
+
+## Changes from Octopus
+
+The Octopus implementation has been modified to support efficient
+dynamic reconfiguration.  The configuration is given in a JSON
+document, rather than YAML.  This makes it convenient to update the
+application over the network using the standard JSON over HTTP
+technique.  The JSON document supports multiple application
+environments. These align with Rails environments when Rails is
+present. Unlike Octopus, this document format does not allow multiple
+'Octopus' evironments.  Instead, there is an `xbar_env` which
+essentially specifies a *different* JSON file to use.
+
+Another change is that shards that are themselves a collection of
+mirrors is supported.  Mirroring does not take place among shards, it
+takes place among the mirrors that constitute a shard.  Replication
+among the mirrors is expected to be handling by some database specific
+technique, such as native *MySQL* replication.
+
+The Octopus `Proxy` class has been split into two classes `Proxy` and
+`Shard`, and into a module `Mapper`.  `Mapper` contains the in-memory
+representation of the current configuration.  Each `Proxy` instance no
+longer has its own copy of the configuration.  There is exactly one
+`Mapper` module for the entire application, necessarily true, since
+its a *module*.  As before there is one `Proxy` instance per thread,
+which allows per-thread state to be stored in the instance.  A `Proxy`
+instance references a collection of `Shard` instances.  Each `Shard`
+instance manages a set of replicas for that shard.
+
+The concept of a *group* as used in Octopus is mostly gone.  Vestiges
+of it exist in the migration code.  Thus a migration can still specify
+that it should take place on multiple shards.  In the general case, a
+group concept would fork writes to send them to multiple shards.  This 
+is only practical if the writes are performed in parallel in separate
+threads.  The current `Proxy` would become a group manager and we'd
+have a hierarchy something like this:
+
+    Proxy (set of Groups) -> Group (set of Shards) -> Shard (set of replicas)
+	
+This complexity is too great for the first version of `XBar`.
+
+## Concepts
+
+### Mapper
+
+The mapper is a module that maintains the state of the database servers, shards,
+replication, and databases.  It is configured via a JSON document.  The JSON
+document can be in the local file system, or it can be delivered over HTTP.  A
+new JSON document can be installed at any time.  The mapper does not maintain
+any per-thread state. All the proxies get their mapping information directly
+from the mapper.  Some information is cached in each proxy, optimized for use in
+the proxy.  Thus when the JSON document is changed, the mapper will notify each
+proxy to rebuild its state.
+
+### Proxy
+
+An instance of the `Proxy` object exists per thread.  Thus thread-local state
+can be kept directly in the proxy, and the proxy can refer to global state in
+the mapper (and in the `XBar` module itself).  Each proxy registers with the
+mapper when it is created so that the mapper can notify the proxy of changes to
+the global configuration.
+
+A proxy is only responsible for managing shards, that is, except for some
+initialization code, it has no knowledge of replicas within a shard.  It has a
+list, the `shard_list`, that maps shard names to `Shard` objects.  For each SQL
+statement, block of statements, or transaction, the proxy choose a shard and
+deletegates the operation to the shard.  The shard, in turn, chooses the particular
+replica to use.
+
+### Shard
+
+The term *shard* is used as it is used in MongoDB.  A shard acts like an
+independent database.  In our case, the replication within the shard is handled
+by some external means such as native MySQL replication.  Nevertheless, XBar
+knows about the replicas within the shard, and it knows which replica is the
+replication master. This allows XBar to direct transactions to the master shard,
+and reads to the (eventually-consistent) replicas within the shard.
+
+The shard is a per-thread object, referenced only from a proxy.  Thus the
+overall structure is tree-like.  One mapper references a collection of proxies
+(one per thread), and each proxy references a collection of shards.  Each shard
+object references a list of `ActiveRecord::ConnectionAdapters::ConnectionPool`
+instances that the shard uses to select connections to replicas within the
+shard.
+
+### Environments
+
+XBar has three concepts all called *environments*.  First, there is the
+*rails_env* that is inherited from Rails when this gem is included in a Rails
+application.  Second, the *xbar_env* is the name of the configuration file that
+is currently in effect.  In the case where the configuration was loaded via a
+JSON document over HTTP, a name for the *xbar_env* is generated in some other
+way (TBD).  Finally, there is the *app_env*, that functions much like
+*rails_env* when Rails is not present.  When Rails is present, *app_env* is read
+only and always has the same value as *rails_env*.  When Rails is not present,
+*app_env* can be selected by the user.  In either case, its value should be name
+of an environment stanza in the current configuration file.
+
+## Getting Started
+
+The *examples* directory at the top level of the source distribution
+has some stand-alone examples that show how to set up and use `XBar`.
+
+## Using XBar
+
+XBar adds a method to each `ActiveRecord` Class and object: the using
+method is used to select the shard like this:
+
+```ruby
+    User.where(:name => "Thiago").limit(3).using(:slave_one)
+```
+
+XBar also supports queries within a block. When you pass a block to
+the using method, all queries inside the block will be sent to the
+specified shard.
+
+```ruby
+    XBar.using(:slave_two) do
+      User.create(:name => "Mike")
+    end
+```
+
+Each model instance knows which shard it came from so this will work
+automatically:
+
+```ruby
+    # This will find the user in the shard1
+    @user = User.using(:shard1).find_by_name("Joao")
+
+    # This will find the user in the master database
+    @user2 = User.find_by_name("Jose")
+
+    #Sets the name
+    @user.name = "Mike"
+
+    # Save the user in the correct shard, shard1.
+    @user.save
+```
+
+### Migrations
+
+In migrations, you also have access to the using method. The syntax is
+basically the same. This migration will run in the brazil and canada
+shards.
+
+```ruby
+    class CreateUsersOnBothShards < ActiveRecord::Migration
+      using(:brazil, :canada)
+
+      def self.up
+        User.create!(:name => "Both")
+      end
+
+      def self.down
+        User.delete_all
+      end
+    end
+```
+
+### Rails Controllers
+
+If you want to send a specified action, or all actions from a
+controller, to a specific shard, use this syntax:
+
+```ruby
+    class ApplicationController < ActionController::Base
+      around_filter :select_shard
+
+      def select_shard(&block)
+        XBar.using(:brazil, &block)
+      end
+    end
+```
+
+## Exception with Idle Connections
+
+Sometimes, when a connection isn't used for much time,
+this will makes ActiveRecord raising an exception. if you have this
+kind of applications, please, add the following line to your
+configuration:
+
+```ruby
+    verify_connection: true
+```
+
+This will tell XBar to verify the connection before sending the query.
+
+## Mixing XBar with the Rails Multiple Database Mechanism
+
+If you want to set a custom connection to a specific model, use this
+syntax:
+
+```ruby
+    #This class sets its own connection
+    # establish_connection will not work, use xbar_establish_connection instead.
+    class CustomConnection < ActiveRecord::Base
+      xbar_establish_connection(:adapter => "mysql", :database => "xbar_shard2")
+    end
+```

data/Rakefile

@@ -1 +1,337 @@
-require "bundler/gem_tasks"
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+# require 'metric_fu'
+require 'appraisal'
+
+task :default => :spec
+
+# MetricFu::Mapperuration.run do |config|
+#   config.metrics = [:churn,:flay, :flog, :reek, :roodi, :saikuro]
+#   config.graphs  = [:flog, :flay, :reek, :roodi]
+#   config.flay    = { :dirs_to_flay => ['spec', 'lib']  }
+#   config.flog    = { :dirs_to_flog => ['spec', 'lib']  }
+#   config.reek    = { :dirs_to_reek => ['spec', 'lib']  }
+#   config.roodi   = { :dirs_to_roodi => ['spec', 'lib'] }
+#   config.churn   = { :start_date => "1 year ago", :minimum_churn_count => 10 }
+# end
+
+RSpec::Core::RakeTask.new(:spec) do |spec|
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+end
+
+namespace :test do
+
+  desc 'Create usage_statistics table'
+  task :add_stats do
+    require 'active_record'
+    require "xbar"
+    XBar.directory = File.expand_path("../spec", __FILE__)
+    XBar::Mapper.reset(xbar_env: 'default', app_env: 'test')
+    
+    class CreateUsageStatistics < ActiveRecord::Migration
+      using :master
+      def up
+        create_table(:usage_statistics) do |t|
+          t.string :shard_name
+          t.string :method
+          t.string :adapter
+          t.string :username
+          t.string :thread_id
+          t.integer :port
+          t.string :host
+          t.string :database_name
+        end
+      end
+    end 
+    CreateUsageStatistics.migrate(:up)
+  end
+
+  desc 'Drop usage_statistics table'
+  task :drop_stats do
+    require 'active_record'
+    require "xbar"
+    XBar.directory = File.expand_path("../spec", __FILE__)
+    XBar::Mapper.reset(xbar_env: 'default', app_env: 'test')
+    
+    class CreateUsageStatistics < ActiveRecord::Migration
+      using :master
+      def down
+        drop_table(:usage_statistics)
+      end
+    end    
+    CreateUsageStatistics.migrate(:down)
+  end
+
+  desc 'Drop the databases for tests'
+  task :drop do
+    mysql_user = ENV['MYSQL_USER'] || "root"
+    postgres_user = ENV['POSTGRES_USER'] || "postgres"
+
+    require 'active_record'
+    require "xbar"
+    XBar.directory = File.expand_path("../spec", __FILE__)
+    XBar::Mapper.reset(xbar_env: 'default', app_env: 'test')
+    
+    %w(mysql_m london_s canada_1 brazil_1 china_1 china_2).each do |key|
+      config = XBar::Mapper.connections[key].spec.config
+      opts = "-h#{config[:host]} -P#{config[:port]} -u#{config[:username]}"
+      opts += " -p#{config[:password]}" if config[:password] &&
+        config[:password] != ""
+        
+      db = config[:database]
+      cmd = "mysqladmin #{opts} -f drop #{db}"
+      %x( #{cmd} )
+    end
+    
+    %x{ mysqladmin -uroot -f drop rogue }
+    
+    %w(moscow_s russia_1 russia_2 russia_3).each do |key|
+      config = XBar::Mapper.connections[key].spec.config
+      %x( dropdb -U #{postgres_user} #{config[:database]} )
+    end
+      
+    %x( rm -f /tmp/paris.sqlite3 )
+
+  end
+
+  desc 'Create databases for the tests'
+  task :create do
+    mysql_user = ENV['MYSQL_USER'] || "root"
+    postgres_user = ENV['POSTGRES_USER'] || "postgres"
+    
+    require 'active_record'
+    require "xbar"
+    XBar.directory = File.expand_path("../spec", __FILE__)
+    XBar::Mapper.reset(xbar_env: 'default', app_env: 'test')
+    
+    %w(brazil_1 canada_1 mysql_m london_s china_1 china_2).each do |key|
+      config = XBar::Mapper.connections[key].spec.config
+      opts = "-h#{config[:host]} -P#{config[:port]} -u#{config[:username]}"
+      opts += " -p#{config[:password]}" if config[:password ] &&
+        config[:password] != ""
+        
+      db = config[:database]
+      sql = "create DATABASE #{db} DEFAULT CHARACTER SET utf8 " +
+        "DEFAULT COLLATE utf8_unicode_ci"
+      %x( echo #{sql} | mysql #{opts} )
+    end
+    
+    %x{ mysqladmin -uroot create rogue }
+    
+    %w(moscow_s russia_1 russia_2 russia_3).each do |key|
+      config = XBar::Mapper.connections[key].spec.config
+      %x( createdb -E UTF8 -U #{postgres_user} -T template0 #{config[:database]} )
+    end
+  end
+  
+  desc 'Build test tables'
+  task :tables do
+    
+    require 'active_record'
+    require "xbar"
+    XBar.directory = File.expand_path("../spec", __FILE__)
+    XBar::Mapper.reset(xbar_env: 'default', app_env: 'test')
+    
+    class BlankModel < ActiveRecord::Base; end;
+
+    %w(master london paris moscow canada brazil russia_east
+      russia_west russia_central china_east china_west).each do |shard|
+    
+      BlankModel.using(shard).connection.
+        initialize_schema_migrations_table
+
+      ## Find the best way to build tables on the master of each shard. ##
+      ## We should use migrations instead of this error-prone way.      ##
+
+      BlankModel.using(shard).connection.create_table(:users) do |u|
+        u.string :name
+        u.integer :number
+        u.boolean :admin
+      end
+
+      BlankModel.using(shard).connection.create_table(:clients) do |u|
+        u.string :country
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:cats) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:items) do |u|
+        u.string :name
+        u.integer :client_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:computers) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:keyboards) do |u|
+        u.string :name
+        u.integer :computer_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:roles) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:permissions) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:permissions_roles, :id => false) do |u|
+        u.integer :role_id
+        u.integer :permission_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:assignments) do |u|
+        u.integer :programmer_id
+        u.integer :project_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:programmers) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:projects) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard).connection.create_table(:comments) do |u|
+        u.string :name
+        u.string :commentable_type
+        u.integer :commentable_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:parts) do |u|
+        u.string :name
+        u.integer :item_id
+      end
+
+      BlankModel.using(shard).connection.create_table(:yummy) do |u|
+        u.string :name
+      end
+    end
+  end
+
+  desc 'Prepare the test databases'
+  task :prepare => [:drop, :create, :tables]
+end
+
+namespace :db do
+  desc 'Build the databases for tests'
+
+  task :build_databases do
+    mysql_user = ENV['MYSQL_USER'] || "root"
+    postgres_user = ENV['POSTGRES_USER'] || "postgres"
+    (1..5).each do |idx|
+      %x( echo "create DATABASE xbar_shard#{idx} DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_unicode_ci " | mysql --user=#{mysql_user})
+    end
+
+    %x( createdb -E UTF8 -U #{postgres_user} -T template0 xbar_shard1 )
+  end
+
+  desc 'Drop the tests databases'
+  task :drop_databases do
+    mysql_user = ENV['MYSQL_USER'] || "root"
+    postgres_user = ENV['POSTGRES_USER'] || "postgres"
+    (1..5).each do |idx|
+      %x( mysqladmin --user=#{mysql_user} -f drop xbar_shard#{idx} )
+    end
+
+    %x( dropdb -U #{postgres_user} xbar_shard1 )
+    %x( rm -f /tmp/database.sqlite3 )
+  end
+
+  desc 'Create tables on tests databases'
+  task :create_tables do
+    Dir.chdir(File.expand_path(File.dirname(__FILE__) + "/spec"))
+    # require 'active_support/core_ext/class/inheritable_attributes'
+    require 'active_record'
+    require "xbar"
+    [:master, :brazil, :canada, :russia, :alone_shard, :postgresql_shard, :sqlite_shard].each do |shard_symbol|
+      # Rails 3.1 needs to do some introspection around the base class, which requires
+      # the model be a descendent of ActiveRecord::Base.
+      class BlankModel < ActiveRecord::Base; end;
+
+      BlankModel.using(shard_symbol).connection.initialize_schema_migrations_table()
+
+      BlankModel.using(shard_symbol).connection.create_table(:users) do |u|
+        u.string :name
+        u.integer :number
+        u.boolean :admin
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:clients) do |u|
+        u.string :country
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:cats) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:items) do |u|
+        u.string :name
+        u.integer :client_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:computers) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:keyboards) do |u|
+        u.string :name
+        u.integer :computer_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:roles) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:permissions) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:permissions_roles, :id => false) do |u|
+        u.integer :role_id
+        u.integer :permission_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:assignments) do |u|
+        u.integer :programmer_id
+        u.integer :project_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:programmers) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:projects) do |u|
+        u.string :name
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:comments) do |u|
+        u.string :name
+        u.string :commentable_type
+        u.integer :commentable_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:parts) do |u|
+        u.string :name
+        u.integer :item_id
+      end
+
+      BlankModel.using(shard_symbol).connection.create_table(:yummy) do |u|
+        u.string :name
+      end
+    end
+  end
+
+  desc 'Prepare the test databases'
+  task :prepare => [:drop_databases, :build_databases, :create_tables]
+end
+

data/examples/README

@@ -0,0 +1,5 @@
+Set your environment variable BUNDLE_GEMFILE to one of the
+gemfiles in ../gemfiles, then do
+
+bundle exec ruby example1.rb
+

data/examples/config/simple.json

@@ -0,0 +1,22 @@
+{
+    "__COMMENT": "Simple SQLite French Environment",
+
+    "connections": {
+        "paris_m": { "adapter": "sqlite3", "database": "/tmp/paris.sqlite3"},
+	"france_1": { "adapter": "sqlite3", "database": "/tmp/france_1.sqlite3"},
+	"france_2": { "adapter": "sqlite3", "database": "/tmp/france_2.sqlite3"},
+	"france_3": { "adapter": "sqlite3", "database": "/tmp/france_3.sqlite3"}
+    },
+
+    "environments": {
+        "test": {
+            "shards": {
+                "master": "paris_m",
+                "france": ["france_1", "france_2", "france_3"],
+                "france_nord": "france_1",
+		"france_central": "france_2",
+                "france_sud": "france_3"
+            }
+        }
+    }
+}

data/examples/example1.rb

@@ -0,0 +1,34 @@
+require File.expand_path(File.join(File.dirname(__FILE__),  'setup'))
+
+module Examples
+  Setup.start('simple', 'test', 1)
+
+  # Define the model to let us access the 'users' table through ActiveRecord.
+  class User < ActiveRecord::Base; end
+
+  # Everything is now set up. Remember that the three french shards don't 
+  # really replicate.  XBar is set up to *think* that they do.  We know
+  # better, and can check that reads and writes go to the shards that 
+  # we think that they should.  They curious results below wouldn't happen
+  # if replication is really taking place (modulo the eventual-consistency
+  # problem).
+
+  User.using(:france_central).create!([{name: "central1"}, {name: "central2"}])
+  XBar.using(:france_sud) do
+    User.create!([{name: "sud1"}, {name: "sud2"}, {name: "sud3"}])
+  end
+  User.using(:france).create!(name: "nord1")
+
+  f1 = User.using(:france).all.size # 2
+  f2 = User.using(:france).all.size # 3
+  f3 = User.using(:france).all.size # 2
+  f4 = User.using(:france).all.size # 3
+  n1 = User.using(:france_nord).all.size # 1
+  c1 = User.using(:france_central).all.size # 2
+  s1 = User.using(:france_sud).all.size # 3
+
+  puts [f1, f2, f3, f4, n1, c1, s1].to_s # [2, 3, 2, 3, 1, 2, 3]
+
+  Setup.stop(1)
+end
+

data/examples/migrations/1_create_users.rb

@@ -0,0 +1,10 @@
+class CreateUsers < ActiveRecord::Migration
+  using :france_nord, :france_central, :france_sud
+  def change
+    create_table :users do |t|
+      t.string :name
+
+      t.timestamps
+    end
+  end
+end

data/examples/setup.rb

@@ -0,0 +1,43 @@
+require 'active_support'
+require 'active_record'
+require 'xbar'
+
+# This file demonstrates a self-contained use of XBar.  The only support
+# is a subdirectory 'config' that holds a JSON configuration file, and
+# a subdirectory 'migrations' that holds the one migration that we plan
+# to use. 
+
+module Examples
+  module Setup
+
+    MIGRATIONS_ROOT = File.expand_path(File.join(File.dirname(__FILE__),  'migrations'))
+
+    def self.start(xbar_env, app_env, version = nil)
+
+      # This must agree with what's in the 'simple' JSON config file.  Make
+      # sure that we're starting with a clean slate.
+      %x{ rm -f /tmp/paris.sqlite3 /tmp/france_1.sqlite3 \
+        /tmp/france_2.sqlite3 /tmp/france_3.sqlite3 }
+
+      # This directory should have a subdirectory called 'config' which
+      # actually holds the config files.
+      XBar.directory = File.expand_path(File.dirname(__FILE__))
+
+      # Initialize the mapper with the 'test' environment from the 'simple' 
+      # configuration file.
+      XBar::Mapper.reset(xbar_env: xbar_env, app_env: app_env)
+
+      # Use a migration to create initial table(s) to work with.
+      if version
+        ActiveRecord::Migrator.run(:up, MIGRATIONS_ROOT, version)
+      end
+    end
+
+    def self.stop(version = nil)
+      if version
+        ActiveRecord::Migrator.run(:down, MIGRATIONS_ROOT, version)
+      end
+    end
+
+  end
+end