rails-sharding 0.1.0

data/.gitignore

@@ -0,0 +1,12 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/spec/fixtures/shards.yml
+/spec/fixtures/schemas
+/tmp/
+.byebug_history

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.ruby-gemset

@@ -0,0 +1 @@
+rails-sharding

data/.ruby-version

@@ -0,0 +1 @@
+ruby-2.3.1

data/.travis.yml

@@ -0,0 +1,10 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.3.1
+services:
+  - mysql
+before_install: gem install bundler -v 1.12.4
+before_script:
+  - cp spec/fixtures/shards.yml.travis spec/fixtures/shards.yml
+  - rake db:test:prepare

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in rails-sharding.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Henrique Gubert
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,156 @@
+# Rails::Sharding
+
+[![Build Status](https://travis-ci.org/hsgubert/rails-sharding.svg?branch=master)](https://travis-ci.org/hsgubert/rails-sharding)
+[![Code Climate](https://codeclimate.com/github/hsgubert/rails-sharding/badges/gpa.svg)](https://codeclimate.com/github/hsgubert/rails-sharding)
+
+
+Simple and robust sharding for Rails, including Migrations and ActiveRecord extensions
+
+This gems allows you to easily create extra databases to your rails application, and freely allocate ActiveRecord instances to any of the databases. It also provides rake tasks and migrations to help you manage the schema by shard groups.
+
+After you have setup your shards, accessing them is as simple as:
+```ruby
+  new_user = User.using_shard(:shard_group1, :shard1).create(username: 'x')
+  loaded_user = User.using_shard(:shard_group1, :shard1).where(username: 'x').first
+```
+
+You can also use the block syntax, where all your queries inside will be directed to the correct shard:
+```ruby
+  Rails::Sharding.using_shard(:shard_group1, :shard1) do
+    new_user = User.create(username: 'x')
+    loaded_user = User.where(username: 'x').first
+    billing_infos = loaded_user.billing_infos.all
+  end
+```
+
+You can also pick and choose which models will be shardable, so that all the models that are not shardable will still be retrieved from the master database, even if inside a using_shard block.
+
+## Compatibility
+As of now this gem has been tested only with Rails 4.2. It does not work yet with Rails 5.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'rails-sharding'
+```
+
+And then execute:
+```
+bundle
+```
+
+## Creating Shards
+This gem helps you create shards that are additional and completely separate from your master database. The master database is the one that is created and managed through rails, and is the default storage for all your models.
+
+To start with the rails-sharding gem, run the command
+```
+rails g rails_sharding:scaffold
+```
+
+This will generate a `config/shards.yml.example` like this:
+```ruby
+default: &default
+  adapter: mysql2
+  encoding: utf8
+  reconnect: false
+  pool: 5
+  username: ___
+  password: ___
+  socket: /var/run/mysqld/mysqld.sock
+
+development:
+  shard_group1:
+    shard1:
+      <<: *default
+      database: group1_shard1_development
+    shard2:
+      <<: *default
+      database: group1_shard2_development
+...
+```
+
+Rename it to `config/shards.yml` and change it to your database configuration. This example file defines a single shard group (named `shard_group1`) containing two shards (`shard1` and `shard2`). A shard group is simply a set of shards that should have the same schema.
+
+When you're ready to create the shards run
+```
+rake shards:create
+```
+
+## Migrating Shards
+Go to the directory `db/shards_migrations/shard_group1` and add all migrations that you want to run on the shards of `shard_group1`. By design, all shards in a same group should always have the same schema. For example, add the following migration to your `db/shards_migrations/shard_group1`:
+```ruby
+# 20160808000000_create_users.rb
+class CreateClients < ActiveRecord::Migration
+  def up
+    create_table :users do |t|
+      t.string :username, :limit => 100
+      t.timestamps
+    end
+  end
+
+  def down
+    drop_table :users
+  end
+end
+```
+
+Then run:
+```
+rake shards:migrate
+```
+
+All the shards will be migrated, and one schema file will be dumped for each of the shards (just like rails would do for your master database). You can see the schema of the shards in `db/shards_schemas/shard_group1/`, and it will be something like:
+```ruby
+ActiveRecord::Schema.define(version: 20160808000000) do
+
+  create_table "users", force: :cascade do |t|
+    t.string   "username",       limit: 100
+    t.datetime "created_at"
+    t.datetime "updated_at"
+  end
+
+end
+```
+
+## Other rake tasks
+The rails-sharding gem offers several rake tasks analogous to the ones offered by ActiveRecord:
+```
+rake shards:create                                      
+rake shards:drop                                        
+rake shards:migrate                                     
+rake shards:migrate:down                                
+rake shards:migrate:redo                                
+rake shards:migrate:reset                               
+rake shards:migrate:up                                  
+rake shards:rollback                                    
+rake shards:schema:dump                                 
+rake shards:schema:load                                 
+rake shards:test:load_schema                            
+rake shards:test:prepare                                
+rake shards:test:purge               
+rake shards:version
+```
+
+They work just the same as the tasks `rake:db:...` but they operate on all shards of all shard groups. If you want to run a rake task just to a specific shard group or shard you can use the `SHARD_GROUP` and `SHARD` options:
+```
+rake shards:migrate SHARD_GROUP=shard_group_1
+rake shards:migrate SHARD_GROUP=shard_group_1 SHARD=shard1
+```
+
+
+## Development and Contributing
+
+After checking out the repo, run `bundle` to install gems and run `rake db:test:prepare` to create the test shards. Then, run `rspec` to run the tests.
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/hsgubert/rails-sharding.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+
+## Acknowledgements
+
+This gem was inspired and based on several other gems like: [octopus](https://github.com/thiagopradi/octopus), [shard_handler](https://github.com/locaweb/shard_handler) and [active_record_shards](https://github.com/zendesk/active_record_shards).

data/Rakefile

@@ -0,0 +1,52 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+
+# defines an environment task so we can run rake tasks from lib/tasks/rails-sharding.rake.
+# They require the :environment task, which is defined by rails, but we don need anything
+task :environment do
+  # do nothing
+end
+
+namespace :db do
+  namespace :test do
+
+    desc 'Loads gem test environment and rake tasks from gem'
+    task :load_env do
+      require 'rspec'
+
+      # requires spec helper but ensures no test coverage is reported to codeclimate
+      ENV['CODECLIMATE_REPO_TOKEN'] = nil
+      require './spec/spec_helper'
+      
+      load 'lib/tasks/rails-sharding.rake'
+    end
+
+    desc "Creates database shards for testing the gem"
+    task create: [:load_env] do
+      # simply calls shards:create as it will work properly after spec_helper
+      # changed the shards configuration
+      Rake::Task['shards:create'].invoke
+    end
+
+    desc "Drops database shards for testing the gem"
+    task drop: [:load_env] do
+      Rake::Task['shards:drop'].invoke
+    end
+
+    desc "Migrates database shards for testing the gem"
+    task migrate: [:load_env] do
+      Rake::Task['shards:migrate'].invoke
+    end
+
+    desc "Prepares database shards for testing the gem (this will clear and recreate database)"
+    task prepare: [:load_env] do
+      Rake::Task['db:test:drop'].invoke
+      Rake::Task['db:test:create'].invoke
+      Rake::Task['db:test:migrate'].invoke
+    end
+  end
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "rails/sharding"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/generators/scaffold_generator.rb

@@ -0,0 +1,27 @@
+require 'rails/generators'
+
+module RailsSharding
+  class ScaffoldGenerator < Rails::Generators::Base
+    source_root File.expand_path('../templates', __FILE__)
+
+    def copy_initializer
+      copy_file 'rails-sharding_initializer.rb', 'config/initializers/rails-sharding.rb'
+    end
+
+    def copy_configuration_file_and_example
+      copy_file 'shards.yml.example', Rails::Sharding::Config.shards_config_file + '.example'
+      copy_file 'shards.yml.example', Rails::Sharding::Config.shards_config_file
+    end
+
+    def add_configuration_to_gitignore
+      append_to_file '.gitignore' do
+        "\n" + Rails::Sharding::Config.shards_config_file
+      end
+    end
+
+    def create_migrations_and_schema_directory
+      empty_directory Rails::Sharding::Config.shards_migrations_dir
+      empty_directory Rails::Sharding::Config.shards_schemas_dir
+    end
+  end
+end

data/lib/generators/templates/rails-sharding_initializer.rb

@@ -0,0 +1,4 @@
+
+Rails::Sharding.setup do |config|
+
+end

data/lib/generators/templates/shards.yml.example

@@ -0,0 +1,35 @@
+default: &default
+  adapter: mysql2
+  encoding: utf8
+  reconnect: false
+  pool: 5
+  username: ___
+  password: ___
+  socket: /var/run/mysqld/mysqld.sock
+
+development:
+  shard_group1:
+    shard1:
+      <<: *default
+      database: group1_shard1_development
+    shard2:
+      <<: *default
+      database: group1_shard2_development
+
+test:
+  shard_group1:
+    shard1:
+      <<: *default
+      database: group1_shard1_test
+    shard2:
+      <<: *default
+      database: group1_shard2_test
+
+production:
+  shard_group1:
+    shard1:
+      <<: *default
+      database: group1_shard1
+    shard2:
+      <<: *default
+      database: group1_shard2

data/lib/rails/sharding.rb

@@ -0,0 +1,18 @@
+require 'rails'
+
+require 'rails/sharding/version'
+require 'rails/sharding/core'
+require 'generators/scaffold_generator'
+
+require 'rails/sharding/railtie' if defined?(Rails::Railtie)
+
+module Rails
+  module Sharding
+
+    # delegates all methods to Core, to shorten method calls
+    def self.method_missing(method_sym, *arguments, &block)
+      Core.send(method_sym, *arguments, &block)
+    end
+
+  end
+end

data/lib/rails/sharding/active_record_extensions.rb

@@ -0,0 +1,90 @@
+
+module Rails::Sharding
+  module ActiveRecordExtensions
+    # Will automatically add the #using_shard method to all ActiveRecord scopes
+    # (including has_many and habtm relations).
+    #
+    # Despite the fact that the #using_shard method will exist for all scopes, it
+    # will only have effect when loading/saving models that include the
+    # Rails::Sharding::Shardable module
+    def self.extend_active_record_scope
+      # avoinds duplicate extension
+      return if ActiveRecord::Base.respond_to? :using_shard
+
+      # Includes #using_shard in ActiveRecord::Base models (both classes and instances)
+      ActiveRecord::Base.extend ScopeMethods
+      ActiveRecord::Base.include ScopeMethods
+      ActiveRecord::Base.extend CaseFixer
+
+      # Includes #using_shard scope method in scopes
+      ActiveRecord::Relation.include ScopeMethods
+      ActiveRecord::Relation.extend CaseFixer
+
+      # Includes #using_shard scope method in has_many and habtm relations
+      ActiveRecord::Scoping.include ScopeMethods
+      ActiveRecord::Scoping.extend CaseFixer
+    end
+
+    module ScopeMethods
+      def using_shard(shard_group, shard_name)
+        if block_given?
+          raise Errors::WrongUsageError,
+            "#{name}.using is not allowed to receive a block, it works just like a regular scope.\nIf you are trying to scope everything to a specific shard, use Shards::Core.using_shard instead."
+        end
+
+        ScopeProxy.new(shard_group, shard_name, self)
+      end
+    end
+
+    # Return value of the #using_shard scope method. Allows us to chain the shard
+    # choice with other ActiveRecord scopes
+    class ScopeProxy
+      attr_accessor :original_scope
+
+      def initialize(shard_group, shard_name, original_scope)
+        @shard_group = shard_group
+        @shard_name = shard_name
+        @original_scope = original_scope
+      end
+
+      # if using_shard is called twice in a chain, just replaces configuration
+      def using_shard(shard_group, shard_name)
+        @shard_group = shard_group
+        @shard_name = shard_name
+        self
+      end
+
+      def method_missing(method, *args, &block)
+        # runs any method chained in the correct shard
+        result = Core.using_shard(@shard_group, @shard_name) do
+          @original_scope.send(method, *args, &block)
+        end
+
+        # if result is still a scope (responds to to_sql), update original scope
+        # and return proxy to continue chaining
+        if result.respond_to?(:to_sql)
+          @original_scope = result
+          return self
+        end
+
+        result
+      end
+
+      # Delegates == to method_missing so that User.using_scope(:a,:b).where(:name => "Mike")
+      # gets run in the correct shard context when #== is evaluated.
+      def ==(other)
+        method_missing(:==, other)
+      end
+      alias_method :eql?, :==
+    end
+
+    # Fixes case when behavior when ScopeProxy is passed to case
+    # (otherwise classes don't match)
+    module CaseFixer
+      def ===(other)
+        other = other.original_scope while other === ScopeProxy
+        super
+      end
+    end
+  end
+end