master_slave_adapter_soundcloud 0.1.0 → 0.1.1

data/Readme.md

@@ -0,0 +1,180 @@
+# Replication Aware Master Slave Adapter
+
+Improved version of the [master_slave_adapter plugin][1], packaged as a gem.
+
+## Features
+
+1. automatic selection of master or slave connection: `with_consistency`
+2. manual selection of master or slave connection: `with_master`, `with_slave`
+3. transaction callbacks: `on_commit`, `on_rollback`
+4. also:
+  * support for multiple slaves
+  * (partial) support for [database_cleaner][2]
+
+### Automatic Selection of Master or Slave
+
+* _note that this feature currently only works with MySQL_
+* _see also this [blog post][3] for a more detailed explanation_
+
+The adapter will run all reads against a slave database, unless a) the read is inside an open transaction or b) the
+adapter determines that the slave lags behind the master _relative to the last write_. For this to work, an initial
+initial consistency requirement ("`Clock`") must be passed to the adapter. Based on this clock value, the adapter
+determines if a (randomly chosen) slave meets this requirement. If not, all statements are executed against master,
+otherwise, the slave connection is used until either a transaction is opened or a write occurs. After a successful write
+or transaction, the adapter determines a new consistency requirement, which is returned and can be used for subsequent
+operations. Note that after a write or transaction, the adapter keeps using the master connection.
+
+As an example, a Rails application could run the following function as an `around_filter`:
+
+```ruby
+def with_consistency_filter
+  if logged_in?
+    # it's a good idea to use this feature on a per-user basis
+    cache_key = [ CACHE_NAMESPACE, current_user.id.to_s ].join(":")
+
+    clock = cached_clock(cache_key) ||
+      ActiveRecord::Base.connection.master_clock
+
+    new_clock = ActiveRecord::Base.with_consistency(clock) do
+        # inside the controller, ActiveRecord models can be used just as normal.
+        # The adapter will take care of choosing the right connection.
+        yield
+      end
+
+    [ new_clock, clock ].compact.max.tap do |c|
+      cache_clock!(cache_key, c)
+    end if new_clock != clock
+  else
+    # anonymous users will have to wait until the slaves have caught up
+    with_slave { yield }
+  end
+end
+```
+
+Note that we use the current `master_clock` as a reference point. This will give the user a recent view of the data,
+possibly reading from master, and if no write occurs inside the `with_consistency` block, we have a reasonable value to
+cache and reuse on subsequent requests. Alternatively, we could have used
+`ActiveRecord::ConnectionAdapters::MasterSlaveAdapter::Clock.zero` to indicate no particular consistency requirement.
+Since `with_consistency` blocks can be nested, the controller code could later decide to require a more recent view on
+the data.
+
+### Manual Selection of Master or Slave
+
+The original functionality of the adapter has been preserved:
+
+```ruby
+ActiveRecord::Base.with_master do
+  # everything inside here will go to master
+end
+
+ActiveRecord::Base.with_slave do
+  # everything inside here will go to one of the slaves
+  # opening a transaction or writing will switch to master
+  # for the rest of the block
+end
+```
+
+`with_master`, `with_slave` as well as `with_consistency` can be nested deliberately.
+
+### Transaction Callbacks
+
+This feature was originally developed at [SoundCloud][4] for the standard `MysqlAdapter`. It allows arbitrary blocks of
+code to be deferred for execution until the next transaction completes (or rolls back).
+
+```irb
+irb> ActiveRecord::Base.on_commit { puts "COMMITTED!" }
+irb> ActiveRecord::Base.on_rollback { puts "ROLLED BACK!" }
+irb> ActiveRecord::Base.connection.transaction do
+irb*   # ...
+irb> end
+COMMITTED!
+=> nil
+irb> ActiveRecord::Base.connection.transaction do
+irb*   # ...
+irb*   raise "failed operation"
+irb> end
+ROLLED BACK!
+# stack trace omitted
+=> nil
+```
+
+Note that a transaction callback will be fired only *once*, so you might want to do:
+
+```ruby
+class MyModel
+  after_save do
+    connection.on_commit do
+      # ...
+    end
+  end
+end
+```
+
+### Support for Multiple Slaves
+
+The adapter keeps a list of slave connections (see *Configuration*) and chooses randomly between them. The selection is
+made at the beginning of a `with_slave` or `with_consistency` block and doesn't change until the block returns. Hence, a
+nested `with_slave` or `with_consistency` might run against a different slave.
+
+### Database Cleaner
+
+At [SoundCloud][4], we're using [database_cleaner][2]'s 'truncation strategy' to wipe the database between [cucumber][5]
+'feature's. As our cucumber suite proved valuable while testing the `with_consistency` feature, we had to support
+`truncate_table` as an `ActiveRecord::Base.connection` instance method. We might add other strategies if there's enough
+interest.
+
+## Configuration
+
+Example configuration for the development environment in `database.yml`:
+
+```yaml
+development:
+  adapter: master_slave          # use master_slave adapter
+  connection_adapter: mysql      # actual adapter to use (only mysql is supported atm)
+  disable_connection_test: false # when an instance is checked out from the connection pool,
+                                 # we check if the connections are still alive, reconnecting if necessary
+  # these values are picked up as defaults in the 'master' and 'slaves' sections:
+  database: aweapp_development
+  username: aweappuser
+  password: s3cr3t
+
+  master:
+    host: masterhost
+    username: readwrite_user     # override default value
+
+  slaves:
+    - host: slave01
+    - host: slave02
+```
+
+## Installation
+
+Using plain rubygems:
+
+```sh
+$ gem install master_slave_adapter_soundcloud
+```
+
+Using bundler:
+
+```sh
+$ cat >> Gemfile
+gem 'master_slave_adapter_soundcloud', '~> 0.1', :require => 'master_slave_adaper'
+^D
+$ bundle install
+```
+
+## Credits
+
+* Maurício Lenhares - _original master_slave_adapter plugin_
+* Torsten Curdt     - _with_consistency, maintainership & open source licenses_
+* Sean Treadway     - _chief everything & transaction callbacks_
+* Kim Altintop      - _strong lax monoidal endofunctors_
+* Omid Aladini      - _chief operator & everything else_
+
+
+[1]: https://github.com/mauricio/master_slave_adapter
+[2]: https://github.com/bmabey/database_cleaner
+[3]: http://www.yourdailygeekery.com/2011/06/14/master-slave-consistency.html
+[4]: http://backstage.soundcloud.com
+[5]: http://cukes.info

data/VERSION

@@ -1 +1 @@
-0.1.0
+0.1.1

data/master_slave_adapter.gemspec

@@ -18,7 +18,7 @@ Gem::Specification.new do |s|
   s.require_path = 'lib'
 
   s.required_ruby_version     = '>= 1.9.2'
-  s.required_rubygems_version = '1.3.7'
+  s.required_rubygems_version = '>= 1.3.7'
   s.add_development_dependency 'rspec'
 
   s.add_dependency 'activerecord', '= 2.3.9'

metadata

@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
   segments: 
   - 0
   - 1
-  - 0
-  version: 0.1.0
+  - 1
+  version: 0.1.1
 platform: ruby
 authors: 
 - Mauricio Linhares
@@ -61,8 +61,8 @@ files:
 - .gitignore
 - Gemfile
 - LICENSE
-- README
 - Rakefile
+- Readme.md
 - VERSION
 - lib/active_record/connection_adapters/master_slave_adapter.rb
 - lib/master_slave_adapter.rb
@@ -90,7 +90,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement 
   none: false
   requirements: 
-  - - "="
+  - - ">="
     - !ruby/object:Gem::Version 
       segments: 
       - 1

data/README

@@ -1,43 +0,0 @@
-master_slave_adapter
-====
-
-This simple plugin acts as a common ActiveRecord adapter and allows you to
-setup a master-slave environment using any database you like (and is supported
-by ActiveRecord).
-
-This plugin works by handling two connections, one to a master database,
-that will receive all non-"SELECT" statements, and another to a slave database
-that that is going to receive all SELECT statements. It also tries to do as
-little black magic as possible, it works just like any other ActiveRecord database
-adapter and performs no monkeypatching at all, so it's easy and simple to use
-and understand.
-
-The master database connection will also receive SELECT calls if a transaction
-is active at the moment or if a command is executed inside a "with_master" block:
-
-ActiveRecord::Base.with_master do # :with_master instructs the adapter
-    @users = User.all             # to use the master connection inside the block
-end
-
-To use this adapter you just have to install the plugin:
-
-ruby script/plugin install git://github.com/tcurdt/master_slave_adapter_mauricio.git
-
-And then configure it at your database.yml file:
-
-development:
-  database: sample_development
-  username: root
-  adapter: master_slave             # the adapter must be set to "master_slave"
-  host: 10.21.34.80
-  master_slave_adapter: mysql       # here's where you'll place the real database adapter name
-  disable_connection_test: true     # this will disable the connection test before use,
-                                    # can possibly improve the performance but you could also
-                                    # hit stale connections, default is false
-  eager_load_connections: true      # connections are lazy loaded by default, you can load gem eagerly setting this to true
-  master:                           # and here's where you'll add the master database configuration
-    database: talkies_development   # you shouldn't specify an "adapter" here, the
-    username: root                  # value at "master_slave_adapter" is going to be used
-    host: 10.21.34.82
-    adapter: postgresql             # you can use another adapter for the master connection if needed
-                                    # if you don't set it the "master_slave_adapter" property will be used