atduskgreg-slipcover 0.2.0

data/README.rdoc

@@ -0,0 +1,19 @@
+Two simple scripts for helping with CouchDB clustering and parallelization:
+
+=Slipcover
+
+- run a single query across a multi-member cluser (i.e. a group of CouchDBs) and zip up the results
+
+=California King
+
+- run a series of queries in parallel against a single CouchDB
+
+=Details
+
+Checkout slipcover.doctest and california_king.doctest for usage and explanation.
+
+To run the doctests:
+
+  gem install rubydoctest
+  cd slipcover
+  rubydoctest doctest/*.doctest

data/doctest/california_king.doctest

@@ -0,0 +1,35 @@
+  >> $:.unshift File.dirname(__FILE__)
+	>> require 'doctest_helper'
+
+## CalforniaKing ##
+
+
+- named after the hugest mattress you can get.
+  - where Slipcover lets you run a single query against a cluster, CaliforniaKing runs multiple simultaneous queries against a single CouchDb
+  - furnish the king with an array of queries, and it will execute them across a series of threads (because Erlang/CouchDB eats up concurrency like a king)
+  - using CaliforniaKing you can usually get done in seconds what might take minutes, without threading.
+  
+# Example: run a collection of queries in parallel against a single db.
+
+Require CalforniaKing:
+
+	>> require 'lib/california_king'
+	>> cr = CouchRest.new('localhost:5984')
+	>> cr.database('california_king-test').delete! rescue nil
+	>> db = cr.create_db('california_king-test')
+
+Create a bunch of docs:
+
+  >> docs = []
+  >> 400.times{|n|docs.push({:number => n})}
+  >> saved = db.bulk_save(docs)
+  >> saved['new_revs'].length
+  => 400
+  
+Retrieve them using 10 threads:
+
+  >> queries =  saved['new_revs'].collect{|r|[:get, r['id']]}
+  >> king = CaliforniaKing.new('localhost:5984','california_king-test', 10)
+  >> results = king.query queries
+  >> results.collect{|doc|doc['number']}.uniq.length
+  => 400

data/doctest/doctest_helper.rb

@@ -0,0 +1,11 @@
+$:.unshift File.dirname(__FILE__) + "/.."
+
+class Proc
+  def raises_error?(err)
+    begin
+      self.call
+    rescue Exception => e
+      e.is_a? err
+    end
+  end
+end

data/doctest/slipcover.doctest

@@ -0,0 +1,76 @@
+  >> $:.unshift File.dirname(__FILE__)
+	>> require 'doctest_helper'
+
+## Slipcover ##
+
+- treat a series of objects as a cluster by:
+ - systematically calling the same method on all of them in parallel
+ - zipping up the results
+
+# Example: access a cluster of CouchDB databases accessed via CouchRest (http://github.com/jchris/couchrest/tree/master)
+
+Require Slipcover and CouchRest:
+
+	>> require 'lib/slipcover'
+	>> require 'couchrest'
+
+Create the members of the cluster (these could be on different hosts, but we'll just simulate that here):
+  	 
+	>> cr1 = CouchRest.new('localhost:5984')
+	>> cr1.database('slipcover-test').delete! rescue nil
+	>> cr1.create_db('slipcover-test')
+
+Create cluster member two.
+	
+	>> cr2 = CouchRest.new('127.0.0.1:5984')
+	>> cr2.database('slipcover-test2').delete! rescue nil
+	>> cr2.create_db('slipcover-test2') 
+	>> db1 = cr1.database('slipcover-test')
+	>> db2 = cr2.database('slipcover-test2')
+	>> saved = db1.save({"test"=>"doc"})
+	>> saved['ok']
+	=> true
+
+Assign them to Slipcover for management:
+        
+	>> cluster = Slipcover.new( [db1, db2] ) 
+
+By default, our cluster will re-raise any errors that occur in individual members
+	
+	>> lambda{ cluster.get( saved['id'] )}.raises_error? RestClient::ResourceNotFound
+	=> true
+
+but, if we want to ignore certain errors in the members (like in this case where we only want to hear back from the member of the cluster that actually has the document we're looking for), we can tell Slipcover to silence errors of a certain type
+
+	>> cluster.silenced_errors << RestClient::ResourceNotFound	
+	=> [RestClient::ResourceNotFound]
+
+and then getting a document that's present on only one of the members will return the document without any noise from the other cluster members:
+  
+	>> result = cluster.get( saved['id'] )
+	>> result.first['test']
+	=> "doc" 
+
+If members raise other errors that aren't included in the list to be silenced, however, they will bring things to a halt. For example, if we add another member to the cluster on a broken connection
+
+	>> db3 = CouchRest.new('broken-socket').database('no-couch-here')
+	>> cluster.add_member db3
+
+then the resulting errors will still raise:
+
+	>> lambda{ cluster.get( saved['id'] )}.raises_error? Object
+	=> true
+
+Let's remove this broken cluster member so we can continue our tests:
+
+	>> cluster.remove_member{ |m| m.host == 'broken-socket' }
+	>> cluster.members.length == 2
+	=> true
+
+We could also remove a particular member if we had it handy
+
+	>> cluster.add_member db3
+	>> cluster.remove_member db3
+	>> cluster.members.length == 2
+	=> true
+

data/lib/california_king.rb

@@ -0,0 +1,51 @@
+require 'rubygems'
+require 'enumerator'
+require 'couchrest'
+
+class CaliforniaKing
+  attr_accessor :silenced_errors, :width
+  
+  def initialize(server, dbname, width=7)
+    @server = server
+    @dbname = dbname
+    @width = width
+    @silenced_errors = []
+  end
+  
+  def query(queries)
+    results = []
+    slice_size = queries.length.to_f / @width
+    # each thread gets a slice to process, so it doesn't have to wait on others
+    threads = []
+    puts "slicing #{queries.length} queries into slices of size #{slice_size}"
+    queries.each_slice([slice_size.round,1].max) do |qs|
+      puts "feeding #{qs.length} queries to thread #{threads.length}"
+      threads << Thread.new(database) do |db|
+        qs.each do |q|
+          method = q.shift
+          begin
+            results << db.send(method, *q)
+            $stdout.putc '.'
+            $stdout.flush
+          rescue Exception => e
+            raise e unless silenced_errors_include? e
+          end
+        end
+        puts "thread finished"
+      end
+    end
+    threads.each{|t| t.join}
+    results
+  end
+  
+  private
+  
+  def database
+    CouchRest.new(@server).database(@dbname)
+  end
+  
+  def silenced_errors_include? e
+    @silenced_errors.any?{|eklass| e.is_a? eklass}
+  end
+
+end

data/lib/slipcover.rb

@@ -0,0 +1,43 @@
+class Slipcover
+  attr_accessor :silenced_errors, :members
+  
+  def initialize(members)
+    @members = Array(members)
+    @silenced_errors = []
+  end
+
+  def add_member member
+    @members << member
+  end
+  
+  def remove_member(member=nil, &block)
+    @members.delete(member) if member
+    @members.reject!{ |m| block.call(m) } if block_given?
+  end
+  
+  
+  def method_missing(method, *args, &block)
+    results = []
+    threads = []
+        
+    @members.each do |m|
+      threads << Thread.new(m) do |member|
+        begin
+          results << member.send(method, *args)
+        rescue Exception => e
+          raise e unless silenced_errors_include? e
+        end
+      end
+    end
+      
+    threads.each{|t| t.join}
+    results
+  end
+  
+  private
+  
+  def silenced_errors_include? e
+    @silenced_errors.any?{|eklass| e.is_a? eklass}
+  end
+
+end

data/slipcover.gemspec

@@ -0,0 +1,20 @@
+Gem::Specification.new do |s|
+
+  s.name = "slipcover"
+  s.version = "0.2.0"
+  s.date = "2008-09-09"
+  s.summary = "CouchDB clustering and parallelization."
+  s.email = "greg@grabb.it"
+  s.homepage = "http://github.com/atduskgreg/slipcover"
+  s.description = "Slipcover runs a single query across a multi-member cluser (i.e. a group of CouchDBs) and zip up the results. CaliforniaKing runs a series of queries in parallel against a single CouchDB."
+  s.has_rdoc = false
+  s.authors = ["Greg Borenstein", "J. Chris Anderson"]
+  s.files = %w{
+    lib/slipcover.rb lib/california_king.rb
+    README.rdoc 
+    slipcover.gemspec
+    doctest/slipcover.doctest doctest/california_king.doctest doctest/doctest_helper.rb
+    }
+  s.require_path = "lib"
+  s.add_dependency("couchrest", [">= 0.9"])
+end

metadata

@@ -0,0 +1,68 @@
+--- !ruby/object:Gem::Specification 
+name: atduskgreg-slipcover
+version: !ruby/object:Gem::Version 
+  version: 0.2.0
+platform: ruby
+authors: 
+- Greg Borenstein
+- J. Chris Anderson
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-09-09 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: couchrest
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0.9"
+    version: 
+description: Slipcover runs a single query across a multi-member cluser (i.e. a group of CouchDBs) and zip up the results. CaliforniaKing runs a series of queries in parallel against a single CouchDB.
+email: greg@grabb.it
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- lib/slipcover.rb
+- lib/california_king.rb
+- README.rdoc
+- slipcover.gemspec
+- doctest/slipcover.doctest
+- doctest/california_king.doctest
+- doctest/doctest_helper.rb
+has_rdoc: false
+homepage: http://github.com/atduskgreg/slipcover
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: CouchDB clustering and parallelization.
+test_files: []
+