netinlet-sweat_shop 1.1.0

data/History.txt

@@ -0,0 +1,6 @@
+=== 1.0.0 / 2009-01-13
+
+* 1 major enhancement
+
+  * Birthday!
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Amos Elliston
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,105 @@
+# SweatShop
+
+SweatShop provides an api to background resource intensive tasks. Much of the api design was copied from Workling, with a few tweaks.
+Currently, it runs rabbitmq and kestrel, but it can support any number of queues.
+
+## Installing 
+
+    gem install sweat_shop
+    freeze in your rails directory
+    cd vendor/gems/sweat_shop
+    rake setup
+
+## Writing workers
+
+Put `email_worker.rb` into app/workers and sublcass `SweatShop::Worker`:
+
+    class EmailWorker
+      def send_mail(to)
+        user = User.find_by_id(to)
+        Mailer.deliver_welcome(to)
+      end
+    end
+
+Then, anywhere in your app you can execute:
+
+    EmailWorker.async_send_mail(1)
+
+The `async` signifies that this task will be placed on a queue to be serviced by the EmailWorker possibly on another machine. You can also
+call:
+
+    EmailWorker.send_mail(1)
+
+That will do the work immediately, without placing the task on the queue. You can also define a `queue_group` at the top of the file
+which will allow you to split workers out into logical groups. This is important if you have various machines serving different
+queues. 
+
+## Running the queue
+
+SweatShop has been tested with Rabbit and Kestrel, but it will also work with Starling. Please use the following resources to install the server:
+
+Kestrel:
+http://github.com/robey/kestrel/tree/master
+
+Rabbit:
+http://github.com/ezmobius/nanite/tree/master
+
+config/sweatshop.yml specifies the machine address of the queue (default localhost:5672). You can also specify the queue type with the queue param.
+
+## Running the workers
+
+Assuming you ran `rake setup` in Rails, you can type:
+
+    script/sweatshop
+
+By default, the script will run all workers defined in the app/workers dir. Every task will be processed on each queue using a round-robin algorithm. You can also add the `-d` flag which will put the worker in daemon mode. The daemon also takes other params.  Add a `-h` for more details.
+
+    script/sweatshop -d
+    script/sweatshop -d stop
+
+If you would like to run SweatShop as a daemon on a linux machine, use the initd.sh script provided in the sweat_shop/script dir.
+
+# Serializers
+
+Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
+
+The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
+are available out of the box, Marshal, JSON & YAML.  
+
+To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your 
+own custom serializer.
+
+You may also write your own custom serializer classes by extending SweatShop::Serializer.  
+
+You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer.  Notice
+you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml.  This default short name
+is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e. 
+SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted.  You can
+override this default generated name by using the declaration in the top of your class:
+	serializer_name :foo 
+	
+Where :foo is what you want to register the serializer as.
+
+This is a complete example of how to write a custom serializer:
+
+   class MyCustomSerializer < SweatShop::Serializer
+      serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
+
+      def self.serialize(data)
+        ... # custom serialize the data
+      end
+      
+      def self.deserialize(data)
+        ... # custom deserialize the data
+      end
+   end
+
+# REQUIREMENTS
+
+    i_can_daemonize
+    memcache (for kestrel)
+    carrot (for rabbit)
+
+# LICENSE
+
+Copyright (c) 2009 Amos Elliston, Geni.com; Published under The MIT License, see License

data/Rakefile

@@ -0,0 +1,42 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rcov/rcovtask'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "sweat_shop"
+    s.summary = %Q{SweatShop is a simple asynchronous worker queue build on top of rabbitmq/ampq}
+    s.email = "doug@netinlet.com"
+    s.homepage = "http://github.com/netinlet/sweat_shop"
+    s.description = "Amos Elliston's sweat_shop + Custom Serialization"
+    s.authors = ["Amos Elliston", "Doug Bryant"]
+    s.files =  FileList["[A-Z]*", "{lib,test,config}/**/*"]
+    s.add_dependency('famoseagle-carrot', '= 0.7.0')
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+Rake::TestTask.new
+
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'new_project'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+Rcov::RcovTask.new do |t|
+  t.libs << 'test'
+  t.test_files = FileList['test/**/*_test.rb']
+  t.verbose = true
+end
+
+task :default => :test
+
+task :setup do
+  require File.dirname(__FILE__) + '/install'
+end

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:major: 1
+:minor: 1
+:patch: 0

data/bin/sweatd

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+require File.dirname(__FILE__) + '/../lib/sweat_shop/sweatd'

data/config/defaults.yml

@@ -0,0 +1,10 @@
+# default options
+default: 
+  queue: rabbit
+  host: localhost
+  port: 5672
+  user: 'guest'
+  pass: 'guest'
+  vhost: '/'
+  serializer: 'marshal'
+enable: true

data/config/sweatshop.yml

@@ -0,0 +1,18 @@
+development:
+  default:
+    queue: rabbit
+    host: localhost 
+    port: 5672
+  enable: true
+test:
+  default:
+    queue: rabbit
+    host: localhost 
+    port: 5672
+  enable: true
+production:
+  default:
+    queue: rabbit
+    host: localhost 
+    port: 5672
+  enable: true

data/lib/message_queue/base.rb

@@ -0,0 +1,17 @@
+module MessageQueue
+  class Base
+    attr_reader :opts
+    def queue_size(queue);    end
+    def enqueue(queue, data, serializer); end
+    def dequeue(queue, serializer);       end
+    def confirm(queue);       end
+    def subscribe(queue);     end
+    def delete(queue);        end
+    def client;               end
+    def stop;                 end
+
+    def subscribe?
+      false
+    end
+  end
+end

data/lib/message_queue/kestrel.rb

@@ -0,0 +1,32 @@
+module MessageQueue
+  class Kestrel < Base
+    def initialize(opts)
+      @servers = opts['servers']
+    end
+
+    def queue_size(queue)
+      size  = 0
+      stats = client.stats
+      servers.each do |server|
+        size += stats[server]["queue_#{queue}_items"].to_i
+      end
+      size
+    end
+
+    def enqueue(queue, data, serializer)
+      client.set(queue, data)
+    end
+
+    def dequeue(queue, serializer)
+      client.get("#{queue}/open")
+    end
+
+    def confirm(queue)
+      client.get("#{queue}/close")
+    end
+
+    def client
+      @client ||= MemCache.new(servers) 
+    end
+  end
+end

data/lib/message_queue/rabbit.rb

@@ -0,0 +1,72 @@
+require 'carrot'
+module MessageQueue
+  class Rabbit < Base
+
+    def initialize(opts={})
+      @opts = opts
+    end
+
+    def delete(queue)
+      send_command do
+        client.queue(queue).delete
+      end
+    end
+
+    def queue_size(queue)
+      send_command do
+        client.queue(queue).message_count
+      end
+    end
+
+    def enqueue(queue, data, serializer)
+      send_command do 
+        client.queue(queue, :durable => true).publish(serializer.serialize(data), :persistent => true)
+      end
+    end
+
+    def dequeue(queue, serializer)
+      send_command do
+        task = client.queue(queue).pop(:ack => true)
+        return unless task
+        serializer.deserialize(task)
+      end
+    end
+
+    def confirm(queue)
+      send_command do
+        client.queue(queue).ack
+      end
+    end
+
+    def send_command(&block)
+      retried = false
+      begin
+        block.call
+      rescue Carrot::AMQP::Server::ServerDown => e
+        if not retried
+          puts "Error #{e.message}. Retrying..."
+          @client = nil
+          retried = true
+          retry
+        else
+          raise e
+        end
+      end
+    end
+
+    def client
+      @client ||= Carrot.new(
+        :host   => @opts['host'], 
+        :port   => @opts['port'].to_i, 
+        :user   => @opts['user'], 
+        :pass   => @opts['pass'], 
+        :vhost  => @opts['vhost'],
+        :insist => @opts['insist']
+      ) 
+    end
+
+    def stop
+      client.stop
+    end
+  end
+end

data/lib/serializers/json_serializer.rb

@@ -0,0 +1,39 @@
+gem 'json'
+require 'json'
+require 'json/add/rails'
+require 'json/add/core'
+
+module SweatShop
+  module Serializers
+    class JsonSerializer < SweatShop::Serializer
+      class << self
+      
+        def serialize(payload)
+          if payload.respond_to?(:to_json)
+            payload.to_json
+          else
+            JSON.generate(payload)
+          end
+        end
+      
+        def deserialize(payload)
+          symbolize_keys(JSON.parse(payload))
+        end
+        
+        protected 
+        # another straight out of rails land - ActiveSupport::CoreExtensions::Hash::Keys
+        def symbolize_keys(data)
+          if data.is_a?(Hash)
+            data.inject({}) do |options, (key, value)|
+              options[(key.to_sym rescue key) || key] = value
+              options
+            end
+          else
+            data
+          end
+        end
+        
+      end
+    end
+  end
+end

data/lib/serializers/marshal_serializer.rb

@@ -0,0 +1,15 @@
+module SweatShop
+  module Serializers
+    class MarshalSerializer < SweatShop::Serializer
+      class << self
+        def serialize(payload)
+          Marshal.dump(payload)
+        end
+      
+        def deserialize(payload)
+          Marshal.load(payload)
+        end
+      end
+    end
+  end
+end

data/lib/serializers/serializer.rb

@@ -0,0 +1,108 @@
+# Serializers can be swapped out on a per worker basis and set globally to a default via the config file.
+# 
+# The default serializer is Marshal to maintain backward compatibility with existing apps. Three serializers
+# are available out of the box, Marshal, JSON & YAML.  
+# 
+# To change the default serializer, add a key [serializer] to the config file with a value of %w{marshal json yaml} or your 
+# own custom serializer.
+# 
+# You may also write your own custom serializer classes by extending SweatShop::Serializer.  
+# 
+# You can give your serializers an arbitrary class name, but the default pattern is to name it <somename>Serializer.  Notice
+# you can specify the Marshal, JSON or YAML serializer with the name :marshal, :json, or :yaml.  This default short name
+# is generated by underscoring the class name and removing the "Serializer" from the end of the class name, i.e. 
+# SweatShop::Serializers::JsonSerializer becomes :json and My::Custom::EncryptedSerializer would become :encrypted.  You can
+# override this default generated name by using the declaration in the top of your class:
+#   serializer_name :foo 
+#   
+# Where :foo is what you want to register the serializer as.
+# 
+# This is a complete example of how to write a custom serializer:
+# 
+#    class MyCustomSerializer < SweatShop::Serializer
+#       serializer_name :foo # overrides the generated default name of :my_custom based on the above rules
+# 
+#       def self.serialize(data)
+#         ... # custom serialize the data
+#       end
+#       
+#       def self.deserialize(data)
+#         ... # custom deserialize the data
+#       end
+#    end
+
+module SweatShop
+  class Serializer
+    
+    def Serializer.inherited(subklass)
+      register(subklass.get_name, subklass)
+    end
+    
+    def Serializer.serializers
+      @serializers ||= {}
+    end
+
+    def Serializer.default
+      @default_serializer ||= begin
+        if s = config['serializer']
+          unless serializers.has_key?(s.to_sym)
+            raise RuntimeError, "Unknown serializer [#{s}]"
+          end
+          log("Using [:#{s}] as the Default Serializer")
+          serializers[s.to_sym]
+        else
+          log("No Default Serializer specified.  Using [:marshal]")
+          serializers[:marshal]
+        end
+      end
+    end
+    
+    def Serializer.default=(serializer_name)
+      @default_serializer = serializers[serializer_name.to_sym]
+    end
+
+    def self.serializer_name(name)
+      @serializer_name = name
+      register(name, self)
+    end
+    
+    def self.get_name
+      @serializer_name || generate_serializer_name(self.to_s)
+    end
+
+    def self.log(msg)
+      SweatShop.log(msg)
+    end
+    
+    private
+    
+    def Serializer.register(name, klass)
+      # classes get inherited before the name is set - update it by removing first
+      Serializer.serializers.delete(generate_serializer_name(klass.to_s)) 
+      Serializer.serializers[name] = klass
+    end
+    
+    def self.config
+      SweatShop.config
+    end
+    
+    
+    def self.generate_serializer_name(name)
+      underscore(name.to_s.gsub(/Serializer$/, "")).split("/").last.to_sym    
+    end
+
+    # Taken straight out of the rails Inflector module
+    def self.underscore(camel_cased_word)
+      camel_cased_word.to_s.gsub(/::/, '/').
+      gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
+      gsub(/([a-z\d])([A-Z])/,'\1_\2').
+      tr("-", "_").
+      downcase
+    end
+  end
+end
+
+
+require File.dirname(__FILE__) +  '/marshal_serializer'
+require File.dirname(__FILE__) +  '/json_serializer'
+require File.dirname(__FILE__) +  '/yaml_serializer'