em-resque 1.0.0.beta1

data/HISTORY.md

@@ -0,0 +1,3 @@
+## 1.0.0 (2011-XX-XX)
+
+* First release.

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) Julius de Bruijn
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,275 @@
+EM::Resque
+==========
+
+EM::Resque is an addition to [Resque][0] for asynchronic processing of the
+background jobs created by Resque. It works like the original Resque worker,
+but runs inside an [EventMachine][1] and uses the same process instead of
+forking a new one for every job. It can run N workers inside one process, it
+packs all of them in Ruby fibers. The library is meant for small but IO-heavy
+jobs, which won't use much of CPU power in the running server.
+
+Use cases
+---------
+
+EM::Resque is good for processing background jobs which are doing lots of IO.
+The evented nature of the reactor core is great when accessing third
+party services with HTTP or doing lots of database-intensive work. When
+combined with a connection pool to a SQL server it lets you easily control the
+amount of connections, being at the same time extremely scalable.
+
+Overview
+--------
+
+EM::Resque jobs are created and queued like with the synchronous version. When
+queued, one of the workers in the fiber pool will pick it up and process
+the job.
+
+When making IO actions inside a job, it should never block the other workers. E.g.
+database operations etc. should be handled with libraries that support EventMachine
+to allow concurrent processing.
+
+Resque jobs are Ruby classes (or modules) which respond to the
+`perform` method. Here's an example:
+
+``` ruby
+class Pinger
+  @queue = :ping_publisher
+
+  def self.perform(url)
+    self.url = url
+    self.result = EventMachine::HttpRequest.new(url).get.response
+    self.save
+  end
+end
+```
+
+The `@queue` class instance variable determines which queue `Pinger`
+jobs will be placed in. Queues are arbitrary and created on the fly -
+you can name them whatever you want and have as many as you want.
+
+To place an `Pinger` job on the `ping_publisher` queue, we might add this
+to our application's pre-existing `Callback` class:
+
+``` ruby
+class Callback
+  def async_ping_publisher
+    Resque.enqueue(Pinger, self.callback_url)
+  end
+end
+```
+
+Now when we call `callback.async_ping_publisher` in our
+application, a job will be created and placed on the `ping_publisher`
+queue.
+
+For more use cases please refer [the original Resque manual][0].
+
+Let's start 100 async workers to work on `ping_publisher` jobs:
+
+    $ cd app_root
+    $ QUEUE=file_serve FIBERS=100 rake em_resque:work
+
+This starts the EM::Resque process and loads 100 fibers with a worker inside
+each fiber and tells them to work off the `ping_publisher` queue. As soon as
+one of the workers is doing it's first IO action it will go to a "yield" mode
+to get data back from the IO and allow another one to start a new job. The
+event loop resumes the worker when it has some data back from the IO action.
+
+The workers also reserve the jobs for them so the other workers won't touch them.
+
+Workers can be given multiple queues (a "queue list") and run on
+multiple machines. In fact they can be run anywhere with network
+access to the Redis server.
+
+Jobs
+----
+
+What should you run in the background with EM::Resque? Anything with lots of
+IO and which takes any time at all. Best use case is gathering data and sending
+pings to 3rd party services, which might or might not answer in a decent time.
+
+At SponsorPay we use EM::Resque to process the following types of jobs:
+
+* Simple messaging between our frontend and backend softwares
+* Pinging publishers and affiliate networks 
+
+We're handling a tremendious amount of traffic with a bit over 100 workers,
+using a lot less of database connections, memory and cpu power compared to the
+synchronous and forking Resque or Delayed Job.
+
+All the environment options from the original Resque work also in EM::Resque.
+There are also couple of more variables.
+
+### The amount of fibers
+
+The number of fibers for the current process is set in FIBERS variable. One
+fiber equals one worker. They are all polling the same queue and terminated
+when the main process terminates. The default value is 1.
+
+    $ QUEUE=ping_publisher FIBERS=50 rake em_resque:work
+
+### The amount of green threads
+
+EventMachine has an option to use defer for long-running processes to be run in
+a different thread. The default value is 20.
+
+    $ QUEUE=ping_publisher CONCURRENCY=20 rake em_resque:work
+
+### Signals
+
+EM:Resque workers respond to a few different signals:
+
+* `QUIT` / `TERM` / `INT` - Wait for workers to finish processing then exit
+
+The Front End
+-------------
+
+EM::Resque uses the same frontend as Resque. 
+
+EM::Resque Dependencies
+-----------------------
+
+    $ gem install bundler
+    $ bundle install
+
+Installing EM::Resque
+---------------------
+
+### In a Rack app, as a gem
+
+First install the gem.
+
+    $ gem install em-resque
+
+Next include it in your application.
+
+``` ruby
+require 'em-resque'
+```
+
+Now start your application:
+
+    rackup config.ru
+
+That's it! You can now create EM::Resque jobs from within your app.
+
+To start a worker, create a Rakefile in your app's root (or add this
+to an existing Rakefile):
+
+``` ruby
+require 'your/app'
+require 'em-resque/tasks'
+```
+
+Now:
+
+    $ QUEUE=* FIBERS=50 rake em_resque:work
+
+Alternately you can define a `resque:setup` hook in your Rakefile if you
+don't want to load your app every time rake runs.
+
+### In a Rails 3 app, as a gem
+
+*EM::Resque is not supporting Rails with Rake at the moment. Needs more work.*
+
+To run EM::Resque with your Rails application, you need a specified script to
+load all the needed libraries and start the workers.
+
+*script/resque_async.rb*
+
+``` ruby
+RAILS_ENV = ENV['RAILS_ENV'] || 'development_async'
+RAILS_ROOT = Dir.pwd
+
+require 'rubygems'
+require 'yaml'
+require 'uri'
+require 'em-resque'
+require 'em-resque/worker_machine'
+require 'em-resque/task_helper'
+require 'resque-retry'
+require 'em-synchrony'
+require 'em-synchrony/connection_pool'
+require 'em-synchrony/mysql2'
+
+Dir.glob(File.join(RAILS_ROOT, 'lib', 'async_worker', '**', '*.rb')).sort.each{|f| require File.expand_path(f)}
+
+resque_config = YAML.load_file("#{RAILS_ROOT}/config/resque.yml")
+proxy_config = YAML.load_file("#{RAILS_ROOT}/config/proxy.yml")
+PROXY = proxy_config ? proxy_config[RAILS_ENV] : nil
+
+EM::Resque.redis = resque_config[RAILS_ENV]
+EM::Resque::WorkerMachine.new(TaskHelper.parse_opts_from_env).start
+```
+
+You can start the script with the same environment variables as with the Rake
+task.
+
+Now we have our own minimal ORM backed with em-powered mysql connection pool to
+handle our models, but there's a library in [em-synchrony][2] called
+em-activerecord which can be combined with [async mysql2][3] library to handle
+sql connections inside the EventMachine. 
+
+Configuration
+-------------
+
+You may want to change the Redis host and port Resque connects to, or
+set various other options at startup.
+
+EM::Resque has a `redis` setter which can be given a string or a Redis
+object. This means if you're already using Redis in your app, EM::Resque
+can re-use the existing connection. EM::Resque is using the non-blocking
+em-redis by default.
+
+String: `Resque.redis = 'localhost:6379'`
+
+Redis: `Resque.redis = $redis`
+
+TODO: Better configuration instructions.
+
+Namespaces
+----------
+
+If you're running multiple, separate instances of Resque you may want
+to namespace the keyspaces so they do not overlap. This is not unlike
+the approach taken by many memcached clients.
+
+This feature is provided by the [redis-namespace][rs] library, which
+Resque uses by default to separate the keys it manages from other keys
+in your Redis server.
+
+Simply use the `EM::Resque.redis.namespace` accessor:
+
+``` ruby
+EM::Resque.redis.namespace = "resque:SponsorPay"
+```
+
+We recommend sticking this in your initializer somewhere after Redis
+is configured.
+
+Contributing
+------------
+
+1. Fork EM::Resque
+2. Create a topic branch - `git checkout -b my_branch`
+3. Push to your branch - `git push origin my_branch`
+4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
+5. That's it!
+
+Meta
+----
+
+* Code: `git clone git://github.com/SponsorPay/em-resque.git`
+* Home: <http://github.com/SponsorPay/em-resque>
+* Bugs: <http://github.com/SponsorPay/em-resque/issues>
+* Gems: <http://gemcutter.org/gems/em-resque>
+
+Author
+------
+
+Julius de Bruijn :: julius.bruijn@sponsorpay.com :: @pimeys
+
+[0]: http://github.com/defunkt/resque
+[1]: http://rubyeventmachine.com/
+[2]: https://github.com/igrigorik/em-synchrony
+[3]: https://github.com/brianmario/mysql2/blob/master/lib/mysql2/em.rb

data/Rakefile

@@ -0,0 +1,77 @@
+#
+# Setup
+#
+
+$LOAD_PATH.unshift 'lib'
+require 'em-resque/tasks'
+
+def command?(command)
+  system("type #{command} > /dev/null 2>&1")
+end
+
+
+#
+# Tests
+#
+
+require 'rake/testtask'
+
+task :default => :test
+
+if command?(:rg)
+  desc "Run the test suite with rg"
+  task :test do
+    Dir['test/**/*_test.rb'].each do |f|
+      sh("rg #{f}")
+    end
+  end
+else
+  Rake::TestTask.new do |test|
+    test.libs << "test"
+    test.test_files = FileList['test/**/*_test.rb']
+  end
+end
+
+if command? :kicker
+  desc "Launch Kicker (like autotest)"
+  task :kicker do
+    puts "Kicking... (ctrl+c to cancel)"
+    exec "kicker -e rake test lib examples"
+  end
+end
+
+
+#
+# Install
+#
+
+task :install => [ 'redis:install', 'dtach:install' ]
+
+
+#
+# Documentation
+#
+
+begin
+  require 'sdoc_helpers'
+rescue LoadError
+end
+
+
+#
+# Publishing
+#
+
+desc "Push a new version to Gemcutter"
+task :publish do
+  require 'resque/version'
+
+  sh "gem build resque.gemspec"
+  sh "gem push resque-#{Resque::Version}.gem"
+  sh "git tag v#{Resque::Version}"
+  sh "git push origin v#{Resque::Version}"
+  sh "git push origin master"
+  sh "git clean -fd"
+  exec "rake pages"
+end
+

data/lib/em-resque/task_helper.rb

@@ -0,0 +1,14 @@
+class TaskHelper
+  def self.parse_opts_from_env
+    integer_keys = %w(CONCURRENCY INTERVAL FIBERS)
+    string_keys = %w(QUEUE QUEUES PIDFILE)
+    bool_keys = %w(LOGGING VERBOSE VVERBOSE)
+
+    ENV.reduce({}) do |acc, (k, v)|
+      acc = acc.merge(k.downcase.to_sym => v.to_i) if integer_keys.any?{|ik| ik == k}
+      acc = acc.merge(k.downcase.to_sym => v.to_s) if string_keys.any?{|sk| sk == k}
+      acc = acc.merge(k.downcase.to_sym => v == '1' || v.downcase == 'true') if bool_keys.any?{|bk| bk == k}
+      acc
+    end
+  end
+end

data/lib/em-resque/tasks.rb

@@ -0,0 +1,14 @@
+require 'em-synchrony'
+require 'em-resque/worker_machine'
+require 'em-resque/task_helper'
+
+namespace :em_resque do
+  task :setup
+
+  desc "Start an async Resque worker"
+  task :work => [ :setup ] do
+    require 'em-resque'
+
+    EM::Resque::WorkerMachine.new(TaskHelper.parse_opts_from_env).start
+  end
+end

data/lib/em-resque/version.rb

@@ -0,0 +1,5 @@
+module EventMachine
+  module Resque
+    Version = VERSION = '1.0.0.beta1'
+  end
+end

data/lib/em-resque/worker.rb

@@ -0,0 +1,36 @@
+require 'resque'
+
+# A non-forking version of Resque worker, which handles waiting with
+# a non-blocking version of sleep. 
+class EventMachine::Resque::Worker < Resque::Worker
+  # Overwrite system sleep with the non-blocking version
+  def sleep(interval)
+    EM::Synchrony.sleep interval
+  end
+  
+  # Be sure we're never forking
+  def fork
+    nil
+  end
+
+  # Simpler startup
+  def startup
+    register_worker
+    @cant_fork = true
+    $stdout.sync = true
+  end
+
+  # Tell Redis we've processed a job.
+  def processed!
+    Resque::Stat << "processed"
+    Resque::Stat << "processed:#{self}"
+    Resque::Stat << "processed_#{job['queue']}"
+  end
+
+  # The string representation is the same as the id for this worker instance.
+  # Can be used with Worker.find
+  def to_s
+    "#{super}:#{Fiber.current.object_id}"
+  end
+  alias_method :id, :to_s
+end

data/lib/em-resque/worker_machine.rb

@@ -0,0 +1,114 @@
+require 'em-synchrony'
+require 'em-resque'
+require 'em-resque/worker'
+
+module EventMachine
+  module Resque
+    # WorkerMachine is an EventMachine with Resque workers wrapped in Ruby
+    # fibers.
+    #
+    # An instance contains the workers and a system monitor running inside an
+    # EventMachine. The monitoring takes care of stopping the machine when all
+    # workers are shut down.
+
+    class WorkerMachine
+      # Initializes the machine, creates the fibers and workers, traps quit
+      # signals and prunes dead workers
+      #
+      # == Options
+      # concurrency::  The number of green threads inside the machine (default 20)
+      # interval::     Time in seconds how often the workers check for new work
+      #                (default 5)
+      # fibers_count:: How many fibers (and workers) to be run inside the
+      #                machine (default 1)
+      # queues::       Which queues to poll (default all)
+      # verbose::      Verbose log output (default false)
+      # vverbose::     Even more verbose log output (default false)
+      # pidfile::      The file to save the process id number
+      def initialize(opts = {})
+        @concurrency = opts[:concurrency] || 20
+        @interval = opts[:interval] || 5
+        @fibers_count = opts[:fibers] || 1
+        @queues = opts[:queue] || opts[:queues] || '*'
+        @verbose = opts[:logging] || opts[:verbose] || false
+        @very_verbose = opts[:vverbose] || false
+        @pidfile = opts[:pidfile]
+
+        raise(ArgumentError, "Should have at least one fiber") if @fibers_count.to_i < 1
+
+        build_workers
+        build_fibers
+        trap_signals
+        prune_dead_workers
+      end
+
+      # Start the machine and start polling queues.
+      def start
+        EM.synchrony do
+          @fibers.each(&:resume)
+          system_monitor.resume
+        end
+      end
+
+      # Stop the machine.
+      def stop
+        @workers.each(&:shutdown)
+        File.delete(@pidfile) if @pidfile
+      end
+
+      def fibers
+        @fibers || []
+      end
+
+      def workers
+        @workers || []
+      end
+
+      private
+
+      # Builds the workers to poll the given queues.
+      def build_workers
+        queues = @queues.to_s.split(',')
+
+        @workers = (1..@fibers_count.to_i).map do
+          worker = EM::Resque::Worker.new(*queues)
+          worker.verbose = @verbose
+          worker.very_verbose = @very_verbose
+
+          worker
+        end
+      end
+
+      # Builds the fibers to contain the built workers.
+      def build_fibers
+        @fibers = @workers.map do |worker|
+          Fiber.new do
+            worker.log "starting async worker #{worker}"
+            worker.work(@interval)
+          end
+        end
+      end
+
+      # Traps signals TERM, INT and QUIT to stop the machine.
+      def trap_signals
+        ['TERM', 'INT', 'QUIT'].each { |signal| trap(signal) { stop } }
+      end
+
+      # Deletes worker information from Redis if there's now processes for
+      # their pids.
+      def prune_dead_workers
+        @workers.first.prune_dead_workers if @workers.size > 0
+      end
+
+      # Shuts down the machine if all fibers are dead.
+      def system_monitor
+        Fiber.new do
+          loop do
+            EM.stop unless fibers.any?(&:alive?)
+            EM::Synchrony.sleep 1
+          end
+        end
+      end
+    end
+  end
+end

data/lib/em-resque.rb

@@ -0,0 +1,26 @@
+require 'resque'
+require 'em-synchrony/em-redis'
+
+module EM::Resque
+  extend Resque
+  def redis=(server)
+    case server
+    when String
+      if server =~ /redis\:\/\//
+        redis = EM::Protocols::Redis.connect(:url => server, :thread_safe => true)
+      else
+        server, namespace = server.split('/', 2)
+        host, port, db = server.split(':')
+        redis = EM::Protocols::Redis.new(:host => host, :port => port,
+                                         :thread_safe => true, :db => db)
+      end
+      namespace ||= :resque
+
+      @redis = Redis::Namespace.new(namespace, :redis => redis)
+    when Redis::Namespace
+      @redis = server
+    else
+      @redis = Redis::Namespace.new(:resque, :redis => server)
+    end
+  end
+end

data/lib/tasks/em-resque.rake

@@ -0,0 +1,2 @@
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../../lib'
+require 'em-resque/tasks'

data/test/em-resque_test.rb

@@ -0,0 +1,21 @@
+require 'test_helper'
+
+context "Resque" do
+  setup do
+    EM::Resque.redis.flushall
+  end
+
+  test "can put jobs to a queue" do
+    assert EM::Resque.enqueue(TestJob, 420, 'foo')
+  end
+
+  test "can read jobs from a queue" do
+    EM::Resque.enqueue(TestJob, 420, 'foo')
+
+    job = EM::Resque.reserve(:jobs)
+
+    assert_equal TestJob, job.payload_class
+    assert_equal 420, job.args[0]
+    assert_equal 'foo', job.args[1]
+  end
+end

data/test/redis-test-cluster.conf

@@ -0,0 +1,115 @@
+# Redis configuration file example
+
+# By default Redis does not run as a daemon. Use 'yes' if you need it.
+# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
+daemonize yes
+
+# When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
+# You can specify a custom pid file location here.
+pidfile ./test/redis-test-cluster.pid
+
+# Accept connections on the specified port, default is 6379
+port 9737
+
+# If you want you can bind a single interface, if the bind option is not
+# specified all the interfaces will listen for connections.
+#
+# bind 127.0.0.1
+
+# Close the connection after a client is idle for N seconds (0 to disable)
+timeout 300
+
+# Save the DB on disk:
+#
+#   save <seconds> <changes>
+#
+#   Will save the DB if both the given number of seconds and the given
+#   number of write operations against the DB occurred.
+#
+#   In the example below the behaviour will be to save:
+#   after 900 sec (15 min) if at least 1 key changed
+#   after 300 sec (5 min) if at least 10 keys changed
+#   after 60 sec if at least 10000 keys changed
+save 900 1
+save 300 10
+save 60 10000
+
+# The filename where to dump the DB
+dbfilename dump-cluster.rdb
+
+# For default save/load DB in/from the working directory
+# Note that you must specify a directory not a file name.
+dir ./test/
+
+# Set server verbosity to 'debug'
+# it can be one of:
+# debug (a lot of information, useful for development/testing)
+# notice (moderately verbose, what you want in production probably)
+# warning (only very important / critical messages are logged)
+loglevel debug
+
+# Specify the log file name. Also 'stdout' can be used to force
+# the demon to log on the standard output. Note that if you use standard
+# output for logging but daemonize, logs will be sent to /dev/null
+logfile stdout
+
+# Set the number of databases. The default database is DB 0, you can select
+# a different one on a per-connection basis using SELECT <dbid> where
+# dbid is a number between 0 and 'databases'-1
+databases 16
+
+################################# REPLICATION #################################
+
+# Master-Slave replication. Use slaveof to make a Redis instance a copy of
+# another Redis server. Note that the configuration is local to the slave
+# so for example it is possible to configure the slave to save the DB with a
+# different interval, or to listen to another port, and so on.
+
+# slaveof <masterip> <masterport>
+
+################################## SECURITY ###################################
+
+# Require clients to issue AUTH <PASSWORD> before processing any other
+# commands.  This might be useful in environments in which you do not trust
+# others with access to the host running redis-server.
+#
+# This should stay commented out for backward compatibility and because most
+# people do not need auth (e.g. they run their own servers).
+
+# requirepass foobared
+
+################################### LIMITS ####################################
+
+# Set the max number of connected clients at the same time. By default there
+# is no limit, and it's up to the number of file descriptors the Redis process
+# is able to open. The special value '0' means no limts.
+# Once the limit is reached Redis will close all the new connections sending
+# an error 'max number of clients reached'.
+
+# maxclients 128
+
+# Don't use more memory than the specified amount of bytes.
+# When the memory limit is reached Redis will try to remove keys with an
+# EXPIRE set. It will try to start freeing keys that are going to expire
+# in little time and preserve keys with a longer time to live.
+# Redis will also try to remove objects from free lists if possible.
+#
+# If all this fails, Redis will start to reply with errors to commands
+# that will use more memory, like SET, LPUSH, and so on, and will continue
+# to reply to most read-only commands like GET.
+#
+# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
+# 'state' server or cache, not as a real DB. When Redis is used as a real
+# database the memory usage will grow over the weeks, it will be obvious if
+# it is going to use too much memory in the long run, and you'll have the time
+# to upgrade. With maxmemory after the limit is reached you'll start to get
+# errors for write operations, and this may even lead to DB inconsistency.
+
+# maxmemory <bytes>
+
+############################### ADVANCED CONFIG ###############################
+
+# Glue small output buffers together in order to send small replies in a
+# single TCP packet. Uses a bit more CPU but most of the times it is a win
+# in terms of number of queries per second. Use 'yes' if unsure.
+glueoutputbuf yes

data/test/redis-test.conf

@@ -0,0 +1,115 @@
+# Redis configuration file example
+
+# By default Redis does not run as a daemon. Use 'yes' if you need it.
+# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
+daemonize yes
+
+# When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
+# You can specify a custom pid file location here.
+pidfile ./test/redis-test.pid
+
+# Accept connections on the specified port, default is 6379
+port 9736
+
+# If you want you can bind a single interface, if the bind option is not
+# specified all the interfaces will listen for connections.
+#
+# bind 127.0.0.1
+
+# Close the connection after a client is idle for N seconds (0 to disable)
+timeout 300
+
+# Save the DB on disk:
+#
+#   save <seconds> <changes>
+#
+#   Will save the DB if both the given number of seconds and the given
+#   number of write operations against the DB occurred.
+#
+#   In the example below the behaviour will be to save:
+#   after 900 sec (15 min) if at least 1 key changed
+#   after 300 sec (5 min) if at least 10 keys changed
+#   after 60 sec if at least 10000 keys changed
+save 900 1
+save 300 10
+save 60 10000
+
+# The filename where to dump the DB
+dbfilename dump.rdb
+
+# For default save/load DB in/from the working directory
+# Note that you must specify a directory not a file name.
+dir ./test/
+
+# Set server verbosity to 'debug'
+# it can be one of:
+# debug (a lot of information, useful for development/testing)
+# notice (moderately verbose, what you want in production probably)
+# warning (only very important / critical messages are logged)
+loglevel debug
+
+# Specify the log file name. Also 'stdout' can be used to force
+# the demon to log on the standard output. Note that if you use standard
+# output for logging but daemonize, logs will be sent to /dev/null
+logfile stdout
+
+# Set the number of databases. The default database is DB 0, you can select
+# a different one on a per-connection basis using SELECT <dbid> where
+# dbid is a number between 0 and 'databases'-1
+databases 16
+
+################################# REPLICATION #################################
+
+# Master-Slave replication. Use slaveof to make a Redis instance a copy of
+# another Redis server. Note that the configuration is local to the slave
+# so for example it is possible to configure the slave to save the DB with a
+# different interval, or to listen to another port, and so on.
+
+# slaveof <masterip> <masterport>
+
+################################## SECURITY ###################################
+
+# Require clients to issue AUTH <PASSWORD> before processing any other
+# commands.  This might be useful in environments in which you do not trust
+# others with access to the host running redis-server.
+#
+# This should stay commented out for backward compatibility and because most
+# people do not need auth (e.g. they run their own servers).
+
+# requirepass foobared
+
+################################### LIMITS ####################################
+
+# Set the max number of connected clients at the same time. By default there
+# is no limit, and it's up to the number of file descriptors the Redis process
+# is able to open. The special value '0' means no limts.
+# Once the limit is reached Redis will close all the new connections sending
+# an error 'max number of clients reached'.
+
+# maxclients 128
+
+# Don't use more memory than the specified amount of bytes.
+# When the memory limit is reached Redis will try to remove keys with an
+# EXPIRE set. It will try to start freeing keys that are going to expire
+# in little time and preserve keys with a longer time to live.
+# Redis will also try to remove objects from free lists if possible.
+#
+# If all this fails, Redis will start to reply with errors to commands
+# that will use more memory, like SET, LPUSH, and so on, and will continue
+# to reply to most read-only commands like GET.
+#
+# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
+# 'state' server or cache, not as a real DB. When Redis is used as a real
+# database the memory usage will grow over the weeks, it will be obvious if
+# it is going to use too much memory in the long run, and you'll have the time
+# to upgrade. With maxmemory after the limit is reached you'll start to get
+# errors for write operations, and this may even lead to DB inconsistency.
+
+# maxmemory <bytes>
+
+############################### ADVANCED CONFIG ###############################
+
+# Glue small output buffers together in order to send small replies in a
+# single TCP packet. Uses a bit more CPU but most of the times it is a win
+# in terms of number of queries per second. Use 'yes' if unsure.
+glueoutputbuf yes

data/test/task_helpers_test.rb

@@ -0,0 +1,32 @@
+require 'test_helper'
+require 'em-resque/task_helper'
+
+context "TaskHelper" do
+  setup do
+    ENV['CONCURRENCY'] = '20'
+    ENV['INTERVAL'] = '5'
+    ENV['FIBERS'] = '5'
+    ENV['QUEUE'] = 'foo'
+    ENV['QUEUES'] = 'foo, bar'
+    ENV['PIDFILE'] = '/foo/bar'
+    ENV['LOGGING'] = '1'
+    ENV['VERBOSE'] = 'true'
+    ENV['VVERBOSE'] = 'false'
+
+    @valid_opts = {
+      :concurrency => ENV['CONCURRENCY'].to_i,
+      :interval => ENV['INTERVAL'].to_i,
+      :fibers => ENV['FIBERS'].to_i,
+      :queue => ENV['QUEUE'],
+      :queues => ENV['QUEUES'],
+      :pidfile => ENV['PIDFILE'],
+      :logging => true,
+      :verbose => true,
+      :vverbose => false }
+  end
+
+  test "can parse all parameters correctly" do
+    opts = TaskHelper.parse_opts_from_env
+    @valid_opts.each {|k,v| assert_equal v, opts[k]}
+  end
+end

data/test/test_helper.rb

@@ -0,0 +1,99 @@
+require 'rubygems'
+require 'bundler'
+Bundler.setup(:default, :test)
+Bundler.require(:default, :test)
+
+dir = File.dirname(File.expand_path(__FILE__))
+$LOAD_PATH.unshift dir + '/../lib'
+$TESTING = true
+require 'test/unit'
+
+begin
+  require 'leftright'
+rescue LoadError
+end
+
+#
+# make sure we can run redis
+#
+
+if !system("which redis-server")
+  puts '', "** can't find `redis-server` in your path"
+  puts "** try running `sudo rake install`"
+  abort ''
+end
+
+
+##
+# test/spec/mini 3
+# http://gist.github.com/25455
+# chris@ozmm.org
+#
+def context(*args, &block)
+  return super unless (name = args.first) && block
+  require 'test/unit'
+  klass = Class.new(defined?(ActiveSupport::TestCase) ? ActiveSupport::TestCase : Test::Unit::TestCase) do
+    def self.test(name, &block)
+      define_method("test_#{name.gsub(/\W/,'_')}", &block) if block
+    end
+    def self.xtest(*args) end
+    def self.setup(&block) define_method(:setup, &block) end
+    def self.teardown(&block) define_method(:teardown, &block) end
+  end
+  (class << klass; self end).send(:define_method, :name) { name.gsub(/\W/,'_') }
+
+  klass.class_eval &block
+
+  # XXX: In 1.8.x, not all tests will run unless anonymous classes are kept in scope.
+  ($test_classes ||= []) << klass
+end
+
+##
+# Helper to perform job classes
+#
+module PerformJob
+  def perform_job(klass, *args)
+    resque_job = Resque::Job.new(:testqueue, 'class' => klass, 'args' => args)
+    resque_job.perform
+  end
+end
+
+#
+# fixture classes
+#
+
+class TestJob
+  @queue = 'jobs'
+
+  def self.perform(number, string)
+  end
+end
+
+class FailJob
+  @queue = 'jobs'
+
+  def self.perform(number, string)
+    raise Exception
+  end
+end
+
+def with_failure_backend(failure_backend, &block)
+  previous_backend = Resque::Failure.backend
+  Resque::Failure.backend = failure_backend
+  yield block
+ensure
+  Resque::Failure.backend = previous_backend
+end
+
+class Time
+  # Thanks, Timecop
+  class << self
+    alias_method :now_without_mock_time, :now
+
+    def now_with_mock_time
+      $fake_time || now_without_mock_time
+    end
+
+    alias_method :now, :now_with_mock_time
+  end
+end

data/test/worker_machine_test.rb

@@ -0,0 +1,19 @@
+require 'test_helper'
+require 'em-resque/worker_machine'
+
+context 'WorkerMachine' do
+  test 'should initialize itself' do
+    machine = EM::Resque::WorkerMachine.new
+
+    assert_equal 1, machine.fibers.count
+    assert_equal 1, machine.workers.count
+    assert_equal Fiber, machine.fibers.first.class
+    assert_equal EM::Resque::Worker, machine.workers.first.class
+  end
+
+  test 'should not run with under one fibers' do
+    assert_raise(ArgumentError, "Should have at least one fiber") do
+      machine = EM::Resque::WorkerMachine.new :fibers => 0
+    end
+  end
+end

data/test/worker_test.rb

@@ -0,0 +1,46 @@
+require 'test_helper'
+require 'em-resque/worker'
+
+context "Worker" do
+  setup do
+    EM::Resque.redis.flushall
+  end
+
+  test "processes jobs" do
+    EM.synchrony do
+      EM::Resque.enqueue(TestJob, 420, 'foo')
+      worker = EM::Resque::Worker.new('*')
+      worker.work(0)
+
+      assert_equal 1, EM::Resque.info[:processed]
+
+      worker.shutdown!
+      EM.stop
+    end
+  end
+
+  test "logs the processed queue" do
+    EM.synchrony do
+      EM::Resque.enqueue(TestJob, 420, 'test processed')
+      worker = EM::Resque::Worker.new('*')
+      worker.work(0)
+
+      assert_equal 1, EM::Resque.redis.get("stat:processed_jobs").to_i
+
+      worker.shutdown!
+      EM.stop
+    end
+  end
+
+  test "fails bad jobs" do
+    EM.synchrony do
+      EM::Resque.enqueue(FailJob, 420, "foo")
+      worker = EM::Resque::Worker.new('*')
+      worker.work(0)
+
+      assert_equal 1, Resque::Failure.count
+      worker.shutdown!
+      EM.stop
+    end
+  end
+end

metadata

@@ -0,0 +1,109 @@
+--- !ruby/object:Gem::Specification
+name: em-resque
+version: !ruby/object:Gem::Version
+  version: 1.0.0.beta1
+  prerelease: 6
+platform: ruby
+authors:
+- Julius de Bruijn
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-01-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: &5999720 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.19.0
+  type: :runtime
+  prerelease: false
+  version_requirements: *5999720
+- !ruby/object:Gem::Dependency
+  name: em-synchrony
+  requirement: &5999200 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: *5999200
+- !ruby/object:Gem::Dependency
+  name: em-redis
+  requirement: &5990080 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *5990080
+description: ! "    Em-resque is a version of Resque, which offers non-blocking and
+  non-forking\n    workers. The idea is to have fast as possible workers for tasks
+  with lots of\n    IO like pinging third party servers or hitting the database.\n\n
+  \   The async worker is using fibers through Synchrony library to reduce the amount\n
+  \   of callback functions. There's one fiber for worker and if one of the workers\n
+  \   is blocking, it will block all the workers at the same time.\n\n    The idea
+  to use this version and not the regular Resque is to reduce the amount\n    of SQL
+  connections for high-load services. Using one process for many workers\n    gives
+  a better control to the amount of SQL connections.\n\n    For using Resque please
+  refer the original project.\n\n    https://github.com/defunkt/resque/\n\n    The
+  library adds two rake tasks over Resque:\n\n      * resque:work_async for working
+  inside the EventMachine\n"
+email: julius.bruijn@sponsorpay.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- LICENSE
+- README.markdown
+files:
+- README.markdown
+- Rakefile
+- LICENSE
+- HISTORY.md
+- lib/em-resque/version.rb
+- lib/em-resque/worker.rb
+- lib/em-resque/tasks.rb
+- lib/em-resque/worker_machine.rb
+- lib/em-resque/task_helper.rb
+- lib/tasks/em-resque.rake
+- lib/em-resque.rb
+- test/worker_machine_test.rb
+- test/em-resque_test.rb
+- test/redis-test.conf
+- test/task_helpers_test.rb
+- test/redis-test-cluster.conf
+- test/worker_test.rb
+- test/test_helper.rb
+homepage: http://github.com/SponsorPay/em-resque
+licenses: []
+post_install_message: 
+rdoc_options:
+- --charset=UTF-8
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>'
+    - !ruby/object:Gem::Version
+      version: 1.3.1
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: Em-resque is an async non-forking version of Resque
+test_files: []