mongo-resque 1.17.1

data/HISTORY.md

@@ -0,0 +1,278 @@
+## 1.17.1 (2011-05-30)
+
+* More monit typo fixes
+* Web UI: Wrap preformatted arguments
+
+## 1.16.1 (2011-05-18)
+
+Exlusive to resque-mongo:
+* BREAKING CHANGE! Resque.mongo now expects a Mongo::DB object and nothing else
+* Minor api cleanups (removed undocumented methods to get closer to original resque)
+* Did NOT make the switch to use hoptoad_notifier for the hoptoad failure backend
+
+From original resque:
+* Added `Worker#paused?` method
+* Bugfix: Properly reseed random number generator after forking.
+* Bugfix: Monit example stdout/stderr redirection
+* Bugfix: Removing single failure now works with multiple failure backends
+* Web: 'Remove Queue' now requires confirmation
+* Web: Favicon!
+* Web Bugfix: Dates display in Safari
+* Web Bugfix: Dates display timezone
+* Web Bugfix: Race condition querying working workers
+* Web Bugfix: Fix polling /workers/all in resque-web
+
+## 1.15.0 (2011-03-18)
+
+* Fixed Sinatra 1.2 compatibility
+
+## 1.14.0 (2011-03-17)
+
+* Sleep interval can now be a float
+* Added Resque.inline to allow in-process performing of jobs (for testing)
+* Fixed tests for Ruby 1.9.2
+* Added Resque.validate(klass) to validate a Job
+* Decode errors are no longer ignored to help debugging
+* Web: Sinatra 1.2 compatibility
+* Fixed after_enqueue hook to actually run in `Resque.enqueue`
+* Fixed very_verbose timestamps to use 24 hour time (AM/PM wasn't included)
+* Fixed monit example
+* Fixed Worker#pid
+
+## 1.13.0 (2011-02-07)
+
+* Bugfix: worker pruning
+
+## 1.12.0 (2011-02-03)
+
+* Added pidfile writing from `rake resque:work`
+* Added Worker#pid method
+* Bugfix: Errors in failure backend are rescue'd
+* Bugfix: Non-working workers no longer counted in "working" count
+* Bugfix: Don't think resque-web is a worker
+
+## 1.11.0 (2010-08-23)
+
+* Web UI: Group /workers page by hostnames
+
+## 1.10.0 (2010-08-23)
+
+* Support redis:// string format in `Resque.redis=`
+* Using new cross-platform JSON gem.
+* Added `after_enqueue` plugin hook.
+* Added `shutdown?` method which can be overridden.
+* Added support for the "leftright" gem when running tests.
+* Grammarfix: In the README
+
+## 1.9.10 (2010-08-06)
+
+* Bugfix: before_fork should get passed the job
+
+## 1.9.9 (2010-07-26)
+
+* Depend on redis-namespace 0.8.0
+* Depend on json_pure instead of json (for JRuby compat)
+* Bugfix: rails_env display in stats view
+
+## 1.9.8 (2010-07-20)
+
+* Bugfix: Worker.all should never return nil
+* monit example: Fixed Syntax Error and adding environment to the rake task
+* redis rake task: Fixed typo in copy command
+
+## 1.9.7 (2010-07-09)
+
+* Improved memory usage in Job.destroy
+* redis-namespace 0.7.0 now required
+* Bugfix: Reverted $0 changes
+* Web Bugfix: Payload-less failures in the web ui work
+
+## 1.9.6 (2010-06-22)
+
+* Bugfix: Rakefile logging works the same as all the other logging
+
+## 1.9.5 (2010-06-16)
+
+* Web Bugfix: Display the configured namespace on the stats page
+* Revert Bugfix: Make ps -o more cross platform friendly
+
+## 1.9.4 (2010-06-14)
+
+* Bugfix: Multiple failure backend gets exception information when created
+
+## 1.9.3 (2010-06-14)
+
+* Bugfix: Resque#queues always returns an array
+
+## 1.9.2 (2010-06-13)
+
+* Bugfix: Worker.all returning nil fix
+* Bugfix: Make ps -o more cross platform friendly
+
+## 1.9.1 (2010-06-04)
+
+* Less strict JSON dependency
+* Included HISTORY.md in gem
+
+## 1.9.0 (2010-06-04)
+
+* Redis 2 support
+* Depend on redis-namespace 0.5.0
+* Added Resque::VERSION constant (alias of Resque::Version)
+* Bugfix: Specify JSON dependency
+* Bugfix: Hoptoad plugin now works on 1.9
+
+## 1.8.5 (2010-05-18)
+
+* Bugfix: Be more liberal in which Redis clients we accept.
+
+## 1.8.4 (2010-05-18)
+
+* Try to resolve redis-namespace dependency issue
+
+## 1.8.3 (2010-05-17)
+
+* Depend on redis-rb ~> 1.0.7
+
+## 1.8.2 (2010-05-03)
+
+* Bugfix: Include "tasks/" dir in RubyGem
+
+## 1.8.1 (2010-04-29)
+
+* Bugfix: Multiple failure backend did not support requeue-ing failed jobs
+* Bugfix: Fix /failed when error has no backtrace
+* Bugfix: Add `Redis::DistRedis` as a valid client
+
+## 1.8.0 (2010-04-07)
+
+* Jobs that never complete due to killed worker are now failed.
+* Worker "working" state is now maintained by the parent, not the child.
+* Stopped using deprecated redis.rb methods
+* `Worker.working` race condition fixed
+* `Worker#process` has been deprecated.
+* Monit example fixed
+* Redis::Client and Redis::Namespace can be passed to `Resque.redis=`
+
+## 1.7.1 (2010-04-02)
+
+* Bugfix: Make job hook execution order consistent
+* Bugfix: stdout buffering in child process
+
+## 1.7.0 (2010-03-31)
+
+* Job hooks API. See docs/HOOKS.md.
+* web: Hovering over dates shows a timestamp
+* web: AJAXify retry action for failed jobs
+* web bugfix: Fix pagination bug
+
+## 1.6.1 (2010-03-25)
+
+* Bugfix: Workers may not be clearing their state correctly on
+  shutdown
+* Added example monit config.
+* Exception class is now recorded when an error is raised in a
+  worker.
+* web: Unit tests
+* web: Show namespace in header and footer
+* web: Remove a queue
+* web: Retry failed jobs
+
+## 1.6.0 (2010-03-09)
+
+* Added `before_first_fork`, `before_fork`, and `after_fork` hooks.
+* Hoptoad: Added server_environment config setting
+* Hoptoad bugfix: Don't depend on RAILS_ROOT
+* 1.8.6 compat fixes
+
+## 1.5.2 (2010-03-03)
+
+* Bugfix: JSON check was crazy.
+
+## 1.5.1 (2010-03-03)
+
+* `Job.destroy` and `Resque.dequeue` return the # of destroyed jobs.
+* Hoptoad notifier improvements
+* Specify the namespace with `resque-web` by passing `-N namespace`
+* Bugfix: Don't crash when trying to parse invalid JSON.
+* Bugfix: Non-standard namespace support
+* Web: Red backgound for queue "failed" only shown if there are failed jobs.
+* Web bugfix: Tabs highlight properly now
+* Web bugfix: ZSET partial support in stats
+* Web bugfix: Deleting failed jobs works again
+* Web bugfix: Sets (or zsets, lists, etc) now paginate.
+
+## 1.5.0 (2010-02-17)
+
+* Version now included in procline, e.g. `resque-1.5.0: Message`
+* Web bugfix: Ignore idle works in the "working" page
+* Added `Resque::Job.destroy(queue, klass, *args)`
+* Added `Resque.dequeue(klass, *args)`
+
+## 1.4.0 (2010-02-11)
+
+* Fallback when unable to bind QUIT and USR1 for Windows and JRuby.
+* Fallback when no `Kernel.fork` is provided (for IronRuby).
+* Web: Rounded corners in Firefox
+* Cut down system calls in `Worker#prune_dead_workers`
+* Enable switching DB in a Redis server from config
+* Support USR2 and CONT to stop and start job processing.
+* Web: Add example failing job
+* Bugfix: `Worker#unregister_worker` shouldn't call `done_working`
+* Bugfix: Example god config now restarts Resque properly.
+* Multiple failure backends now permitted.
+* Hoptoad failure backend updated to new API
+
+## 1.3.1 (2010-01-11)
+
+* Vegas bugfix: Don't error without a config
+
+## 1.3.0 (2010-01-11)
+
+* Use Vegas for resque-web
+* Web Bugfix: Show proper date/time value for failed_at on Failures
+* Web Bugfix: Make the / route more flexible
+* Add Resque::Server.tabs array (so plugins can add their own tabs)
+* Start using [Semantic Versioning](http://semver.org/)
+
+## 1.2.4 (2009-12-15)
+
+* Web Bugfix: fix key links on stat page
+
+## 1.2.3 (2009-12-15)
+
+* Bugfix: Fixed `rand` seeding in child processes.
+* Bugfix: Better JSON encoding/decoding without Yajl.
+* Bugfix: Avoid `ps` flag error on Linux
+* Add `PREFIX` observance to `rake` install tasks.
+
+## 1.2.2 (2009-12-08)
+
+* Bugfix: Job equality was not properly implemented.
+
+## 1.2.1 (2009-12-07)
+
+* Added `rake resque:workers` task for starting multiple workers.
+* 1.9.x compatibility
+* Bugfix: Yajl decoder doesn't care about valid UTF-8
+* config.ru loads RESQUECONFIG if the ENV variable is set.
+* `resque-web` now sets RESQUECONFIG
+* Job objects know if they are equal.
+* Jobs can be re-queued using `Job#recreate`
+
+## 1.2.0 (2009-11-25)
+
+* If USR1 is sent and no child is found, shutdown.
+* Raise when a job class does not respond to `perform`.
+* Added `Resque.remove_queue` for deleting a queue
+
+## 1.1.0 (2009-11-04)
+
+* Bugfix: Broken ERB tag in failure UI
+* Bugfix: Save the worker's ID, not the worker itself, in the failure module
+* Redesigned the sinatra web interface
+* Added option to clear failed jobs
+
+## 1.0.0 (2009-11-03)
+
+* First release.

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Chris Wanstrath
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,76 @@
+Mongo-Resque
+============
+
+Mongo-Resque is a fork of Resque based on MongoDB instead of Redis.
+
+All work on this project is sponsored by the online video platform [Streamio](http://streamio.com).
+
+[![Streamio](http://d253c4ja9jigvu.cloudfront.net/assets/small-logo.png)](http://streamio.com)
+
+Check out the [ORIGINAL_README][0] included in this repository
+for the general Resque lowdown.
+
+What did you guys do to Resque?
+===============================
+
+Apart from transparently replacing the redis backend with mongodb
+(where each queue has a corresponding mongo collection) this fork
+also features delayed jobs if you want to schedule your jobs.
+
+Note that this fork is different from IGO's fork in that queue
+collections are namespaced with 'resque.queues.' to make it possible
+to use the same database for your application as Resque (this might
+still not be the best idea though - behold the stern warnings below).
+
+Original resque is currently using hoptoad_notifier in its Hoptoad Failure
+Backend. This fork has not implemented this change as I'm undecided wether 
+the change was for the better or not (we avoid dependency troubles this way).
+
+Delayed Jobs
+------------
+
+If your job class indicates that @delayed_jobs = true, you can queue
+delayed jobs.  These jobs will not be popped off the queue until the
+Time indicated in arg[0][:delay_until] has come.  Note that you must
+call Resque.enable_delay(:queue) before enququing any delayed jobs, to
+ensure that the performance impact on other queues is minimal.
+
+Configuration
+=============
+
+Resque.redis= has been replaced with Resque.mongo= and expects a Mongo::DB
+object as an argument.
+
+    Resque.mongo = Mongo::Connection.new.db("my_awesome_queue")
+
+Stern Warnings
+==============
+
+Sometimes, Mongo-Resque will drop a queue collection, or create some
+indexes, or otherwise manipulate its database. For this reason, it is
+STRONGLY recommended that you give it its own database in mongo.
+
+All jobs should be queued via Resque.enqueue.  All arguments passed to
+this method must be BSON-encodable. Mongo-Resque does not serialize
+your objects for you.  Arrays, Hashes, Strings, Numbers, and Times
+are all ok, so don't worry.
+
+Many of the new queue-level features require the first argument of
+your perform method to be an options hash.  In fact, if you just start
+making all your perform()s take one param, that is an options hash,
+you'll probably save yourself some pain.
+
+Mongo-Resque will not create any indexes on your queues, only on its
+meta-data.  You will need to create any indexes you want.  Normally,
+This is not a problem, because you aren't querying by keys, but you may
+want to create indexes on the class key in some circumstances.  If you 
+use the unique or delay features, you may want some additional indexes, 
+depending on the nature of your workload.  Paranoid?  Test enqueuing and 
+processing all your jobs, and run with --notablescans.  Learn the profiler,
+and use it often.
+
+Specifically, a queue with many long-delayed jobs will result in slower queue pops
+for all jobs using that queue.  Index delay_until in the case of
+thousands of delayed jobs.
+
+[0]: https://github.com/dbackeus/resque-mongo/blob/master/ORIGINAL_README.markdown

data/bin/resque

@@ -0,0 +1,67 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+require 'rubygems'
+require 'resque'
+
+def kill(worker)
+  abort "** resque kill WORKER_ID" if worker.nil?
+  pid = worker.split(':')[1].to_i
+
+  begin
+    Process.kill("KILL", pid)
+    puts "** killed #{worker}"
+  rescue Errno::ESRCH
+    puts "** worker #{worker} not running"
+  end
+
+  remove worker
+end
+
+def remove(worker)
+  abort "** resque remove WORKER_ID" if worker.nil?
+
+  Resque.remove_worker(worker)
+  puts "** removed #{worker}"
+end
+
+def list
+  if Resque.workers.any?
+    Resque.workers.each do |worker|
+      puts "#{worker} (#{worker.state})"
+    end
+  else
+    puts "None"
+  end
+end
+
+def queues
+  Resque.queues.each do |queue|
+    puts"Queue #{queue}: #{Resque.size(queue)}"
+  end
+end
+
+if (i = ARGV.index('-r')) && ARGV[i+1]
+  Resque.redis = ARGV[i+1]
+  ARGV.delete_at(i)
+  ARGV.delete_at(i+1)
+end
+
+case ARGV[0]
+when 'kill'
+  kill ARGV[1]
+when 'remove'
+  remove ARGV[1]
+when 'list'
+  list
+when 'queues'
+  queues
+else
+  puts "Usage: resque [-r redis_host:redis_port] COMMAND [option]"
+  puts
+  puts "Commands:"
+  puts "  remove WORKER   Removes a worker"
+  puts "  kill WORKER     Kills a worker"
+  puts "  list            Lists known workers"
+  puts "  queues          checks queue depth"
+end

data/bin/resque-web

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
+begin
+  require 'vegas'
+rescue LoadError
+  require 'rubygems'
+  require 'vegas'
+end  
+require 'resque/server'
+
+
+Vegas::Runner.new(Resque::Server, 'resque-web', {
+  :before_run => lambda {|v|
+    path = (ENV['RESQUECONFIG'] || v.args.first)
+    load path.to_s.strip if path
+  }
+}) do |runner, opts, app|
+  opts.on('-N NAMESPACE', "--namespace NAMESPACE", "set the Mongo database") {|namespace|
+    runner.logger.info "Using Mongo database '#{namespace}'"
+    Resque.mongo = Resque.mongo.conn.db(namespace)
+  }
+end

data/docs/HOOKS.md

@@ -0,0 +1,132 @@
+Resque Hooks
+============
+
+You can customize Resque or write plugins using its hook API. In many
+cases you can use a hook rather than mess with Resque's internals.
+
+For a list of available plugins see
+<http://wiki.github.com/defunkt/resque/plugins>.
+
+
+Worker Hooks
+------------
+
+If you wish to have a Proc called before the worker forks for the
+first time, you can add it in the initializer like so:
+
+    Resque.before_first_fork do
+      puts "Call me once before the worker forks the first time"
+    end
+
+You can also run a hook before _every_ fork:
+
+    Resque.before_fork do |job|
+      puts "Call me before the worker forks"
+    end
+
+The `before_fork` hook will be run in the **parent** process. So, be
+careful - any changes you make will be permanent for the lifespan of
+the worker.
+
+And after forking:
+
+    Resque.after_fork do |job|
+      puts "Call me after the worker forks"
+    end
+
+The `after_fork` hook will be run in the child process and is passed
+the current job. Any changes you make, therefor, will only live as
+long as the job currently being processes.
+
+All worker hooks can also be set using a setter, e.g.
+
+    Resque.after_fork = proc { puts "called" }
+
+
+Job Hooks
+---------
+
+Plugins can utilize job hooks to provide additional behavior. A job
+hook is a method name in the following format:
+
+    HOOKNAME_IDENTIFIER
+
+For example, a `before_perform` hook which adds locking may be defined
+like this:
+
+    def before_perform_with_lock(*args)
+      set_lock!
+    end
+
+Once this hook is made available to your job (either by way of
+inheritence or `extend`), it will be run before the job's `perform`
+method is called. Hooks of each type are executed in alphabetical order,
+so `before_perform_a` will always be executed before `before_perform_b`.
+An unnamed hook (`before_perform`) will be executed first.
+
+The available hooks are:
+
+* `after_enqueue`: Called with the job args after a job is placed on the queue.
+  Any exception raised propagates up to the code which queued the job.
+
+* `before_perform`: Called with the job args before perform. If it raises
+  `Resque::Job::DontPerform`, the job is aborted. If other exceptions
+  are raised, they will be propagated up the the `Resque::Failure`
+  backend.
+
+* `after_perform`: Called with the job args after it performs. Uncaught
+  exceptions will propagate up to the `Resque::Failure` backend.
+
+* `around_perform`: Called with the job args. It is expected to yield in order
+  to perform the job (but is not required to do so). It may handle exceptions
+  thrown by `perform`, but any that are not caught will propagate up to the
+  `Resque::Failure` backend.
+
+* `on_failure`: Called with the exception and job args if any exception occurs
+  while performing the job (or hooks).
+
+Hooks are easily implemented with superclasses or modules. A superclass could
+look something like this.
+
+    class LoggedJob
+      def self.before_perform_log_job(*args)
+        Logger.info "About to perform #{self} with #{args.inspect}"
+      end
+    end
+
+    class MyJob < LoggedJob
+      def self.perform(*args)
+        ...
+      end
+    end
+
+Modules are even better because jobs can use many of them.
+
+    module ScaledJob
+      def after_enqueue_scale_workers(*args)
+        Logger.info "Scaling worker count up"
+        Scaler.up! if Redis.info[:pending].to_i > 25
+      end
+    end
+
+    module LoggedJob
+      def before_perform_log_job(*args)
+        Logger.info "About to perform #{self} with #{args.inspect}"
+      end
+    end
+
+    module RetriedJob
+      def on_failure_retry(e, *args)
+        Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
+        Resque.enqueue self, *args
+      end
+    end
+
+    class MyJob
+      extend LoggedJob
+      extend RetriedJob
+      extend ScaledJob
+      def self.perform(*args)
+        ...
+      end
+    end

data/docs/PLUGINS.md

@@ -0,0 +1,93 @@
+Resque Plugins
+==============
+
+Resque encourages plugin development. For a list of available plugins,
+please see <http://wiki.github.com/defunkt/resque/plugins>.
+
+The `docs/HOOKS.md` file included with Resque documents the available
+hooks you can use to add or change Resque functionality. This document
+describes best practice for plugins themselves.
+
+
+Version
+-------
+
+Plugins should declare the major.minor version of Resque they are
+known to work with explicitly in their README.
+
+For example, if your plugin depends on features in Resque 2.1, please
+list "Depends on Resque 2.1" very prominently near the beginning of
+your README.
+
+Because Resque uses [Semantic Versioning][sv], you can safely make the
+following assumptions:
+
+* Your plugin will work with 2.2, 2.3, etc - no methods will be
+  removed or changed, only added.
+* Your plugin might not work with 3.0+, as APIs may change or be
+  removed.
+
+
+Namespace
+---------
+
+All plugins should live under the `Resque::Plugins` module to avoid
+clashing with first class Resque constants or other Ruby libraries.
+
+Good:
+
+* Resque::Plugins::Lock
+* Resque::Plugins::FastRetry
+
+Bad:
+
+* Resque::Lock
+* ResqueQueue
+
+
+Gem Name
+--------
+
+Gem names should be in the format of `resque-FEATURE`, where `FEATURE`
+succinctly describes the feature your plugin adds to Resque.
+
+Good:
+
+* resque-status
+* resque-scheduler
+
+Bad:
+
+* multi-queue
+* defunkt-resque-lock
+
+
+Hooks
+-----
+
+Job hook names should be namespaced to work properly.
+
+Good:
+
+* before_perform_lock
+* around_perform_check_status
+
+Bad:
+
+* before_perform
+* on_failure
+
+
+Lint
+----
+
+Plugins should test compliance to this document using the
+`Resque::Plugin.lint` method.
+
+For example:
+
+    assert_nothing_raised do
+      Resque::Plugin.lint(Resque::Plugins::Lock)
+    end
+
+[sv]: http://semver.org/

data/lib/resque/errors.rb

@@ -0,0 +1,13 @@
+module Resque
+  # Raised whenever we need a queue but none is provided.
+  class NoQueueError < RuntimeError; end
+
+  # Raised when trying to create a job without a class
+  class NoClassError < RuntimeError; end
+
+  # Raised when a worker was killed while processing a job.
+  class DirtyExit < RuntimeError; end
+
+  #raised for bad delay queue integrity
+  class QueueError < RuntimeError; end
+end