resque 1.26.pre.0 → 1.26.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f654ccc902abcf33c2340b1e7a718224404880aa
+  data.tar.gz: bb83a2fabd7698745fcde434567c85f4852b48a4
+SHA512:
+  metadata.gz: b4c51510f6ce42a7c4b2b31bda3c5758d91e1503fbdc7fac70ae976ef02aa3d519258e38da5a74def54db4487eb673df309b16b3b09b9bc9870beb844f474dd5
+  data.tar.gz: bda8063a0120e1a9e12c483b0a8c0aac1c472104f129f36d1867d2b75663ab2d712c6c965d1735edc1b3f436cc4272f9d1f9037cc3af1e5608c9ada00592949e

data/HISTORY.md

@@ -1,29 +1,42 @@
-## 1.26.0 (TBD)
-
- * FIX: use Socket.hostname instead of shelling out in order to avoid
-   errors caused when too many workers are started simultaneously. #1152
-   Backport of fix from 2.x master. (@hone, @RunsFor)
- * FIX: infinite recusion when applying a workaround to make resque
-   backwards-compatible with <= 1.24. #1150 (@yaauie)
- * FIX: add/fix test coverage for worker pruning and be a little more
-   aggressive in what we prune. (@yaauie)
- * FIX: require yaml library for show_args support in resque-web. #1143
-   (@yaauie)
- * FIX: README and documentation link fixes. #1130 (@cade)
- * Feature: implementation of Backend connection with Hash, using the same
-   connection hash supported by redis-rb #1140 (@lucidstack)
+## 1.27.0 (TBD)
+
+None yet!
+
+## 1.26.0 (2016-03-10)
+
+This changelog is a bit incomplete. We will be much stricter about the changelog for
+the next release.
+
+* rake: when BACKGROUND, ensure worker.reconnect after daemonizing (@yaauie)
+* worker: when searching workers, Worker#pid properly reflects pid of the found
+  worker instead of the current pid. (@yaauie)
+* Add support for RESQUE_PROCLINE_PREFIX environment variable to prefix
+  procline strings with a static identifier. (@rbroemeling)
+* Resque::Worker logs errors at the correct logging level (@chrisccerami)
+
+## 1.25.2 (2014-3-3)
+
+* Respect TERM_CHILD setting when not forking (@ggilder)
+* implementation of backend connection with a hash (Andrea Rossi)
+* require yaml for show_args support (@yaauie)
+* use redis-namespace 1.3 (Andrea Rossi)
+* fix DOCS link in README (@cade)
+* Fix worker prune test to actually run its assertion & cover reality. (@yaauie)
+* Eliminate infinite recursion when Resque::Helpers mixed into Resque (@yaml)
+* use ruby, avoid shelling out. google++ (@hone)
+* Failed Assertions Don't Fail Tests :rage: (@yaauie)
 
 ## 1.25.1 (2013-9-26)
 
 * Actually require Forwardable from the standard library.
 
-## 1.25.0 (TBD)
+## 1.25.0 (2013-4-16)
 * Updates fork method so [resque-multi-job-forks](https://github.com/stulentsev/resque-multi-job-forks)
   monkey patching works again. See discussion at https://github.com/defunkt/resque/pull/895 for more
   context (@jonhyman)
 * Use Redis.pipelined to group batches of redis commands.
   https://github.com/resque/resque/pull/902 (@jonhyman)
-* Fixed uninitialize constant for the module/class that contains the perform 
+* Fixed uninitialize constant for the module/class that contains the perform
   method causing job failures to no be reported, #792 (@sideshowcoder)
 * Fix Resque::Failure::Base.all to have the correct signature.
   (@rentalutions)

data/README.markdown

@@ -1,6 +1,10 @@
 Resque
 ======
 
+[![Gem Version](https://badge.fury.io/rb/resque.svg)](https://rubygems.org/gems/resque)
+[![Build Status](https://travis-ci.org/resque/resque.svg)](https://travis-ci.org/resque/resque)
+[![Coverage Status](https://coveralls.io/repos/github/resque/resque/badge.svg?branch=1-x-stable)](https://coveralls.io/r/resque/resque?branch=1-x-stable)
+
 Resque (pronounced like "rescue") is a Redis-backed library for creating
 background jobs, placing those jobs on multiple queues, and processing
 them later.
@@ -193,7 +197,18 @@ If a job raises an exception, it is logged and handed off to the
 `Resque::Failure` module. Failures are logged either locally in Redis
 or using some different backend.
 
-For example, Resque ships with Airbrake support.
+For example, Resque ships with Airbrake support. To configure it, put
+the following into an initialisation file or into your rake job:
+
+``` ruby
+# send errors which occur in background jobs to redis and airbrake
+require 'resque/failure/multiple'
+require 'resque/failure/redis'
+require 'resque/failure/airbrake'
+
+Resque::Failure::Multiple.classes = [Resque::Failure::Redis, Resque::Failure::Airbrake]
+Resque::Failure.backend = Resque::Failure::Multiple
+```
 
 Keep this in mind when writing your jobs: you may want to throw
 exceptions you would not normally throw in order to assist debugging.
@@ -258,6 +273,13 @@ variable.
 
     $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
 
+If you want Resque to log to a file, in Rails do:
+
+```ruby
+# config/initializers/resque.rb
+Resque.logger = Logger.new(Rails.root.join('log', "#{Rails.env}_resque.log"))
+```
+
 ### Process IDs (PIDs)
 
 There are scenarios where it's helpful to record the PID of a resque
@@ -429,9 +451,41 @@ then `CONT` to start it again.
 
 ### Mysql::Error: MySQL server has gone away
 
-If your workers remain idle for too long they may lose their MySQL
-connection. If that happens we recommend using [this
-Gist](http://gist.github.com/238999).
+If your workers remain idle for too long they may lose their MySQL connection. Depending on your version of Rails, we recommend the following:
+
+#### Rails 3.x
+In your `perform` method, add the following line:
+
+``` ruby
+class MyTask
+  def self.perform
+    ActiveRecord::Base.verify_active_connections!
+    # rest of your code
+  end
+end
+```
+
+The Rails doc says the following about `verify_active_connections!`:
+
+    Verify active connections and remove and disconnect connections associated with stale threads.
+
+#### Rails 4.x
+
+In your `perform` method, instead of `verify_active_connections!`, use:
+
+``` ruby
+class MyTask
+  def self.perform
+    ActiveRecord::Base.clear_active_connections!
+    # rest of your code
+  end
+end
+```
+
+From the Rails docs on [`clear_active_connections!`](http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html#method-i-clear_active_connections-21):
+
+    Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.
+
 
 
 The Front End
@@ -440,7 +494,7 @@ The Front End
 Resque comes with a Sinatra-based front end for seeing what's up with
 your queue.
 
-![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
+![The Front End](https://camo.githubusercontent.com/64d150a243987ffbc33f588bd6d7722a0bb8d69a/687474703a2f2f7475746f7269616c732e6a756d7073746172746c61622e636f6d2f696d616765732f7265737175655f6f766572766965772e706e67)
 
 ### Standalone
 
@@ -818,7 +872,7 @@ sort it out.
 Contributing
 ------------
 
-Read the [Contributing][cb] wiki page first.
+Read [CONTRIBUTING.md](CONTRIBUTING.md) first.
 
 Once you've made your great commits:
 

data/Rakefile

@@ -11,6 +11,10 @@ def command?(command)
   system("type #{command} > /dev/null 2>&1")
 end
 
+require 'rubygems'
+require 'bundler/setup'
+require 'bundler/gem_tasks'
+
 
 #
 # Tests
@@ -51,20 +55,3 @@ begin
   require 'sdoc_helpers'
 rescue LoadError
 end
-
-
-#
-# Publishing
-#
-
-desc "Push a new version to Gemcutter"
-task :publish do
-  require 'resque/version'
-
-  sh "gem build resque.gemspec"
-  sh "gem push resque-#{Resque::Version}.gem"
-  sh "git tag v#{Resque::Version}"
-  sh "git push origin v#{Resque::Version}"
-  sh "git push origin master"
-  sh "git clean -fd"
-end

data/bin/resque-web

@@ -24,4 +24,8 @@ Vegas::Runner.new(Resque::Server, 'resque-web', {
     runner.logger.info "Using Redis connection '#{redis_conf}'"
     Resque.redis = redis_conf
   }
+  opts.on('-a url-prefix', "--append url-prefix", "set reverse_proxy friendly prefix to links") {|url_prefix|
+    runner.logger.info "Using URL Prefix '#{url_prefix}'"
+    Resque::Server.url_prefix = url_prefix
+  }
 end

data/lib/resque.rb

@@ -22,6 +22,7 @@ require 'resque/plugin'
 require 'resque/vendor/utf8_util'
 
 module Resque
+  include Helpers
   extend self
 
   # Given a Ruby object, returns a string suitable for storage in a
@@ -49,20 +50,55 @@ module Resque
     end
   end
 
-  extend ::Forwardable
-
-  def self.config=(options = {})
-    @config = Config.new(options)
+  # Given a word with dashes, returns a camel cased version of it.
+  #
+  # classify('job-name') # => 'JobName'
+  def classify(dashed_word)
+    dashed_word.split('-').each { |part| part[0] = part[0].chr.upcase }.join
   end
 
-  def self.config
-    @config ||= Config.new
-  end
+  # Tries to find a constant with the name specified in the argument string:
+  #
+  # constantize("Module") # => Module
+  # constantize("Test::Unit") # => Test::Unit
+  #
+  # The name is assumed to be the one of a top-level constant, no matter
+  # whether it starts with "::" or not. No lexical context is taken into
+  # account:
+  #
+  # C = 'outside'
+  # module M
+  #   C = 'inside'
+  #   C # => 'inside'
+  #   constantize("C") # => 'outside', same as ::C
+  # end
+  #
+  # NameError is raised when the constant is unknown.
+  def constantize(camel_cased_word)
+    camel_cased_word = camel_cased_word.to_s
+
+    if camel_cased_word.include?('-')
+      camel_cased_word = classify(camel_cased_word)
+    end
+
+    names = camel_cased_word.split('::')
+    names.shift if names.empty? || names.first.empty?
 
-  def self.configure
-    yield config
+    constant = Object
+    names.each do |name|
+      args = Module.method(:const_get).arity != 1 ? [false] : []
+
+      if constant.const_defined?(name, *args)
+        constant = constant.const_get(name)
+      else
+        constant = constant.const_missing(name)
+      end
+    end
+    constant
   end
 
+  extend ::Forwardable
+
   # Accepts:
   #   1. A 'hostname:port' String
   #   2. A 'hostname:port:db' String (to select the Redis db)
@@ -116,6 +152,19 @@ module Resque
   # Set or retrieve the current logger object
   attr_accessor :logger
 
+  DEFAULT_HEARTBEAT_INTERVAL = 60
+  DEFAULT_PRUNE_INTERVAL = DEFAULT_HEARTBEAT_INTERVAL * 5
+
+  attr_writer :heartbeat_interval
+  def heartbeat_interval
+    @heartbeat_interval || DEFAULT_HEARTBEAT_INTERVAL
+  end
+
+  attr_writer :prune_interval
+  def prune_interval
+    @prune_interval || DEFAULT_PRUNE_INTERVAL
+  end
+
   # The `before_first_fork` hook will be run in the **parent** process
   # only once, before forking to run the first job. Be careful- any
   # changes you make will be permanent for the lifespan of the
@@ -168,8 +217,10 @@ module Resque
     block ? register_hook(:before_pause, block) : hooks(:before_pause)
   end
 
-  # Set the after_pause proc.
-  attr_writer :before_pause
+  # Register a before_pause proc.
+  def before_pause=(block)
+    register_hook(:before_pause, block)
+  end
 
   # The `after_pause` hook will be run in the parent process after the
   # worker has paused (via SIGCONT).
@@ -177,8 +228,10 @@ module Resque
     block ? register_hook(:after_pause, block) : hooks(:after_pause)
   end
 
-  # Set the after_continue proc.
-  attr_writer :after_pause
+  # Register an after_pause proc.
+  def after_pause=(block)
+    register_hook(:after_pause, block)
+  end
 
   def to_s
     "Resque Client connected to #{redis_id}"
@@ -211,9 +264,10 @@ module Resque
   #
   # Returns nothing
   def push(queue, item)
+    encoded = encode(item)
     redis.pipelined do
       watch_queue(queue)
-      redis.rpush "queue:#{queue}", encode(item)
+      redis.rpush "queue:#{queue}", encoded
     end
   end
 
@@ -391,7 +445,7 @@ module Resque
     queue ||= queue_from_class(klass)
 
     if !queue
-      raise NoQueueError.new("Jobs must be placed onto a queue.")
+      raise NoQueueError.new("Jobs must be placed onto a queue. No queue could be inferred for class #{klass}")
     end
 
     if klass.to_s.empty?
@@ -428,7 +482,7 @@ module Resque
   # Returns a hash, similar to redis-rb's #info, of interesting stats.
   def info
     return {
-      :pending   => queues.inject(0) { |m,k| m + size(k) },
+      :pending   => queue_sizes.inject(0) { |sum, (queue_name, queue_size)| sum + queue_size },
       :processed => Stat[:processed],
       :queues    => queues.size,
       :workers   => workers.size.to_i,
@@ -447,6 +501,47 @@ module Resque
     end
   end
 
+  # Returns a hash, mapping queue names to queue sizes
+  def queue_sizes
+    queue_names = queues
+
+    sizes = redis.pipelined do
+      queue_names.each do |name|
+        redis.llen("queue:#{name}")
+      end
+    end
+
+    Hash[queue_names.zip(sizes)]
+  end
+
+  # Returns a hash, mapping queue names to (up to `sample_size`) samples of jobs in that queue
+  def sample_queues(sample_size = 1000)
+    queue_names = queues
+
+    samples = redis.pipelined do
+      queue_names.each do |name|
+        key = "queue:#{name}"
+        redis.llen(key)
+        redis.lrange(key, 0, sample_size - 1)
+      end
+    end
+
+    hash = {}
+
+    queue_names.zip(samples.each_slice(2).to_a) do |queue_name, (queue_size, serialized_samples)|
+      samples = serialized_samples.map do |serialized_sample|
+        Job.decode(serialized_sample)
+      end
+
+      hash[queue_name] = {
+        :size => queue_size,
+        :samples => samples
+      }
+    end
+
+    hash
+  end
+
   private
 
   # Register a new proc as a hook. If the block is nil this is the
@@ -468,6 +563,11 @@ module Resque
     @hooks && @hooks[name] = []
   end
 
+  # Retrieve all hooks
+  def hooks
+    @hooks || {}
+  end
+
   # Retrieve all hooks of a given name.
   def hooks(name)
     (@hooks && @hooks[name]) || []

data/lib/resque/errors.rb

@@ -7,6 +7,7 @@ module Resque
 
   # Raised when a worker was killed while processing a job.
   class DirtyExit < RuntimeError; end
+  class PruneDeadWorkerDirtyExit < DirtyExit; end
 
   # Raised when child process is TERM'd so job can rescue this to do shutdown work.
   class TermException < SignalException; end

data/lib/resque/failure.rb

@@ -90,24 +90,30 @@ module Resque
       backend.clear(queue)
     end
 
-    def self.requeue(id)
-      backend.requeue(id)
+    def self.requeue(id, queue = nil)
+      backend.requeue(id, queue)
     end
 
-    def self.remove(id)
-      backend.remove(id)
+    def self.remove(id, queue = nil)
+      backend.remove(id, queue)
     end
-    
+
     # Requeues all failed jobs in a specific queue.
     # Queue name should be a string.
     def self.requeue_queue(queue)
       backend.requeue_queue(queue)
     end
 
+    # Requeues all failed jobs
+    def self.requeue_all
+      backend.requeue_all
+    end
+
     # Removes all failed jobs in a specific queue.
     # Queue name should be a string.
     def self.remove_queue(queue)
       backend.remove_queue(queue)
     end
+
   end
 end