capistrano-resque 0.1.0 → 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2e15d3f8b0d54a6296f6bf508c33dc7314f335dd
+  data.tar.gz: 1f59211f7df131c3ec3067d8d48915c8ca82d2ea
+SHA512:
+  metadata.gz: 15c30dc1bb42ca0b7891732921eebb365fe4ac5698a13c747f282b3c7a9bcf4a987e7f2c4f4ddcc5c5fbdeecfecf135f8f4de413266b971254c1eae17dd491b7
+  data.tar.gz: a2bfd6313c338ffef06a2f9173faf76c2b9cb94a5b4eac6a04902e45fa91f0d76a5ee3b8a67124de4021189da259b3acde1049e8aa90a19db28b321944d12087

data/Changelog.md

@@ -1,17 +1,19 @@
+# Unreleased
+* Added support for Capistrano 3.0
+* Set MUTE environment variable for resque_scheduler
+* Added a `resque_environment_task` option to load the `environment` rake task before running Resque workers
+* Add a resque:scheduler:status task
+
 # 0.1.0
-Interval is configurable
-Fix issue where pids weren't created correctly
+* Interval is configurable
+* Fix issue where pids weren't created correctly
 
 # 0.0.9
-
-Start workers in threads
+* Start workers in threads
 
 # 0.0.8
-
-Using stable branch of Resque, rather than the released gem, to take advantage of logging ability, losing shell redirection
-Using SIGQUIT to kill processes as they aren't terminating properly
-
+* Using stable branch of Resque, rather than the released gem, to take advantage of logging ability, losing shell redirection
+* Using SIGQUIT to kill processes as they aren't terminating properly
 
 # 0.0.7
-
-Different workers for different roles
+* Different workers for different roles

data/README.md

@@ -1,6 +1,13 @@
 # Capistrano Resque
 
-Basic tasks for putting some Resque in your Cap.
+Basic tasks for putting some Resque in your Cap. This should be fully compatible with both Capistrano 2.x and 3.x,
+but if you run into any issues please report them.
+
+### In your Gemfile:
+
+```
+gem "capistrano-resque", "~> 0.2.0", require: false
+```
 
 ### In your Capfile:
 
@@ -8,6 +15,10 @@ Basic tasks for putting some Resque in your Cap.
 require "capistrano-resque"
 ```
 
+Note: You must tell Bundler not to automatically require the file (by using `require: false`),
+otherwise the gem will try to load the Capistrano tasks outside of the context of running
+the `cap` command (e.g. running `rails console`).
+
 ### In your deploy.rb:
 
 ```
@@ -15,6 +26,13 @@ role :resque_worker, "app_domain"
 role :resque_scheduler, "app_domain"
 
 set :workers, { "my_queue_name" => 2 }
+
+# To ensure resque can start/stop properly between deploys, you'll want to make
+# sure to link the `tmp/pids` directory.
+set :linked_dirs, %w(tmp/pids)
+
+# Uncomment this line if your workers need access to the Rails environment:
+# set :resque_environment_task, true
 ```
 
 You can also specify multiple queues and the number of workers
@@ -30,6 +48,48 @@ The above will start five workers in total:
  * one listening on the `search_index, cache_warming` queue
  * three listening on the `mailing` queue
 
+### Multiple Servers/Roles
+
+You can also start up workers on multiple servers/roles:
+
+```
+role :worker_server_A,  <server-ip-A>
+role :worker_servers_B_and_C,  [<server-ip-B>, <server-ip-C>]
+
+set :workers, {
+  worker_server_A: {
+    "archive" => 1,
+    "mailing" => 1
+  },
+  worker_servers_B_and_C: {
+    "search_index" => 1,
+  }
+}
+```
+
+The above will start four workers in total:
+
+ * one `archive` on Server A
+ * one `mailing` on Server A
+ * one `search_index` on Server B
+ * one `search_index` on Server C
+
+### Rails Environment
+
+With Rails, Resque requires loading the Rails environment task to have access to your models, etc. (e.g. `QUEUE=* rake environment resque:work`). However, Resque is often used without Rails (and even if you are using Rails, you may not need/want to load the Rails environment). As such, the `environment` task is not automatically included.
+
+If you would like to load the `environment` task automatically, add this to your `deploy.rb`:
+
+```
+set :resque_environment_task, true
+```
+
+If you would like your workers to use a different Rails environment than your actual Rails app:
+
+```
+set :resque_rails_env, "my_resque_env"
+```
+
 ### The tasks
 
 Running cap -vT | grep resque should give you...
@@ -53,6 +113,7 @@ add the following line to your `deploy.rb`:
 ```
 after "deploy:restart", "resque:restart"
 ```
+
 ### Logging
 
 Backgrounding and logging are current sticking points. I'm using the HEAD of resque's 1-x-stable branch for the 0.0.8 release because it has some new logging functions not yet slated for a resque release.
@@ -60,7 +121,7 @@ Backgrounding and logging are current sticking points. I'm using the HEAD of res
 In your Gemfile, you will need to specify:
 
 ```
-gem 'resque', :git => 'git://github.com/defunkt/resque.git', :branch => '1-x-stable'
+gem 'resque', :git => 'git://github.com/resque/resque.git', :branch => '1-x-stable'
 ```
 
 Also, you will need to include:
@@ -73,11 +134,21 @@ Resque.logger = Logger.new("new_resque_log_file")
 
 The chatter on: https://github.com/defunkt/resque/pull/450 gives more information. If using HEAD of this resque branch doesn't work for you, then pin to v0.0.7 of this project.
 
+### Redirecting output
+
+Due to issues in the way Resque 1.x handles background processes, we automatically redirect stderr and stdout to `/dev/null`.
+
+If you'd like to capture this output instead, just specify a log file:
+
+```ruby
+set :resque_log_file, "log/resque.log"
+```
+
 ### Limitations
 
-Starting workers is done concurently via capistrano and you are limited by ssh connections limit on your server (default limit is 10)
+Starting workers is done concurrently via Capistrano and you are limited by ssh connections limit on your server (default limit is 10)
 
-in order to use more workers please change your sshd configurtion (/etc/ssh/sshd_config)
+To to use more workers, please change your sshd configuration (/etc/ssh/sshd_config)
 
     MaxStartups 100
 

data/lib/capistrano-resque.rb

@@ -1,2 +1,8 @@
 require "capistrano-resque/version"
-require "capistrano-resque/capistrano_integration"
+
+if defined?(Capistrano::VERSION) && Gem::Version.new(Capistrano::VERSION).release >= Gem::Version.new("3.0.0")
+  load File.expand_path("../capistrano-resque/tasks/capistrano-resque.rake", __FILE__)
+else
+  require "capistrano-resque/capistrano_integration"
+end
+

data/lib/capistrano-resque/capistrano_integration.rb

@@ -9,6 +9,16 @@ module CapistranoResque
         _cset(:workers, {"*" => 1})
         _cset(:resque_kill_signal, "QUIT")
         _cset(:interval, "5")
+        _cset(:resque_environment_task, false)
+        _cset(:resque_log_file, "/dev/null")
+
+        def rails_env
+          fetch(:resque_rails_env, fetch(:rails_env, "production"))
+        end
+
+        def output_redirection
+          ">> #{fetch(:resque_log_file)} 2>> #{fetch(:resque_log_file)}"
+        end
 
         def workers_roles
           return workers.keys if workers.first[1].is_a? Hash
@@ -35,26 +45,41 @@ module CapistranoResque
         def start_command(queue, pid)
           "cd #{current_path} && RAILS_ENV=#{rails_env} QUEUE=\"#{queue}\" \
            PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 INTERVAL=#{interval} \
-           #{fetch(:bundle_cmd, "bundle")} exec rake resque:work"
+           #{fetch(:bundle_cmd, "bundle")} exec rake \
+           #{"environment" if fetch(:resque_environment_task)} \
+           resque:work #{output_redirection}"
         end
 
         def stop_command
           "if [ -e #{current_path}/tmp/pids/resque_work_1.pid ]; then \
            for f in `ls #{current_path}/tmp/pids/resque_work*.pid`; \
-             do #{try_sudo} kill -s #{resque_kill_signal} `cat $f` \
-             && rm $f ;done \
+             do \
+               if kill -0 `cat $f`> /dev/null 2>&1; then \
+                 kill -s #{resque_kill_signal} `cat $f` \
+                 && rm $f \
+               ;else \
+                 echo 'Resque was not running, cleaning up stale PID file' \
+                 && rm $f \
+               ;fi \
+             ;done \
+           ;fi"
+        end
+
+        def status_scheduler
+          "if [ -e #{current_path}/tmp/pids/scheduler.pid ]; then \
+             ps -p $(cat #{current_path}/tmp/pids/scheduler.pid) | sed -n 2p \
            ;fi"
         end
 
         def start_scheduler(pid)
           "cd #{current_path} && RAILS_ENV=#{rails_env} \
-           PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 \
-           #{fetch(:bundle_cmd, "bundle")} exec rake resque:scheduler"
+           PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 MUTE=1 \
+           #{fetch(:bundle_cmd, "bundle")} exec rake resque:scheduler #{output_redirection}"
         end
 
         def stop_scheduler(pid)
           "if [ -e #{pid} ]; then \
-            #{try_sudo} kill $(cat #{pid}) ; rm #{pid} \
+            kill -s #{resque_kill_signal} $(cat #{pid}) ; rm #{pid} \
            ;fi"
         end
 
@@ -99,6 +124,11 @@ module CapistranoResque
           end
 
           namespace :scheduler do
+            desc "See current scheduler status"
+            task :status, :roles => :resque_scheduler do
+              run(status_scheduler)
+            end
+
             desc "Starts resque scheduler with default configs"
             task :start, :roles => :resque_scheduler do
               pid = "#{current_path}/tmp/pids/scheduler.pid"

data/lib/capistrano-resque/tasks/capistrano-resque.rake

@@ -0,0 +1,141 @@
+namespace :load do
+  task :defaults do
+    set :workers, {"*" => 1}
+    set :resque_kill_signal, "QUIT"
+    set :interval, "5"
+    set :resque_environment_task, false
+    set :resque_log_file, "/dev/null"
+  end
+end
+
+namespace :resque do
+  def rails_env
+    fetch(:resque_rails_env) ||
+      fetch(:rails_env) ||       # capistrano-rails doesn't automatically set this (yet),
+      fetch(:stage)              # so we need to fall back to the stage.
+  end
+
+  def output_redirection
+    ">> #{fetch(:resque_log_file)} 2>> #{fetch(:resque_log_file)}"
+  end
+
+  def workers_roles
+    return fetch(:workers).keys if fetch(:workers).first[1].is_a? Hash
+    [:resque_worker]
+  end
+
+  def for_each_workers(&block)
+    if fetch(:workers).first[1].is_a? Hash
+      workers_roles.each do |role|
+        yield(role.to_sym, fetch(:workers)[role.to_sym])
+      end
+    else
+      yield(:resque_worker, fetch(:workers))
+    end
+  end
+
+  desc "See current worker status"
+  task :status do
+    on roles(*workers_roles) do
+      if test "[ -e #{current_path}/tmp/pids/resque_work_1.pid ]"
+        within current_path do
+          files = capture(:ls, "-1 tmp/pids/resque_work*.pid")
+          files.each_line do |file|
+            info capture(:ps, "-f -p $(cat #{file.chomp}) | sed -n 2p")
+          end
+        end
+      end
+    end
+  end
+
+  desc "Start Resque workers"
+  task :start do
+    for_each_workers do |role, workers|
+      on roles(role) do
+        worker_id = 1
+        workers.each_pair do |queue, number_of_workers|
+          info "Starting #{number_of_workers} worker(s) with QUEUE: #{queue}"
+          number_of_workers.times do
+            pid = "./tmp/pids/resque_work_#{worker_id}.pid"
+            within current_path do
+              execute :rake, %{RAILS_ENV=#{rails_env} QUEUE="#{queue}" PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 INTERVAL=#{fetch(:interval)} #{"environment" if fetch(:resque_environment_task)} resque:work #{output_redirection}}
+            end
+            worker_id += 1
+          end
+        end
+      end
+    end
+  end
+
+  # See https://github.com/defunkt/resque#signals for a descriptions of signals
+  # QUIT - Wait for child to finish processing then exit (graceful)
+  # TERM / INT - Immediately kill child then exit (stale or stuck)
+  # USR1 - Immediately kill child but don't exit (stale or stuck)
+  # USR2 - Don't start to process any new jobs (pause)
+  # CONT - Start to process new jobs again after a USR2 (resume)
+  desc "Quit running Resque workers"
+  task :stop do
+    on roles(*workers_roles) do
+      if test "[ -e #{current_path}/tmp/pids/resque_work_1.pid ]"
+        within current_path do
+          pids = capture(:ls, "-1 tmp/pids/resque_work*.pid").lines.map(&:chomp)
+          pids.each do |pid_file|
+            pid = capture(:cat, pid_file)
+            if test "kill -0 #{pid} > /dev/null 2>&1"
+              execute :kill, "-s #{fetch(:resque_kill_signal)} #{pid} && rm #{pid_file}"
+            else
+              info "Process #{pid} from #{pid_file} is not running, cleaning up stale PID file"
+              execute :rm, pid_file
+            end
+          end
+        end
+      else
+        info "No resque PID files found"
+      end
+    end
+  end
+
+  desc "Restart running Resque workers"
+  task :restart do
+    invoke "resque:stop"
+    invoke "resque:start"
+  end
+
+  namespace :scheduler do
+    desc "See current scheduler status"
+    task :status do
+      on roles :resque_scheduler do
+        pid = "#{current_path}/tmp/pids/scheduler.pid"
+        if test "[ -e #{pid} ]"
+          info capture(:ps, "-f -p $(cat #{pid}) | sed -n 2p")
+        end
+      end
+    end
+
+    desc "Starts resque scheduler with default configs"
+    task :start do
+      on roles :resque_scheduler do
+        pid = "#{current_path}/tmp/pids/scheduler.pid"
+        within current_path do
+          execute :rake, %{RAILS_ENV=#{rails_env} PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 MUTE=1 resque:scheduler #{output_redirection}}
+        end
+      end
+    end
+
+    desc "Stops resque scheduler"
+    task :stop do
+      on roles :resque_scheduler do
+        pid = "#{current_path}/tmp/pids/scheduler.pid"
+        if test "[ -e #{pid} ]"
+          execute :kill, "-s #{fetch(:resque_kill_signal)} $(cat #{pid}); rm #{pid}"
+        end
+      end
+    end
+
+    task :restart do
+      invoke "resque:scheduler:stop"
+      invoke "resque:scheduler:start"
+    end
+  end
+end
+

data/lib/capistrano-resque/version.rb

@@ -1,5 +1,5 @@
 module CapistranoResque
   unless defined?(::CapistranoResque::VERSION)
-    VERSION = "0.1.0".freeze
+    VERSION = "0.2.0".freeze
   end
 end

metadata

@@ -1,62 +1,55 @@
 --- !ruby/object:Gem::Specification
 name: capistrano-resque
 version: !ruby/object:Gem::Version
-  version: 0.1.0
-  prerelease: 
+  version: 0.2.0
 platform: ruby
 authors:
 - Steven Shingler
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-02-03 00:00:00.000000000 Z
+date: 2014-07-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: capistrano
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: resque
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: resque-scheduler
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: Capistrano plugin that integrates Resque server tasks.
@@ -65,7 +58,7 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
+- ".gitignore"
 - Changelog.md
 - Gemfile
 - Gemfile.lock
@@ -75,29 +68,29 @@ files:
 - capistrano-resque.gemspec
 - lib/capistrano-resque.rb
 - lib/capistrano-resque/capistrano_integration.rb
+- lib/capistrano-resque/tasks/capistrano-resque.rake
 - lib/capistrano-resque/version.rb
 homepage: https://github.com/sshingler/capistrano-resque
 licenses: []
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.23
+rubygems_version: 2.4.1
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: Resque integration for Capistrano
 test_files: []