capistrano-resque 0.1.0 → 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 2e15d3f8b0d54a6296f6bf508c33dc7314f335dd
4
+ data.tar.gz: 1f59211f7df131c3ec3067d8d48915c8ca82d2ea
5
+ SHA512:
6
+ metadata.gz: 15c30dc1bb42ca0b7891732921eebb365fe4ac5698a13c747f282b3c7a9bcf4a987e7f2c4f4ddcc5c5fbdeecfecf135f8f4de413266b971254c1eae17dd491b7
7
+ data.tar.gz: a2bfd6313c338ffef06a2f9173faf76c2b9cb94a5b4eac6a04902e45fa91f0d76a5ee3b8a67124de4021189da259b3acde1049e8aa90a19db28b321944d12087
data/Changelog.md CHANGED
@@ -1,17 +1,19 @@
1
+ # Unreleased
2
+ * Added support for Capistrano 3.0
3
+ * Set MUTE environment variable for resque_scheduler
4
+ * Added a `resque_environment_task` option to load the `environment` rake task before running Resque workers
5
+ * Add a resque:scheduler:status task
6
+
1
7
  # 0.1.0
2
- Interval is configurable
3
- Fix issue where pids weren't created correctly
8
+ * Interval is configurable
9
+ * Fix issue where pids weren't created correctly
4
10
 
5
11
  # 0.0.9
6
-
7
- Start workers in threads
12
+ * Start workers in threads
8
13
 
9
14
  # 0.0.8
10
-
11
- Using stable branch of Resque, rather than the released gem, to take advantage of logging ability, losing shell redirection
12
- Using SIGQUIT to kill processes as they aren't terminating properly
13
-
15
+ * Using stable branch of Resque, rather than the released gem, to take advantage of logging ability, losing shell redirection
16
+ * Using SIGQUIT to kill processes as they aren't terminating properly
14
17
 
15
18
  # 0.0.7
16
-
17
- Different workers for different roles
19
+ * Different workers for different roles
data/README.md CHANGED
@@ -1,6 +1,13 @@
1
1
  # Capistrano Resque
2
2
 
3
- Basic tasks for putting some Resque in your Cap.
3
+ Basic tasks for putting some Resque in your Cap. This should be fully compatible with both Capistrano 2.x and 3.x,
4
+ but if you run into any issues please report them.
5
+
6
+ ### In your Gemfile:
7
+
8
+ ```
9
+ gem "capistrano-resque", "~> 0.2.0", require: false
10
+ ```
4
11
 
5
12
  ### In your Capfile:
6
13
 
@@ -8,6 +15,10 @@ Basic tasks for putting some Resque in your Cap.
8
15
  require "capistrano-resque"
9
16
  ```
10
17
 
18
+ Note: You must tell Bundler not to automatically require the file (by using `require: false`),
19
+ otherwise the gem will try to load the Capistrano tasks outside of the context of running
20
+ the `cap` command (e.g. running `rails console`).
21
+
11
22
  ### In your deploy.rb:
12
23
 
13
24
  ```
@@ -15,6 +26,13 @@ role :resque_worker, "app_domain"
15
26
  role :resque_scheduler, "app_domain"
16
27
 
17
28
  set :workers, { "my_queue_name" => 2 }
29
+
30
+ # To ensure resque can start/stop properly between deploys, you'll want to make
31
+ # sure to link the `tmp/pids` directory.
32
+ set :linked_dirs, %w(tmp/pids)
33
+
34
+ # Uncomment this line if your workers need access to the Rails environment:
35
+ # set :resque_environment_task, true
18
36
  ```
19
37
 
20
38
  You can also specify multiple queues and the number of workers
@@ -30,6 +48,48 @@ The above will start five workers in total:
30
48
  * one listening on the `search_index, cache_warming` queue
31
49
  * three listening on the `mailing` queue
32
50
 
51
+ ### Multiple Servers/Roles
52
+
53
+ You can also start up workers on multiple servers/roles:
54
+
55
+ ```
56
+ role :worker_server_A, <server-ip-A>
57
+ role :worker_servers_B_and_C, [<server-ip-B>, <server-ip-C>]
58
+
59
+ set :workers, {
60
+ worker_server_A: {
61
+ "archive" => 1,
62
+ "mailing" => 1
63
+ },
64
+ worker_servers_B_and_C: {
65
+ "search_index" => 1,
66
+ }
67
+ }
68
+ ```
69
+
70
+ The above will start four workers in total:
71
+
72
+ * one `archive` on Server A
73
+ * one `mailing` on Server A
74
+ * one `search_index` on Server B
75
+ * one `search_index` on Server C
76
+
77
+ ### Rails Environment
78
+
79
+ With Rails, Resque requires loading the Rails environment task to have access to your models, etc. (e.g. `QUEUE=* rake environment resque:work`). However, Resque is often used without Rails (and even if you are using Rails, you may not need/want to load the Rails environment). As such, the `environment` task is not automatically included.
80
+
81
+ If you would like to load the `environment` task automatically, add this to your `deploy.rb`:
82
+
83
+ ```
84
+ set :resque_environment_task, true
85
+ ```
86
+
87
+ If you would like your workers to use a different Rails environment than your actual Rails app:
88
+
89
+ ```
90
+ set :resque_rails_env, "my_resque_env"
91
+ ```
92
+
33
93
  ### The tasks
34
94
 
35
95
  Running cap -vT | grep resque should give you...
@@ -53,6 +113,7 @@ add the following line to your `deploy.rb`:
53
113
  ```
54
114
  after "deploy:restart", "resque:restart"
55
115
  ```
116
+
56
117
  ### Logging
57
118
 
58
119
  Backgrounding and logging are current sticking points. I'm using the HEAD of resque's 1-x-stable branch for the 0.0.8 release because it has some new logging functions not yet slated for a resque release.
@@ -60,7 +121,7 @@ Backgrounding and logging are current sticking points. I'm using the HEAD of res
60
121
  In your Gemfile, you will need to specify:
61
122
 
62
123
  ```
63
- gem 'resque', :git => 'git://github.com/defunkt/resque.git', :branch => '1-x-stable'
124
+ gem 'resque', :git => 'git://github.com/resque/resque.git', :branch => '1-x-stable'
64
125
  ```
65
126
 
66
127
  Also, you will need to include:
@@ -73,11 +134,21 @@ Resque.logger = Logger.new("new_resque_log_file")
73
134
 
74
135
  The chatter on: https://github.com/defunkt/resque/pull/450 gives more information. If using HEAD of this resque branch doesn't work for you, then pin to v0.0.7 of this project.
75
136
 
137
+ ### Redirecting output
138
+
139
+ Due to issues in the way Resque 1.x handles background processes, we automatically redirect stderr and stdout to `/dev/null`.
140
+
141
+ If you'd like to capture this output instead, just specify a log file:
142
+
143
+ ```ruby
144
+ set :resque_log_file, "log/resque.log"
145
+ ```
146
+
76
147
  ### Limitations
77
148
 
78
- Starting workers is done concurently via capistrano and you are limited by ssh connections limit on your server (default limit is 10)
149
+ Starting workers is done concurrently via Capistrano and you are limited by ssh connections limit on your server (default limit is 10)
79
150
 
80
- in order to use more workers please change your sshd configurtion (/etc/ssh/sshd_config)
151
+ To to use more workers, please change your sshd configuration (/etc/ssh/sshd_config)
81
152
 
82
153
  MaxStartups 100
83
154
 
@@ -1,2 +1,8 @@
1
1
  require "capistrano-resque/version"
2
- require "capistrano-resque/capistrano_integration"
2
+
3
+ if defined?(Capistrano::VERSION) && Gem::Version.new(Capistrano::VERSION).release >= Gem::Version.new("3.0.0")
4
+ load File.expand_path("../capistrano-resque/tasks/capistrano-resque.rake", __FILE__)
5
+ else
6
+ require "capistrano-resque/capistrano_integration"
7
+ end
8
+
@@ -9,6 +9,16 @@ module CapistranoResque
9
9
  _cset(:workers, {"*" => 1})
10
10
  _cset(:resque_kill_signal, "QUIT")
11
11
  _cset(:interval, "5")
12
+ _cset(:resque_environment_task, false)
13
+ _cset(:resque_log_file, "/dev/null")
14
+
15
+ def rails_env
16
+ fetch(:resque_rails_env, fetch(:rails_env, "production"))
17
+ end
18
+
19
+ def output_redirection
20
+ ">> #{fetch(:resque_log_file)} 2>> #{fetch(:resque_log_file)}"
21
+ end
12
22
 
13
23
  def workers_roles
14
24
  return workers.keys if workers.first[1].is_a? Hash
@@ -35,26 +45,41 @@ module CapistranoResque
35
45
  def start_command(queue, pid)
36
46
  "cd #{current_path} && RAILS_ENV=#{rails_env} QUEUE=\"#{queue}\" \
37
47
  PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 INTERVAL=#{interval} \
38
- #{fetch(:bundle_cmd, "bundle")} exec rake resque:work"
48
+ #{fetch(:bundle_cmd, "bundle")} exec rake \
49
+ #{"environment" if fetch(:resque_environment_task)} \
50
+ resque:work #{output_redirection}"
39
51
  end
40
52
 
41
53
  def stop_command
42
54
  "if [ -e #{current_path}/tmp/pids/resque_work_1.pid ]; then \
43
55
  for f in `ls #{current_path}/tmp/pids/resque_work*.pid`; \
44
- do #{try_sudo} kill -s #{resque_kill_signal} `cat $f` \
45
- && rm $f ;done \
56
+ do \
57
+ if kill -0 `cat $f`> /dev/null 2>&1; then \
58
+ kill -s #{resque_kill_signal} `cat $f` \
59
+ && rm $f \
60
+ ;else \
61
+ echo 'Resque was not running, cleaning up stale PID file' \
62
+ && rm $f \
63
+ ;fi \
64
+ ;done \
65
+ ;fi"
66
+ end
67
+
68
+ def status_scheduler
69
+ "if [ -e #{current_path}/tmp/pids/scheduler.pid ]; then \
70
+ ps -p $(cat #{current_path}/tmp/pids/scheduler.pid) | sed -n 2p \
46
71
  ;fi"
47
72
  end
48
73
 
49
74
  def start_scheduler(pid)
50
75
  "cd #{current_path} && RAILS_ENV=#{rails_env} \
51
- PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 \
52
- #{fetch(:bundle_cmd, "bundle")} exec rake resque:scheduler"
76
+ PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 MUTE=1 \
77
+ #{fetch(:bundle_cmd, "bundle")} exec rake resque:scheduler #{output_redirection}"
53
78
  end
54
79
 
55
80
  def stop_scheduler(pid)
56
81
  "if [ -e #{pid} ]; then \
57
- #{try_sudo} kill $(cat #{pid}) ; rm #{pid} \
82
+ kill -s #{resque_kill_signal} $(cat #{pid}) ; rm #{pid} \
58
83
  ;fi"
59
84
  end
60
85
 
@@ -99,6 +124,11 @@ module CapistranoResque
99
124
  end
100
125
 
101
126
  namespace :scheduler do
127
+ desc "See current scheduler status"
128
+ task :status, :roles => :resque_scheduler do
129
+ run(status_scheduler)
130
+ end
131
+
102
132
  desc "Starts resque scheduler with default configs"
103
133
  task :start, :roles => :resque_scheduler do
104
134
  pid = "#{current_path}/tmp/pids/scheduler.pid"
@@ -0,0 +1,141 @@
1
+ namespace :load do
2
+ task :defaults do
3
+ set :workers, {"*" => 1}
4
+ set :resque_kill_signal, "QUIT"
5
+ set :interval, "5"
6
+ set :resque_environment_task, false
7
+ set :resque_log_file, "/dev/null"
8
+ end
9
+ end
10
+
11
+ namespace :resque do
12
+ def rails_env
13
+ fetch(:resque_rails_env) ||
14
+ fetch(:rails_env) || # capistrano-rails doesn't automatically set this (yet),
15
+ fetch(:stage) # so we need to fall back to the stage.
16
+ end
17
+
18
+ def output_redirection
19
+ ">> #{fetch(:resque_log_file)} 2>> #{fetch(:resque_log_file)}"
20
+ end
21
+
22
+ def workers_roles
23
+ return fetch(:workers).keys if fetch(:workers).first[1].is_a? Hash
24
+ [:resque_worker]
25
+ end
26
+
27
+ def for_each_workers(&block)
28
+ if fetch(:workers).first[1].is_a? Hash
29
+ workers_roles.each do |role|
30
+ yield(role.to_sym, fetch(:workers)[role.to_sym])
31
+ end
32
+ else
33
+ yield(:resque_worker, fetch(:workers))
34
+ end
35
+ end
36
+
37
+ desc "See current worker status"
38
+ task :status do
39
+ on roles(*workers_roles) do
40
+ if test "[ -e #{current_path}/tmp/pids/resque_work_1.pid ]"
41
+ within current_path do
42
+ files = capture(:ls, "-1 tmp/pids/resque_work*.pid")
43
+ files.each_line do |file|
44
+ info capture(:ps, "-f -p $(cat #{file.chomp}) | sed -n 2p")
45
+ end
46
+ end
47
+ end
48
+ end
49
+ end
50
+
51
+ desc "Start Resque workers"
52
+ task :start do
53
+ for_each_workers do |role, workers|
54
+ on roles(role) do
55
+ worker_id = 1
56
+ workers.each_pair do |queue, number_of_workers|
57
+ info "Starting #{number_of_workers} worker(s) with QUEUE: #{queue}"
58
+ number_of_workers.times do
59
+ pid = "./tmp/pids/resque_work_#{worker_id}.pid"
60
+ within current_path do
61
+ execute :rake, %{RAILS_ENV=#{rails_env} QUEUE="#{queue}" PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 INTERVAL=#{fetch(:interval)} #{"environment" if fetch(:resque_environment_task)} resque:work #{output_redirection}}
62
+ end
63
+ worker_id += 1
64
+ end
65
+ end
66
+ end
67
+ end
68
+ end
69
+
70
+ # See https://github.com/defunkt/resque#signals for a descriptions of signals
71
+ # QUIT - Wait for child to finish processing then exit (graceful)
72
+ # TERM / INT - Immediately kill child then exit (stale or stuck)
73
+ # USR1 - Immediately kill child but don't exit (stale or stuck)
74
+ # USR2 - Don't start to process any new jobs (pause)
75
+ # CONT - Start to process new jobs again after a USR2 (resume)
76
+ desc "Quit running Resque workers"
77
+ task :stop do
78
+ on roles(*workers_roles) do
79
+ if test "[ -e #{current_path}/tmp/pids/resque_work_1.pid ]"
80
+ within current_path do
81
+ pids = capture(:ls, "-1 tmp/pids/resque_work*.pid").lines.map(&:chomp)
82
+ pids.each do |pid_file|
83
+ pid = capture(:cat, pid_file)
84
+ if test "kill -0 #{pid} > /dev/null 2>&1"
85
+ execute :kill, "-s #{fetch(:resque_kill_signal)} #{pid} && rm #{pid_file}"
86
+ else
87
+ info "Process #{pid} from #{pid_file} is not running, cleaning up stale PID file"
88
+ execute :rm, pid_file
89
+ end
90
+ end
91
+ end
92
+ else
93
+ info "No resque PID files found"
94
+ end
95
+ end
96
+ end
97
+
98
+ desc "Restart running Resque workers"
99
+ task :restart do
100
+ invoke "resque:stop"
101
+ invoke "resque:start"
102
+ end
103
+
104
+ namespace :scheduler do
105
+ desc "See current scheduler status"
106
+ task :status do
107
+ on roles :resque_scheduler do
108
+ pid = "#{current_path}/tmp/pids/scheduler.pid"
109
+ if test "[ -e #{pid} ]"
110
+ info capture(:ps, "-f -p $(cat #{pid}) | sed -n 2p")
111
+ end
112
+ end
113
+ end
114
+
115
+ desc "Starts resque scheduler with default configs"
116
+ task :start do
117
+ on roles :resque_scheduler do
118
+ pid = "#{current_path}/tmp/pids/scheduler.pid"
119
+ within current_path do
120
+ execute :rake, %{RAILS_ENV=#{rails_env} PIDFILE=#{pid} BACKGROUND=yes VERBOSE=1 MUTE=1 resque:scheduler #{output_redirection}}
121
+ end
122
+ end
123
+ end
124
+
125
+ desc "Stops resque scheduler"
126
+ task :stop do
127
+ on roles :resque_scheduler do
128
+ pid = "#{current_path}/tmp/pids/scheduler.pid"
129
+ if test "[ -e #{pid} ]"
130
+ execute :kill, "-s #{fetch(:resque_kill_signal)} $(cat #{pid}); rm #{pid}"
131
+ end
132
+ end
133
+ end
134
+
135
+ task :restart do
136
+ invoke "resque:scheduler:stop"
137
+ invoke "resque:scheduler:start"
138
+ end
139
+ end
140
+ end
141
+
@@ -1,5 +1,5 @@
1
1
  module CapistranoResque
2
2
  unless defined?(::CapistranoResque::VERSION)
3
- VERSION = "0.1.0".freeze
3
+ VERSION = "0.2.0".freeze
4
4
  end
5
5
  end
metadata CHANGED
@@ -1,62 +1,55 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: capistrano-resque
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
5
- prerelease:
4
+ version: 0.2.0
6
5
  platform: ruby
7
6
  authors:
8
7
  - Steven Shingler
9
8
  autorequire:
10
9
  bindir: bin
11
10
  cert_chain: []
12
- date: 2013-02-03 00:00:00.000000000 Z
11
+ date: 2014-07-25 00:00:00.000000000 Z
13
12
  dependencies:
14
13
  - !ruby/object:Gem::Dependency
15
14
  name: capistrano
16
15
  requirement: !ruby/object:Gem::Requirement
17
- none: false
18
16
  requirements:
19
- - - ! '>='
17
+ - - ">="
20
18
  - !ruby/object:Gem::Version
21
19
  version: '0'
22
20
  type: :runtime
23
21
  prerelease: false
24
22
  version_requirements: !ruby/object:Gem::Requirement
25
- none: false
26
23
  requirements:
27
- - - ! '>='
24
+ - - ">="
28
25
  - !ruby/object:Gem::Version
29
26
  version: '0'
30
27
  - !ruby/object:Gem::Dependency
31
28
  name: resque
32
29
  requirement: !ruby/object:Gem::Requirement
33
- none: false
34
30
  requirements:
35
- - - ! '>='
31
+ - - ">="
36
32
  - !ruby/object:Gem::Version
37
33
  version: '0'
38
34
  type: :runtime
39
35
  prerelease: false
40
36
  version_requirements: !ruby/object:Gem::Requirement
41
- none: false
42
37
  requirements:
43
- - - ! '>='
38
+ - - ">="
44
39
  - !ruby/object:Gem::Version
45
40
  version: '0'
46
41
  - !ruby/object:Gem::Dependency
47
42
  name: resque-scheduler
48
43
  requirement: !ruby/object:Gem::Requirement
49
- none: false
50
44
  requirements:
51
- - - ! '>='
45
+ - - ">="
52
46
  - !ruby/object:Gem::Version
53
47
  version: '0'
54
48
  type: :runtime
55
49
  prerelease: false
56
50
  version_requirements: !ruby/object:Gem::Requirement
57
- none: false
58
51
  requirements:
59
- - - ! '>='
52
+ - - ">="
60
53
  - !ruby/object:Gem::Version
61
54
  version: '0'
62
55
  description: Capistrano plugin that integrates Resque server tasks.
@@ -65,7 +58,7 @@ executables: []
65
58
  extensions: []
66
59
  extra_rdoc_files: []
67
60
  files:
68
- - .gitignore
61
+ - ".gitignore"
69
62
  - Changelog.md
70
63
  - Gemfile
71
64
  - Gemfile.lock
@@ -75,29 +68,29 @@ files:
75
68
  - capistrano-resque.gemspec
76
69
  - lib/capistrano-resque.rb
77
70
  - lib/capistrano-resque/capistrano_integration.rb
71
+ - lib/capistrano-resque/tasks/capistrano-resque.rake
78
72
  - lib/capistrano-resque/version.rb
79
73
  homepage: https://github.com/sshingler/capistrano-resque
80
74
  licenses: []
75
+ metadata: {}
81
76
  post_install_message:
82
77
  rdoc_options: []
83
78
  require_paths:
84
79
  - lib
85
80
  required_ruby_version: !ruby/object:Gem::Requirement
86
- none: false
87
81
  requirements:
88
- - - ! '>='
82
+ - - ">="
89
83
  - !ruby/object:Gem::Version
90
84
  version: '0'
91
85
  required_rubygems_version: !ruby/object:Gem::Requirement
92
- none: false
93
86
  requirements:
94
- - - ! '>='
87
+ - - ">="
95
88
  - !ruby/object:Gem::Version
96
89
  version: '0'
97
90
  requirements: []
98
91
  rubyforge_project:
99
- rubygems_version: 1.8.23
92
+ rubygems_version: 2.4.1
100
93
  signing_key:
101
- specification_version: 3
94
+ specification_version: 4
102
95
  summary: Resque integration for Capistrano
103
96
  test_files: []