sidekiq 0.11.1 → 0.11.2

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

data/Changes.md CHANGED
@@ -1,4 +1,15 @@
1
- HEAD
1
+ 0.11.2
2
+ -----------
3
+
4
+ - Implement "safe shutdown". The messages for any workers that
5
+ are stil busy when we hit the TERM timeout will be requeued in
6
+ Redis so the messages are not lost when the Sidekiq process exits.
7
+ [#110]
8
+ - Work around Celluloid's small 4kb stack limit [#115]
9
+ - Add support for a custom Capistrano role to limit Sidekiq to
10
+ a set of machines. [#113]
11
+
12
+ 0.11.1
2
13
  -----------
3
14
 
4
15
  - Fix fetch breaking retry when used with Redis namespaces. [#109]
@@ -13,10 +24,7 @@ HEAD
13
24
  As a side effect of this change, the client API works on Ruby 1.8.
14
25
  It's not officially supported but should work [#103]
15
26
  - NO POLL! Sidekiq no longer polls Redis, leading to lower network
16
- utilization and lower latency for message processing. As a side
17
- effect of this change, queue weights are no longer supported. If you
18
- wish to process multiple queues, list them in the order you want
19
- them processed: `sidekiq -q critical -q high -q default -q low`
27
+ utilization and lower latency for message processing.
20
28
  - Add --version CLI option
21
29
 
22
30
  0.10.1
data/README.md CHANGED
@@ -3,26 +3,28 @@ Sidekiq
3
3
 
4
4
  Simple, efficient message processing for Ruby.
5
5
 
6
+ Sidekiq uses threads to handle many messages at the same time in the
7
+ same process. It integrates tightly with Rails 3 to make background
8
+ message processing dead simple.
9
+
6
10
  Sidekiq is compatible with Resque. It uses the exact same
7
11
  message format as Resque so it can integrate into an existing Resque processing farm.
8
12
  You can have Sidekiq and Resque run side-by-side at the same time and
9
13
  use the Resque client to enqueue messages in Redis to be processed by Sidekiq.
10
14
 
11
- At the same time, Sidekiq uses multithreading so it much more memory efficient than Resque (which forks a new process for every job).
12
- You'll find that you might need 50 200MB resque processes to peg your CPU
13
- whereas one 300MB Sidekiq process will peg the same CPU and perform the
14
- same amount of work. Please see [my blog post on Resque's memory
15
+ At the same time, Sidekiq uses multithreading so it much more memory efficient than
16
+ Resque (which forks a new process for every job). You'll find that you might need
17
+ 50 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg
18
+ the same CPU and perform the same amount of work. Please see [my blog post on Resque's memory
15
19
  efficiency](http://blog.carbonfive.com/2011/09/16/improving-resques-memory-efficiency/)
16
20
  and how I was able to shrink a Carbon Five client's resque processing farm
17
21
  from 9 machines to 1 machine.
18
22
 
19
- In sum, if your jobs are well-behaved and threadsafe, Sidekiq is probably a good replacement for Resque. If your jobs are not thread-safe or they leak memory, you may want to continue using Resque, because its forking model gives you more protection.
20
-
21
23
 
22
24
  Requirements
23
25
  -----------------
24
26
 
25
- I test on Ruby 1.9.3 and JRuby 1.6.5 in 1.9 mode. Other versions/VMs are
27
+ I test on Ruby 1.9.3 and JRuby 1.6.x in 1.9 mode. Other versions/VMs are
26
28
  untested but I will do my best to support them. Ruby 1.8 is not supported.
27
29
 
28
30
 
@@ -57,4 +59,3 @@ Author
57
59
  -----------------
58
60
 
59
61
  Mike Perham, [@mperham](https://twitter.com/mperham), [http://mikeperham.com](http://mikeperham.com)
60
-
@@ -15,7 +15,7 @@ module Sidekiq
15
15
  :concurrency => 25,
16
16
  :require => '.',
17
17
  :environment => nil,
18
- :timeout => 5,
18
+ :timeout => 8,
19
19
  :enable_rails_extensions => true,
20
20
  }
21
21
 
@@ -3,27 +3,28 @@ Capistrano::Configuration.instance.load do
3
3
  after "deploy", "sidekiq:restart"
4
4
 
5
5
  _cset(:sidekiq_timeout) { 10 }
6
+ _cset(:sidekiq_role) { :app }
6
7
 
7
8
  namespace :sidekiq do
8
9
 
9
10
  desc "Quiet sidekiq (stop accepting new work)"
10
- task :quiet do
11
+ task :quiet, :roles => lambda { fetch(:sidekiq_role) } do
11
12
  run "cd #{current_path} && if [ -f #{current_path}/tmp/pids/sidekiq.pid ]; then bundle exec sidekiqctl quiet #{current_path}/tmp/pids/sidekiq.pid ; fi"
12
13
  end
13
14
 
14
15
  desc "Stop sidekiq"
15
- task :stop do
16
+ task :stop, :roles => lambda { fetch(:sidekiq_role) } do
16
17
  run "cd #{current_path} && if [ -f #{current_path}/tmp/pids/sidekiq.pid ]; then bundle exec sidekiqctl stop #{current_path}/tmp/pids/sidekiq.pid #{fetch :sidekiq_timeout} ; fi"
17
18
  end
18
19
 
19
20
  desc "Start sidekiq"
20
- task :start do
21
+ task :start, :roles => lambda { fetch(:sidekiq_role) } do
21
22
  rails_env = fetch(:rails_env, "production")
22
23
  run "cd #{current_path} ; nohup bundle exec sidekiq -e #{rails_env} -C #{current_path}/config/sidekiq.yml -P #{current_path}/tmp/pids/sidekiq.pid >> #{current_path}/log/sidekiq.log 2>&1 &"
23
24
  end
24
25
 
25
26
  desc "Restart sidekiq"
26
- task :restart do
27
+ task :restart, :roles => lambda { fetch(:sidekiq_role) } do
27
28
  stop
28
29
  start
29
30
  end
@@ -67,6 +67,7 @@ module Sidekiq
67
67
  poller.terminate
68
68
  @manager.stop!(:shutdown => true, :timeout => options[:timeout])
69
69
  @manager.wait(:shutdown)
70
+ exit(0)
70
71
  end
71
72
  end
72
73
 
@@ -16,7 +16,7 @@ module Sidekiq
16
16
  def initialize(mgr, queues)
17
17
  @mgr = mgr
18
18
  @queues = queues.map { |q| "queue:#{q}" }
19
- @num_queues = queues.uniq.size
19
+ @unique_queues = @queues.uniq
20
20
  end
21
21
 
22
22
  # Fetching is straightforward: the Manager makes a fetch
@@ -49,9 +49,9 @@ module Sidekiq
49
49
  # recreate the queue command each time we invoke Redis#blpop
50
50
  # to honor weights and avoid queue starvation.
51
51
  def queues_cmd
52
- cmd = @queues.sample(@num_queues)
53
- cmd << TIMEOUT
54
- cmd
52
+ queues = @queues.sample(@unique_queues.size).uniq
53
+ queues.concat(@unique_queues - queues)
54
+ queues << TIMEOUT
55
55
  end
56
56
  end
57
57
  end
@@ -25,6 +25,7 @@ module Sidekiq
25
25
  @count = options[:concurrency] || 25
26
26
  @done_callback = nil
27
27
 
28
+ @in_progress = {}
28
29
  @done = false
29
30
  @busy = []
30
31
  @fetcher = Fetcher.new(current_actor, options[:queues])
@@ -39,29 +40,22 @@ module Sidekiq
39
40
  @done = true
40
41
 
41
42
  @fetcher.terminate if @fetcher.alive?
43
+
44
+ logger.info { "Shutting down #{@ready.size} quiet workers" }
42
45
  @ready.each { |x| x.terminate if x.alive? }
43
46
  @ready.clear
44
47
 
45
- redis do |conn|
48
+ logger.debug { "Clearing workers in redis" }
49
+ Sidekiq.redis do |conn|
46
50
  workers = conn.smembers('workers')
47
51
  workers.each do |name|
48
52
  conn.srem('workers', name) if name =~ /:#{process_id}-/
49
53
  end
50
54
  end
51
55
 
52
- if shutdown
53
- if @busy.empty?
54
- # after(0) needed to avoid deadlock in Celluoid after USR1 + TERM
55
- return after(0) { signal(:shutdown) }
56
- else
57
- logger.info { "Pausing #{timeout} seconds to allow workers to finish..." }
58
- end
59
-
60
- after(timeout) do
61
- @busy.each { |x| x.terminate if x.alive? }
62
- signal(:shutdown)
63
- end
64
- end
56
+ return after(0) { signal(:shutdown) } if @busy.empty?
57
+ logger.info { "Pausing up to #{timeout} seconds to allow workers to finish..." }
58
+ hard_shutdown_in(timeout) if shutdown
65
59
  end
66
60
  end
67
61
 
@@ -76,6 +70,7 @@ module Sidekiq
76
70
  def processor_done(processor)
77
71
  watchdog('Manager#processor_done died') do
78
72
  @done_callback.call(processor) if @done_callback
73
+ @in_progress.delete(processor.object_id)
79
74
  @busy.delete(processor)
80
75
  if stopped?
81
76
  processor.terminate if processor.alive?
@@ -89,6 +84,7 @@ module Sidekiq
89
84
 
90
85
  def processor_died(processor, reason)
91
86
  watchdog("Manager#processor_died died") do
87
+ @in_progress.delete(processor.object_id)
92
88
  @busy.delete(processor)
93
89
 
94
90
  unless stopped?
@@ -103,6 +99,7 @@ module Sidekiq
103
99
  def assign(msg, queue)
104
100
  watchdog("Manager#assign died") do
105
101
  processor = @ready.pop
102
+ @in_progress[processor.object_id] = [msg, queue]
106
103
  @busy << processor
107
104
  processor.process!(MultiJson.decode(msg), queue)
108
105
  end
@@ -110,6 +107,29 @@ module Sidekiq
110
107
 
111
108
  private
112
109
 
110
+ def hard_shutdown_in(delay)
111
+ watchdog("Manager#watch_for_shutdown died") do
112
+ after(delay) do
113
+ # We've reached the timeout and we still have busy workers.
114
+ # They must die but their messages shall live on.
115
+ logger.info("Still waiting for #{@busy.size} busy workers")
116
+
117
+ Sidekiq.redis do |conn|
118
+ @busy.each do |processor|
119
+ # processor is an actor proxy and we can't call any methods
120
+ # that would go to the actor (since it's busy). Instead
121
+ # we'll use the object_id to track the worker's data here.
122
+ msg, queue = @in_progress[processor.object_id]
123
+ conn.lpush("queue:#{queue}", msg)
124
+ end
125
+ end
126
+ logger.info("Pushed #{@busy.size} messages back to Redis")
127
+
128
+ after(0) { signal(:shutdown) }
129
+ end
130
+ end
131
+ end
132
+
113
133
  def dispatch
114
134
  return if stopped?
115
135
  # This is a safety check to ensure we haven't leaked
@@ -30,9 +30,11 @@ module Sidekiq
30
30
  def process(msg, queue)
31
31
  klass = constantize(msg['class'])
32
32
  worker = klass.new
33
- stats(worker, msg, queue) do
34
- Sidekiq.server_middleware.invoke(worker, msg, queue) do
35
- worker.perform(*msg['args'])
33
+ defer do
34
+ stats(worker, msg, queue) do
35
+ Sidekiq.server_middleware.invoke(worker, msg, queue) do
36
+ worker.perform(*msg['args'])
37
+ end
36
38
  end
37
39
  end
38
40
  @boss.processor_done!(current_actor)
@@ -6,7 +6,9 @@ module Sidekiq
6
6
  class RedisConnection
7
7
  def self.create(options={})
8
8
  url = options[:url] || ENV['REDISTOGO_URL'] || 'redis://localhost:6379/0'
9
- size = (options[:size] || Sidekiq.options[:concurrency] || 25)
9
+ # need a connection for Fetcher and Retry
10
+ size = options[:size] || (Sidekiq.options[:concurrency] + 2)
11
+
10
12
  ConnectionPool.new(:timeout => 1, :size => size) do
11
13
  build_client(url, options[:namespace])
12
14
  end
@@ -19,6 +19,10 @@ module Sidekiq
19
19
  # assert_equal 1, HardWorker.jobs.size
20
20
  # assert_equal :something, HardWorker.jobs[0]['args'][0]
21
21
  #
22
+ # assert_equal 0, Sidekiq::Extensions::DelayedMailer.jobs.size
23
+ # MyMailer.delayed.send_welcome_email('foo@example.com')
24
+ # assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs.size
25
+ #
22
26
  module ClassMethods
23
27
  alias_method :perform_async_old, :perform_async
24
28
  def perform_async(*args)
@@ -1,3 +1,3 @@
1
1
  module Sidekiq
2
- VERSION = "0.11.1"
2
+ VERSION = "0.11.2"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.11.1
4
+ version: 0.11.2
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,11 +9,11 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2012-04-04 00:00:00.000000000 Z
12
+ date: 2012-04-07 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: redis
16
- requirement: &70115690910620 !ruby/object:Gem::Requirement
16
+ requirement: &70143968211720 !ruby/object:Gem::Requirement
17
17
  none: false
18
18
  requirements:
19
19
  - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
21
21
  version: '0'
22
22
  type: :runtime
23
23
  prerelease: false
24
- version_requirements: *70115690910620
24
+ version_requirements: *70143968211720
25
25
  - !ruby/object:Gem::Dependency
26
26
  name: redis-namespace
27
- requirement: &70115690910180 !ruby/object:Gem::Requirement
27
+ requirement: &70143968227640 !ruby/object:Gem::Requirement
28
28
  none: false
29
29
  requirements:
30
30
  - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
32
32
  version: '0'
33
33
  type: :runtime
34
34
  prerelease: false
35
- version_requirements: *70115690910180
35
+ version_requirements: *70143968227640
36
36
  - !ruby/object:Gem::Dependency
37
37
  name: connection_pool
38
- requirement: &70115690926040 !ruby/object:Gem::Requirement
38
+ requirement: &70143968227140 !ruby/object:Gem::Requirement
39
39
  none: false
40
40
  requirements:
41
41
  - - ~>
@@ -43,10 +43,10 @@ dependencies:
43
43
  version: 0.9.0
44
44
  type: :runtime
45
45
  prerelease: false
46
- version_requirements: *70115690926040
46
+ version_requirements: *70143968227140
47
47
  - !ruby/object:Gem::Dependency
48
48
  name: celluloid
49
- requirement: &70115690925540 !ruby/object:Gem::Requirement
49
+ requirement: &70143968226640 !ruby/object:Gem::Requirement
50
50
  none: false
51
51
  requirements:
52
52
  - - ~>
@@ -54,10 +54,10 @@ dependencies:
54
54
  version: 0.10.0
55
55
  type: :runtime
56
56
  prerelease: false
57
- version_requirements: *70115690925540
57
+ version_requirements: *70143968226640
58
58
  - !ruby/object:Gem::Dependency
59
59
  name: multi_json
60
- requirement: &70115690925160 !ruby/object:Gem::Requirement
60
+ requirement: &70143968226260 !ruby/object:Gem::Requirement
61
61
  none: false
62
62
  requirements:
63
63
  - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
65
65
  version: '0'
66
66
  type: :runtime
67
67
  prerelease: false
68
- version_requirements: *70115690925160
68
+ version_requirements: *70143968226260
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: minitest
71
- requirement: &70115690924700 !ruby/object:Gem::Requirement
71
+ requirement: &70143968225800 !ruby/object:Gem::Requirement
72
72
  none: false
73
73
  requirements:
74
74
  - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
76
76
  version: '0'
77
77
  type: :development
78
78
  prerelease: false
79
- version_requirements: *70115690924700
79
+ version_requirements: *70143968225800
80
80
  - !ruby/object:Gem::Dependency
81
81
  name: sinatra
82
- requirement: &70115690924280 !ruby/object:Gem::Requirement
82
+ requirement: &70143968225380 !ruby/object:Gem::Requirement
83
83
  none: false
84
84
  requirements:
85
85
  - - ! '>='
@@ -87,10 +87,10 @@ dependencies:
87
87
  version: '0'
88
88
  type: :development
89
89
  prerelease: false
90
- version_requirements: *70115690924280
90
+ version_requirements: *70143968225380
91
91
  - !ruby/object:Gem::Dependency
92
92
  name: slim
93
- requirement: &70115690923860 !ruby/object:Gem::Requirement
93
+ requirement: &70143968224960 !ruby/object:Gem::Requirement
94
94
  none: false
95
95
  requirements:
96
96
  - - ! '>='
@@ -98,10 +98,10 @@ dependencies:
98
98
  version: '0'
99
99
  type: :development
100
100
  prerelease: false
101
- version_requirements: *70115690923860
101
+ version_requirements: *70143968224960
102
102
  - !ruby/object:Gem::Dependency
103
103
  name: rake
104
- requirement: &70115690923440 !ruby/object:Gem::Requirement
104
+ requirement: &70143968224540 !ruby/object:Gem::Requirement
105
105
  none: false
106
106
  requirements:
107
107
  - - ! '>='
@@ -109,10 +109,10 @@ dependencies:
109
109
  version: '0'
110
110
  type: :development
111
111
  prerelease: false
112
- version_requirements: *70115690923440
112
+ version_requirements: *70143968224540
113
113
  - !ruby/object:Gem::Dependency
114
114
  name: actionmailer
115
- requirement: &70115690922940 !ruby/object:Gem::Requirement
115
+ requirement: &70143968224040 !ruby/object:Gem::Requirement
116
116
  none: false
117
117
  requirements:
118
118
  - - ~>
@@ -120,10 +120,10 @@ dependencies:
120
120
  version: '3'
121
121
  type: :development
122
122
  prerelease: false
123
- version_requirements: *70115690922940
123
+ version_requirements: *70143968224040
124
124
  - !ruby/object:Gem::Dependency
125
125
  name: activerecord
126
- requirement: &70115690922440 !ruby/object:Gem::Requirement
126
+ requirement: &70143968223540 !ruby/object:Gem::Requirement
127
127
  none: false
128
128
  requirements:
129
129
  - - ~>
@@ -131,7 +131,7 @@ dependencies:
131
131
  version: '3'
132
132
  type: :development
133
133
  prerelease: false
134
- version_requirements: *70115690922440
134
+ version_requirements: *70143968223540
135
135
  description: Simple, efficient message processing for Ruby
136
136
  email:
137
137
  - mperham@gmail.com