sidekiq-scheduler 0.0.0 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +111 -1
- data/lib/sidekiq-scheduler/cli.rb +24 -0
- data/lib/sidekiq-scheduler/client.rb +87 -0
- data/lib/sidekiq-scheduler/manager.rb +86 -0
- data/lib/sidekiq-scheduler/version.rb +1 -1
- data/lib/sidekiq-scheduler/worker.rb +18 -0
- data/lib/sidekiq-scheduler.rb +4 -2
- data/test/cli_test.rb +33 -0
- data/test/client_test.rb +118 -0
- data/test/config.yml +10 -0
- data/test/fake_env.rb +0 -0
- data/test/manager_test.rb +66 -0
- data/test/support/my_worker.rb +3 -0
- data/test/test_helper.rb +7 -6
- metadata +26 -12
- data/test/sidekiq-scheduler_test.rb +0 -7
data/README.md
CHANGED
@@ -1,3 +1,113 @@
|
|
1
1
|
# SidekiqScheduler
|
2
2
|
|
3
|
-
|
3
|
+
## Description
|
4
|
+
|
5
|
+
sidekiq-scheduler is an extension to [Sidekiq](http://github.com/mperham/sidekiq)
|
6
|
+
that adds support for queueing jobs in the future.
|
7
|
+
|
8
|
+
At the moment job scheduling is only supported in a delayed fashion. Replacing cron
|
9
|
+
is not the intention of this gem. Delayed jobs are Sidekiq jobs that you want to run
|
10
|
+
at some point in the future.
|
11
|
+
|
12
|
+
The syntax is pretty explanatory:
|
13
|
+
|
14
|
+
MyWorker.perform_in(5.days, 'arg1', 'arg2') # run a job in 5 days
|
15
|
+
# or
|
16
|
+
MyWorker.perform_at(5.days.from_now, 'arg1', 'arg2') # run job at a specific time
|
17
|
+
|
18
|
+
## Installation
|
19
|
+
|
20
|
+
# Rails 3.x: add it to your Gemfile
|
21
|
+
gem 'sidekiq-scheduler'
|
22
|
+
|
23
|
+
# Starting the scheduler
|
24
|
+
bundle exec sidekiq-scheduler
|
25
|
+
|
26
|
+
The scheduler will perform identically to a normal sidekiq worker with
|
27
|
+
an additional scheduler thread being run - in the default configuration
|
28
|
+
this will result in 25 worker threads being available on the scheduler
|
29
|
+
node but all normal configuration options apply.
|
30
|
+
|
31
|
+
NOTE: Since it's currently not possible to hook into the default option
|
32
|
+
parsing provided by sidekiq you will need to use a configuration file to
|
33
|
+
override the scheduler options. Currently the only option available is
|
34
|
+
|
35
|
+
resolution: <seconds between schedule runs>
|
36
|
+
|
37
|
+
The scheduling thread will sleep this many seconds between looking for
|
38
|
+
jobs that need moving to the worker queue. The default is 5 seconds
|
39
|
+
which should be fast enough for almost all uses.
|
40
|
+
|
41
|
+
NOTE: You DO NOT want to run more than one instance of the scheduler. Doing
|
42
|
+
so will result in the same job being queued multiple times. You only need one
|
43
|
+
instance of the scheduler running per application, regardless of number of servers.
|
44
|
+
|
45
|
+
NOTE: If the scheduler thread goes down for whatever reason, the delayed items
|
46
|
+
that should have fired during the outage will fire once the scheduler is
|
47
|
+
started back up again (even if it is on a new machine).
|
48
|
+
|
49
|
+
## Delayed jobs
|
50
|
+
|
51
|
+
Delayed jobs are one-off jobs that you want to be put into a queue at some point
|
52
|
+
in the future. The classic example is sending email:
|
53
|
+
|
54
|
+
MyWorker.perform_in(5.days, current_user.id)
|
55
|
+
|
56
|
+
This will store the job for 5 days in the Sidekiq delayed queue at which time
|
57
|
+
the scheduler will pull it from the delayed queue and put it in the appropriate
|
58
|
+
work queue for the given job. It will then be processed as soon as a worker is
|
59
|
+
available (just like any other Sidekiq job).
|
60
|
+
|
61
|
+
NOTE: The job does not fire **exactly** at the time supplied. Rather, once that
|
62
|
+
time is in the past, the job moves from the delayed queue to the actual work
|
63
|
+
queue and will be completed as workers are free to process it.
|
64
|
+
|
65
|
+
Also supported is `MyWork.perform_at` which takes a timestamp to queue the job.
|
66
|
+
|
67
|
+
The delayed queue is stored in redis and is persisted in the same way the
|
68
|
+
standard Sidekiq jobs are persisted (redis writing to disk). Delayed jobs differ
|
69
|
+
from scheduled jobs in that if your scheduler process is down or workers are
|
70
|
+
down when a particular job is supposed to be processed, they will simply "catch up"
|
71
|
+
once they are started again. Jobs are guaranteed to run (provided they make it
|
72
|
+
into the delayed queue) after their given queue_at time has passed.
|
73
|
+
|
74
|
+
One other thing to note is that insertion into the delayed queue is O(log(n))
|
75
|
+
since the jobs are stored in a redis sorted set (zset). I can't imagine this
|
76
|
+
being an issue for someone since redis is stupidly fast even at log(n), but full
|
77
|
+
disclosure is always best.
|
78
|
+
|
79
|
+
### Removing Delayed jobs
|
80
|
+
|
81
|
+
If you have the need to cancel a delayed job, you can do it like this:
|
82
|
+
|
83
|
+
# after you've enqueued a job like:
|
84
|
+
MyWorker.perform_at(5.days.from_now, 'arg1', 'arg2')
|
85
|
+
# remove the job with exactly the same parameters:
|
86
|
+
MyWorker.remove_delayed(<timestamp>, 'arg1', 'arg2')
|
87
|
+
|
88
|
+
## Note on Patches / Pull Requests
|
89
|
+
|
90
|
+
* Fork the project.
|
91
|
+
* Make your feature addition or bug fix.
|
92
|
+
* Add tests for it. This is important so I don't break it in a future version unintentionally.
|
93
|
+
* Commit, do not mess with rakefile, version, or history.
|
94
|
+
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
|
95
|
+
* Send me a pull request. Bonus points for topic branches.
|
96
|
+
|
97
|
+
## Credits
|
98
|
+
|
99
|
+
This work is a partial port of [resque-scheduler](https://github.com/bvandenbos/resque-scheduler) by Ben VandenBos.
|
100
|
+
Modified to work with the Sidekiq queueing library by Morton Jonuschat.
|
101
|
+
|
102
|
+
## Maintainers
|
103
|
+
|
104
|
+
* [Morton Jonuschat](https://github.com/yabawock)
|
105
|
+
|
106
|
+
## License
|
107
|
+
|
108
|
+
MIT License
|
109
|
+
|
110
|
+
## Copyright
|
111
|
+
|
112
|
+
Copyright 2012 Morton Jonuschat
|
113
|
+
Some parts copyright 2010 Ben VandenBos
|
@@ -0,0 +1,24 @@
|
|
1
|
+
require 'sidekiq-scheduler/manager'
|
2
|
+
require 'sidekiq/cli'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module CLI
|
6
|
+
def self.included(base)
|
7
|
+
base.class_eval do
|
8
|
+
alias_method :run_manager, :run
|
9
|
+
alias_method :run, :run_scheduler
|
10
|
+
end
|
11
|
+
end
|
12
|
+
|
13
|
+
def run_scheduler
|
14
|
+
scheduler_options = { :scheduler => true, :resolution => 5 }
|
15
|
+
scheduler_options.merge!(options)
|
16
|
+
scheduler = SidekiqScheduler::Manager.new(scheduler_options)
|
17
|
+
scheduler.start!
|
18
|
+
run_manager
|
19
|
+
scheduler.stop!
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
Sidekiq::CLI.send(:include, SidekiqScheduler::CLI)
|
@@ -0,0 +1,87 @@
|
|
1
|
+
require 'sidekiq-scheduler/worker'
|
2
|
+
require 'sidekiq/client'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module Client
|
6
|
+
# Example usage:
|
7
|
+
# Sidekiq::Client.delayed_push('my_queue', Time.now + 600, 'class' => MyWorker, 'args' => ['foo', 1, :bat => 'bar'])
|
8
|
+
def delayed_push(queue=nil, timestamp, item)
|
9
|
+
raise(ArgumentError, "Message must be a Hash of the form: { 'class' => SomeClass, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash)
|
10
|
+
raise(ArgumentError, "Message must include a class and set of arguments: #{item.inspect}") if !item['class'] || !item['args']
|
11
|
+
|
12
|
+
timestamp = timestamp.to_i
|
13
|
+
|
14
|
+
item['queue'] = queue.to_s if queue
|
15
|
+
item['class'] = item['class'].to_s if !item['class'].is_a?(String)
|
16
|
+
|
17
|
+
# Add item to the list for this timestamp
|
18
|
+
Sidekiq.redis.rpush("delayed:#{timestamp}", MultiJson.encode(item))
|
19
|
+
|
20
|
+
# Add timestamp to zset. Score and value are based on the timestamp
|
21
|
+
# as querying will be based on that
|
22
|
+
Sidekiq.redis.zadd('delayed_queue_schedule', timestamp, timestamp)
|
23
|
+
end
|
24
|
+
|
25
|
+
def remove_scheduler_queue(timestamp)
|
26
|
+
key = "delayed:#{timestamp}"
|
27
|
+
if 0 == Sidekiq.redis.llen(key)
|
28
|
+
Sidekiq.redis.del(key)
|
29
|
+
Sidekiq.redis.zrem('delayed_queue_schedule', timestamp)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
# Example usage:
|
34
|
+
# Sidekiq::Client.remove_all_delayed(MyWorker, 'foo', 1, :bat => 'bar')
|
35
|
+
#
|
36
|
+
# Returns the number of jobs removed
|
37
|
+
#
|
38
|
+
# This method can be very expensive since it needs to scan
|
39
|
+
# through the delayed queues of all timestamps
|
40
|
+
def remove_all_delayed(klass, *args)
|
41
|
+
remove_all_delayed_from_queue(nil, klass, *args)
|
42
|
+
end
|
43
|
+
|
44
|
+
# Example usage:
|
45
|
+
# Sidekiq::Client.remove_all_delayed('foo', MyWorker, 'foo', 1, :bat => 'bar')
|
46
|
+
#
|
47
|
+
# Returns the number of jobs removed
|
48
|
+
#
|
49
|
+
# This method can be very expensive since it needs to scan
|
50
|
+
# through the delayed queues of all timestamps
|
51
|
+
def remove_all_delayed_from_queue(queue, klass, *args)
|
52
|
+
count = 0
|
53
|
+
item = {'class' => klass.to_s, 'args' => args}
|
54
|
+
item['queue'] = queue.to_s if queue
|
55
|
+
search = MultiJson.encode(item)
|
56
|
+
Array(Sidekiq.redis.keys("delayed:*")).each do |key|
|
57
|
+
count += Sidekiq.redis.lrem(key, 0, search)
|
58
|
+
end
|
59
|
+
count
|
60
|
+
end
|
61
|
+
|
62
|
+
# Example usage:
|
63
|
+
# Sidekiq::Client.remove_delayed(Time.now + 600, MyWorker, 'foo', 1, :bat => 'bar')
|
64
|
+
#
|
65
|
+
# Returns the number of jobs removed
|
66
|
+
def remove_delayed(timestamp, klass, *args)
|
67
|
+
remove_delayed_from_queue(nil, timestamp, klass, *args)
|
68
|
+
end
|
69
|
+
|
70
|
+
|
71
|
+
# Example usage:
|
72
|
+
# Sidekiq::Client.remove_delayed('foo', Time.now + 600, MyWorker, 'foo', 1, :bat => 'bar')
|
73
|
+
#
|
74
|
+
# Returns the number of jobs removed
|
75
|
+
def remove_delayed_from_queue(queue, timestamp, klass, *args)
|
76
|
+
timestamp = timestamp.to_i
|
77
|
+
item = {'class' => klass.to_s, 'args' => args}
|
78
|
+
item['queue'] = queue.to_s if queue
|
79
|
+
search = MultiJson.encode(item)
|
80
|
+
count = Sidekiq.redis.lrem("delayed:#{timestamp}", 0, search)
|
81
|
+
remove_scheduler_queue(timestamp)
|
82
|
+
count
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
Sidekiq::Client.send(:extend, SidekiqScheduler::Client)
|
@@ -0,0 +1,86 @@
|
|
1
|
+
require 'celluloid'
|
2
|
+
require 'redis'
|
3
|
+
require 'multi_json'
|
4
|
+
|
5
|
+
require 'sidekiq/util'
|
6
|
+
|
7
|
+
module SidekiqScheduler
|
8
|
+
|
9
|
+
##
|
10
|
+
# The delayed job router in the system. This
|
11
|
+
# manages the scheduled jobs pushed messages
|
12
|
+
# from Redis onto the work queues
|
13
|
+
#
|
14
|
+
class Manager
|
15
|
+
include Sidekiq::Util
|
16
|
+
include Celluloid
|
17
|
+
|
18
|
+
def initialize(options={})
|
19
|
+
logger.info "Booting sidekiq scheduler #{SidekiqScheduler::VERSION} with Redis at #{redis.client.location}"
|
20
|
+
logger.debug { options.inspect }
|
21
|
+
@enabled = options[:scheduler]
|
22
|
+
@resolution = options[:resolution] || 5
|
23
|
+
end
|
24
|
+
|
25
|
+
def stop
|
26
|
+
@enabled = false
|
27
|
+
end
|
28
|
+
|
29
|
+
def start
|
30
|
+
schedule(true)
|
31
|
+
end
|
32
|
+
|
33
|
+
def reset
|
34
|
+
clear_scheduled_work
|
35
|
+
end
|
36
|
+
|
37
|
+
private
|
38
|
+
|
39
|
+
def clear_scheduled_work
|
40
|
+
queues = redis.zrange('delayed_queue_schedule', 0, -1).to_a
|
41
|
+
redis.del(*queues.map { |t| "delayed:#{t}" }) unless queues.empty?
|
42
|
+
redis.del('delayed_queue_schedule')
|
43
|
+
end
|
44
|
+
|
45
|
+
def find_scheduled_work(timestamp)
|
46
|
+
loop do
|
47
|
+
break logger.debug("Finished processing queue for timestamp #{timestamp}") unless msg = redis.lpop("delayed:#{timestamp}")
|
48
|
+
item = MultiJson.decode(msg)
|
49
|
+
queue = item.delete('queue')
|
50
|
+
Sidekiq::Client.push(queue, item)
|
51
|
+
end
|
52
|
+
Sidekiq::Client.remove_scheduler_queue(timestamp)
|
53
|
+
end
|
54
|
+
|
55
|
+
def find_next_timestamp
|
56
|
+
timestamp = redis.zrangebyscore('delayed_queue_schedule', '-inf', Time.now.to_i, :limit => [0, 1])
|
57
|
+
if timestamp.is_a?(Array)
|
58
|
+
timestamp = timestamp.first
|
59
|
+
end
|
60
|
+
timestamp.to_i unless timestamp.nil?
|
61
|
+
end
|
62
|
+
|
63
|
+
def schedule(run_loop = false)
|
64
|
+
watchdog("Fatal error in sidekiq, scheduler loop died") do
|
65
|
+
return if stopped?
|
66
|
+
|
67
|
+
# Dispatch loop
|
68
|
+
loop do
|
69
|
+
break logger.debug('no scheduler queues to process') unless timestamp = find_next_timestamp
|
70
|
+
find_scheduled_work(timestamp)
|
71
|
+
end
|
72
|
+
|
73
|
+
# This is the polling loop that ensures we check Redis every
|
74
|
+
# second for work, even if there was nothing to do this time
|
75
|
+
# around.
|
76
|
+
after(@resolution) do
|
77
|
+
schedule(run_loop)
|
78
|
+
end if run_loop
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
def stopped?
|
83
|
+
!@enabled
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
require 'sidekiq-scheduler/client'
|
2
|
+
require 'sidekiq/worker'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module Worker
|
6
|
+
module ClassMethods
|
7
|
+
def perform_at(timestamp, *args)
|
8
|
+
Sidekiq::Client.delayed_push(timestamp, 'class' => self.name, 'args' => args)
|
9
|
+
end
|
10
|
+
|
11
|
+
def perform_in(seconds_from_now, *args)
|
12
|
+
Sidekiq::Client.delayed_push(Time.now + seconds_from_now, 'class' => self.name, 'args' => args)
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
Sidekiq::Worker::ClassMethods.send(:include, SidekiqScheduler::Worker::ClassMethods)
|
data/lib/sidekiq-scheduler.rb
CHANGED
data/test/cli_test.rb
ADDED
@@ -0,0 +1,33 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'sidekiq-scheduler/cli'
|
3
|
+
require 'tempfile'
|
4
|
+
|
5
|
+
class CliTest < MiniTest::Unit::TestCase
|
6
|
+
describe 'with cli' do
|
7
|
+
before do
|
8
|
+
@cli = new_cli
|
9
|
+
end
|
10
|
+
|
11
|
+
describe 'with config file' do
|
12
|
+
before do
|
13
|
+
@cli.parse(['sidekiq', '-C', './test/config.yml'])
|
14
|
+
end
|
15
|
+
|
16
|
+
it 'sets the resolution of the scheduler timer' do
|
17
|
+
assert_equal 30, Sidekiq.options[:resolution]
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def new_cli
|
22
|
+
cli = Sidekiq::CLI.new
|
23
|
+
def cli.die(code)
|
24
|
+
@code = code
|
25
|
+
end
|
26
|
+
|
27
|
+
def cli.valid?
|
28
|
+
!@code
|
29
|
+
end
|
30
|
+
cli
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
data/test/client_test.rb
ADDED
@@ -0,0 +1,118 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'timecop'
|
3
|
+
|
4
|
+
class ClientTest < MiniTest::Unit::TestCase
|
5
|
+
describe 'with real redis' do
|
6
|
+
before do
|
7
|
+
Sidekiq.redis = { :url => 'redis://localhost/sidekiq_test' }
|
8
|
+
Sidekiq.redis.flushdb
|
9
|
+
end
|
10
|
+
|
11
|
+
it 'removes scheduled messages and returns count' do
|
12
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
13
|
+
assert_equal 1, Sidekiq::Client.remove_all_delayed('Foo', 1, 2)
|
14
|
+
end
|
15
|
+
|
16
|
+
it 'removes scheduled messages for a queue and returns count' do
|
17
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
18
|
+
assert_equal 1, Sidekiq::Client.remove_all_delayed_from_queue('foo', 'Foo', 1, 2)
|
19
|
+
end
|
20
|
+
|
21
|
+
it 'removes only selected scheduled messages' do
|
22
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
23
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 4])
|
24
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 4])
|
25
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [5, 6])
|
26
|
+
assert_equal 0, Sidekiq::Client.remove_all_delayed('Foo')
|
27
|
+
end
|
28
|
+
|
29
|
+
it 'removes messages in different timestamp queues' do
|
30
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
31
|
+
Sidekiq::Client.delayed_push(1331284492, 'class' => 'Foo', 'args' => [3, 4])
|
32
|
+
Sidekiq::Client.delayed_push(1331284493, 'class' => 'Foo', 'args' => [3, 4])
|
33
|
+
Sidekiq::Client.delayed_push(1331284493, 'class' => 'Foo', 'args' => [5, 6])
|
34
|
+
assert_equal 2, Sidekiq::Client.remove_all_delayed('Foo', 3, 4)
|
35
|
+
end
|
36
|
+
|
37
|
+
it 'removes messages from specified timestamp' do
|
38
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
39
|
+
Sidekiq::Client.delayed_push(1331284492, 'class' => 'Foo', 'args' => [1, 2])
|
40
|
+
assert_equal 1, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 1, 2)
|
41
|
+
assert_equal 1, Sidekiq.redis.llen('delayed:1331284492')
|
42
|
+
end
|
43
|
+
|
44
|
+
it 'removes messages for a queue from specified timestamp' do
|
45
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
46
|
+
Sidekiq::Client.delayed_push('foo', 1331284492, 'class' => 'Foo', 'args' => [1, 2])
|
47
|
+
assert_equal 1, Sidekiq::Client.remove_delayed_from_queue('foo', 1331284491, 'Foo', 1, 2)
|
48
|
+
assert_equal 1, Sidekiq.redis.llen('delayed:1331284492')
|
49
|
+
end
|
50
|
+
|
51
|
+
it 'removes nothing if no message is found' do
|
52
|
+
assert_equal 0, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 3, 4)
|
53
|
+
end
|
54
|
+
|
55
|
+
it 'removes only messages with matching arguments' do
|
56
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
57
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 2])
|
58
|
+
assert_equal 0, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 3, 4)
|
59
|
+
assert_equal 2, Sidekiq.redis.llen('delayed:1331284491')
|
60
|
+
end
|
61
|
+
|
62
|
+
it 'removes empty scheduler queues' do
|
63
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
64
|
+
assert_equal 1, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 1, 2)
|
65
|
+
assert !Sidekiq.redis.exists('delayed:1331284491')
|
66
|
+
assert_equal 0, Sidekiq.redis.zcard('delayed_scheduler_queue')
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
describe 'with mock redis' do
|
71
|
+
before do
|
72
|
+
@redis = MiniTest::Mock.new
|
73
|
+
def @redis.multi; yield; end
|
74
|
+
def @redis.set(*); true; end
|
75
|
+
def @redis.sadd(*); true; end
|
76
|
+
def @redis.srem(*); true; end
|
77
|
+
def @redis.get(*); nil; end
|
78
|
+
def @redis.del(*); nil; end
|
79
|
+
def @redis.incrby(*); nil; end
|
80
|
+
def @redis.setex(*); nil; end
|
81
|
+
def @redis.expire(*); true; end
|
82
|
+
def @redis.with_connection; yield self; end
|
83
|
+
Sidekiq.instance_variable_set(:@redis, @redis)
|
84
|
+
end
|
85
|
+
|
86
|
+
it 'pushes delayed messages to redis' do
|
87
|
+
@redis.expect :rpush, 1, ['delayed:1331284491', String]
|
88
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', 1331284491, 1331284491]
|
89
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
90
|
+
@redis.verify
|
91
|
+
end
|
92
|
+
|
93
|
+
it 'removes empty scheduler queues' do
|
94
|
+
@redis.expect :llen, 0, ['delayed:1331284491']
|
95
|
+
@redis.expect :zrem, 1, ['delayed_queue_schedule', 1331284491]
|
96
|
+
Sidekiq::Client.remove_scheduler_queue(1331284491)
|
97
|
+
@redis.verify
|
98
|
+
end
|
99
|
+
|
100
|
+
it 'handles perform_at' do
|
101
|
+
@redis.expect :rpush, 1, ['delayed:1331284491', String]
|
102
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', 1331284491, 1331284491]
|
103
|
+
MyWorker.perform_at(1331284491, 1, 2)
|
104
|
+
@redis.verify
|
105
|
+
end
|
106
|
+
|
107
|
+
it 'handles perform_in' do
|
108
|
+
Timecop.freeze(Time.now) do
|
109
|
+
timestamp = Time.now + 30
|
110
|
+
@redis.expect :rpush, 1, ["delayed:#{timestamp.to_i}", String]
|
111
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', timestamp.to_i, timestamp.to_i]
|
112
|
+
MyWorker.perform_in(30, 1, 2)
|
113
|
+
@redis.verify
|
114
|
+
end
|
115
|
+
end
|
116
|
+
|
117
|
+
end
|
118
|
+
end
|
data/test/config.yml
ADDED
data/test/fake_env.rb
ADDED
File without changes
|
@@ -0,0 +1,66 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'sidekiq'
|
3
|
+
require 'sidekiq/manager'
|
4
|
+
|
5
|
+
class ManagerTest < MiniTest::Unit::TestCase
|
6
|
+
describe 'with redis' do
|
7
|
+
before do
|
8
|
+
Sidekiq.redis = { :url => 'redis://localhost/sidekiq_test' }
|
9
|
+
@scheduler = SidekiqScheduler::Manager.new
|
10
|
+
@redis = Sidekiq.redis
|
11
|
+
@redis.flushdb
|
12
|
+
$processed = 0
|
13
|
+
$mutex = Mutex.new
|
14
|
+
end
|
15
|
+
|
16
|
+
class IntegrationWorker
|
17
|
+
include Sidekiq::Worker
|
18
|
+
|
19
|
+
def perform(a, b)
|
20
|
+
$mutex.synchronize do
|
21
|
+
$processed += 1
|
22
|
+
end
|
23
|
+
a + b
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
it 'detects an empty schedule run' do
|
28
|
+
assert_nil @scheduler.send(:find_next_timestamp)
|
29
|
+
end
|
30
|
+
|
31
|
+
it 'processes only jobs that are due' do
|
32
|
+
timestamp = Time.now + 600
|
33
|
+
Sidekiq::Client.delayed_push(:foo, timestamp, 'class' => IntegrationWorker, 'args' => [1,2])
|
34
|
+
assert_nil @scheduler.send(:find_next_timestamp)
|
35
|
+
end
|
36
|
+
|
37
|
+
it 'processes queues in the right order' do
|
38
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
39
|
+
Sidekiq::Client.delayed_push(:foo, 1331284492, 'class' => IntegrationWorker, 'args' => [1,2])
|
40
|
+
|
41
|
+
assert_equal 1331284491, @scheduler.send(:find_next_timestamp)
|
42
|
+
end
|
43
|
+
|
44
|
+
it 'moves jobs from the scheduler queues to the worker queues' do
|
45
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
46
|
+
|
47
|
+
@scheduler.send(:find_scheduled_work, 1331284491)
|
48
|
+
|
49
|
+
assert_equal 0, @redis.llen("delayed:1331284491")
|
50
|
+
assert_equal 1, @redis.llen("queue:foo")
|
51
|
+
end
|
52
|
+
|
53
|
+
it 'resets the scheduler queue' do
|
54
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
55
|
+
Sidekiq::Client.delayed_push(:foo, 1331284492, 'class' => IntegrationWorker, 'args' => [1,2])
|
56
|
+
Sidekiq::Client.delayed_push(:foo, 1331284493, 'class' => IntegrationWorker, 'args' => [1,2])
|
57
|
+
|
58
|
+
@scheduler.reset
|
59
|
+
|
60
|
+
assert_equal 0, @redis.zcard('delayed_queue_schedule')
|
61
|
+
assert !@redis.exists('delayed:1331284491')
|
62
|
+
assert !@redis.exists('delayed:1331284492')
|
63
|
+
assert !@redis.exists('delayed:1331284493')
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
data/test/test_helper.rb
CHANGED
@@ -1,10 +1,11 @@
|
|
1
|
-
|
2
|
-
|
1
|
+
require 'minitest/unit'
|
2
|
+
require 'minitest/pride'
|
3
|
+
require 'minitest/autorun'
|
4
|
+
require 'sidekiq-scheduler'
|
3
5
|
|
4
|
-
require
|
5
|
-
|
6
|
-
|
7
|
-
Rails.backtrace_cleaner.remove_silencers!
|
6
|
+
require 'sidekiq/util'
|
7
|
+
Sidekiq::Util.logger.level = Logger::ERROR
|
8
8
|
|
9
9
|
# Load support files
|
10
10
|
Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each { |f| require f }
|
11
|
+
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-scheduler
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.1.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,11 +9,11 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-03-
|
12
|
+
date: 2012-03-13 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sidekiq
|
16
|
-
requirement: &
|
16
|
+
requirement: &70235778635560 !ruby/object:Gem::Requirement
|
17
17
|
none: false
|
18
18
|
requirements:
|
19
19
|
- - ! '>='
|
@@ -21,10 +21,10 @@ dependencies:
|
|
21
21
|
version: 0.8.0
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
|
-
version_requirements: *
|
24
|
+
version_requirements: *70235778635560
|
25
25
|
- !ruby/object:Gem::Dependency
|
26
26
|
name: rake
|
27
|
-
requirement: &
|
27
|
+
requirement: &70235778634920 !ruby/object:Gem::Requirement
|
28
28
|
none: false
|
29
29
|
requirements:
|
30
30
|
- - ! '>='
|
@@ -32,10 +32,10 @@ dependencies:
|
|
32
32
|
version: '0'
|
33
33
|
type: :development
|
34
34
|
prerelease: false
|
35
|
-
version_requirements: *
|
35
|
+
version_requirements: *70235778634920
|
36
36
|
- !ruby/object:Gem::Dependency
|
37
37
|
name: timecop
|
38
|
-
requirement: &
|
38
|
+
requirement: &70235778634440 !ruby/object:Gem::Requirement
|
39
39
|
none: false
|
40
40
|
requirements:
|
41
41
|
- - ! '>='
|
@@ -43,7 +43,7 @@ dependencies:
|
|
43
43
|
version: '0'
|
44
44
|
type: :development
|
45
45
|
prerelease: false
|
46
|
-
version_requirements: *
|
46
|
+
version_requirements: *70235778634440
|
47
47
|
description: Light weight job scheduling extension for Sidekiq that adds support for
|
48
48
|
queueing items in the future.
|
49
49
|
email:
|
@@ -52,13 +52,22 @@ executables: []
|
|
52
52
|
extensions: []
|
53
53
|
extra_rdoc_files: []
|
54
54
|
files:
|
55
|
+
- lib/sidekiq-scheduler/cli.rb
|
56
|
+
- lib/sidekiq-scheduler/client.rb
|
57
|
+
- lib/sidekiq-scheduler/manager.rb
|
55
58
|
- lib/sidekiq-scheduler/version.rb
|
59
|
+
- lib/sidekiq-scheduler/worker.rb
|
56
60
|
- lib/sidekiq-scheduler.rb
|
57
61
|
- lib/tasks/sidekiq-scheduler_tasks.rake
|
58
62
|
- MIT-LICENSE
|
59
63
|
- Rakefile
|
60
64
|
- README.md
|
61
|
-
- test/
|
65
|
+
- test/cli_test.rb
|
66
|
+
- test/client_test.rb
|
67
|
+
- test/config.yml
|
68
|
+
- test/fake_env.rb
|
69
|
+
- test/manager_test.rb
|
70
|
+
- test/support/my_worker.rb
|
62
71
|
- test/test_helper.rb
|
63
72
|
homepage: https://github.com/yabawock/sidekiq-scheduler
|
64
73
|
licenses: []
|
@@ -74,7 +83,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
74
83
|
version: '0'
|
75
84
|
segments:
|
76
85
|
- 0
|
77
|
-
hash: -
|
86
|
+
hash: -2626338027335115584
|
78
87
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
79
88
|
none: false
|
80
89
|
requirements:
|
@@ -83,7 +92,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
83
92
|
version: '0'
|
84
93
|
segments:
|
85
94
|
- 0
|
86
|
-
hash: -
|
95
|
+
hash: -2626338027335115584
|
87
96
|
requirements: []
|
88
97
|
rubyforge_project:
|
89
98
|
rubygems_version: 1.8.17
|
@@ -91,5 +100,10 @@ signing_key:
|
|
91
100
|
specification_version: 3
|
92
101
|
summary: Light weight job scheduling extension for Sidekiq
|
93
102
|
test_files:
|
94
|
-
- test/
|
103
|
+
- test/cli_test.rb
|
104
|
+
- test/client_test.rb
|
105
|
+
- test/config.yml
|
106
|
+
- test/fake_env.rb
|
107
|
+
- test/manager_test.rb
|
108
|
+
- test/support/my_worker.rb
|
95
109
|
- test/test_helper.rb
|