sidekiq-scheduler 0.0.0 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +111 -1
- data/lib/sidekiq-scheduler/cli.rb +24 -0
- data/lib/sidekiq-scheduler/client.rb +87 -0
- data/lib/sidekiq-scheduler/manager.rb +86 -0
- data/lib/sidekiq-scheduler/version.rb +1 -1
- data/lib/sidekiq-scheduler/worker.rb +18 -0
- data/lib/sidekiq-scheduler.rb +4 -2
- data/test/cli_test.rb +33 -0
- data/test/client_test.rb +118 -0
- data/test/config.yml +10 -0
- data/test/fake_env.rb +0 -0
- data/test/manager_test.rb +66 -0
- data/test/support/my_worker.rb +3 -0
- data/test/test_helper.rb +7 -6
- metadata +26 -12
- data/test/sidekiq-scheduler_test.rb +0 -7
data/README.md
CHANGED
@@ -1,3 +1,113 @@
|
|
1
1
|
# SidekiqScheduler
|
2
2
|
|
3
|
-
|
3
|
+
## Description
|
4
|
+
|
5
|
+
sidekiq-scheduler is an extension to [Sidekiq](http://github.com/mperham/sidekiq)
|
6
|
+
that adds support for queueing jobs in the future.
|
7
|
+
|
8
|
+
At the moment job scheduling is only supported in a delayed fashion. Replacing cron
|
9
|
+
is not the intention of this gem. Delayed jobs are Sidekiq jobs that you want to run
|
10
|
+
at some point in the future.
|
11
|
+
|
12
|
+
The syntax is pretty explanatory:
|
13
|
+
|
14
|
+
MyWorker.perform_in(5.days, 'arg1', 'arg2') # run a job in 5 days
|
15
|
+
# or
|
16
|
+
MyWorker.perform_at(5.days.from_now, 'arg1', 'arg2') # run job at a specific time
|
17
|
+
|
18
|
+
## Installation
|
19
|
+
|
20
|
+
# Rails 3.x: add it to your Gemfile
|
21
|
+
gem 'sidekiq-scheduler'
|
22
|
+
|
23
|
+
# Starting the scheduler
|
24
|
+
bundle exec sidekiq-scheduler
|
25
|
+
|
26
|
+
The scheduler will perform identically to a normal sidekiq worker with
|
27
|
+
an additional scheduler thread being run - in the default configuration
|
28
|
+
this will result in 25 worker threads being available on the scheduler
|
29
|
+
node but all normal configuration options apply.
|
30
|
+
|
31
|
+
NOTE: Since it's currently not possible to hook into the default option
|
32
|
+
parsing provided by sidekiq you will need to use a configuration file to
|
33
|
+
override the scheduler options. Currently the only option available is
|
34
|
+
|
35
|
+
resolution: <seconds between schedule runs>
|
36
|
+
|
37
|
+
The scheduling thread will sleep this many seconds between looking for
|
38
|
+
jobs that need moving to the worker queue. The default is 5 seconds
|
39
|
+
which should be fast enough for almost all uses.
|
40
|
+
|
41
|
+
NOTE: You DO NOT want to run more than one instance of the scheduler. Doing
|
42
|
+
so will result in the same job being queued multiple times. You only need one
|
43
|
+
instance of the scheduler running per application, regardless of number of servers.
|
44
|
+
|
45
|
+
NOTE: If the scheduler thread goes down for whatever reason, the delayed items
|
46
|
+
that should have fired during the outage will fire once the scheduler is
|
47
|
+
started back up again (even if it is on a new machine).
|
48
|
+
|
49
|
+
## Delayed jobs
|
50
|
+
|
51
|
+
Delayed jobs are one-off jobs that you want to be put into a queue at some point
|
52
|
+
in the future. The classic example is sending email:
|
53
|
+
|
54
|
+
MyWorker.perform_in(5.days, current_user.id)
|
55
|
+
|
56
|
+
This will store the job for 5 days in the Sidekiq delayed queue at which time
|
57
|
+
the scheduler will pull it from the delayed queue and put it in the appropriate
|
58
|
+
work queue for the given job. It will then be processed as soon as a worker is
|
59
|
+
available (just like any other Sidekiq job).
|
60
|
+
|
61
|
+
NOTE: The job does not fire **exactly** at the time supplied. Rather, once that
|
62
|
+
time is in the past, the job moves from the delayed queue to the actual work
|
63
|
+
queue and will be completed as workers are free to process it.
|
64
|
+
|
65
|
+
Also supported is `MyWork.perform_at` which takes a timestamp to queue the job.
|
66
|
+
|
67
|
+
The delayed queue is stored in redis and is persisted in the same way the
|
68
|
+
standard Sidekiq jobs are persisted (redis writing to disk). Delayed jobs differ
|
69
|
+
from scheduled jobs in that if your scheduler process is down or workers are
|
70
|
+
down when a particular job is supposed to be processed, they will simply "catch up"
|
71
|
+
once they are started again. Jobs are guaranteed to run (provided they make it
|
72
|
+
into the delayed queue) after their given queue_at time has passed.
|
73
|
+
|
74
|
+
One other thing to note is that insertion into the delayed queue is O(log(n))
|
75
|
+
since the jobs are stored in a redis sorted set (zset). I can't imagine this
|
76
|
+
being an issue for someone since redis is stupidly fast even at log(n), but full
|
77
|
+
disclosure is always best.
|
78
|
+
|
79
|
+
### Removing Delayed jobs
|
80
|
+
|
81
|
+
If you have the need to cancel a delayed job, you can do it like this:
|
82
|
+
|
83
|
+
# after you've enqueued a job like:
|
84
|
+
MyWorker.perform_at(5.days.from_now, 'arg1', 'arg2')
|
85
|
+
# remove the job with exactly the same parameters:
|
86
|
+
MyWorker.remove_delayed(<timestamp>, 'arg1', 'arg2')
|
87
|
+
|
88
|
+
## Note on Patches / Pull Requests
|
89
|
+
|
90
|
+
* Fork the project.
|
91
|
+
* Make your feature addition or bug fix.
|
92
|
+
* Add tests for it. This is important so I don't break it in a future version unintentionally.
|
93
|
+
* Commit, do not mess with rakefile, version, or history.
|
94
|
+
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
|
95
|
+
* Send me a pull request. Bonus points for topic branches.
|
96
|
+
|
97
|
+
## Credits
|
98
|
+
|
99
|
+
This work is a partial port of [resque-scheduler](https://github.com/bvandenbos/resque-scheduler) by Ben VandenBos.
|
100
|
+
Modified to work with the Sidekiq queueing library by Morton Jonuschat.
|
101
|
+
|
102
|
+
## Maintainers
|
103
|
+
|
104
|
+
* [Morton Jonuschat](https://github.com/yabawock)
|
105
|
+
|
106
|
+
## License
|
107
|
+
|
108
|
+
MIT License
|
109
|
+
|
110
|
+
## Copyright
|
111
|
+
|
112
|
+
Copyright 2012 Morton Jonuschat
|
113
|
+
Some parts copyright 2010 Ben VandenBos
|
@@ -0,0 +1,24 @@
|
|
1
|
+
require 'sidekiq-scheduler/manager'
|
2
|
+
require 'sidekiq/cli'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module CLI
|
6
|
+
def self.included(base)
|
7
|
+
base.class_eval do
|
8
|
+
alias_method :run_manager, :run
|
9
|
+
alias_method :run, :run_scheduler
|
10
|
+
end
|
11
|
+
end
|
12
|
+
|
13
|
+
def run_scheduler
|
14
|
+
scheduler_options = { :scheduler => true, :resolution => 5 }
|
15
|
+
scheduler_options.merge!(options)
|
16
|
+
scheduler = SidekiqScheduler::Manager.new(scheduler_options)
|
17
|
+
scheduler.start!
|
18
|
+
run_manager
|
19
|
+
scheduler.stop!
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
Sidekiq::CLI.send(:include, SidekiqScheduler::CLI)
|
@@ -0,0 +1,87 @@
|
|
1
|
+
require 'sidekiq-scheduler/worker'
|
2
|
+
require 'sidekiq/client'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module Client
|
6
|
+
# Example usage:
|
7
|
+
# Sidekiq::Client.delayed_push('my_queue', Time.now + 600, 'class' => MyWorker, 'args' => ['foo', 1, :bat => 'bar'])
|
8
|
+
def delayed_push(queue=nil, timestamp, item)
|
9
|
+
raise(ArgumentError, "Message must be a Hash of the form: { 'class' => SomeClass, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash)
|
10
|
+
raise(ArgumentError, "Message must include a class and set of arguments: #{item.inspect}") if !item['class'] || !item['args']
|
11
|
+
|
12
|
+
timestamp = timestamp.to_i
|
13
|
+
|
14
|
+
item['queue'] = queue.to_s if queue
|
15
|
+
item['class'] = item['class'].to_s if !item['class'].is_a?(String)
|
16
|
+
|
17
|
+
# Add item to the list for this timestamp
|
18
|
+
Sidekiq.redis.rpush("delayed:#{timestamp}", MultiJson.encode(item))
|
19
|
+
|
20
|
+
# Add timestamp to zset. Score and value are based on the timestamp
|
21
|
+
# as querying will be based on that
|
22
|
+
Sidekiq.redis.zadd('delayed_queue_schedule', timestamp, timestamp)
|
23
|
+
end
|
24
|
+
|
25
|
+
def remove_scheduler_queue(timestamp)
|
26
|
+
key = "delayed:#{timestamp}"
|
27
|
+
if 0 == Sidekiq.redis.llen(key)
|
28
|
+
Sidekiq.redis.del(key)
|
29
|
+
Sidekiq.redis.zrem('delayed_queue_schedule', timestamp)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
# Example usage:
|
34
|
+
# Sidekiq::Client.remove_all_delayed(MyWorker, 'foo', 1, :bat => 'bar')
|
35
|
+
#
|
36
|
+
# Returns the number of jobs removed
|
37
|
+
#
|
38
|
+
# This method can be very expensive since it needs to scan
|
39
|
+
# through the delayed queues of all timestamps
|
40
|
+
def remove_all_delayed(klass, *args)
|
41
|
+
remove_all_delayed_from_queue(nil, klass, *args)
|
42
|
+
end
|
43
|
+
|
44
|
+
# Example usage:
|
45
|
+
# Sidekiq::Client.remove_all_delayed('foo', MyWorker, 'foo', 1, :bat => 'bar')
|
46
|
+
#
|
47
|
+
# Returns the number of jobs removed
|
48
|
+
#
|
49
|
+
# This method can be very expensive since it needs to scan
|
50
|
+
# through the delayed queues of all timestamps
|
51
|
+
def remove_all_delayed_from_queue(queue, klass, *args)
|
52
|
+
count = 0
|
53
|
+
item = {'class' => klass.to_s, 'args' => args}
|
54
|
+
item['queue'] = queue.to_s if queue
|
55
|
+
search = MultiJson.encode(item)
|
56
|
+
Array(Sidekiq.redis.keys("delayed:*")).each do |key|
|
57
|
+
count += Sidekiq.redis.lrem(key, 0, search)
|
58
|
+
end
|
59
|
+
count
|
60
|
+
end
|
61
|
+
|
62
|
+
# Example usage:
|
63
|
+
# Sidekiq::Client.remove_delayed(Time.now + 600, MyWorker, 'foo', 1, :bat => 'bar')
|
64
|
+
#
|
65
|
+
# Returns the number of jobs removed
|
66
|
+
def remove_delayed(timestamp, klass, *args)
|
67
|
+
remove_delayed_from_queue(nil, timestamp, klass, *args)
|
68
|
+
end
|
69
|
+
|
70
|
+
|
71
|
+
# Example usage:
|
72
|
+
# Sidekiq::Client.remove_delayed('foo', Time.now + 600, MyWorker, 'foo', 1, :bat => 'bar')
|
73
|
+
#
|
74
|
+
# Returns the number of jobs removed
|
75
|
+
def remove_delayed_from_queue(queue, timestamp, klass, *args)
|
76
|
+
timestamp = timestamp.to_i
|
77
|
+
item = {'class' => klass.to_s, 'args' => args}
|
78
|
+
item['queue'] = queue.to_s if queue
|
79
|
+
search = MultiJson.encode(item)
|
80
|
+
count = Sidekiq.redis.lrem("delayed:#{timestamp}", 0, search)
|
81
|
+
remove_scheduler_queue(timestamp)
|
82
|
+
count
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
Sidekiq::Client.send(:extend, SidekiqScheduler::Client)
|
@@ -0,0 +1,86 @@
|
|
1
|
+
require 'celluloid'
|
2
|
+
require 'redis'
|
3
|
+
require 'multi_json'
|
4
|
+
|
5
|
+
require 'sidekiq/util'
|
6
|
+
|
7
|
+
module SidekiqScheduler
|
8
|
+
|
9
|
+
##
|
10
|
+
# The delayed job router in the system. This
|
11
|
+
# manages the scheduled jobs pushed messages
|
12
|
+
# from Redis onto the work queues
|
13
|
+
#
|
14
|
+
class Manager
|
15
|
+
include Sidekiq::Util
|
16
|
+
include Celluloid
|
17
|
+
|
18
|
+
def initialize(options={})
|
19
|
+
logger.info "Booting sidekiq scheduler #{SidekiqScheduler::VERSION} with Redis at #{redis.client.location}"
|
20
|
+
logger.debug { options.inspect }
|
21
|
+
@enabled = options[:scheduler]
|
22
|
+
@resolution = options[:resolution] || 5
|
23
|
+
end
|
24
|
+
|
25
|
+
def stop
|
26
|
+
@enabled = false
|
27
|
+
end
|
28
|
+
|
29
|
+
def start
|
30
|
+
schedule(true)
|
31
|
+
end
|
32
|
+
|
33
|
+
def reset
|
34
|
+
clear_scheduled_work
|
35
|
+
end
|
36
|
+
|
37
|
+
private
|
38
|
+
|
39
|
+
def clear_scheduled_work
|
40
|
+
queues = redis.zrange('delayed_queue_schedule', 0, -1).to_a
|
41
|
+
redis.del(*queues.map { |t| "delayed:#{t}" }) unless queues.empty?
|
42
|
+
redis.del('delayed_queue_schedule')
|
43
|
+
end
|
44
|
+
|
45
|
+
def find_scheduled_work(timestamp)
|
46
|
+
loop do
|
47
|
+
break logger.debug("Finished processing queue for timestamp #{timestamp}") unless msg = redis.lpop("delayed:#{timestamp}")
|
48
|
+
item = MultiJson.decode(msg)
|
49
|
+
queue = item.delete('queue')
|
50
|
+
Sidekiq::Client.push(queue, item)
|
51
|
+
end
|
52
|
+
Sidekiq::Client.remove_scheduler_queue(timestamp)
|
53
|
+
end
|
54
|
+
|
55
|
+
def find_next_timestamp
|
56
|
+
timestamp = redis.zrangebyscore('delayed_queue_schedule', '-inf', Time.now.to_i, :limit => [0, 1])
|
57
|
+
if timestamp.is_a?(Array)
|
58
|
+
timestamp = timestamp.first
|
59
|
+
end
|
60
|
+
timestamp.to_i unless timestamp.nil?
|
61
|
+
end
|
62
|
+
|
63
|
+
def schedule(run_loop = false)
|
64
|
+
watchdog("Fatal error in sidekiq, scheduler loop died") do
|
65
|
+
return if stopped?
|
66
|
+
|
67
|
+
# Dispatch loop
|
68
|
+
loop do
|
69
|
+
break logger.debug('no scheduler queues to process') unless timestamp = find_next_timestamp
|
70
|
+
find_scheduled_work(timestamp)
|
71
|
+
end
|
72
|
+
|
73
|
+
# This is the polling loop that ensures we check Redis every
|
74
|
+
# second for work, even if there was nothing to do this time
|
75
|
+
# around.
|
76
|
+
after(@resolution) do
|
77
|
+
schedule(run_loop)
|
78
|
+
end if run_loop
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
def stopped?
|
83
|
+
!@enabled
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
require 'sidekiq-scheduler/client'
|
2
|
+
require 'sidekiq/worker'
|
3
|
+
|
4
|
+
module SidekiqScheduler
|
5
|
+
module Worker
|
6
|
+
module ClassMethods
|
7
|
+
def perform_at(timestamp, *args)
|
8
|
+
Sidekiq::Client.delayed_push(timestamp, 'class' => self.name, 'args' => args)
|
9
|
+
end
|
10
|
+
|
11
|
+
def perform_in(seconds_from_now, *args)
|
12
|
+
Sidekiq::Client.delayed_push(Time.now + seconds_from_now, 'class' => self.name, 'args' => args)
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
Sidekiq::Worker::ClassMethods.send(:include, SidekiqScheduler::Worker::ClassMethods)
|
data/lib/sidekiq-scheduler.rb
CHANGED
data/test/cli_test.rb
ADDED
@@ -0,0 +1,33 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'sidekiq-scheduler/cli'
|
3
|
+
require 'tempfile'
|
4
|
+
|
5
|
+
class CliTest < MiniTest::Unit::TestCase
|
6
|
+
describe 'with cli' do
|
7
|
+
before do
|
8
|
+
@cli = new_cli
|
9
|
+
end
|
10
|
+
|
11
|
+
describe 'with config file' do
|
12
|
+
before do
|
13
|
+
@cli.parse(['sidekiq', '-C', './test/config.yml'])
|
14
|
+
end
|
15
|
+
|
16
|
+
it 'sets the resolution of the scheduler timer' do
|
17
|
+
assert_equal 30, Sidekiq.options[:resolution]
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def new_cli
|
22
|
+
cli = Sidekiq::CLI.new
|
23
|
+
def cli.die(code)
|
24
|
+
@code = code
|
25
|
+
end
|
26
|
+
|
27
|
+
def cli.valid?
|
28
|
+
!@code
|
29
|
+
end
|
30
|
+
cli
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
data/test/client_test.rb
ADDED
@@ -0,0 +1,118 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'timecop'
|
3
|
+
|
4
|
+
class ClientTest < MiniTest::Unit::TestCase
|
5
|
+
describe 'with real redis' do
|
6
|
+
before do
|
7
|
+
Sidekiq.redis = { :url => 'redis://localhost/sidekiq_test' }
|
8
|
+
Sidekiq.redis.flushdb
|
9
|
+
end
|
10
|
+
|
11
|
+
it 'removes scheduled messages and returns count' do
|
12
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
13
|
+
assert_equal 1, Sidekiq::Client.remove_all_delayed('Foo', 1, 2)
|
14
|
+
end
|
15
|
+
|
16
|
+
it 'removes scheduled messages for a queue and returns count' do
|
17
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
18
|
+
assert_equal 1, Sidekiq::Client.remove_all_delayed_from_queue('foo', 'Foo', 1, 2)
|
19
|
+
end
|
20
|
+
|
21
|
+
it 'removes only selected scheduled messages' do
|
22
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
23
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 4])
|
24
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 4])
|
25
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [5, 6])
|
26
|
+
assert_equal 0, Sidekiq::Client.remove_all_delayed('Foo')
|
27
|
+
end
|
28
|
+
|
29
|
+
it 'removes messages in different timestamp queues' do
|
30
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
31
|
+
Sidekiq::Client.delayed_push(1331284492, 'class' => 'Foo', 'args' => [3, 4])
|
32
|
+
Sidekiq::Client.delayed_push(1331284493, 'class' => 'Foo', 'args' => [3, 4])
|
33
|
+
Sidekiq::Client.delayed_push(1331284493, 'class' => 'Foo', 'args' => [5, 6])
|
34
|
+
assert_equal 2, Sidekiq::Client.remove_all_delayed('Foo', 3, 4)
|
35
|
+
end
|
36
|
+
|
37
|
+
it 'removes messages from specified timestamp' do
|
38
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
39
|
+
Sidekiq::Client.delayed_push(1331284492, 'class' => 'Foo', 'args' => [1, 2])
|
40
|
+
assert_equal 1, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 1, 2)
|
41
|
+
assert_equal 1, Sidekiq.redis.llen('delayed:1331284492')
|
42
|
+
end
|
43
|
+
|
44
|
+
it 'removes messages for a queue from specified timestamp' do
|
45
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
46
|
+
Sidekiq::Client.delayed_push('foo', 1331284492, 'class' => 'Foo', 'args' => [1, 2])
|
47
|
+
assert_equal 1, Sidekiq::Client.remove_delayed_from_queue('foo', 1331284491, 'Foo', 1, 2)
|
48
|
+
assert_equal 1, Sidekiq.redis.llen('delayed:1331284492')
|
49
|
+
end
|
50
|
+
|
51
|
+
it 'removes nothing if no message is found' do
|
52
|
+
assert_equal 0, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 3, 4)
|
53
|
+
end
|
54
|
+
|
55
|
+
it 'removes only messages with matching arguments' do
|
56
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
57
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [3, 2])
|
58
|
+
assert_equal 0, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 3, 4)
|
59
|
+
assert_equal 2, Sidekiq.redis.llen('delayed:1331284491')
|
60
|
+
end
|
61
|
+
|
62
|
+
it 'removes empty scheduler queues' do
|
63
|
+
Sidekiq::Client.delayed_push(1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
64
|
+
assert_equal 1, Sidekiq::Client.remove_delayed(1331284491, 'Foo', 1, 2)
|
65
|
+
assert !Sidekiq.redis.exists('delayed:1331284491')
|
66
|
+
assert_equal 0, Sidekiq.redis.zcard('delayed_scheduler_queue')
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
describe 'with mock redis' do
|
71
|
+
before do
|
72
|
+
@redis = MiniTest::Mock.new
|
73
|
+
def @redis.multi; yield; end
|
74
|
+
def @redis.set(*); true; end
|
75
|
+
def @redis.sadd(*); true; end
|
76
|
+
def @redis.srem(*); true; end
|
77
|
+
def @redis.get(*); nil; end
|
78
|
+
def @redis.del(*); nil; end
|
79
|
+
def @redis.incrby(*); nil; end
|
80
|
+
def @redis.setex(*); nil; end
|
81
|
+
def @redis.expire(*); true; end
|
82
|
+
def @redis.with_connection; yield self; end
|
83
|
+
Sidekiq.instance_variable_set(:@redis, @redis)
|
84
|
+
end
|
85
|
+
|
86
|
+
it 'pushes delayed messages to redis' do
|
87
|
+
@redis.expect :rpush, 1, ['delayed:1331284491', String]
|
88
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', 1331284491, 1331284491]
|
89
|
+
Sidekiq::Client.delayed_push('foo', 1331284491, 'class' => 'Foo', 'args' => [1, 2])
|
90
|
+
@redis.verify
|
91
|
+
end
|
92
|
+
|
93
|
+
it 'removes empty scheduler queues' do
|
94
|
+
@redis.expect :llen, 0, ['delayed:1331284491']
|
95
|
+
@redis.expect :zrem, 1, ['delayed_queue_schedule', 1331284491]
|
96
|
+
Sidekiq::Client.remove_scheduler_queue(1331284491)
|
97
|
+
@redis.verify
|
98
|
+
end
|
99
|
+
|
100
|
+
it 'handles perform_at' do
|
101
|
+
@redis.expect :rpush, 1, ['delayed:1331284491', String]
|
102
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', 1331284491, 1331284491]
|
103
|
+
MyWorker.perform_at(1331284491, 1, 2)
|
104
|
+
@redis.verify
|
105
|
+
end
|
106
|
+
|
107
|
+
it 'handles perform_in' do
|
108
|
+
Timecop.freeze(Time.now) do
|
109
|
+
timestamp = Time.now + 30
|
110
|
+
@redis.expect :rpush, 1, ["delayed:#{timestamp.to_i}", String]
|
111
|
+
@redis.expect :zadd, 1, ['delayed_queue_schedule', timestamp.to_i, timestamp.to_i]
|
112
|
+
MyWorker.perform_in(30, 1, 2)
|
113
|
+
@redis.verify
|
114
|
+
end
|
115
|
+
end
|
116
|
+
|
117
|
+
end
|
118
|
+
end
|
data/test/config.yml
ADDED
data/test/fake_env.rb
ADDED
File without changes
|
@@ -0,0 +1,66 @@
|
|
1
|
+
require 'test_helper'
|
2
|
+
require 'sidekiq'
|
3
|
+
require 'sidekiq/manager'
|
4
|
+
|
5
|
+
class ManagerTest < MiniTest::Unit::TestCase
|
6
|
+
describe 'with redis' do
|
7
|
+
before do
|
8
|
+
Sidekiq.redis = { :url => 'redis://localhost/sidekiq_test' }
|
9
|
+
@scheduler = SidekiqScheduler::Manager.new
|
10
|
+
@redis = Sidekiq.redis
|
11
|
+
@redis.flushdb
|
12
|
+
$processed = 0
|
13
|
+
$mutex = Mutex.new
|
14
|
+
end
|
15
|
+
|
16
|
+
class IntegrationWorker
|
17
|
+
include Sidekiq::Worker
|
18
|
+
|
19
|
+
def perform(a, b)
|
20
|
+
$mutex.synchronize do
|
21
|
+
$processed += 1
|
22
|
+
end
|
23
|
+
a + b
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
it 'detects an empty schedule run' do
|
28
|
+
assert_nil @scheduler.send(:find_next_timestamp)
|
29
|
+
end
|
30
|
+
|
31
|
+
it 'processes only jobs that are due' do
|
32
|
+
timestamp = Time.now + 600
|
33
|
+
Sidekiq::Client.delayed_push(:foo, timestamp, 'class' => IntegrationWorker, 'args' => [1,2])
|
34
|
+
assert_nil @scheduler.send(:find_next_timestamp)
|
35
|
+
end
|
36
|
+
|
37
|
+
it 'processes queues in the right order' do
|
38
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
39
|
+
Sidekiq::Client.delayed_push(:foo, 1331284492, 'class' => IntegrationWorker, 'args' => [1,2])
|
40
|
+
|
41
|
+
assert_equal 1331284491, @scheduler.send(:find_next_timestamp)
|
42
|
+
end
|
43
|
+
|
44
|
+
it 'moves jobs from the scheduler queues to the worker queues' do
|
45
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
46
|
+
|
47
|
+
@scheduler.send(:find_scheduled_work, 1331284491)
|
48
|
+
|
49
|
+
assert_equal 0, @redis.llen("delayed:1331284491")
|
50
|
+
assert_equal 1, @redis.llen("queue:foo")
|
51
|
+
end
|
52
|
+
|
53
|
+
it 'resets the scheduler queue' do
|
54
|
+
Sidekiq::Client.delayed_push(:foo, 1331284491, 'class' => IntegrationWorker, 'args' => [1,2])
|
55
|
+
Sidekiq::Client.delayed_push(:foo, 1331284492, 'class' => IntegrationWorker, 'args' => [1,2])
|
56
|
+
Sidekiq::Client.delayed_push(:foo, 1331284493, 'class' => IntegrationWorker, 'args' => [1,2])
|
57
|
+
|
58
|
+
@scheduler.reset
|
59
|
+
|
60
|
+
assert_equal 0, @redis.zcard('delayed_queue_schedule')
|
61
|
+
assert !@redis.exists('delayed:1331284491')
|
62
|
+
assert !@redis.exists('delayed:1331284492')
|
63
|
+
assert !@redis.exists('delayed:1331284493')
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
data/test/test_helper.rb
CHANGED
@@ -1,10 +1,11 @@
|
|
1
|
-
|
2
|
-
|
1
|
+
require 'minitest/unit'
|
2
|
+
require 'minitest/pride'
|
3
|
+
require 'minitest/autorun'
|
4
|
+
require 'sidekiq-scheduler'
|
3
5
|
|
4
|
-
require
|
5
|
-
|
6
|
-
|
7
|
-
Rails.backtrace_cleaner.remove_silencers!
|
6
|
+
require 'sidekiq/util'
|
7
|
+
Sidekiq::Util.logger.level = Logger::ERROR
|
8
8
|
|
9
9
|
# Load support files
|
10
10
|
Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each { |f| require f }
|
11
|
+
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-scheduler
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.1.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,11 +9,11 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-03-
|
12
|
+
date: 2012-03-13 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sidekiq
|
16
|
-
requirement: &
|
16
|
+
requirement: &70235778635560 !ruby/object:Gem::Requirement
|
17
17
|
none: false
|
18
18
|
requirements:
|
19
19
|
- - ! '>='
|
@@ -21,10 +21,10 @@ dependencies:
|
|
21
21
|
version: 0.8.0
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
|
-
version_requirements: *
|
24
|
+
version_requirements: *70235778635560
|
25
25
|
- !ruby/object:Gem::Dependency
|
26
26
|
name: rake
|
27
|
-
requirement: &
|
27
|
+
requirement: &70235778634920 !ruby/object:Gem::Requirement
|
28
28
|
none: false
|
29
29
|
requirements:
|
30
30
|
- - ! '>='
|
@@ -32,10 +32,10 @@ dependencies:
|
|
32
32
|
version: '0'
|
33
33
|
type: :development
|
34
34
|
prerelease: false
|
35
|
-
version_requirements: *
|
35
|
+
version_requirements: *70235778634920
|
36
36
|
- !ruby/object:Gem::Dependency
|
37
37
|
name: timecop
|
38
|
-
requirement: &
|
38
|
+
requirement: &70235778634440 !ruby/object:Gem::Requirement
|
39
39
|
none: false
|
40
40
|
requirements:
|
41
41
|
- - ! '>='
|
@@ -43,7 +43,7 @@ dependencies:
|
|
43
43
|
version: '0'
|
44
44
|
type: :development
|
45
45
|
prerelease: false
|
46
|
-
version_requirements: *
|
46
|
+
version_requirements: *70235778634440
|
47
47
|
description: Light weight job scheduling extension for Sidekiq that adds support for
|
48
48
|
queueing items in the future.
|
49
49
|
email:
|
@@ -52,13 +52,22 @@ executables: []
|
|
52
52
|
extensions: []
|
53
53
|
extra_rdoc_files: []
|
54
54
|
files:
|
55
|
+
- lib/sidekiq-scheduler/cli.rb
|
56
|
+
- lib/sidekiq-scheduler/client.rb
|
57
|
+
- lib/sidekiq-scheduler/manager.rb
|
55
58
|
- lib/sidekiq-scheduler/version.rb
|
59
|
+
- lib/sidekiq-scheduler/worker.rb
|
56
60
|
- lib/sidekiq-scheduler.rb
|
57
61
|
- lib/tasks/sidekiq-scheduler_tasks.rake
|
58
62
|
- MIT-LICENSE
|
59
63
|
- Rakefile
|
60
64
|
- README.md
|
61
|
-
- test/
|
65
|
+
- test/cli_test.rb
|
66
|
+
- test/client_test.rb
|
67
|
+
- test/config.yml
|
68
|
+
- test/fake_env.rb
|
69
|
+
- test/manager_test.rb
|
70
|
+
- test/support/my_worker.rb
|
62
71
|
- test/test_helper.rb
|
63
72
|
homepage: https://github.com/yabawock/sidekiq-scheduler
|
64
73
|
licenses: []
|
@@ -74,7 +83,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
74
83
|
version: '0'
|
75
84
|
segments:
|
76
85
|
- 0
|
77
|
-
hash: -
|
86
|
+
hash: -2626338027335115584
|
78
87
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
79
88
|
none: false
|
80
89
|
requirements:
|
@@ -83,7 +92,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
83
92
|
version: '0'
|
84
93
|
segments:
|
85
94
|
- 0
|
86
|
-
hash: -
|
95
|
+
hash: -2626338027335115584
|
87
96
|
requirements: []
|
88
97
|
rubyforge_project:
|
89
98
|
rubygems_version: 1.8.17
|
@@ -91,5 +100,10 @@ signing_key:
|
|
91
100
|
specification_version: 3
|
92
101
|
summary: Light weight job scheduling extension for Sidekiq
|
93
102
|
test_files:
|
94
|
-
- test/
|
103
|
+
- test/cli_test.rb
|
104
|
+
- test/client_test.rb
|
105
|
+
- test/config.yml
|
106
|
+
- test/fake_env.rb
|
107
|
+
- test/manager_test.rb
|
108
|
+
- test/support/my_worker.rb
|
95
109
|
- test/test_helper.rb
|