sidekiq-cron 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 0e24b22d1f58e374c9ea85f606f3a86e01c204c6
4
+ data.tar.gz: af7f0ea32e39568a6f04981703088cf9c5fb4692
5
+ SHA512:
6
+ metadata.gz: 4283ebe1d86984162381e640bad0d447b7fd2b4b25fd24f0a5750381fcd2aa74ab1432c3918015789bd549635928afbe84a102956c52a0ba79e7383850154854
7
+ data.tar.gz: 38e0734ff2253e123fe012687153cad465a06b187cda5806ac05baac3984ef24fbb14b072f522d2e759b821b0989c6cd4ca538c249efb43067946635d70d2911
@@ -1,10 +1,9 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 1.9.3
4
3
  - jruby-19mode
5
4
  - rbx-19mode
6
5
  - 2.0.0
7
- - 2.1.0
6
+ - 2.2.2
8
7
  services:
9
8
  - redis-server
10
9
  branches:
@@ -19,4 +18,4 @@ env:
19
18
  matrix:
20
19
  allow_failures:
21
20
  - rvm: jruby-19mode
22
- - rvm: rbx-19mode
21
+ - rvm: rbx-19mode
data/Changes.md CHANGED
@@ -1,8 +1,13 @@
1
+ v 0.3.0
2
+ -------
1
3
 
4
+ - suport for Active Job
5
+ - sidekiq cron web ui needs to be loaded by: require 'sidekiq/cron/web'
6
+ - add load_from_hash! and load_from_array! which cleanup jobs before adding new ones
2
7
 
3
8
  v 0.1.1
4
9
  -------
5
10
 
6
11
  - add Web fontend with enabled/disable job, unqueue now, delete job
7
12
  - add cron poller - enqueu cro jobs
8
- - add cron job - save all needed data to redis
13
+ - add cron job - save all needed data to redis
data/Gemfile CHANGED
@@ -1,30 +1,31 @@
1
1
  source 'https://rubygems.org'
2
2
 
3
3
  gem 'tilt' , '< 2.0.0'
4
- gem "sidekiq", ">= 2.17.3"
4
+ gem 'sidekiq', '>= 2.17.3'
5
5
  gem 'rufus-scheduler', '>= 2.0.24'
6
6
 
7
7
  group :development do
8
- gem "bundler"
9
- gem "simplecov"
8
+ gem 'bundler'
9
+ gem 'simplecov'
10
10
 
11
11
  gem 'shoulda-context'
12
- gem "turn"
13
12
 
14
13
  gem 'rack'
15
14
  gem 'rack-test'
16
15
 
17
- gem "jeweler", "~> 1.8.3"
16
+ gem 'jeweler', '~> 1.8.3'
18
17
 
19
- gem "sdoc" # sdoc -N .
18
+ gem 'minitest'
19
+ gem 'test-unit'
20
+ gem 'sdoc' # sdoc -N .
20
21
 
21
- gem "slim"
22
- gem "sinatra"
22
+ gem 'slim'
23
+ gem 'sinatra'
23
24
 
24
25
  gem 'mocha'
25
26
  gem 'coveralls'
26
27
 
27
- gem "shotgun"
28
+ gem 'shotgun'
28
29
 
29
30
  # gem 'guard'
30
31
  # gem 'guard-minitest'
data/README.md CHANGED
@@ -2,16 +2,28 @@ Sidekiq-Cron [![Gem Version](https://badge.fury.io/rb/sidekiq-cron.png)](http://
2
2
  ================================================================================================================================================================================================================================================================================================================================================================================================================================================
3
3
 
4
4
 
5
- Add-on for [Sidekiq](http://sidekiq.org)
5
+ An scheduling add-on for [Sidekiq](http://sidekiq.org).
6
6
 
7
- Allows you to schedule recurring jobs for sidekiq workers using cron notation _* * * * *_.
7
+ Runs a thread along side Sidekiq workers to schedule jobs at specified times (using cron notation `* * * * *` parsed by [Rufus-Scheduler](https://github.com/jmettraux/rufus-scheduler), more about [cron notation](http://www.nncron.ru/help/EN/working/cron-format.htm).
8
+
9
+ Checks for new jobs to schedule every 10 seconds and doesn't schedule the same job multiple times when more than one Sidekiq worker is running.
10
+
11
+ Scheduling jobs are added only when at least one sidekiq process is running.
12
+
13
+ If you want to know how scheduling work check [out under the hood](#under-the-hood)
14
+
15
+ Works with Active jobs (Rails 4.2+)
8
16
 
9
17
  Requirements
10
18
  -----------------
11
19
 
12
20
  - Redis 2.4 or greater is required.
13
- - Sidekiq 2.13.1 or grater is required.
21
+ - Sidekiq 2.17.3 or grater is required.
14
22
 
23
+ Change Log
24
+ ----------
25
+ before upgrading to new version pls read:
26
+ [Change Log](https://github.com/ondrejbartas/sidekiq-cron/raw/master/Changes.md)
15
27
 
16
28
  Installation
17
29
  ------------
@@ -20,7 +32,7 @@ Installation
20
32
 
21
33
  or add to your Gemfile
22
34
 
23
- gem "sidekiq-cron", "~> 0.1.0"
35
+ gem "sidekiq-cron", "~> 0.3.0"
24
36
 
25
37
 
26
38
  Getting Started
@@ -42,6 +54,35 @@ _Job properties_:
42
54
  }
43
55
  ```
44
56
 
57
+ ### Time, cron and sidekiq-cron
58
+
59
+ Cron line is allways evaluated against UTC time. So if you are Prague (timezone +02:00) and you want job to be qneueued at 8:30 AM
60
+ You will need to adjust cronline to `30 6 * * *`.
61
+
62
+ ### What objects/classes can be scheduled
63
+ #### Sidekiq Worker
64
+ In example we were using: `HardWorker` which loooks like:
65
+ ```ruby
66
+ class HardWorker
67
+ include Sidekiq::Worker
68
+ def perform(*args)
69
+ # do something
70
+ end
71
+ end
72
+ ```
73
+
74
+ #### Active Job Worker
75
+ You can schedule: `ExampleJob` which loooks like:
76
+ ```ruby
77
+ class ExampleJob < ActiveJob::Base
78
+ queue_as :default
79
+
80
+ def perform(*args)
81
+ # Do something
82
+ end
83
+ end
84
+ ```
85
+
45
86
  #### Adding Cron job:
46
87
  ```ruby
47
88
 
@@ -111,9 +152,16 @@ array = [
111
152
  Sidekiq::Cron::Job.load_from_array array
112
153
  ```
113
154
 
155
+ Bang methods will remove jobs that are not present in given hash/array
156
+ updates jobs with same names and create new ones.
157
+ ```ruby
158
+ Sidekiq::Cron::Job#load_from_hash! hash
159
+ Sidekiq::Cron::Job#load_from_array! array
160
+ ```
161
+
114
162
  or from YML (same notation as Resque-scheduler)
115
163
  ```yaml
116
- #config/shedule.yml
164
+ #config/schedule.yml
117
165
 
118
166
  my_first_job:
119
167
  cron: "*/5 * * * *"
@@ -124,7 +172,7 @@ second_job:
124
172
  cron: "*/30 * * * *"
125
173
  class: "HardWorker"
126
174
  queue: hard_worker_long
127
- args:
175
+ args:
128
176
  hard: "stuff"
129
177
  ```
130
178
 
@@ -137,8 +185,6 @@ if File.exists?(schedule_file)
137
185
  end
138
186
  ```
139
187
 
140
-
141
-
142
188
  #### Finding jobs
143
189
  ```ruby
144
190
  #return array of all jobs
@@ -191,10 +237,38 @@ Just start sidekiq workers by:
191
237
  ### Web Ui for Cron Jobs
192
238
 
193
239
  If you are using sidekiq web ui and you would like to add cron josb to web too,
194
- add `require 'sidekiq-cron'` after `require 'sidekiq/web'`.
240
+ add `require 'sidekiq/cron/web'` after `require 'sidekiq/web'`.
195
241
  By this you will get:
196
242
  ![Web UI](https://github.com/ondrejbartas/sidekiq-cron/raw/master/examples/web-cron-ui.png)
197
243
 
244
+ ### Forking Processes
245
+
246
+ If you're using a forking web server like Unicorn you may run into an issue where the Redis connection is used
247
+ before the process forks, causing the following exception
248
+
249
+ Redis::InheritedError: Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.
250
+
251
+ to occcur. To avoid this, wrap your job creation in the a call to `Sidekiq.configure_server`:
252
+
253
+ ```ruby
254
+ Sidekiq.configure_server do |config|
255
+ schedule_file = "config/schedule.yml"
256
+
257
+ if File.exists?(schedule_file)
258
+ Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
259
+ end
260
+ end
261
+ ```
262
+
263
+ Note that this API is only available in Sidekiq 3.x.x.
264
+
265
+ ## Under the hood
266
+
267
+ When you start sidekiq process it starts one thread with Sidekiq::Poller instance, which perform adding of scheduled jobs to queues, retryes etc.
268
+
269
+ Sidekiq-Cron add itself into this start procedure and start another thread with Sidekiq::Cron::Poler which checks all enabled sidekiq cron jobs evry 10 seconds,
270
+ if they should be added to queue (their cronline matches time of check).
271
+
198
272
 
199
273
 
200
274
  ## Contributing to sidekiq-cron
data/Rakefile CHANGED
@@ -1,7 +1,9 @@
1
1
  # encoding: utf-8
2
2
 
3
3
  require 'rubygems'
4
+ require 'bundler/setup'
4
5
  require 'bundler'
6
+
5
7
  begin
6
8
  Bundler.setup(:default, :development)
7
9
  rescue Bundler::BundlerError => e
@@ -58,4 +60,4 @@ namespace :test do
58
60
  t.warning = false
59
61
  t.verbose = false
60
62
  end
61
- end
63
+ end
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.2.0
1
+ 0.3.0
@@ -1,14 +1,8 @@
1
- begin
2
- require "sidekiq/web"
3
- rescue LoadError
4
- # client-only usage
5
- end
6
-
7
1
  require "sidekiq/cron/job"
8
- require "sidekiq/cron/web_extension"
9
2
 
10
3
  #require poller only if celluloid is defined
11
4
  if defined?(Celluloid)
5
+ require 'celluloid/autostart'
12
6
  require "sidekiq/cron/poller"
13
7
  require "sidekiq/cron/launcher"
14
8
  end
@@ -17,14 +11,3 @@ module Sidekiq
17
11
  module Cron
18
12
  end
19
13
  end
20
-
21
- if defined?(Sidekiq::Web)
22
- Sidekiq::Web.register Sidekiq::Cron::WebExtension
23
-
24
- if Sidekiq::Web.tabs.is_a?(Array)
25
- # For sidekiq < 2.5
26
- Sidekiq::Web.tabs << "cron"
27
- else
28
- Sidekiq::Web.tabs["Cron"] = "cron"
29
- end
30
- end
@@ -11,7 +11,7 @@ module Sidekiq
11
11
  extend Util
12
12
 
13
13
  #how long we would like to store informations about previous enqueues
14
- REMEMBER_THRESHOLD = 24 * 60 * 60
14
+ REMEMBER_THRESHOLD = 24 * 60 * 60
15
15
 
16
16
  #crucial part of whole enquing job
17
17
  def should_enque? time
@@ -45,12 +45,36 @@ module Sidekiq
45
45
  def enque! time = Time.now
46
46
  @last_enqueue_time = time
47
47
 
48
- Sidekiq::Client.push(@message.is_a?(String) ? Sidekiq.load_json(@message) : @message)
48
+ if defined?(ActiveJob::Base) && @klass.to_s.constantize < ActiveJob::Base
49
+ Sidekiq::Client.push(active_job_message)
50
+ else
51
+ Sidekiq::Client.push(sidekiq_worker_message)
52
+ end
49
53
 
50
54
  save
51
55
  logger.debug { "enqueued #{@name}: #{@message}" }
52
56
  end
53
57
 
58
+ # siodekiq worker message
59
+ def sidekiq_worker_message
60
+ @message.is_a?(String) ? Sidekiq.load_json(@message) : @message
61
+ end
62
+
63
+ # active job has different structure how it is loading data from sidekiq
64
+ # queue, it createaswrapper arround job
65
+ def active_job_message
66
+ {
67
+ 'class' => 'ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper',
68
+ 'queue' => @queue,
69
+ 'args' => [{
70
+ 'job_class' => @klass,
71
+ 'job_id' => SecureRandom.uuid,
72
+ 'queue_name' => @queue,
73
+ 'arguments' => @args
74
+ }]
75
+ }
76
+ end
77
+
54
78
  # load cron jobs from Hash
55
79
  # input structure should look like:
56
80
  # {
@@ -73,6 +97,12 @@ module Sidekiq
73
97
  load_from_array array
74
98
  end
75
99
 
100
+ # like to {#load_from_hash}
101
+ # If exists old jobs in redis but removed from args, destroy old jobs
102
+ def self.load_from_hash! hash
103
+ destroy_removed_jobs(hash.keys)
104
+ load_from_hash(hash)
105
+ end
76
106
 
77
107
  # load cron jobs from Array
78
108
  # input structure should look like:
@@ -99,6 +129,13 @@ module Sidekiq
99
129
  errors
100
130
  end
101
131
 
132
+ # like to {#load_from_array}
133
+ # If exists old jobs in redis but removed from args, destroy old jobs
134
+ def self.load_from_array! array
135
+ job_names = array.map { |job| job["name"] }
136
+ destroy_removed_jobs(job_names)
137
+ load_from_array(array)
138
+ end
102
139
 
103
140
  # get all cron jobs
104
141
  def self.all
@@ -194,7 +231,7 @@ module Sidekiq
194
231
  when String
195
232
  begin
196
233
  @klass.constantize.get_sidekiq_options.merge(message_data)
197
- rescue
234
+ rescue
198
235
  #Unknown class
199
236
  message_data.merge("queue"=>"default")
200
237
  end
@@ -203,7 +240,11 @@ module Sidekiq
203
240
 
204
241
  #override queue if setted in config
205
242
  #only if message is hash - can be string (dumped JSON)
206
- message_data['queue'] = args['queue'] if args['queue']
243
+ if args['queue']
244
+ @queue = message_data['queue'] = args['queue']
245
+ else
246
+ @queue = message_data['queue'] || default
247
+ end
207
248
 
208
249
  #dump message as json
209
250
  @message = message_data
@@ -211,7 +252,7 @@ module Sidekiq
211
252
 
212
253
  end
213
254
 
214
- def status
255
+ def status
215
256
  @status
216
257
  end
217
258
 
@@ -258,7 +299,7 @@ module Sidekiq
258
299
  }
259
300
  end
260
301
 
261
- def errors
302
+ def errors
262
303
  @errors ||= []
263
304
  end
264
305
 
@@ -268,9 +309,9 @@ module Sidekiq
268
309
 
269
310
  errors << "'name' must be set" if @name.nil? || @name.size == 0
270
311
  if @cron.nil? || @cron.size == 0
271
- errors << "'cron' must be set"
312
+ errors << "'cron' must be set"
272
313
  else
273
- begin
314
+ begin
274
315
  cron = Rufus::Scheduler::CronLine.new(@cron)
275
316
  cron.next_time(Time.now)
276
317
  rescue Exception => e
@@ -283,11 +324,21 @@ module Sidekiq
283
324
  end
284
325
  end
285
326
 
286
- errors << "'klass' (or class) must be set" if @klass.nil? || @klass.size == 0
327
+ errors << "'klass' (or class) must be set" unless klass_valid
287
328
 
288
329
  !errors.any?
289
330
  end
290
331
 
332
+ def klass_valid
333
+ case @klass
334
+ when Class
335
+ true
336
+ when String
337
+ @klass.size > 0
338
+ else
339
+ end
340
+ end
341
+
291
342
  # add job to cron jobs
292
343
  # input:
293
344
  # name: (string) - name of job
@@ -315,7 +366,7 @@ module Sidekiq
315
366
  end
316
367
  logger.info { "Cron Jobs - add job with name: #{@name}" }
317
368
  end
318
-
369
+
319
370
  # remove job from cron jobs by name
320
371
  # input:
321
372
  # first arg: name (string) - name of job (must be same - case sensitive)
@@ -323,7 +374,7 @@ module Sidekiq
323
374
  Sidekiq.redis do |conn|
324
375
  #delete from set
325
376
  conn.srem self.class.jobs_key, redis_key
326
-
377
+
327
378
  #delete runned timestamps
328
379
  conn.del job_enqueued_key
329
380
 
@@ -341,6 +392,14 @@ module Sidekiq
341
392
  logger.info { "Cron Jobs - deleted all jobs" }
342
393
  end
343
394
 
395
+ # remove "removed jobs" between current jobs and new jobs
396
+ def self.destroy_removed_jobs new_job_names
397
+ current_job_names = Sidekiq::Cron::Job.all.map(&:name)
398
+ removed_job_names = current_job_names - new_job_names
399
+ removed_job_names.each { |j| Sidekiq::Cron::Job.destroy(j) }
400
+ removed_job_names
401
+ end
402
+
344
403
  # Parse cron specification '* * * * *' and returns
345
404
  # time when last run should be performed
346
405
  def last_time now = Time.now
@@ -372,7 +431,7 @@ module Sidekiq
372
431
  def not_enqueued_after?(time)
373
432
  @last_enqueue_time.nil? || @last_enqueue_time < last_time(time)
374
433
  end
375
-
434
+
376
435
  # Try parsing inbound args into an array.
377
436
  # args from Redis will be encoded JSON;
378
437
  # try to load JSON, then failover