mini_scheduler 0.14.0 → 0.16.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (10) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +9 -0
  3. data/Gemfile +0 -2
  4. data/README.md +20 -11
  5. data/Rakefile +1 -0
  6. data/lib/mini_scheduler/manager.rb +66 -30
  7. data/lib/mini_scheduler/schedule_info.rb +2 -2
  8. data/lib/mini_scheduler/version.rb +1 -1
  9. data/mini_scheduler.gemspec +12 -10
  10. metadata +42 -36
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 319678529749db26d855d097741f1ff271bb3f19e87276eb764ede430f831d0e
4
- data.tar.gz: e01a47b807bb03ee1f2f97a36f55d95663cb9a79b5cb87bb6fef64c45969c961
3
+ metadata.gz: e4a38d3c8bcf38814218e73edefc423c504c8469b3c9e3c05a08bb6b0012dd47
4
+ data.tar.gz: 13a72f979b4007dd48910a5b03efafab049643e2944058be2a01602855e87393
5
5
  SHA512:
6
- metadata.gz: 581b32d6f15869169797af99f68c3ffc4c137e54404b77a2f62948d8e448e822821fa24a56571d66355c96e16d2e38e321995b2b9f606e01caaeeec8aa8bc74a
7
- data.tar.gz: 81467c469e97e95510dff3749d3efeedd7eb58344bb7ff30357979c05b8398fc9a75f282f8a8a60f0f8241861edafbcf4a85ff1091557f44236cc19d6fba5a01
6
+ metadata.gz: d748b674b67d3faf5e1507d5b51d2dfaa1585cf9315d93f8b634faaaf9d2942667f17b345ace2375cfa452412ddb59aac67ed98bb704de831f924f9f4545b700
7
+ data.tar.gz: 0cae809fb72df9043374a451cfa8e906e434a460201e3383bd54c60c2e4acc92147a83af8887377f3ba42bd661b104a9b7f1e500a80a1c249683fb72f406dbca
data/CHANGELOG.md CHANGED
@@ -1,3 +1,12 @@
1
+ # 0.16.0 - 2023-05-17
2
+
3
+ - Support Redis gem version 5
4
+
5
+ # 0.15.0 - 2022-11-17
6
+
7
+ - Fix data inconsistencies when Redis fails during jobs (#19)
8
+ - Update minimum Ruby version to 2.7
9
+
1
10
  # 0.14.0 - 2022-06-20
2
11
 
3
12
  - Fix compatibility with Sidekiq 6.5 (#15)
data/Gemfile CHANGED
@@ -1,6 +1,4 @@
1
1
  # frozen_string_literal: true
2
2
  source 'https://rubygems.org'
3
3
 
4
- git_source(:github) { 'https://github.com/discourse/mini_scheduler' }
5
-
6
4
  gemspec
data/README.md CHANGED
@@ -1,17 +1,15 @@
1
1
  [![Build Status](https://github.com/discourse/mini_scheduler/workflows/CI/badge.svg)](https://github.com/discourse/mini_scheduler/actions)
2
2
  [![Gem Version](https://badge.fury.io/rb/mini_scheduler.svg)](https://rubygems.org/gems/mini_scheduler)
3
3
 
4
- # mini_scheduler
4
+ # MiniScheduler
5
5
 
6
6
  MiniScheduler adds recurring jobs to [Sidekiq](https://sidekiq.org/).
7
7
 
8
-
9
-
10
8
  ## Installation
11
9
 
12
10
  Add this line to your application's Gemfile:
13
11
 
14
- ```ruby
12
+ ```rb
15
13
  gem 'mini_scheduler'
16
14
  ```
17
15
 
@@ -25,8 +23,8 @@ Or install it yourself as:
25
23
 
26
24
  In a Rails application, create files needed in your application to configure mini_scheduler:
27
25
 
28
- bin/rails g mini_scheduler:install
29
- rake db:migrate
26
+ $ bin/rails g mini_scheduler:install
27
+ $ bin/rails db:migrate
30
28
 
31
29
  An initializer is created named `config/initializers/mini_scheduler.rb` which lists all the configuration options.
32
30
 
@@ -34,7 +32,7 @@ An initializer is created named `config/initializers/mini_scheduler.rb` which li
34
32
 
35
33
  By default each instance of MiniScheduler will run with a single worker. To amend this behavior:
36
34
 
37
- ```
35
+ ```rb
38
36
  if Sidekiq.server? && defined?(Rails)
39
37
  Rails.application.config.after_initialize do
40
38
  MiniScheduler.start(workers: 5)
@@ -48,7 +46,7 @@ This is useful for cases where you have extremely long running tasks that you wo
48
46
 
49
47
  Create jobs with a recurring schedule like this:
50
48
 
51
- ```ruby
49
+ ```rb
52
50
  class MyHourlyJob
53
51
  include Sidekiq::Worker
54
52
  extend MiniScheduler::Schedule
@@ -63,12 +61,23 @@ end
63
61
 
64
62
  Options for schedules:
65
63
 
66
- * **queue** followed by a queue name, like "queue :email", default queue is "default"
67
- * **every** followed by a duration in seconds, like "every 1.hour".
68
- * **daily at:** followed by a duration since midnight, like "daily at: 12.hours", to run only once per day at a specific time.
64
+ - **queue** followed by a queue name, like "queue :email", default queue is "default"
65
+ - **every** followed by a duration in seconds, like "every 1.hour".
66
+ - **daily at:** followed by a duration since midnight, like "daily at: 12.hours", to run only once per day at a specific time.
69
67
 
70
68
  To view the scheduled jobs, their history, and the schedule, go to sidekiq's web UI and look for the "Scheduler" tab at the top.
71
69
 
70
+ To enable this view in Sidekiq, add `require "mini_scheduler/web"` to `routes.rb`:
71
+
72
+ ```rb
73
+ require "sidekiq/web"
74
+ require "mini_scheduler/web"
75
+
76
+ Rails.application.routes.draw do
77
+ ...
78
+ end
79
+ ```
80
+
72
81
  ## How to reach us
73
82
 
74
83
  If you have questions about using mini_scheduler or found a problem, you can find us at https://meta.discourse.org.
data/Rakefile CHANGED
@@ -14,6 +14,7 @@ rescue Bundler::BundlerError => error
14
14
  end
15
15
 
16
16
  RSpec::Core::RakeTask.new(:spec)
17
+ task test: :spec
17
18
 
18
19
  desc "Default: run tests"
19
20
  task default: [ :spec ]
data/lib/mini_scheduler/manager.rb CHANGED
@@ -19,6 +19,7 @@ module MiniScheduler
19
19
  @mutex.synchronize do
20
20
  repair_queue
21
21
  reschedule_orphans
22
+ ensure_worker_threads
22
23
  end
23
24
  end
24
25
  end
@@ -30,40 +31,67 @@ module MiniScheduler
30
31
  sleep (@manager.keep_alive_duration / 2)
31
32
  end
32
33
  end
33
- @threads = []
34
- manager.workers.times do
35
- @threads << Thread.new do
36
- while !@stopped
37
- process_queue
38
- end
39
- end
40
- end
34
+ ensure_worker_threads
41
35
  end
42
36
 
43
- def keep_alive
44
- @manager.keep_alive
37
+ def keep_alive(*ids)
38
+ @manager.keep_alive(*ids)
45
39
  rescue => ex
46
- MiniScheduler.handle_job_exception(ex, message: "Scheduling manager keep-alive")
40
+ MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler keep_alive")
47
41
  end
48
42
 
49
43
  def repair_queue
50
44
  @manager.repair_queue
51
45
  rescue => ex
52
- MiniScheduler.handle_job_exception(ex, message: "Scheduling manager queue repair")
46
+ MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler repair_queue")
53
47
  end
54
48
 
55
49
  def reschedule_orphans
56
50
  @manager.reschedule_orphans!
57
51
  rescue => ex
58
- MiniScheduler.handle_job_exception(ex, message: "Scheduling manager orphan rescheduler")
52
+ MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler reschedule_orphans")
53
+ end
54
+
55
+ def ensure_worker_threads
56
+ @threads ||= []
57
+ @threads.delete_if { |t| !t.alive? }
58
+ (@manager.workers - @threads.size).times do
59
+ @threads << Thread.new { worker_loop }
60
+ end
61
+ rescue => ex
62
+ MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler ensure_worker_threads")
63
+ end
64
+
65
+ def worker_loop
66
+ set_current_worker_thread_id!
67
+ keep_alive(current_worker_thread_id)
68
+ while !@stopped
69
+ begin
70
+ process_queue
71
+ rescue => ex
72
+ MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler worker_loop")
73
+ break # Data could be in a bad state - stop the thread
74
+ end
75
+ end
59
76
  end
60
77
 
61
78
  def hostname
62
79
  @hostname
63
80
  end
64
81
 
65
- def process_queue
82
+ def current_worker_thread_id
83
+ Thread.current[:mini_scheduler_worker_thread_id]
84
+ end
85
+
86
+ def set_current_worker_thread_id!
87
+ Thread.current[:mini_scheduler_worker_thread_id] = "#{@manager.identity_key}:thread_#{SecureRandom.alphanumeric(10)}"
88
+ end
66
89
 
90
+ def worker_thread_ids
91
+ @threads.filter(&:alive?).filter_map { |t| t[:mini_scheduler_worker_thread_id] }
92
+ end
93
+
94
+ def process_queue
67
95
  klass = @queue.deq
68
96
  # hack alert, I need to both deq and set @running atomically.
69
97
  @running = true
@@ -78,6 +106,7 @@ module MiniScheduler
78
106
 
79
107
  begin
80
108
  info.prev_result = "RUNNING"
109
+ info.current_owner = current_worker_thread_id
81
110
  @mutex.synchronize { info.write! }
82
111
 
83
112
  if @manager.enable_stats
@@ -92,7 +121,7 @@ module MiniScheduler
92
121
 
93
122
  klass.new.perform
94
123
  rescue => e
95
- MiniScheduler.handle_job_exception(e, message: "Running a scheduled job", job: { "class" => klass })
124
+ MiniScheduler.handle_job_exception(e, message: "Error while running a scheduled job", job: { "class" => klass })
96
125
 
97
126
  error = "#{e.class}: #{e.message} #{e.backtrace.join("\n")}"
98
127
  failed = true
@@ -113,8 +142,6 @@ module MiniScheduler
113
142
  attempts(3) do
114
143
  @mutex.synchronize { info.write! }
115
144
  end
116
- rescue => ex
117
- MiniScheduler.handle_job_exception(ex, message: "Processing scheduled job queue")
118
145
  ensure
119
146
  @running = false
120
147
  if defined?(ActiveRecord::Base)
@@ -163,14 +190,16 @@ module MiniScheduler
163
190
  end
164
191
  end
165
192
 
166
- def attempts(n)
167
- n.times {
168
- begin
169
- yield; break
170
- rescue
171
- sleep Random.rand
172
- end
173
- }
193
+ def attempts(max_attempts)
194
+ attempt = 0
195
+ begin
196
+ yield
197
+ rescue
198
+ attempt += 1
199
+ raise if attempt >= max_attempts
200
+ sleep Random.rand
201
+ retry
202
+ end
174
203
  end
175
204
 
176
205
  end
@@ -314,8 +343,11 @@ module MiniScheduler
314
343
  60
315
344
  end
316
345
 
317
- def keep_alive
318
- redis.setex identity_key, keep_alive_duration, ""
346
+ def keep_alive(*ids)
347
+ ids = [identity_key, *@runner.worker_thread_ids] if ids.size == 0
348
+ ids.each do |identity_key|
349
+ redis.setex identity_key, keep_alive_duration, ""
350
+ end
319
351
  end
320
352
 
321
353
  def lock
@@ -344,16 +376,20 @@ module MiniScheduler
344
376
  schedules
345
377
  end
346
378
 
347
- @mutex = Mutex.new
379
+ @class_mutex = Mutex.new
348
380
  def self.seq
349
- @mutex.synchronize do
381
+ @class_mutex.synchronize do
350
382
  @i ||= 0
351
383
  @i += 1
352
384
  end
353
385
  end
354
386
 
387
+ @@identity_key_mutex = Mutex.new
355
388
  def identity_key
356
- @identity_key ||= "_scheduler_#{hostname}:#{Process.pid}:#{self.class.seq}:#{SecureRandom.hex}"
389
+ return @identity_key if @identity_key
390
+ @@identity_key_mutex.synchronize do
391
+ @identity_key ||= "_scheduler_#{hostname}:#{Process.pid}:#{self.class.seq}:#{SecureRandom.hex}"
392
+ end
357
393
  end
358
394
 
359
395
  def self.lock_key(queue)
data/lib/mini_scheduler/schedule_info.rb CHANGED
@@ -104,7 +104,7 @@ module MiniScheduler
104
104
  current_owner: @current_owner
105
105
  }.to_json
106
106
 
107
- redis.zadd queue_key, @next_run, @klass if @next_run
107
+ redis.zadd queue_key, @next_run.to_s, @klass.to_s if @next_run
108
108
  end
109
109
 
110
110
  def del!
@@ -135,7 +135,7 @@ module MiniScheduler
135
135
  private
136
136
  def clear!
137
137
  redis.del key
138
- redis.zrem queue_key, @klass
138
+ redis.zrem queue_key, @klass.to_s
139
139
  end
140
140
 
141
141
  end
data/lib/mini_scheduler/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module MiniScheduler
4
- VERSION = "0.14.0"
4
+ VERSION = "0.16.0"
5
5
  end
data/mini_scheduler.gemspec CHANGED
@@ -15,18 +15,20 @@ Gem::Specification.new do |spec|
15
15
  spec.homepage = "https://github.com/discourse/mini_scheduler"
16
16
  spec.license = "MIT"
17
17
 
18
+ spec.required_ruby_version = ">= 2.7.0"
19
+
18
20
  spec.files = `git ls-files`.split($/).reject { |s| s =~ /^(spec|\.)/ }
19
21
  spec.require_paths = ["lib"]
20
22
 
21
- spec.add_dependency "sidekiq", ">= 4.2.3"
23
+ spec.add_runtime_dependency "sidekiq", ">= 4.2.3", "< 7.0"
22
24
 
23
- spec.add_development_dependency "pg", ">= 1.0"
24
- spec.add_development_dependency "activesupport", ">= 5.2"
25
- spec.add_development_dependency "rspec"
26
- spec.add_development_dependency "mocha"
27
- spec.add_development_dependency "guard"
28
- spec.add_development_dependency "guard-rspec"
29
- spec.add_development_dependency "mock_redis"
30
- spec.add_development_dependency "rake"
31
- spec.add_development_dependency 'rubocop-discourse'
25
+ spec.add_development_dependency "pg", "~> 1.0"
26
+ spec.add_development_dependency "activesupport", "~> 7.0"
27
+ spec.add_development_dependency "rspec", "~> 3.0"
28
+ spec.add_development_dependency "mocha", "~> 2.0"
29
+ spec.add_development_dependency "guard", "~> 2.0"
30
+ spec.add_development_dependency "guard-rspec", "~> 4.0"
31
+ spec.add_development_dependency "redis", ">= 4.0"
32
+ spec.add_development_dependency "rake", "~> 13.0"
33
+ spec.add_development_dependency "rubocop-discourse", "= 3.2.0"
32
34
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: mini_scheduler
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.14.0
4
+ version: 0.16.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sam Saffron
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2022-06-20 00:00:00.000000000 Z
12
+ date: 2023-05-17 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: sidekiq
@@ -18,6 +18,9 @@ dependencies:
18
18
  - - ">="
19
19
  - !ruby/object:Gem::Version
20
20
  version: 4.2.3
21
+ - - "<"
22
+ - !ruby/object:Gem::Version
23
+ version: '7.0'
21
24
  type: :runtime
22
25
  prerelease: false
23
26
  version_requirements: !ruby/object:Gem::Requirement
@@ -25,132 +28,135 @@ dependencies:
25
28
  - - ">="
26
29
  - !ruby/object:Gem::Version
27
30
  version: 4.2.3
31
+ - - "<"
32
+ - !ruby/object:Gem::Version
33
+ version: '7.0'
28
34
  - !ruby/object:Gem::Dependency
29
35
  name: pg
30
36
  requirement: !ruby/object:Gem::Requirement
31
37
  requirements:
32
- - - ">="
38
+ - - "~>"
33
39
  - !ruby/object:Gem::Version
34
40
  version: '1.0'
35
41
  type: :development
36
42
  prerelease: false
37
43
  version_requirements: !ruby/object:Gem::Requirement
38
44
  requirements:
39
- - - ">="
45
+ - - "~>"
40
46
  - !ruby/object:Gem::Version
41
47
  version: '1.0'
42
48
  - !ruby/object:Gem::Dependency
43
49
  name: activesupport
44
50
  requirement: !ruby/object:Gem::Requirement
45
51
  requirements:
46
- - - ">="
52
+ - - "~>"
47
53
  - !ruby/object:Gem::Version
48
- version: '5.2'
54
+ version: '7.0'
49
55
  type: :development
50
56
  prerelease: false
51
57
  version_requirements: !ruby/object:Gem::Requirement
52
58
  requirements:
53
- - - ">="
59
+ - - "~>"
54
60
  - !ruby/object:Gem::Version
55
- version: '5.2'
61
+ version: '7.0'
56
62
  - !ruby/object:Gem::Dependency
57
63
  name: rspec
58
64
  requirement: !ruby/object:Gem::Requirement
59
65
  requirements:
60
- - - ">="
66
+ - - "~>"
61
67
  - !ruby/object:Gem::Version
62
- version: '0'
68
+ version: '3.0'
63
69
  type: :development
64
70
  prerelease: false
65
71
  version_requirements: !ruby/object:Gem::Requirement
66
72
  requirements:
67
- - - ">="
73
+ - - "~>"
68
74
  - !ruby/object:Gem::Version
69
- version: '0'
75
+ version: '3.0'
70
76
  - !ruby/object:Gem::Dependency
71
77
  name: mocha
72
78
  requirement: !ruby/object:Gem::Requirement
73
79
  requirements:
74
- - - ">="
80
+ - - "~>"
75
81
  - !ruby/object:Gem::Version
76
- version: '0'
82
+ version: '2.0'
77
83
  type: :development
78
84
  prerelease: false
79
85
  version_requirements: !ruby/object:Gem::Requirement
80
86
  requirements:
81
- - - ">="
87
+ - - "~>"
82
88
  - !ruby/object:Gem::Version
83
- version: '0'
89
+ version: '2.0'
84
90
  - !ruby/object:Gem::Dependency
85
91
  name: guard
86
92
  requirement: !ruby/object:Gem::Requirement
87
93
  requirements:
88
- - - ">="
94
+ - - "~>"
89
95
  - !ruby/object:Gem::Version
90
- version: '0'
96
+ version: '2.0'
91
97
  type: :development
92
98
  prerelease: false
93
99
  version_requirements: !ruby/object:Gem::Requirement
94
100
  requirements:
95
- - - ">="
101
+ - - "~>"
96
102
  - !ruby/object:Gem::Version
97
- version: '0'
103
+ version: '2.0'
98
104
  - !ruby/object:Gem::Dependency
99
105
  name: guard-rspec
100
106
  requirement: !ruby/object:Gem::Requirement
101
107
  requirements:
102
- - - ">="
108
+ - - "~>"
103
109
  - !ruby/object:Gem::Version
104
- version: '0'
110
+ version: '4.0'
105
111
  type: :development
106
112
  prerelease: false
107
113
  version_requirements: !ruby/object:Gem::Requirement
108
114
  requirements:
109
- - - ">="
115
+ - - "~>"
110
116
  - !ruby/object:Gem::Version
111
- version: '0'
117
+ version: '4.0'
112
118
  - !ruby/object:Gem::Dependency
113
- name: mock_redis
119
+ name: redis
114
120
  requirement: !ruby/object:Gem::Requirement
115
121
  requirements:
116
122
  - - ">="
117
123
  - !ruby/object:Gem::Version
118
- version: '0'
124
+ version: '4.0'
119
125
  type: :development
120
126
  prerelease: false
121
127
  version_requirements: !ruby/object:Gem::Requirement
122
128
  requirements:
123
129
  - - ">="
124
130
  - !ruby/object:Gem::Version
125
- version: '0'
131
+ version: '4.0'
126
132
  - !ruby/object:Gem::Dependency
127
133
  name: rake
128
134
  requirement: !ruby/object:Gem::Requirement
129
135
  requirements:
130
- - - ">="
136
+ - - "~>"
131
137
  - !ruby/object:Gem::Version
132
- version: '0'
138
+ version: '13.0'
133
139
  type: :development
134
140
  prerelease: false
135
141
  version_requirements: !ruby/object:Gem::Requirement
136
142
  requirements:
137
- - - ">="
143
+ - - "~>"
138
144
  - !ruby/object:Gem::Version
139
- version: '0'
145
+ version: '13.0'
140
146
  - !ruby/object:Gem::Dependency
141
147
  name: rubocop-discourse
142
148
  requirement: !ruby/object:Gem::Requirement
143
149
  requirements:
144
- - - ">="
150
+ - - '='
145
151
  - !ruby/object:Gem::Version
146
- version: '0'
152
+ version: 3.2.0
147
153
  type: :development
148
154
  prerelease: false
149
155
  version_requirements: !ruby/object:Gem::Requirement
150
156
  requirements:
151
- - - ">="
157
+ - - '='
152
158
  - !ruby/object:Gem::Version
153
- version: '0'
159
+ version: 3.2.0
154
160
  description: Adds recurring jobs for Sidekiq
155
161
  email:
156
162
  - neil.lalonde@discourse.org
@@ -190,7 +196,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
190
196
  requirements:
191
197
  - - ">="
192
198
  - !ruby/object:Gem::Version
193
- version: '0'
199
+ version: 2.7.0
194
200
  required_rubygems_version: !ruby/object:Gem::Requirement
195
201
  requirements:
196
202
  - - ">="