mini_scheduler 0.15.0 → 0.17.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -2
- data/Gemfile +1 -1
- data/README.md +15 -15
- data/app/models/mini_scheduler/stat.rb +2 -3
- data/lib/generators/mini_scheduler/install/install_generator.rb +8 -5
- data/lib/generators/mini_scheduler/install/templates/mini_scheduler_initializer.rb +1 -3
- data/lib/mini_scheduler/distributed_mutex.rb +4 -9
- data/lib/mini_scheduler/manager.rb +163 -90
- data/lib/mini_scheduler/schedule.rb +1 -4
- data/lib/mini_scheduler/schedule_info.rb +17 -25
- data/lib/mini_scheduler/version.rb +1 -1
- data/lib/mini_scheduler/web.rb +6 -10
- data/lib/mini_scheduler.rb +8 -15
- data/mini_scheduler.gemspec +12 -11
- metadata +24 -10
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: '08b8517cbb1f57938ca8337f54bd804e402d176070934398dfd53ab1b00b5b2f'
|
4
|
+
data.tar.gz: 2d70f148236b793e71d276e5acd72403d4945a23d33f36121f081ec7b2ee3f0e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: fa584d657e08dfbcfd0f0f7dc05d7da2b4884423a7140afbeedf8381b5dc475823c312e0c08e0ecf0a7461fc0c6cc36f39e19df0d4a80d35b03135431d300a64
|
7
|
+
data.tar.gz: a1c34182f330304932e95a94c75a9366444be9bbedf88fc41cb3c8d359de13d5f1bdda957966b1399b66d96e901dcc165f34d5f456f6e521e070ac539ea5f107
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,12 @@
|
|
1
|
+
# 0.17.0 - 2024-08-06
|
2
|
+
|
3
|
+
- Add `MiniScheduler::Manager.discover_running_scheduled_jobs` API to allow running scheduled jobs to easily be discovered on the
|
4
|
+
current host.
|
5
|
+
|
6
|
+
# 0.16.0 - 2023-05-17
|
7
|
+
|
8
|
+
- Support Redis gem version 5
|
9
|
+
|
1
10
|
# 0.15.0 - 2022-11-17
|
2
11
|
|
3
12
|
- Fix data inconsistencies when Redis fails during jobs (#19)
|
@@ -10,7 +19,7 @@
|
|
10
19
|
# 0.13.0 - 2020-11-30
|
11
20
|
|
12
21
|
- Fix exception code so it has parity with Sidekiq 4.2.3 and up, version bump cause
|
13
|
-
minimum version of Sikekiq changed.
|
22
|
+
minimum version of Sikekiq changed.
|
14
23
|
|
15
24
|
# 0.12.3 - 2020-10-15
|
16
25
|
|
@@ -31,7 +40,7 @@ minimum version of Sikekiq changed.
|
|
31
40
|
# 0.11.0 - 2019-06-24
|
32
41
|
|
33
42
|
- Correct situation where distributed mutex could end in a tight loop when
|
34
|
-
|
43
|
+
redis could not be contacted
|
35
44
|
|
36
45
|
# 0.9.2 - 2019-04-26
|
37
46
|
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,17 +1,15 @@
|
|
1
1
|
[![Build Status](https://github.com/discourse/mini_scheduler/workflows/CI/badge.svg)](https://github.com/discourse/mini_scheduler/actions)
|
2
2
|
[![Gem Version](https://badge.fury.io/rb/mini_scheduler.svg)](https://rubygems.org/gems/mini_scheduler)
|
3
3
|
|
4
|
-
#
|
4
|
+
# MiniScheduler
|
5
5
|
|
6
6
|
MiniScheduler adds recurring jobs to [Sidekiq](https://sidekiq.org/).
|
7
7
|
|
8
|
-
|
9
|
-
|
10
8
|
## Installation
|
11
9
|
|
12
10
|
Add this line to your application's Gemfile:
|
13
11
|
|
14
|
-
```
|
12
|
+
```rb
|
15
13
|
gem 'mini_scheduler'
|
16
14
|
```
|
17
15
|
|
@@ -25,8 +23,8 @@ Or install it yourself as:
|
|
25
23
|
|
26
24
|
In a Rails application, create files needed in your application to configure mini_scheduler:
|
27
25
|
|
28
|
-
bin/rails g mini_scheduler:install
|
29
|
-
|
26
|
+
$ bin/rails g mini_scheduler:install
|
27
|
+
$ bin/rails db:migrate
|
30
28
|
|
31
29
|
An initializer is created named `config/initializers/mini_scheduler.rb` which lists all the configuration options.
|
32
30
|
|
@@ -34,7 +32,7 @@ An initializer is created named `config/initializers/mini_scheduler.rb` which li
|
|
34
32
|
|
35
33
|
By default each instance of MiniScheduler will run with a single worker. To amend this behavior:
|
36
34
|
|
37
|
-
```
|
35
|
+
```rb
|
38
36
|
if Sidekiq.server? && defined?(Rails)
|
39
37
|
Rails.application.config.after_initialize do
|
40
38
|
MiniScheduler.start(workers: 5)
|
@@ -48,7 +46,7 @@ This is useful for cases where you have extremely long running tasks that you wo
|
|
48
46
|
|
49
47
|
Create jobs with a recurring schedule like this:
|
50
48
|
|
51
|
-
```
|
49
|
+
```rb
|
52
50
|
class MyHourlyJob
|
53
51
|
include Sidekiq::Worker
|
54
52
|
extend MiniScheduler::Schedule
|
@@ -63,21 +61,23 @@ end
|
|
63
61
|
|
64
62
|
Options for schedules:
|
65
63
|
|
66
|
-
|
67
|
-
|
68
|
-
|
64
|
+
- **queue** followed by a queue name, like "queue :email", default queue is "default"
|
65
|
+
- **every** followed by a duration in seconds, like "every 1.hour".
|
66
|
+
- **daily at:** followed by a duration since midnight, like "daily at: 12.hours", to run only once per day at a specific time.
|
69
67
|
|
70
68
|
To view the scheduled jobs, their history, and the schedule, go to sidekiq's web UI and look for the "Scheduler" tab at the top.
|
71
69
|
|
72
70
|
To enable this view in Sidekiq, add `require "mini_scheduler/web"` to `routes.rb`:
|
73
|
-
|
74
|
-
|
71
|
+
|
72
|
+
```rb
|
75
73
|
require "sidekiq/web"
|
76
74
|
require "mini_scheduler/web"
|
75
|
+
|
77
76
|
Rails.application.routes.draw do
|
78
|
-
|
79
|
-
|
77
|
+
...
|
78
|
+
end
|
80
79
|
```
|
80
|
+
|
81
81
|
## How to reach us
|
82
82
|
|
83
83
|
If you have questions about using mini_scheduler or found a problem, you can find us at https://meta.discourse.org.
|
@@ -3,11 +3,10 @@
|
|
3
3
|
if defined?(ActiveRecord::Base)
|
4
4
|
module MiniScheduler
|
5
5
|
class Stat < ActiveRecord::Base
|
6
|
-
|
7
|
-
self.table_name = 'scheduler_stats'
|
6
|
+
self.table_name = "scheduler_stats"
|
8
7
|
|
9
8
|
def self.purge_old
|
10
|
-
where(
|
9
|
+
where("started_at < ?", 1.months.ago).delete_all
|
11
10
|
end
|
12
11
|
end
|
13
12
|
end
|
@@ -1,14 +1,14 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require
|
4
|
-
require
|
5
|
-
require
|
3
|
+
require "rails/generators"
|
4
|
+
require "rails/generators/migration"
|
5
|
+
require "active_record"
|
6
6
|
|
7
7
|
module MiniScheduler
|
8
8
|
module Generators
|
9
9
|
class InstallGenerator < ::Rails::Generators::Base
|
10
10
|
include Rails::Generators::Migration
|
11
|
-
source_root File.expand_path(
|
11
|
+
source_root File.expand_path("../templates", __FILE__)
|
12
12
|
desc "Generate files for MiniScheduler"
|
13
13
|
|
14
14
|
def self.next_migration_number(path)
|
@@ -17,7 +17,10 @@ module MiniScheduler
|
|
17
17
|
end
|
18
18
|
|
19
19
|
def copy_migrations
|
20
|
-
migration_template(
|
20
|
+
migration_template(
|
21
|
+
"create_mini_scheduler_stats.rb",
|
22
|
+
"db/migrate/create_mini_scheduler_stats.rb",
|
23
|
+
)
|
21
24
|
end
|
22
25
|
|
23
26
|
def copy_initializer_file
|
@@ -2,7 +2,8 @@
|
|
2
2
|
|
3
3
|
module MiniScheduler
|
4
4
|
class DistributedMutex
|
5
|
-
class Timeout < StandardError
|
5
|
+
class Timeout < StandardError
|
6
|
+
end
|
6
7
|
|
7
8
|
@default_redis = nil
|
8
9
|
|
@@ -15,7 +16,7 @@ module MiniScheduler
|
|
15
16
|
end
|
16
17
|
|
17
18
|
def initialize(key, redis)
|
18
|
-
raise ArgumentError.new(
|
19
|
+
raise ArgumentError.new("redis argument is nil") if redis.nil?
|
19
20
|
@key = key
|
20
21
|
@redis = redis
|
21
22
|
@mutex = Mutex.new
|
@@ -32,7 +33,6 @@ module MiniScheduler
|
|
32
33
|
attempts = 0
|
33
34
|
sleep_duration = BASE_SLEEP_DURATION
|
34
35
|
while !try_to_get_lock
|
35
|
-
|
36
36
|
sleep(sleep_duration)
|
37
37
|
|
38
38
|
if sleep_duration < MAX_SLEEP_DURATION
|
@@ -44,7 +44,6 @@ module MiniScheduler
|
|
44
44
|
end
|
45
45
|
|
46
46
|
yield
|
47
|
-
|
48
47
|
ensure
|
49
48
|
@redis.del @key
|
50
49
|
@mutex.unlock
|
@@ -62,9 +61,7 @@ module MiniScheduler
|
|
62
61
|
@redis.watch @key
|
63
62
|
time = @redis.get @key
|
64
63
|
if time && time.to_i < Time.now.to_i
|
65
|
-
got_lock = @redis.multi
|
66
|
-
@redis.set @key, Time.now.to_i + 60
|
67
|
-
end
|
64
|
+
got_lock = @redis.multi { @redis.set @key, Time.now.to_i + 60 }
|
68
65
|
end
|
69
66
|
ensure
|
70
67
|
@redis.unwatch
|
@@ -73,7 +70,5 @@ module MiniScheduler
|
|
73
70
|
|
74
71
|
got_lock
|
75
72
|
end
|
76
|
-
|
77
73
|
end
|
78
|
-
|
79
74
|
end
|
@@ -12,32 +12,34 @@ module MiniScheduler
|
|
12
12
|
@manager = manager
|
13
13
|
@hostname = manager.hostname
|
14
14
|
|
15
|
-
@recovery_thread =
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
15
|
+
@recovery_thread =
|
16
|
+
Thread.new do
|
17
|
+
while !@stopped
|
18
|
+
sleep 60
|
19
|
+
|
20
|
+
@mutex.synchronize do
|
21
|
+
repair_queue
|
22
|
+
reschedule_orphans
|
23
|
+
ensure_worker_threads
|
24
|
+
end
|
23
25
|
end
|
24
26
|
end
|
25
|
-
|
26
|
-
@keep_alive_thread =
|
27
|
-
|
28
|
-
|
29
|
-
keep_alive
|
27
|
+
|
28
|
+
@keep_alive_thread =
|
29
|
+
Thread.new do
|
30
|
+
while !@stopped
|
31
|
+
@mutex.synchronize { keep_alive }
|
32
|
+
sleep(@manager.keep_alive_duration / 2)
|
30
33
|
end
|
31
|
-
sleep (@manager.keep_alive_duration / 2)
|
32
34
|
end
|
33
|
-
|
35
|
+
|
34
36
|
ensure_worker_threads
|
35
37
|
end
|
36
38
|
|
37
39
|
def keep_alive(*ids)
|
38
40
|
@manager.keep_alive(*ids)
|
39
41
|
rescue => ex
|
40
|
-
MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler
|
42
|
+
MiniScheduler.handle_job_exception(ex, message: "Error during MiniScheduler keep_alive")
|
41
43
|
end
|
42
44
|
|
43
45
|
def repair_queue
|
@@ -49,27 +51,36 @@ module MiniScheduler
|
|
49
51
|
def reschedule_orphans
|
50
52
|
@manager.reschedule_orphans!
|
51
53
|
rescue => ex
|
52
|
-
MiniScheduler.handle_job_exception(
|
54
|
+
MiniScheduler.handle_job_exception(
|
55
|
+
ex,
|
56
|
+
message: "Error during MiniScheduler reschedule_orphans",
|
57
|
+
)
|
53
58
|
end
|
54
59
|
|
55
60
|
def ensure_worker_threads
|
56
61
|
@threads ||= []
|
57
62
|
@threads.delete_if { |t| !t.alive? }
|
58
|
-
(@manager.workers - @threads.size).times
|
59
|
-
@threads << Thread.new { worker_loop }
|
60
|
-
end
|
63
|
+
(@manager.workers - @threads.size).times { @threads << Thread.new { worker_loop } }
|
61
64
|
rescue => ex
|
62
|
-
MiniScheduler.handle_job_exception(
|
65
|
+
MiniScheduler.handle_job_exception(
|
66
|
+
ex,
|
67
|
+
message: "Error during MiniScheduler ensure_worker_threads",
|
68
|
+
)
|
63
69
|
end
|
64
70
|
|
65
71
|
def worker_loop
|
66
72
|
set_current_worker_thread_id!
|
67
73
|
keep_alive(current_worker_thread_id)
|
74
|
+
|
68
75
|
while !@stopped
|
69
76
|
begin
|
70
77
|
process_queue
|
71
78
|
rescue => ex
|
72
|
-
MiniScheduler.handle_job_exception(
|
79
|
+
MiniScheduler.handle_job_exception(
|
80
|
+
ex,
|
81
|
+
message: "Error during MiniScheduler worker_loop",
|
82
|
+
)
|
83
|
+
|
73
84
|
break # Data could be in a bad state - stop the thread
|
74
85
|
end
|
75
86
|
end
|
@@ -84,7 +95,9 @@ module MiniScheduler
|
|
84
95
|
end
|
85
96
|
|
86
97
|
def set_current_worker_thread_id!
|
87
|
-
Thread.current[
|
98
|
+
Thread.current[
|
99
|
+
:mini_scheduler_worker_thread_id
|
100
|
+
] = "#{@manager.identity_key}:thread_#{SecureRandom.alphanumeric(10)}"
|
88
101
|
end
|
89
102
|
|
90
103
|
def worker_thread_ids
|
@@ -93,6 +106,7 @@ module MiniScheduler
|
|
93
106
|
|
94
107
|
def process_queue
|
95
108
|
klass = @queue.deq
|
109
|
+
|
96
110
|
# hack alert, I need to both deq and set @running atomically.
|
97
111
|
@running = true
|
98
112
|
|
@@ -110,22 +124,30 @@ module MiniScheduler
|
|
110
124
|
@mutex.synchronize { info.write! }
|
111
125
|
|
112
126
|
if @manager.enable_stats
|
113
|
-
stat =
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
127
|
+
stat =
|
128
|
+
MiniScheduler::Stat.create!(
|
129
|
+
name: klass.to_s,
|
130
|
+
hostname: hostname,
|
131
|
+
pid: Process.pid,
|
132
|
+
started_at: Time.now,
|
133
|
+
live_slots_start: GC.stat[:heap_live_slots],
|
134
|
+
)
|
120
135
|
end
|
121
136
|
|
122
137
|
klass.new.perform
|
123
138
|
rescue => e
|
124
|
-
MiniScheduler.handle_job_exception(
|
139
|
+
MiniScheduler.handle_job_exception(
|
140
|
+
e,
|
141
|
+
message: "Error while running a scheduled job",
|
142
|
+
job: {
|
143
|
+
"class" => klass,
|
144
|
+
},
|
145
|
+
)
|
125
146
|
|
126
147
|
error = "#{e.class}: #{e.message} #{e.backtrace.join("\n")}"
|
127
148
|
failed = true
|
128
149
|
end
|
150
|
+
|
129
151
|
duration = ((Time.now.to_f - start) * 1000).to_i
|
130
152
|
info.prev_duration = duration
|
131
153
|
info.prev_result = failed ? "FAILED" : "OK"
|
@@ -135,15 +157,14 @@ module MiniScheduler
|
|
135
157
|
duration_ms: duration,
|
136
158
|
live_slots_finish: GC.stat[:heap_live_slots],
|
137
159
|
success: !failed,
|
138
|
-
error: error
|
160
|
+
error: error,
|
139
161
|
)
|
140
162
|
MiniScheduler.job_ran&.call(stat)
|
141
163
|
end
|
142
|
-
attempts(3)
|
143
|
-
@mutex.synchronize { info.write! }
|
144
|
-
end
|
164
|
+
attempts(3) { @mutex.synchronize { info.write! } }
|
145
165
|
ensure
|
146
166
|
@running = false
|
167
|
+
|
147
168
|
if defined?(ActiveRecord::Base)
|
148
169
|
ActiveRecord::Base.connection_handler.clear_active_connections!
|
149
170
|
end
|
@@ -163,10 +184,11 @@ module MiniScheduler
|
|
163
184
|
|
164
185
|
enq(nil)
|
165
186
|
|
166
|
-
kill_thread =
|
167
|
-
|
168
|
-
|
169
|
-
|
187
|
+
kill_thread =
|
188
|
+
Thread.new do
|
189
|
+
sleep 0.5
|
190
|
+
@threads.each(&:kill)
|
191
|
+
end
|
170
192
|
|
171
193
|
@threads.each(&:join)
|
172
194
|
kill_thread.kill
|
@@ -179,29 +201,24 @@ module MiniScheduler
|
|
179
201
|
end
|
180
202
|
|
181
203
|
def wait_till_done
|
182
|
-
while !@queue.empty? && !(@queue.num_waiting > 0)
|
183
|
-
sleep 0.001
|
184
|
-
end
|
204
|
+
sleep 0.001 while !@queue.empty? && !(@queue.num_waiting > 0)
|
185
205
|
# this is a hack, but is only used for test anyway
|
186
206
|
# if tests fail that depend on this we are forced to increase it.
|
187
207
|
sleep 0.010
|
188
|
-
while @running
|
189
|
-
sleep 0.001
|
190
|
-
end
|
208
|
+
sleep 0.001 while @running
|
191
209
|
end
|
192
210
|
|
193
211
|
def attempts(max_attempts)
|
194
212
|
attempt = 0
|
195
213
|
begin
|
196
214
|
yield
|
197
|
-
rescue
|
215
|
+
rescue StandardError
|
198
216
|
attempt += 1
|
199
217
|
raise if attempt >= max_attempts
|
200
218
|
sleep Random.rand
|
201
219
|
retry
|
202
220
|
end
|
203
221
|
end
|
204
|
-
|
205
222
|
end
|
206
223
|
|
207
224
|
def self.without_runner
|
@@ -233,11 +250,7 @@ module MiniScheduler
|
|
233
250
|
end
|
234
251
|
|
235
252
|
def hostname
|
236
|
-
@hostname ||=
|
237
|
-
`hostname`.strip
|
238
|
-
rescue
|
239
|
-
"unknown"
|
240
|
-
end
|
253
|
+
@hostname ||= self.class.hostname
|
241
254
|
end
|
242
255
|
|
243
256
|
def schedule_info(klass)
|
@@ -249,15 +262,11 @@ module MiniScheduler
|
|
249
262
|
end
|
250
263
|
|
251
264
|
def ensure_schedule!(klass)
|
252
|
-
lock
|
253
|
-
schedule_info(klass).schedule!
|
254
|
-
end
|
265
|
+
lock { schedule_info(klass).schedule! }
|
255
266
|
end
|
256
267
|
|
257
268
|
def remove(klass)
|
258
|
-
lock
|
259
|
-
schedule_info(klass).del!
|
260
|
-
end
|
269
|
+
lock { schedule_info(klass).del! }
|
261
270
|
end
|
262
271
|
|
263
272
|
def reschedule_orphans!
|
@@ -268,18 +277,20 @@ module MiniScheduler
|
|
268
277
|
end
|
269
278
|
|
270
279
|
def reschedule_orphans_on!(hostname = nil)
|
271
|
-
redis
|
272
|
-
|
273
|
-
|
274
|
-
|
275
|
-
|
276
|
-
|
277
|
-
|
278
|
-
info.prev_result
|
279
|
-
|
280
|
-
|
280
|
+
redis
|
281
|
+
.zrange(Manager.queue_key(queue, hostname), 0, -1)
|
282
|
+
.each do |key|
|
283
|
+
klass = get_klass(key)
|
284
|
+
next unless klass
|
285
|
+
info = schedule_info(klass)
|
286
|
+
|
287
|
+
if %w[QUEUED RUNNING].include?(info.prev_result) &&
|
288
|
+
(!info.current_owner || !redis.get(info.current_owner))
|
289
|
+
info.prev_result = "ORPHAN"
|
290
|
+
info.next_run = Time.now.to_i
|
291
|
+
info.write!
|
292
|
+
end
|
281
293
|
end
|
282
|
-
end
|
283
294
|
end
|
284
295
|
|
285
296
|
def get_klass(name)
|
@@ -289,10 +300,14 @@ module MiniScheduler
|
|
289
300
|
end
|
290
301
|
|
291
302
|
def repair_queue
|
292
|
-
|
293
|
-
|
303
|
+
if redis.exists?(self.class.queue_key(queue)) ||
|
304
|
+
redis.exists?(self.class.queue_key(queue, hostname))
|
305
|
+
return
|
306
|
+
end
|
294
307
|
|
295
|
-
self
|
308
|
+
self
|
309
|
+
.class
|
310
|
+
.discover_schedules
|
296
311
|
.select { |schedule| schedule.queue == queue }
|
297
312
|
.each { |schedule| ensure_schedule!(schedule) }
|
298
313
|
end
|
@@ -310,9 +325,7 @@ module MiniScheduler
|
|
310
325
|
|
311
326
|
if due.to_i <= Time.now.to_i
|
312
327
|
klass = get_klass(key)
|
313
|
-
if !klass || (
|
314
|
-
(klass.is_per_host && !hostname) || (hostname && !klass.is_per_host)
|
315
|
-
)
|
328
|
+
if !klass || ((klass.is_per_host && !hostname) || (hostname && !klass.is_per_host))
|
316
329
|
# corrupt key, nuke it (renamed job or something)
|
317
330
|
redis.zrem Manager.queue_key(queue, hostname), key
|
318
331
|
return
|
@@ -345,9 +358,7 @@ module MiniScheduler
|
|
345
358
|
|
346
359
|
def keep_alive(*ids)
|
347
360
|
ids = [identity_key, *@runner.worker_thread_ids] if ids.size == 0
|
348
|
-
ids.each
|
349
|
-
redis.setex identity_key, keep_alive_duration, ""
|
350
|
-
end
|
361
|
+
ids.each { |identity_key| redis.setex identity_key, keep_alive_duration, "" }
|
351
362
|
end
|
352
363
|
|
353
364
|
def lock
|
@@ -357,7 +368,9 @@ module MiniScheduler
|
|
357
368
|
end
|
358
369
|
|
359
370
|
def self.discover_queues
|
360
|
-
|
371
|
+
queues = Set.new
|
372
|
+
ObjectSpace.each_object(MiniScheduler::Schedule).each { |schedule| queues << schedule.queue }
|
373
|
+
queues
|
361
374
|
end
|
362
375
|
|
363
376
|
def self.discover_schedules
|
@@ -376,6 +389,73 @@ module MiniScheduler
|
|
376
389
|
schedules
|
377
390
|
end
|
378
391
|
|
392
|
+
def self.hostname
|
393
|
+
@hostname ||=
|
394
|
+
begin
|
395
|
+
require "socket"
|
396
|
+
Socket.gethostname
|
397
|
+
rescue => e
|
398
|
+
begin
|
399
|
+
`hostname`.strip
|
400
|
+
rescue => e
|
401
|
+
"unknown_host"
|
402
|
+
end
|
403
|
+
end
|
404
|
+
end
|
405
|
+
|
406
|
+
# Discover running scheduled jobs on the current host.
|
407
|
+
#
|
408
|
+
# @example
|
409
|
+
#
|
410
|
+
# MiniScheduler::Manager.discover_running_scheduled_jobs
|
411
|
+
#
|
412
|
+
# @return [Array<Hash>] an array of hashes representing the running scheduled jobs.
|
413
|
+
# @option job [Class] :class The class of the scheduled job.
|
414
|
+
# @option job [Time] :started_at The time when the scheduled job started.
|
415
|
+
# @option job [String] :thread_id The ID of the worker thread running the job.
|
416
|
+
# The thread can be identified by matching the `:mini_scheduler_worker_thread_id` thread variable with the ID.
|
417
|
+
def self.discover_running_scheduled_jobs
|
418
|
+
hostname = self.hostname
|
419
|
+
|
420
|
+
schedule_keys =
|
421
|
+
discover_schedules.reduce({}) do |acc, klass|
|
422
|
+
acc[klass] = if klass.is_per_host
|
423
|
+
self.schedule_key(klass, hostname)
|
424
|
+
else
|
425
|
+
self.schedule_key(klass)
|
426
|
+
end
|
427
|
+
|
428
|
+
acc
|
429
|
+
end
|
430
|
+
|
431
|
+
running_scheduled_jobs = []
|
432
|
+
|
433
|
+
schedule_keys
|
434
|
+
.keys
|
435
|
+
.zip(MiniScheduler.redis.mget(*schedule_keys.values))
|
436
|
+
.each do |scheduled_job_class, scheduled_job_info|
|
437
|
+
next if scheduled_job_info.nil?
|
438
|
+
|
439
|
+
parsed =
|
440
|
+
begin
|
441
|
+
JSON.parse(scheduled_job_info, symbolize_names: true)
|
442
|
+
rescue JSON::ParserError
|
443
|
+
nil
|
444
|
+
end
|
445
|
+
|
446
|
+
next if parsed.nil?
|
447
|
+
next if parsed[:prev_result] != "RUNNING"
|
448
|
+
|
449
|
+
running_scheduled_jobs << {
|
450
|
+
class: scheduled_job_class,
|
451
|
+
started_at: Time.at(parsed[:prev_run]),
|
452
|
+
thread_id: parsed[:current_owner],
|
453
|
+
}
|
454
|
+
end
|
455
|
+
|
456
|
+
running_scheduled_jobs
|
457
|
+
end
|
458
|
+
|
379
459
|
@class_mutex = Mutex.new
|
380
460
|
def self.seq
|
381
461
|
@class_mutex.synchronize do
|
@@ -388,7 +468,8 @@ module MiniScheduler
|
|
388
468
|
def identity_key
|
389
469
|
return @identity_key if @identity_key
|
390
470
|
@@identity_key_mutex.synchronize do
|
391
|
-
@identity_key ||=
|
471
|
+
@identity_key ||=
|
472
|
+
"_scheduler_#{hostname}:#{Process.pid}:#{self.class.seq}:#{SecureRandom.hex}"
|
392
473
|
end
|
393
474
|
end
|
394
475
|
|
@@ -397,19 +478,11 @@ module MiniScheduler
|
|
397
478
|
end
|
398
479
|
|
399
480
|
def self.queue_key(queue, hostname = nil)
|
400
|
-
|
401
|
-
"_scheduler_queue_#{queue}_#{hostname}_"
|
402
|
-
else
|
403
|
-
"_scheduler_queue_#{queue}_"
|
404
|
-
end
|
481
|
+
hostname ? "_scheduler_queue_#{queue}_#{hostname}_" : "_scheduler_queue_#{queue}_"
|
405
482
|
end
|
406
483
|
|
407
484
|
def self.schedule_key(klass, hostname = nil)
|
408
|
-
|
409
|
-
"_scheduler_#{klass}_#{hostname}"
|
410
|
-
else
|
411
|
-
"_scheduler_#{klass}"
|
412
|
-
end
|
485
|
+
hostname ? "_scheduler_#{klass}_#{hostname}" : "_scheduler_#{klass}"
|
413
486
|
end
|
414
487
|
end
|
415
488
|
end
|
@@ -1,16 +1,13 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module MiniScheduler::Schedule
|
4
|
-
|
5
4
|
def queue(value = nil)
|
6
5
|
@queue = value.to_s if value
|
7
6
|
@queue ||= "default"
|
8
7
|
end
|
9
8
|
|
10
9
|
def daily(options = nil)
|
11
|
-
if options
|
12
|
-
@daily = options
|
13
|
-
end
|
10
|
+
@daily = options if options
|
14
11
|
@daily
|
15
12
|
end
|
16
13
|
|
@@ -2,11 +2,7 @@
|
|
2
2
|
|
3
3
|
module MiniScheduler
|
4
4
|
class ScheduleInfo
|
5
|
-
attr_accessor :next_run,
|
6
|
-
:prev_run,
|
7
|
-
:prev_duration,
|
8
|
-
:prev_result,
|
9
|
-
:current_owner
|
5
|
+
attr_accessor :next_run, :prev_run, :prev_duration, :prev_result, :current_owner
|
10
6
|
|
11
7
|
def initialize(klass, manager)
|
12
8
|
@klass = klass
|
@@ -25,7 +21,7 @@ module MiniScheduler
|
|
25
21
|
@prev_duration = data["prev_duration"]
|
26
22
|
@current_owner = data["current_owner"]
|
27
23
|
end
|
28
|
-
rescue
|
24
|
+
rescue StandardError
|
29
25
|
# corrupt redis
|
30
26
|
@next_run = @prev_run = @prev_result = @prev_duration = @current_owner = nil
|
31
27
|
end
|
@@ -40,17 +36,14 @@ module MiniScheduler
|
|
40
36
|
|
41
37
|
def valid_every?
|
42
38
|
return false unless @klass.every
|
43
|
-
!!@prev_run &&
|
44
|
-
@prev_run <= Time.now.to_i &&
|
39
|
+
!!@prev_run && @prev_run <= Time.now.to_i &&
|
45
40
|
@next_run < @prev_run + @klass.every * (1 + @manager.random_ratio)
|
46
41
|
end
|
47
42
|
|
48
43
|
def valid_daily?
|
49
44
|
return false unless @klass.daily
|
50
45
|
return true if !@prev_run && @next_run && @next_run <= (Time.now + 1.day).to_i
|
51
|
-
!!@prev_run &&
|
52
|
-
@prev_run <= Time.now.to_i &&
|
53
|
-
@next_run < @prev_run + 1.day
|
46
|
+
!!@prev_run && @prev_run <= Time.now.to_i && @next_run < @prev_run + 1.day
|
54
47
|
end
|
55
48
|
|
56
49
|
def schedule_every!
|
@@ -63,9 +56,7 @@ module MiniScheduler
|
|
63
56
|
# this can look a bit confusing, but @next_run above could be off
|
64
57
|
# if prev_run is off, so this ensures it ends up correct and in the
|
65
58
|
# future
|
66
|
-
if !valid?
|
67
|
-
@next_run = Time.now.to_i + 300 * Random.rand
|
68
|
-
end
|
59
|
+
@next_run = Time.now.to_i + 300 * Random.rand if !valid?
|
69
60
|
end
|
70
61
|
|
71
62
|
def schedule_daily!
|
@@ -96,15 +87,16 @@ module MiniScheduler
|
|
96
87
|
|
97
88
|
def write!
|
98
89
|
clear!
|
99
|
-
redis.set key,
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
90
|
+
redis.set key,
|
91
|
+
{
|
92
|
+
next_run: @next_run,
|
93
|
+
prev_run: @prev_run,
|
94
|
+
prev_duration: @prev_duration,
|
95
|
+
prev_result: @prev_result,
|
96
|
+
current_owner: @current_owner,
|
97
|
+
}.to_json
|
98
|
+
|
99
|
+
redis.zadd queue_key, @next_run.to_s, @klass.to_s if @next_run
|
108
100
|
end
|
109
101
|
|
110
102
|
def del!
|
@@ -133,10 +125,10 @@ module MiniScheduler
|
|
133
125
|
end
|
134
126
|
|
135
127
|
private
|
128
|
+
|
136
129
|
def clear!
|
137
130
|
redis.del key
|
138
|
-
redis.zrem queue_key, @klass
|
131
|
+
redis.zrem queue_key, @klass.to_s
|
139
132
|
end
|
140
|
-
|
141
133
|
end
|
142
134
|
end
|
data/lib/mini_scheduler/web.rb
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
# Based off sidetiq https://github.com/tobiassvn/sidetiq/blob/master/lib/sidetiq/web.rb
|
3
3
|
module MiniScheduler
|
4
4
|
module Web
|
5
|
-
VIEWS = File.expand_path(
|
5
|
+
VIEWS = File.expand_path("views", File.dirname(__FILE__)) unless defined?(VIEWS)
|
6
6
|
|
7
7
|
def self.find_schedules_by_time
|
8
8
|
Manager.discover_schedules.sort do |a, b|
|
@@ -19,7 +19,6 @@ module MiniScheduler
|
|
19
19
|
end
|
20
20
|
|
21
21
|
def self.registered(app)
|
22
|
-
|
23
22
|
app.helpers do
|
24
23
|
def sane_time(time)
|
25
24
|
return unless time
|
@@ -31,7 +30,7 @@ module MiniScheduler
|
|
31
30
|
if duration < 1000
|
32
31
|
"#{duration}ms"
|
33
32
|
else
|
34
|
-
"#{
|
33
|
+
"#{"%.2f" % (duration / 1000.0)} secs"
|
35
34
|
end
|
36
35
|
end
|
37
36
|
end
|
@@ -39,24 +38,22 @@ module MiniScheduler
|
|
39
38
|
app.get "/scheduler" do
|
40
39
|
MiniScheduler.before_sidekiq_web_request&.call
|
41
40
|
@schedules = Web.find_schedules_by_time
|
42
|
-
erb File.read(File.join(VIEWS,
|
41
|
+
erb File.read(File.join(VIEWS, "scheduler.erb")), locals: { view_path: VIEWS }
|
43
42
|
end
|
44
43
|
|
45
44
|
app.get "/scheduler/history" do
|
46
45
|
MiniScheduler.before_sidekiq_web_request&.call
|
47
46
|
@schedules = Manager.discover_schedules
|
48
47
|
@schedules.sort_by!(&:to_s)
|
49
|
-
@scheduler_stats = Stat.order(
|
48
|
+
@scheduler_stats = Stat.order("started_at desc")
|
50
49
|
|
51
50
|
@filter = params[:filter]
|
52
51
|
names = @schedules.map(&:to_s)
|
53
52
|
@filter = nil if !names.include?(@filter)
|
54
|
-
if @filter
|
55
|
-
@scheduler_stats = @scheduler_stats.where(name: @filter)
|
56
|
-
end
|
53
|
+
@scheduler_stats = @scheduler_stats.where(name: @filter) if @filter
|
57
54
|
|
58
55
|
@scheduler_stats = @scheduler_stats.limit(200)
|
59
|
-
erb File.read(File.join(VIEWS,
|
56
|
+
erb File.read(File.join(VIEWS, "history.erb")), locals: { view_path: VIEWS }
|
60
57
|
end
|
61
58
|
|
62
59
|
app.post "/scheduler/:name/trigger" do
|
@@ -71,7 +68,6 @@ module MiniScheduler
|
|
71
68
|
|
72
69
|
redirect "#{root_path}scheduler"
|
73
70
|
end
|
74
|
-
|
75
71
|
end
|
76
72
|
end
|
77
73
|
end
|
data/lib/mini_scheduler.rb
CHANGED
@@ -1,18 +1,17 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
require "mini_scheduler/engine"
|
3
|
-
require
|
4
|
-
require
|
5
|
-
require
|
6
|
-
require
|
7
|
-
require
|
3
|
+
require "mini_scheduler/schedule"
|
4
|
+
require "mini_scheduler/schedule_info"
|
5
|
+
require "mini_scheduler/manager"
|
6
|
+
require "mini_scheduler/distributed_mutex"
|
7
|
+
require "sidekiq"
|
8
8
|
|
9
9
|
begin
|
10
|
-
require
|
10
|
+
require "sidekiq/exception_handler"
|
11
11
|
rescue LoadError
|
12
12
|
end
|
13
13
|
|
14
14
|
module MiniScheduler
|
15
|
-
|
16
15
|
def self.configure
|
17
16
|
yield self
|
18
17
|
end
|
@@ -69,18 +68,12 @@ module MiniScheduler
|
|
69
68
|
Manager.discover_queues.each do |queue|
|
70
69
|
manager = Manager.new(queue: queue, workers: workers)
|
71
70
|
|
72
|
-
schedules.each
|
73
|
-
if schedule.queue == queue
|
74
|
-
manager.ensure_schedule!(schedule)
|
75
|
-
end
|
76
|
-
end
|
71
|
+
schedules.each { |schedule| manager.ensure_schedule!(schedule) if schedule.queue == queue }
|
77
72
|
|
78
73
|
Thread.new do
|
79
74
|
while true
|
80
75
|
begin
|
81
|
-
if !self.skip_schedule || !self.skip_schedule.call
|
82
|
-
manager.tick
|
83
|
-
end
|
76
|
+
manager.tick if !self.skip_schedule || !self.skip_schedule.call
|
84
77
|
rescue => e
|
85
78
|
# the show must go on
|
86
79
|
handle_job_exception(e, message: "While ticking scheduling manager")
|
data/mini_scheduler.gemspec
CHANGED
@@ -1,19 +1,19 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
lib = File.expand_path("../lib", __FILE__)
|
4
|
-
$LOAD_PATH.unshift(lib)
|
4
|
+
$LOAD_PATH.unshift(lib) if !$LOAD_PATH.include?(lib)
|
5
5
|
require "mini_scheduler/version"
|
6
6
|
|
7
7
|
Gem::Specification.new do |spec|
|
8
|
-
spec.name
|
9
|
-
spec.version
|
10
|
-
spec.authors
|
11
|
-
spec.email
|
8
|
+
spec.name = "mini_scheduler"
|
9
|
+
spec.version = MiniScheduler::VERSION
|
10
|
+
spec.authors = ["Sam Saffron", "Neil Lalonde"]
|
11
|
+
spec.email = ["neil.lalonde@discourse.org"]
|
12
12
|
|
13
|
-
spec.summary
|
14
|
-
spec.description
|
15
|
-
spec.homepage
|
16
|
-
spec.license
|
13
|
+
spec.summary = "Adds recurring jobs for Sidekiq"
|
14
|
+
spec.description = "Adds recurring jobs for Sidekiq"
|
15
|
+
spec.homepage = "https://github.com/discourse/mini_scheduler"
|
16
|
+
spec.license = "MIT"
|
17
17
|
|
18
18
|
spec.required_ruby_version = ">= 2.7.0"
|
19
19
|
|
@@ -28,7 +28,8 @@ Gem::Specification.new do |spec|
|
|
28
28
|
spec.add_development_dependency "mocha", "~> 2.0"
|
29
29
|
spec.add_development_dependency "guard", "~> 2.0"
|
30
30
|
spec.add_development_dependency "guard-rspec", "~> 4.0"
|
31
|
-
spec.add_development_dependency "redis", "
|
31
|
+
spec.add_development_dependency "redis", ">= 4.0"
|
32
32
|
spec.add_development_dependency "rake", "~> 13.0"
|
33
|
-
spec.add_development_dependency "rubocop-discourse", "=
|
33
|
+
spec.add_development_dependency "rubocop-discourse", "= 3.8.1"
|
34
|
+
spec.add_development_dependency "syntax_tree"
|
34
35
|
end
|
metadata
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: mini_scheduler
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.17.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sam Saffron
|
8
8
|
- Neil Lalonde
|
9
|
-
autorequire:
|
9
|
+
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2024-08-08 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sidekiq
|
@@ -119,14 +119,14 @@ dependencies:
|
|
119
119
|
name: redis
|
120
120
|
requirement: !ruby/object:Gem::Requirement
|
121
121
|
requirements:
|
122
|
-
- - "
|
122
|
+
- - ">="
|
123
123
|
- !ruby/object:Gem::Version
|
124
124
|
version: '4.0'
|
125
125
|
type: :development
|
126
126
|
prerelease: false
|
127
127
|
version_requirements: !ruby/object:Gem::Requirement
|
128
128
|
requirements:
|
129
|
-
- - "
|
129
|
+
- - ">="
|
130
130
|
- !ruby/object:Gem::Version
|
131
131
|
version: '4.0'
|
132
132
|
- !ruby/object:Gem::Dependency
|
@@ -149,14 +149,28 @@ dependencies:
|
|
149
149
|
requirements:
|
150
150
|
- - '='
|
151
151
|
- !ruby/object:Gem::Version
|
152
|
-
version:
|
152
|
+
version: 3.8.1
|
153
153
|
type: :development
|
154
154
|
prerelease: false
|
155
155
|
version_requirements: !ruby/object:Gem::Requirement
|
156
156
|
requirements:
|
157
157
|
- - '='
|
158
158
|
- !ruby/object:Gem::Version
|
159
|
-
version:
|
159
|
+
version: 3.8.1
|
160
|
+
- !ruby/object:Gem::Dependency
|
161
|
+
name: syntax_tree
|
162
|
+
requirement: !ruby/object:Gem::Requirement
|
163
|
+
requirements:
|
164
|
+
- - ">="
|
165
|
+
- !ruby/object:Gem::Version
|
166
|
+
version: '0'
|
167
|
+
type: :development
|
168
|
+
prerelease: false
|
169
|
+
version_requirements: !ruby/object:Gem::Requirement
|
170
|
+
requirements:
|
171
|
+
- - ">="
|
172
|
+
- !ruby/object:Gem::Version
|
173
|
+
version: '0'
|
160
174
|
description: Adds recurring jobs for Sidekiq
|
161
175
|
email:
|
162
176
|
- neil.lalonde@discourse.org
|
@@ -188,7 +202,7 @@ homepage: https://github.com/discourse/mini_scheduler
|
|
188
202
|
licenses:
|
189
203
|
- MIT
|
190
204
|
metadata: {}
|
191
|
-
post_install_message:
|
205
|
+
post_install_message:
|
192
206
|
rdoc_options: []
|
193
207
|
require_paths:
|
194
208
|
- lib
|
@@ -203,8 +217,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
203
217
|
- !ruby/object:Gem::Version
|
204
218
|
version: '0'
|
205
219
|
requirements: []
|
206
|
-
rubygems_version: 3.
|
207
|
-
signing_key:
|
220
|
+
rubygems_version: 3.5.11
|
221
|
+
signing_key:
|
208
222
|
specification_version: 4
|
209
223
|
summary: Adds recurring jobs for Sidekiq
|
210
224
|
test_files: []
|